Development Documentation

In addition to version control (which we've got) and bug tracking, which of course we're making, the third most common tool used when developing large projects is some sort of easy to maintain documentation system - usually a wiki.  Therefore I have decided to set up a wiki to maintain a high level overview of the development of Abundant.  I think this would be a very good idea for all of our projects, as maintaining documentation as we develop our code will help improve the quality of our work.

I have decided to use TiddlyWiki for Abundant.  There are several reasons:

• Easy to set up
  The entire wiki consists of one HTML file and one JAR, there is no installation or system requirements.  Furthermore, setup and configuration is a breeze, with less than half an hour's work I had customized my TiddlyWiki to a desirable structure for software development.
• Compatible with version control
  The wiki is tracked by Mercurial and located right next to the source code.  This means this documentation always goes with the code, and tracking and merging changes works just like any other file.
• Lightweight bug tracker
  Although Abundant has far loftier goals than TiddlyWiki can ultimately fulfill, it will make a perfectly suitable issue tracker until Abundant is ready for dogfooding.  Similarly, this is one of the biggest reasons I think others should look into using TiddlyWiki or some other documentation tool, as it will provide for some very nice issue tracking as they develop their project.

Additionally, since TiddlyWiki is just an HTML file, you can view and browse it straight from the source code.  If you are interested in playing with TiddlyWiki as a development documenter, you may like to download the first version of the wiki which is structured with development in mind and an issue tracker page created.

Note: Presently, the links to the Abundant repository are password protected.  Anyone reading this blog is welcome to come talk to me and I'll tell you the password, but otherwise it will be publicly accessible soon enough.

Data Storage

One of the fundamental challenges of Abundant is storing bug data.  There are several conflicting challenges that must all be met in order to store data in ways that address all needs.

The data format must:

Be Human Readable
In order to best integrate with version control, we want the data format to be text based (as opposed to in some sort of database) so that users can see the bugs and track changes directly in the version control system.  Furthermore, version control works best with text content, as it is easier to compare changes automatically.

Be Able to Store Metadata
A lesson learned from developing b is that metadata (assigned-to, bug status, etc.) needs to stick with the actual data like issue description and comments.  In b metadata was stored separately in order to ease caching, but I believe it makes more sense to keep cached data for speed separate from all the actual data.  The data structure therefore needs to store structured data that is machine readable, at the same time as remaining human readable.

Be Fast to Parse
We want this to be scalable and that means fast access to any bug.  This can largely be done with untracked cache files, but assuming each bug is stored in its own file, each file should be fast to display without caching.  Listing, browsing, and filtering can be assisted with some sort of caching or indexing mechanism that will remain invisible to the user.


These factors affect what formats we can use, some options include:

Plain Text / Custom Format
A plain text file is the most human readable, but the least computer parseable.  This what was used for b, text in the file was split into sections denoted by titles square brackets, and any section that wasn't empty was displayed, and metadata was stored in a separate file.

XML / JSON / Other Standard Data Format
A structured text file which will contain both user content and metadata in one file per bug.  The primary advantage is this will make working with the data very easy, as there exist powerful parsers for standard data formats in both Python and most other languages.  This seems like a better option than a custom format.

A potential limitation of this method (and to some extend the one above) is in how Mercurial stores changes.  More details can be found in this email thread.

A File-based Database
Another alternative is using a database system that works well with version control.  This might be somewhat ideal, as it would (presumably) work flawlessly with Mercurial's change tracking, and also be efficient.  The problem with this method is I don't know of such a tool.  It seems fairly likely to me that such a thing exists, but I don't know of it.


These are all the challenges and options that come to mind, what are your thoughts?

Proposal

Well, I have done a truly terrible job keeping this blog up to date over the last semester, but I've done a lot of work and finalized what I will be working on, so I give to you my proposal for Abundant, a distributed issue tracker!

The Proposal:


The Presentation - Scribd seems to not like the font I used for some reason, click the 'Download' link below if it looks bad:


Unfortunately, apparently Google doesn't support uploading anything other than images and videos to blogger, so the links above point to Scribd and potentially could break in the future.  Hopefully not.

System.out.println("Hello world");

Hello all!

This blog is intended to be a place for me to chronicle the work I will be doing over the next school year developing my senior project in the Computer Science department at Willamette University. With apologies to non-technical readers, this blog is likely to be somewhat dense and is not intended for a general public readership.

At this point, I am exploring different potential ideas for what I could be working on. I have two ideas at this point, neither of which are fully formed, but both would be enjoyable, interesting, and challenging.

Developing an Operating System and Multi-Threaded Virtual Machine

My first idea, which I have had since sophomore year when I took CS353 and developed the PC231, a simple virtual machine to develop assembly code on, is to expand upon the PC231 in the following ways:

• Improve the architecture (larger word size, two's complement, more hardware commands, etc.) - potentially/probably trying to replicate a more common architecture
• Develop a limited kernel to handle memory management and execution control
• Develop a rudimentary operating system to allow the user to execute arbitrary commands
• Develop a virtual file system to integrate with the virtual machine

This would be a fairly new field for me - outside of CS353 I have never done any kind of assembly code work or low-level programming. Exploring this area would be interesting, however my limited experience and knowledge makes it difficult to predict if this is a reasonable goal, and if the amount of work is feasible.

Develop A Lightweight Issue Tracker for Use With Distributed Revision Control Systems

Over the summer, I developed an extension for Mercurial, a distributed revision control system. This extension is a lightweight, command-line bug tracker called b. I would like to develop this into a free standing tool which can be used anywhere, with few system requirements, no setup or configuration necessary, and a minimal footprint. Specifically, I intend to do the following:

• Remove Mercurial dependency and turn b into a standalone program
• Enable it to integrate cleanly with the major distributed version control tools (Mercurial, Git, Bazaar)
• Develop an internal web server to provide a web interface to view and manage issues
• Develop plugins for major IDEs (Eclipse, others if time permits)
• Develop a graphical front-end for non-command line use

The different facets of this project would allow me to explore many fields at once, ranging from Python programming to web development to software management concepts. This would be an interesting and valuable project, both personally - because I would love to use such a tool - and potentially others would find it useful as well.

And these are my two ideas at this point. Comments, questions, and alternative ideas are welcome!