←  Back to blog

Agile Documentation for Modern Development


Written by Geoffrey Chan

May 28, 2015

Documenting a project is often an afterthought. Dreaded and neglected, the process sluggishly commences after most development is complete. It’s a colossal task that regularly concludes with an inadequate resource that dissuades rather than encourages other developers from using your code.

A Multifaceted Problem

This documentation conundrum, as we see it, can be defined as follows:

  • A large development to documentation gap leads to incomplete coverage
  • Documenting at the end of numerous development cycles becomes an overwhelming and monotonous task producing slapdash results
  • Coordination across developers is crucial to collaborate on functional overlap, ensure documentation consistency, and enable everyone to be involved – yet it is not addressed effectively by the tools we found
  • Agile development practices promote smaller, quicker releases – and while many software solutions support agile development, agile documentation is effectively ignored
  • The processes and results from internal documentation are generally different from the processes and results of public documentation – a scenario that discourages thorough effort when documenting

We faced this challenge by turning to some of the many documentation tools out there – and there are fantastic ones. But we still struggled with the fundamental problem of process. Without a toolset that helped integrate documenting into our day-to-day development routine, we were left with all of the above problems. So, we did things our way.

The Fix

Accompanying our 1.1 release of Fuse, we completely overhauled our documentation process to make sure it’s as agile as our development process. We think our solution is pretty good for collaborating on and maintaining up-to-date documentation. The following mantra was our guide:

Developers write the best documentation – use familiar tools that fit into your development process.

This mantra led us to use a minimalist toolbox of Sphinx and git along with our own bit to keep things seamless. The below diagram provides an overview of this process from end-to-end:

Documentation Workflow

Sphinx

Sphinx is an exceptionally hackable static site generator that is well loved by the open source Python community. Sphinx relies on reStructuredText, an expressive markup language that is highly readable and logical. This markup is then rendered from standard and custom directives into publishable HTML output. Styles are applied separately.

Sphinx does two things really well for us:

  1. Separation of concern – an expressive yet writable format that requires no design input
  2. Custom rendering – an extensible format that lets us define custom markup and output

Sphinx does the heavy lifting and lets our developers use a friendly format that requires no design forethought. For example, the following block of documentation:

Screen Shot 2015-05-01 at 1.29.08 PM

Is actually generated by the following reStructuredText:

Query Content by ID
===================

.. get:: /contents/(key)

    Return content details about a specific result as a :js:data:`content`
    object.

    :qparam q: The search query (only used for highlighting).
    :qparam boolean highlight: Turn on/off highlighting of search terms.
    :qparam list attributes: Selected attributes to return in content
        results.

This takes us a long way in completely de-coupling style and content when writing documentation – and since Sphinx is infinitely extensible, we make the code fit our needs. Our designers worry about styles in a different work thread. This helps us get out updates to documentation with as few roadblocks as possible.

git

If there’s any universally loved software by developers of all disciplines, it’s git. Using git to version control our documentation enables our developers full access to the workflows and tools that they know best.

Git provides us with collaboration and version management that easily fits into our existing development process. We branch documentation additions and open pull requests so the team can review, ensure consistency, and provide feedback. We tag our documentation to match our software version. If disaster strikes, we have all of the tools in git’s repertoire to get us back into an operable state.

To break it down, git enables:

  1. Organization – branches, commits, and tags keep us well oiled
  2. Team collaboration – we all pitch in to make sure it’s great
  3. Version management – git is a distributed version control system after all
  4. Proven workflows – git has proven standards and conventions that developers love

Here it is in action:

Pull Request Example

Putting it all Together

Git and Sphinx bring us a long way to solving the documentation conundrum. With a light application layer, we’re able to orchestrate these tools and fill in the gaps to create a seamless, agile, and awesome process.

  1. We defined conventions to be used with reStructuredText along with custom directives in Sphinx to output the desired HTML
  2. Our designers and front end developers went to town and created a beautiful and functional experience using this output
  3. A dedicated git repository houses all of the reStructuredText documentation files
  4. Listeners, using git hooks, trigger Sphinx builds on our dev site whenever a commit is pushed to this repo
  5. Our process declares that all additions or changes to documentation be branched and issued pull requests before merging into master
  6. Collaboration ensues, internal documentation is available on our dev server, and this process repeats until a new release is published
  7. We tag that moment in history with the new release and issue a command to our application that uses Sphinx and git to rebuild all versions of our documentation on our staging server
  8. Once we’re comfortable, production deployment is a simple command that also backs up the current version for quick rollbacks
  9. Rinse and repeat, this process is completely integrated with our development cycles

Now, our documentation gets the same love (or at least close to) as the rest of our software development process. With the abundance of tools out there, this process will continue to evolve and improve. We’re already planning to add documentation to our continuous integration servers so everyone will know if something breaks and we can automate most of the process enforcement.

Interested in implementing something like this at your organization? Drop us a line and we’ll be more than happy to share our work. In the coming months, you may even see an open source project from us. Oh, and make sure to check out our new documentation here.