Nicholas Haddock

Miserable earthworms!

Tetrane historically is an R&D company: we try things out, implement new ideas, improve our software incrementally based on customer's input or on our own perception of where the technology can be pushed. But at some point your cool, bleeding-edge software becomes a product, and you face the usual questions: when will we ship? What confidence do we have in our roadmap? What are the priorities?

As such, Tetrane is now partially shifting its development process to that of an actual software shop. I say partially because we will keep a balance between a down-to-earth, software-shop-like hard-working model and the more relaxed, creative, risky, will-produce-unexpected-things, plain fun one that is fundamental to Tetrane's identity and strength.

The needs

More specifically, we had decided a while back to meet a rather precise deadline. We already had defined a rather clear (and ambitious!) set of tasks and priorities, so during the 8-week-long development run that ensued, we faced rather obvious questions:

A haddock fish

This looks fishy

  • What is the global amount of work still ahead of us?
  • How can we best distribute the work chunks amongst us?
  • Are there tasks that take more or less time than expected?
  • Are there hidden tasks that eat up time?
  • How can we take unrelated or unexpected events (sickness, client meetings) into account?

There are directions we decided not to take:

  • we didn't try to go the Agile way, because we had no previous experience with it, and our tasks were already set anyway.
  • we didn't simply set up a Gantt diagram, because tools at our disposal felt way to clunky to use to be practical in the very short term. Besides, there was little dependency between the various tasks.

Instead, we figured it would be pretty straightforward to connect to our redmine database, extract the relevant data and compute graphs to get a visual overview. This DIY approach would also mean learning exactly what we are dealing with in the process, what the right metrics are and their limits.

That is how haddock came to life. I can't quite remember why we picked that name, though.

The captain's toolbox

To extract the raw data we used the python-redmine library: it exposes the REST API from your redmine server, and can be used like this:

# Connect to redmine API
red_handle = Redmine('https://your-server/redmine/', key='*****',
              requests={'verify': False}) # self-signed certificate

# Get base objects
red_project = obj.red_handle.project.get("My project")

for version in obj.red_project.versions:
    if version.name == "Target Version":
        red_version = version.id

issues = red_handle.issue.filter(project_id=red_project.identifier,
                                  version_id=red_version, status_id="*")

# Extract required data from issue
for i in issues:
    issue = translate_issue(i)

I won't go into details because this is very specific to what we need, but it is really quite straightforward to use.

The next thing we needed was a way to display graphs easily. For a quick PoC I found pygal to be a nice-looking and easy-to-use alternative to plotly. Here is an example of code that creates a single line of stacked values:

# Build a unique line graph with global times

style = Style()
style.colors = [unparse_color(0, 200, 200, 1, 'rgb'),
                #[...]
                unparse_color(200, 200, 0, 0, 'rgb')]

pygal_graph = pygal.HorizontalStackedBar(style=style, height=160, width=1600)

pygal_graph.add("Spent on v1", [int(days_loggued_v1)])
#[...]
pygal_graph.add("Max time left", [int(days_margin)])
pygal_graph.title = 'Current project status (in man-days)'

Although in the end, I needed a different kind of flexibility than what is provided by regular plotting libraries: I ended up cooking a custom stackable horizontal bar class in PyQt4, and converted the previous code into the following:

# Build a unique line graph with global times

def build_graph(self):
    self.entry_list = []

    # Add data
    self.add_to_list(QColor(0, 200, 200), days_loggued_v1, "Spent",
                     "%d man-days loggued on the project")
     #[...]
    self.add_to_list(QColor(200, 200, 0), days_margin, "Margin",
                     "%d man-days available until shipping")
    self.add_stacked_bars(self.entry_list)

    # Add time markers
    self.add_date(self._project.days_since_beginning, 'Today')
    self.add_date(self._project.total_estimated_days, 'End of estimated tasks')

    self.add_background_area(0-margin, self.bar_max+margin*2, 0-margin/3, 1+margin*2/3,
                             "Project time consumption in man-days", Qt.white)

The actual views

The global view

Battle of Solebay - Sir Richard Haddock

Ten thousand thundering typhoons!

The first information we desperatly needed was a view of the globally estimated amount of work. I specifically wanted a plain, clear, colored view of the project status so we could all get a sense of the project's health: when you are knee-deep in your work, it can be hard to take the time to step back and think on a project's level – which could lead to disaster. This translates into the following time metrics, in man-hours:

  • Time actually spent on the project
  • Time spent on other tasks (or time off)
  • Time left, according to the estimations
  • Margin (which at the time knowingly included the planned test phase – there was no way to plan something ahead of time)

Which ended being something like the following pseudo-code:

# Parse the time log and count hours spent on project or on other tasks
for user_log_entries in time_entries_per_user:
    for entry in user_log_entries:
        if entry['task_id'] in all_project_issues:
            project_past_spent_hours += entry['hours']
        else:
            other_past_spent_hours += entry['hours']

# Aggregate tasks estimations
for task in roadmap_issues:
        project_estimated_hours += task['est_spent_time']

# Get logical info from that
total_past_spent_hours = days_since_beginning*team_count*7.0
past_unloggued_hours = total_past_spent_hours - other_past_spent_hours - project_past_spent_hours
total_estimated_hours = project_estimated_hours + other_past_spent_hours
total_estimated_hours_left = total_estimated_hours - total_past_spent_hours
margin_estimated_hours = days_for_project*team_count*7.0 - total_estimated_hours

Believe me or not, it took several iterations to get both the metrics and their corresponding code right, but in the process I gained confidence in the concepts I was handling – which was the whole point of the exercise. It finally gave the following global view of our project status.

Global project view

The global view of our tool

The tasks view

Now that I was assured that our plan wasn't going straight into a wall, it was time to go a little further. The next thing I needed was a detailed view of the tasks at hand:

  • One bar per task with a percentage of time consumption
  • Sorted by assignee when planned
  • Unassigned tasks set apart
  • The ability to move the tasks around and experiment with distribution.

You might note that such a view would be unusable with a high number of tasks. We had to regroup tickets under bigger "roadmap tasks", which are easier to reason about anyway; this view will only work on those.

PyQt's canvas made the displaying easy, and again the result was quite straightforward:

Task distribution view

Task distribution

Note that finished tasks appear as bright green, so as to empahize on the positive feeling of marking something as done.

The timelog view

This is perhaps the most controversial part of Haddock. I wanted a simple way for everyone to know what others were working on; I also wanted to give an incentive to keep daily time logs up-to-date, so that the time estimations could be correctly updated (especially when spending significant time on tasks outside the project's scope). I decided to build a comprehensive time log from the redmine data:

Project time log

The time log view

Each line represents a team member, and each column a day. Each day is divided in 7 blocks. You can see the current day with a slightly larger border. On the right, we stacked the remaining time of each task that is assigned to that team member.

I feared it might come as a bit too intrusive but people seem to have accepted it pretty well, and it actually serves its purpose:

  • The view is quite clear, and makes it as cool as possible to track your time (which isn't much, but still)
  • The black holes remind you to log your time
  • It makes log errors easy to spot and fix
  • The time spent on unrelated tasks stands out as it should: see the 13th of february.
  • This is currenly the only way to see if a team member is overloaded to the point of missing the deadline.

We ended up using it during our weekly monday-morning briefing as a good reminder of what had happened the week before.

What is to come

This is as far as I could go on the time we budgeted for that tool, but it served its purpose quite well: we were able to keep an eye on the project's status.

There are various things that we felt were missing though, which we may implement at some point:

  • It would have been cool to be able to log time directly from inside the tool.
  • The task view isn't the best way to represents tasks on the long run: we need a way to see the remaining time left. Currently this information is only shown in the time log view, but the latter is too detailed for that purpose.
  • Tasks with a defined starting date must be taken into account, so we can plan holidays or other things ahead.
  • The model doesn't work well with a huge tasks that is to be divided between members: you can see the "Various boat repairs" task above: it included a lot of small tickets that people worked on from time to time, and it didn't reflect well in the task view.

What about you? Did you build specific time management tools tailored to your needs? Or did you find a tool that matched your workflow? Share with us in the comments!


Comments

comments powered by Disqus