CodeSpeed2: First Graphs

First Graphs

In week 2 of CodeSpeed2, the rewrite of CodeSpeed for pypy and the PSF, we got our first graphs up.

Malte & Matt just having coaxed a benchmark run into the first bar chart

By the end of the week, we had the first cut on three graphs.

Revision timings

Normalised data

History graph

These are being drawn from bench mark runner output data submitted via the Cyclone endpoint, persisted to MongoDB, fetched from the API, represented in an Angular.js object and rendered with d3.js.

API

The API is served up via Cylcone and so far we have the endpoints below.

Upload

/api/raw_benchmark_runner_baseline_output
/api/raw_benchmark_runner_output

Retrieval

/api/c_python_normalized_benchmarks
/api/benchmark_history
/api/benchmark_result

For usage details, tests are in /src/test in the repository bitbucket.org/pypy/codespeed2.

Taxonomy

Also in this week we had some meaningful discussion about what terms mean in this context.

Best summarised in the following sentence:

The runner runs Benchmarks (which are Modules e.g. ai.py) and produces Benchmark_Runner_Output data which contains [one or more] BenchMark_Results

A key observation: the system under test is pypy and the modules (e.g. ai) are the benchmarks being run against that.

Up next: User Experience / Flow

In the next week we'll be crafting a first take on how users select and navigate data. D3 gives us some interesting options for hyper-linking in the SVG itself.

Read More

CodeSpeed2: Inception

What're we doing in the CodeSpeed re-write? Well, to get us all on the bus, in our first iteration we started off the first day with an inception deck.

The Inception Deck

As discussed in Jonathan Rasmusson's Agile Samurai:

Many projects get killed before they even get out of the starting blocks.
This is mostly because of the following reasons:

1. They fail to ask the right questions.
2. They don’t have the courage to ask the tough ones.

So, seeing as we're a courageous bunch and wanted to avoid the fail, we started with an inception deck.

Why are we here?

This project is funded by the PSF and donated time from Information Logistics and is beeing overseen by Maciej Fijalkowski.
The answer we came up with to the question "why are we here?" was:

Because PSF likes us enough to rewrite codespeed
So that: it has tests on python code and javascript that let you do TDD
So that: Maciej or any interested dev can easily test and implement a feature in an afternoon... and know the answer to: "Does it work?"

Meet the Neighbours

IL Team: Matt, Malte, Dave

PyPy devs: Maciej, Alex, ...

PSF: in particular: Jessie, Brett CPython test

Greater Community:
Twisted: JP Calderon,
Apple: Glyph,
Linaro,
ARM/Raspberry,
Calxeda
CTPUG: Simon & DavidM

Show the solution

"munch munch munch" -- fijall

Technologies we chose to get started:

• cyclone io
• mongo db
• angular.js
• d3.js

What's going to give?

We'd rather deliver features more slowly and have tests, quality is very important. We are building dev tools here.

The product can be extended after we're done with this phase.

What's it going to take?

We have 6 weeks

1: API serving, receiving uploads
2: First graphs
3: ++
4-6: Smart search, statistics

What Keeps you up at night?

Not much really, Excited.

Elevator Pitch

For maciej, codespeed2 is a benchmark run visualiser that lets you understand performance.

Unlike codespeed it is extensible.

And, we're off!

With that quick summary done, we all headed off into the first week.

In the first week: we got the backend up and running, accepting all the different formats from the runner and persisting to MongoDB. Check out bitbucket.org/pypy/codespeed2 for the blow-by-blow.

Next post: week 2 update.

Read More