GSoC - Rewriting and in-browser testing of BD rendering

Posted on July 5, 2016 by Christian Fischer
Tags: gsoc

The last few weeks I’ve worked on rewriting the drawing/rendering code in BioDalliance. The reason for this is that there have been reasons to add functionality to the renderer, but this has been difficult due to the way the drawing code was integrated with the rest of BD - that is, in a deep and complex way.

The first feature I added was support for horizontal rulers in tracks, see the red and gray ones for significant and suggestive LRS as in this plot. This was fairly easy to add, after digging through the respective parts of the code in BD. I extended the functionality somewhat by supporting arbitrary drawing callbacks, which were called after the main rendering of the track was complete. However, this was not enough for all the functionality that I wanted to add.

The main feature I wanted to add was the possibility of drawing multiple tiers in the same track - as seen in the plot linked above. My initial attempt at adding this was to write a new tier, which would contain and draw several tiers by combining their fetched data. However, this went poorly, as it proved difficult to get the fetching of data to work properly. After spending a fair bit of time on it, I abandoned this strategy.

Instead, I decided rewrite the drawing and rendering functions, to make it easier to modify them. This also proved to be difficult, as the corresponding parts of the BD code are complex and have clearly grown organically over the years, making them quite entangled with both each other, as well as other parts of BD. Still, this felt doable, and I’ve made progress.

To ease the writing of all this code, I decided to use ECMAScript 6, that is, the latest specification of Javascript, rather than the version commonly in use today. The ES6 code is transpiled down to Javascript, and the result is fully compatible with both the rest of the BD code and the major browsers supported by BD.

ES6 adds a lot of features to the language, a good overview can be found here.

The two I’ve used the most, and found very helpful, are the ‘let’ keyword, and arrow functions. The ‘let’ keyword can be used instead of ‘var’ when declaring variables, and is block scoped, which ‘var’ isn’t. Names defined with ‘let’ cannot be redefined, either, nor can they be used before they are defined. I find that this makes the scope much easier to reason about, and reduces the mental overhead required to understand what’s going on.

Arrow functions fulfill a similar role. They are a new way of defining anonymous functions, which is both a bit easier to write, but, mainly, arrow functions don’t redefine the ‘this’ keyword. Meaning if you write ‘this’ in an arrow function, you get the object that the arrow function is defined in, rather than the arrow function itself, as is the case with anonymous functions defined using the ‘function’ keyword.

Aside from those, the built in support for modules is nice, and also works perfectly with require(), which is what is used for modules in the rest of the BioDalliance code.

At this point I also added ESLint to the equation, which as the name implies is a linting tool for JS and ES. It’s not much, but it gives at least some static code analysis, which is helpful.

With these new tools in hand, I set about rewriting the rendering code.

I’d already located the parts where the rendering was being done, now I had to find the code that called those parts. This turned out to be tougher than expected. Most of the drawing was being called from a function called defaultTierRenderer(), which in turn called a draw() function on the tier that was to be drawn. However, I also found places where the draw() function was being called directly. There was also a chain of five or so functions that actually fetched the data to be displayed, which sent the rendering callback from one to the next. The main problem I had with that is that I wanted different tracks to have different renderers.

To consolidate all this, I decided on a simple interface to be shared by renderers. A renderer is a module exporting at least two functions, renderTier() and drawTier(). They replace the defaultTierRenderer() and tier.draw() functions, respectively. A browser has a default renderer which is used by all tiers, however tiers can override this by providing their own renderer. The callback chain, defaultTierRenderer() and tier.draw() were all replaced by calls to the tier’s respective functions. At this point all seemed to work great. It was time to start actually rewriting the renderer.

To begin with, the only changes I made were replacing ‘var’ with ‘let’ and removing unused variables, as reported by ESLint. The second step was to rewrite the many loops used. I’ve done my best to replace them with calls to forEach(), and using arrow functions. I have also tried to give longer, more descriptive names to the variables used, mainly to improve my understanding of what is going on.

Some examples of what I’ve changed in the rendering code follows.

In cbrowser.js

Before

Browser.prototype.refreshTier = function(tier, tierCallback) {
    tierCallback = tierCallback || defaultTierRenderer;
    if (this.knownSpace) {
        this.knownSpace.invalidate(tier, tierCallback);
    }
}

After:

Browser.prototype.refreshTier = function(tier, tierCallback) {
    var renderer = this.getTierRenderer(tier);
    var renderCallback = tierCallback || renderer.renderTier;
    if (this.knownSpace) {
        this.knownSpace.invalidate(tier, renderCallback);
    }
}
Set of renderers
this.renderers =
    { 'default': DefaultRenderer,
      'dummy': DummyRenderer,
      'multi': MultiRenderer,
      'old': OldRenderer,
      'test': TestRenderer
    };

this.defaultRenderer = DefaultRenderer;

The renderer can be set for the whole browser or tier-wise, by giving the respective configuration object the ‘renderer’ property, with the value being one of the keys in the ‘renderers’ object above.

KSpace

Didn’t change much here, other than removing the tierCallback parameter from many functions. As far as I can tell, it was always the renderer callback anyway - now the appropriate renderer for the given tier is used. #### Before

KnownSpace.prototype.retrieveFeatures = function(tiers, chr, min, max, scale, tierCallback) {
...
    this.startFetchesForTiers(tiers, tierCallback);
...
}

KnownSpace.prototype.startFetchesForTiers = function(tiers, tierCallback) {
...
    tierCallback(ex, tier);
...
}

After

KnownSpace.prototype.retrieveFeatures = function(tiers, chr, min, max, scale) {
...
    this.startFetchesForTiers(tiers);
...
}

KnownSpace.prototype.startFetchesForTiers = function(tiers) {
...
    var tierRenderer = tiers[t].browser.getTierRenderer(tiers[t]);
    tierRenderer.renderTier(ex, tier);
...
}

Rendering loops

A couple of examples of rewritten loops from the rendering functions. As you can see, the biggest differences are replacing var with let, as well as making use of forEach whenever possible, together with arrow functions.

Before

for (var uft in tier.ungroupedFeatures) {
    var ufl = tier.ungroupedFeatures[uft];

    for (var pgid = 0; pgid < ufl.length; ++pgid) {
        var f = ufl[pgid];
        if (f.parts) {
            continue;
        }

        var style = tier.styleForFeature(f);
        if (!style)
            continue;

        if (style.glyph == 'LINEPLOT') {
            pusho(gbsFeatures, style.id, f);
            gbsStyles[style.id] = style;
        } else {
            var g = glyphForFeature(f, 0, style, tier);
            if (g)
                glyphs.push(g);
        }
    }
}

After

for (let uft in tier.ungroupedFeatures) {
    let ufl = tier.ungroupedFeatures[uft];
    ufl.forEach(f => {
        let style = tier.styleForFeature(f);

        if (f.parts || !style)
            return;

        if (style.glyph === 'LINEPLOT') {
            pusho(gbsFeatures, style.id, f);
            gbsStyles[style.id] = style;
        } else {
            let glyph = glyphForFeature(canvas, f, 0, style, tier);
            if (glyph)
                glyphs.push(glyph);
        }
    });
}

This second example threw me for a loop, if you pardon the pun, and it took a bit before I understood what was going on. I didn’t even know that Javascript had labels!

Before

GLYPH_LOOP:
  for (var i = 0; i < glyphs.length; ++i) {
      var g = glyphs[i];
      if (g.bump) {
          hasBumpedFeatures = true;
      }
      if (g.bump && (tier.bumped || tier.dasSource.collapseSuperGroups)) {
          for (var sti = 0; sti < bumpedSTs.length;  ++sti) {
              var st = bumpedSTs[sti];
              if (st.hasSpaceFor(g)) {
                  st.add(g);
                  continue GLYPH_LOOP;
              }
          }
          if (bumpedSTs.length >= subtierMax) {
              subtiersExceeded = true;
          } else {
              var st = new SubTier();
              st.add(g);
              bumpedSTs.push(st);
          }
      } else {
          unbumpedST.add(g);
      }
}

After

glyphs.forEach(glyph => {
    if (glyph.bump &&
        tier.bumped ||
        tier.dasSource.collapseSuperGroups) {

        let glyphTier = bumpedSTs.find(st => st.hasSpaceFor(glyph));

        if (glyphTier) {
            glyphTier.add(glyph);
        } else if (bumpedSTs.length >= subtierMax) {
            subtiersExceeded = true;
        } else {
            let subtier = new SubTier();
            subtier.add(glyph);
            bumpedSTs.push(subtier);
        }
    } else {
        unbumpedST.add(glyph);
    }
});

This all seemed to go well - the browser rendered as it should, at least as far as I could see. Of course “seemed to go well” is not strict enough when writing code, and as I began to make more drastic changes to the code, it was time to add tests.

The largest parts of the rendering and drawing code do not actually draw anything, but instead transform data in preparation of drawing it to the canvas. These are also the most complex parts, and the most important to test, as it would otherwise be quite difficult to get the new implementation to work as it should.

To test the new renderer I wrote another rendering module, which calls the old and new rendering functions on the same tier data, and then compares the results. Currently it does so by recursively checking the contents of the tiers’ subtier property, as that appears to be the most important one, looking at the function that’s currently being tested. It will likely have to be extended in the future, but the basic functionality is there.

Here oldDrawFeatureTier is the old drawFeatureTier function, from the original BD code. This test tests that function only, for now. The subtiers are saved by serialization to and from JSON - clunky, but it works as a way to copy objects in JS.

oldDrawFeatureTier(oldTier);
if (oldTier.subtiers) {
    oldStAfter = JSON.parse(JSON.stringify(oldTier.subtiers));
} else {
    oldStAfter = oldTier.subtiers;
}

The drawFeatureTier in the new renderer is called in the same way, with the subtiers serialized and saved. The subtiers are then compared, using a function I wrote to recursively compare the properties/elements of objects/arrays.

function compareObjects(o1, o2, depth=0) {
    if (typeof(o1) !== typeof(o2)) {
        console.log("type mismatch!");
        return false;
    } else {
        if (o1 === null || o1 === undefined && o1 === o2) {
            console.log("objects null or undefined");
        }
        if (o1 instanceof Array &&
            o2 instanceof Array) {
            console.log("comparing arrays");
            if (o1.length === o2.length) {
                for (let i = 0; i < o1.length; i++) {
                    console.log("recursing on element #" + i);
                    if (!compareObjects(o1[i], o2[i], depth+1))
                        break;
                }
            } else {
                console.log("Arrays of different length!");
            }
        } else if (typeof(o1) === "object" && o1 && o2) {
            console.log("comparing objects");
            if (Object.keys(o1).length === Object.keys(o2).length ||
                Object.keys(o1).every(k => k in o2)) {
                console.log("keys equal, comparing values");
                for (let k in o1) {
                    console.log("recursing on key " + k);
                    if (!compareObjects(o1[k], o2[k], depth+1))
                        break;
                }

            } else {
                console.log("Objects have different keys:");
            }

        } else {
            return o1 === o2;
        }
    }
}

The actual function has a few more prints to help find where the issue is. It may also be of note that the function stops whenever it finds something that doesn’t match.

As you can see, the testing code currently does not use any testing framework. Instead, it’s just a web page that’s to be opened in a web browser, with the output being in the browser console. Very simple, but it works. Configuring additional tests - such as using more or different tiers - is trivial, as it only consists of changing the browser configuration, just like when configuring BD normally.