Philadelphia Software Developer

A year or two ago, I saw a Microsoft’s director deliver the keynote at a conference. He claimed at the time that Microsoft had entered a new era of supporting open source development, which raised a few eyebrows considering their history. This was at a Solr conference, which made me think it’d be interesting to see if you could build a search engine to prove out these claims.

Microsoft’s position is not one of altruism; they freely admit that this is borne of a desire to make various systems work well on Azure, which you can see from their open source marketing materials, where they have developed a sudden interest in PHP, Node.JS, Python, etc.

Microsoft started building Azure in 2008, making it commercially available in 2010, and in 2012 they released an update which allowed for Node.js and PHP.

I built a tool to prove this out, searching the history of several popular open source repositories, including the ones Microsoft mentions specifically. You can see here that Microsoft had a number of commits from 2010-2012, with the peak in 2011. This is likely more of a tactical play on their part than a long-term strategy, but better than some alternatives they’ve pursed in the past.

The numbers above are counts of commits against the following Github repositories with an @microsoft.com email address (so possibly there are some left out, for instance if someone emailed in a patch).

The search repository includes the following projects: Drupal, Git, Lucene, Lucene.NET, Solr, Mono, Node.JS, PHP, Postgres, and Vagrant. Collectively this represents 2.1 GB Git History, including 232,839 commits – this compresses to 132 MB when indexed in Solr.

The indexing process is quite simple, using a simple Java class to copy data from Git to Solr, which you can also use to “>find who are the senior developers on various projects:

Inferring a commit author’s company can be trivially accomplished from an email address:

// x.y@google.com => google.com
String company = emailAddress.split("@")[1];
 
if (company.contains("."))
{	
  // abc.com => google
  company = company.substring(0, company.lastIndexOf("."));
}
 
return company;

In the end what this shows is not just that Microsoft has made some overtures to the rest of the software world, but also that you can readily combine some journalistic research processes with coding to produce interesting results – any activity which occurs in public and on a regular basis can be queried to seek answers, given the will and the patience to look, combined with some basic technical knowledge.

If you want to look into this further, you may find the following resources of interest:

• Copying data from Git to Solr [github page]
• Mining Source Code Repositories with Solr[talk]
• Full Text Indexing PDFs in Javascript

I went to see a great talk by Ryan Joy (MSFT) and Keith Casey (Twilio), at Refresh Austin, on what developer evangelists do. I found this surprisingly interesting; this combination of programming and marketing knowledge is a fairly rare and high bar for a developer. Typically the people found for these positions are already active in the community in some way, either people who already exhibit leadership by volunteering a lot of time working with a product (writing blog posts, giving talks, etc), or people who have a lot of connections. For instance, in Microsoft case, the goal was to have access to a lot of developers not already in the Microsoft ecosystem.

These jobs involve a lot of travel, and have high burnout rates- Twilio, for instance cover ~60 events / year / person. Consequently they recommend hiring people near airport hubs, to avoid layovers. Once hiring, there are different models for this- clearly it doesn’t make sense to hire a developer evangelist in the city where you are headquartered, but the size of the region a person covers can vary widely from a single tech city like Austin to an entire state, like Texas. In addition to a lot of travel, the actual work ranges from holding events, light customer support, to dealing with a lot of internal and external politics.

In compensation for these challenges, the position of developer evangelist gives the person with the job a lot of new connections they might not already have, since they are working in a cross-department way (e.g. marketing + product teams). For instance, in sponsoring an event, a developer evangelist might work with multiple departments to provide sponsorships. Interestingly a lot of developer evangelists know each other – their typical goal is to make meetups and conferences successful, which sometimes means referring conference organizers to other companies that do sponsorships.

The companies that do have heavy developer marketing programs clearly require a narrow combination of product features; an easy to use API, plus a reasonably short sales cycle. In the case of Twilio, the goal is to help developers make heavy use of their existing APIs, as they are pricing based on usage, so successful developers = money for twilio. These marketing programs do double-duty, getting both new customers and allowing for soft-interviews (e.g.: whoever wins a hackathon has a very visible achievement and likely an online demo).

Developer evangelists are relatively rare, and spend some time working with volunteers. These include groups like conference organizers, students, and professionals looking for exposure for themselves. In many cases what a conference organizer thinks they have to offer doesn’t benefit companies the way you’d think: twilio, for instance, has way more twitter followers than any conference or meetup, so providing them access to your twitter is not that valuable. In that vein, Microsoft doesn’t really need brand exposure (putting their logo on banners). They much prefer to be able to run hackathons or contests, where they can purchase access to interact with developers.

One way they facilitate this is with canned programs; for instance providing demos, videos, and training to developers who want to hold their own events. This way, you can start your own meetup, and often the company you are indirectly promoting can chip in for food.

“Long-term” sales focus for these guys seems to be in the range of months to a year, which makes sense in fast growing start-ups, but I can’t help but feel like the 5-20 year range is important as well, but most of those benefits are reaped by the developer evangelist themselves, rather than their fast-moving employer. The feeling of the guys who gave this talk was that the industry is going to move beyond “hackathons,” but no one knows what the next thing will be.

ETL (“extract, transform, load”) come in many shapes, sizes, and product types, and occur under many names – “data migration” projects, business intelligence software, analytics, reporting, scraping, database upgrades, and so on. I’ve collected some notes, attempting to classify these projects by their principal attributes, so that you can estimate the risks and plan the testing process for similar projects- if you have other additions to this list, please add comments below.

Types of ETL Processes

• Data migrations from an unknown source to a known source
  • Consulting projects
  • Generic migration products (http://wiki.postgresql.org/wiki/Converting_from_other_Databases_to_PostgreSQL)
  • Data from acquisition => existing product
  • Conversion from paper to electronic systems
• Data migrations from unknown, unstructured data into “something useful”
  • Scraped data => Reports (see Ahrefs, MixRank)
  • Scanned data => Reports (see Westlaw)
• Data migrations from known datasets to known destinations
  • Old version of existing product => New product (v1 => v2)
  • Loading data from {cassandra, postgres, solr, mongo …} => {cassandra, postgres, solr, mongo, …}

Risks

• Unknown source data – Highest Risk
  • Risks can be hard to articulate without seeing the source system
  • In theory risks can be mitigated with industry standards
  • Often Better to do by hand for small data
  • Can be difficult / impossible to script
  • Source data usually missing columns
  • Source data usually has data values that don’t match
  • Source and destination data handle gaps differently
  • Great for people who bill by the hour
  • Finite data with a large number of edge cases
  • Performance will likely not degrade over time
  • May add junk data to an existing system and permanently degrade performance
• Medium Risk (Scraped/Scanned Data)
  • Easier to articulate risks, because people understand the nature of scanning, etc
  • May have no metadata
  • Some data is impossible to recover (missing pages, things stored in cabinets)
  • Wide variety of formats (scraped data will vary over time on a site, same for scanned forms)
  • Data volume is typically limited by what humans can generate. Can still be big (e.g. all court cases), but still easier than “all stars” or “all clicks”
  • Volume of support calls may not justify the effort at 95%, but that may be all that’s achievable
  • Infinite number of subtle edge cases (infinite = you will always find more)
  • Performance will degrade over time if the system is in use
• Known to Known system – Medium/Low risk
  • Near real time may be expected but be difficult
  • Risk of data falling behind in dest. system
  • If this is done because of data volume, certain features will not be available to users (e.g. medians? or other more O(x) heavy operations)
  • You control the data, but there is communication risk among developers on a team
  • Subtle edge cases are very visible
  • Some edge cases will be known, but more will develop over time
  • Performance will degrade over time if the system is in use

Testing ETL processes

• Completeness of migrated data
  • Can be fairly easy to test (SELECT count(*) FROM … in each system)
  • Better to migrate slightly more data than needed into the warehouse – helps avoid rework and easier to tell that it’s all there
  • Even if you populate everything to a warehouse you can still cut it down when you push it to the system end users see (e.g. if OLAP)
• Performance of ETL process
  • Near real time?
  • Stop the world vs. incremental
  • Under load (e.g. sharing population of warehouse with some external volume, e.g. used for marketing system)
  • Each of these generates extra test cases
  • Is it worth NoSQL solutions – pain of additional server setup, use of newer/less tested software to spray work out and duplicate everything vs. having control/centralized database/more query features
• Quality of transformation
  • This is never-ending
  • This is harder (i.e. impossible) if it is done once and never revisited
  • Requirements changes must propagate to every corner of the system
  • The end result system that receives the migrated data must handle every idiosyncracy in the source data
  • Fail early and risk stopping often vs. failing late, and risk not finding defects
  • Diff before/after for two implementations
  • Need a thin layer of processing over before/after to coerce into something sane (widen date range, handle nulls, add guaranteed sort order, add subsetting of data, must be rows rather than counts)

In my last article I presented a simple tool for discovering senior developers from source code history, which imports Git history into a search engine. This lets you do searches like “who is our Sharepoint expert,” or “who fixed all our Oracle problems.” There are a few code search products that handle similiar problems, e.g. Krugle, Github, and Fisheye, however they tend to be focused on who made specific changes, rather than finding out who ran a project.

Version control history typically shows file changes using text diffs – in theory we could instead reconstruct a file before and after a change, then compare the parse trees instead. This is impractical in the general case, and if you read on I’ll show how to handle this correctly.

These results can be obtained from commit history, which includes metadata for commits (e.g. the author and commit comments), as well as the actual changes.

Typically full-text search engines (e.g. Solr) use natural language techniques to make text searchable, assuming that the text in question is English. NLP as a discipline is largely a list of language-specific tricks. This is a lossy process, where words are changed or removed, so that “similar” words are actually treated the same:

The dogs ran home => dog run home
The dog runs home => dog run home

This naturally resembles commit messages, but not so much code. In this article I’m going to show how to extract comments (a common request is searching Javadocs), but you could easily use the same technique to pull keywords and other code features.

The following diagram shows the indexing process. Our ETL process receives a sequence of file diffs and processes each in sequence. The API allows you to pull just the lines that changed, or the entire file, so you need to think about this in advance.

Here we have a few test cases for comments. If this was all you wanted, these should be relatively easy to find with a regular expression:

@@ -1,9 +1,9 @@
/* 
* Here is a multi-line comment.
*/
String a = "x"; // here is a comment at the end of a line
 
-String b /* a comment in a line */ = "y"; 
+String b /* a <b>long</b> comment in a line */ = "y"; 
 
// Commented out code:
// String c = "d";

This sample is output from git, and shows a few interesting problems. One is determining whether a line is really English or just commented code. When someone adds a single word to a comment, do we credit them the entire comment (remember, I’m trying to find out who a project lead was)? Even then, we need enough lines of context to find the beginning and end of the comment.

These can be resolved by agreeing on business rules for the product being built; true code parsing is much harder in the general case. The following is a sample from the Documentation for JSX, one of many new languages that compile to Javascript.

var MarkdownEditor = React.createClass({
  render: function() {
    return (
      <div className="MarkdownEditor">
        <h3>Input</h3>
        <div
          className="content"
          dangerouslySetInnerHTML={{
            __html: converter.makeHtml(this.state.value)
          }}
        />
      </div>
    );
  }
});

This has xml inside Javascript (!), which is modified at compile time to do something sane. With all the new languages like this, you’d be hard pressed to build a generalized solution to code parsing, and anyway, in my experience, large software projects all eventually acquire their own domain-specific languages.

Having established that accurate code-parsing is out of the question, we might look at how syntax highlighters work. Most programming blogs have tools to do highlight snippets, which is essentially the same problem, since the highlighter can’t resolve API references, and has to deal with typos. There are numerous IDEs and plugins which support highlighting (e.g ctags in Vim) so clearly this can’t be as hard of a problem as I’ve made it out to be. With some quick searching you’ll find there are at least a half dozen different tools to do this – many are Javascript libraries which like to consume and produce HTML.

Shown here is an example in Node.js, using highlight.js to tokenize and highlight code:

var hljs = require('highlight.js');
 
var fs = require('fs');
fs.readFile('./reveal.js', 'utf8', function read(err, code) {
  if (err) {
    throw err;
  }
 
  var output = hljs.highlightAuto(code).value;
  output = output.replace(/<\/span>/g, '</span>\n');
 
  var spans = new RegExp('<span class="(.+)">(.+)</span>', 'g');
  output.replace(spans, function(match) {
    var spans = new RegExp('<span class="(.+)">(.+)</span>', 'g');
    var values = spans.exec(match);
 
    var type = values[1];
    var string = values[2];
 
    var document = {
      type: type,
      string: string
    };
 
    console.log(JSON.stringify(document));
  });
});

Since the library only gives you HTML, you have to do a little regex trickery to get useful output. As you can see below it gives pretty good results (anything it doesn’t recognize it skips). When you’re populating a search engine’s index, it typically doesn’t matter what order the words go in, unless you search against the same field you display.

{"type":"comment","string":"// Checks if reveal.js has been loaded"}
{"type":"function","string":"function"}
{"type":"params","string":"()"}
{"type":"keyword","string":"return"}
{"type":"comment","string":"// Forward event binding to the reveal DOM element"}
{"type":"function","string":"function"}
{"type":"params","string":"( type, listener, useCapture )"}

In the highlight.js code, we can see this works by applying regexes, like I suggested above:

 this.C_LINE_COMMENT_MODE = {
    className: 'comment',
    begin: '//', end: '$'
  };
  this.C_BLOCK_COMMENT_MODE = {
    className: 'comment',
    begin: '/\\*', end: '\\*/'
  };
  this.HASH_COMMENT_MODE = {
    className: 'comment',
    begin: '#', end: '$'
  };

With this information in hand, you could easily filter to just comments, and push that data into a full-text index (the code here is for Lunr.js, a Javascript library that resembles Lucene).

var index = lunr(function () {
  this.field('commments')
  this.ref('id')
});
 
index.add({
  id: 1,
  comments: '// Checks if reveal.js has been loaded'
});
 
index.add({
  id: 2,
  commit_message: '// Forward event binding to the reveal DOM element',
  commit: ''
});

This works well for building a search engine for developers, but if this were a customer facing project and this was 95% correct, you have to decide if it’s worth the cost of the call volume for the remaining 5%. In a search engine that starts with scraped or scanned data, the support issue is also an inevitability, and must be planned for – any project that does data migration must consider the specific cases it intends to solve.

If you’re interested in this project, check out slides from my talk, “Mining Source Code Repositories with Solr“, or the code for the project on github.

In a software company that does consulting, it’s often valuable for engineers to look at a change and know if it was done for a particular client – for instance, if an API feature does not appear to be used, it may actually be in use by a specific client. That said, the person who made the change may be a junior developer, and you’d prefer to know who the lead for a project was.

Similarly, when someone calls the company, it’s also helpful to know who to route phone calls to – this is often the case when a client doesn’t have a problem for a year or two after project was built, and since the person who picks up the phone could be a marketing or administrative person receiving the call and not someone with knowledge of the project.

I built a small application to do this, with the architecture pictured here:

I found that typically you can find who is the lead on a project, merely based on the number of commits they’ve made referencing various topics, as long as you make good use of the available data. To do these searches we want to use the full-text indexing capacity of Solr to filter down commits. However, this is different from a typical search, where you’re looking for the commits individually, as we’re displaying facet results.

The backend schema for this is fairly simple:

<field name="id" type="string" indexed="true" stored="false"/>
<field name="author" type="string" indexed="true" stored="true"/> 
<field name="company" type="string" indexed="true" stored="true"/> 
<field name="year" type="string" indexed="true" stored="true"/> 
<field name="email" type="string" indexed="true" stored="true"/> 
<field name="message" type="string" indexed="true" stored="true"/> 
<field name="search" type="string" indexed="true" stored="true" />

To populate this database, we need to do some work using JGit, a Java library maintained as part of Eclipse. To traverse source code history, you need to pick a starting point and work backwards – if you were to parse every branch, you would need to de-duplicate older commits.

FileRepositoryBuilder builder = new FileRepositoryBuilder();
Repository repository = 
  builder.setGitDir(new File(path))
    .build();
 
RevWalk walk = new RevWalk(repository);
 
for (Ref ref : repository.getAllRefs().values()) {
  if ("HEAD".equals(ref.getName())) {
    walk.markStart(walk.parseCommit(ref.getObjectId()));
    break;
  }
}
 
for (RevCommit commit : walk) {
  ...
}

The final loop steps over each file in the commit – you can construct patches here if you wish. For this application, it’s helpful to remove very large commits, where someone might have made mass refactorings, and to hide file deletions. (That said, there may be some value in reporting on who does mass changes – this might also identify your senior developers).

As we loop over the commits, we can construct “documents” to send to Solr- these are just maps containing all the attributes we want to save.

// Connections happen over HTTP:
HttpSolrServer server = new HttpSolrServer(
     "http://localhost:8080/solr");
 
Collection docs = new ArrayList();
 
SolrInputDocument doc = new SolrInputDocument();
 
// ID for the document should have enough information to 
// find it in the source data:
doc.addField("id", remoteUrl + "." + commit.getId());
 
doc.addField("author", commit.getAuthorIdent().getName());
doc.addField("email", commit.getAuthorIdent().getEmailAddress());
doc.addField("message", commit.getFullMessage());
 
// Any data we let the user search against is included in this value:
doc.addField("search", search);

The final search results look like this:

Notably, all these people are knowledgeable about SQL, but from different angles: a committer on Postgres, a DBA, a principal on MySQL, and people who built PHP drivers for Oracle and SQL Server. Thus, through fairly simple means, and a few hours of work, you can quickly built a reporting system which identifies key players in an organization- If you’re interested in this project, check out slides from my talk, “Mining Source Code Repositories with Solr“, or the code for the project on github.

I recently attended a talk by a sales engineer for Acunu (http://www.acunu.com/), an analytics platform for Cassandra. I came away with a couple interesting notes:
- The product aims to build data cubes for you in a “big data” scenario
- Operating on the principle that disk space is cheap, they increment lots of counters on insert
- These counters correspond to specific cube + filter + grouping scenarios
- It sounds like treating cells in a database as counters is a Cassandra feature (I think I’d heard similar about Redis)
- They can also do “normal” count queries – if you had 20 dimensions, you’d be forced to build 20! cubes otherwise
- Like all cubes, this can still cause issues with double-counting (if you added two cubes) so it’s not really preferable if you work with “small” data
- Their UI has some out of the box charting capacity (with High Charts)
- Pricing is based on usage (i.e. rather than cores)
- This is possibly a competitor to MetaMarkets Druid (http://metamarkets.com/product/technology/), which I covered previously

What would be interesting here, for further research, is to compare the performance of NoSql counters and see if similar functionality can be had with a relational database.

The slides for my talk are available here – http://www.garysieling.com/jug. Please note that the demo in the slides is live during the talk, and won’t function here.

“Secrets of the Javascript Ninja,” is a relatively new Javascript book (2013), written by developers with deep browser programming experience, John Resig and Bear Bibeault, It covers lessons learned over many years maintaining Javascript libraries. A lot of the practical guidance is valuable, even if you’re an experienced developer – it includes a lot of notes about common problems developers encounter in browser APIs, many items which are otherwise difficult to learn without hard experience. If you’re primary interest is server-side Javascript, the relevant examples are likely too basic, unless you’re unfamiliar with things like closures.

This is a very practically minded book, with lots of code samples. The book is divided into categories referred to as “Training,” “Apprentice,” “Ninja,” and “Master, ” roughly splitting sections by the complexity and sophistication of material presented. While I find the “Ninja” metaphor goofy, it is not overly pervasive in the book except as a literary prop (note the cover image is a Samurai). The progression of difficulty is consistent with what you might expect from people who work on a library like jQuery: “Master,” for instance, involves patching a browser to handle events and CSS selectors correctly, rather than managing codebases of tens or hundreds of thousands of lines of Javascript code.

For someone who has written a lot of Javascript code, there will be chapters you can skip or skim. I’ve seen some of the material before – having read library documentation, I’ve read numerous discussions of OOP in Javascript, but I found the chapter on Timers invaluable. The authors are big proponents of regular expressions, which I’ve found to be atypical in most code I’ve seen, and thus presents an interesting perspective.

Since the book is relatively new, the authors structure certain chapters around how one might patch useful new constructs into legacy browsers (e.g. event handling improvements), which provides a window into how they think about design, what new features are coming, and how the new features actually work. This isn’t about design patterns per se though, it is about specific APIs that have cause a lot of pain in the past (or currently), and are being improved.

If you’re not currently steeped in browser coding issues, or are only causually, “Secrets of the Javascript Ninja” could easily save you the cost in time through understanding common nuances of brower APIs.

One of the challenges in supporting complex applications is to answer questions like “Why doesn’t my document show up in search results,” or “why isn’t this event in my report?”

Even though database operations are often modeled with set theory, returning the inverse of a query is not sufficient.

Consider, however, a hypothetical ORM layer, where one specifies what is needed, and the ORM builds SQL for you:

query = {
  select: ['document_id', 'document_name', 'document_author', 'document_description'],
  where: [
    {column: 'company', op: '=', value: 'ACME Corp'},
    {column: 'creation_date', op: '>', value: '1/1/2013'}
  ]
};

The simplest ORM layers will just concatenate this into SQL according to known formatting rules; more complex systems handle ACLs, construct joins, or encapsulate rules for tuning queries.

// Vastly oversimplified way of constructing SQL
function sql(query) {
  var result =
    "SELECT " + query.select.join(", ") +
    " WHERE ";
 
  for (var i = 0; i < query.where.length; i++) 
    result += query.where[i].column + 
              query.where[i].op +
              "'" + query.where[i].value + "'"
}
 
function runQuery(query) {
  var q = sql(query); 
  // run query
  ...
  // return results
}

Now, let’s consider the support question, “where is document X”. Hopefully it is expressed in this form and not “there were more things yesterday,” although you could extend this technique to handle scenarios like that.

// Treat query objects as immutable
function copyQuery(query) {
  return {
    select: query.select.slice(0),
    where: query.where.slice(0)
  }
};
 
function getTestQuery(query, id) {
  var result = copyQuery(query);
  result.where.push(
    {column: 'document_id', op: '=', value: id}
  );
  return result;
}

If the earlier query was fast, this should be as well. You want to run this query first, to prove that the user’s ticket is actually caused by data, rather than misconfigured caching or a higher level defect.

From here, this becomes a simple bread-first search:

function removeWhere(query, i) {
 return {
    select: query.select.slice(0),
    where: query.where.slice(0).splice(i, i+1)
  } 
}
 
var tests = [];
tests.push(removeWhere(query, 0));
tests.push(removeWhere(query, 1));
// etc

Then, all you have to do is run through these tests until you get a result, essentially the same as what a developer would do by hand. Looks simple, right?

The reason you want to do a bread-first search is that for very complex queries (dozens of where clauses), removing fields will tend on average to make things slower. There is a risk, for instance, of removing an indexed field. Ideally this “why not” analysis then becomes baked into your ORM, which would work well if it knows which fields are indexed.

An alternate, general case scenario would be for the ORM to run a database execution plan and apply some heuristics, e.g. if it changes structurally from the original, put it later in the queue.

A final consideration is that of ACLs: it may be mandatory for this support functionality to be baked into an ORM layer, as you’d want to run this with an “ignore permissions” option, to determine if ACLs are the cause. The analysis of “why doesn’t this user have permissions” is too complex (and often this is a business process or training issue), and better answered through other means.

This technique lets you unlock some of the power of your ORM layer (assuming it has any to begin with), automating typical support work through code. In practice I’ve yet to see this done, but it seems like a very practical feature for the next generation of ORMs to consider, as unglamorous problems like support are not always prime topics of discussion among developers.

This fairly simple example makes use of the core Java APIs to read files, demonstrating how to construct a base-64 encoded version, and then generating the original file again.

package com.garysieling {
  import java.io.{File, FileInputStream, FileOutputStream}
  import sun.misc.{BASE64Encoder, BASE64Decoder}
 
  object Base64 extends App {
    val filename = """C:\IdeaProjects\base64\tests.pdf"""
    Console.println("Testing " + filename)
 
    val file = new File(filename)
    val in = new FileInputStream(file)
    val bytes = new Array[Byte](file.length.toInt)
    in.read(bytes)
    in.close()
 
    val encodedFile = new File(filename + ".base64")
    val encoded = 
       new BASE64Encoder()
         .encode(bytes)
         .replace("\n", "")
         .replace("\r", "")
    val encodedStream = new FileOutputStream(encodedFile)
    encodedStream.write(encoded.getBytes)
    encodedStream.close()
 
    val decodedFile = new File(filename + ".decoded")
    val decoded = new BASE64Decoder().decodeBuffer(encoded)
    val decodedStream = new FileOutputStream(decodedFile)
    decodedStream.write(decoded)
    decodedStream.close()
  }
}