Scala fill example

“Fill” is a static collection method, as one of several ways to instantiate a static list.

If you want a list of several values, you could do something like this:

val x = List(0, 0, 0, 0, 0)

But, fill makes this easier:

scala> List.fill(0)(5)
res22: List[Int] = List(0, 0, 0, 0, 0)

You can pass anything into this:

scala> List.fill(5)("test")
res21: List[String] = List(test, test, test, test, test)

You can even make multi-dimensional arrays this way:

scala> List.fill(2, 3)(0)
res26: List[List[Int]] = List(List(0, 0, 0), List(0, 0, 0))
 
scala> List.fill(2, 3, 4)(0)
res27: List[List[List[Int]]] = 
  List(
    List(
      List(0, 0, 0, 0), 
      List(0, 0, 0, 0), 
      List(0, 0, 0, 0)), 
    List(List(0, 0, 0, 0),
         List(0, 0, 0, 0), 
         List(0, 0, 0, 0)))

Listing object methods and class structure in scala

From scala, you can list all the methods on an object (essentially the same as pressing tab in the repl), by doing this:

List(1,2,3)
  .getClass.getMethods()

You can get the full method / class list by doing the following:

List(1,2,3)
  .map( 
    (method) => { List(method.getDeclaringClass, method.getName) }
  ).map(println)

Which will give you the following result, neatly sorted by the class hierarchy:

List(class scala.collection.immutable.$colon$colon, head)
List(class scala.collection.immutable.$colon$colon, tail)
List(class scala.collection.immutable.$colon$colon, tail)
List(class scala.collection.immutable.$colon$colon, tail)
List(class scala.collection.immutable.$colon$colon, productIterator)
List(class scala.collection.immutable.$colon$colon, productPrefix)
List(class scala.collection.immutable.$colon$colon, productElement)
List(class scala.collection.immutable.$colon$colon, productArity)
List(class scala.collection.immutable.$colon$colon, tl)
List(class scala.collection.immutable.$colon$colon, isEmpty)
List(class scala.collection.immutable.$colon$colon, copy$default$1)
List(class scala.collection.immutable.$colon$colon, copy$default$2)
List(class scala.collection.immutable.$colon$colon, copy)
List(class scala.collection.immutable.$colon$colon, tl_$eq)
List(class scala.collection.immutable.$colon$colon, tl$1)
List(class scala.collection.immutable.List, exists)
List(class scala.collection.immutable.List, writeReplace)
List(class scala.collection.immutable.List, empty)
List(class scala.collection.immutable.List, empty)
List(class scala.collection.immutable.List, slice)
List(class scala.collection.immutable.List, slice)
List(class scala.collection.immutable.List, slice)
List(class scala.collection.immutable.List, map)
List(class scala.collection.immutable.List, toList)
List(class scala.collection.immutable.List, canBuildFrom)
List(class scala.collection.immutable.List, apply)
List(class scala.collection.immutable.List, apply)
List(class scala.collection.immutable.List, $plus$plus)
List(class scala.collection.immutable.List, $plus$colon)
List(class scala.collection.immutable.List, flatMap)
List(class scala.collection.immutable.List, forall)
List(class scala.collection.immutable.List, foreach)
List(class scala.collection.immutable.List, collect)
List(class scala.collection.immutable.List, hashCode)
List(class scala.collection.immutable.List, length)
List(class scala.collection.immutable.List, concat)
List(class scala.collection.immutable.List, contains)
List(class scala.collection.immutable.List, iterator)
List(class scala.collection.immutable.List, find)
List(class scala.collection.immutable.List, fill)
List(class scala.collection.immutable.List, fill)
List(class scala.collection.immutable.List, fill)
List(class scala.collection.immutable.List, fill)
List(class scala.collection.immutable.List, fill)
List(class scala.collection.immutable.List, reverse)
List(class scala.collection.immutable.List, reverse)
List(class scala.collection.immutable.List, drop)
List(class scala.collection.immutable.List, drop)
List(class scala.collection.immutable.List, drop)
List(class scala.collection.immutable.List, $colon$colon)
List(class scala.collection.immutable.List, takeWhile)
List(class scala.collection.immutable.List, takeWhile)
List(class scala.collection.immutable.List, takeWhile)
List(class scala.collection.immutable.List, seq)
List(class scala.collection.immutable.List, seq)
List(class scala.collection.immutable.List, seq)
List(class scala.collection.immutable.List, seq)
List(class scala.collection.immutable.List, seq)
List(class scala.collection.immutable.List, seq)
List(class scala.collection.immutable.List, seq)
List(class scala.collection.immutable.List, seq)
List(class scala.collection.immutable.List, seq)
List(class scala.collection.immutable.List, ReusableCBF)
List(class scala.collection.immutable.List, tabulate)
List(class scala.collection.immutable.List, tabulate)
List(class scala.collection.immutable.List, tabulate)
List(class scala.collection.immutable.List, tabulate)
List(class scala.collection.immutable.List, tabulate)
List(class scala.collection.immutable.List, range)
List(class scala.collection.immutable.List, range)
List(class scala.collection.immutable.List, iterate)
List(class scala.collection.immutable.List, scala$collection$LinearSeqOptimized$$super$sameElements)
List(class scala.collection.immutable.List, unapplySeq)
List(class scala.collection.immutable.List, sameElements)
List(class scala.collection.immutable.List, foldLeft)
List(class scala.collection.immutable.List, reduceLeft)
List(class scala.collection.immutable.List, reduceRight)
List(class scala.collection.immutable.List, last)
List(class scala.collection.immutable.List, dropRight)
List(class scala.collection.immutable.List, dropRight)
List(class scala.collection.immutable.List, lengthCompare)
List(class scala.collection.immutable.List, isDefinedAt)
List(class scala.collection.immutable.List, isDefinedAt)
List(class scala.collection.immutable.List, segmentLength)
List(class scala.collection.immutable.List, indexWhere)
List(class scala.collection.immutable.List, lastIndexWhere)
List(class scala.collection.immutable.List, thisCollection)
List(class scala.collection.immutable.List, thisCollection)
List(class scala.collection.immutable.List, thisCollection)
List(class scala.collection.immutable.List, thisCollection)
List(class scala.collection.immutable.List, toCollection)
List(class scala.collection.immutable.List, toCollection)
List(class scala.collection.immutable.List, toCollection)
List(class scala.collection.immutable.List, toCollection)
List(class scala.collection.immutable.List, corresponds)
List(class scala.collection.immutable.List, toSeq)
List(class scala.collection.immutable.List, toSeq)
List(class scala.collection.immutable.List, toSeq)
List(class scala.collection.immutable.List, parCombiner)
List(class scala.collection.immutable.List, companion)
List(class scala.collection.immutable.List, $colon$colon$colon)
List(class scala.collection.immutable.List, reverse_$colon$colon$colon)
List(class scala.collection.immutable.List, mapConserve)
List(class scala.collection.immutable.List, take)
List(class scala.collection.immutable.List, take)
List(class scala.collection.immutable.List, take)
List(class scala.collection.immutable.List, takeRight)
List(class scala.collection.immutable.List, takeRight)
List(class scala.collection.immutable.List, splitAt)
List(class scala.collection.immutable.List, dropWhile)
List(class scala.collection.immutable.List, dropWhile)
List(class scala.collection.immutable.List, span)
List(class scala.collection.immutable.List, foldRight)
List(class scala.collection.immutable.List, stringPrefix)
List(class scala.collection.immutable.List, toStream)
List(class scala.collection.immutable.List, toTraversable)
List(class scala.collection.immutable.List, groupBy)
List(class scala.collection.immutable.List, toIterable)
List(class scala.collection.immutable.List, view)
List(class scala.collection.immutable.List, view)
List(class scala.collection.immutable.List, view)
List(class scala.collection.immutable.List, view)
List(class scala.collection.AbstractSeq, prefixLength)
List(class scala.collection.AbstractSeq, lift)
List(class scala.collection.AbstractSeq, orElse)
List(class scala.collection.AbstractSeq, equals)
List(class scala.collection.AbstractSeq, toString)
List(class scala.collection.AbstractSeq, indexOf)
List(class scala.collection.AbstractSeq, indexOf)
List(class scala.collection.AbstractSeq, startsWith)
List(class scala.collection.AbstractSeq, startsWith)
List(class scala.collection.AbstractSeq, endsWith)
List(class scala.collection.AbstractSeq, lastIndexOf)
List(class scala.collection.AbstractSeq, lastIndexOf)
List(class scala.collection.AbstractSeq, size)
List(class scala.collection.AbstractSeq, diff)
List(class scala.collection.AbstractSeq, runWith)
List(class scala.collection.AbstractSeq, permutations)
List(class scala.collection.AbstractSeq, combinations)
List(class scala.collection.AbstractSeq, reverseMap)
List(class scala.collection.AbstractSeq, patch)
List(class scala.collection.AbstractSeq, updated)
List(class scala.collection.AbstractSeq, padTo)
List(class scala.collection.AbstractSeq, union)
List(class scala.collection.AbstractSeq, intersect)
List(class scala.collection.AbstractSeq, distinct)
List(class scala.collection.AbstractSeq, reverseIterator)
List(class scala.collection.AbstractSeq, indexOfSlice)
List(class scala.collection.AbstractSeq, indexOfSlice)
List(class scala.collection.AbstractSeq, lastIndexOfSlice)
List(class scala.collection.AbstractSeq, lastIndexOfSlice)
List(class scala.collection.AbstractSeq, containsSlice)
List(class scala.collection.AbstractSeq, sortWith)
List(class scala.collection.AbstractSeq, sorted)
List(class scala.collection.AbstractSeq, indices)
List(class scala.collection.AbstractSeq, sortBy)
List(class scala.collection.AbstractSeq, $colon$plus)
List(class scala.collection.AbstractSeq, apply$mcZD$sp)
List(class scala.collection.AbstractSeq, apply$mcDD$sp)
List(class scala.collection.AbstractSeq, apply$mcFD$sp)
List(class scala.collection.AbstractSeq, apply$mcID$sp)
List(class scala.collection.AbstractSeq, apply$mcJD$sp)
List(class scala.collection.AbstractSeq, apply$mcVD$sp)
List(class scala.collection.AbstractSeq, apply$mcZF$sp)
List(class scala.collection.AbstractSeq, apply$mcDF$sp)
List(class scala.collection.AbstractSeq, apply$mcFF$sp)
List(class scala.collection.AbstractSeq, apply$mcIF$sp)
List(class scala.collection.AbstractSeq, apply$mcJF$sp)
List(class scala.collection.AbstractSeq, apply$mcVF$sp)
List(class scala.collection.AbstractSeq, apply$mcZI$sp)
List(class scala.collection.AbstractSeq, apply$mcDI$sp)
List(class scala.collection.AbstractSeq, apply$mcFI$sp)
List(class scala.collection.AbstractSeq, apply$mcII$sp)
List(class scala.collection.AbstractSeq, apply$mcJI$sp)
List(class scala.collection.AbstractSeq, apply$mcVI$sp)
List(class scala.collection.AbstractSeq, apply$mcZJ$sp)
List(class scala.collection.AbstractSeq, apply$mcDJ$sp)
List(class scala.collection.AbstractSeq, apply$mcFJ$sp)
List(class scala.collection.AbstractSeq, apply$mcIJ$sp)
List(class scala.collection.AbstractSeq, apply$mcJJ$sp)
List(class scala.collection.AbstractSeq, apply$mcVJ$sp)
List(class scala.collection.AbstractSeq, compose)
List(class scala.collection.AbstractSeq, andThen)
List(class scala.collection.AbstractSeq, andThen)
List(class scala.collection.AbstractSeq, indexWhere)
List(class scala.collection.AbstractSeq, lastIndexWhere)
List(class scala.collection.AbstractSeq, applyOrElse)
List(class scala.collection.AbstractSeq, view)
List(class scala.collection.AbstractSeq, view)
List(class scala.collection.AbstractIterable, copyToArray)
List(class scala.collection.AbstractIterable, canEqual)
List(class scala.collection.AbstractIterable, zip)
List(class scala.collection.AbstractIterable, toIterable)
List(class scala.collection.AbstractIterable, toIterator)
List(class scala.collection.AbstractIterable, grouped)
List(class scala.collection.AbstractIterable, sliding)
List(class scala.collection.AbstractIterable, sliding)
List(class scala.collection.AbstractIterable, zipWithIndex)
List(class scala.collection.AbstractIterable, zipAll)
List(class scala.collection.AbstractTraversable, nonEmpty)
List(class scala.collection.AbstractTraversable, copyToArray)
List(class scala.collection.AbstractTraversable, copyToArray)
List(class scala.collection.AbstractTraversable, mkString)
List(class scala.collection.AbstractTraversable, mkString)
List(class scala.collection.AbstractTraversable, mkString)
List(class scala.collection.AbstractTraversable, fold)
List(class scala.collection.AbstractTraversable, flatten)
List(class scala.collection.AbstractTraversable, filterNot)
List(class scala.collection.AbstractTraversable, withFilter)
List(class scala.collection.AbstractTraversable, count)
List(class scala.collection.AbstractTraversable, min)
List(class scala.collection.AbstractTraversable, max)
List(class scala.collection.AbstractTraversable, toArray)
List(class scala.collection.AbstractTraversable, init)
List(class scala.collection.AbstractTraversable, filter)
List(class scala.collection.AbstractTraversable, headOption)
List(class scala.collection.AbstractTraversable, toSet)
List(class scala.collection.AbstractTraversable, toSet)
List(class scala.collection.AbstractTraversable, newBuilder)
List(class scala.collection.AbstractTraversable, sum)
List(class scala.collection.AbstractTraversable, genericBuilder)
List(class scala.collection.AbstractTraversable, repr)
List(class scala.collection.AbstractTraversable, toTraversable)
List(class scala.collection.AbstractTraversable, groupBy)
List(class scala.collection.AbstractTraversable, isTraversableAgain)
List(class scala.collection.AbstractTraversable, hasDefiniteSize)
List(class scala.collection.AbstractTraversable, $plus$plus$colon)
List(class scala.collection.AbstractTraversable, $plus$plus$colon)
List(class scala.collection.AbstractTraversable, partition)
List(class scala.collection.AbstractTraversable, scan)
List(class scala.collection.AbstractTraversable, scanLeft)
List(class scala.collection.AbstractTraversable, scanRight)
List(class scala.collection.AbstractTraversable, lastOption)
List(class scala.collection.AbstractTraversable, sliceWithKnownDelta)
List(class scala.collection.AbstractTraversable, sliceWithKnownBound)
List(class scala.collection.AbstractTraversable, tails)
List(class scala.collection.AbstractTraversable, inits)
List(class scala.collection.AbstractTraversable, to)
List(class scala.collection.AbstractTraversable, reversed)
List(class scala.collection.AbstractTraversable, collectFirst)
List(class scala.collection.AbstractTraversable, $div$colon)
List(class scala.collection.AbstractTraversable, $colon$bslash)
List(class scala.collection.AbstractTraversable, reduceLeftOption)
List(class scala.collection.AbstractTraversable, reduceRightOption)
List(class scala.collection.AbstractTraversable, reduce)
List(class scala.collection.AbstractTraversable, reduceOption)
List(class scala.collection.AbstractTraversable, aggregate)
List(class scala.collection.AbstractTraversable, product)
List(class scala.collection.AbstractTraversable, maxBy)
List(class scala.collection.AbstractTraversable, minBy)
List(class scala.collection.AbstractTraversable, copyToBuffer)
List(class scala.collection.AbstractTraversable, toIndexedSeq)
List(class scala.collection.AbstractTraversable, toBuffer)
List(class scala.collection.AbstractTraversable, toVector)
List(class scala.collection.AbstractTraversable, toMap)
List(class scala.collection.AbstractTraversable, toMap)
List(class scala.collection.AbstractTraversable, addString)
List(class scala.collection.AbstractTraversable, addString)
List(class scala.collection.AbstractTraversable, addString)
List(class scala.collection.AbstractTraversable, par)
List(class scala.collection.AbstractTraversable, unzip)
List(class scala.collection.AbstractTraversable, unzip3)
List(class scala.collection.AbstractTraversable, transpose)
List(class java.lang.Object, wait)
List(class java.lang.Object, wait)
List(class java.lang.Object, wait)
List(class java.lang.Object, getClass)
List(class java.lang.Object, notify)
List(class java.lang.Object, notifyAll)

Fixing the error “cannot create /dev/null: Permission denied”, “Operation not permitted” restarting Apache

If you try to restart apache without sudo, like this:

sshuser@makingbeehives:~$ /etc/init.d/apache2 restart

You may receive the following error message:

/etc/init.d/apache2: 229: /etc/init.d/apache2: cannot create /dev/null: Permission denied
/usr/sbin/apache2ctl: 87: ulimit: error setting limit (Operation not permitted)
[Mon Feb 08 21:49:45 2016] [error] VirtualHost 173.255.224.150:80 -- mixing * ports and non-* ports with a NameVirtualHost address is not supported, proceeding with undefined results
Syntax OK
   ...fail!

The solution is to simply sudo the command:

sshuser@makingbeehives:~$ sudo /etc/init.d/apache2 restart

Scala array indexing

Scala arrays are 0-based and are accessed with parenthesis, rather than square brackets:

val x = Array(1,2,3)
 
x(0)

You can also index them inline, or use the same syntax to instantiate other collections:

scala> List(1,2,3)(0)
res10: Int = 1
 
scala> List(1,2,3)(1)
res11: Int = 2

Unlike languages like Python and R, collections will not wrap around:

scala> List(1,2,3)(3)
java.lang.IndexOutOfBoundsException: 3
  at scala.collection.LinearSeqOptimized$class.apply(LinearSeqOptimized.scala:51)
  at scala.collection.immutable.List.apply(List.scala:83)
  ... 33 elided
 
scala> List(1,2,3)(-1)
java.lang.IndexOutOfBoundsException: -1
  at scala.collection.LinearSeqOptimized$class.apply(LinearSeqOptimized.scala:51)
  at scala.collection.immutable.List.apply(List.scala:83)
  ... 33 elided

Using RabbitMQ for ETL of cloud storage APIs

Message queues are used as a communication between pieces of software, e.g. on a web site these are often used to request that a job be run in the background.

Compared to a database, they are optimized for message throughput and provide some mechanisms to send messages to many consumers – this could be useful if you were replicating data to many locations.

RabbitMQ is a popular message queuing product, and is fairly easy to use. In this article, I’ll show how to use it in a piece of software that pulls documents from DropBox, OneDrive, or Google Docs and indexes them for a search engine in Solr.

RabbitMQ lets us save a message string. If you use JSON, you can serialize any class, so for this example we’ll define something that takes an account ID for our application, the type of document store we’re indexing, and the OAuth credentials to DropBox.

public class IndexQueueMessage
{
  public String accountId { get; set; }
  public SearchableApplication requestedIndex { get; set; }
  public String user_key { get; set; } 
  public String user_token { get; set; } 
}

Connecting and receiving messages becomes very simple.

Queues are made to collect and pass on messages. The naming of these is one of the more important features, and the closest thing I’ve seen to a database design problem in message queueing – for instance, if you made one of these per account/tenant in your application, you could expose statistics about usage or charge people based on the quality of service you provide them (more indexers for their account)

var factory = new ConnectionFactory() 
{ 
  HostName = "tasks.cloudapp.net", 
  UserName = "user", 
  Password = "pwd"
};
 
factory.AutomaticRecoveryEnabled = true;
 
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
  var queue = indexer.GetType().ToString().ToLower() + "_indexer";
 
  channel.QueueDeclare(queue: queue,
    durable: true,
    exclusive: false,
    autoDelete: false,
    arguments: null);
 
  var consumer = new QueueingBasicConsumer(channel);
 
  channel.BasicConsume(
    queue: queue,
    noAck: false, // false = send acknowledgement from the job
    consumer: consumer
  );
 
  while (true)
  {
     ...
  }

To retrieve a message, we get the message bytes and deserialize it:

var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
 
try
{
  var body = ea.Body;
  var messageJson = Encoding.UTF8.GetString(body);
 
  executeMessage(messageJson, indexer);
}
finally
{
  debug("Acking job: " + ea.DeliveryTag);
  channel.BasicAck(
    deliveryTag: ea.DeliveryTag,
    multiple: false
  );
}

The sending side does something similar:

channel.BasicPublish(
  exchange: "",
  routingKey: app.ToString().ToLower() + "_indexer",
  basicProperties: properties,
  body: body);

When we receive messages, we can deserialize them to our message type:

try
{
  var jss = new System.Web.Script.Serialization.JavaScriptSerializer();
  var message = jss.Deserialize<IndexQueueMessage>(messageJson);
 
  routeType(message.type, indexer, message);
}
catch (Exception e)
{
  error(e);
}

Finally, we need to route these to application code:

if (MESSAGE_START.Equals(type))
{
  indexer.startIndex(message);
} 
else if (MESSAGE_FOLDER.Equals(type)) 
{
  indexer.indexFolder(message);
} 
else if (MESSAGE_CONTENT.Equals(type)) 
{
  indexer.indexContent(message);
} 
else if (MESSAGE_RECENT.Equals(type)) 
{
  indexer.indexRecent(message);
}
else if (MESSAGE_NEXT.Equals(type)) 
{
  indexer.indexNext(message);
}

There are several things that are neat about this, which were not immediately apparent to me until I set this up.

One really nice feature (compared to cron) is that you can get events to run nearly instantaneously from the time they are requested.

If you use a common message queue, a web application could have clients that use many backend clients – as an example, you could use a pre-built image resizing utility in Go, a Java based natural language processing tool, and computer vision libraries in Python, with JSON as the interop language.

This design also allows for instant parallelism – you can simply connect more processing nodes to the cluster. I’ve found that many long running ETL scripts tend to accumulate resources (memory, disk, file handles, etc) if not carefully written. A nice side benefit of this design is that you can subdivide these types of problems by having scripts run a set number of jobs and shut down – it also makes failing and re-trying much simpler (especially in the out of memory case).

Extracting images from PDFs in C#

TinyMCE is a Javascript rich-text editor that allows for a lot of extensibility. For example, this is a screenshot of what it looks like in my WordPress installation:

 

wordpress-tinymce

There are a lot of customization hooks, and because it’s used in WordPress, it gets a lot of maintenance. For instance, it supports pasting from Word pretty well. There is even a flag you can enable to allow people to paste in images (this base64 encodes them in the page).

The toughest area to work on is adding a media library, since that is typically dependent on the backend services (i.e. you’d need a different implementation depending on whether you use PHP, C#, Javascript etc, and Postgres / MySQL / Mongo).

WordPress lets you drag and drop images onto the page, and then they go into a paged list of images that you can search later.

wordpress-upload

One thing that surprises me that more people don’t do is to let you upload PDFs and automatically extract the images, but as I’ve found out, this turns out to be a little more difficult than I’d anticipated.

In .NET, there are a few options for PDF libraries, including PDFBox, Aspose, iTextSharp. I believe thatall three are originally Java libraries, which adds some complexity.

PDFBox is an Apache library, so it is the “cheapest”, but only if you value your time low. In order to use PDFBox, you have to run the library through IKVM (or download a copy from someone who has). IKVM converts bytecode from Java to .NET, and adds a ton of libraries to replace the JDK. Unfortunately it’s a big pain to do interop, since you need to write wrapper classes for things like streams. Image processing depends on AWT, which didn’t work in the versions of PDFBox I found, and after a few hours of poking at this I abandoned this approach.

iTextSharp is licensed under a license that lets you look at it, but if you want to use this without releasing your source you’d need to purchase a license. At one point it was licensed more openly. Someone ported this to C# under the old license and added it to Nuget, which is an option for testing. At this point this is probably missing a lot of bug fixes (you can certainly find many Stackoverflow posts where iText reps say this).

There are a lot of examples of how to do this already written, so for completeness, this is a good one to start from:

The biggest problem I’ve had with this is that it appears that images in PDFs may include transparency, and this information can get lost in translation on the way out.

Fixing error: “output is not a tty” in Node

If you write a Node script the prints output to the screen, and try to pipe the output to a file, you can get an error:

$ node extract.js > ideas.csv
output is not a tty

In my case, I was trying to write out a CSV file:

for (var k = 0; k < all.length; k++) {
  console.log(all[k]); 
}

The solution to this is to open the file directly and write to it:

var fs = require('fs');
var results = '';
 
for (var k = 0; k < all.length; k++) {
  results += all[k] + "\n";
}
 
fs.writeFile("ideas.csv", results);

The reason this doesn’t work as expected appears to be that ‘git bash’ is passing the command through to the Windows cmd shell, and not handling the output stream correctly. Writing directly to a file avoids the issue entirely.

Migrating Node.js apps from Heroku to Dokku

Heroku (and Dokku) make for an awesome development environment- the most well-known feature being the use of “git push” to deploy changes to a server. For simple projects, I’ve found that the deployment time is as fast as compilation times on JVM based projects, except that at the end you have a working environment.

Knowing you can deploy an environment at any time during a project means you can spin up as many environments as you want later, and it forces you to think through migrations in your checkins.

The downside of Heroku is that you are somewhat locked in, and if you want to build many small applications as a hobbyist, it adds up ($7/mo to keep one app running – the free ones shut down all the time). On the other hand, you could just set up a VM to run several apps, which guarantees many wasted hours on dependency hell problems (every time you update a shared library, you risk breaking what you have previously built).

Conceptually this setup is not that complex, but there are a lot of moving parts to get right, and it isn’t that satisfying to work on vs. actually building things.

To build a system like Heroku where you can push from git, you’d start by checking a script into the repository that runs when code is pushed to one of the remote repositories (a post-commit hook). This would kick off the deployment.

Heroku also has you check in a script to the root of your repository that defines what you want to run (a bash script with something like “node index.js” and any arguments). This lets them control the deployment, and lets the developer control what actually runs.

Virtualization at the operating system-level saves you from dependency issues (called “Linux Containers” or “LXC”), without having to build and secure a VM for every app. Linux containers seem pretty complex and I would assume there are some good ways to screw this up from a security perspective. It is new enough that tools and best practices aren’t as mature and clear as the older generation, so it’s easy to get mired in all the options (the few hours I spent trying to get Docker working have yet to pay off)

There is an open-source script called Dokku (a Star Wars reference, perhaps?), which ties all these pieces together for you, and tries to act as much like Heroku as possible. Consequently you can use a cheap virtual machine, and then upgrade it when you need real infrastructure. The marketing materials for Dokku seem quite slick, so I suspect that one or more companies is sponsoring it (probably someone like Digital Ocean), although I haven’t been able to find evidence of this.

Digital Ocean has a pre-built Dokku image you can use at $5/mo. While they’ve had some growing pains, they typically give out credits when they have issues. If you follow them on Facebook and subscribe to their give out a lot of coupons as well, which has made the $5 VMs basically free – clearly a loss leader for them (feel free to use my referral link – you get $10 off to start, and I get another $25)

Regardless of who you use, if you set up a Dokku image, there is a page where you upload your SSH public key:
dokku1

This lets it accepts your “git push.”

If you want to use subdomains for each app like Heroku does, you’ll need to add them to your DNS manually. I put Cloudflare on any site I can, if only because it has the best DNS management UI I’ve ever seen (and they don’t do referral links, unfortunately!):

dokku2

Once you set these two things up, ssh into the VM.

You can run create commands for each app you want to configure:

 dokku apps:create editor
 dokku apps:create meeting
 dokku apps:create gmail-search
 dokku apps:create document-search

This will let your VM receive pushes.

Then, to push changes, you need to set up the remote:

git remote add dokku dokku@apps.garysieling.com:generic-search-ui

If you add this to an existing app, you can push to both Heroku and Dokku simultaneously by doing this:

git push heroku master
git push dokku master

You will see the same sort of commit log on the remote that you see with Heroku:

gary@gary-PC MINGW64 /d/projects/heroku/editor (master)
$ git push dokku master
Counting objects: 451, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (350/350), done.
Writing objects: 100% (451/451), 47.84 KiB | 0 bytes/s, done.
Total 451 (delta 305), reused 138 (delta 91)
-----> Cleaning up...
-----> Building editor from herokuish...
-----> Setting config vars
       CURL_CONNECT_TIMEOUT: 5
-----> Setting config vars
       CURL_TIMEOUT: 30
-----> Adding BUILD_ENV to build environment...
-----> Node.js app detected

-----> Creating runtime environment

       NPM_CONFIG_LOGLEVEL=error
       NPM_CONFIG_PRODUCTION=true
       NODE_ENV=production
       NODE_MODULES_CACHE=true

-----> Installing binaries
       engines.node (package.json):  5.5.0
       engines.npm (package.json):   unspecified (use default)

       Downloading and installing node 5.5.0...
       Using default npm version: 3.3.12

-----> Restoring cache
       Skipping cache restore (new runtime signature)

-----> Building dependencies
       Pruning any extraneous modules
       Installing node modules (package.json)
       editor@1.0.0 /tmp/build
       +-- body-parser@1.14.2
       ...


-----> Caching build
       Clearing previous node cache
       Saving 2 cacheDirectories (default):
       - node_modules
       - bower_components (nothing to cache)

-----> Build succeeded!
       +-- body-parser@1.14.2
       +-- cookie-parser@1.4.1
       +-- ejs@2.3.4
       +-- errorhandler@1.4.3
       +-- express@4.13.4
       +-- express-session@1.13.0
       +-- foreman@1.4.1
       +-- http@0.0.0
       +-- lodash@4.0.0
       `-- morgan@1.6.1

-----> Discovering process types
       Procfile declares types -> web
-----> Releasing editor (dokku/editor:latest)...
-----> Deploying editor (dokku/editor:latest)...
-----> DOKKU_SCALE file not found in app image. Generating one based on Procfile...
-----> New DOKKU_SCALE file generated
=====> web=1
-----> Running pre-flight checks
       For more efficient zero downtime deployments, create a file CHECKS.
       See http://dokku.viewdocs.io/dokku/checks-examples.md for examples
       CHECKS file not found in container: Running simple container check...
-----> Waiting for 10 seconds ...
-----> Default container check successful!
=====> editor container output:
       Node app is running on port 5000
=====> end editor container output
-----> Running post-deploy
=====> renaming container (3735de61fe0a) suspicious_euclid to editor.web.1
-----> Creating new /home/dokku/editor/VHOST...
-----> Setting config vars
       DOKKU_NGINX_PORT: 80
-----> Configuring editor.garysieling.com...
       (using /var/lib/dokku/plugins/available/
               nginx-vhosts/templates/nginx.conf.template)
-----> Creating http nginx.conf
-----> Running nginx-pre-reload
       Reloading nginx
-----> Setting config vars
       DOKKU_APP_RESTORE: 1
=====> Application deployed:
       http://editor.garysieling.com

To dokku@apps.garysieling.com:editor
 * [new branch]      master -> master

Then, if you need a database, you can install this as well:

sudo dokku plugin:install https://github.com/dokku/dokku-postgres.git
 
dokku postgres:link editor editor

I name my database and my apps the same, which makes the above command a little unclear.

Dokku supports the same concept of environment configuration variables that Heroku does (i.e. your database connection string, API keys, etc should never be checked in). When you run the above commands, you will get log entries like this:

no config vars for generic-search-ui
-----&gt; Setting config vars
       DATABASE_URL: postgres://postgres:baf4@dokku-editor:5432/editor
-----&gt; Restarting app generic-search-ui
App generic-search-ui has not been deployed

If you are migrating an application and change the DNS, you will need to update any references in external services – e.g. typekit, the twitter developer API, Google Developer keys, etc.

Once you’ve set this up, there are some useful troubleshooting commands. E.g. list installed apps:

dokku apps
root@ubuntu-512mb-nyc2-01:~# dokku apps
=====&gt; My Apps
editor
generic-search
generic-search-ui

Viewing logs is a little more complex – you need to list the running processes, and get them using docker:

root@ubuntu-512mb-nyc2-01:~# docker ps
CONTAINER ID        IMAGE                            COMMAND                  CREATED              STATUS              PORTS               NAMES
fcc5604164c9        dokku/generic-search-ui:latest   "/start web"             About a minute ago   Up About a minute                       generic-search-ui.web.1
e4e42ace32f2        postgres:9.5.0                   "/docker-entrypoint.s"   About an hour ago    Up About an hour    5432/tcp            dokku.postgres.generic-search-ui
697e11c55807        dokku/editor:latest              "/start web"             2 days ago           Up 2 days                               editor.web.1
 
docker attach fcc5604164c9

And there you have it- your own PaaS, with limited dependencies on external companies for hosting.

,

Testing against multiple jQuery versions

I recently took over maintenance of a jQuery UI plugin that lets you highlight bits of a textarea. Since it’s a UI plugin, most of the testing is done by visual inspection, and supporting different versions of jQuery presents an interesting testing challenge, since jQuery is one of the first things to load on the page.

jquery-highlight

This is the code that generates the above example:

function runTest() {
  $('#demo1').highlightTextarea({
      words: {
        color: '#ADF0FF',
        words: ['Lorem ipsum','vulputate']
      },
      debug: true
  });
}

It turns out that you can dynamically add scripts to a page, so rather than including jQuery in a script tag, we can import it after the page loads.

Normally you might use “$(document).ready” to run initialization code, but you can use window.onload directly instead:

window.onload = loadJavascript;

Inside the runTests function, you need the ability to append a script, and call a callback once it loads, since this is an asynchronous process.

function appendScript(src, callback) {
  var head = document.getElementsByTagName('head')[0];
 
  var elt = document.createElement("script");
  elt.type = "text/javascript";
  elt.src = src;
 
  elt.onload = function() {
    callback();
  }
 
  head.appendChild(elt);
}

To make the page load correctly, I’ve set this up so you can specify the jQuery version on the URL. Once jQuery loads, it triggers jQuery UI loading, and then finally our own scritpts.

function loadJavascript() {
  // this sets the default
  var jqv = "jquery-1.11.1.min.js";
 
  try {
    jqv = window.location.search.split('?')[1].split('=')[1];
  } catch (e) {}
 
  appendScript("http://code.jquery.com/" + jqv,
    function() {
      $('#jqv').val(jqv);
 
      appendScript(
        "http://code.jquery.com/ui/1.10.4/jquery-ui.min.js",
      function() {
        appendScript("../jquery.highlighttextarea.js",
        runTest
      )})});
}

Once this works, you can add a dropdown with all the jQuery versions you want to support, and set the page to refresh when one is selected:

function setJQuery(newVersion) {
  var jqv = $('#jqv').val();
 
  window.location = 'index.html?jqv=' + jqv;
}

For completeness, here is the dropdown:

<select id="jqv" onchange="setJQuery()">
  <option value="jquery-2.2.0.js">jQuery Core 2.2.0</option>
  <option value="jquery-2.1.4.js">jQuery Core 2.1.4</option>
  <option value="jquery-2.1.3.js">jQuery Core 2.1.3</option>
  <option value="jquery-2.1.2.js">jQuery Core 2.1.2</option>
  <option value="jquery-2.1.1.js">jQuery Core 2.1.1</option>
  <option value="jquery-2.1.1-rc2.js">jQuery Core 2.1.1-rc2</option>
  <option value="jquery-2.1.1-rc1.js">jQuery Core 2.1.1-rc1</option>
  <option value="jquery-2.1.1-beta1.js">jQuery Core 2.1.1-beta1</option>
  <option value="jquery-2.1.0.js">jQuery Core 2.1.0</option>
  <option value="jquery-2.1.0-rc1.js">jQuery Core 2.1.0-rc1</option>
  <option value="jquery-2.1.0-beta3.js">jQuery Core 2.1.0-beta3</option>
  <option value="jquery-2.1.0-beta2.js">jQuery Core 2.1.0-beta2</option>
  <option value="jquery-2.1.0-beta1.js">jQuery Core 2.1.0-beta1</option>
  <option value="jquery-2.0.3.js">jQuery Core 2.0.3</option>
  <option value="jquery-2.0.2.js">jQuery Core 2.0.2</option>
  <option value="jquery-2.0.1.js">jQuery Core 2.0.1</option>
  <option value="jquery-2.0.0.js">jQuery Core 2.0.0</option>
  <option value="jquery-2.0.0-beta3.js">jQuery Core 2.0.0-beta3</option>
  <option value="jquery-2.0.0b2.js">jQuery Core 2.0.0-b2</option>
  <option value="jquery-2.0.0b1.js">jQuery Core 2.0.0-b1</option>
  <option value="jquery-1.12.0.js">jQuery Core 1.12.0</option>
  <option value="jquery-1.11.3.js">jQuery Core 1.11.3</option>
  <option value="jquery-1.11.2.js">jQuery Core 1.11.2</option>
  <option value="jquery-1.11.1.js">jQuery Core 1.11.1</option>
  <option value="jquery-1.11.1-rc2.js">jQuery Core 1.11.1-rc2</option>
  <option value="jquery-1.11.1-rc1.js">jQuery Core 1.11.1-rc1</option>
  <option value="jquery-1.11.1-beta1.js">jQuery Core 1.11.1-beta1</option>
  <option value="jquery-1.11.0.js">jQuery Core 1.11.0</option>
  <option value="jquery-1.11.0-rc1.js">jQuery Core 1.11.0-rc1</option>
  <option value="jquery-1.11.0-beta3.js">jQuery Core 1.11.0-beta3</option>
  <option value="jquery-1.11.0-beta2.js">jQuery Core 1.11.0-beta2</option>
  <option value="jquery-1.11.0-beta1.js">jQuery Core 1.11.0-beta1</option>
  <option value="jquery-1.10.2.js">jQuery Core 1.10.2</option>
  <option value="jquery-1.10.1.js">jQuery Core 1.10.1</option>
  <option value="jquery-1.10.0.js">jQuery Core 1.10.0</option>
  <option value="jquery-1.10.0-beta1.js">jQuery Core 1.10.0-beta1</option>
  <option value="jquery-1.9.1.js">jQuery Core 1.9.1</option>
  <option value="jquery-1.9.0.js">jQuery Core 1.9.0</option>
  <option value="jquery-1.9.0rc1.js">jQuery Core 1.9.0-rc1</option>
  <option value="jquery-1.9.0b1.js">jQuery Core 1.9.0-b1</option>
  <option value="jquery-1.8.3.js">jQuery Core 1.8.3</option>
  <option value="jquery-1.8.2.js">jQuery Core 1.8.2</option>
  <option value="jquery-1.8.1.js">jQuery Core 1.8.1</option>
  <option value="jquery-1.8.0.js">jQuery Core 1.8.0</option>
  <option value="jquery-1.8rc1.js">jQuery Core 1.8.0-rc1</option>
  <option value="jquery-1.8b2.js">jQuery Core 1.8.0-b2</option>
  <option value="jquery-1.8b1.js">jQuery Core 1.8.0-b1</option>
  <option value="jquery-1.7.2.js">jQuery Core 1.7.2</option>
  <option value="jquery-1.7.2rc1.js">jQuery Core 1.7.2-rc1</option>
  <option value="jquery-1.7rc1.js">jQuery Core 1.7.0-rc1</option>
  <option value="jquery-1.7b2.js">jQuery Core 1.7.0-b2</option>
  <option value="jquery-1.7b1.js">jQuery Core 1.7.0-b1</option>
  <option value="jquery-1.6.4.js">jQuery Core 1.6.4</option>
  <option value="jquery-1.6.4rc1.js">jQuery Core 1.6.4-rc1</option>
  <option value="jquery-1.6.3.js">jQuery Core 1.6.3</option>
  <option value="jquery-1.6.3rc1.js">jQuery Core 1.6.3-rc1</option>
  <option value="jquery-1.6.2.js">jQuery Core 1.6.2</option>
  <option value="jquery-1.6.2rc1.js">jQuery Core 1.6.2-rc1</option>
  <option value="jquery-1.6.1.js">jQuery Core 1.6.1</option>
  <option value="jquery-1.6.1rc1.js">jQuery Core 1.6.1-rc1</option>
  <option value="jquery-1.6.js">jQuery Core 1.6.0</option>
  <option value="jquery-1.6rc1.js">jQuery Core 1.6.0-rc1</option>
  <option value="jquery-1.6b1.js">jQuery Core 1.6.0-b1</option>
  <option value="jquery-1.5.2.js">jQuery Core 1.5.2</option>
  <option value="jquery-1.5.2rc1.js">jQuery Core 1.5.2-rc1</option>
  <option value="jquery-1.5.1.js">jQuery Core 1.5.1</option>
  <option value="jquery-1.5.1rc1.js">jQuery Core 1.5.1-rc1</option>
  <option value="jquery-1.5.js">jQuery Core 1.5.0</option>
  <option value="jquery-1.5rc1.js">jQuery Core 1.5.0-rc1</option>
  <option value="jquery-1.5b1.js">jQuery Core 1.5.0-b1</option>
  <option value="jquery-1.4.4.js">jQuery Core 1.4.4</option>
  <option value="jquery-1.4.4rc3.js">jQuery Core 1.4.4-rc3</option>
  <option value="jquery-1.4.4rc2.js">jQuery Core 1.4.4-rc2</option>
  <option value="jquery-1.4.4rc1.js">jQuery Core 1.4.4-rc1</option>
  <option value="jquery-1.4.3.js">jQuery Core 1.4.3</option>
  <option value="jquery-1.4.3rc2.js">jQuery Core 1.4.3-rc2</option>
  <option value="jquery-1.4.3rc1.js">jQuery Core 1.4.3-rc1</option>
  <option value="jquery-1.4.2.js">jQuery Core 1.4.2</option>
  <option value="jquery-1.4.1.js">jQuery Core 1.4.1</option>
  <option value="jquery-1.4.js">jQuery Core 1.4.0</option>
  <option value="jquery-1.4rc1.js">jQuery Core 1.4.0-rc1</option>
  <option value="jquery-1.4a2.js">jQuery Core 1.4.0-a2</option>
  <option value="jquery-1.4a1.js">jQuery Core 1.4.0-a1</option>
  <option value="jquery-1.3.2.js">jQuery Core 1.3.2</option>
  <option value="jquery-1.3.1.js">jQuery Core 1.3.1</option>
  <option value="jquery-1.3.1rc1.js">jQuery Core 1.3.1-rc1</option>
  <option value="jquery-1.3.js">jQuery Core 1.3.0</option>
  <option value="jquery-1.3rc2.js">jQuery Core 1.3.0-rc2</option>
  <option value="jquery-1.3rc1.js">jQuery Core 1.3.0-rc1</option>
  <option value="jquery-1.3b2.js">jQuery Core 1.3.0-b2</option>
  <option value="jquery-1.3b1.js">jQuery Core 1.3.0-b1</option>
  <option value="jquery-1.2.6.js">jQuery Core 1.2.6</option>
  <option value="jquery-1.2.5.js">jQuery Core 1.2.5</option>
  <option value="jquery-1.2.4.js">jQuery Core 1.2.4</option>
  <option value="jquery-1.2.4b.js">jQuery Core 1.2.4-b</option>
  <option value="jquery-1.2.4a.js">jQuery Core 1.2.4-a</option>
  <option value="jquery-1.2.3.js">jQuery Core 1.2.3</option>
  <option value="jquery-1.2.3b.js">jQuery Core 1.2.3-b</option>
  <option value="jquery-1.2.3a.js">jQuery Core 1.2.3-a</option>
  <option value="jquery-1.2.3.js">jQuery Core 1.2.3</option>
  <option value="jquery-1.2.3b.js">jQuery Core 1.2.3-b</option>
  <option value="jquery-1.2.3a.js">jQuery Core 1.2.3-a</option>
  <option value="jquery-1.2.2.js">jQuery Core 1.2.2</option>
  <option value="jquery-1.2.2b2.js">jQuery Core 1.2.2-b2</option>
  <option value="jquery-1.2.2b.js">jQuery Core 1.2.2-b</option>
  <option value="jquery-1.2.1.js">jQuery Core 1.2.1</option>
  <option value="jquery-1.2.js">jQuery Core 1.2.0</option>
  <option value="jquery-1.1.4.js">jQuery Core 1.1.4</option>
  <option value="jquery-1.1.3.js">jQuery Core 1.1.3</option>
  <option value="jquery-1.1.3a.js">jQuery Core 1.1.3-a</option>
  <option value="jquery-1.1.2.js">jQuery Core 1.1.2</option>
  <option value="jquery-1.1.1.js">jQuery Core 1.1.1</option>
  <option value="jquery-1.1.js">jQuery Core 1.1.0</option>
  <option value="jquery-1.1b.js">jQuery Core 1.1.0-b</option>
  <option value="jquery-1.1a.js">jQuery Core 1.1.0-a</option>
  <option value="jquery-1.0.4.js">jQuery Core 1.0.4</option>
  <option value="jquery-1.0.3.js">jQuery Core 1.0.3</option>
  <option value="jquery-1.0.2.js">jQuery Core 1.0.2</option>
  <option value="jquery-1.0.1.js">jQuery Core 1.0.1</option>
  <option value="jquery-1.0.js">jQuery Core 1.0.0</option>
</select>

Lodash debounce example

To ensure that a Javascript function only gets called once ever few seconds, you can run run it through debounce:

function run() {
   console.log('abc');
}
 
var realFunction = 
  _.debounce(run, 1500);
 
realFunction();
realFunction();
realFunction();

Output is:

abc

In this case, the function will only get run once.

If you pass an argument, it will be sent through to the function, but still only one call is made. This is why these work well as click handlers.

function run(a) {
   console.log(a);
}
 
var realFunction = 
  _.debounce(run, 1500);
 
realFunction('a');
realFunction('b');
realFunction('c');

Output is:

c

By default, debounce waits the interval before making the call.

If you want it to run immediately without a pause (if possible) do this:

var realFunction = 
  _.debounce(run, 1500, {leading: true});
 
realFunction('a'); 
realFunction('b'); 
realFunction('c');

Prints out:

a