Youtube-dl in AWS Lambda

I wrote an AWS lambda to get video metadata from youtube with youtube-dl (Code).

Lessons learned:

  • There are Docker containers that match the AWS environment – these could be good for doing testing on a CI server.
  • If you need to run a binary like ffmpeg, you need a version build for AWS linux – the Exodus bundler made this painless
  • It took me a few iterations to get the right syntax for events coming from dynamodb – if you do test runs of a lambda, make sure to get a good sample event, or you’ll spend a lot of time re-writing your code going from test to real events.
  • I had to write a second script to upload a CSV to DynamoDB. It appears that there is a cap on 15 lambdas running at once (currently unclear what is triggering that limitation)
  • Write I/O is limited – make sure it all goes to /tmp
  • Use the aws cli to count the rows in your dynamodb table to track status (aws dynamodb scan –table-name findlectures-videos –select COUNT)
  • Because the lambda can shell out to run external code, you can run any language, as long as it fits in a 250 MB zip
  • If you use the aws cli npm package, you shouldn’t need any configuration inside your node script, aside from setting a region – it picks up credentials locally from your account.