{"id":5805,"date":"2018-10-23T01:52:22","date_gmt":"2018-10-23T01:52:22","guid":{"rendered":"http:\/\/www.garysieling.com\/blog\/?p=5805"},"modified":"2020-03-30T02:29:58","modified_gmt":"2020-03-30T02:29:58","slug":"youtube-dl-aws-lambda","status":"publish","type":"post","link":"https:\/\/www.garysieling.com\/blog\/youtube-dl-aws-lambda\/","title":{"rendered":"Youtube-dl in AWS Lambda"},"content":{"rendered":"<p>I wrote an AWS lambda to get video metadata from youtube with youtube-dl (<a href=\"https:\/\/github.com\/garysieling\/lambda-video-crawler\">Code<\/a>).<\/p>\n<p>Lessons learned:<\/p>\n<ul>\n<li>There are <a href=\"https:\/\/github.com\/lambci\/docker-lambda\">Docker containers<\/a> that match the AWS environment &#8211; these could be good for doing testing on a CI server.<\/li>\n<li>If you need to run a binary like ffmpeg, you need a version build for AWS linux &#8211; the <a href=\"https:\/\/github.com\/intoli\/exodus\">Exodus bundler<\/a> made this painless<\/li>\n<li>It took me a few iterations to get the right syntax for events coming from dynamodb &#8211; if you do test runs of a lambda, make sure to get a good sample event, or you&#8217;ll spend a lot of time re-writing your code going from test to real events.<\/li>\n<li>I had to write a second script to upload a CSV to DynamoDB. It appears that there is a cap on 15 lambdas running at once (currently unclear what is triggering that limitation)<\/li>\n<li>Write I\/O is limited &#8211; make sure it all goes to \/tmp<\/li>\n<li>Use the aws cli to count the rows in your dynamodb table to track status (aws dynamodb scan &#8211;table-name findlectures-videos &#8211;select COUNT)<\/li>\n<li>Because the lambda can shell out to run external code, you can run any language, as long as it fits in a 250 MB zip<\/li>\n<li>If you use the aws cli npm package, you shouldn&#8217;t need any configuration inside your node script, aside from setting a region &#8211; it picks up credentials locally from your account.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Lessons learned from using youtube-dl on AWS lambda<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[6,8,12],"tags":[71,179,334],"aioseo_notices":[],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/5805"}],"collection":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/comments?post=5805"}],"version-history":[{"count":1,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/5805\/revisions"}],"predecessor-version":[{"id":6429,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/5805\/revisions\/6429"}],"wp:attachment":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/media?parent=5805"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/categories?post=5805"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/tags?post=5805"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}