{"id":5004,"date":"2016-08-25T01:13:24","date_gmt":"2016-08-25T01:13:24","guid":{"rendered":"http:\/\/www.garysieling.com\/blog\/?p=5004"},"modified":"2020-03-30T02:41:29","modified_gmt":"2020-03-30T02:41:29","slug":"uploading-json-files-to-dynamodb-from-python","status":"publish","type":"post","link":"https:\/\/www.garysieling.com\/blog\/uploading-json-files-to-dynamodb-from-python\/","title":{"rendered":"Uploading JSON files to DynamoDB from Python"},"content":{"rendered":"<p>Posting JSON to DynamoDB through the AWS CLI can fail due to Unicode errors, so it may be worth importing your data manually through Python.<\/p>\n<p>Fortunately this is relatively simple &#8211; you need to do this first:<\/p>\n<pre lang=\"bash\">pip install boto3\n<\/pre>\n<p>Then you can run this code to loop over all files in a directory and upload them:<\/p>\n<pre lang=\"bash\">from __future__ import print_function\nimport boto3\nimport json\nimport decimal\nimport os \n\ndynamodb = boto3.resource('dynamodb', region_name='us-east-1')\n\ntable = dynamodb.Table('talks')\n\nrootdir = \"d:\\\\projects\\\\image-annotation\\\\data\\\\talks\\\\json\\\\1\"\nfor subdir, dirs, files in os.walk(rootdir):\n  for file in files:\n    selected = subdir + \"\\\\\" + file\n    try:\n      with open(selected, encoding=\"UTF-8\") as json_file:\n        row = json.load(json_file, parse_float = decimal.Decimal)\n        row['id_i'] = str(row['id_i'])\n\n        #print(row)\n\n        table.put_item(\n          Item=row\n        )\n    except:\n      print(\"error: \" + selected)\n<\/pre>\n<p>Note here that I&#8217;ve chosen &#8220;id_i&#8221; as my primary key and set it up as a string in AWS, which is why it needs to be monkey-patched here.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Uploading data to AWS DynamoDB through Python<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"footnotes":""},"categories":[6,12],"tags":[71,179,392,447],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 4.9.9 - aioseo.com -->\n\t<meta name=\"description\" content=\"Uploading data to AWS DynamoDB through Python\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"gary\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/www.garysieling.com\/blog\/uploading-json-files-to-dynamodb-from-python\/\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 4.9.9\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"Gary Sieling - Software Engineer\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"Uploading JSON files to DynamoDB from Python - Gary Sieling\" \/>\n\t\t<meta property=\"og:description\" content=\"Uploading data to AWS DynamoDB through Python\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/www.garysieling.com\/blog\/uploading-json-files-to-dynamodb-from-python\/\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2016-08-25T01:13:24+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-03-30T02:41:29+00:00\" \/>\n\t\t<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n\t\t<meta name=\"twitter:title\" content=\"Uploading JSON files to DynamoDB from Python - Gary Sieling\" \/>\n\t\t<meta name=\"twitter:description\" content=\"Uploading data to AWS DynamoDB through Python\" \/>\n\t\t<script type=\"application\/ld+json\" class=\"aioseo-schema\">\n\t\t\t{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"BlogPosting\",\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/uploading-json-files-to-dynamodb-from-python\\\/#blogposting\",\"name\":\"Uploading JSON files to DynamoDB from Python - Gary Sieling\",\"headline\":\"Uploading JSON files to DynamoDB from Python\",\"author\":{\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/author\\\/gary\\\/#author\"},\"publisher\":{\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/#organization\"},\"datePublished\":\"2016-08-25T01:13:24+00:00\",\"dateModified\":\"2020-03-30T02:41:29+00:00\",\"inLanguage\":\"en-US\",\"commentCount\":1,\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/uploading-json-files-to-dynamodb-from-python\\\/#webpage\"},\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/uploading-json-files-to-dynamodb-from-python\\\/#webpage\"},\"articleSection\":\"Data Science, How-To, aws, dynamodb, nosql, python\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/uploading-json-files-to-dynamodb-from-python\\\/#breadcrumblist\",\"itemListElement\":[{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog#listItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.garysieling.com\\\/blog\",\"nextItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/category\\\/data-science\\\/#listItem\",\"name\":\"Data Science\"}},{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/category\\\/data-science\\\/#listItem\",\"position\":2,\"name\":\"Data Science\",\"item\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/category\\\/data-science\\\/\",\"nextItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/uploading-json-files-to-dynamodb-from-python\\\/#listItem\",\"name\":\"Uploading JSON files to DynamoDB from Python\"},\"previousItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog#listItem\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/uploading-json-files-to-dynamodb-from-python\\\/#listItem\",\"position\":3,\"name\":\"Uploading JSON files to DynamoDB from Python\",\"previousItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/category\\\/data-science\\\/#listItem\",\"name\":\"Data Science\"}}]},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/#organization\",\"name\":\"Gary Sieling\",\"description\":\"Software Engineer\",\"url\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/author\\\/gary\\\/#author\",\"url\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/author\\\/gary\\\/\",\"name\":\"gary\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/uploading-json-files-to-dynamodb-from-python\\\/#authorImage\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/0be925276d848ffe98a6a9dc8cf33e67?s=96&d=identicon&r=g\",\"width\":96,\"height\":96,\"caption\":\"gary\"}},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/uploading-json-files-to-dynamodb-from-python\\\/#webpage\",\"url\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/uploading-json-files-to-dynamodb-from-python\\\/\",\"name\":\"Uploading JSON files to DynamoDB from Python - Gary Sieling\",\"description\":\"Uploading data to AWS DynamoDB through Python\",\"inLanguage\":\"en-US\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/#website\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/uploading-json-files-to-dynamodb-from-python\\\/#breadcrumblist\"},\"author\":{\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/author\\\/gary\\\/#author\"},\"creator\":{\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/author\\\/gary\\\/#author\"},\"datePublished\":\"2016-08-25T01:13:24+00:00\",\"dateModified\":\"2020-03-30T02:41:29+00:00\"},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/\",\"name\":\"Gary Sieling\",\"description\":\"Software Engineer\",\"inLanguage\":\"en-US\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.garysieling.com\\\/blog\\\/#organization\"}}]}\n\t\t<\/script>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"Uploading JSON files to DynamoDB from Python - Gary Sieling","description":"Uploading data to AWS DynamoDB through Python","canonical_url":"https:\/\/www.garysieling.com\/blog\/uploading-json-files-to-dynamodb-from-python\/","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"BlogPosting","@id":"https:\/\/www.garysieling.com\/blog\/uploading-json-files-to-dynamodb-from-python\/#blogposting","name":"Uploading JSON files to DynamoDB from Python - Gary Sieling","headline":"Uploading JSON files to DynamoDB from Python","author":{"@id":"https:\/\/www.garysieling.com\/blog\/author\/gary\/#author"},"publisher":{"@id":"https:\/\/www.garysieling.com\/blog\/#organization"},"datePublished":"2016-08-25T01:13:24+00:00","dateModified":"2020-03-30T02:41:29+00:00","inLanguage":"en-US","commentCount":1,"mainEntityOfPage":{"@id":"https:\/\/www.garysieling.com\/blog\/uploading-json-files-to-dynamodb-from-python\/#webpage"},"isPartOf":{"@id":"https:\/\/www.garysieling.com\/blog\/uploading-json-files-to-dynamodb-from-python\/#webpage"},"articleSection":"Data Science, How-To, aws, dynamodb, nosql, python"},{"@type":"BreadcrumbList","@id":"https:\/\/www.garysieling.com\/blog\/uploading-json-files-to-dynamodb-from-python\/#breadcrumblist","itemListElement":[{"@type":"ListItem","@id":"https:\/\/www.garysieling.com\/blog#listItem","position":1,"name":"Home","item":"https:\/\/www.garysieling.com\/blog","nextItem":{"@type":"ListItem","@id":"https:\/\/www.garysieling.com\/blog\/category\/data-science\/#listItem","name":"Data Science"}},{"@type":"ListItem","@id":"https:\/\/www.garysieling.com\/blog\/category\/data-science\/#listItem","position":2,"name":"Data Science","item":"https:\/\/www.garysieling.com\/blog\/category\/data-science\/","nextItem":{"@type":"ListItem","@id":"https:\/\/www.garysieling.com\/blog\/uploading-json-files-to-dynamodb-from-python\/#listItem","name":"Uploading JSON files to DynamoDB from Python"},"previousItem":{"@type":"ListItem","@id":"https:\/\/www.garysieling.com\/blog#listItem","name":"Home"}},{"@type":"ListItem","@id":"https:\/\/www.garysieling.com\/blog\/uploading-json-files-to-dynamodb-from-python\/#listItem","position":3,"name":"Uploading JSON files to DynamoDB from Python","previousItem":{"@type":"ListItem","@id":"https:\/\/www.garysieling.com\/blog\/category\/data-science\/#listItem","name":"Data Science"}}]},{"@type":"Organization","@id":"https:\/\/www.garysieling.com\/blog\/#organization","name":"Gary Sieling","description":"Software Engineer","url":"https:\/\/www.garysieling.com\/blog\/"},{"@type":"Person","@id":"https:\/\/www.garysieling.com\/blog\/author\/gary\/#author","url":"https:\/\/www.garysieling.com\/blog\/author\/gary\/","name":"gary","image":{"@type":"ImageObject","@id":"https:\/\/www.garysieling.com\/blog\/uploading-json-files-to-dynamodb-from-python\/#authorImage","url":"https:\/\/secure.gravatar.com\/avatar\/0be925276d848ffe98a6a9dc8cf33e67?s=96&d=identicon&r=g","width":96,"height":96,"caption":"gary"}},{"@type":"WebPage","@id":"https:\/\/www.garysieling.com\/blog\/uploading-json-files-to-dynamodb-from-python\/#webpage","url":"https:\/\/www.garysieling.com\/blog\/uploading-json-files-to-dynamodb-from-python\/","name":"Uploading JSON files to DynamoDB from Python - Gary Sieling","description":"Uploading data to AWS DynamoDB through Python","inLanguage":"en-US","isPartOf":{"@id":"https:\/\/www.garysieling.com\/blog\/#website"},"breadcrumb":{"@id":"https:\/\/www.garysieling.com\/blog\/uploading-json-files-to-dynamodb-from-python\/#breadcrumblist"},"author":{"@id":"https:\/\/www.garysieling.com\/blog\/author\/gary\/#author"},"creator":{"@id":"https:\/\/www.garysieling.com\/blog\/author\/gary\/#author"},"datePublished":"2016-08-25T01:13:24+00:00","dateModified":"2020-03-30T02:41:29+00:00"},{"@type":"WebSite","@id":"https:\/\/www.garysieling.com\/blog\/#website","url":"https:\/\/www.garysieling.com\/blog\/","name":"Gary Sieling","description":"Software Engineer","inLanguage":"en-US","publisher":{"@id":"https:\/\/www.garysieling.com\/blog\/#organization"}}]},"og:locale":"en_US","og:site_name":"Gary Sieling - Software Engineer","og:type":"article","og:title":"Uploading JSON files to DynamoDB from Python - Gary Sieling","og:description":"Uploading data to AWS DynamoDB through Python","og:url":"https:\/\/www.garysieling.com\/blog\/uploading-json-files-to-dynamodb-from-python\/","article:published_time":"2016-08-25T01:13:24+00:00","article:modified_time":"2020-03-30T02:41:29+00:00","twitter:card":"summary_large_image","twitter:title":"Uploading JSON files to DynamoDB from Python - Gary Sieling","twitter:description":"Uploading data to AWS DynamoDB through Python"},"aioseo_meta_data":{"post_id":"5004","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"BlogPosting","isEnabled":true},"graphs":[]},"schema_type":"default","schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"limit_modified_date":false,"created":"2023-02-04 16:05:58","updated":"2026-07-06 02:11:54","ai":null,"breadcrumb_settings":null,"seo_analyzer_scan_date":null},"aioseo_breadcrumb":"<div class=\"aioseo-breadcrumbs\"><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"https:\/\/www.garysieling.com\/blog\" title=\"Home\">Home<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">&raquo;<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"https:\/\/www.garysieling.com\/blog\/category\/data-science\/\" title=\"Data Science\">Data Science<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">&raquo;<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\tUploading JSON files to DynamoDB from Python\n\t\t<\/span><\/div>","aioseo_breadcrumb_json":[{"label":"Home","link":"https:\/\/www.garysieling.com\/blog"},{"label":"Data Science","link":"https:\/\/www.garysieling.com\/blog\/category\/data-science\/"},{"label":"Uploading JSON files to DynamoDB from Python","link":"https:\/\/www.garysieling.com\/blog\/uploading-json-files-to-dynamodb-from-python\/"}],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/5004"}],"collection":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/comments?post=5004"}],"version-history":[{"count":1,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/5004\/revisions"}],"predecessor-version":[{"id":6462,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/5004\/revisions\/6462"}],"wp:attachment":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/media?parent=5004"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/categories?post=5004"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/tags?post=5004"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}