{"id":5004,"date":"2016-08-25T01:13:24","date_gmt":"2016-08-25T01:13:24","guid":{"rendered":"http:\/\/www.garysieling.com\/blog\/?p=5004"},"modified":"2020-03-30T02:41:29","modified_gmt":"2020-03-30T02:41:29","slug":"uploading-json-files-to-dynamodb-from-python","status":"publish","type":"post","link":"https:\/\/www.garysieling.com\/blog\/uploading-json-files-to-dynamodb-from-python\/","title":{"rendered":"Uploading JSON files to DynamoDB from Python"},"content":{"rendered":"<p>Posting JSON to DynamoDB through the AWS CLI can fail due to Unicode errors, so it may be worth importing your data manually through Python.<\/p>\n<p>Fortunately this is relatively simple &#8211; you need to do this first:<\/p>\n<pre lang=\"bash\">pip install boto3\n<\/pre>\n<p>Then you can run this code to loop over all files in a directory and upload them:<\/p>\n<pre lang=\"bash\">from __future__ import print_function\nimport boto3\nimport json\nimport decimal\nimport os \n\ndynamodb = boto3.resource('dynamodb', region_name='us-east-1')\n\ntable = dynamodb.Table('talks')\n\nrootdir = \"d:\\\\projects\\\\image-annotation\\\\data\\\\talks\\\\json\\\\1\"\nfor subdir, dirs, files in os.walk(rootdir):\n  for file in files:\n    selected = subdir + \"\\\\\" + file\n    try:\n      with open(selected, encoding=\"UTF-8\") as json_file:\n        row = json.load(json_file, parse_float = decimal.Decimal)\n        row['id_i'] = str(row['id_i'])\n\n        #print(row)\n\n        table.put_item(\n          Item=row\n        )\n    except:\n      print(\"error: \" + selected)\n<\/pre>\n<p>Note here that I&#8217;ve chosen &#8220;id_i&#8221; as my primary key and set it up as a string in AWS, which is why it needs to be monkey-patched here.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Uploading data to AWS DynamoDB through Python<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[6,12],"tags":[71,179,392,447],"aioseo_notices":[],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/5004"}],"collection":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/comments?post=5004"}],"version-history":[{"count":1,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/5004\/revisions"}],"predecessor-version":[{"id":6462,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/posts\/5004\/revisions\/6462"}],"wp:attachment":[{"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/media?parent=5004"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/categories?post=5004"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.garysieling.com\/blog\/wp-json\/wp\/v2\/tags?post=5004"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}