{"id":6046,"date":"2018-12-05T03:30:34","date_gmt":"2018-12-05T03:30:34","guid":{"rendered":"http:\/\/www.garysieling.com\/blog\/?p=6046"},"modified":"2018-12-05T03:30:34","modified_gmt":"2018-12-05T03:30:34","slug":"retinanet-dockerfile-with-gpu-support","status":"publish","type":"post","link":"https:\/\/www.garysieling.com\/blog\/retinanet-dockerfile-with-gpu-support\/","title":{"rendered":"Retinanet Dockerfile with GPU support"},"content":{"rendered":"

Retinanet is an object detection model that is supposed to be suitable for tagging objects in videos. To use this, you need to install the keras-retinanet<\/a> project from github. <\/p>\n

The following example shows how to train this, taken from the excellent pyimagesearch<\/a> book:<\/p>\n

\nretinanet-train \\\n  --batch-size 4 \\\n  --steps 349 \\\n  --epochs 50 \\\n  --weights logos\/resnet50_coco_best_v2.1.0.h5 \\\n  --snapshot-path $DIR\/snapshots \\\n  csv logos\/retinanet_train.csv \\\n  logos\/retinanet_classes.csv\n<\/pre>\n
Under the hood this uses Tensorflow, which is easy to get running against a CPU (and the default install doesn’t use Intel vector instructions, so even there it’s slower than it needs to be)<\/p>\n
When I ran the above command it estimated 5 hours per epoch – 50 training epochs would take around ten days. Based on my laptop’s electricity consumption, this would consume around $5.25 of electricity ($0.21\/kWh).<\/p>\n
The “easy” answer is to use a GPU (1080 Ti), which will bring the time to 5 minutes per epoch (4.1 hours for 50 epochs).<\/p>\n
While it may look like this saves 9.75 days, you could easily spend that time just installing dependencies: reinstalling your OS to a version CUDA supports (I chose Ubuntu 18.04), identifying which of the six ways to install CUDA works best (the .run file, if you want Steam to keep working), building a custom Tensorflow version (if using CUDA 10, which isn’t officially supported yet), and identifying the correct version of Google’s build tool (bazel) to build Tensorflow.<\/p>\n
Enter Docker.<\/p>\n
Tensorflow provides convenient pre-built Docker containers, so you can avoid a lot of this hassle. <\/p>\n
\ndocker run --rm -it --runtime=nvidia tensorflow\/tensorflow:latest-gpu-py3\n<\/pre>\n
To use these, you must install the NVIDIA Docker runtime<\/a>, so you do need to have CUDA configured, but this lets you avoid the pain of configuring the downstream libraries (Tensorflow, mxnet, etc) by delegating it to library maintainers.<\/p>\n
You may be wondering how this integrates with your main machine: you want tagged images and the model stored outside a container, with minimal I\/O overhead. Because we’re using the Linux version of Docker we avoid all the I\/O problems of OSX Docker versions<\/a> (while there are workarounds, you can waste a few hours the first time you hit these problems).<\/p>\n
To use this to run Keras \/ retinanet, we also need opencv, but the version on pip seems to work OK for this:<\/p>\n
\nFROM tensorflow\/tensorflow:latest-gpu-py3\nRUN apt-get clean && \\\n    apt-get update && \\\n    apt-get upgrade -y && \\\n    apt-get install -y git\nRUN git clone https:\/\/github.com\/fizyr\/keras-retinanet\nRUN cd keras-retinanet && pip install . --user\nRUN pip install opencv-python\nRUN apt-get install -y libsm6 libxrender1\nENTRYPOINT [\"\/bin\/bash\", \"-c\", \"\/root\/.local\/bin\/retinanet-train ${*}\", \"--\"]\n<\/pre>\n
We’ll take any arguments and pass them on to retinanet, so you can treat this as if it was the equivalent command line app running locally.<\/p>\n
\ndocker build . -t retinanet-train\n<\/pre>\n
I found it convenient to trick the Docker container into looking like it’s file system had the same structure as my laptop – so the volume mount is the output of “pwd -P” (a deep link into a drive partition).<\/p>\n
\nDIR=$(pwd -P) docker run --rm -it \\\n  --runtime=nvidia \\\n  -v $DIR:$DIR \\\n  retinanet-train \\\n  --batch-size 4 \\\n  --steps 349 \\\n  --epochs 50 \\\n  --weights $DIR\/logos\/resnet50_coco_best_v2.1.0.h5 \\\n  --snapshot-path $DIR\/snapshots \\\n  csv $DIR\/logos\/retinanet_train.csv \\\n  $DIR\/logos\/retinanet_classes.csv\n<\/pre>\n
Once you do this it works like a charm – you’ll be building new models in no time.<\/p>\n","protected":false},"excerpt":{"rendered":"
Setting up a development environment to run keras-retinanet<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"footnotes":""},"categories":[12],"tags":[120,173,328,353,407,447,540],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t