{"id":969,"date":"2013-05-02T03:27:01","date_gmt":"2013-05-02T03:27:01","guid":{"rendered":"http:\/\/garysieling.com\/blog\/?p=969"},"modified":"2013-05-02T03:27:01","modified_gmt":"2013-05-02T03:27:01","slug":"case-study-10x-file-copy-performance-with-robocopy","status":"publish","type":"post","link":"https:\/\/www.garysieling.com\/blog\/case-study-10x-file-copy-performance-with-robocopy\/","title":{"rendered":"Case Study: 10x File Copy Performance with Robocopy"},"content":{"rendered":"
Source data:<\/p>\n
\n
~500,000 folders (court cases)<\/span><\/li>\n
~2.5-3 million documents<\/li>\n
Source drives is replicated x2 with RAID<\/li>\n
Copying to NAS over GB ethernet<\/li>\n
Initial un-tuned copy was set to take ~2 weeks (after switching to Robocopy – before, it was painful just to do an ls)<\/li>\n
Final copy took ~24 hours<\/li>\n<\/ul>\n
Monitoring:<\/p>\n
\n
Initially I saw 20-40 Kbps in traffic in DD-WRT, clearly too low. After some changes this is still generally low, but with spikes up to 650 Kbps.<\/span><\/li>\n
CPU use – 4\/8 cores in use, even with >8 threads assigned to Robocopy<\/li>\n
In Computer Management -> Performance monitoring, the disk being copied is Reading as fast as it can (set to 100 all the time)<\/li>\n
The number called “Split IO \/ second” is very high much of the time. Research indicates this could be improved with defrag (though this might take me months to complete).<\/li>\n<\/ul>\n
Filesystem Lessions:<\/p>\n
\n
NTFS can hold a folder with large numbers of files but takes forever to enumerate<\/span><\/li>\n
When you enumerate a directory in NTFS (e.g. by opening it in Windows Explorer), Windows appears to lock the folder(!) which pauses any copy\/ls operations<\/li>\n
The copy does not appear to be I\/O bound – even setting Robocopy to use many threads, only 4\/8 cores are in use at 5-15% per each.<\/li>\n
ext4 (destination system) supports up to 64,000 items per folder, any more and you get an error.<\/li>\n