Case Study: 10x File Copy Performance with Robocopy

Source data:

  • ~500,000 folders (court cases)
  • ~2.5-3 million documents
  • Source drives is replicated x2 with RAID
  • Copying to NAS over GB ethernet
  • Initial un-tuned copy was set to take ~2 weeks (after switching to Robocopy – before, it was painful just to do an ls)
  • Final copy took ~24 hours


  • Initially I saw 20-40 Kbps in traffic in DD-WRT, clearly too low. After some changes this is still generally low, but with spikes up to 650 Kbps.
  • CPU use – 4/8 cores in use, even with >8 threads assigned to Robocopy
  • In Computer Management -> Performance monitoring, the disk being copied is Reading as fast as it can (set to 100 all the time)
  • The number called “Split IO / second” is very high much of the time. Research indicates this could be improved with defrag (though this might take me months to complete).

Filesystem Lessions:

  • NTFS can hold a folder with large numbers of files but takes forever to enumerate
  • When you enumerate a directory in NTFS (e.g. by opening it in Windows Explorer), Windows appears to lock the folder(!) which pauses any copy/ls operations
  • The copy does not appear to be I/O bound – even setting Robocopy to use many threads, only 4/8 cores are in use at 5-15% per each.
  • ext4 (destination system) supports up to 64,000 items per folder, any more and you get an error.
  • I split all 500k items into groups of 256*256 at random (for instance one might open \36\0f to see a half dozen items). These are split up using md5 on the folder names – basically this uses the filesystem as a tree map.
  • One nice consequence of this is that you can estimate how far along the process is by looking at how many folders have been copied (85/256 -> 33%, etc)

Robocopy Options:

  • Robocopy lets you remove the console logging, with /LOG:output.txt
  • Robocopy lets you set the number of threads it uses. By default this is 8, it seemed to run faster with > 8, but only the first few threads made any difference.

To investigate:

  • Ways of using virtual filesystems – it’d be nice to continue using wget to download, but split up large folders into batches for scraping. 
  • One possibility is to use wget through VirtualBox, since there are more linux based virtual filesystems – not sure on the performance ovehead