Quantcast

Maximum PC

It is currently Fri Aug 22, 2014 5:12 am

All times are UTC - 8 hours




Post new topic Reply to topic  [ 2 posts ] 
Author Message
 Post subject: An Analogy: Why File Quantity Kills Transfer Rates
PostPosted: Mon Feb 04, 2013 3:42 pm 
Smithfield
Smithfield

Joined: Sun Jun 18, 2006 7:37 pm
Posts: 5219
I've pushed it around on a few topics, but perhaps this analogy could shed light on why having a lot of files kills your transfer speed as opposed to one big file of the same size.

You're sorting files for a company, you got all the cabinet space you need, but because it's a big company, they're pretty anal about how to index things. Let's say at the minimum, you have to index by Department -> Project -> Date. And these files could be anything from memos, reports, notes, etc. Now you're given two tasks: organize 10,000 separate, distinct sheets of paper; or organize 34 300 page (10,200 pages total) notebooks. Which one do you think will take you less time?

Even though it's roughly the same amount of data, one will take longer because there's more overhead.


Top
  Profile  
 
 Post subject: Re: An Analogy: Why File Quantity Kills Transfer Rates
PostPosted: Mon Feb 04, 2013 4:00 pm 
Clawhammer
Clawhammer
User avatar

Joined: Fri Sep 23, 2005 6:22 pm
Posts: 4406
Location: In the closet
Good analogy. For the visual...

Image

The reasons I've come to appreciate are most hdd's are well defragmented thus copying a single large file will be a continuous read with very little movement for the heads while various small files are more likely to be spread all over it causing the head to do a lot of jumping for every file. Also copying files is more than reading it’s data at one place and write it to another. There are metadata (security options, file attributes, nnn) which needs to be read from the source, written to the target and centralized data like the Master File Table needs to be updated for every single file.

If you copy files to another File System there are also other operations necessary for example FAT32 doesn’t support all attributes and metadata NTFS and has a different cluster size. These are things the system has to take care of too. And if you copy files to another system or volume over the network additional things come into play like validation, etc.

This is why a common backup strategy is to compress all data locally [into a single large compressed or image file] and then transfer this one big file over the network to the storage.


Top
  Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 2 posts ] 

All times are UTC - 8 hours


Who is online

Users browsing this forum: No registered users and 7 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group