Fragmentation

[BobH]

In trying to overcome some nagging performance issues on this HP laptop, I've run Defraggler repeatedly trying to defragment the hard drive. It seems each time I run the analysis step the number of fragmented files increases and the total number of segments increases. After running defrag overnight and turning off the computer when finished, if I start Defraggler before running any other applications (after starting Windows XP Pro SP3) there are still a few dozen fragmented files and several thousands segments.

While I'm not convinced that these numbers represent a problem, I'm curious as to why there are any fragmented files left and why the ratio of fragmented files to affected segments remains relatively high. For example, I'm currently showing 14 fragmented files containing 1.2 GB over 2002 segments. Is this normal or indicative of a problem?

TIA

[JoeP]

It is probably not worth worrying about. There are certain files/folders that are never going to be defragged by a defrag program running with Windows booted. Do you have a list of the the files that are showing as fragmented?

Joe

[BobH]

It is probably not worth worrying about. There are certain files/folders that are never going to be defragged by a defrag program running with Windows booted. Do you have a list of the the files that are showing as fragmented?

Joe

Thanks for the response, Joe!

Since posting, I went through each fragmented file and defragged it. There were some very large Thunderbird files (Inbox, Sent, etc.) that I successfully defragged individually. There were dozens and dozens of very small Firefox profile files that existed as 2 fragments. Those files were roughly 25KB to 50KB in size. I persisted and defragged every file individually until all were fixed. I then shut down and restarted windows and there were fewer than 10 fragmented files.

I'm assuming that something about the way in which Acronis cloned the drive from the old one and the way in which I managed to partition it, somehow created a lot of fragments. Probably, I should have defragged as exhaustively before cloning the drive.

As I said, I don't think this is a problem. It does, though, seem that the hesitation in Firefox has gone away after cleaning up all those fragmented profile files. I'll have to remember to watch this over the next few day.

Thanks again, for the response.

[ChrisGreaves]

I'm curious as to why there are any fragmented files left

Have you tried statring a session by defragmenting FreeSpace?
The menu system varies with different versions of Defraggler.
One theory is that by defragmenting FreeSpace first you give defraggler lots of empty-sandbox space in which to dump defragmented files.

4.JPG

Me, I think that defragmentaion is over-rated nowadays, what with large RAM, cache, fast drives etc.

[viking33]

.
While I'm not convinced that these numbers represent a problem, I'm curious as to why there are any fragmented files left and why the ratio of fragmented files to affected segments remains relatively high. For example, I'm currently showing 14 fragmented files containing 1.2 GB over 2002 segments. Is this normal or indicative of a problem?

TIA

Another thing to consider is how much free space you have on the drive. Some defrggers seem to want more free space than others in order to do a complete defrag. Doing single files seem to change the equation a bit and the program is more likely to finish it up for you.

[BobH]

Thank you for the suggestions, guys!

The drive in question is a relatively new 500GB drive with a very large 450GB partition and a small 50GB partition. A 160GB drive 3/4 full was cloned to the 450GB partition. Defraggler shows that partition as 1/4 full at the moment; so there are over 300GB of free space - most of it probably virgin space.

I am not given the option to defrag anything other than the drives partitioned on the device. Also, I'm a little puzzled as to why one would need to defrag clear space. Can clear space be fragmented? Or, is it simply available and non-contiguous. If you defrag a device, aren't you moving data to eliminate non-contiguous data which usually results in contiguous clear space? :insert scratching head smiley here:

[HansV]

We do have a :scratch: smiley:

Perhaps this image will make the idea of defragmenting files and free space a bit clearer:

x260.jpg

[viking33]

I'm assuming that something about the way in whiAcronisnis cloned the drive from the old one and the way in which I managed to partition it, somehow created a lot of fragments. Probably, I should hadefraggedged as exhaustively before cloning the drive.

When Acronis creates an image or a cloned copy, it is an EXACT COPY of the original disk. IF it fragged before, it will fragged after.

[BobH]

Thanks, Hans!

The graphic you posted is a much better demonstration of my question about defragging clear space than my words. Looking at it, I can see that if the defrag software first determines whether sectors are 'free' or 'in use', then determines the general density and number of free sectors, that an option to defrag clear space makes more sense. I have always thought of defragging in the sense of placing, contiguously, sectors that contain data. The end result of moving like data to contiguous sectors ultimately results in unused sectors also being contiguous. It seems that the choice of using active sectors or free sectors as the means of examination should be determined by the number and dispersion of each type.

Am I correct in assuming that files are not mixed on sectors and that defragging takes place only at the sector level and not at some lower level? Also, am I correct in assuming that 'free' sectors does not mean that they are data free (virgin) but rather means that they are no longer addressable and might contain data from earlier assignments of those sectors? It would not matter to future disk writes that the tracks actually contain binary digits but it would matter if those tracks' addresses were in use by the MFT or other NTFS functions. If not in use, they are 'free' to have the bits set on or off according to the next write but they might not be 'clear'. But, if in use by the file system, they are not 'free' to be written and are not 'clear'. This, as I understand it, is the reason that forensic tools can take data from files that have been 'erased'. Only the bits directing the file system to organize them coherently into files have been changed when files are erased but the actual data encoding remains. To truly erase a disk drive, one must use software to turn all bits on all tracks in all sectors to either the on or the off state thus removing character representation.

Please tell me where I err as I remain eager to learn even in my dotage.

Again, thanks for the post.

[HansV]

Physical sectors aren't really relevant; the operating system works with clusters. On most hard disks nowadays, the cluster size is 4,096 bytes (4 kB). This means that files are read and written in chunks of 4 kB. If you save a text file of 160 bytes (the size of a text message), the operating system will reserve one cluster for it; the remaining 3,936 bytes in that cluster will not be used to store other files.

You're correct that unused clusters aren't necessarily really empty. When a file is deleted or moved, the operating system does not erase the clusters that were used by the file, it merely marks them as not in use, and therefore available to the system. That's why utilities such as Recuva can recover (some) deleted files - they try to reconstruct which clusters were used by a deleted file.

There are utilities that overwrite unused clusters a number of times with 0s, so that "undelete" utilities such as Recuva cannot retrieve deleted files any more. But specialized hardware/software can often still recover some of the original information.

[BobH]

I'm assuming that something about the way in which Acronis cloned the drive from the old one and the way in which I managed to partition it, somehow created a lot of fragments. Probably, I should have defragged as exhaustively before cloning the drive.

When Acronis creates an image or a cloned copy, it is an EXACT COPY of the original disk. IF it fragged before, it will fragged after.

Yes! That is my understanding, too. But, I did not see an option to create a larger partition when I cloned the old drive and assumed that I might have done something in the process that caused Acronis to perform differently. Also, I used Paragon's Partition Manager to change the partition size afterward. I was supposing that the processes might have been my error.

Although the original symptoms haven't disappeared entirely, I am getting much better performance even in Firefox. I'm reasonably certain that, by defragging, I took away some variables contributing to the problem (the dozens of Firefox profile files that existed in 2 fragments) but believe that I now need to turn my attention to Firefox.

The defrag adventure has been educational.

[BobH]

Physical sectors aren't really relevant; the operating system works with clusters. On most hard disks nowadays, the cluster size is 4,096 bytes (4 kB). This means that files are read and written in chunks of 4 kB. If you save a text file of 160 bytes (the size of a text message), the operating system will reserve one cluster for it; the remaining 3,936 bytes in that cluster will not be used to store other files.

You're correct that unused clusters aren't necessarily really empty. When a file is deleted or moved, the operating system does not erase the clusters that were used by the file, it merely marks them as not in use, and therefore available to the system. That's why utilities such as Recuva can recover (some) deleted files - they try to reconstruct which clusters were used by a deleted file.

There are utilities that overwrite unused clusters a number of times with 0s, so that "undelete" utilities such as Recuva cannot retrieve deleted files any more. But specialized hardware/software can often still recover some of the original information.

Aaaah! I was suffering from a misapprehension about the way file systems address data. Perhaps I have some residual disinformation or perhaps I never fully understood it. I did a little googling and found a graphic that gave me a better picture understanding of clusters in relation to tracks and sectors. :insert light-bulb-over-the-head smiley here:

Thanks again, Hans!

[ChrisGreaves]

Also, I'm a little puzzled as to why one would need to defrag clear space.

Try thinking of your hard drive as having 1,001 files on it.
1,000 files are what you and I call "A useful file of data", that is, if any part of any one of those 1,000 "files" got clobbered, you'd be unhappy.
Think of the 1,001st file as "all the other stuff that is of no use to me".

That is, think of "freespace" as a very large file, none of which you care about.

Then defragging freespace while leaving files fragmented says "First lets get all the free space into one large and useful chunk (and therefore by definition, all 1,000 useful files into another single chunk).

[BobH]

Also, I'm a little puzzled as to why one would need to defrag clear space.

Try thinking of your hard drive as having 1,001 files on it.
1,000 files are what you and I call "A useful file of data", that is, if any part of any one of those 1,000 "files" got clobbered, you'd be unhappy.
Think of the 1,001st file as "all the other stuff that is of no use to me".

That is, think of "freespace" as a very large file, none of which you care about.

Then defragging freespace while leaving files fragmented says "First lets get all the free space into one large and useful chunk (and therefore by definition, all 1,000 useful files into another single chunk).

Hi Chris!

I understand the concept, but I can't see the practical application.

In your analogy, those first 1000 files would be OK if they never became fragmented; but if you use and update them they will become fragmented. By defragging the freespace, the 1000 - which you use and update - remain fragmented and (arguably) affect performance while the 1001 which you don't use - and by extension don't update - don't become fragmented ergo don't affect performance. Further, if there is any fragmentation of the 1000 to begin with and only the 1001 (freespace) is defragmented, then you end with space not to be used clearly contiguous but data that is used non-contiguous (fragmented) and slowing retrieval.

Perhaps I'm missing the point entirely. At my age that happens more frequently than I like to admit. If the rationale for defragging is to improve performance and/or efficiency of disk utilization, it seems that the action should focus on disk files by frequency of use and file size which would make concern over freespace nil.

Show me the fallacies in my logic, please. Or perhaps I'm flogging an insensate equine.

[JoeP]

A couple of things to remember.

With the spacious modern drives fragmentation is less an issue than it used to be. I've seen many people post that defragging once a month is more than sufficient. I personally will never buy or use another third party defrag program on any new home systems. Whatever is supplied with the OS will be fine.

In your example remember that the 1000 files will only become fragmented if the size is increased beyond the current clusters that are allocated. Unless you increase the size of a file each time it is updated there will be some files that never become fragemented because the size is relatively static even though the file is updated. There will be others that become fragmented over time as data is gradually added. There will be some that are fragmented each time an update is done.

Most people would defrag files in additon to defragging free space. Theoretically, you'd get the best results. Whether or not the most updated files get defragged or not will most likely not be noticable. There have been studies done that show most people unable to really tell a difference until there is a 15% deviation from prior performance. Most people want to believe that defragging is a magic elixir when it comes to performance. I'd be very surprised if you get anywhere near a 15% performance improvement by defragging.

Joe

[BobH]

Hi Joe!

I suspect that you are correct about the effect of fragmentation on large HDDs; however, as Hans pointed out, NTSF assigns clusters in 4KB segments. That means that to the extent that file sizes are less than 4K, there is unused space. With the proliferation of files for software modules - typically and ideally quite small - there is a proliferation of unused, allocated space making the effective utilization of disk space less efficient. One of nature's paradoxes, I guess. This problem is compounded by the propensity of browsers and email clients to use small data chunks to minimize IO time (there's a term that only old programmers from the '70s will tremble over). The old TINSTAAFL dictum at work.

True, modern disk drives are quite fast and data density is (to us old guys) unbelievably high. These argue positively for your position. To model and test to resolve the issue is far more expensive than would be reasonable. It is far easier and less expensive simply to buy the ever more dense, ever less costly HDDs. Your statement about updating files not increasing their size gives me some pause. Logically it is consistent to hypothesize that an update could simply change or even remove data (leaving the file the same size or smaller); however in 45 years of experience with computers and their use, I've rarely experienced stasis or shrinkage in file size. They grow. That's a primary reason for the demand for larger drives, IME.

I'm not arguing for or against defragmentation. I was simply quite curious about the original results I got after trying defragmentation as a diagnostic tool and potential remediation. After pursuing the issue to satisfy my curiosity (which ultimately involved defragging files individually), I perceived an improvement in my browser operation. I remain unsure whether or not defragging improved performance or even caused my perception of improved performance. Because I found literally dozens and dozens of fragmented Firefox (browser) profile files (the provenance of which were and remain unknown to me), I speculated that Firefox was perhaps either bloating, seeking, or in some manner slowing down because of them.

Finally, on the question of free space and its import with regard to fragmentation (i.e., non-contiguity of data), I remain perplexed by the idea of defragging free space. I thought that by definition 'free space' was space that had no relationship with any other space; or, said another way, has no bearing on contiguity either of data or non-data. As I understand it, free space is space that NTSF (or other file systems) track as being available for use. Because it is in no way indexed to data or files, I don't see how it can have any impact on data transport or disk functionality unless it is dispersed across the tracks and sectors in such a manner as to be obstructive. But, then, defragging the data and not the free space would remove the obstruction. Besides, don't defrag routines move free space to the 'end' of the drive instead of leaving it interspersed? It is the data that is sought and written when the file system functions are invoked that matters, not the space that is not used - neither read nor written. Except as free space diminishes and is not available for RAM paging or file growth as needed by the OS and applications, free space does not matter. As RAM grows larger, space needed for paging increases; but RAM hasn't grown as rapidly (natural limitations imposed by addressability issues based on architecture - 8 bit, 16 bit, 32 bit, 64 bit) as the growth in disk density, speed, and size of HDDs has; so I deduce that paging is not a major issue where a physical drive is say, less than 75%, utilized. Paging would be a problem, as would reading and writing data, only as devices approach full capacity utilization.

Realizing that this is very pedantic, I've pursued it both for my personal understanding and (as Hans reminds me) for informing anyone coming later searching for discussions on disk fragmentation and file system resource allocation.

Again, thanks for the post. It gave me reason to re-examine my understanding in writing this response.

[StuartR]

...I thought that by definition 'free space' was space that had no relationship with any other space; or, said another way, has no bearing on contiguity either of data or non-data. As I understand it, free space is space that NTSF (or other file systems) track as being available for use. Because it is in no way indexed to data or files, I don't see how it can have any impact on data transport or disk functionality...

Fragmented free space can have a very big impact on the time needed to create or extend files. Suppose you copy a 100MByte file and the biggest chunks of free space are only 5 MBytes. Not only does the new file end up fragmented into 20 pieces, but also the file system has lots of work to do finding all the space and managing the metadata to create the new file.

[ChrisGreaves]

I understand the concept, but I can't see the practical application.

You got it! (I was posting late at night after a long day; my brain was defragmented!)
I think of FreeSpace as a (usually) large file "owned" by the operating system.
It is, after all, a collection of chunks of disk space, just as is any of the 1,000 files "owned" by us.

Many people treat freespace as something mystical. It ain't.

It's a "file", just like any other file, only that you and I can't really do anything with it.

The system can use it as a sandbox for defragmenting files and, of course, creating new files in a blissfully unfragmented space.

Here's a thought-experiment for you, since you have shown us that you're good at thinking:

Whatever the units of writing are (bytes, sectors, clusters etc.) imagine that the freespace is fragmented so that freespace has no two units together. That's a extremely fragmented freespace, right?
Now imagine writing a user-owned file of 3 units into that free space.
That file is going to occupy 3 units of free space.
But by our definition, those three units are non-contiguous, so it follows that the new file is going to be instantly defrgmented, right?

"Should've defrgamented free space first".

I truly hope this helps.

p.s. For an illustration of what fragmentation can do to your double-bass, see this post

[ChrisGreaves]

A couple of things to remember.

I concur with all of this.
I hold that:-
(1) Modern machines with fast drives (7,200 rpm), smaller-diameter disks (reduced seek time), denser packing and 2GB RAM for a day-long disk cache are not affected with read-write times anywhere near as much as the DOS-based PC of 1980.
(2) Sun Microsystems did a study 10(?) years ago that showed that 95%(?) of files are never referenced after 5(?) days are elapsed. My question-marks because I can't find the survey to quote to y'all.
(3) Once a read-only file (e.g. part of the OS or applications, but especially large DLLs) has been written to disk, it can't get defragmented! So defragging ferespace prior to a machine update makes the most sense.
(4) For most people, defragging is something they can do that looks useful when things aren't going right. "Poke it with a stick and see if that makes it move" is the zoo-like analogy.

[ChrisGreaves]

... I found literally dozens and dozens of fragmented Firefox (browser) profile files (the provenance of which were and remain unknown to me), I speculated that Firefox was perhaps either bloating, seeking, or in some manner slowing down because of them.

I'd like to posit that we are now moving into a different realm of defragmentation, but a more important one.
I develop VBA applications, especially in Microsoft Word. Word is famous for corrupted templates, to the extent that a rather nifty little utility Robe Bovey's code cleaner has sprouted. In essence, the bloated code module files are written out to TXT files then read back in, greatly reducing the size of the code and eliminating the numerous links that chain over deleted code fragments in Word/VBA's scheme of maintaining source code.
It sounds as if FireFox too might suffer from some form of bloating as changed options are chained-around and the new values written to the end of the file.
In both cases, I feel that as the chaining increases, so does the risk of making the program code trip up. If there is any form of weakness in the Word/VBA code handler, excessive chaining will find it over geological time.
Firefox probably doesn't have anything like Windows's disk cache, and so as the profile files become convoluted, perhaps Firefox struggles on an instance-by-instance to resolve chains around previous values?

Both examples (Word/VBA and Firefox) could be said to be part of a meta-problem of defragmentation, but the Word/VBA and Firefox examples are more likely to cost us than the disk defragmentation, Im My Humble Opinion.