Defragmenting FreeSpace - RANT

[ChrisGreaves]

My mind has been pestering about defragmentation for about twenty-five years now.

(1) Disks are so much faster nowadays (reduced diameter, lighter materials, faster heads, better disk I/O driver software, more RAM, better buffering, ...) that the time taken to assemble a file from multiple fragments seems trivial. In this sense defragmentation seems to be a waste of processing time, and even when run at 2a.m. is akin to a government make-work feel-good project.
(2) defragmentation is more wear and tear on the mechanical components, and brings closer the day when the drive dies. In this sense defragmentation can be seen as hastening the death of the drive.

That said, my Excel workbook (Health.xls=125 fragments this morning) bugs the heck out of me. I feel certain that MSExcel uses a shotgun approach to storage. Other applications seem to make better use of FreeSpace (daily2.bmp=2 fragments as a result of a batch file DOS COPY command).

I would have thought that by now an application, whether O/S or user, could request “x bytes storage please”, and be assigned a contiguous block. (How I hated those IBM DOS JCL statements with the disk allocation statements on punched cards; punch ‘em wrong and you clobbered your boss’s memos! Well, I did) Best-fit algorithms have been around since at least 1968, for I remember a CDC-3300 System Engineer working on one. And best-fit algorithms are simple enough to be written in VBA(grin)

I have been playing with Windows7 “defrag t: /u /x” to defragment FreeSpace, but after such a run, Defraggler’s map shows that the FreeSpace is still fragmented across a drive.

Why wouldn’t a defragmentation routine treat FreeSpace as a single file (same logic as a useful file, surely) and make the effort to collect FreeSpace into one contiguous block (OK, skirting the issue of system files) so that applications that did request file space, could be assigned a contiguous block and be un-fragmented at time of creation.

I have lost the reference to the SUN Microsystems report from around 2000AD where they said that some high percentage of files were not accessed after some days - 95% never accessed after 5 days was the idea - which would strengthen the argument for assigning contiguous space, and hence for implementing a better job of defragmenting Free Space.

Defragmenting the folders (“up-stream access”) is an arguably beneficial task.

Cheers
Chris

[HansV]

In recent versions of Windows, there is very little point in using a defragment utility. The OS manages disk space automatically, and performs some defragmenting once a week (if I remember correctly).

[ChrisGreaves]

In recent versions of Windows, there is very little point in using a defragment utility. The OS manages disk space automatically, and performs some defragmenting once a week (if I remember correctly).

[pessimistic] But I bet this new Windows won't allocate a file to a contiguous area.[/pessimistic]

If, as in my example, Excel was coerced ("THEY should do something about it") into locating a chunk of FreeSpace sufficient for the 2MB worksheet, then there would be less need for defragmentation.

Defragmentation would still be needed, but this whole business of fragmenting a file so that it needs to be de fragmented is a make-work project for the computer.

I know I am fighting a lost cause.

Cheers
Chris

[stuck]

I know life's short but the milliseconds I might convert from 'waiting for a fragmented file to load' into time 'in which to do fun things' are not worth the pain and effort of trying to defragment the fragmented file. Consequently I gave up even thinking about defragmenting, let alone doing it, some time back in the XP era.

Ken

[DaveA]

With the defrag of Windows 10, one must leave the machine on, as it is done when there is no action.
So one needs to leave their machine on for a couple days, then it just may get defragged.
We leave ours on 7/24 with no sleep, hibernation or suspend settings turn on.

[Argus]

I agree with you Ken. It is possible that "old code" makes bad decisions how it allocates file data etc. but I've completely left it to the OS. It is possible that there are some rare cases where it might be important (and moving/resizing partitions etc.), but not in my use. The speed of modern CPUs and SSDs (which should not be defragmented), and the OS’s native defrag of HDDs works well together.

(A different thing, related to load time etc. in a "mixed" environment with SSDs and HDDs, is that I am (now) used to the speed of the SSDs, but then the HDDs (at times) take their time to start spinning, more so depending on type (on the other hand the "green" ones et al used for backups etc. are not accessed frequently.)

[ChrisGreaves]

I know life's short but the milliseconds I might convert from 'waiting for a fragmented file to load' into time 'in which to do fun things' are not worth the pain and effort of trying to defragment the fragmented file. Consequently I gave up even thinking about defragmenting, let alone doing it, some time back in the XP era.

Hi Ken, I quite agree with your time analysis. I spend more time reading (let alone two-fingered typing) emails to warrant the savings via defragmentation.

I am more concerned with my logic/thinking.

I can't come up with any other explanation except that (in my case) Excel is served 200 fragments for its file, whereas COPY is served just one fragment.
I can't work out why Windows should serve differently.

For years I've wondered why C:\Windows *.DLL and *.EXE get defragmented.
Once I have installed and defragmented, I don't expect DLL & EXE to be fragmented unless I download updates.
To me this seems as if stable files are being shunted around the hard drive AN D getting fragmented in the process.

In short, those of us (not me) who feel the need to defragment are doing so because Windows is making matters worse!

Cheers
Chris

[ChrisGreaves]

With the defrag of Windows 10, one must leave the machine on, as it is done when there is no action. So one needs to leave their machine on for a couple days, then it just may get defragged. We leave ours on 7/24 with no sleep, hibernation or suspend settings turn on.

I am by now jaded enough to see the Windows Logic in this:-
If you are working so hard that your machine doesn't get a rest, then we won't defragment and ultimately something will go wrong and you'll have to restore from your backup, and if nothing else, that will give you the break from work you obviously need so much.
Cheers
Chris

[StuartR]

Disk fragmentation is not a significant issue. Even if a file has hundreds of fragments this is not likely to cause significant performance issues.

[ChrisGreaves]

Disk fragmentation is not a significant issue. Even if a file has hundreds of fragments this is not likely to cause significant performance issues.

Basically, yes, I agree, although that Excel2003 sheet takes a couple of seconds to load from its (sometimes) 200 fragments

I think I am more annoyed by Windows7HP not assigning Free Space.
My largest contiguous block of Free Space is just under 4GB.
It seems strange to me to see a workbook scattered amongst 200 fragments when there is a dollop of Free Space available.

Cheers
Chris

[JoeP]

If you want to try to get the file into fewer fragments try this, Leave the current file alone. Try using "Save As" giving the file a new name. Check the fragmentation. If you like what happened dekete the original file and then rename the "new" file to the original name.

[ChrisGreaves]

If you want to try to get the file into fewer fragments try this, Leave the current file alone. Try using "Save As" giving the file a new name. Check the fragmentation. If you like what happened dekete the original file and then rename the "new" file to the original name.

Hi Joe. Good suggestion.
But ... (grin!)
If the user can save a file as a single chunk (not fragmented) by issuing a File, saveAs, why can't the operating system save a file as a single chunk (not fragmented) when the user issues a File, Save?

I am pretty sure that I could defragment ALL files on my hard drive by reformatting the drive and using RoboCopy to re-create the data, but when I have Acres And Acres of Free Space, why fragment files needlessly?
Cheers
Chris

[JoeP]

"Save As" is a new file and the OS looks for a place to hold it all. "Save" is replace if it already exists. My guess is that when the file already exists it is more costly in time and overhead to delete the "old" file and save the "new" file than to just extend the file as needed.

[ChrisGreaves]

"Save As" is a new file and the OS looks for a place to hold it all. "Save" is replace if it already exists. My guess is that when the file already exists it is more costly in time and overhead to delete the "old" file and save the "new" file than to just extend the file as needed.

Yes.
But.
Up to and over 200 fragmented segments can't be all that much more efficient, and it does just delay the work to be done in defragmenting.

Your argument/methods for the user being able to save as a complete file are valid; but it still leaves me with the question of why the o/s can't do this for me.

Cheers
Chris

[ChrisGreaves]

If you want to try to get the file into fewer fragments try this, Leave the current file alone. Try using "Save As" giving the file a new name. Check the fragmentation. If you like what happened dekete the original file and then rename the "new" file to the original name.

So then last night I had an idea:
Each month I save my monthly financial workbook (for December: 2019_12.xls) with a new name “Trial.xls” and do a trial balance before the pension comes in. This gives me a pre-processor check on account balances, so that on pension day I can quickly reconcile my accounts (and breathe my monthly sigh of relief!).
I had entered the day’s disbursements and issued a File, Save, then thought “Why don’t I try Joe’s idea and issue a saveAs?”, so I immediately issued a File, saveAs and saved a fresh copy of 2019_12.xls (Note: NOT the original workbook Health.xls with which I started this thread) in my daily blotter folder which was T:\Blotter\201912187. I can rename this second copy to Trial.xls later …

Defraggler.png

There is dear old Health.xls with its (by now customary) 125 fragments, and then 2019_12.xls in its proper folder Admin\2019, and immediately after it the saveAs copy in the blotter folder.
Both 2019_12 workbooks have 25 fragments.
?!!???
This suggests to me that regardless of Save/saveAs from Excel, Excel decides that this specific workbook can be written to disk in a series of twenty-five ouput/writing commands, and so Excel issues twenty-five separate calls to Windows (Supervisor? Kernel? IO-Control System, whatever deals with hard drive communication), and that part of the o/s does NOT receive a call to write 25 chunks to disk, let alone a call to write 382KB to disk, but receives 25 separate calls to write 25 apparently unrelated chunks to disk.
That is, Windows is not given a command “write this 382KB to disk”, but a series of commands “write these 2KB to disk”, and then a Windows heartbeat later “write these 17KB to disk”, and later “write these 13KB to disk” and so on.

It seems that in this specific case, saveAs was not able to issue a call that Windows could have used to grab a 382KB defragmented space.
Rather, perhaps, Windows was as smart as it could be (We’ll never know!) and found a chunk of free space exactly 2KB size and made good use of it; (“that’s one less chunk of free space floating around!”) And then Windows found a chunk of free space exactly 17KB; (“that’s one less chunk of free space floating around!”) Then a chunk of free space exactly 13KB; (“that’s one less chunk of free space floating around!”) and so on.

So maybe the culprit is un-cooperative Excel2003, saying that it needs twenty-five separate file-chunks written to disk, and the culprit is not Windows maliciously willfully fragmenting the 382KB file into 25 pieces.

Assuming that I make no major structural changes to my financial workbook, I predict that for the next few months, Save/saveAs will produce 25 fragments regardless of how I save it, or how frequently I save it (typically once per day).
Cheers
Chris

P.S. There might be significance in my (original) Health.XLS nowadays sticking at 125 segments, no more, no less. Years ago I used to record my weight each morning (Hence “health”) but now I record the electricty meter reading and the living-room temperature each morning (economic health I suppose). Perhaps I should include a “segments” reading each day and see if/when Health.xls makes a quantum leap.

P.P.S.

Defraggler02.png

So then I thought that this thread is surely worth a masters, even a PhD perhaps.
I loaded Excel2003 and immediately saved the Empty00 workbook, shut down Excel.
Repeated but this time Empty01 included a single cell with two words of text “Almost Empty”.
Repeated but this time Empty02 was populated three cells with an age calculation.
All three workbooks occupy four fragments on my hard drive.
Your challenge: Repeat this experiment with whatever version of Excel YOU use and let me know how many fragments for each of the three workbooks.
FWIW my screen snapshots are from Defraggler 2.19.982(64 bit) Free version.

P.P.P.S I think it is probably too early in the game to challenge you to reduce my attached three workbooks to single-segment files.
Chris

[JoeP]

Excel does not decide where to physically write the file. Windows does that. I guess Windows just picks an available free space chunk and starts there. If the file fits fine. If not you get fragments.

I won't try your test because I have an SSD and fragmentation generally doesn't matter with an SSD. I just let Windows do its thing.

[ChrisGreaves]

I have worked out an explanation that satisfies me but is probably incorrect:
My workbook is saved as 18 fragments consistently.
Suppose six sheets in a workbook, each sheet containing numeric data, calculated expressions (formulas), and formatting.
Excel writes the workbook out in assembled chunks:-

First: Excel writes the numeric data of worksheet 1 to disk; Excel sends a chunk of data to Windows saying “Please write this to disk”, and Windows honours the request. A single chunk of data, representing the numeric data in Sheet1, is written to disk.
Second: Excel writes the formulas of worksheet 1 to disk; Excel sends a chunk of data to Windows saying “Please write this to disk”, and Windows honours the request. A second chunk of data, representing the numeric data in Sheet1, is written to disk.
Third: Excel writes the formatting of worksheet 1 to disk; Excel sends a chunk of data to Windows saying “Please write this to disk”, and Windows honours the request. A third chunk of data, representing the formatting in Sheet1, is written to disk.
…
Eighteenth: Excel writes the formatting of worksheet 6 to disk; Excel sends a chunk of data to Windows saying “Please write this to disk”, and Windows honours the request. An eighteenth and final chunk of data, representing the formatting in Sheet6, is written to disk.

I don’t say that this is what happens, but I suspect that there is some sort of breakdown like this, and that Windows might be doing the best it can, given that it is handed eighteen apparently unrelated “write” tasks to perform, each request coming from an unseen client at the bottom of the management tree.

Cheers
Chris