How best to partition a large HDD

[zxcvbob]

Now that my spare desktop computer (I mostly use it for testing, and doing World Community Grid projects) is up and running with a 2TB disk drive, I'm wondering what the best strategy is for partitioning the drive.  What brought this up (now that the computer is already setup), I tried to install Lotus Smart Suite last week to test if it's compatible with Windows 10.  We need a way to convert some old LWP documents at work to HTML or DOC format.  There's a good story there, maybe I can tell it someday...   Lotus would not install because it says the drive doesn't have enough space .   So I created a 10 GB virtual drive and tried installing on that, and it still failed because now it said my *system* drive doesn't have enough space.   

It installs and seems to run just fine on another W10 machine with a 500GB drive, BTW.

I don't really care about the Lotus software, but I wonder if it makes sense anyway to have a modest system partition, maybe 300 GB, and use the rest as a data partition.  It might make backups easier, and clean reinstalls without touching the data disk.  And I may have other old software that doesn't comprehend drives over 1TB.

[Brad Johnson]

If memory serves there is may be an issue with Win10 not allowing the installer to create a Lotus folder in the root directory. Will it let you change the install folder destination?

Also, I kinda recall something about Lotus having a drive size limitation due to it's FAT32-era origins. If that's the case, you'll have to set up a <32GB partition for it to work. Not certain about that, though, so more checking is needed to verify.

Brad

[Regolith]

For HDD's, the fastest portion of the disk is usually first 30-50% of the drive, so you usually want to partition it so that you have the fastest portion of the drive allocated to your OS and other programs that require it, and the rest for data storage.

Before I replaced it with an SDD, I had a 2TB Seagate Barracuda that I ran a benchmark on, and it showed that performance started to really drop off after around 600GB, so for that drive I allocated the first 600GB for my OS and video games, and used the rest of the drive for storage.

There are several programs you can download that will run a test to benchmark your drive's performance so you can figure out where to put your partitions.  IIRC I used the trial version of HD Tune Pro to run the test.

[cordex]

If you are concerned about performance, buy an SSD and put your OS and applications on that.

I have so far never partitioned a personal hard drive and ultimately decided that it was the right move. There can be cases with servers where it can make sense, but then only rarely and there are usually better options these days.

When I built my most recent computer I bought a decent sized M.2 NVMe drive for OS and applications and a big HDD for data. That gives me the best performance and flexibility, I think.

[zxcvbob]

When I built my most recent computer I bought a decent sized M.2 NVMe drive for OS and applications and a big HDD for data. That gives me the best performance and flexibility, I think.

I've been thinking about adding a small SSD just for the OS and a few games, and putting everything else on the big HDD.  There's an extra drve bay and connectors, and I have a spare 2.5" to 3.5" mounting adapter.  But I use this computer so little, it will take me a while to get around tuit.

[zxcvbob]

I've been thinking about adding a small SSD just for the OS and a few games, and putting everything else on the big HDD.  There's an extra drve bay and connectors, and I have a spare 2.5" to 3.5" mounting adapter.  But I use this computer so little, it will take me a while to get around tuit.

I just ordered a 120GB Intel SSD.  That's a little small, I was looking for a 180GB, but should be enough for a boot drive.

[AZRedhawk44]

Assuming your 2TB drive is a HDD and not an SSD:

Create a C: drive that is somewhere around 250-500GB. 

Create another partition that is 2-4x the amount of RAM you have in the system.  Any drive letter will do, but D: is usually next.  Use this partition for your Windows Virtual Memory / Paging / Swap assignment.

Finally, create a data volume.  E: or whatever letter you like.  This has the rest of the drive's space.

If you ever have to re-install Windows and the C: partition is just hosed, be sure you have all your data you actually like backed up to E:.

Do not install software to C: if you are able to avoid it.  Always install to E:.  Leave C: open to contain all the windows update crap.

The reason the 2nd partition is your virtual memory, is because you want your paging file to be as fast as possible, and volumes on hard drives are faster the closer to the spindle center the data is stored.  Your C: drive will be fastest because it's first, your swap is next, and your data volume is next.

In general, it's bad to have a ginormous C: volume with no secondary volumes, particularly in server administration but the techniques have relevance to desktop management too.

[zxcvbob]

Assuming your 2TB drive is a HDD and not an SSD:

Create a C: drive that is somewhere around 250-500GB.  

Create another partition that is 2-4x the amount of RAM you have in the system.  Any drive letter will do, but D: is usually next.  Use this partition for your Windows Virtual Memory / Paging / Swap assignment.

Finally, create a data volume.  E: or whatever letter you like.  This has the rest of the drive's space.

If you ever have to re-install Windows and the C: partition is just hosed, be sure you have all your data you actually like backed up to E:.

Do not install software to C: if you are able to avoid it.  Always install to E:.  Leave C: open to contain all the windows update crap.

The reason the 2nd partition is your virtual memory, is because you want your paging file to be as fast as possible, and volumes on hard drives are faster the closer to the spindle center the data is stored.  Your C: drive will be fastest because it's first, your swap is next, and your data volume is next.

In general, it's bad to have a ginormous C: volume with no secondary volumes, particularly in server administration but the techniques have relevance to desktop management too.

That's exactly the type of info I was looking for.  Thanks.  :)  How do I tell windoze to use a partition for virtual memory?  Is that an option under the Disk Management tool?  Does it get a drive letter assignment?  (I know you said it did, just double-checking)

ETA: Nevermind about the virtual memory, I think I figured it out already 

[AZRedhawk44]

On windows 10, virtual memory is configured from Control Panel - System - Advanced System Settings (on left side of screen) - Performance Settings - Advanced - Virtual Memory (Change button).

Uncheck the "Automatically manage paging file size for all drives" checkbox.

Select the C: drive paging file and select the "No paging file" radio button, then hit the Set button. 
Select the D: drive and either set a Custom Size 2-4x your amount of RAM (equal to the size of that volume), or just hit System Managed Size radio button, and hit Set.

OK - OK - OK - OK - etc until all the windows go away again.


You don't want paging on C: because it's a dynamically sized component and you don't want to be fighting service pack storage, OS components, .net elements, those annoying apps that won't install anywhere but into C: and so on with paging file requirements.  500GB is plenty for OS, Win10 can be installed onto as small as a 32GB HDD, only needing 16GB for a stripped install and some local unmanaged paging.

You also want paging to be as physically close to all disk sectors that involve paging as possible (i.e. as unfragmented as possible).  Mixing the paging file into the C: drive introduces a lot of fragmentation potential as pagefile.sys is allocated based on swapfile needs and memory pressure.  If the D: drive is reserved for that and is a fixed size and the only place for it, it remains less fragmented and doesn't introduce fragmentation pressure to other files.

[AZRedhawk44]

That's exactly the type of info I was looking for.  Thanks.  :)  How do I tell windoze to use a partition for virtual memory?  Is that an option under the Disk Management tool?  Does it get a drive letter assignment?  (I know you said it did, just double-checking)

ETA: Nevermind about the virtual memory, I think I figured it out already 

As far as whether the 2nd partition needs a D: assignment and NTFS formatting... it's been a while since I've set up a new server.  I don't remember off the top of my head.

If you can get by without assigning a drive letter and formatting, that's better, because it'll stop you (or another user) from accidentally treating that volume like a data volume.  Just not sure if the pagefile can be created and accessed on a raw file system rather than NTFS (and thus an assigned drive letter or mount point).

[AZRedhawk44]

I also don't adhere to a lot of what I'm talking about here since when I spin up new servers, their volumes are on a rather powerful SAN rather than on a single physical SATA HDD.  LUN allocation and RAID level make considerations like the ones I mentioned rather trivial.  Though I still definitely differentiate between OS and Data volumes.

And, all of the laptops I've spec'ed and configured in the last 5 years for my company all have SSD's.  Smaller single C: volume, 1TB or less, so no point diversifying the data volume from the OS volume or swap.  You just run into situations where you choke either the OS or the data volume.

[zxcvbob]

As far as whether the 2nd partition needs a D: assignment and NTFS formatting... it's been a while since I've set up a new server.  I don't remember off the top of my head.

If you can get by without assigning a drive letter and formatting, that's better, because it'll stop you (or another user) from accidentally treating that volume like a data volume.  Just not sure if the pagefile can be created and accessed on a raw file system rather than NTFS (and thus an assigned drive letter or mount point).

I'll probably assign it a very high drive letter (maybe V: ) and then create a fixed pagefile that takes up the entire partition.  Then there's no room left for somebody stupid (like me at some future date) to copy files to it.  Possibly I can make that drive hidden, but I doubt it.  Fun things to try.

[zxcvbob]

I found out how to hide drive partitions from Windows Explorer, but they are still accessable if you know the drive letter:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\Explorer]
"NoDrives"=dword:00000000

"NoDrives" is a bitmap.  00000001 is drive A, 00000002 is B, 00000004 is C, etc, and you OR them toegether to hide multiple drives.

If you have a shiny new (and empty) HDD with just one partition, and you write a file to it (like a fixed pagefile) that goes at the beginning of the disk, right?  And since it's a fixed size it should not get fragmented.  I'm wondering if a separate partition for the pagefile is necessary, or if you just need to create it first.

[AZRedhawk44]

I found out how to hide drive partitions from Windows Explorer, but they are still accessable if you know the drive letter:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\Explorer]
"NoDrives"=dword:00000000

"NoDrives" is a bitmap.  00000001 is drive A, 00000002 is B, 00000004 is C, etc, and you OR them toegether to hide multiple drives.

If you have a shiny new (and empty) HDD with just one partition, and you write a file to it (like a fixed pagefile) that goes at the beginning of the disk, right?  And sense it's a fixed size it should not get fragmented.  I'm wondering if a separate partition for the pagefile is necessary, or if you just need to create it first.

I remember looking at visual representations of disk defragmenter in Win95/98 days.  Large files, even on partitions with enormous amounts of free space, would often be fragmented.  Not sure why.  Also not sure if later versions of NTFS (granted that old Win95 era stuff was FAT) have improved on fragmentation tendencies.

Something else to consider when formatting the various partitions/volumes is block size (allocation unit size).  In particular, when I format a SQL Server data volume, I make my NTFS block size 64KB, because it matches a concept in SQL Server called an extent, which is the smallest unit of storage that can be allocated in that application.  The default NTFS block size today is still 8KB, IIRC.  Maybe they've finally updated that in Win10 and the latest version of Windows server, but I doubt it.

If you save a 128MB file, in 8KB that becomes 16,000 blocks.  Lots of opportunities for those 16,000 blocks to be spread around on the drive wherever the NTFS engine decides to shoehorn them in.

If you save a 128MB file in 64KB blocks that becomes 2000 blocks.  Lot less potential for fragmentation.

Of course, if you deal with a lot of files that are smaller than 64KB, you have a lot of wasted white space that you cannot recover.  Nowadays that isn't much of an issue.  Even a blank MS Word docx file is about 11KB.

[AZRedhawk44]

I misremembered.  NTFS default today is 4KB, not 8KB.

https://www.howtogeek.com/136078/what-should-i-set-the-allocation-unit-size-to-when-formatting/

In general, if you deal with large files on your data volume, you want large allocation unit size.  MP3, video, photo, application installs, etc.

[zxcvbob]

I installed the SSD last night; reinitialized the HDD and installed W'10 on the SSD.  The system boots noticeably faster now (but not as fast as I expected), but the most surprising difference is how much quieter it is.  The HDD was making a lot of noise when the heads or arms were moving, and they were almost constantly moving (seeking, I guess).  No "clicks" or other bad sounds, just a lot of activity.  The OS is on the SSD now and the program and data directories are on the HDD (and the swap file is on the HDD, more on that later), and now all that commotion has stopped.  The computer wasn't really idle, it was running World Community Grid workloads, but all that is still on the HDD.   So I don't know what all the activity was.

I created a 30GB NTFS partition at the beginning of the HDD and put my pagefile on that.  Windows wouldn't let me use all the space available but I used most of it.  Now I keep getting helpful notifications from Explorer that my P: drive is almost full and I should delete unecessary files.  (of course it's almost full)  I dismiss the message, and after a while (maybe when I sign off/on, I don't know) it comes back again.  The P: drive is hidden using that NoDrives thing in regedit, but Explorer still knows it's there.  I turned off all Explorer notifications to make it stop pestering me, but I dunno if that was a good idea.

I initialized both drives to GPT instead of MBR.  Windows Setup didn't like that for the SSD, but it didn't insist that I change it.  Did it not like GPT just because the drive is so small?  It actually recommended GPT for the large drive.  Or perhaps what it didn't like was i was installing on drive 1 instead of drive 0.  (are those number assigned based on which SATA socket on the motherboard the drive is connected to?)

I've been experimenting using the MKLINK /J command to try to trick software installers -- especially Chrome because it insists on using the system drive -- into using my big D: drive but not having a lot of luck.  (MKLINK creates symbolic links, hard links, and something called a junction which I think is just like a hard link but for directories)  I created the junction and it looked okay, but when I ran the chrome installer it failed but didn't tell me what the failure was.  BTW, MKLINK is not available in the Windows Powershell, but it is still in CMD.exe.  So open a Powershell in the directory you want to work in, then type CMD to start the command processor, and now you can use mklink.

[AZRedhawk44]

I installed the SSD last night; reinitialized the HDD and installed W'10 on the SSD.  The system boots noticeably faster now (but not as fast as I expected), but the most surprising difference is how much quieter it is.  The HDD was making a lot of noise when the heads or arms were moving, and they were almost constantly moving (seeking, I guess).  No "clicks" or other bad sounds, just a lot of activity.  The OS is on the SSD now and the program and data directories are on the HDD (and the swap file is on the HDD, more on that later), and now all that commotion has stopped.  The computer wasn't really idle, it was running World Community Grid workloads, but all that is still on the HDD.   So I don't know what all the activity was.

I created a 30GB NTFS partition at the beginning of the HDD and put my pagefile on that.  Windows wouldn't let me use all the space available but I used most of it.  Now I keep getting helpful notifications from Explorer that my P: drive is almost full and I should delete unecessary files.  (of course it's almost full)  I dismiss the message, and after a while (maybe when I sign off/on, I don't know) it comes back again.  The P: drive is hidden using that NoDrives thing in regedit, but Explorer still knows it's there.  I turned off all Explorer notifications to make it stop pestering me, but I dunno if that was a good idea.

I initialized both drives to GPT instead of MBR.  Windows Setup didn't like that for the SSD, but it didn't insist that I change it.  Did it not like GPT just because the drive is so small?  It actually recommended GPT for the large drive.  Or perhaps what it didn't like was i was installing on drive 1 instead of drive 0.  (are those number assigned based on which SATA socket on the motherboard the drive is connected to?)

I've been experimenting using the MKLINK /J command to try to trick software installers -- especially Chrome because it insists on using the system drive -- into using my big D: drive but not having a lot of luck.  (MKLINK creates symbolic links, hard links, and something called a junction which I think is just like a hard link but for directories)  I created the junction and it looked okay, but when I ran the chrome installer it failed but didn't tell me what the failure was.  BTW, MKLINK is not available in the Windows Powershell, but it is still in CMD.exe.  So open a Powershell in the directory you want to work in, then type CMD to start the command processor, and now you can use mklink.

Maybe modify the P: volume so it's large enough that you no longer get the nag message?

Never had a problem with GPT on SSD's.  GPT is for particularly large drives over 2TB, and SSD's don't really fit that profile yet on the current mainstream market, but they're compatible with GPT.

Your motherboard and BIOS will dictate the order of SATA connections, not Windows.  So plug selection does matter for drive 0 and drive 1 assignment.

It's possible to mount a new drive or new partition into the C: drive folder structure (or into any other drive letter/folder), much like a Linux mount point.  For instance, I can create a c:\Users\AZRedhawk44\Multimedia folder, and then format a new drive and mount its empty space at C:\Users\AZRedhawk44\Multimedia rather than a new drive letter like M:.  Anything stored in that folder takes away from empty space on the new drive, not from the C: drive.  I don't have experience with the MKLINK command, but it sounds similar.