Linux and . files

[zahc]

I bought a new hard drive. I wanted to migrate to it. I installed a fresh installation of Ubuntu on it, with a separate home partition.

I booted my old hard drive.

I mounted my new, freshly installed hard drive's home partition to /newdrive.

I ran rm -r /newdrive/chaz/*. I wanted to remove all the empty directories like Music, Videos, etc.

I ran cp -r /home/chaz/* /newdrive/chaz/. This should copy everything but the .whatever configuration files, right?

When this finished running, I ran

du -h /home/chaz
782G
du -h /newdrive/chaz/
773G

Why would they be that much different? My first guess was that my old install had tons of .whatever configuration files, where my new install was pretty clean. But how in the world could there be 9GB of them? What else could account for the differences?

[Harold Tuttle]

maybe the format or block size had a different minimum file size

[zahc]

Well, they are both ext4 file systems, but the one I'm migrating FROM (the larger one) was encrypted and the other one wasn't. I guess that could account for it; I knew encryption had a performance overhead but I didn't know there was a spatial overhead.

[CNYCacher]

Some of the fancier file managers use a .dir folder in your home drive for the "Trash"

[lee n. field]

I bought a new hard drive. I wanted to migrate to it. I installed a fresh installation of Ubuntu on it, with a separate home partition.

I booted my old hard drive.

I mounted my new, freshly installed hard drive's home partition to /newdrive.

I ran rm -r /newdrive/chaz/*. I wanted to remove all the empty directories like Music, Videos, etc.

I ran cp -r /home/chaz/* /newdrive/chaz/. This should copy everything but the .whatever configuration files, right?

...

Eh?  Not the way I'd do it.

[zahc]

Well, I'm open to suggestions. I want to remove my home directory encryption, and I want to add the capacity of the new hard drive to my system. There's no way to un-encrypt a file system unless you have an equal-sized place to copy it all to. That's why I decided I would copy it to the new hard drive, and then erase the old, encrypted one.

I never found a .trash file, by the way, so I still don't know where the 9GB went.

[roo_ster]

Well, I'm open to suggestions. I want to remove my home directory encryption, and I want to add the capacity of the new hard drive to my system. There's no way to un-encrypt a file system unless you have an equal-sized place to copy it all to. That's why I decided I would copy it to the new hard drive, and then erase the old, encrypted one.

I never found a .trash file, by the way, so I still don't know where the 9GB went.

If that is indeed true of your implementation, your implementation is awful.  Most I have used can be started, stopped, unencrypted, whatever with no big whoop.  Kinda like defragging a Windows drive.

[zahc]

Well, I'm using the default encryption scheme that comes with ubuntu. The few instructions I have found for removing the encryption all involve moving your data (or at least backing it up for safety) to another location, and my hard drive is full, so there is nowhere to move it.

[CNYCacher]

If that is indeed true of your implementation, your implementation is awful.  Most I have used can be started, stopped, unencrypted, whatever with no big whoop.  Kinda like defragging a Windows drive.

I don't see how an encryption system worth using could possibly be decrypted on the fly.  Unless you are dynamically growing and shrinking the partitions and filesystems the entire time, as well as needing some seed space to get started

[tyme]

Probably cache directories and temp files for your web browser and other things.

#!/bin/zsh
du -s ~/.*(/) | sort -n

A better way to mirror a directory tree is to use rsync, although it may not work with extended attributes or other exotic features depending on the filesystems involved.

rsync -avn /home/chaz /newdrive/
means "replicate the directory named chaz into the directory /newdrive/

alternately,
rsync -avn /home/chaz/ /newdrive/chaz/

adding a / after chaz means "replicate the contents of the directory chaz (into the directory /newdrive/chaz/).  Adding a / to the first command line would turn /newdrive/ into your new home dir, rather than what you want, which is to make /newdrive/chaz/ your new home dir.

Drop the n to do it for real (-av), and add --progress to show progress bars, and add --delete to delete files at the destination when they don't exist at the source.

I don't see how an encryption system worth using could possibly be decrypted on the fly.

http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation#Counter_.28CTR.29

You're probably thinking of the traditional CBC mode that works well only for data streams.  And ECB mode which is simply horrid.

I suppose an encrypted filesystem could theoretically use extended attributes to mark files as encrypted or not, and decrypt files one at a time so as not to require enough space to replicate all the files at once, but there are probably policy reasons for not doing that.  It would mean users couldn't easily tell which files are and aren't encrypted without using special tools to check extended attributes.  Most people probably want to assume that if they mount an encrypted filesystem, all files are encrypted all the time.

[zahc]

I don't want to copy the . configuration files to the new hard drive. Doing that broke my desktop and I couldn't get it fixed by deleting .gconf, .gnome and .gnome2. I'm content to start over will all my settings. Will rsync -av copy the . hidden directories? I RTFMP but it wasn't explicit as to whether it did or not.

[roo_ster]

rsync FTW, when it comes to backing up data manually or in a script.

zahc:

tyme is showing you how to determine which dot directories are largest and if any could possibly be the culprit, not how to copy them.

#!/bin/zsh
du -s ~/.*(/) | sort -n

I'd add a "-h" to the du command, but I don't need them too accurate and I like K, M, G indicators.

tyme:

What makes you desire to call out zsh?

<snoops around for zsh factoids>

(Relative to my preferred tcsh)
Sharing of command history among all running shells
Floating point arithmetic
Scientific notation
Better arrays

Looks neat, but I suppose its scripting syntax is a derivative of bourne & bash.  I despise sh/bash script syntax.

I get around tcsh's arithmetic limitations with one alias, so it hasn't been a limitation I care about.

[KD5NRH]

Well, I'm using the default encryption scheme that comes with ubuntu. The few instructions I have found for removing the encryption all involve moving your data (or at least backing it up for safety) to another location, and my hard drive is full, so there is nowhere to move it.

Just pipe everything through uuencode and send it to the printer.  You can reenter it after you get the drive ready.

[zahc]

Of course I would never do that; I would have my wife reenter it.

[tyme]

I'd add a "-h" to the du command, but I don't need them too accurate and I like K, M, G indicators.

tyme:

What makes you desire to call out zsh?

The -h would screw up sorting.

zsh allows *(/) to match only directories, *(.) to match files, and *(*) to match executables.  I'm not aware of how to replicate that behavior in bash.  *(/) simplifies the output.

[CNYCacher]

The -h would screw up sorting.

zsh allows *(/) to match only directories, *(.) to match files, and *(*) to match executables.  I'm not aware of how to replicate that behavior in bash.  *(/) simplifies the output.

I like this one:

sudo apt-get install filelight
filelight .

[Nitrogen]

if both are filesystems, use dump/restore.

[roo_ster]

The -h would screw up sorting.

zsh allows *(/) to match only directories, *(.) to match files, and *(*) to match executables.  I'm not aware of how to replicate that behavior in bash.  *(/) simplifies the output.

D'oh.

Hmm, that is nice behavior.

[CNYCacher]

D'oh.

Hmm, that is nice behavior.

Indeed.

It's one of those "I didn't know I wanted it until I realized I didn't have it" kinda things. . .