I tried to use logrotate.conf to automatically clear out some big log files, but ended up both losing the space the files were using and not being able to get it back.
First problem is how to see how the disk is being used. There are programs to do this called xdu (which I couldn’t get compiled, though I didn’t try very hard) and xdiskusage, which I ended up using. The output of xdiskusage is this:
Notice the 12.86GB at permission denied. This is disk space that is being used, but not associated with any filesystem. To find out if this space is due to deleted files, use lsof.
[root@uct3-edge3 maryh]# lsof|grep deleted sh 15711 root 1w REG 8,2 8525310901 2541174 /var/log/dCacheDomain.log.2 (deleted) sh 15711 root 2w REG 8,2 8525310901 2541174 /var/log/dCacheDomain.log.2 (deleted) java 15715 root 1w REG 8,2 8525310901 2541174 /var/log/dCacheDomain.log.2 (deleted) java 15715 root 2w REG 8,2 8525310901 2541174 /var/log/dCacheDomain.log.2 (deleted) sh 15902 root 1w REG 8,2 5257653371 2541172 /var/log/httpdDomain.log.2 (deleted) sh 15902 root 2w REG 8,2 5257653371 2541172 /var/log/httpdDomain.log.2 (deleted) java 15906 root 1w REG 8,2 5257653371 2541172 /var/log/httpdDomain.log.2 (deleted) java 15906 root 2w REG 8,2 5257653371 2541172 /var/log/httpdDomain.log.2 (deleted)
If I had accidentally deleted these files and wanted to recover them, I could run this command:
$ cp /proc/15906/fd/1 myfile
This command would have to be run for each file. The 15906 is the number from the second column in the lsof command. And the 1 is the number from the fourth column (1w or 2w). Make sure you don’t run cp -a because that won’t work.
I, however, just want to recover the space from the deleted files. I can either reboot or restart the process that was writing to the file.