As a Linux administrator, I recently encountered a situation where one of my servers was displaying low disk utilization. To rectify the issue, I decided to employ a strategy of decluttering by removing any redundant files, including expired logs.
In order to accomplish this task, I first established a secure remote connection to the server using ssh. Then, I embarked on the following series of steps to reclaim the precious disk space and restore optimal performance to my Linux system.
1. Show the disk space information.
df -h
# Filesystem Size Used Avail Use% Mounted on
# /dev/vda1 40G 33G 4.5G 89% /
# devtmpfs 1.9G 0 1.9G 0% /dev
# tmpfs 1.9G 0 1.9G 0% /dev/shm
# tmpfs 1.9G 484K 1.9G 1% /run
# tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
# tmpfs 380M 0 380M 0% /run/user/1005
# tmpfs 380M 0 380M 0% /run/user/0
I discovered that the server had only 4.5GB of disk space remaining. This prompted me to take immediate action and commence the process of freeing up disk space by eliminating unnecessary files.
2. Find out the biggest files and directories.
du -Sh / | sort -rh | head -n 15
# du: cannot access ‘/proc/13476/task/13476/fd/4’: No such file or directory
# du: cannot access ‘/proc/13476/task/13476/fdinfo/4’: No such file or directory
# du: cannot access ‘/proc/13476/fd/4’: No such file or directory
# du: cannot access ‘/proc/13476/fdinfo/4’: No such file or directory
# 1.3G /usr/local/mysql/bin
# 1008M /usr/local/mysql/lib
# 622M /root/oneinstack/src
# 497M /var/log/journal/a9fe31fd96c841a5a16f9c54110b66cc
# 174M /boot
# 121M /data/mysql
# 114M /usr/lib64
# 114M /usr/bin
# 102M /usr/lib/locale
# 86M /usr/java/jdk1.7.0_80/jre/lib
# 76M /usr/local/cloudmonitor/jre/lib/i386
What’s wrong ????
There seems no big files or directories. (It’s impossible according to my experience!!!)
3. Find out the data that was deleted but not released by FileSystem.
lsof | grep '(deleted)'
# gunicorn 21228 root 7u REG 253,1 0 665601 /tmp/wgunicorn-AcuOTP (deleted)
# gunicorn 21230 root 7u REG 253,1 0 665590 /tmp/wgunicorn-ddaaqtnp (deleted)
# gunicorn 21230 root 8u REG 253,1 0 665593 /tmp/wgunicorn-lpv_r06g (deleted)
# gunicorn 21230 root 9u REG 253,1 0 665596 /tmp/wgunicorn-kqmta562 (deleted)
# gunicorn 21230 root 10u REG 253,1 0 665603 /tmp/wgunicorn-fnusav1e (deleted)
# nginx 21495 root 5w REG 253,1 26287067976 1709016 /var/log/nginx/access.log (deleted)
# nginx 21496 root 5w REG 253,1 26287067976 1709016 /var/log/nginx/access.log (deleted)
# nginx 21497 root 5w REG 253,1 26287067976 1709016 /var/log/nginx/access.log (deleted)
# gunicorn 27996 root 6u REG 253,1 0 665594 /tmp/wgunicorn-7CgVKM (deleted)
# gunicorn 27996 root 7u REG 253,1 0 665601 /tmp/wgunicorn-AcuOTP (deleted)
It shows that the issue was caused by a failure to properly purge nginx logs from the system.
4. Restart nginx process to release the logs.
systemctl restart nginx
5. Confirm the storage is released.
df -h
# Filesystem Size Used Avail Use% Mounted on
# /dev/vda1 40G 8.4G 29G 23% /
# devtmpfs 1.9G 0 1.9G 0% /dev
# tmpfs 1.9G 0 1.9G 0% /dev/shm
# tmpfs 1.9G 484K 1.9G 1% /run
# tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
# tmpfs 380M 0 380M 0% /run/user/1005
# tmpfs 380M 0 380M 0% /run/user/0