Error: No space left on the device when starting/stopping services only

I’m using Amazon SSM Agent for connecting to EC2 instance via securely. Amazon SSM Agent is also able to connect to the EC2 instance console via aws.amazon.com web console. So, this is my preferred way.

About a few days ago, when I tried to connect to EC2 via Amazon SSM Agent it doesn’t respond and after that, I decided to connect via a regular way, yes with SSH.

Of course, first of all, I want to restart Amazon SSM Agent and I see this:

[[email protected] ~] systemctl restart amazon-ssm-agent.service
Error: No space left on device

I didn’t wait for this error message about disk status. And want to check disk usage. Now, I see this:

[[email protected] ~] df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 960M 0 960M 0% /dev
tmpfs 978M 0 978M 0% /dev/shm
tmpfs 978M 420K 978M 1% /run
tmpfs 978M 0 978M 0% /sys/fs/cgroup
/dev/nvme0n1p1 8.0G 2.6G 5.5G 32% /
tmpfs 196M 0 196M 0% /run/user/1000

Oops! It looks like, this error message is not about disk usage but I have to make sure. Now, time is checking inodes.

[[email protected] ~] df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
devtmpfs 245750 304 245446 1% /dev
tmpfs 250254 2 250252 1% /dev/shm
tmpfs 250254 388 249866 1% /run
tmpfs 250254 16 250238 1% /sys/fs/cgroup
/dev/nvme0n1p1 4193216 67672 4125544 2% /
tmpfs 250254 1 250253 1% /run/user/1000

Now that I could confirm it is not inodes too I did further research and I found there exist known bug reports for this same issue where systemctl action of “start”, “stop” and “reload” will show a “No space left on device” error, but actions of “enable”, “disable” and “kill” do not. In the bugs, it is reported that this error occurs is a result of the inotify “max_user_watches” limit. inotify has a limit on the number of files and directories it can monitor for changes. To silence the error you need to increase the value of “/proc/sys/fs/inotify/max_user_watches” to allow more files and directories to be added and monitored for changes.

To change the value in realtime you can run:

echo 1048576 > /proc/sys/fs/inotify/max_user_watches

To persistently loads this same increased value after a reboot or stop/start action is performed:

Add line “fs.inotify.max_user_watches=1048576” to /etc/sysctl.conf You can check your sysctl configuration with sysctl -p command.

References:
https://bugzilla.redhat.com/show_bug.cgi?id=894483
https://bugzilla.redhat.com/show_bug.cgi?id=1452933

Conclusion