I’m using Amazon SSM Agent for connecting to EC2 instance via securely. Amazon SSM Agent is also able to connect to the EC2 instance console via aws.amazon.com web console. So, this is my preferred way.
About a few days ago, when I tried to connect to EC2 via Amazon SSM Agent it doesn’t respond and after that, I decided to connect via a regular way, yes with SSH.
Of course, first of all, I want to restart Amazon SSM Agent and I see this:
[root@i-0cd9514c60d532e78 ~] systemctl restart amazon-ssm-agent.service Error: No space left on device
I didn’t wait for this error message about disk status. And want to check disk usage. Now, I see this:
[root@i-0cd9514c60d532e78 ~] df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 960M 0 960M 0% /dev tmpfs 978M 0 978M 0% /dev/shm tmpfs 978M 420K 978M 1% /run tmpfs 978M 0 978M 0% /sys/fs/cgroup /dev/nvme0n1p1 8.0G 2.6G 5.5G 32% / tmpfs 196M 0 196M 0% /run/user/1000
Oops! It looks like, this error message is not about disk usage but I have to make sure. Now, time is checking inodes.
[root@i-0cd9514c60d532e78 ~] df -i Filesystem Inodes IUsed IFree IUse% Mounted on devtmpfs 245750 304 245446 1% /dev tmpfs 250254 2 250252 1% /dev/shm tmpfs 250254 388 249866 1% /run tmpfs 250254 16 250238 1% /sys/fs/cgroup /dev/nvme0n1p1 4193216 67672 4125544 2% / tmpfs 250254 1 250253 1% /run/user/1000
Now that I could confirm it is not inodes too I did further research and I found there exist known bug reports for this same issue where systemctl action of “start”, “stop” and “reload” will show a “No space left on device” error, but actions of “enable”, “disable” and “kill” do not. In the bugs, it is reported that this error occurs is a result of the inotify “max_user_watches” limit. inotify has a limit on the number of files and directories it can monitor for changes. To silence the error you need to increase the value of “/proc/sys/fs/inotify/max_user_watches” to allow more files and directories to be added and monitored for changes.
To change the value in realtime you can run:
echo 1048576 > /proc/sys/fs/inotify/max_user_watches
To persistently loads this same increased value after a reboot or stop/start action is performed:
Add line “
/etc/sysctl.conf You can check your sysctl configuration with
sysctl -p command.