I’m using Amazon SSM Agent for connecting to EC2 instance via securely. Amazon SSM Agent is also able to connect to the EC2 instance console via aws.amazon.com web console. So, this is my preferred way.
About a few days ago, when I tried to connect to EC2 via Amazon SSM Agent it doesn’t respond and after that, I decided to connect via a regular way, yes with SSH.
Of course, first of all, I want to restart Amazon SSM Agent and I see this:
[[email protected] ~] systemctl restart amazon-ssm-agent.service
Error: No space left on deviceI didn’t wait for this error message about disk status. And want to check disk usage. Now, I see this:
[[email protected] ~] df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 960M 0 960M 0% /dev
tmpfs 978M 0 978M 0% /dev/shm
tmpfs 978M 420K 978M 1% /run
tmpfs 978M 0 978M 0% /sys/fs/cgroup
/dev/nvme0n1p1 8.0G 2.6G 5.5G 32% /
tmpfs 196M 0 196M 0% /run/user/1000Oops! It looks like, this error message is not about disk usage but I have to make sure. Now, time is checking inodes.
[[email protected] ~] df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
devtmpfs 245750 304 245446 1% /dev
tmpfs 250254 2 250252 1% /dev/shm
tmpfs 250254 388 249866 1% /run
tmpfs 250254 16 250238 1% /sys/fs/cgroup
/dev/nvme0n1p1 4193216 67672 4125544 2% /
tmpfs 250254 1 250253 1% /run/user/1000Now that I could confirm it is not inodes too I did further research and I found there exist known bug reports for this same issue where systemctl action of “start”, “stop” and “reload” will show a “No space left on device” error, but actions of “enable”, “disable” and “kill” do not. In the bugs, it is reported that this error occurs is a result of the inotify “max_user_watches” limit. inotify has a limit on the number of files and directories it can monitor for changes. To silence the error you need to increase the value of “/proc/sys/fs/inotify/max_user_watches” to allow more files and directories to be added and monitored for changes.
To change the value in realtime you can run:
echo 1048576 > /proc/sys/fs/inotify/max_user_watchesTo persistently loads this same increased value after a reboot or stop/start action is performed:
Add line “fs.inotify.max_user_watches=1048576” to /etc/sysctl.conf You can check your sysctl configuration with sysctl -p command.
References:
https://bugzilla.redhat.com/show_bug.cgi?id=894483
https://bugzilla.redhat.com/show_bug.cgi?id=1452933
Written By
For some reason, you have to extend your ec2 Linux server with our reboot and securely. In this port, you will learn how you can extend your ec2 Linux disk on aws.In this example, you will see extend the disk size from 80GiB to 120GiB.Step 1: Modify VolumeLogin to your AWS Console and find your ec2's volume (disk). Here is the tip; in the left pane, there is an Elastic Block Store > Vol...
Today, I've played with AWS RDS with MariaDB. In the normally, AWS is created just one user for access AWS RDS. Here is the steps for create a new grant user on AWS RDS with MariaDB.mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, PROCESS, REF...
If you are serving static content directly Amazon S3 via CloudFront (CDN) you should probably get an error about fonts, images or other static contents to deliver for your users. Actually, it’s not complicated. You can fix this issue on your Amazon S3 Bucket CORS...