Skip to content

Ercan Ermis

notes for everyone

Menu
  • AWS
  • Kubernetes
  • Linux
  • DevOps
  • Docker
  • GCP
  • Uncategorized
  • Contact Me
Menu

Error: No space left on the device when starting/stopping services only

Posted on March 1, 2020April 24, 2021 by Ercan

I’m using Amazon SSM Agent for connecting to EC2 instance via securely. Amazon SSM Agent is also able to connect to the EC2 instance console via aws.amazon.com web console. So, this is my preferred way.

About a few days ago, when I tried to connect to EC2 via Amazon SSM Agent it doesn’t respond and after that, I decided to connect via a regular way, yes with SSH.

Of course, first of all, I want to restart Amazon SSM Agent and I see this:

[[email protected] ~] systemctl restart amazon-ssm-agent.service
Error: No space left on device

I didn’t wait for this error message about disk status. And want to check disk usage. Now, I see this:

[[email protected]c60d532e78 ~] df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 960M 0 960M 0% /dev
tmpfs 978M 0 978M 0% /dev/shm
tmpfs 978M 420K 978M 1% /run
tmpfs 978M 0 978M 0% /sys/fs/cgroup
/dev/nvme0n1p1 8.0G 2.6G 5.5G 32% /
tmpfs 196M 0 196M 0% /run/user/1000

Oops! It looks like, this error message is not about disk usage but I have to make sure. Now, time is checking inodes.

[[email protected] ~] df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
devtmpfs 245750 304 245446 1% /dev
tmpfs 250254 2 250252 1% /dev/shm
tmpfs 250254 388 249866 1% /run
tmpfs 250254 16 250238 1% /sys/fs/cgroup
/dev/nvme0n1p1 4193216 67672 4125544 2% /
tmpfs 250254 1 250253 1% /run/user/1000

Now that I could confirm it is not inodes too I did further research and I found there exist known bug reports for this same issue where systemctl action of “start”, “stop” and “reload” will show a “No space left on device” error, but actions of “enable”, “disable” and “kill” do not. In the bugs, it is reported that this error occurs is a result of the inotify “max_user_watches” limit. inotify has a limit on the number of files and directories it can monitor for changes. To silence the error you need to increase the value of “/proc/sys/fs/inotify/max_user_watches” to allow more files and directories to be added and monitored for changes.

To change the value in realtime you can run:

echo 1048576 > /proc/sys/fs/inotify/max_user_watches

To persistently loads this same increased value after a reboot or stop/start action is performed:

Add line “fs.inotify.max_user_watches=1048576” to /etc/sysctl.conf You can check your sysctl configuration with sysctl -p command.

References:
https://bugzilla.redhat.com/show_bug.cgi?id=894483
https://bugzilla.redhat.com/show_bug.cgi?id=1452933

Conclusion

Share on Social Media
twitter facebook linkedin reddit

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recent Posts

  • What is an Internet Gateway in AWS?
  • What are Route Tables on AWS VPC?
  • What is Subnet on AWS VPC?
  • What is AWS VPC?
  • Prevent nginx from caching DNS for proxy
  • Scaling PHP Applications on AWS
  • Create an S3 Bucket and Set a Policy via CLI
  • Issue a Let’s Encrypt SSL with the AWS Route53
  • Automate Let’s Encrypt SSL on AWS Application Load Balancer
  • Let’s Learn Kubernetes – Part 3
  • Deploy a website to S3 and CloudFront with Bitbucket Pipelines
  • Let’s Learn Kubernetes – Part 2
  • Protect your AWS Account with specified IPs
  • Let’s Learn Kubernetes – Part 1
  • Differences between AWS CLI v1 and v2
  • SSL CA Problem on CentOS7 Docker and Solution
  • What do I have?
  • Deploy HA nginx to AWS ECS with Geolocation Routing via Terraform
  • Deploy nginx docker to AWS ECS with Terraform Automation
  • How to Install Node Exporter on Linux Server
  • FortiClient Problem on M1 MacBookPro Problem Solution
  • Connect your AWS to GCP with Terraform via IPSec Site-to-Site VPN
  • Google Cloud Platform Automation with Terraform Easily
  • How to secure your Amazon Web Services account
  • Install UGREEN USB Ethernet Adapter on macOS
  • Redirect 301 HTTPS on App Engine with nginx on Google Cloud Platform
  • Set two different Target Groups on AWS Load Balancer with Terraform
  • Extend your ec2 Linux disk without reboot on Amazon Web Services
  • Create a New Grant User on AWS RDS (MariaDB)
  • Amazon S3 CORS Settings with CloudFront on Amazon Web Services
  • Take your GitLab backup everyday if it works in Docker
  • Find large files in CentOS, ubuntu and MacOS easily
  • Fix “Error: rpmdb open failed” on CentOS or Amazon Linux 2
  • Error: No space left on the device when starting/stopping services only
  • Juniper SRX110H-VA VDSL2 Configuration Step by Step
  • Enable Logrotation for Docker Containers
  • Download specific file extension via wget easily on terminal
  • Find the exact size of certain files in Linux via terminal
  • Disable SELinux on CentOS 7 or CentOS 8
  • Hello Blog!

Tag Cloud

active-active amazon linux 2 amazon web services automation aws basics bug centos centos7 cloud cloudfront container containers crud curl deployment devops docker ec2 ecs fargate file size gcp gitlab google cloud platform iam policy k8s kubernetes kubernetes architecture kubernetes basics linux macos network nginx pipeline replicaset route table s3 security terraform ubuntu vpc vpn wget yum

Archive

  • January 2023 (2)
  • December 2022 (2)
  • August 2022 (2)
  • July 2022 (3)
  • June 2022 (6)
  • March 2022 (1)
  • July 2021 (1)
  • May 2021 (5)
  • April 2021 (5)
  • February 2021 (1)
  • January 2021 (1)
  • September 2020 (2)
  • July 2020 (1)
  • April 2020 (1)
  • March 2020 (1)
  • February 2020 (1)
  • November 2019 (5)
©2023 Ercan Ermis | Built using WordPress and Responsive Blogily theme by Superb