Issue
We run the follwoing on a debian 11.8 system:
# dpkg -l | grep gitlab
ii gitlab-ee 16.7.0-ee.0 amd64 GitLab Enterprise Edition (including NGINX, Postgres, Redis)
ii gitlab-runner 16.7.0 amd64 GitLab Runner
After the latest upgrade of gitlab-ee it does not run anymore. The website response ist 502 
In the logs I found many lines like this:
{"severity":"ERROR","time":"2023-12-23T23:39:53.299Z","exception.class":"IOError","exception.message":"Can't reserve 4096 bytes for memory-mapped file in /run/gitlab/puma/gauge_all_puma_6-0.db: ENOSPC: No space left on device","exception.backtrace":["config/initializers/7_prometheus_metrics.rb:82:in `block in \u003ctop (required)\u003e'","lib/gitlab/cluster/lifecycle_events.rb:173:in `block in call'","lib/gitlab/cluster/lifecycle_events.rb:172:in `each'","lib/gitlab/cluster/lifecycle_events.rb:172:in `call'","lib/gitlab/cluster/lifecycle_events.rb:133:in `do_worker_start'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null}
{"severity":"ERROR","time":"2023-12-23T23:43:05.910Z","exception.class":"IOError","exception.message":"Can't reserve 4096 bytes for memory-mapped file in /run/gitlab/puma/gauge_max_puma_1-0.db: ENOSPC: No space left on device","exception.backtrace":["config/initializers/7_prometheus_metrics.rb:82:in `block in \u003ctop (required)\u003e'","lib/gitlab/cluster/lifecycle_events.rb:173:in `block in call'","lib/gitlab/cluster/lifecycle_events.rb:172:in `each'","lib/gitlab/cluster/lifecycle_events.rb:172:in `call'","lib/gitlab/cluster/lifecycle_events.rb:133:in `do_worker_start'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null}
{"severity":"ERROR","time":"2023-12-23T23:48:48.758Z","exception.class":"IOError","exception.message":"Can't reserve 4096 bytes for memory-mapped file in /run/gitlab/puma/gauge_max_puma_0-0.db: ENOSPC: No space left on device","exception.backtrace":["config/initializers/7_prometheus_metrics.rb:82:in `block in \u003ctop (required)\u003e'","lib/gitlab/cluster/lifecycle_events.rb:173:in `block in call'","lib/gitlab/cluster/lifecycle_events.rb:172:in `each'","lib/gitlab/cluster/lifecycle_events.rb:172:in `call'","lib/gitlab/cluster/lifecycle_events.rb:133:in `do_worker_start'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null}
{"severity":"ERROR","time":"2023-12-23T23:49:25.184Z","exception.class":"IOError","exception.message":"Can't reserve 4096 bytes for memory-mapped file in /run/gitlab/puma/counter_puma_8-0.db: ENOSPC: No space left on device","exception.backtrace":["config/initializers/7_prometheus_metrics.rb:82:in `block in \u003ctop (required)\u003e'","lib/gitlab/cluster/lifecycle_events.rb:173:in `block in call'","lib/gitlab/cluster/lifecycle_events.rb:172:in `each'","lib/gitlab/cluster/lifecycle_events.rb:172:in `call'","lib/gitlab/cluster/lifecycle_events.rb:133:in `do_worker_start'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null}
{"severity":"ERROR","time":"2023-12-23T23:50:04.281Z","exception.class":"IOError","exception.message":"Can't reserve 4096 bytes for memory-mapped file in /run/gitlab/puma/histogram_puma_7-0.db: ENOSPC: No space left on device","exception.backtrace":["config/initializers/7_prometheus_metrics.rb:82:in `block in \u003ctop (required)\u003e'","lib/gitlab/cluster/lifecycle_events.rb:173:in `block in call'","lib/gitlab/cluster/lifecycle_events.rb:172:in `each'","lib/gitlab/cluster/lifecycle_events.rb:172:in `call'","lib/gitlab/cluster/lifecycle_events.rb:133:in `do_worker_start'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null}
{"severity":"ERROR","time":"2023-12-23T23:50:04.306Z","exception.class":"IOError","exception.message":"Can't reserve 4096 bytes for memory-mapped file in /run/gitlab/puma/counter_puma_8-0.db: ENOSPC: No space left on device","exception.backtrace":["config/initializers/7_prometheus_metrics.rb:82:in `block in \u003ctop (required)\u003e'","lib/gitlab/cluster/lifecycle_events.rb:173:in `block in call'","lib/gitlab/cluster/lifecycle_events.rb:172:in `each'","lib/gitlab/cluster/lifecycle_events.rb:172:in `call'","lib/gitlab/cluster/lifecycle_events.rb:133:in `do_worker_start'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null}
{"severity":"ERROR","time":"2023-12-23T23:50:43.720Z","exception.class":"IOError","exception.message":"Can't reserve 4096 bytes for memory-mapped file in /run/gitlab/puma/counter_puma_8-0.db: ENOSPC: No space left on device","exception.backtrace":["config/initializers/7_prometheus_metrics.rb:82:in `block in \u003ctop (required)\u003e'","lib/gitlab/cluster/lifecycle_events.rb:173:in `block in call'","lib/gitlab/cluster/lifecycle_events.rb:172:in `each'","lib/gitlab/cluster/lifecycle_events.rb:172:in `call'","lib/gitlab/cluster/lifecycle_events.rb:133:in `do_worker_start'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":null,"tags.correlation_id":null}
But when I check my "run" directory it looks like this:
# df -h
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 6.5M 6.3G 1% /run
/dev/md2 906G 609G 252G 71% /
tmpfs 32G 52K 32G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/md1 485M 105M 355M 23% /boot
tmpfs 6.3G 0 6.3G 0% /run/user/1000
I already tried to restart gitlab, reboot, prune old docker files, disabled prometheus in gitlab.rb with prometheus_monitoring['enable'] = false and reconfiguring gitlab and restarting after this. Nothing helps.
How can I investigate this problem further?
Solution
I faced similar problem.
In /var/log/gitlab/puma/puma_stderr.log I had such errors:
Cannot allocate memory - fork(2) (Errno::ENOMEM)
My system has got 16GB RAM. I didn't find any reason why the memory couldn't be allocated. Finally I reduced number of worker processes for Puma:
/etc/gitlab/gitlab.rb
-- # puma['worker_processes'] = 2
++ puma['worker_processes'] = 1
then
gitlab-ctl reconfigure
gitlab-ctl restart puma
and GitLab is back, up and running.
Though, it's a workaround only. Why it doesn't work with 2 is still unknown for me.
Answered By - Karel Pičman Answer Checked By - Candace Johnson (WPSolving Volunteer)