Issue
I have a server running Ubuntu 20.04, nginx and varnish with letsencrypt SSL.
I have installed PHPMyAdmin to the default server block and have another server block for my actual website running magento 2.4.3. This was all working fine until i installed varnish and now when trying to load either site i get the 503 Backend fetch failed error and when i try and load then on port 8080 from the URL i get a "this page cannot be reach" error.
I have configured nginx to listen on port 8080 for both sites, i have set varnish to listen on port 80. I have edited the vcl generated by magento to set the host and port to 127.0.0.1 and 8080 respectively as shown on https://devdocs.magento.com/guides/v2.4/config-guide/varnish/config-varnish-configure.html.
The varnishlog show the backend is unhealthy error but i dont know how to solve this.
The output of netstat -tulpn is:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22007 0.0.0.0:* LISTEN 48993/sshd: /usr/sb
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 302464/nginx: maste
tcp 0 0 127.0.0.1:6082 0.0.0.0:* LISTEN 304955/varnishd
tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN 42533/beam.smp
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 44657/redis-server
tcp 0 0 127.0.0.1:9100 0.0.0.0:* LISTEN 14734/noderig
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 304955/varnishd
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 302464/nginx: maste
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 162503/pure-ftpd (S
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 843/systemd-resolve
tcp6 0 0 :::22007 :::* LISTEN 48993/sshd: /usr/sb
tcp6 0 0 :::443 :::* LISTEN 302464/nginx: maste
tcp6 0 0 :::5672 :::* LISTEN 42533/beam.smp
tcp6 0 0 :::3306 :::* LISTEN 158505/mysqld
tcp6 0 0 ::1:6379 :::* LISTEN 44657/redis-server
tcp6 0 0 :::80 :::* LISTEN 304955/varnishd
tcp6 0 0 :::8080 :::* LISTEN 302464/nginx: maste
As you can see, varnishd is running on port 80 and nginx on 8080 but its almost as if port 8080 isnt listening. I have ufw enabled and "Nginx Full" is set to allow and i have also tried adding a rule to allow 8080 but this made no difference. Disabling varnish and trying again to load both sites on port 8080 results in the same "page cannot be reached" error so i dont believe this to be a varnish issue, but an nginx one, but why isnt it listen on port 8080?
If i try curl from the local server accessing localhost:8080 for example, it works and loads the default nginx page as expected. My PHPMyAdmin install is in a folder and that also works by using curl to access localhost/phpmyadmin. Does this mean it is then a firewall issue? Disabling ufw didnt help.
My mysite.conf my magento site is:
upstream fastcgi_backend {
server unix:/run/php/php7.4-fpm.sock;
}
server {
listen 8080;
listen [::]:8080;
server_name staging.clicksaveandprint.com;
return 404; # managed by Certbot
set $MAGE_ROOT /var/www/clicksaveandprint.com;
access_log /var/log/nginx/clicksaveandprint.com.log;
error_log /var/log/nginx/clicksaveandprint.com.err;
include /var/www/clicksaveandprint.com/nginx.conf;
if ($host = staging.clicksaveandprint.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
# PHP entry point for main application
location ~ (index|get|static|report|404|503)\.php$ {
try_files $uri $uri/ =404;
fastcgi_pass fastcgi_backend;
fastcgi_buffers 1024 4k;
fastcgi_param PHP_FLAG "session.auto_start=off \n suhosin.session.cryptua=off";
fastcgi_param PHP_VALUE "memory_limit=4096M \n max_execution_time=18000";
fastcgi_read_timeout 6000s;
fastcgi_connect_timeout 6000s;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param HTTPS on;
include fastcgi_params;
}
}
server {
listen [::]:443 ssl ipv6only=on; # managed by Certbot
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/staging.clicksaveandprint.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/staging.clicksaveandprint.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
# OCSP stapling
ssl_stapling on;
ssl_stapling_verify on;
location / {
proxy_pass http://127.0.0.1;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Ssl-Offloaded "1";
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Port 443;
#proxy_hide_header X-Varnish;
#proxy_hide_header Via;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
}
}
magento nginx.conf. This is the default one from magento with no changes:
## Example configuration:
# upstream fastcgi_backend {
# # use tcp connection
# # server 127.0.0.1:9000;
# # or socket
# server unix:/var/run/php/php7.4-fpm.sock;
# }
# server {
# listen 80;
# server_name mage.dev;
# set $MAGE_ROOT /var/www/magento2;
# set $MAGE_DEBUG_SHOW_ARGS 0;
# include /vagrant/magento2/nginx.conf.sample;
# }
#
## Optional override of deployment mode. We recommend you use the
## command 'bin/magento deploy:mode:set' to switch modes instead.
##
## set $MAGE_MODE default; # or production or developer
##
## If you set MAGE_MODE in server config, you must pass the variable into the
## PHP entry point blocks, which are indicated below. You can pass
## it in using:
##
## fastcgi_param MAGE_MODE $MAGE_MODE;
##
## In production mode, you should uncomment the 'expires' directive in the /static/ location block
# Modules can be loaded only at the very beginning of the Nginx config file, please move the line below to the main config file
# load_module /etc/nginx/modules/ngx_http_image_filter_module.so;
root $MAGE_ROOT/pub;
index index.php;
autoindex off;
charset UTF-8;
error_page 404 403 = /errors/404.php;
#add_header "X-UA-Compatible" "IE=Edge";
# Deny access to sensitive files
location /.user.ini {
deny all;
}
# PHP entry point for setup application
location ~* ^/setup($|/) {
root $MAGE_ROOT;
location ~ ^/setup/index.php {
fastcgi_pass fastcgi_backend;
fastcgi_param PHP_FLAG "session.auto_start=off \n suhosin.session.cryptua=off";
fastcgi_param PHP_VALUE "memory_limit=756M \n max_execution_time=600";
fastcgi_read_timeout 600s;
fastcgi_connect_timeout 600s;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
location ~ ^/setup/(?!pub/). {
deny all;
}
location ~ ^/setup/pub/ {
add_header X-Frame-Options "SAMEORIGIN";
}
}
# PHP entry point for update application
location ~* ^/update($|/) {
root $MAGE_ROOT;
location ~ ^/update/index.php {
fastcgi_split_path_info ^(/update/index.php)(/.+)$;
fastcgi_pass fastcgi_backend;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_path_info;
include fastcgi_params;
}
# Deny everything but index.php
location ~ ^/update/(?!pub/). {
deny all;
}
location ~ ^/update/pub/ {
add_header X-Frame-Options "SAMEORIGIN";
}
}
location / {
try_files $uri $uri/ /index.php$is_args$args;
}
location /pub/ {
location ~ ^/pub/media/(downloadable|customer|import|custom_options|theme_customization/.*\.xml) {
deny all;
}
alias $MAGE_ROOT/pub/;
add_header X-Frame-Options "SAMEORIGIN";
}
location /static/ {
# Uncomment the following line in production mode
# expires max;
# Remove signature of the static files that is used to overcome the browser cache
location ~ ^/static/version\d*/ {
rewrite ^/static/version\d*/(.*)$ /static/$1 last;
}
location ~* \.(ico|jpg|jpeg|png|gif|svg|svgz|webp|avif|avifs|js|css|eot|ttf|otf|woff|woff2|html|json|webmanifest)$ {
add_header Cache-Control "public";
add_header X-Frame-Options "SAMEORIGIN";
expires +1y;
if (!-f $request_filename) {
rewrite ^/static/(version\d*/)?(.*)$ /static.php?resource=$2 last;
}
}
location ~* \.(zip|gz|gzip|bz2|csv|xml)$ {
add_header Cache-Control "no-store";
add_header X-Frame-Options "SAMEORIGIN";
expires off;
if (!-f $request_filename) {
rewrite ^/static/(version\d*/)?(.*)$ /static.php?resource=$2 last;
}
}
if (!-f $request_filename) {
rewrite ^/static/(version\d*/)?(.*)$ /static.php?resource=$2 last;
}
add_header X-Frame-Options "SAMEORIGIN";
}
location /media/ {
## The following section allows to offload image resizing from Magento instance to the Nginx.
## Catalog image URL format should be set accordingly.
## See https://docs.magento.com/user-guide/configuration/general/web.html#url-options
# location ~* ^/media/catalog/.* {
#
# # Replace placeholders and uncomment the line below to serve product images from public S3
# # See examples of S3 authentication at https://github.com/anomalizer/ngx_aws_auth
# # resolver 8.8.8.8;
# # proxy_pass https://<bucket-name>.<region-name>.amazonaws.com;
#
# set $width "-";
# set $height "-";
# if ($arg_width != '') {
# set $width $arg_width;
# }
# if ($arg_height != '') {
# set $height $arg_height;
# }
# image_filter resize $width $height;
# image_filter_jpeg_quality 90;
# }
try_files $uri $uri/ /get.php$is_args$args;
location ~ ^/media/theme_customization/.*\.xml {
deny all;
}
location ~* \.(ico|jpg|jpeg|png|gif|svg|svgz|webp|avif|avifs|js|css|eot|ttf|otf|woff|woff2)$ {
add_header Cache-Control "public";
add_header X-Frame-Options "SAMEORIGIN";
expires +1y;
try_files $uri $uri/ /get.php$is_args$args;
}
location ~* \.(zip|gz|gzip|bz2|csv|xml)$ {
add_header Cache-Control "no-store";
add_header X-Frame-Options "SAMEORIGIN";
expires off;
try_files $uri $uri/ /get.php$is_args$args;
}
add_header X-Frame-Options "SAMEORIGIN";
}
location /media/customer/ {
deny all;
}
location /media/downloadable/ {
deny all;
}
location /media/import/ {
deny all;
}
location /media/custom_options/ {
deny all;
}
location /errors/ {
location ~* \.xml$ {
deny all;
}
}
# PHP entry point for main application
location ~ ^/(index|get|static|errors/report|errors/404|errors/503|health_check)\.php$ {
try_files $uri =404;
fastcgi_pass fastcgi_backend;
fastcgi_buffers 16 16k;
fastcgi_buffer_size 32k;
fastcgi_param PHP_FLAG "session.auto_start=off \n suhosin.session.cryptua=off";
fastcgi_param PHP_VALUE "memory_limit=756M \n max_execution_time=18000";
fastcgi_read_timeout 600s;
fastcgi_connect_timeout 600s;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
gzip on;
gzip_disable "msie6";
gzip_comp_level 6;
gzip_min_length 1100;
gzip_buffers 16 8k;
gzip_proxied any;
gzip_types
text/plain
text/css
text/js
text/xml
text/javascript
application/javascript
application/x-javascript
application/json
application/xml
application/xml+rss
image/svg+xml;
gzip_vary on;
# Banned locations (only reached if the earlier PHP entry point regexes don't match)
location ~* (\.php$|\.phtml$|\.htaccess$|\.git) {
deny all;
}
EDIT: adding port 8080 to ufw does now allow my phpmyadmin to work when connecting to port 8080, but i still get a 503 error when connecting to anything on port 80 with varnish enabled.
Output of sudo varnishlog -g raw -i backend_health:
varnishlog -g raw -i backend_health
0 Backend_health - default Still sick 4---X-R- 0 5 10 0.000624 0.000000 HTTP/1.1 404 Not Found
0 Backend_health - default Still sick 4---X-R- 0 5 10 0.000621 0.000000 HTTP/1.1 404 Not Found
0 Backend_health - default Still sick 4---X-R- 0 5 10 0.000652 0.000000 HTTP/1.1 404 Not Found
0 Backend_health - default Still sick 4---X-R- 0 5 10 0.000210 0.000000 HTTP/1.1 404 Not Found
0 Backend_health - default Still sick 4---X-R- 0 5 10 0.000214 0.000000 HTTP/1.1 404 Not Found
The output for command sudo varnishlog -g request -q "RespStatus == 503":
sudo varnishlog -g request -q "RespStatus == 503"
* << Request >> 11
- Begin req 10 rxreq
- Timestamp Start: 1641551164.244422 0.000000 0.000000
- Timestamp Req: 1641551164.244422 0.000000 0.000000
- VCL_use boot
- ReqStart 127.0.0.1 39792 a0
- ReqMethod GET
- ReqURL /
- ReqProtocol HTTP/1.0
- ReqHeader Host: staging.clicksaveandprint.com
- ReqHeader X-Forwarded-Host: staging.clicksaveandprint.com
- ReqHeader X-Real-IP: 81.107.192.226
- ReqHeader X-Forwarded-For: 81.107.192.226
- ReqHeader Ssl-Offloaded: 1
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-Port: 443
- ReqHeader X-Forwarded-Proto: https
- ReqHeader Connection: close
- ReqHeader Cache-Control: max-age=0
- ReqHeader Upgrade-Insecure-Requests: 1
- ReqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
- ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
- ReqHeader Sec-GPC: 1
- ReqHeader Sec-Fetch-Site: none
- ReqHeader Sec-Fetch-Mode: navigate
- ReqHeader Sec-Fetch-User: ?1
- ReqHeader Sec-Fetch-Dest: document
- ReqHeader Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
- ReqHeader Cookie: __tawkuuid=e::clicksaveandprint.com::n8RwZ5uipY2iBtEtSiR+wc7TP1jlMwD51gtYez+PiWZ4Xr/t3VxVX/BMIhUzfv0U::2; _ga=GA1.2.428376681.1618304758; _ga_01K4WXXY6B=GS1.1.1641392250.103.1.1641392407.60; form_key=VKzAhiJUg3HyT7jg; AMCVS_8F99160E571FC0427F0001
- ReqUnset X-Forwarded-For: 81.107.192.226
- ReqHeader X-Forwarded-For: 81.107.192.226, 127.0.0.1
- VCL_call RECV
- ReqHeader grace: none
- ReqURL /
- ReqUnset Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Encoding: gzip
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
- VCL_return fetch
- Link bereq 12 fetch
- Timestamp Fetch: 1641551164.245036 0.000614 0.000614
- RespProtocol HTTP/1.1
- RespStatus 503
- RespReason Backend fetch failed
- RespHeader Date: Fri, 07 Jan 2022 10:26:04 GMT
- RespHeader Server: Varnish
- RespHeader content-type: text/html; charset=utf-8
- RespHeader Retry-After: 5
- RespHeader X-Varnish: 11
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/6.2)
- VCL_call DELIVER
- RespUnset Age: 0
- RespHeader Pragma: no-cache
- RespHeader Expires: -1
- RespHeader Cache-Control: no-store, no-cache, must-revalidate, max-age=0
- RespUnset Server: Varnish
- RespUnset X-Varnish: 11
- RespUnset Via: 1.1 varnish (Varnish/6.2)
- VCL_return deliver
- Timestamp Process: 1641551164.245126 0.000704 0.000090
- Filters
- RespHeader Content-Length: 279
- RespHeader Connection: close
- Timestamp Resp: 1641551164.245228 0.000807 0.000103
- ReqAcct 1233 0 1233 264 279 543
- End
** << BeReq >> 12
-- Begin bereq 11 fetch
-- VCL_use boot
-- Timestamp Start: 1641551164.244718 0.000000 0.000000
-- BereqMethod GET
-- BereqURL /
-- BereqProtocol HTTP/1.0
-- BereqHeader Host: staging.clicksaveandprint.com
-- BereqHeader X-Forwarded-Host: staging.clicksaveandprint.com
-- BereqHeader X-Real-IP: 81.107.192.226
-- BereqHeader Ssl-Offloaded: 1
-- BereqHeader X-Forwarded-Proto: https
-- BereqHeader X-Forwarded-Port: 443
-- BereqHeader X-Forwarded-Proto: https
-- BereqHeader Upgrade-Insecure-Requests: 1
-- BereqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
-- BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
-- BereqHeader Sec-GPC: 1
-- BereqHeader Sec-Fetch-Site: none
-- BereqHeader Sec-Fetch-Mode: navigate
-- BereqHeader Sec-Fetch-User: ?1
-- BereqHeader Sec-Fetch-Dest: document
-- BereqHeader Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
-- BereqHeader Cookie: __tawkuuid=e::clicksaveandprint.com::n8RwZ5uipY2iBtEtSiR+wc7TP1jlMwD51gtYez+PiWZ4Xr/t3VxVX/BMIhUzfv0U::2; _ga=GA1.2.428376681.1618304758; _ga_01K4WXXY6B=GS1.1.1641392250.103.1.1641392407.60; form_key=VKzAhiJUg3HyT7jg; AMCVS_8F99160E571FC0427F0001
-- BereqHeader X-Forwarded-For: 81.107.192.226, 127.0.0.1
-- BereqHeader grace: none
-- BereqHeader Accept-Encoding: gzip
-- BereqProtocol HTTP/1.1
-- BereqHeader X-Varnish: 12
-- VCL_call BACKEND_FETCH
-- VCL_return fetch
-- FetchError backend default: unhealthy
-- Timestamp Beresp: 1641551164.244824 0.000107 0.000107
-- Timestamp Error: 1641551164.244834 0.000116 0.000010
-- BerespProtocol HTTP/1.1
-- BerespStatus 503
-- BerespReason Service Unavailable
-- BerespReason Backend fetch failed
-- BerespHeader Date: Fri, 07 Jan 2022 10:26:04 GMT
-- BerespHeader Server: Varnish
-- VCL_call BACKEND_ERROR
-- BerespHeader content-type: text/html; charset=utf-8
-- BerespHeader Retry-After: 5
-- VCL_return deliver
-- Storage malloc Transient
-- Length 279
-- BereqAcct 0 0 0 0 0 0
-- End
Solution
Can you please provide the output of the following command?
sudo varnishlog -g raw -i backend_health
This command will provide logging information on the status of the health probe checks.
The fact that you get a 503 Backend fetch failed probably means the backend is not available.
If the backend health logging is inconclusive, we can always look at the transactions that resulted in that 503 Backend fetch failed error.
You can run the following command for that:
sudo varnishlog -g request -q "RespStatus == 503"
Please provide some logging output:
- A couple of lines from the backend health logging command
- A full log transaction for a request that triggered the
503 Backend fetch failed
UPDATE
I check the output of your VSL logs and the health check returns an HTTP 404 Not Found error. This means that the /health_check.php resource cannot be found.
That's a common error in Magento. As described in https://www.varnish-software.com/developers/tutorials/configuring-varnish-magento/#fixing-the-backend-health-checks-for-magento-24 the health check endpoint depends on your Magento version.
Please replace the URL of your health probe from /health_check.php to /pub/health_check.php or vice versa.
Just ensure that the health check URL returns an HTTP 200 OK and Varnish should be able to consider the backend healthy.
Answered By - Thijs Feryn Answer Checked By - Clifford M. (WPSolving Volunteer)