[SOLVED] Reasons for differing behavior between Apache reverse proxy with and without SSL

Issue

I've been working on a local reverse proxy that routes traffic between two local Apache installations (each running a different version of mod_wsgi, which is the reason for the bifurcation). I want this reverse proxy to work whether the requests are HTTP or HTTPS.

However, when using SSL, the Location response header isn't being modified (properly) by ProxyPassReverse.

Below are the VirtualHost definitions for HTTP and HTTPS traffic, respectively:

<VirtualHost *:80>
        # Proxy traffic for Version 6 with an alias of: 6x/
        ProxyPass /6x/ http://localhost:10090/
        ProxyPassReverse /6x/ http://localhost:10090/

        # Proxy traffic for previous versions with aliases of: 5x/, 4x/, and /
        ProxyPass /5x/ http://localhost:10080/
        ProxyPassReverse /5x/ http://localhost:10080/
        ProxyPass /4x/ http://localhost:10080/
        ProxyPassReverse /4x/ http://localhost:10080/
        ProxyPass / http://localhost:10080/
        ProxyPassReverse / http://localhost:10080/
</VirtualHost>

<IfModule mod_ssl.c>
        <VirtualHost *:443>
                ServerName snakeoil.us.com

                ProxyPreserveHost on
                ProxyRequests off
                SSLEngine on
                SSLProxyEngine on
                SSLProxyVerify none
                SSLProxyCheckPeerCN off
                SSLProxyCheckPeerName off
                SSLProxyCheckPeerExpire off

                SSLCertificateFile /etc/ssl/certs/snakeoil.crt
                SSLCertificateKeyFile /etc/ssl/certs/snakeoil.key
                SSLCertificateChainFile /etc/ssl/certs/bundle-client.crt

                # Proxy traffic for Version 6 with an alias of: 6x/
                ProxyPass /6x/ https://localhost:10453/
                ProxyPassReverse /6x/ https://localhost:10453/

                # Proxy traffic for previous versions with aliases of: 5x/, 4x/, and /
                ProxyPass /5x/ https://localhost:10443/
                ProxyPassReverse /5x/ https://localhost:10443/
                ProxyPass /4x/ https://localhost:10443/
                ProxyPassReverse /4x/ https://localhost:10443/
                ProxyPass / https://localhost:10443/
                ProxyPassReverse / https://localhost:10443/
        </VirtualHost>
</IfModule>

When I access the url http://snakeoil.us.com/6x/snk610/index, the location header comes back as: Location: http://snakeoil.us.com/6x/snk610/index.

However, when I access the url https://snakeoil.us.com/6x/snk610/index, the location header comes back as: Location: https://snakeoil.us.com/snk610/index, which results in a 404 since only one of the two local Apache instances (the one associated with the 6x route) being proxied recognizes the snk610 alias (and it isn't the instance being routed to in this case).

The bottom line is that the HTTP VirtualHost definition proxies requests between the two local Apache instances without fail. However, the HTTPS VirtualHost definition does not and it isn't clear to me what causes this discrepancy.

Solution

Managed to find the solution. In retrospect, it should have been more obvious.

On the Apache instances being proxied to, I changed the access_log format to be the following:

LogFormat "%h %l %u %t \"%r\" %>s %b   -->   ResponseLocation: '%{Location}o'" common

This causes the outgoing response location to be logged.

Here is the output from the Apache HTTP instance (being proxied to):

[snake6x@test1 httpd6x]$ grep "ResponseLocation: 'http" logs/access_log
::1 - - [06/May/2020:15:43:25 -0400] "GET /snk610 HTTP/1.1" 301 233   -->   ResponseLocation: 'http://localhost:10090/snk610/index'
::1 - - [06/May/2020:15:43:30 -0400] "GET /snk610/index HTTP/1.1" 302 247   -->   ResponseLocation: 'http://localhost:10090/snk610/login?params=&message=&redirect_to=index'
::1 - - [06/May/2020:15:43:32 -0400] "POST /snk610/auth?redirect_to=index&params= HTTP/1.1" 302 204   -->   ResponseLocation: 'http://localhost:10090/snk610/index'

From the above, you can see that the response location header looks as expected, i.e. ProxyPassReverse should be able to successfully make its replacement.

Conversely, here is the output from the Apache HTTPS instance (being proxied to):

[snake6x@test1 httpd]$ grep "ResponseLocation: 'http" logs/ssl_request_log
[06/May/2020:19:53:38 +0000] ::1 "GET /snk610 HTTP/1.1" 240 2645788   -->   ResponseLocation: 'https://snakeoil.us.com/snk610/index'
[06/May/2020:19:56:21 +0000] ::1 "GET /snk610/index HTTP/1.1" 254 2682899   -->   ResponseLocation: 'https://snakeoil.us.com/snk610/login?params=&message=&redirect_to=index'
[06/May/2020:19:56:23 +0000] ::1 "POST /snk610/auth?redirect_to=index&params= HTTP/1.1" 240 752392   -->   ResponseLocation: 'https://snakeoil.us.com/snk610/index'

From the above, you can see that the server name has been substituted for the incoming host name in the response location header. This is what was causing ProxyPassReverse to fail to replace outgoing hostname (on the reverse proxy server).

I resolved this problem by explicitly updating the outgoing location header on the server being proxied to:

    # Since this server has a proxy immediately in front of it, we need the outgoing
    # location to match the incoming location. However, the ServerName tag will
    # cause the incoming location to be changed to include the ServerName, which will
    # cause the upstream ProxyPassReverse to fail to update the outgoing location
    # properly.
    #
    # This Header modification replaces the outgoing ServerName with the incoming
    # name.
    #
    # FIXME: There is surely a better way to do this with a variable that contains
    # the incoming host
    Header edit Location ^https://snakeoil.us.com:443 https://localhost:10453
    Header edit Location ^https://snakeoil.us.com https://localhost:10453

Answered By - djkern

Answer Checked By - Mary Flores (WPSolving Volunteer)