Shopware HTTP Cache with Varnish XKey

Foreword

Shopware removes the original method for using Varnish as HTTP cache with version v6.7.0 and provides an alternative via Varnish XKey. Varnish XKey is part of the Varnish Module Collection, which is not integrated into Varnish itself. We provide a corresponding Debian repository here, which enables the installation of the Varnish Module Collection via apt and thus simplifies the integration of Varnish XKey.

For the configuration of Varnish with XKey, Shopware provides a complete configuration in the Shopware developer documentation:

https://developer.shopware.com/docs/guides/hosting/infrastructure/reverse-http-cache.html#configure-shopware

Prerequisite

• Varnish server with Varnish 7.6
• Varnish Modules Collection

Trusted Proxies

Enter the source IP address of the proxy server in the configuration TRUSTED_PROXIES. We recommend operation within a creoline VPC network in order to demilitarize the internal infrastructure.

As of Shopware 6.6, a framework.yaml must first be configured, which activates the Shopware internal TRUSTED_PROXIES configuration.

# config/packages/framework.yaml

framework:
    trusted_proxies: '%env(TRUSTED_PROXIES)%'

Example configuration via .env:

TRUSTED_PROXIES=127.0.0.1,10.20.0.1/32,10.20.0.2/32

In this example configuration, the two Varnish instances 10.20.0.1 and 10.20.0.2 are authorized as proxies.

More information about the Trusted Proxies can be found in the official Symfony documentation at Symfony Proxy Docs

Shopware 6 XKey configuration

In order for Shopware 6 to set the correct Cache-Control header, the configuration of the reverse proxy is required. The following configuration can be used to ensure that Shopware sets the value public, must-revalidate instead of the Cache-Control header value private, no-cache and invalidates the caches accordingly in the Varnish servers via BAN requests.

Create the following file and add the following content accordingly:

nano config/packages/shopware.yml

# Be aware that the configuration key changed from storefront.reverse_proxy to shopware.http_cache.reverse_proxy starting with Shopware 6.6
shopware:
  http_cache:
      reverse_proxy:
        enabled: true
        use_varnish_xkey: true
        hosts:
          - '10.20.0.2'

In addition, the environment variable SHOPWARE_HTTP_CACHE_ENABLED=1 must be set in the .env file.

Varnish 7.6 Configuration

The Varnish configuration can be edited and rolled out directly via the configuration module in the Customer Center. Navigate to Server → Your server → Configuration files → Varnish, to customize the Varnish configuration.

Example configuration of the default.vcl of Varnish with XKey Hard-Purge:

vcl 4.1;

import std;
import xkey;
import cookie;

# Specify your app nodes here. Use round-robin balancing to add more than one.
backend default {
    .host = "XX.XX.XX.XX";
    .port = "80";
}

# ACL for purgers IP. (This needs to contain app server ips)
acl purgers {
    "127.0.0.1";
    "localhost";
    "::1";
}

sub vcl_recv {
    # Handle PURGE
    if (req.method == "PURGE") {
        if (client.ip !~ purgers) {
            return (synth(403, "Forbidden"));
        }
        if (req.http.xkey) {
            set req.http.n-gone = xkey.purge(req.http.xkey);

            return (synth(200, "Invalidated "+req.http.n-gone+" objects"));
        } else {
            return (purge);
        }
    }

    if (req.method == "BAN") {
        if (!client.ip ~ purgers) {
            return (synth(403, "Forbidden"));
        }

        ban("req.url ~ "+req.url);
        return (synth(200, "BAN URLs containing (" + req.url + ") done."));
    }

    # Only handle relevant HTTP request methods
    if (req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "PATCH" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE") {
          return (pipe);
    }

    if (req.http.Authorization) {
        return (pass);
    }

    # We only deal with GET and HEAD by default
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    # Always pass these paths directly to php without caching
    # Note: virtual URLs might bypass this rule (e.g. /en/checkout)
    if (req.url ~ "^/(checkout|account|admin|api)(/.*)?$") {
        return (pass);
    }

    cookie.parse(req.http.cookie);

    set req.http.cache-hash = cookie.get("sw-cache-hash");
    set req.http.currency = cookie.get("sw-currency");
    set req.http.states = cookie.get("sw-states");

    if (req.url == "/widgets/checkout/info" && !req.http.states ~ "cart-filled") {
        return (synth(204, ""));
    }

    # Ignore query strings that are only necessary for the js on the client. Customize as needed.
    if (req.url ~ "(\?|&)(pk_campaign|piwik_campaign|pk_kwd|piwik_kwd|pk_keyword|pixelId|kwid|kw|adid|chl|dv|nk|pa|camid|adgid|cx|ie|cof|siteurl|utm_[a-z]+|_ga|gclid)=") {
        # see rfc3986#section-2.3 "Unreserved Characters" for regex
        set req.url = regsuball(req.url, "(pk_campaign|piwik_campaign|pk_kwd|piwik_kwd|pk_keyword|pixelId|kwid|kw|adid|chl|dv|nk|pa|camid|adgid|cx|ie|cof|siteurl|utm_[a-z]+|_ga|gclid)=[A-Za-z0-9\-\_\.\~%]+&?", "");
    }

    set req.url = regsub(req.url, "(\?|\?&|&&)$", "");

    # Normalize query arguments
    set req.url = std.querysort(req.url);

    # Set a header announcing Surrogate Capability to the origin
    set req.http.Surrogate-Capability = "shopware=ESI/1.0";

    # Make sure that the client ip is forward to the client.
    if (req.http.x-forwarded-for) {
        set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
    } else {
        set req.http.X-Forwarded-For = client.ip;
    }

    return (hash);
}

sub vcl_hash {
    # Consider Shopware HTTP cache cookies
    if (req.http.cache-hash != "") {
        hash_data("+context=" + req.http.cache-hash);
    } elseif (req.http.currency != "") {
        hash_data("+currency=" + req.http.currency);
    }
}

sub vcl_hit {
  # Consider client states for response headers
  if (req.http.states) {
     if (req.http.states ~ "logged-in" && obj.http.sw-invalidation-states ~ "logged-in" ) {
        return (pass);
     }

     if (req.http.states ~ "cart-filled" && obj.http.sw-invalidation-states ~ "cart-filled" ) {
        return (pass);
     }
  }
}

sub vcl_backend_fetch {
    unset bereq.http.cache-hash;
    unset bereq.http.currency;
    unset bereq.http.states;
}

sub vcl_backend_response {
    # Serve stale content for three days after object expiration
    set beresp.grace = 3d;

    unset beresp.http.X-Powered-By;
    unset beresp.http.Server;

    if (beresp.http.Surrogate-Control ~ "ESI/1.0") {
        unset beresp.http.Surrogate-Control;
        set beresp.do_esi = true;
        return (deliver);
    }

    if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }

    if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
        unset beresp.http.Set-Cookie;
    }
}

sub vcl_deliver {
    ## we don't want the client to cache
    if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(theme|media|thumbnail|bundles)/") {
        set resp.http.Pragma = "no-cache";
        set resp.http.Expires = "-1";
        set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
    }

    # invalidation headers are only for internal use
    unset resp.http.sw-invalidation-states;
    unset resp.http.xkey;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.Link;
}

Soft-Purge vs. Hard-Purge

The above Varnish configuration uses hard purges by default, which ensures that a page is removed from the cache and the next request takes longer. To counteract this, soft-purges can be used, which deliver the outdated page to the client once and update the cache for this website in the background.

Soft-purges can be used with an adjustment to the Varnish configuration, see Line 27 in the above-mentioned Varnish configuration:

# Hard-Purge
set req.http.n-gone = xkey.purge(req.http.xkey);

# Soft-Purge
set req.http.n-gone = xkey.softpurge(req.http.xkey);

Debugging

The default configuration removes all HTTP headers except for the Age, which is used to determine the cache age. An Age of 0 means that the cache does not work.

\This is usually because the Cache-Control: public HTTP header has not been set by the web application.

\nTo check this, the command curl can be used as follows:

curl -vvv -H 'Host: <sales-channel-domain>' <app-server-ip> 1> /dev/null

which returns the following response:

< HTTP/1.1 200 OK
< Cache-Control: public, s-maxage=7200
< Content-Type: text/html; charset=UTF-8
< Xkey: theme.sw-logo-desktop, ...

\nIf the Cache-Control: public or xkey: ... HTTP header cannot be found in the response, this is probably due to a faulty configuration in the web application itself.

Check the configuration of Shopware 6 to see if the Reverse Proxy Mode has been activated correctly.

Error analysis

HTTP status 503 backend fetch failed on individual article and/or category pages

Depending on the requirements of your store, it may be that the size of the HTTP header exceeds the standard size configured in Varnish for one or more item and category pages.

Check

Connect to your Varnish server via SSH and execute the following command:

curl -I -H "Host: <your-domain.tld>" -H "X-Forwarded-Proto: https" http://<app-server-ip>/path/with/error/503 | wc -c

If the result exceeds 8192, the Varnish Systemd service must be adjusted here, whereby new standard sizes are set.

Solution

Execute the following command as root user:

systemctl edit varnish

\nCustomize the ExecStart setting as follows:

[Service]
ExecStart=/usr/sbin/varnishd \
...
+ -p http_req_hdr_len=32k \
+ -p http_resp_hdr_len=32k \
+ -p http_resp_size=64k \
...
Restart=on-failure

Then reload the Varnish instance to apply the configuration.

systemctl reload varnish