Shopware HTTP Cache with Varnish XKey

Starting with version 6.7.0, Shopware has removed Redis as a dependency for using Varnish as an HTTP cache and provides an alternative via Varnish XKey. Varnish XKey is part of the Varnish Module Collection, which is not integrated into Varnish Cache itself. We provide a corresponding Debian repository that allows you to install the Varnish Module Collection via apt, thereby simplifying the integration of Varnish XKey.

For configuring Varnish with XKey, Shopware provides a complete configuration guide in the Shopware developer documentation:

https://developer.shopware.com/docs/guides/hosting/infrastructure/reverse-http-cache.html#configure-shopware

Prerequisites

• Shopware 6 Cluster

• Varnish server with Varnish

• Varnish Modules Collection

  \

Trusted Proxies

Enter the source IP address of the proxy server in the TRUSTED_PROXIES configuration. We recommend operating within a creoline VPC network to demilitarize the internal infrastructure.

Starting with Shopware 6.6, you must first configure a framework.yaml file to enable the Shopware-internal TRUSTED_PROXIES configuration.

# config/packages/framework.yaml

framework:
    trusted_proxies: '%env(TRUSTED_PROXIES)%'

Example configuration via .env:

TRUSTED_PROXIES=127.0.0.1,10.20.0.1/32,10.20.0.2/32

In this example configuration, the two Varnish instances 10.20.0.1 and 10.20.0.2 are authorized as proxies.

For more information on trusted proxies, see the official Symfony documentation at Symfony Proxy Docs.

Shopware 6 XKey Configuration

To ensure that Shopware 6 sets the correct Cache-Control header, you must configure the reverse proxy. The following configuration ensures that Shopware sets the Cache-Control header value to public, no-cache, and invalidates the caches accordingly on the Varnish servers via BAN requests.

Create the following file and insert the following content:

nano config/packages/shopware.yml

# Note that the configuration key changed from `storefront.reverse_proxy` to `shopware.http_cache.reverse_proxy` starting with Shopware 6.6
shopware:
  http_cache:
      reverse_proxy:
 enabled: true
 use_varnish_xkey: true
 hosts:
 # The names for App-Slave servers start at 2, since No. 1 is the App-Master server
 - 'http://10.20.0.XX' # <- Varnish cache for App Slave 2 (10.20.0.X)
 - 'http://10.20.0.ZZ' # <- Varnish cache for App Slave 3 (10.20.0.Z)

Additionally, the environment variable SHOPWARE_HTTP_CACHE_ENABLED=1 must be set in the .env file.

Varnish Configuration

The Varnish configuration can be edited and deployed directly via the configuration module in the Customer Center. Navigate to Server → Server X → Configuration Files → Varnish, to customize the Varnish configuration.

Shopware 6 Varnish XKey VCL

The latest Shopware 6 Varnish VCL can be downloaded directly from GitHub:

vcl 4.1;

import std;
import xkey;
import cookie;

# Specify your app nodes here. Use round-robin balancing to add more than one.
backend default {
    .host = "__SHOPWARE_BACKEND_HOST__";
    .port = "__SHOPWARE_BACKEND_PORT__";
}

# ACL for purgers' IPs. (This must contain app server IPs)
acl purgers {
    "127.0.0.1";
    "localhost";
    "::1";
    __SHOPWARE_ALLOWED_PURGER_IP__;
}

sub vcl_recv {
    # Handle PURGE
    if (req.method == "PURGE") {
 if (client.ip !~ purgers) {
 return (synth(403, "Forbidden"));
 }
        if (req.http.xkey) {
 set req.http.n-gone = xkey.purge(req.http.xkey);

 return (synth(200, "Invalidated "+req.http.n-gone+" objects"));
        } else {
 return (purge);
 }
    }

 if (req.method == "BAN") {
 if (client.ip !~ purgers) {
 return (synth(403, "Forbidden"));
 }

        ban("req.url ~ " + req.url);
 return (synth(200, "BAN URLs containing (" + req.url + ") done."));
    }

 # Only handle relevant HTTP request methods
    if (req.method != "GET" &&
 req.method != "HEAD" &&
 req.method != "PUT" &&
 req.method != "POST" &&
 req.method != "PATCH" &&
        req.method != "TRACE" &&
 req.method != "OPTIONS" &&
 req.method != "DELETE") {
 return (pipe);
    }

 if (req.http.Authorization) {
 return (pass);
    }

 # We only handle GET and HEAD by default
    if (req.method != "GET" && req.method != "HEAD") {
 return (pass);
    }

 # Micro-optimization: Always pass these paths directly to PHP without caching
    # to prevent hashing and cache lookup overhead
    # Note: virtual URLs might bypass this rule (e.g., /en/checkout)
    if (req.url ~ "^/(checkout|account|admin|api)(/.*)?$") {
 return (pass);
    }

    # Static assets do not require any cookies. Stripping them ensures a stable cache
    # key when assets are served from the same domain as the storefront, so the
    # hash does not vary based on sw-cache-hash or sw-currency.
    if (req.url ~ "^/(media|thumbnail|theme|bundles)/") {
 unset req.http.Cookie;
    }

 cookie.parse(req.http.cookie);

    # Set the cache-hash cookie value to the header for hashing based on the `vary` header
    # If the header is provided directly, it takes precedence
    if (!req.http.sw-cache-hash) {
 set req.http.sw-cache-hash = cookie.get("sw-cache-hash");
    }

 set req.http.currency = cookie.get("sw-currency");
    set req.http.states = cookie.get("sw-states");

    if (req.url == "/widgets/checkout/info" && (req.http.sw-cache-hash == "" || (cookie.isset("sw-states") && req.http.states !~ "cart-filled"))) {
 return (synth(204, ""));
    }

 #  Ignore query strings that are only necessary for the client-side JavaScript. Customize as needed.
    if (req.url ~ "(\?|&)(pk_campaign|piwik_campaign|pk_kwd|piwik_kwd|pk_keyword|piwik_keyword|mtm_campaign|matomo_campaign|mtm_cid|matomo_cid|mtm_kwd|matomo_kwd|mtm_keyword|matomo_keyword|mtm_source|matomo_source|mtm_medium|matomo_medium|mtm_content|matomo_content|mtm_group|matomo_group|mtm_placement|matomo_placement|pixelId|kwid|kw|chl|dv|nk|pa|camid|adgid|yclid|utm_term|utm_source|utm_medium|utm_campaign|utm_content|cx|ie|cof|siteurl|_ga|adgroupid|campaignid|adid|utm_id|utm_source_platform|utm_creative_format|utm_marketing_tactic|_gl|gclsrc|gPromoCode|gQT|gclid|srsltid|dclid|gbraid|wbraid|gad_source|gad_campaignid|fbclid|fb_action_ids|fb_action_types|fb_source|mc_cid|mc_eid|_bta_tid|_bta_c|trk_contact|trk_msg|trk_module|trk_sid|gdfms|gdftrk|gdffi|_ke|_kx|redirect_log_mongo_id|redirect_mongo_id|sb_referer_host|mkwid|pcrid|ef_id|s_kwcid|msclkid|dm_i|epik|pp|twclid|hsa_cam|hsa_grp|hsa_mt|hsa_src|hsa_ad|hsa_acc|hsa_net|hsa_kw|hsa_tgt|hsa_ver|_branch_match_id|mkevt|mkcid|mkrid|campid|toolid|customid|igshid|si|ttclid|ScCid|rtid|irclickid|klar_source|klar_cpid|klar_adid)=") {
 # see RFC 3986, Section 2.3 "Unreserved Characters" for the regex
 set req.url = regsuball(req.url, "(pk_campaign|piwik_campaign|pk_kwd|piwik_kwd|pk_keyword|piwik_keyword|mtm_campaign|matomo_campaign|mtm_cid|matomo_cid|mtm_kwd|matomo_kwd|mtm_keyword|matomo_keyword|mtm_source|matomo_source|mtm_medium|matomo_medium|mtm_content|matomo_content|mtm_group|matomo_group|mtm_placement|matomo_placement|pixelId|kwid|kw|chl|dv|nk|pa|camid|adgid|yclid|utm_term|utm_source|utm_medium|utm_campaign|utm_content|cx|ie|cof|siteurl|_ga|adgroupid|campaignid|adid|utm_id|utm_source_platform|utm_creative_format|utm_marketing_tactic|_gl|gclsrc|gPromoCode|gQT|gclid|srsltid|dclid|gbraid|wbraid|gad_source|gad_campaignid|fbclid|fb_action_ids|fb_action_types|fb_source|mc_cid|mc_eid|_bta_tid|_bta_c|trk_contact|trk_msg|trk_module|trk_sid|gdfms|gdftrk|gdffi|_ke|_kx|redirect_log_mongo_id|redirect_mongo_id|sb_referer_host|mkwid|pcrid|ef_id|s_kwcid|msclkid|dm_i|epik|pp|twclid|hsa_cam|hsa_grp|hsa_mt|hsa_src|hsa_ad|hsa_acc|hsa_net|hsa_kw|hsa_tgt|hsa_ver|_branch_match_id|mkevt|mkcid|mkrid|campid|toolid|customid|igshid|si|ttclid|ScCid|rtid|irclickid|klar_source|klar_cpid|klar_adid)=[A-Za-z0-9\-\_\.\~%]+&?", "");
    }

    set req.url = regsub(req.url, "(\?|\?&|&)$", "");

 # Normalize query arguments
    set req.url = std.querysort(req.url);

    # Set a header announcing Surrogate Capability to the origin
    set req.http.Surrogate-Capability = "shopware=ESI/1.0";

 # Ensure that the client's IP address is forwarded to the origin.
    if (req.http.x-forwarded-for) {
        set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
    } else {
 set req.http.X-Forwarded-For = client.ip;
    }

    return (hash);
}

sub vcl_hash {
    # Take Shopware HTTP cache cookies into account
    if (req.http.sw-cache-hash != "") {
 hash_data("+context=" + req.http.sw-cache-hash);
    } elseif (req.http.currency != "") {
 hash_data("+currency=" + req.http.currency);
    }
}

sub vcl_hit {
  # Consider client states for response headers
  if (req.http.states) {
 if (req.http.states ~ "logged-in" && obj.http.sw-invalidation-states ~ "logged-in" ) {
 return (pass);
 }

 if (req.http.states ~ "cart-filled" && obj.http.sw-invalidation-states ~ "cart-filled" ) {
 return (pass);
 }
  }
}

sub vcl_backend_fetch {
    unset bereq.http.currency;
    unset bereq.http.states;
}

sub vcl_backend_response {
    # Serve stale content for three days after object expiration
    set beresp.grace = 3d;

 unset beresp.http.X-Powered-By;
    unset beresp.http.Server;

 # This should happen before any early return via `deliver`, so that ESI can still be processed
    if (beresp.http.Surrogate-Control ~ "ESI/1.0") {
 unset beresp.http.Surrogate-Control;
        set beresp.do_esi = true;
    }

 # Reducing the "hit-for-miss" duration for dynamically uncacheable responses
    if (beresp.http.sw-dynamic-cache-bypass == "1") {
        # Mark as "Hit-For-Miss" for the next n seconds
 set beresp.ttl = 1s;
 set beresp.uncacheable = true;
 unset beresp.http.sw-dynamic-cache-bypass;
 return (deliver);
    }

 if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
 set beresp.do_gzip = true;
    }

    if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
 unset beresp.http.Set-Cookie;
    }
}

sub vcl_deliver {
    ## We don't want the client to cache anything except assets and store-api responses
    if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(theme|media|thumbnail|bundles|store-api)/") {
 set resp.http.Pragma = "no-cache";
        set resp.http.Expires = "-1";
 set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
    }

 # invalidation headers are for internal use only
    unset resp.http.sw-invalidation-states;
    unset resp.http.xkey;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.Link;
}c

Defining the Backend

To facilitate simple vertical scaling in our clusters, a Varnish cache is always operated in a 1:1 ratio with the app slave servers. This means that if there are 2 app slave servers, there must also be 2 Varnish cache servers in the setup.

In the default backend, the app slave server with a 1:1 mapping is specified. Example for Varnish server #2:

backend default {
    .host = "10.20.0.Y"; # <- VPC IP address of App Slave #2
    .port = "80"; # <- Web server port for 10.20.0.Y 
}

Define purgers

Purgers are the servers authorized to clear the cache via a BAN request. These are primarily the app servers. All app servers must be specified.

# ACL for purgers' IPs. (This must contain app server IPs)
acl purgers {
    "127.0.0.1";
    "localhost";
    "::1";
    "10.20.0.X"; # <- VPC IP address of App Master #1
    "10.20.0.Y"; # <- VPC IP address of App Slave #2
    "10.20.0.Z"; # <- VPC IP address of App Slave #3
#   ...    
}

Enable the load balancer for BAN requests via URL

Varnish XKey also supports BANs via the path shop.creoline-demo.de/products, so that only specific sections of the website are removed from the HTTP cache. Often, the ban request is resolved directly via the domain name, which is why the load balancer must also be configured as a purger in this case.

# ACL for purgers' IPs. (This must contain app server IPs)
acl purgers {
    "127.0.0.1";
    "localhost";
    "::1";
    "10.20.0.X"; # <- VPC IP address of App Master #1
    "10.20.0.Y"; # <- VPC IP address of App Slave #2
    "10.20.0.Z"; # <- VPC IP address of App Slave #3
#   ...
    "10.20.0.1"; # <- VPC IP address of the load balancer  
}

Soft Purge vs. Hard Purge

The Varnish configuration above uses hard purges by default, which ensures that a page is removed from the cache and that the next request takes longer. To counteract this, you can use soft purges, which first serve the outdated page to the client and then update the cache for that webpage in the background.

Soft purges can be enabled by modifying the Varnish configuration; see line 27 in the Varnish configuration above:

# Hard purge
set req.http.n-gone = xkey.purge(req.http.xkey);

# Soft purge
set req.http.n-gone = xkey.softpurge(req.http.xkey);

HTTP Cache for Logged-In Users or Visitors with Shopping Carts

By default, the Shopware HTTP cache is only available to non-logged-in visitors without shopping cart contents. If you do not use any customizations for logged-in users, the HTTP cache can also be enabled for logged-in users with shopping carts.

# config/packages/prod/shopware.yaml
shopware:
    cache:
 invalidation:
 http_cache: []

Debugging

The default configuration removes all HTTP headers except for Age, which is used to determine the cache age. An Age⁣ of 0 means that the cache is not working. This is usually because the Cache-Control: public HTTP header was not set by the web application.

To verify this, use the curl command as follows:

curl -vvv -H 'Host: <sales-channel-domain>' <app-server-ip> 1> /dev/null

which should return the following response:

< HTTP/1.1 200 OK
< Cache-Control: public, s-maxage=7200
< Content-Type: text/html; charset=UTF-8
< Xkey: theme.sw-logo-desktop, ...

If the Cache-Control: public or xkey: … HTTP header is not present in the response, this is likely due to an incorrect configuration in the web application itself.

Check the Shopware 6 configuration to ensure that reverse proxy mode has been correctly enabled in shopware.yaml.

Error Analysis

HTTP Status 503 Backend Fetch Failed on Individual Product and/or Category Pages

Depending on your store’s requirements, the size of the HTTP header may exceed the default size configured in Varnish for one or more product and category pages.

Check

Connect to your Varnish server via SSH and run the following command:

curl -I -H "Host: <sales-channel-domain>" -H "X-Forwarded-Proto: https" http://<app-server-ip>/path/with/error/503 | wc -c

If the result exceeds 8192, you’ll need to adjust the Varnish cache Systemd service to set new default sizes.

Solution

Run the following command as the root user:

systemctl edit varnish

Adjust the ExecStart setting as follows:

[Service]
ExecStart=/usr/sbin/varnishd -a :80 -a localhost:8443,PROXY -f /etc/varnish/default.vcl -P %t/%N/varnishd.pid -p feature=+http2 -s malloc,2g \
 -p http_req_hdr_len=32k \
         -p http_resp_hdr_len=32k \
 -p http_resp_size=64k
Restart=on-failure

Next, reload the Varnish cache to apply the configuration.

systemctl reload varnish

HTTP Status Error 502 Bad Gateway

Checking FastCGI Proxy Buffers

It is possible that the size of the FastCGI proxy buffers in Nginx is insufficient for Varnish XKey, which is why the following settings must be specified in the server or location directive.

Check the corresponding Nginx error log file for the following error message:

grep "upstream sent too big header" /var/log/nginx/error.log

Solution

Increase the following settings within your Nginx Server or Location directive as follows:

nano /etc/nginx/conf.d/website.conf

Option 1: Server Directive

server {
 ...
    fastcgi_buffers 8 16k;
    fastcgi_buffer_size 32k;
 ...
}

Option 2: Location Directive

location XXX {
 ...
    fastcgi_buffers 8 16k;
    fastcgi_buffer_size 32k;
 ...
}

Test the configuration and then reload Nginx:

nginx -t && systemctl reload nginx