This page concerns regular tasks to be perform on Varnish.

Contents

  1. Good debugging tools
    1. trace requests only from one IP
    2. trace backend requests from that IP
    3. test VCL syntax
    4. Show requests grouped by host or URL pattern
  2. Upgrades
  3. Tâches avancées
    1. avoir des stats de varnish par munin
    2. Adding a site to varnish, detailed configuration
      1. Varnish configuration
        1. VCL for Pressflow
        2. VCL for basic Drupal
          1. Advanced drupal (w/ x-forward-proto support)
          2. Config côté Drupal
        3. Config côté Spip
        4. Wordpress
          1. WP plugin: Proxy Cache Purge
          2. Attention: certains plugins empêchent le caching!
          3. PolyLang: work around pour site multilangue
      2. Putting in production: enabling DNS
        1. "Normal" configuration
        2. AlternC specific configuration
    3. Flusher la cache de Varnish
      1. Flusher le cache d'un seul site
        1. Varnish 3.x+ incluant 6.5
        2. Varnish 2.x
      2. Permettre à un client de flusher la cache
    4. Inspecting load balancer status
    5. Watching the logs, stats
    6. Common configuration snippets
      1. Custom error pages
      2. Load balancing
      3. Changing the backend depending on the URL
      4. High-performance tweaks
      5. Apache LogFormat to log the X-Forwarded-For
  4. Troubleshooting
    1. 503 errors
  5. Testing and benchmarking
  6. References

Good debugging tools

Those are life savers:

trace requests only from one IP

varnishlog -c -m ReqStart:66.130.160.12

varnishlog -c -o SessionOpen 66.130.160.12

(Found that tip here) (Tip for varnish 3 found here)

trace backend requests from that IP

This makes it possible to see, for example, only requests that you're issuing (to make debugging easier): 

varnishlog -b -q 'BeReqHeader ~ "X-Forwarded-For: 184.160.35.43"'

You can also complexify matching using the VSL query language.

See: https://varnish-cache.org/docs/6.0/reference/vsl.html#vsl-7

test VCL syntax

varnishd -C -f /etc/varnish/default.vcl

(Found that in http://aaronbonner.io/post/14125553826/testing-varnish-vcl-syntax)

Show requests grouped by host or URL pattern

This tool helps with grouping information from varnishlog and presents it in an easy format (still untested!):

https://github.com/xcir/varnishHostStat

Upgrades

Just apt-get install and fix the syntax errors you find. Look into site_cache/files in puppet for previous upgrades and upstream documentation.

Tâches avancées

avoir des stats de varnish par munin

On peut avoir des stats de varnish en utilisant les plugins de munin. On a ça de codé dans puppet. 

include cache::varnish::munin

exemple: https://stats.koumbit.net/varnish-day.html

Adding a site to varnish, detailed configuration

Varnish configuration

If you need a special configuration, you add it to a domain-specific if clause. The VCL (Varnish Configuration Language) will then vary according to the site being optimised.

VCL for Pressflow

Pressflow is a distribution of Drupal with integrated performance, scalability, availability, and testing enhancements. As such, it allows caches such as Varnish to cache cacheable content very easily. We still need to modify the Varnish VCL to have this actually work properly. Our config is based on this configuration and is deployed by puppet on both servers.

(!) By default, we apply the above configuration to all hosts, so we support Pressflow out of the box. -- TheAnarcat 2011-02-01 21:41:40

(!) We also use ceres as a backend so if your site is in a pressflow platform on aegir, there should be no modification necessary on the varnish server(s).

VCL for basic Drupal

The following configuration will completely remove cookies from all requests in Varnish, which will break all dynamic functionality on the website (logins, etc) but yield major performance improvements. The idea here is that the "dynamic" website is edited on another domain that doesn't point to varnish. 

sub vcl_recv {
        if (req.http.host ~ "^(.*\.)?example.net$") {
                set req.backend = example;
                remove req.http.Set-Cookie;
                remove req.http.cookie;
                lookup;
        }
}

sub vcl_fetch {
        if (req.http.host ~ "^(.*\.)?example.net$" && obj.http.Set-Cookie) {
                insert;
        }
}

Advanced drupal (w/ x-forward-proto support)

sub vcl_deliver {
  if (obj.hits > 0) {
    set resp.http.X-Varnish-Cache = "HIT";
  }
  else {
    set resp.http.X-Varnish-Cache = "MISS";
  }
}

sub vcl_recv {
  // AegirVPS only.
  //set req.backend_hint = aegirvps.backend();

  // configuration from https://wiki.fourkitchens.com/display/PF/Configure+Varnish+for+Pressflow
  if (req.method != "GET" &&
    req.method != "HEAD" &&
    req.method != "PUT" &&
    req.method != "POST" &&
    req.method != "TRACE" &&
    req.method != "OPTIONS" &&
    req.method != "DELETE") {
      /* Non-RFC2616 or CONNECT which is weird. */
      return (pipe);
  }

  if (req.method != "GET" && req.method != "HEAD") {
    /* We only deal with GET and HEAD by default */
    return (pass);
  }

  if (req.http.Cookie) {
    // Remove has_js and Google Analytics cookies.
    set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|__utma_a2a|has_js)=[^;]*", "");

    // more removal, from: https://www.varnish-cache.org/trac/wiki/VCLExampleRemovingSomeCookies

    // remove other stats cookies
    set req.http.Cookie = regsuball(req.http.Cookie, "(^|; ) *__utm.=[^;]+;? *", "\1"); # removes all cookies named __utm? (utma, utmb...) - tracking thing
    // Remove all cookies but sessions
    set req.http.Cookie = ";" + req.http.Cookie;
    set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
    set req.http.Cookie = regsuball(req.http.Cookie, ";(SESS[a-zA-Z0-9=]+)=", "; \1=");
    set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");

    // Remove a ";" prefix, if present.
    set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");

    // Remove empty cookies.
    if (req.http.Cookie ~ "^\s*$") {
      unset req.http.Cookie;
    }
  }

  if (req.http.Authorization || req.http.Cookie) {
    /* Not cacheable by default */
    return (pass);
  }

  // Skip the Varnish cache for install, update, and cron
  if (req.url ~ "install\.php|update\.php|cron\.php") {
    return (pass);
  }

  if (req.url ~ "x") {
    ban ("req.url ~ .");
    return(synth(500, "the cache is now purged."));
  }

  // Normalize the Accept-Encoding header
  // as per: http://varnish-cache.org/wiki/FAQ/Compression
  if (req.http.Accept-Encoding) {
    if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
      # No point in compressing these
      unset req.http.Accept-Encoding;
    }
    elsif (req.http.Accept-Encoding ~ "gzip") {
      set req.http.Accept-Encoding = "gzip";
    }
    else {
      # Unknown or deflate algorithm
      unset req.http.Accept-Encoding;
    }
  }

  return (hash);
}


// configuration from https://wiki.fourkitchens.com/display/PF/Configure+Varnish+for+Pressflow
sub vcl_hash {
        hash_data(req.url);
        if (req.http.host) {
          hash_data(req.http.host);
        } else {
          hash_data(server.ip);
        }
        if (req.http.X-Forwarded-Proto ~ "https") {
          hash_data(req.http.X-Forwarded-Proto);
        }
        if (req.http.Cookie) {
                hash_data(req.http.Cookie);
        }
        return (lookup);
}

// Strip any cookies before an image/js/css is inserted into cache.
// configuration from https://wiki.fourkitchens.com/display/PF/Configure+Varnish+for+Pressflow
sub vcl_backend_response {
 if (bereq.url ~ "\.(png|gif|jpg|swf|css|js)$") {
   // For Varnish 2.1 or later, replace obj with beresp:
   unset beresp.http.set-cookie;
 }

 // Let's have a little grace
 set beresp.ttl = 10s;
 set beresp.grace = 1h;
}

(!) This should probably be updated with the information available here: http://varnish-cache.org/wiki/VarnishAndDrupal

Config côté Drupal

Dans local.settings.php 

<?php
global $conf;
global $is_https;
$conf['reverse_proxy'] = TRUE;
$conf['reverse_proxy_addresses'] = array('199.58.80.104', '199.58.80.105', '127.0.0.1');
$conf['reverse_proxy_header'] = 'HTTP_X_FORWARDED_FOR';
// $conf['page_cache_invoke_hooks'] = FALSE;
if (isset($_SERVER['HTTP_X_FORWARDED_PROTO']) && $_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https') {

  // uncomment if using domain access and testing on non-standard ports
  //$hosts = explode(':', $_SERVER['HTTP_HOST']);
  //if ($hosts) {
  //  $_SERVER['HTTP_HOST'] = $hosts[0];
  //}

  $conf['https'] = TRUE;
  $_SERVER['https'] = 'on';
  $is_https = TRUE;
  $base_url = 'https://'.$_SERVER['SERVER_NAME'];
}

// uncomment if using domain access
// include dirname(__FILE__)."/modules/domain/settings.inc";

Config côté Spip

Voir https://wiki.koumbit.net/SpIp#Configuration_en_arri.2BAOg-re_de_varnish

Wordpress

Wordpress is relatively easy to setup for caching. However, it should be noted that by default, wordpress does not send any header for setting max age, so a plain wordpress will not be cached! To make a wordpress site interact correctly with varnish, it is required that a caching plugin be installed in wordpress. Hypercache is a good plugin for this purpose.

The following VCL snippet will remove any cookie sent by client requests so that we can cache the same page for different ppl. If a cookie is set for authenticated users, it'll force requests to be sent to the backend: 

# Drop any cookies sent to Wordpress.
sub vcl_recv {
        if (!(req.url ~ "wp-(login|admin)")) {
                unset req.http.cookie;
        }
}

# Drop any cookies Wordpress tries to send back to the client.
sub vcl_fetch {
        if (!(req.url ~ "wp-(login|admin)")) {
                unset beresp.http.set-cookie;
        }
}

reference: https://www.varnish-cache.org/trac/wiki/VarnishAndWordpress

WP plugin: Proxy Cache Purge

Bon plugin pour gérer le purge des pages. Permet de tester le cache via:  /wp-admin/admin.php?page=varnish-check-caching 

Attention: certains plugins empêchent le caching!

Il y a certain plugin qui empêche le caching avec varnish, notamment par qu'ils utilisent des cookies spéciaux. On peut voir les plugins problématiques dans la page de test.

PolyLang: work around pour site multilangue

Si tu veux que WP puisse cacher un site qui utilise le plugin PolyLang, il faut suivre ce workaround:

https://dev.to/tammalee/busted-how-i-solved-a-varnish-cache-issue-238k https://polylang.pro/doc/is-polylang-compatible-with-the-eu-cookie-law/

Doc, il faut ajouter ceci à wp-config.php: 

define( 'PLL_COOKIE', false);

On va encore avoir l'avertissement dans la page de test pour varnish, mais on voit que ça marche maintenant.

Putting in production: enabling DNS

Our setup is such that we have multiple varnish servers providing the CachingService. We therefore have one IP in the frontend that our LoadBalancingService redirects to the proper cache. That is the "virtual" cache.koumbit.net, which is not a real server, but just an entry in the load balancer.

"Normal" configuration

Normally, this would just be changing the A record of the main domain to point to the caching system while keeping the Apache vhost unchanged, so that it keeps on answering properly to varnish requests.

Basically, the domain still needs to be configured in Apache while the DNS needs to point somewhere else. The way to do this will vary according to the control panel and environment.

AlternC specific configuration

With AlternC, that's simply not possible because the DNS configuration is coupled with the Apache configuration. The workaround is to have a domain (say dynamic.example.com) that serves the dynamic website and is properly configured in AlternC, while the main domain (example.com) is an IP redirection to the caching system. Varnish also needs to be configured to access the server with the right hostname, using a directive like: 

        } elseif (req.http.host ~ "^(.*\.)?example.com$") {
                set req.http.host = "dynamic.example.com";

Since DNS and Apache configurations are coupled in AlternC, care needs to be taken when changing the configuration. It should be a process a bit like this:

  1. add a dynamic.example.com pointer that points to the same place as example.com, test it

  2. 24h in advance, lower the TTLs to 5 minutes or skip to the next step

  3. change the A pointer of example.com to the caching system directly in the zonefile (in /var/alternc/bind/zones/example.com)

  4. wait 5 minutes or 24h, depending if the TTL was changed

  5. change the example.com domain to point to the IP in AlternC

Flusher la cache de Varnish

# varnishadm -T 127.0.0.1:6082 url.purge "."

Pour vider la cache:

Aussi: 

url.purge ^/$

... pour purger le frontpage. L'argument est simplement un URL, donc on peut purger n'importe quel objet de cette façon. Voir la documentation officielle pour plus d'information. Les modules expire ou purge permettent également d'intégrer ceci à Drupal.

Flusher le cache d'un seul site

On peut entrer dans le CLI de Varnish avec la commande (fonctionne avec Varnish 6.5) : 

# varnishadm -T127.0.0.1:6082 -S /etc/varnish/secret

Varnish 3.x+ incluant 6.5

Pour tout flusher pour un certain site: 

varnish> ban req.http.host == "www.example.com"

Pour flusher une page spécifique: 

varnish> ban req.http.host == "www.example.com" && req.url ~ "^/pageblah$"

ref:

Varnish 2.x

varnish> purge req.http.host == www.example.qc.ca

Si tu veut flusher une page spécifique d'un site, il faut utiliser req.url et le syntax regex: 

varnish> purge req.http.host == www.example.qc.ca && req.url ~ ^/pageblah$

Permettre à un client de flusher la cache

encore tiré de: https://www.varnish-cache.org/docs/3.0/tutorial/purging.html 

@@ -4,6 +4,17 @@
 # include "/etc/varnish/conf.d/";
 #
 # but that's not supported by varnish yet
+acl purge {
+       "localhost";
+       "10.10.10.11"/24;
+       "199.58.80.233";
+       "199.58.80.234";
+       "199.58.80.237";
+
 backend default {
         .host = "127.0.0.1";
         .port = "8080";
@@ -63,7 +74,30 @@ director microsites round-robin {
   { .backend = micro; }
 }

+# from https://www.varnish-cache.org/docs/3.0/tutorial/purging.html
+sub vcl_hit {
+        if (req.request == "PURGE") {
+                purge;
+                error 200 "Purged.";
+        }
+}
+
+sub vcl_miss {
+        if (req.request == "PURGE") {
+                purge;
+                error 200 "Purged.";
+        }
+}
+
 sub vcl_recv {
+  # allow PURGE from localhost and 192.168.55...
+  if (req.request == "PURGE") {
+    if (!client.ip ~ purge) {
+      error 405 "Not allowed.";
+    }
+    return (lookup);
+  }
+
   // Set this header so that backends can take evasive maneuvers in case of

Inspecting load balancer status

In load balancing setups (see below for examples), we can inspect the backend status using: 

someserver# varnishadm -S /etc/varnish/secret -T127.0.0.1:6082 backend.list -p
Backend sql01 is Healthy
Current states  good:  5 threshold:  3 window:  5
Average responsetime of good probes: 0.000559
Oldest                                                    Newest
================================================================
---------------------------------------------4444444444444444444 Good IPv4
---------------------------------------------XXXXXXXXXXXXXXXXXXX Good Xmit
---------------------------------------------RRRRRRRRRRRRRRRRRRR Good Recv
------------------------------------------HHHHHHHHHHHHHHHHHHHHHH Happy
Backend web01 is Healthy
Current states  good:  4 threshold:  3 window:  5
Average responsetime of good probes: 0.038032
Oldest                                                    Newest
================================================================
----------------------------------------------444444444444444444 Good IPv4
----------------------------------------------XXXXXXXXXXXXXXXXXX Good Xmit
----------------------------------------------RRRRRRRRRRRRRRR-RR Good Recv
-------------------------------------------HHHHHHHHHHHHHHHHHH-HH Happy
Backend web02 is Sick
Current states  good:  0 threshold:  3 window:  5
Average responsetime of good probes: 0.000000
Oldest                                                    Newest
================================================================
------------------------------------------HHH------------------- Happy
Backend web03 is Sick
Current states  good:  0 threshold:  3 window:  5
Average responsetime of good probes: 0.000000
Oldest                                                    Newest
================================================================
---------------------------------------------HHH---------------- Happy

If you need to interact with a varnish server that's older than 5.0, you might need to use the following command instead: 

varnishadm -S /etc/varnish/secret -T127.0.0.1:6082 debug.health

Also note that on even older setups, you might need to drop the -S /etc/varnish/secret argument.

In the above examples, the 2 first nodes are "healthy" (ie. up and running) while the two last are "sick" (ie. down, non functionnal). In this example, the two last nodes were never configured, which explains why only one line is present.

For this to work, probes need to be configured, see documentation about health checks and documentation for probes for more information.

Watching the logs, stats

Varnish, since it's so high-performance and everything, doesn't log anything to logfiles by default, but instead writes logging information to a shared memory segment. There are a few utilities bundled with varnish to analyse that log.

Les cookies envoyés au client: 

varnishtop -c -d -i TxHeader -I Set-Cookie:

Les URLs populaires: 

varnishtop -d -i RxURL

Dumping the log to disk and reading it: 

  varnishlog -w tmp.log
  varnishlog -r tmp.log  | less

Common configuration snippets

Custom error pages

this allows you to return custom HTML when an error occurs.

Load balancing

backend web01 {
  .host = "10.0.0.1";
  .port = "80";
  .probe = {
                .url = "/";
                .interval = 5s;
                .timeout = 1 s;
                .window = 5;
                .threshold = 3;
  }
}

# ...

director www round-robin {
    { .backend = web01; }
    { .backend = web02; }
    { .backend = web03; }
}

This will probe the three backend nodes and load balance in a round-robin fashion between those. A weight can be put on each node to favor or disfavor them in the array. See this example for more information and this for probe configurations.

Changing the backend depending on the URL

The following configuration will set a different backend depending on the requested URL: 

sub vcl_recv {
    if (req.url ~ "^http://(www.)?example.com/files/.*$") {
       set req.backend = static;
    } else {
       set req.backend = www;
    }
}

Here we wanted to send all static files requests to the static, non-dynamic server to be even faster. We're changing the backend here, but in fact anything could be modified in the request here...

High-performance tweaks

A grace setting is important if we have a *lot* of concurrent connexions.

Otherwise the performance page is instructive and fun to follow.

Apache LogFormat to log the X-Forwarded-For

In file: /etc/apache2/apache2.conf 

LogFormat "%v:%p %h %{X-Forwarded-For}i %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %{X-Forwarded-For}i %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined

The %{X-Forwarded-For}i will show the IP of the visitor that requested the page, since %h will show the IP of the Varnish server. You still want to keep %h, for example, if your SSL requests are hitting Apache directly.

Troubleshooting

503 errors

A website serving 503 errors intermittently is one of the most annoying things to troubleshoot with varnish.

The most common causes for this are:

backend is really slow; varnish backend timeout is reached 

backend "patate" {
  // ...
  .connect_timeut = 10s;
  .first_byte_timeout = 90s;
}

connection reuse might be causing bugs with apache mpm-itk

Varnish tends to reuse HTTP connections to pipe in more than one requests and save time on connection and handshake overhead. This behaviour, however, tends to cause bugs with apache mpm-itk

  1. the first request will make apache spawn a process with a certain UID and the first request is served correctly

  2. a subsequent request hitting a different apache vhost will be reusing the same UID and will thus fail to access some files

So: 

<Virtualhost *:80>
  # ...
  KeepAlive off
</VirtualHost>

c.f.: https://www.section.io/blog/varnish-cache-503-error-guru-meditation/

Testing and benchmarking

See VarnishTesting.

References

CategoryMaintenance

VarnishMaintenance (last edited 2024-05-29 10:05:41 by mathieul)