linux has a limit on the number of concurrent memory mappings. (64k)

include the number of memory maps used by varnishd in the report.

Something like:
root@varnish:/# for i in pidof varnishd; do wc -l /proc/$i/maps; done

might even grep for readwrite mappings, to avoid the binaries/shared ones.

/proc/$$/limits doesn't list cgroup limits, we might want to capture contents in cgroup sysfs (/sys/fs/cgroup) as well.

That is when both are available. netstat will choke on systems with too many open connections.

lkarsten@immer:/work/varnishgather$ python ./varnishgather.py
Running date
Running dmesg
Running free -m
Running lsb_release -a
Running varnishstat -V
Running sysctl -a
Traceback (most recent call last):
File "./varnishgather.py", line 142, in
LoggedCommand(cmd)(tar)
File "./varnishgather.py", line 100, in call
self.run()
File "./varnishgather.py", line 95, in run
p = Popen(self.command, stdout=PIPE, stderr=PIPE)
File "/usr/lib/python2.7/subprocess.py", line 679, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
lkarsten@immer:/work/varnishgather$
lkarsten@immer:~/work/varnishgather$ git log --oneline | head -n1
7da75b6 Add warning and sleep when run as non-root

Do debug 503s it is often not enough with the 20k varnishlog lines included.

It should be an option to make a complete dump of the shmlog (varnishlog -d) and include that in the .tgz file.

Of course this will make the file very big (50-100MB), which may lead to problems distributing it by mail.

When reading a VCL file, a short grep for "include foo" should be done. Any matching files should also be included in the report.

We should include data from vha, if it is in use.

/var/lib/vha-agent/vha-status

Grab the CLI file if you run varnish like this:

varnishd -I /etc/varnish/start.cli -f ""

Mostly to see if the system is using transparent hugetables (AnonHugePages), which is known to cause problems with Varnish on certain systems.

If the varnish agent is installed on the system its configuration should be included in the varnishgather report.

ssia

We have a small file upload service for customers to send us huge files that isn't possible to send via email.

varnishgather could be extended to have a upload-to-url function, where the produced output file could be POST-ed via curl/wget to this URL.

Since this is HTTP we don't have the usual file size limitations that SMTP has.

This will make data collection simpler for the customer/user.

https://github.com/varnish/varnishgather/blob/master/varnishgather#L461 isn't specific enough and will dump the file storage files into the gather

It can be useful to focus on what was tuned: param.show changed

Now that we're making a tarball of the report anyway, it might be as well to just output each test into another file.

That way you don't have to step through the 5k lines of dmesg just to get to varnishstat below.

Varnishlog runs with -d -k 20000, which runs with no problems on a busy system. On an idle system, however, it will hang for a long time while waiting for 20000 requests. Some timeout might be appropriate.

Via the agent, we return a slightly different version of stats, that you get via "varnishstat".

localhost:6085/stats

Potential problems:

• Getting the correct port it is listening on.
• Getting the username and password for use in "-u".

Possible command, if we assume defaults.

curl -u$(cat /etc/varnish/agent_secret) localhost:6085/stats -o stats.json

If the vcl_path is pointing outside of /etc/varnish or /usr, include those directories in the VCLs tar file.

Running iptables (-L etc) on a system without the iptables modules loaded will load the modules. Varnishgather should detect if the modules have already been loaded, and only execute the iptables commands if they have.

The default dmesg output contains timestamps in the form of number of seconds since the system was booted. It is challenging to parse manually.

The -T parameter to dmesg will print human-readable timestamps in the dmesg output, which will make entries in the dmesg output easier to map to entries in other log files.

From the dmesg man page on -T:

Be aware that the timestamp could be inaccurate! The time
source used for the logs is not updated after system SUS-
PEND/RESUME.

Varnish servers are rarely suspended, so this is probably not a problem.

I would like to have where possible the username from the repos url, so we can see in the gather how the machine links to the customer.

I feel we must filter out the password.

Via apt or yum

structure in apt source file:

deb https://USER:[email protected]/ubuntu trusty non-free

structure in yum source file:

baseurl=https://USER:[email protected]/redhat/varnish/el$releasever

apt

/etc/apt/sources.list.d/varnish-XXXX.list

yum

/etc/yum/sources.list.d/varnish-XXXX.list

As Dag pointed out today, there should be some CPU core utilisation numbers in varnishgather.

If varnish is doing excessive regex matches (lots of bans) or somehow has started using a lot of CPU, it isn't really obvious from the current data extracts.

mpstat -A was suggested for Linux. It comes from the sysstat package on rhel and debian, which doesn't seem to be installed by default.

When running the script on RedHat 6.6 it gives a error for the UID variable:

sudo ./varnishgather.sh

./varnishgather.sh: line 43: UID: readonly variable

getarg doesn't handle the case where there is no spaces between the parameter and it's arguments, e.g. -T:6082

varnishgather outputs a .tar.gz file, but the text output isn't clear on if the output is the .log file or not.

• /status output
• logs

Adding two items:

1. "lsof -p $pid"
2. "lsof -p $pid | wc -l"

will help when debugging open files.

nodes.conf is no longer in /etc/vha-agent/, its now in /etc/varnish.

Update the calls from mycat ...service to to be run systemctl cat ...service. This should also result in some lines becoming redundant (and therefore can be removed).

If a customer's environment has outdated ca certs, they will be unable to upload a varnish gather to our filebin service.
It would be nice if we could provide a simple solution for our customers using varnishgather.

For example, adding the -k argument to the script would enable the corresponding insecure mode in curl.
sh ./varnishgather -k -u simukka

Consider including a tar file of the persistence directory via "-p".

Today, this would require some logic to parse the "defaults/varnish-agent" or the "ps aux" to figure out which directory. An easier way in the future, could be to have the agent write to file, or have it respond via a curl request with its location.

Get the version of the varnish-agent if it is running.

varnish-agent -V

It would be helpful if varnishgather output a warning to the user if there is no varnish on the server it is run on.

OpenSSL is a must, but we have other interesting package dependencies.

This is confusing: we either have releases or just use master as the stable branch and forget about tags.

getarg -s might return values per type of argument (malloc, transient, mse, etc.). Eg., getarg -s mse, returns MSE full path. This option is required for the PR (#49).

It would be nice to include resource usage information from sysstat (using sar), if it is enabled.

Currently the .tar.gz files created by varnishgather has a varnishgather/ directory with all the content.

If you uncompress multiple archives in the same directory they will overwrite each other, creating confusion.

Suggestion: let the subdirectory be varnishgather-HOSTNAME/ instead.

Suggest adding the following tests to include which packages are installed on the system:

rpm -qa | grep varnish

dpkg -l | grep varnish

This doesn't work in V4 so we need to update the script to workaround it, perhaps by using a timeout.

problematic line:

include "foo.vcl" // my foo vcl

// is interpreted as a path, chaos ensues

Gather json output via:

varnishstat -j