varnish / varnishgather Goto Github PK
View Code? Open in Web Editor NEWInformation gathering tool for Varnish Cache.
Information gathering tool for Varnish Cache.
linux has a limit on the number of concurrent memory mappings. (64k)
include the number of memory maps used by varnishd in the report.
Something like:
root@varnish:/# for i in pidof varnishd
; do wc -l /proc/$i/maps; done
might even grep for readwrite mappings, to avoid the binaries/shared ones.
/proc/$$/limits
doesn't list cgroup limits, we might want to capture contents in cgroup sysfs (/sys/fs/cgroup
) as well.
That is when both are available. netstat
will choke on systems with too many open connections.
lkarsten@immer:/work/varnishgather$ python ./varnishgather.py/work/varnishgather$
Running date
Running dmesg
Running free -m
Running lsb_release -a
Running varnishstat -V
Running sysctl -a
Traceback (most recent call last):
File "./varnishgather.py", line 142, in
LoggedCommand(cmd)(tar)
File "./varnishgather.py", line 100, in call
self.run()
File "./varnishgather.py", line 95, in run
p = Popen(self.command, stdout=PIPE, stderr=PIPE)
File "/usr/lib/python2.7/subprocess.py", line 679, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
lkarsten@immer:
lkarsten@immer:~/work/varnishgather$ git log --oneline | head -n1
7da75b6 Add warning and sleep when run as non-root
Do debug 503s it is often not enough with the 20k varnishlog lines included.
It should be an option to make a complete dump of the shmlog (varnishlog -d) and include that in the .tgz file.
Of course this will make the file very big (50-100MB), which may lead to problems distributing it by mail.
When reading a VCL file, a short grep for "include foo" should be done. Any matching files should also be included in the report.
We should include data from vha, if it is in use.
/var/lib/vha-agent/vha-status
Grab the CLI file if you run varnish like this:
varnishd -I /etc/varnish/start.cli -f ""
Mostly to see if the system is using transparent hugetables (AnonHugePages), which is known to cause problems with Varnish on certain systems.
If the varnish agent is installed on the system its configuration should be included in the varnishgather report.
ssia
We have a small file upload service for customers to send us huge files that isn't possible to send via email.
varnishgather could be extended to have a upload-to-url function, where the produced output file could be POST-ed via curl/wget to this URL.
Since this is HTTP we don't have the usual file size limitations that SMTP has.
This will make data collection simpler for the customer/user.
https://github.com/varnish/varnishgather/blob/master/varnishgather#L461 isn't specific enough and will dump the file
storage files into the gather
It can be useful to focus on what was tuned: param.show changed
Now that we're making a tarball of the report anyway, it might be as well to just output each test into another file.
That way you don't have to step through the 5k lines of dmesg just to get to varnishstat below.
Varnishlog runs with -d -k 20000
, which runs with no problems on a busy system. On an idle system, however, it will hang for a long time while waiting for 20000 requests. Some timeout might be appropriate.
Via the agent, we return a slightly different version of stats, that you get via "varnishstat".
localhost:6085/stats
Potential problems:
Possible command, if we assume defaults.
curl -u$(cat /etc/varnish/agent_secret) localhost:6085/stats -o stats.json
If the vcl_path is pointing outside of /etc/varnish
or /usr
, include those directories in the VCLs tar file.
Running iptables (-L etc) on a system without the iptables modules loaded will load the modules. Varnishgather should detect if the modules have already been loaded, and only execute the iptables commands if they have.
The default dmesg
output contains timestamps in the form of number of seconds since the system was booted. It is challenging to parse manually.
The -T
parameter to dmesg
will print human-readable timestamps in the dmesg
output, which will make entries in the dmesg
output easier to map to entries in other log files.
From the dmesg
man page on -T
:
Be aware that the timestamp could be inaccurate! The time
source used for the logs is not updated after system SUS-
PEND/RESUME.
Varnish servers are rarely suspended, so this is probably not a problem.
I would like to have where possible the username from the repos url, so we can see in the gather how the machine links to the customer.
I feel we must filter out the password.
Via apt or yum
structure in apt source file:
deb https://USER:[email protected]/ubuntu trusty non-free
structure in yum source file:
baseurl=https://USER:[email protected]/redhat/varnish/el$releasever
/etc/apt/sources.list.d/varnish-XXXX.list
/etc/yum/sources.list.d/varnish-XXXX.list
As Dag pointed out today, there should be some CPU core utilisation numbers in varnishgather.
If varnish is doing excessive regex matches (lots of bans) or somehow has started using a lot of CPU, it isn't really obvious from the current data extracts.
mpstat -A was suggested for Linux. It comes from the sysstat package on rhel and debian, which doesn't seem to be installed by default.
vha-agent -V
When running the script on RedHat 6.6 it gives a error for the UID variable:
./varnishgather.sh: line 43: UID: readonly variable
getarg doesn't handle the case where there is no spaces between the parameter and it's arguments, e.g. -T:6082
varnishgather outputs a .tar.gz file, but the text output isn't clear on if the output is the .log file or not.
Adding two items:
will help when debugging open files.
nodes.conf is no longer in /etc/vha-agent/, its now in /etc/varnish.
Line 470 in a4ebe17
Update the calls from mycat ...service
to to be run systemctl cat ...service
. This should also result in some lines becoming redundant (and therefore can be removed).
If a customer's environment has outdated ca certs, they will be unable to upload a varnish gather to our filebin service.
It would be nice if we could provide a simple solution for our customers using varnishgather.
For example, adding the -k
argument to the script would enable the corresponding insecure mode in curl.
sh ./varnishgather -k -u simukka
Consider including a tar file of the persistence directory via "-p".
Today, this would require some logic to parse the "defaults/varnish-agent" or the "ps aux" to figure out which directory. An easier way in the future, could be to have the agent write to file, or have it respond via a curl request with its location.
Get the version of the varnish-agent if it is running.
varnish-agent -V
It would be helpful if varnishgather output a warning to the user if there is no varnish on the server it is run on.
OpenSSL is a must, but we have other interesting package dependencies.
This is confusing: we either have releases or just use master as the stable branch and forget about tags.
getarg -s
might return values per type of argument (malloc, transient, mse, etc.). Eg., getarg -s mse
, returns MSE full path. This option is required for the PR (#49).
It would be nice to include resource usage information from sysstat (using sar), if it is enabled.
Currently the .tar.gz files created by varnishgather has a varnishgather/ directory with all the content.
If you uncompress multiple archives in the same directory they will overwrite each other, creating confusion.
Suggestion: let the subdirectory be varnishgather-HOSTNAME/ instead.
Suggest adding the following tests to include which packages are installed on the system:
rpm -qa | grep varnish
dpkg -l | grep varnish
This doesn't work in V4 so we need to update the script to workaround it, perhaps by using a timeout.
problematic line:
include "foo.vcl" // my foo vcl
//
is interpreted as a path, chaos ensues
Gather json output via:
varnishstat -j
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.