graphite-project / graphite-web Goto Github PK
View Code? Open in Web Editor NEWA highly scalable real-time graphing system
Home Page: http://graphite.readthedocs.org/
License: Apache License 2.0
A highly scalable real-time graphing system
Home Page: http://graphite.readthedocs.org/
License: Apache License 2.0
I am trying to plot the following scenario -
Metrics
Server1 - Requests Served
Server2 - Requests Served
Total (Server1 + Server2) - Requests Served
Desired Plot
Server1 and Server2 are stacked.
Total is displayed as a line.
Legend - CactiStyle for all three metrics
I use the stacked function to stack individual servers. I notice that CactiStyle legend for the second server displays incorrect stats. It shows the stats as they are on Y-Axis (which is higher because of the stack)
/render?
target=cactiStyle(alias(stacked(derivative(serv1.requests)),'server1'))
&target=cactiStyle(alias(stacked(derivative(serv2.requests)),'server2'))
&target=cactiStyle(alias(derivative(sumSeries(*.requests)),'total'))
This may be an issue with using CactiStyle with the stacked function.
Or maybe I am just taking the wrong approach for the plot.
I can email the image of the graph if required.
I have 6 servers with each a folder within stats.timers.*
also, for any given server, i have 4 folders within stats.timers.<server>.object-server.*
watch what happens:
graphitejs ❯ wget '(...)&target=stats.timers.*.object-server.GET.timing.lower&target=stats.timers.dfvimeodfs5.object-server.*.timing.lower&format=json' -O - | python -mjson.tool | grep target
"target": "stats.timers.dfvimeodfs1.object-server.GET.timing.lower"
"target": "stats.timers.dfvimeodfs2.object-server.GET.timing.lower"
"target": "stats.timers.dfvimeodfs3.object-server.GET.timing.lower"
"target": "stats.timers.dfvimeodfs4.object-server.GET.timing.lower"
"target": "stats.timers.dfvimeodfs5.object-server.GET.timing.lower"
"target": "stats.timers.dfvimeodfs6.object-server.GET.timing.lower"
"target": "stats.timers.dfvimeodfs5.object-server.GET.timing.lower"
"target": "stats.timers.dfvimeodfs5.object-server.HEAD.timing.lower"
"target": "stats.timers.dfvimeodfs5.object-server.PUT.timing.lower"
"target": "stats.timers.dfvimeodfs5.object-server.REPLICATE.timing.lower"
graphitejs ❯ wget '(...)&target=alias(stats.timers.*.object-server.GET.timing.lower,"glob1")&target=alias(stats.timers.dfvimeodfs5.object-server.*.timing.lower,"glob2")&format=json' -O - | python -mjson.tool | grep target
"target": "glob1"
"target": "glob1"
"target": "glob1"
"target": "glob1"
"target": "glob2"
"target": "glob2"
"target": "glob2"
"target": "glob2"
stats.timers.dfvimeodfs5.object-server.GET.timing.lower
is included twice (because the globs overlap)
I think it only has to be included once
Using saved dashboards with multiple graphs (e.g. 9 default sized graphs (3x3)) can be very slow to load and refresh as all 9 graph requests hit apache at once.
Would be good to be able to stagger the requests on first load, but more importantly on refresh/scheduled refresh.
While using a version of trunk pulled on 9/19/12 I have found that when I use the MYSQL database option I am unable to render events in the flot interface. Direct render urls and the normal web interface are all able to find and generate event data on the graphs.
When in flot if I proved '*' as the search in the event field it shows a red bar as if it is unable to find the events. It also fails when searching using a tag name. I can see the events listed in the /event/ interface as expected.
If you use movingAverage
with a wildcard, e.g. movingAverage(web.*.html,20)
, the first N (20) data points are plotted incorrectly.
Here is an example without movingAverage:
This is what the graph looks like using movingAverage:
As you can see, the first 20 data points are somehow averaged together.
16a6868 seems to be the culprit. If I roll back to the commit just before, this bug goes away, and the first 20 data points are null.
Would you consider reverting this (old) commit: 5b40580?
We are doing a ton of 10sec resolution data, and turning on caching brought up this issue. I'd like to not have a fork for those 4 characters.
Thanks!
I suggested this when the Great Github Move happened. I recall agreement that it was a good idea.
Either it got forgotten, is yet to be done but planned, or was decided not to be done.
This issue is for that.
right now, if you select a range where untilTime < oldestTime (the latter defined by now - retention), the /render/
request will fail and the display will break (although graphite-project/whisper#6 alleviates that)
either way it would be nice if the date range selector could visualize how far you can go back, and even prevent you from going further back (because there's no point if (none of the) targetted whisper file(s) only go(es) so far). bonus points for coloring to symbolize accuracy rates
Need to add support for this feature in the master
branch. This issue guarantees that I won't forget it. 🤘
diving into webapp/graphite/render/glyph.py reveals HSV colors are not implemented yet.
one advantage of HSV over RGB is it allows for easier/prettier transitions between red and green, see
http://stackoverflow.com/questions/340209/generate-colors-between-red-and-green-for-a-power-meter
so if anyone feels like contributing and doesn't know where to start...this would be a nice to have.
This is happening in 0.9.10
Reproduced in FF and Chrome
I'm guessing this issue is very similar to the one fixed with this commit: ff8e1a5
When there is a metric that has a "%" in it's name the graphite dashboard won't display anything after clicking on the metric name.
Metric filename example: %cpu.wsp
It seems that the formvalidation for X/Y labels is too strict.
When trying to use a (sensible) y-ax label like: slow request/sec
you get an error "input can only contain letters, numbers, underscores or periods".
While there is no reason why these labels should have any limit. (the full monkey should be available (utf8))
popup: "input can only contain letters, numbers, underscores or periods". is shown
label is updated as requested.
0.9.10
Is there a way of having graphite draw a 1 if a stat is different from 0 and a 0 otherwise?
I tried using the drawAsInfinite function but that actually draws in infinity and doesn't allow me to operations like scale or max on the result.
I am observing two issues with drawAsInfinite
. I see that the Infinite line is drawn at position 0 (horizontally along the bottom) as well as vertically in the graph at the proper location. If at all possible, it would be nice to have infinite lines only draw vertically. If one chooses to use a larger lineWidth
it causes an even more distinct horizontal line to be drawn which can obscure other information.
I believe a similar rendering issue exists for drawAsInfinite
as outlined in #31 for all types of areaModes
. On my graphs that are zoomed in fairly close, and are non stacked graphs I see a small gap in the horizontal line that is drawn. The gap is just to the right of the vertical line and shows up for around a minutes worth of time. If my first issue is resolved then this second observation becomes moot.
areaBetween() draws the two series passed to it as stacked lines with the bottom line set to invisible. Since there are actually 2 sets of stacked lines it ends up stacking 4 lines (with two of them being invisible) - the result is that the area drawn is between the outer band's lower and the inner band's lower, and between the inner band's upper and the outer band's lower. This also affects holtWintersConfidenceArea() as it uses areaBetween
The fetch call in RemoteReader of remote_storage.py fails erratically.
This appears to happen when there are multiple requests for the same set of datapoints at the same time. Examples would be dashboards with multiple graphs that contain the same metrics.
The connection is not defined in some calls back into wait_for_results(). I attempted to correct this before, but the end result is that the waiting requests end up waiting past the timeout and exceptions get raised.
Does anyone have an idea on how to fix this code to be correct in the spirit of the multi-threaded setup that it has?
-Christopher
When i try to set xFormat to:
%a %H:%M
in the composer, the uri is added
xFormat=%25a%20%25H%3A%25M
After that it is not possible to view graph again, as extJS decodeURIComponent
breaks with
ext-base-debug.js:371URIError
Upgraded to 0.9.10 today. Don't recall seeing this problem in 0.9.9.
When another metric in a graph reaches the threshold metric line (same data value or higher), the threshold line is "blocked" by the other graph line and no longer continues being drawn to the right of the blocking line.
I can provide an image if it is necessary to see what I mean.
EDIT: I misspoke a little, turns out the threshold line DOES continue to the right of the "blocking" line... but only for a little bit, and does not traverse the whole graph left to right as it should.
For this test, I Install graphite-web like so:
git clone git://github.com/graphite-project/graphite-web.git
cd graphite-web
git checkout 0.9.x
mkdir -p storage/log/webapp/
cp ./webapp/graphite/local_settings.py.example ./webapp/graphite/local_settings.py
$EDITOR ./webapp/graphite/local_settings.py to set the following two vars:
WHISPER_DIR = '/var/lib/carbon/whisper/' # point to the real production whisper directory
TIME_ZONE = 'US/Eastern' # this is the important part!
python ./webapp/graphite/manage.py syncdb
python ./webapp/graphite/manage.py runserver 0.0.0.0:8000
For example, I create a datapoint with a known value 10, at a specific timestamp:
dieter@dieter-ws ~ date +%s -d '10:00'
1342166400
dieter@dieter-ws ~ date -d @1342166400
Fri Jul 13 10:00:00 CEST 2012
dieter@dieter-ws ~ TZ=UTC date -d @1342166400
Fri Jul 13 08:00:00 UTC 2012
dieter@dieter-ws ~ TZ=US/Eastern date -d @1342166400
Fri Jul 13 04:00:00 EDT 2012
dieter@dieter-ws ~ echo 'dieter.test_tz_bug 10 1342166400' | nc -c nyvimeographitedev1 2003
## verification to be 100% sure the datapoint is recorded correctly:
[dieter@nyvimeographitedev1 ~]$ ./whisper-dump.py /var/lib/carbon/whisper/dieter/test_tz_bug.wsp | grep -C 5 1342166400
points: 1440
retention: 86400
size: 17280
Archive 0 data:
0: 1342166400, 10
1: 0, 0
2: 0, 0
3: 0, 0
4: 0, 0
5: 0, 0
now, I go to http://nyvimeographitedev1:8000, select my dieter.test_tz_bug
metric,
and select the timerange for today (fri 13 juli) from 03:30 AM until 04:30.
that looks like this: http://i.imgur.com/YXurz.png as you can see the webUI claims the used timezone is CEST but it's not correct, it is applying my TIME_ZONE setting (US/Eastern) correctly, as the datapoint I inserted is shown at 4:00AM
the graph generated by the UI is:
http://nyvimeographitedev1:8000/render/?width=586&height=308&_salt=1342198925.045&target=dieter.test_tz_bug&from=03%3A30_20120713&until=04%3A30_20120713
so I can doublecheck this, and indeed: if I add &tz=US/Eastern
to the URL, the image stays the same (meaning the backend applies my timezone setting correctly, and this is how the webUI invoked it, so the webUI should not claim it's CEST)
if I add &tz=Europe/Brussels
(same as CEST for this time of year) the image becomes: http://i.imgur.com/gvaWj.png, showing the datapoint at 10AM (which is what you would expect for CEST, unlike what the webUI originally showed)
In the dashboard area, when trying to create a graph from url any URL which has the '%' symbol won't work. Once you press ok after entering the url nothing happens.
Hello,
I wanted to request CSV data with a specific time zone and found out it does not work. If an image is requested the time zone is set properly, but that is not the case when adding format=csv
in the URL. For example, note the following URLs return the same time series when they should be 3 hours apart.
Instead, they return whatever the configured timezone is in the graphite local_settings.py file.
curl 'http://graphite/render/?target=something.value&from=-10minutes&format=csv&tz=America/Los_Angeles'
curl 'http://graphite/render/?target=something.value&from=-10minutes&format=csv&tz=America/New_York'
Thanks!
At the moment, graphite renders an empty array if you request an URL like target[]=host1.value&target[]=host2.value&format=json
. The form which django/graphite-web needs it would be target=host1.value&target=host2.value&format=json
.
It would be cool if it were possible to use both. According to http://stackoverflow.com/questions/10989342/get-the-javascript-array-in-django this should be possible and not too hard to setup.
Using carbon-relay and 3 instances of carbon-cache behind it. Sending metrics via TCP.
Problem with the twisted daemon. Increasing ulimit doesn't seem to suffice the need.
Restart temporarily fixes issue, and number of open file descriptors gets gobbled up and this error appears.
For example, I may want 1 graph "http request latency for that last 24 hrs" and another "http request latency for the last week" and even "http request latency for the last 30 days" all on the same dashboard
After upgrading from 0.9.10pre with 'pip install' I'm hitting the same bug as reported here: https://bugs.launchpad.net/graphite/+bug/1032771
@wfarr encountered a render bug today when enabling areaMode=stacked
on a chart using drawAsInfinite()
. As you can see, in normal lineMode it looks as you might expect. But with stacking enabled it renders a vertical every ~10 seconds.
target=substr(exclude(sortByMinima(resque.queue.*),"resque.queue.*fs*"),2)
target=lineWidth(alias(color(drawAsInfinite(deploy.foo.*.*),"red"),"deploys"),0.5)
from=-30minutes
title=queue size
areaAlpha=0.5
template=plain
hideLegend=true
target=substr(exclude(sortByMinima(resque.queue.*),"resque.queue.*fs*"),2)
target=lineWidth(alias(color(drawAsInfinite(deploy.foo.*.*),"red"),"deploys"),0.5)
from=-30minutes
title=queue size
areaAlpha=0.5
template=plain
hideLegend=true
areaMode=stacked
see https://github.com/graphite-project/graphite-web/blob/master/webapp/graphite/render/functions.py#L520
series.name = "scale(%s,%.1f)" % (series.name,float(factor))
I'm using scale factors such as 0.05 and 0.01 and notice that graphite shows them as 0.1 and 0.0 respectively (even though the calculation in safeMul()
does use the scale I asked for)
so:
it's not entirely clear why the code is like this (looks like it has been since the very first git commit), but I would suggest to just print each character of factor
literally
It's been about 6 months since 0.9.10 came out. I could really use that nPercentile fix, among other things.
Input can only contain letters, numbers, underscores, or periods.
Click on a graph in a dashboard, render options -> y-axis -> minimum.
local_settings.py should be locatable outside of the installed python source which could be read-only or within a virtualenv. /opt/graphite/conf would be the most obvious default. Additionally, it should be named less generically (graphite_settings.py? graphite.py? graphite_webapp.py?)
graphite.settings
looks for graphite.local_settings
explicitly, so it is impossible to configure without hacking local_settings.py
into the python path somehow.
Some parts of webapp (eg webapp/graphite/settings.py:134
) do not read the STORAGE_DIR value set in carbon.conf
or local_settings.py
. This means that moving the directory must be updated in 3 places:
I will submit a pull request for this once I've made the fix and tested locally.
Specifically, commit adf3f6c does not work with python2.4 anymore because of the import of functools.
My understanding is that the 0.9.x branch is intended to maintain Python 2.4 compatibility.
Hi,
I use graphite 0.9.10 for one month and it works great expect the function related to percentile.
I see that the regression that has been fixed in the following commit : (540fec4) a few month ago.
Can you tell me when you think that a next official release will be done?
If no release is planned yet, can you tell me if it's safe to use the master rather than the version 0.9.10 as of now.
There are 2 issues.
If I use the django admin console to create an event, it won't appear listed in the /events endpoint list. If I create it using the curl POST REST API, then it does appear.
When I create an event (using any of the 2 methods listed above) and then go to the flot interface and attempt to search for the event using the Events input text by entering the tag name, I get a 500 with the following stacktrace:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/opt/graphite/webapp/graphite/events/views.py", line 63, in get_data
return HttpResponse(json.dumps(fetch(request), cls=EventEncoder),
File "/opt/graphite/webapp/graphite/events/views.py", line 82, in fetch
models.Event.find_events(time_from, time_until, tags=tags)]
File "/opt/graphite/webapp/graphite/events/models.py", line 42, in find_events
result = list(query.order_by("when"))
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 84, in __len__
self._result_cache.extend(self._iter)
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 273, in iterator
for row in compiler.results_iter():
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 680, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 725, in execute_sql
sql, params = self.as_sql()
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 68, in as_sql
where, w_params = self.query.where.as_sql(qn=qn, connection=self.connection)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/where.py", line 92, in as_sql
sql, params = child.as_sql(qn=qn, connection=connection)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/where.py", line 95, in as_sql
sql, params = self.make_atom(child, qn, connection)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/where.py", line 142, in make_atom
lvalue, params = lvalue.process(lookup_type, params_or_value, connection)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/where.py", line 329, in process
connection=connection, prepared=True)
File "/usr/lib/python2.7/dist-packages/django/db/models/fields/subclassing.py", line 53, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/db/models/fields/subclassing.py", line 53, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/db/models/fields/subclassing.py", line 53, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 323, in get_db_prep_lookup
return [self.get_db_prep_value(value, connection=connection, prepared=prepared)]
File "/usr/lib/python2.7/dist-packages/django/db/models/fields/subclassing.py", line 53, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 727, in get_db_prep_value
return connection.ops.value_to_db_datetime(value)
File "/usr/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 233, in value_to_db_datetime
raise ValueError("MySQL backend does not support timezone-aware datetimes.")
ValueError: MySQL backend does not support timezone-aware datetimes.
I think it will be much easier if I could see the current metric value when I mouse over the relevant metric graph.
As said in title, metric with parentheses in its name wont show on graph.
Issue detected on Graphite 0.9.10 and reproduced on FF and Chrome.
I have checked whisper file and metric is stored in it, so this looks like issue on front end side.
Metric name example: time_(ms)_avg.
P.S. This looks similar to issue #33, but dont think that fix will apply for this problem since parentheses doesn't need to be encoded.
Best,
Marko
ceres is a required dependency, however its missing from the requirements.txt
Ceres is a noted requirement in check-dependencies.py however its missing from requirements.txt, stopping users from getting up and running with
$ pip install -r requirements.txt
Should the below be added to requirements.txt?
git+git://github.com/graphite-project/ceres.git#egg=ceres
Then the the removal of.
git+git://github.com/graphite-project/whisper.git#egg=whisper
self.yTop
and self.yBottom
are not set when secondYAxis
is true. This causes SVG output in render/glyph.py
to trigger an exception.
Hi,
I've been looking in webapp/graphite/metrics/views.py trying to find a way to get a listing of all metrics in 1 request. (for example in json)
this doesn't seem to be possible yet.
I'm a bit confused, it seems that there are multiple functions that do similar things, by different means.:
find_view
finds metrics by using a Store object from webapp/graphite/storage.py, whereas expand_view
does something similar, as does tree_json
which uses another mechanism, and then there's search
which I haven't figured quite out (it seems to use a textbased index. btw: interestingly on my testing machine the index file does not get created)
please advise how I should proceed.
In the dashboard area, when trying to create a graph from url any URL which has the '%' symbol won't work. Once you press ok after entering the url nothing happens.
The 0.9.10 tagged releases of graphite-web and carbon require Twisted 11.0.0 and 11.1.0, respectively. This means that one will overwrite the version of the other, depending on which you install first.
Also, both require the master version of whisper, rather than a tagged release (presumably you'd want to use 0.9.10 with 0.9.10).
How can we combine data from multiple graphite-web instances?
For ex: We have Data center's in US,EU and Asia. How can we use graphite-web to have a consildated CPU usage view across all my datacenter in one graph?
i think being able to plot bar charts (bin of value against frequency) would be very useful.
We've seen some render errors with stacked graphs when the time range hits an archive boundary and using sortByMaxima
. The results are unpredictable but completely reproducible. Here is a "correct" graph:
Here is the same graph demonstrating the bug:
The url looks like this:
/render?target=substr(sortByMaxima(foo.queue.%7Bwiki%2C%7Dbackups*)%2C2%2C3)&from=-7d&areaMode=stacked
If we shift the range one hour in either direction, the problem is not reproducible.
Hello,
In Graphlot, when x1 is null (which i still have to figure out why), it skips x2 entierly, and the resulting data is wrong.
There's an issue with the data I have and graphlot. In graphite there's no data missing like graphlot.
Here's a little patch 1bacf29, which is far from perfect but shows where the issue is.
I've also attached images showing the data in graphite, then in graphlot with and without the patch.
Regards
In graphite:
In graphlot with the patch:
In graphlot without the patch:
i have go through graphite-web / distro / redhat, only left a postinstall.
As suggest, i think we only need support community distrib fedora. don't need for redhat rhel branch.
here i have found quality spec, please review and let me optimized it.
graphite-web
http://jsteffan.fedorapeople.org/SPECS/graphite-web.spec
carbon
http://jsteffan.fedorapeople.org/SPECS/python-carbon.spec
whisper
http://jsteffan.fedorapeople.org/SPECS/python-whisper.spec
The method in /metrics/views.py
only searches for *.wsp
files. It would be nice to be able to get metrics from a Ceres-based back-end.
I will have a go at this (not sure of timeframe), this ticket is just a place-holder.
when you are on a page like nyvimeographitedev1:8000/graphlot/?(...)&from=03%3A30_20120713&until=04%3A30_20120713
the UI will show (in the lower right block) FROM: 03:30_20120713
and TO: 04:30_20120713
(which is correct), it will also request data like so:
GET http://nyvimeographitedev1:8000/graphlot/rawdata?&from=03:30_20120713&until=04:30_20120713&target=dieter.test_tz_bug
which is correct, the response it gets starts like start": 1342164660, "step": 60, "end": 1342168260
(which is also correct, because my TIME_ZONE is set to US/Eastern
), however, when plotting the data on the graph it shows 7:30 to 8:30 on the x-axis which are UTC times
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.