Git Product home page Git Product logo

cametrics's Introduction

Name

Cametrics

Running Your Own Cametrics

  1. Download from Github (http://github.com/ibolmo/cametrics/zipball/master) or git clone git://github.com/ibolmo/cametrics.git
  2. Rename config.py.sample to config.py and edit the file appropriately
  3. Update app.yaml (in particular the application)

Types

  • none (no statistics, no type)
  • elevation (altitude)
  • zipcode
  • city
  • country
  • accuracy
  • binary (true|false)
  • temperature
  • counter
  • choice

Modify Statistics

Sometimes it would be useful to modify statistics to exclude certain statistics. The following are examples of such filters. Concept from Django Template Filters.

type|filters

Filters

  • off, no statistics
  • number|off

Ontology

All things related to models, objects, or classes and their respected statistics that are auto incremented/decremented/updated. The '*' items are not yet implemented. The '?' items are unverified for appropiateness.

Base (common for all types)

  • value
  • prev*
  • next*
  • created_on
  • statistics
  • first
  • last

Datetime

  • timestamp
  • datetime
  • statistics
    • frequency*
    • years (bucket)
    • months (bucket)
    • days (bucket)
    • hours (bucket)
    • minutes (bucket)
    • seconds (bucket)
    • weekdays (bucket)

Location

  • longitude
  • latitude
  • elevation?
  • statistics
    • area*
    • centroid*
    • speed*
    • distance*
    • displacement*
    • max.longitude
    • min.longitude
    • max.latitude
    • min.latitude
    • geotudes

Coordinate*

  • x
  • y
  • statistics
  • area
  • centroid
  • boundary
  • grids (geotudes)?

Number

  • statistics
    • min
    • max
    • mean
    • sum
    • deviation*
    • mode*
    • median*
    • units (0.1, 1, 10, ...)*

String

  • length?

Interval*

  • start (date)
  • stop (date)
  • duration (number)

Events

Web hooks can be attached for when an event (for a namespace) occurs. Supports standard HTTP methods and position conditions (pre- and post- execution of the method).

# attach(namespace, callback_uri[, method = post|get|put|delete[, condition = post|pre);
attach(namespace, callback_uri)

# detach([namespace, callback_uri[, method = post|get|put|delete[, condition = post|pre]]);
detach(namespace, callback_uri, 'get', 'post')
detach()

Map

The following map/alias between different type of inputs

  • timestamp -> date
  • datetime -> date
  • int -> number
  • integer -> number
  • float -> number
  • long -> number
  • text -> string
  • str -> string
  • gps -> location

Examples

# measure(namespace[, value = 1[, type = number]]);  

measure('execution.start', 1240869175, 'timestamp');
measure('visitor::click');
measure(['participant', guid('[email protected]'), 'join']);

# measure.start|pause|resume|stop(namespace)

UUID

Universally Unique Identifier can be useful for namespacing to get a specific measurement for an object that is known to the campaign but is anonymous to the system. For example, I can track a user's data quality without implicating the user in the backend.

GCharts

Google Charts API is supported by passing all the normal query parameters, as described in the Google Charts API and appending .gc or .gchart to the end of your URL.

For example visit:
http://cametrics.appspot.com/agljYW1ldHJpY3NyFQsSDm15YXBwX2NhbXBhaWduGNEPDA/name/space**.gchart?**cht=p3&chs=250x100

Note

You do not need to include the chd argument since this will be automatically populated by the system.

TODO

  • Timezone for datetime calculations
  • delete Campaign, Storage, and Statistics

Thoughts

  • Olmo: Assuming: we have a good breadth of (statistical) processings for different objects, then incr/decr operations may not be necessary
  • Olmo: Would be interesting to support abritrary/other actions, though.
  • Olmo: Would be interesting to have statistics local to a grouping.
  • Sasank: Delete all data for a campaign
  • Sasank: Delete all data for a namespace
  • Sasank: Query for specific stats, including filtering for withtin a boundary

cametrics's People

Contributors

cketcham avatar ibolmo avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

Forkers

eyuen

cametrics's Issues

Allow BATCH logging

Instead of datum per POST. Would be more efficient to have data (many datum) per POST.

Error on .gc on Appspot

Exception in request:
Traceback (most recent call last):
  File "/base/data/home/apps/cametrics/1.333614277205847806/common/zip-packages/django-1.0.2.zip/django/core/handlers/base.py", line 86, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/base/data/home/apps/cametrics/1.333614277205847806/myapp/views.py", line 60, in measurements
    return renderer.get(format)(request, format, data, stats, data_path)
  File "/base/data/home/apps/cametrics/1.333614277205847806/myapp/renderer.py", line 168, in render
    dqs['chds'] = '%s,%s' % (stats[0].min - 1, stats[0].max) # careful
  File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1255, in __getattr__
    return getattr(super(Expando, self), key)
AttributeError: 'super' object has no attribute 'min'

Timesout

06-08 05:36PM 32.288 /measure/agljYW1ldHJpY3NyEAsSCENhbXBhaWduGN3aAgw//contributions/time/stats/daysth 500 4093ms 12691cpu_ms 9867api_cpu_ms 0kb cametrics.appspot.com gzip(gfe)
131.179.144.121 - - [08/Jun/2009:17:36:36 -0700] "GET /measure/agljYW1ldHJpY3NyEAsSCENhbXBhaWduGN3aAgw//contributions/time/stats/daysth HTTP/1.1" 500 84 - "gzip(gfe)" "cametrics.appspot.com"
D 06-08 05:36PM 32.313
agljYW1ldHJpY3NyEAsSCENhbXBhaWduGN3aAgw, contributions.time, stats.daysth; json
E 06-08 05:36PM 36.348
datastore timeout: operation took too long.
Traceback (most recent call last):
  File "/base/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 501, in __call__
    handler.get(*groups)
  File "/base/data/home/apps/cametrics/3.334062835599350519/measure.py", line 36, in get
    return helper.render_stats(self)
  File "/base/data/home/apps/cametrics/3.334062835599350519/myapp/renderer.py", line 49, in render_stats
    return cls.render(page, stats)
  File "/base/data/home/apps/cametrics/3.334062835599350519/myapp/renderer.py", line 88, in render
    'values': cls.get_values(page.campaign, page.namespace),
  File "/base/data/home/apps/cametrics/3.334062835599350519/myapp/renderer.py", line 75, in get_values
    data = super(JSONRenderer, cls).get_values(campaign, ns, path)
  File "/base/data/home/apps/cametrics/3.334062835599350519/myapp/renderer.py", line 61, in get_values
    return [datum for datum in query] # todo, paginator/generator
  File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1508, in next
    return self.__model_class.from_entity(self.__iterator.next())
  File "/base/python_lib/versions/1/google/appengine/api/datastore.py", line 1584, in next
    self.__buffer = self._Next(self._BUFFER_SIZE)
  File "/base/python_lib/versions/1/google/appengine/api/datastore.py", line 1571, in _Next
    raise _ToDatastoreError(err)
  File "/base/python_lib/versions/1/google/appengine/api/datastore.py", line 2020, in _ToDatastoreError
    raise errors[err.application_error](err.error_detail)
Timeout: datastore timeout: operation took too long.

Running out of Memory with endless recursion

Since the datastore exceptions are caught and are retried by splitting the set in half, you'll end up in an endless loop for datum that are invalid.

Additionally, the second set is never saved because of the endless loop.

DeadlineExceeded for namespace/values

Exception in request:
Traceback (most recent call last):
  File "/base/data/home/apps/cametrics/1.333614277205847806/common/zip-packages/django-1.0.2.zip/django/core/handlers/base.py", line 86, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/base/data/home/apps/cametrics/1.333614277205847806/myapp/views.py", line 60, in measurements
    return renderer.get(format)(request, format, data, stats, data_path)
  File "/base/data/home/apps/cametrics/1.333614277205847806/myapp/renderer.py", line 39, in render
    data_stats = map(Renderer.to_dict, stats)
  File "/base/data/home/apps/cametrics/1.333614277205847806/myapp/renderer.py", line 26, in to_dict
    return datum and datum.to_dict() or {}
  File "/base/data/home/apps/cametrics/1.333614277205847806/myapp/models.py", line 31, in to_dict
    return util.to_dict(self, attr_list, self.to_entity)
  File "/base/data/home/apps/cametrics/1.333614277205847806/util.py", line 33, in to_dict
    init_dict_func(values)
  File "/base/data/home/apps/cametrics/1.333614277205847806/myapp/models.py", line 102, in to_entity
    hist_dict[h.index].append(str(h.datum.key()))
  File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 2642, in __get__
    instance = get(reference_id)
  File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1066, in get
    entities = datastore.Get(keys)
  File "/base/python_lib/versions/1/google/appengine/api/datastore.py", line 219, in Get
    apiproxy_stub_map.MakeSyncCall('datastore_v3', 'Get', req, resp)
  File "/base/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 68, in MakeSyncCall
    apiproxy.MakeSyncCall(service, call, request, response)
  File "/base/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 240, in MakeSyncCall
    stub.MakeSyncCall(service, call, request, response)
  File "/base/python_lib/versions/1/google/appengine/runtime/apiproxy.py", line 182, in MakeSyncCall
    rpc.Wait()
  File "/base/python_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 98, in Wait
    rpc_completed = self._WaitImpl()
  File "/base/python_lib/versions/1/google/appengine/runtime/apiproxy.py", line 104, in _WaitImpl
    rpc_completed = _apphosting_runtime___python__apiproxy.Wait(self)
DeadlineExceededError

On Stress tests found: timeouts

05-25 01:02AM 14.123 /ahBjYW1ldHJpY3Mtc3RyZXNzchULEg5teWFwcF9jYW1wYWlnbhjpBww/contributions/time 500 4155ms 8025cpu_ms 8012api_cpu_ms 0kb cametrics-stress.appspot.com gzip(gfe)
76.73.53.91 - - [25/May/2009:01:02:18 -0700] "POST /ahBjYW1ldHJpY3Mtc3RyZXNzchULEg5teWFwcF9jYW1wYWlnbhjpBww/contributions/time HTTP/1.1" 500 0 - "gzip(gfe)" "cametrics-stress.appspot.com"
E 05-25 01:02AM 18.276
<class 'google.appengine.api.datastore_errors.Timeout'>: 
Traceback (most recent call last):
  File "/base/data/home/apps/cametrics-stress/1.333693860431221035/common/appenginepatch/main.py", line 26, in real_main
    util.run_wsgi_app(application)
  File "/base/python_lib/versions/1/google/appengine/ext/webapp/util.py", line 76, in run_wsgi_app
    result = application(env, _start_response)
  File "/base/data/home/apps/cametrics-stress/1.333693860431221035/common/zip-packages/django-1.0.2.zip/django/core/handlers/wsgi.py", line 239, in __call__
  File "/base/data/home/apps/cametrics-stress/1.333693860431221035/common/zip-packages/django-1.0.2.zip/django/core/handlers/base.py", line 67, in get_response
  File "/base/data/home/apps/cametrics-stress/1.333693860431221035/common/appenginepatch/ragendja/sites/dynamicsite.py", line 21, in process_request
    site = Site.all().filter('domain =', domain).get()
  File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1382, in get
    results = self.fetch(1)
  File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1426, in fetch
    raw = self._get_query().Get(limit, offset)
  File "/base/python_lib/versions/1/google/appengine/api/datastore.py", line 964, in Get
    return self._Run(limit, offset)._Next(limit)
  File "/base/python_lib/versions/1/google/appengine/api/datastore.py", line 908, in _Run
    _ToDatastoreError(err)
  File "/base/python_lib/versions/1/google/appengine/api/datastore.py", line 2020, in _ToDatastoreError
    raise errors[err.application_error](err.error_detail)

Error on DELETE campaign with many entries

Exception in request:
Traceback (most recent call last):
  File "/base/data/home/apps/cametrics/1.333614277205847806/common/zip-packages/django-1.0.2.zip/django/core/handlers/base.py", line 86, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/base/data/home/apps/cametrics/1.333614277205847806/common/zip-packages/django-1.0.2.zip/django/contrib/auth/decorators.py", line 67, in __call__
    return self.view_func(request, *args, **kwargs)
  File "/base/data/home/apps/cametrics/1.333614277205847806/myapp/views.py", line 104, in delete_campaign
    post_delete_redirect=reverse('myapp.views.list_campaigns'))
  File "/base/data/home/apps/cametrics/1.333614277205847806/common/zip-packages/django-1.0.2.zip/django/views/generic/create_update.py", line 222, in delete_object
    obj.delete()
  File "/base/data/home/apps/cametrics/1.333614277205847806/common/appenginepatch/appenginepatcher/patch.py", line 440, in delete
    signals.pre_delete.send(sender=self.__class__, instance=self)
  File "/base/data/home/apps/cametrics/1.333614277205847806/common/zip-packages/django-1.0.2.zip/django/dispatch/dispatcher.py", line 148, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/base/data/home/apps/cametrics/1.333614277205847806/common/appenginepatch/ragendja/dbutils.py", line 426, in cleanup_relations
    rels_seen, to_delete, to_put = get_cleanup_entities(instance)
  File "/base/data/home/apps/cametrics/1.333614277205847806/common/appenginepatch/ragendja/dbutils.py", line 382, in get_cleanup_entities
    raise Exception("Can't delete so many entities at once!")
Exception: Can't delete so many entities at once!

On stress test found: contention

05-25 01:02AM 19.176 /ahBjYW1ldHJpY3Mtc3RyZXNzchULEg5teWFwcF9jYW1wYWlnbhjpBww/356996016212614/contributions/type 200 2726ms 1032cpu_ms 900api_cpu_ms 2kb cametrics-stress.appspot.com gzip(gfe)
76.73.53.91 - - [25/May/2009:01:02:21 -0700] "POST /ahBjYW1ldHJpY3Mtc3RyZXNzchULEg5teWFwcF9jYW1wYWlnbhjpBww/356996016212614/contributions/type HTTP/1.1" 200 2010 - "gzip(gfe)" "cametrics-stress.appspot.com"
E 05-25 01:02AM 21.835
Exception in request:
Traceback (most recent call last):
  File "/base/data/home/apps/cametrics-stress/1.333693860431221035/common/zip-packages/django-1.0.2.zip/django/core/handlers/base.py", line 86, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/base/data/home/apps/cametrics-stress/1.333693860431221035/myapp/views.py", line 68, in measurements
    if (not datum.put()):
  File "/base/data/home/apps/cametrics-stress/1.333693860431221035/common/appenginepatch/appenginepatcher/patch.py", line 434, in put
    created=created, raw=raw)
  File "/base/data/home/apps/cametrics-stress/1.333693860431221035/common/zip-packages/django-1.0.2.zip/django/dispatch/dispatcher.py", line 148, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/base/data/home/apps/cametrics-stress/1.333693860431221035/myapp/models.py", line 106, in cb_calc_statistic
    if (stat.get(datum.type).calculate(datum) is not False):
  File "/base/data/home/apps/cametrics-stress/1.333693860431221035/myapp/stat.py", line 74, in calculate
    cls.tally(stats = datum.stats, name = 'hits', index = datum.value)
  File "/base/data/home/apps/cametrics-stress/1.333693860431221035/myapp/stat.py", line 58, in tally
    if (not hist.put()):
  File "/base/data/home/apps/cametrics-stress/1.333693860431221035/common/appenginepatch/appenginepatcher/patch.py", line 432, in put
    result = old_put(self, *args, **kwargs)
  File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 696, in put
    return datastore.Put(self._entity)
  File "/base/python_lib/versions/1/google/appengine/api/datastore.py", line 166, in Put
    raise _ToDatastoreError(err)
  File "/base/python_lib/versions/1/google/appengine/api/datastore.py", line 2020, in _ToDatastoreError
    raise errors[err.application_error](err.error_detail)
TransactionFailedError: too much contention on these datastore entities. please try again.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.