Git Product home page Git Product logo

vangav / vos_vangav_analytics_writer Goto Github PK

View Code? Open in Web Editor NEW
3.0 2.0 1.0 29.84 MB

vangav open source - analytics writer; generated using vangav backend:

Home Page: https://github.com/vangav/vos_backend

License: MIT License

Shell 2.15% Java 97.16% HTML 0.53% Scala 0.16%
backend generator java service-oriented-architecture scalable performant cassandra client algorithms dispatcher worker-service exceptions geoservices metrics geometry play-framework networking thread-pool tutorial security

vos_vangav_analytics_writer's Introduction

why? vangav analytics is a vangav backend template covering: service oriented architecture and multi-entry-point api; this is also the analytics service used by all vangav's products

vangav analytics writer

vangav analytics backend design

prerequisite

functionality

design philosophy

  • splitting the analytics backend into writer and reader services is done because writer/reader loads can vary significantly (i.e.: writer can take the load of millions of users using analyzed services while the reader can be used by few system admins); so it's easier to handle these loads by deploying a lot more writer than reader services
  • actions are defined in a flexible/generic json file (as explained later in this section) to make it easily extensible in handling analysis as backend services change/grow
  • both of the writer and reader services have actions-config-reload feature to allow modifying actions without having to re-start the services

vangav analytics backend design philosophy

  • handles writing analytics
  • handles reading actions in various ways (e.g.: by year, by day, by category, ...)

actions structure

  • actions.json in both services define how analytics are structured

  • here's a simple example

  {
    "categories": [
      {
        "category_name": "page_views",
        "category_name_short": "PV"
      }
    ],
    "action_classes": [
      {
        "class_name": "backend_page_views",
        "class_prefix": "BPV",
        "class_wide_categories": [
          "PV"
        ],
        "action_ids": [
          {
            "action_id": "quick_start",
            "action_categories": []
          }
        ]
      }
    ]
  }
element explanation
categories defines categories (e.g.: page views, button clicks, ...)
category_name a category name is used mainly for clarity
category_name_short is the one used when querying for categories
action_classes defines classes (e.g.: website, ios app, android app, ...)
class_name a class name is used mainly for clarity
class_prefix is the one used when querying for classes
class_wide_categories categories applied for an action class's action-ids
action_ids defines a single action (e.g.: home-page, download-button, camera-flip, ...)
action_id unique action identifier within its class
action_categories in addition to the class-wide-categories, here one can add extra categories specific to this action

actions structure

overview

  • this service is based on vangav backend's vangav analytics template
  • this service has the 90+% of the vangav backend's generated code + the 10-% of the logic code needed to complete the service

try this service

  1. for first timers - follow the steps in the system requirements tutorial
  2. for first timers - follow the steps in the workspace initialization tutorial
  3. download vos_vangav_analytics_writer.zip and vos_vangav_analytics_reader.zip projects (from the green clone or download button) inside the workspace directory created previously (my_services) and unzip them
  4. rename unzipped directories, remove the -master from their names
  5. in the terminal cd to vos_vangav_analytics_writer/cassandra/cql/
  6. execute ./_start_cassandra.sh to start cassandra
  7. cd to vos_vangav_analytics_writer/cassandra/cql/drop_and_create/
  8. execute the command ./_execute_cql.sh v_analytics_dev.cql to initialize the services' database tables
  9. cd to vos_vangav_analytics_writer and execute ./_run.sh to start the analytics writer service on port 9000
  10. cd to vos_vangav_analytics_reader and execute ./_run.sh 7000 to start the analytics reader service on port 7000
  11. from your prefered client (we recommned postman) start trying the service; refer to the features and service references sections for reference
  • at the end to stop the services: press control + d in the terminal session where each service was started in (9 and 10)
  • to stop cassandra: execute ps auwx | grep cassandra to get cassandra's (pid) then kill -9 (pid) to stop cassandra

eclipse

  • follow the following steps to import the downloaded backend service in eclipse
  1. cd to the service's directory and execute the ./_eclipsify.sh script
  2. file > import > general > existing projects into workspace > next > set "select root directory" to my_services > under projects make sure that vos_calculate_sum is selected > finish
  3. double check the java version used for compiling the project: right click the project > properties > java compiler > enable project specific settings > compiler compliance level > 1.7 or 1.8

covered topics

  • generate multiple services (writer + reader) to work together in a service oriented architecture
  • generic service design (handles any type of analytics)

features

controller feature
record action handles incrementing analytics counters for the specified action
reload properties handles reloading properties files as well as actions.json
controller(s) feature
get total action counters, get annual action counters, get monthly action counters, get daily action counters, get category actions and get current date handle getting analytics data by category, action and/or time
reload properties handles reloading properties files as well as actions.json

service references

reference explanation
routes api routes
setup data contains actions.json used to define the analytics structure
controllers.json api request/response's elements
v_analytics.keyspace v_analytics is the keyspace used for storing analytics
Global.java actions structure initialization
actions contains the json representation of actions.json as well as its loader
controllers api implementation
v_analytics v_analytics cassandra's keyspace client
reference explanation
routes api routes
setup data contains actions.json used to define the analytics structure
controllers.json api request/response's elements
v_analytics.keyspace v_analytics is the keyspace used for storing analytics
Global.java actions structure initialization
actions contains the json representation of actions.json as well as its loader
controllers api implementation
v_analytics v_analytics cassandra's keyspace client

change log

  • this section lists the 10-% code added after vangav backend generated 90+% of the code
file/dir change
setup data added actions.json used to define the analytics structure
Global.java added actions structure initialization
actions added the json representation of actions.json as well as its loader
controllers added the implementation of request processing logic under controller_name/HandlerControllerName.java classes and nested response json structures under controller_name/response_json packages
file/dir change
setup data added actions.json used to define the analytics structure
Global.java added actions structure initialization
actions added the json representation of actions.json as well as its loader
controllers added the implementation of request processing logic under controller_name/HandlerControllerName.java classes and nested response json structures under controller_name/response_json packages

error codes

  • following are the error codes of whatsapp services
class code : sub_code explanation
ActionsManager
400 : 1 invalid actions' category
400 : 2 actions-config must have at least one actions-class
400 : 3 invalid actions class
400 : 4 action class wide category doesn't exist
400 : 5 an action class must have at least one action id
400 : 6 invalid action id
400 : 7 invalid action category prefix
RequestRecordAction
501 : 1 invalid class prefix and/or action id
class code : sub_code explanation
ActionsManager
400 : 1 invalid actions' category
400 : 2 actions-config must have at least one actions-class
400 : 3 invalid actions class
400 : 4 action class wide category doesn't exist
400 : 5 an action class must have at least one action id
400 : 6 invalid action id
400 : 7 invalid action category prefix
RequestGetAnnualActionCounters
501 : 1 invalid actions class prefix
501 : 2 invalid action id
501 : 3 to-year can't be smaller than from-year
501 : 4 can't query more than 100 years per request
RequestGetDailyActionCounters
502 : 1 invalid actions class prefix
502 : 2 invalid action id
502 : 3 can't query more than 365 days
RequestGetMonthlyActionCounters
503 : 1 invalid actions class prefix
503 : 2 invalid action id
503 : 3 can't query more than 120 months
RequestGetTotalActionCounters
504 : 1 actions class prefix
504 : 2 invalid action id
RequestGetCategoryActions
507 : 1 category-name-short doesn't exists

share

facebook share twitter share pinterest share google plus share linkedin share

free consulting

vangav's consultant

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.