gramener / gramex Goto Github PK

• Configuration based
• Allow all in-cell charts and conditional formatting
• Allow filters, table formatting, etc
• Allow multiple sheets, sheet naming, sheet ordering, sheet colouring, etc
• Allow static content. This may be as headers and footers, or as separate sheets
• Allow hyperlinks (internal and external)
• Allow page formatting options
• Allow sheet and workbook protection options

Do not recreate what's in underlying libraries. Just return a manipulable object. Provide an easy mechanism to render it

• Source: https://code.gramener.com/cto/gramex/issues/95
• Author: s.anand
• Created: Sun 09 Jul 2017 05:02

The technical architecture template is a Markdown document that explains the technical solution for the project.

• It is intended to be a live document -- changing as the requirements flow.

• It is a testable document -- i.e. can be translated into test cases directly. Ideally, this should be automatable

• It is a mandatory document -- not filling it should result in test errors

• Source: https://code.gramener.com/cto/gramex/issues/379

• Author: s.anand

• Created: Thu 05 Apr 2018 16:49

FormHandler should allow only specific users to perform specific database actions. For example:

• Some users can only read from a table
• Some users can only read certain columns, or edit certain columns

This requires identifying the users, and identifying their permissions. FormHandler (and gramex.data.filter) should accept an auth object that has 3 keys:

• membership: same as in auth roles. Defines WHO this rule is for. This is a list of attr: values that match the user object. For example:
  • email: '*@gramener.com' -- matches any user where handler.current_user['email'] matches *@gramener.com
  • Multiple values are treated as OR conditions. email: [x@..., y@...] matches both x@... OR y@...
  • Multiple keys are treated as AND conditions. [domain: gramener.com, user: s.anand] means that handler.current_user['user'] MUST equal s.anand AND handler.current_user['domain'] MUST equal gramener.com`
• conditition: same as in auth roles. Defines WHO this rule is for. Typically used INSTEAD of membership if we want advanced rules / custom functions.
• permissions: optional list of rule objects. Defaults to all access. A rule object has:
  • type: allow or deny
  • url: DB URL to allow/deny access
  • sheetname or table or anything else gramex.data.filter accepts
  • column: columns to allow/deny access
  • access: type of access to allow or deny
  • values can be a string or list of strings that support fnmatch style wildcards

condition: ...
membership:
    - col: val
    - col: [val, val]
permission:
    - {type: allow, url: data.db, access: [all = create, read, update, delete]}
    - {type: deny,  url: data.db, table: t1, access: [create]}
    - {type: deny,  url: data.db, table: t2, column: c1, access: [write = update, delete, create]}

To be implemented as gramex.data.access(auth, ...) that returns True / False

• Source: https://code.gramener.com/cto/gramex/issues/610
• Author: s.anand
• Created: Tue 29 Jan 2019 05:31

Example: gramex.data.filter('adl://data-lake-store-name.azuredatalakestore.net/dir/file')

Reference: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_cloud-data-access/content/adls-uri.html

• Source: https://code.gramener.com/cto/gramex/issues/616
• Author: s.anand
• Created: Sun 03 Feb 2019 03:05

Requested by @sundeep.mallu

This would be an added optional parameter for any auth. Perhaps best implemented as a login action.

Recaptcha docs

• Source: https://code.gramener.com/cto/gramex/issues/606
• Author: s.anand
• Created: Mon 28 Jan 2019 05:31

Summary

Adding standard .gitlab/issue_templates/ allows projects to use a standard set of Gitlab issue templates

Benefits

• QA team will find it easier to raise bugs using a consistent format
• Clients, DCs and others will report errors better by filling in all required fields

Usage

The new issue page will have a dropdown listing all templates. Users can choose one of these.

• Source: https://code.gramener.com/cto/gramex/issues/295
• Author: s.anand
• Created: Thu 22 Feb 2018 05:40

Star team requests having an ability to search by user/ pages in LogViewer UI.

• Source: https://code.gramener.com/cto/gramex/issues/350
• Author: sundeep.mallu
• Created: Mon 12 Mar 2018 09:51

Summary

Enhance Gramex big data processing capability by integrating with Spark or Arrow

Benefits

This allows for Gramex to position as a real-time analytics engine platform.

• Source: https://code.gramener.com/cto/gramex/issues/363
• Author: sundeep.mallu
• Created: Wed 14 Mar 2018 17:57

• Source: https://code.gramener.com/cto/gramex/issues/603
• Author: bhanu.k
• Created: Thu 24 Jan 2019 05:30

Summary

Clustering component available as part of Gramex.

Benefits

Existing DataApps can use transformed data to run Clustering algorithm on them. This will enhance the value proposition for Gramex Platform offering.

Usage

Any usecase where segmentation of data into Homogeneous groups is required.

Features like settings page (configure data connection, selection of data elements, selection of ML algorithm to apply) should be part of Gramex and not tied to Clustering tool alone.

• Source: https://code.gramener.com/cto/gramex/issues/357
• Author: sundeep.mallu
• Created: Wed 14 Mar 2018 17:08

Admin2 module doesnot support prepare

The requirement for "No duplicate entry for Username. Unique username" is to ensure that a user has a unique mobile number AND a unique email ID.

admin2

• Source: https://code.gramener.com/cto/gramex/issues/611
• Author: sindhura.ch
• Created: Tue 29 Jan 2019 12:49

Summary

SmartAlert feature should allow individual end users to configure the frequency of emails they opt for.

Benefits

End users can control the volume of alerts they get and when.

Usage

Note from @pratap.vardhan

• Setup sqlite data source to store user preferences
  • user.id | url | config
• Setup functionhandler to call update_schedule
• More like

def update_schedule(handler):
    # key with user.id and page
    key = row[user.id] + row[ur]
    gramex.service.schedule[key] = Task(... email_service ...)
    update_entry_into_db(row)

• Use formhandler to read user's preferences

• Integrate subscription edits from existing user page with endpoints.

• todo: Use formhandler instead of functionhandler to read/edit user preferences with post function to trigger Scheduler API.
  (Currently, formhandler's modify is restricted to reads, we need to extend to edits, to replace the use of functionhandler)

Examples

Sample UI from tableau - https://cdnl.tblsft.com/sites/default/files/blog/dda_opener.gif

• Source: https://code.gramener.com/cto/gramex/issues/406
• Author: sundeep.mallu
• Created: Fri 04 May 2018 11:38

• Source: https://code.gramener.com/cto/gramex/issues/489
• Author: s.anand
• Created: Wed 29 Aug 2018 01:29

• Make sure you setup fresh environment

• Install gramex

• Run nosetests

• Run guide

• Source: https://code.gramener.com/cto/gramex/issues/558

• Author: pratap.vardhan

• Created: Tue 06 Nov 2018 11:57

Summary

Allow LogViewer to render data from other log files

Benefits

Projects can include other logs as part of the same log viewer, such as:

• mobile app download data
• user login / logout data
• error log data
• Windows services data
• alerts log data
• etc

Usage

Create a configuration that accepts

• a CSV glob
• a list of column headers (and an option to define whether the CSV has a header or not)
• with a column identified as the datetime timestamp
• with a set of columns that map to global filters, if applicable (e.g. URL, User, HTTP status, maybe others)
• a series of panels, where each panel has:
  • a set of local filters (e.g. alerts my have alert service, downloads may have device)
  • groupby keys
  • aggregation methods
  • a chart representation with chart format

Given this, Log Viewer will

• pre-aggregate the data source(s) into daily, weekly, monthly, etc

• render the charts based on the configuration

• Source: https://code.gramener.com/cto/gramex/issues/343

• Author: s.anand

• Created: Sun 04 Mar 2018 09:53

Excel is used to create models (data, pivot, charts, controls, what-if scenarios)

We provide an interface to modify model-copy via web apis.

# SET VALUES
model = gramex.excel.open(filename)
model.Sheet1['A3'] = 20    # cell address
model.Sheet1['roi'] = 0.5  # named cell

# model['sheet name']['roi'] = 0.5 
print model.Sheet1['result']  # named range as dataframe 
model.Sheet1['result']  = 1.0

# READ ...
print model.Sheet1['chart1']  # named chart as base64 bitmap READONLY

Driven by com interface.

This most common features required are:

• Get / Set a single cell value
• Get chart content
• Set chart feature

The following are better done within the Excel interface:

• Get / set multiple values based on input. (Use Excel to change other cells based on input)

• Changing fonts, styles, etc. (Use Excel)

• Adding formulas into cells. (Use Excel)

• Source: https://code.gramener.com/cto/gramex/issues/460

• Author: pratap.vardhan

• Created: Thu 05 Jul 2018 08:00

When, new handler spec is added to gramex.yaml or gramex.init(kwargs), gramex reloads all the configurations, which takes time.

We should update config cruds (update, add, delete) atomically. This helps IDE to be performant.

• Source: https://code.gramener.com/cto/gramex/issues/582
• Author: pratap.vardhan
• Created: Fri 04 Jan 2019 04:21

• Users can add comments

• Users can edit or delete their own comments

• Visibility level for comments can be set by developer to Public, Logged-in or role-restricted

• Multiple comment sections allowed per page

• Integrated with Gramex auth

• Flexible UI

  • Embed at the bottom of a page
  • Pop-up against specific components
  • Embed as a sidebar / drawer
  • Customizable CSS
  • Customizable display template

• Ignore optional features

  • Voting / Likes
  • Threaded comments

The main reasons not to use a third-party embedded commenting system or discussion forum are (a) ease of integrating authentication, and (b) being able to install on-prem

• Discourse? SSO is painful. It can't be on-prem.

• Facebook? Only Facebook auth

• Source: https://code.gramener.com/cto/gramex/issues/125

• Author: s.anand

• Created: Fri 18 Aug 2017 05:52

New Logviewer config has a capturehandler at port:9900 this can conflict with someone running their own capturehandler at the same port. Also, if an app is already using capturehandler, should logviewer take that endpoint instead of creating its own?

• Source: https://code.gramener.com/cto/gramex/issues/612
• Author: karmanya.aggarwal
• Created: Wed 30 Jan 2019 06:39

Show a FormHandler table that lists all objects in the gramex cache with:

• key

• type

• size

• an action to delete

• Source: https://code.gramener.com/cto/gramex/issues/450

• Author: s.anand

• Created: Thu 28 Jun 2018 05:10

This issue tracks the current status and next steps on admin module changes.

@sindhura.ch to ask @dipika.verma

• Redirecting to the dashboard features -- Is this the same as "Link to switch from ..." below?

Sindhura to migrate to admin2/ and reproduce the problems below. They may vanish:

• Add or delete users option was not working
• Action option should be aligned properly
• Admin UI changes (facial changes) through user management component
• Assign roles based on lookup tables using auth lookup attributes and manage them using FormHandler table (District, Page, module etc.)
• Link to switch from dashboard to admin and admin to dashboard (recently fixed via #576)

In our next discussion, Anand to guide on how to implementt :

• No duplicate entry for Username. Unique username

• Allow users to register themselves with custom fields with approval

• Add Approval action item in the Action column

• User profile page that users can edit themselves

• Source: https://code.gramener.com/cto/gramex/issues/577

• Author: s.anand

• Created: Tue 18 Dec 2018 06:29

Create gramex.data.table() that creates or alters table column structures

def columns(url, args, meta={}, engine=None, table=None, ext=None, **kwargs):
    '''
    Allows adding / changing / deleting columns in a table in a DataFrame, file
    or database. This is URL friendle.

    This URL structure::

        ?name=col1&type=text&default=&primary=Y&op=add
        &name=col2&type=int&default=0
        &name=col3&op=del

    ...is the same as calling::

        columns(url, table, args={
          'name':     ['col1', 'col2',  'col3'],
          'type':     ['text', 'int',   ''    ]
          'default':  ['',     '0',     ''    ],
          'primary':  ['Y',    '',      ''    ],
          'op':       ['add', 'add',    'del' ],
        })

    The URL query parameters (args) have the following structure:
      - name: column name
      - type: column type. Can be a python or SQLAlchemy at
      - default: blank indicates NULL, any other value is converted to the type
      - primary: blank indicates False, any other value is true
      - op: 'add' (default) or 'del'

    Parameters:

    :arg source url: Pandas DataFrame, sqlalchemy URL, directory or file name,
        ``.format``-ed using ``args``.
    :arg dict args: Column structure query parameters. Pass handler.args or parse_qs results
    :arg dict meta: this dict is updated with metadata during the course of filtering
    :arg string table: table name (if url is an SQLAlchemy URL), ``.format``-ed
        using ``args``.
    :arg string ext: file extension (if url is a file). Defaults to url extension
    :arg dict kwargs: Additional parameters are passed to
        :py:func:`gramex.cache.open` or ``sqlalchemy.create_engine``

    :return: resulting table structure
    '''

• Source: https://code.gramener.com/cto/gramex/issues/493
• Author: s.anand
• Created: Tue 04 Sep 2018 02:48

Provide an ability to show a pop-up periodically to Visual Solution users.

The goal is to capture instant feedback from users by using a simple scale (attractive design) and track it as part of User activity data.

The popup allow for the below configuration

• Customized text content
• Customized response capture - numeric scale or text based scale
• Time interval for showing the popup
• Traffic feedback at userid level/ either via authentication username etc.
• Ability to define the location of popup on the screen
• Allow remind me later or Dont ask buttons
• Option to not ask a user who has already filled in a survey. (But allow multiple surveys - so the same user may respond to different surveys)

DEPENDENCIES - Design team to give provide high fidelity UI for the pop.
28/12 - Hemanth has identified few UI proposals/ questions. Will be sharing them via email.

Suggested questions from Hemanth:

• What is the goal of your today's visit to Dashboard? (Open ended)

• Were you able to complete the purpose of your visit today to Dashboard? (5 point scale)

• What do you like the most/the least about Dashboard>? (Open ended)

• Overall how well does Dashboard meet your needs? (5 point scale)

• How easy was it to find the information you were looking for on Dashboard (5 point scale)

• How easy was it to understand the information on Dashboard? (5 point scale)

• Do you have any feedback on how can we improve Dashboard? (Open ended)

• Source: https://code.gramener.com/cto/gramex/issues/218

• Author: sundeep.mallu

• Created: Fri 08 Dec 2017 10:33

Summary

Create a CMS component in gramex, that will allow users to maintain the homepage(s) content in a google sheet or other soruces.

Benefits

Business Users can modify the content of a Visual story board, without need for a code change/ deployment.

Usage

gramener.com/demo page today is driven off content in a google sheet. We can provide similar CMS capability as part of customer offering.

• Source: https://code.gramener.com/cto/gramex/issues/359
• Author: sundeep.mallu
• Created: Wed 14 Mar 2018 17:50

Summary

For example, we want something like this:

pattern: /(special|img|css)/(.*)
handler: FileHandler
kwargs:
  map:
    /special/(.*): assets/special/$1.html
    /(.*)/(.*): $1/$2.html 
  transform:
    ...

We also want to allow fallback paths. For example: some projects use favicon.ico. Pick it from the project directory. If that doesn't exist, pick it from $GRAMEXPATH.

Benefits

This will help developers create friendly URLs for static files and not have to create multiple FileHandlers.

• Source: https://code.gramener.com/cto/gramex/issues/568
• Author: s.anand
• Created: Wed 28 Nov 2018 12:04

Summary

Alerts should be able to send SMS notifications, apart from emails

Benefits

• Developers get a flexible way
• Users can be notified either on SMS or on email

Usage

The same alert should be able to send SMS and email notifications (based on the same conditions, data, etc.) This can be enabled via:

kwargs:
  sms_service: ...
  sms_to: ...
  sms_subject: ...    # Which defaults to subject
  sms_sender: ...

This needs to be logged as well. Plan a consistent logging mechanism.

• Source: https://code.gramener.com/cto/gramex/issues/328
• Author: s.anand
• Created: Sun 25 Feb 2018 03:01

This is a master issue for current log viewer issues.

Issues

• Move logviewer into admin module
• #464: Show slowest pages by user, etc
• #389 & #350: Build developer interface to log file analysis
• Allow user-level tracking of dashboard usage
• #414: Logviewer startup time is slow
• Needs an info icon with help for every KPI and chart
• Allow custom user interface via logviewer components
• Change the calendar component

Note on the calendar component: pick a from date, pick a to date, continue hovering. This changes the from date. Suggested behavior: don't change the from/to date until clicked.

Data & Code

The log files are at gramex.conf.variables['GRAMEXDATA'] under logs/. Files are:

• requests.csv*: each request & response made to Gramex
• user.csv*: user login and logout information
• alert.csv*: logs alerts data (optional)
• access.csv*: old Gramex logs. Ignore this
• gramex.log*: new Gramex logs. Ignore this

Code is at gramex/apps/logviewer/. @pratap.vardhan is the author.

Needs

We want to create admin/logviewer as a series of individual components that addresses these needs:

• What is the traffic trend:
  • metrics: page views, sessions
  • time periods: daily, weekly, monthly
• What is the site responsiveness trend:
  • metrics: response time
  • time periods: daily, weekly, monthly
• Which are the top items:
  • metrics: page views, sessions
  • groups: page, user, status, ip
• Which are the slow pages:
  • metrics: response time
  • groups: user, page
• Who used which dashboard the most:
  • metrics: page views, sessions, session duration
  • groups: user, page
• Filters
  • time period: current component
  • page: only HTML pages. Prioritize dashboard pages
  • user: grouped by domain, multi-select by domain
  • HTTP status: current component

Principles

• All information is aggregated at a daily level or slower

Next steps

Phase 1: Migrate logviewer backend to admin/log/

• Speak to @dipika.verma to understand the pending issues above
• Speak to @pratap.vardhan to understand the code base
• Create a separate branch called logviewer-admin to migrate logviewer as-is into admin module
• Move the existing logviewer.py and gramex.yaml schedule into admin page and test that the back-end FormHandler endpoints work
• Review with @s.anand

Phase 2: Create simple front-end components in admin/log/

• Create and document the FormHandler tables / charts required to build a custom log viewer interface
• Create a simple log viewer interface that addresses as many needs as possible
• Review with @s.anand and project teams

Phase 3: Integrate all components required into admin/log/ page

• Source: https://code.gramener.com/cto/gramex/issues/585
• Author: s.anand
• Created: Thu 10 Jan 2019 05:22

• Create the repo
• Give access to CTO team
• Add members
• Move master and dev branches to Github
• Protect master and dev
• Set up labels and milestones (@mankoven)
• Create a CSV that has all issues labeled "github" exported. Preserve author, title, date, description, label, milestone. No conversation required
• Augment with cross-linkages to the original issue
• Upload 1 row via API and test
• Upload the rest and test
• Archive the repo, preventing new commits and issues
• Remove access to Gitlab for all except @sanand0
• Move all guide & doc references to code.gramener.com to github.com
• Set up CI on Travis

More project uses MongoDB. This warrants integration with MongoDB and ElasticSearch.

We would want to extend FormHandler API behavior which resides in gramex.data.{filter,update,etc} to connect to MongoDB and ElasticSearch. See docstrings for usage and URL query parameters to understood API.

filtered = gramex.data.filter('connection_string_to_detect', table='table', args=handler.args)

• Extend testcases in testlib/test_data.py and tests/test_formhandler.py for the given connector. These already have bunch of testcases, tested across different connectors, Inherit and extend for new connectors.

Approaches For ElasticSearch:

• Evaluate if sqlalchemy-elasticquery can be used. We directly support sqlalchemy dialects.
• See if elasticsearch-sql has extensive support and if we can integrate with it's ODBC driver. This would be high effort.
• Validate with existing test scenarios.

• Source: https://code.gramener.com/cto/gramex/issues/165
• Author: s.anand
• Created: Sun 29 Oct 2017 03:27

Create a component that indexes structured. Back-end options are:

• PostgreSQL full text search: fast, multi-purpose
• ElasticSearch: feature-rich
  started.html)
• Whoosh: pure Python

An example of real-life usage would be at https://learn.gramener.com/search/

• Source: https://code.gramener.com/cto/gramex/issues/487
• Author: s.anand
• Created: Wed 29 Aug 2018 01:26

The URL filterhandler?_c=Country&_c=State&_c=City displays all Country, State, City ignoring the hierarchy.

But after filterhandler?_c=Country&_c=State&_c=City if we add a &_h=Country>State>City then:

• The result has all distinct Country values. State and City are empty lists
• If the user selects USA as the Country, the URL should be filterhandler?_c=Country&_c=State&_c=City&_h=Country>State>City&Country=USA.
  • This returns all distinct Country values, and all State values for Country=USA. City is an empty list
• If the user selects Utah as the State, the URL should be filterhandler?_c=Country&_c=State&_c=City&_h=Country>State>City&Country=USA&State=Utah.
  • This returns all distinct Country values, and all State values for Country=USA, and all City values for Country=USA&State=Utah

FilterHandlers may contain multiple hierarchies. For example: ?_h=Country>State>City&_h=Category>Brand has 2 heirarchies:

1. Country>State>City
2. Category>Brand

• Source: https://code.gramener.com/cto/gramex/issues/485
• Author: s.anand
• Created: Sun 19 Aug 2018 17:41

• Set up discourse on gramener.com

• Embed discourse on the guide template pages

• Display on all pages publicly

• Source: https://code.gramener.com/cto/gramex/issues/501

• Author: s.anand

• Created: Mon 17 Sep 2018 12:59

Summary

Build a Forecasting component in gramex platform

Benefits

All Visual story boards that have time series data can use this component to build forecasting components.

• Source: https://code.gramener.com/cto/gramex/issues/358
• Author: sundeep.mallu
• Created: Wed 14 Mar 2018 17:10

This feature is required to ensure the Visual Story board built on top of Gramex doesn't expose a vulnerability.

• Source: https://code.gramener.com/cto/gramex/issues/318
• Author: sundeep.mallu
• Created: Fri 23 Feb 2018 16:40

Summary

oEmbed gives the ability to embed visuals seamlessly in blogs, articles, Gramener projects.

Benefits

Maps and charts created from MapViewer UI, Vega chart interface can be easily exposed to projects without additional effort.

Immediate use: Open data platform project, MapViewer UI features export options (image, embed).

Usage

Request for a URL such as https://some-dashboard.gramener.com/oembed?visual=some_random_string_id should return a oEmbed formatted JSON object

This blog post details an example including their implementation in django framework

class OEmbed(APIView):
    def get(self, request):
        # The second param passed to the .get() method is the default value, which
        # is returned if the specified key (first param) isn't found in the dictionary.
        url = request.query_params.get('url', '')
        max_width = request.query_params.get('maxwidth', 0)
        max_height = request.query_params.get('maxheight', 0)
        resp_format = request.query_params.get('format', '')
        referrer = request.query_params.get('referrer', '')

        if resp_format and not resp_format == 'json':
            return Response(data={}, status=501)

        if max_width < 280:
            return Response(data={}, status=501)

        if max_height < 825:
            return Response(data={}, status=501)

        try:
            thread_id = utils.get_thread_id_from_url(url)
        except ObjectDoesNotExist:
            return Response(data={}, status=404)

        try:
            thread = thread_service.get_thread_from_thread_id(request, thread_id)
        except InvalidPermissionError:
            return Response(data={}, status=401)

        width = max_width if (max_width and max_width <= 700) else 700
        height = 825

        resp = thread_service.build_oembed_response(thread, width, height, referrer)
        return Response(data=resp, status=status.HTTP_200_OK)

Examples

For example, FWD:everyone implements oEmbed for a request, https://api.fwdeveryone.com/oembed?url=https://www.fwdeveryone.com/t/e8RFukWTS5Wo54fBNbZ2yQ, and responds as below:

{
   "version": "1.0",
   "type": "rich",

   "provider_name": "FWD:Everyone",
   "provider_url": "https://www.fwdeveryone.com"

   "author_name": "Alex Krupp",
   "author_url": "https://www.fwdeveryone.com/u/alex3917",

    "html": "<iframe src=\"https://oembed.fwdeveryone.com?thread-id=e8RFukWTS5Wo54fBNbZ2yQ\" width=\"700\" height=\"825\" scrolling=\"yes\" frameborder=\"0\" allowfullscreen></iframe>",
    "width": 700,
    "height": 825,

    "thumbnail_url": "https://ddc2txxlo9fx3.cloudfront.net/static/fwd_media_preview.png",
    "thumbnail_width": 280,
    "thumbnail_height": 175,

    "referrer": "",
    "cache_age": 3600,        
}

• Source: https://code.gramener.com/cto/gramex/issues/578
• Author: bhanu.k
• Created: Fri 21 Dec 2018 09:44

Migrate the Gramex FAQ's Google Doc to /guide for easier access.

• https://docs.google.com/document/d/1ogprC1Hl7Q-fL2kxPuAWm8cYGcQ8NZvHLLgIblNzh0Q/edit

• Source: https://code.gramener.com/cto/gramex/issues/590
• Author: sundeep.mallu
• Created: Wed 16 Jan 2019 04:45

Summary

Allow for SQL query editor in IDE/ Query page

Benefits

Allowing for SQL query will expand the no.of users who can go past Query page. Current capability only allows for Python queries. This limits the no.of users who can benefit from the flexibility that IDE allows for.

Usage

Include Pandas SQL for allowing for SQL querying.

• Source: https://code.gramener.com/cto/gramex/issues/609
• Author: sundeep.mallu
• Created: Tue 29 Jan 2019 04:52

While trying to uninstall Gramex from Git Bash, I get stuck at c:\users\gramener\anaconda\lib\site-packages\gramex\transforms.py . It doesn't progress after that.

• Source: https://code.gramener.com/cto/gramex/issues/587
• Author: richie.lionell
• Created: Fri 11 Jan 2019 05:46

• ?_q=text filters rows that match "text" in any of the chosen query columns
• Query columns can be chosen via ?_qc. For example, ?_qc=firstname&_qc=lastname&_q=name searches for "name" in the "firstname" and "lastname" columns.
• In SQLAlchemy core, this would map to columns['firstname'].contains('name') or columns['lastname'].contains('name'). See contains
• _qc defaults to all string columns in the table
• If _qc contains non-string columns, these are added to metadata as meta['ignored'] as ('_qc', col), ...

Implementation notes in gramex/data.py:

• _pop_controls() must pop out _q and _qc as controls
• In _filter_db(), under the # Apply controls, add a if '_q' in controls: section
• In that section, get all text columns.
  • If _qc is in controls, pick only those columns.
  • If any _qc columns are not in controls, add them to meta['ignored'].append(('_qc', qc_ignored_cols))
• Using SQLAlchemy contains, add columns[col].contains(query) for each column, using an OR join.

Origin:

• Source: https://code.gramener.com/cto/gramex/issues/142
• Author: s.anand
• Created: Mon 25 Sep 2017 04:21

Integration with Amazon SNS is a pre-requisite.

• Source: https://code.gramener.com/cto/gramex/issues/320
• Author: sundeep.mallu
• Created: Fri 23 Feb 2018 16:57

Currently we support Info, Shell, Users as options in Admin2 panel.

Is it a good idea to expose session details there as well?

Session details -- how long can users log in to the application (session expiry), purged session duration. These details display the same (User session expiry -- 1 month) instead of user level session details (Bhanu's session will expire on December 11th 2pm since he logged on November 11th 2pm.)

This helps in debugging session expiry related issues.

• Source: https://code.gramener.com/cto/gramex/issues/566
• Author: bhanu.k
• Created: Fri 16 Nov 2018 02:33

FIRST find out why this is important. Whom it will help. Why existing products don't work. Re-use, don't build.

This is an app that should be bundled with Gramex. It allows users to point to directories or databases. It scans all files / tables under these and presents a searchable list of all datasets, and lets them explore and edit these using FormHandler.

We already have a version of this in https://code.gramener.com/soumya.ranjan/databrowse

Features:

• Better UI of FormHandler table API
• A standard API to explore any dataset
• Point Gramex to
  • your data warehouse
  • Hadoop / Spark / cloud environments that have these
  • a folder of Excel sheets
  • a Github repo of data
  • IMPORTANT: common data sources (SAP, Google Analytics, SalesForce, RPA logs, etc)
• Authorized users can explore the data with a simple UI
• Provide dataset profiling like with Pandas profiling or Facet overview
• Charting like Falcon

Examples?

• Redash

• Source: https://code.gramener.com/cto/gramex/issues/188

• Author: s.anand

• Created: Mon 13 Nov 2017 06:36

Summary

Build a Text Classification handler in Gramex as a component. This component can be embedded into a DataApp to perform text classification operation.

Benefits

Text Classification capability is one of the most commonly used ML feature across businesses. By having this model built as component in Gramex, we can offer value propositions to the Visual Story boards that we build for customers.

Usage

Potential usage in engagements like PMOsocial where each tweet is classified as positive/ negative/ neutral.

Breakdown of the feature into granular scope of work

Backend - Handler - Accept post request, take meta data (json, key=sentence, key=classiciation)
- get request (json object. )
- Bulk classifire (classify text as an array)
- Handler should be able to invoke any model - scikit-learn, cloud API - google ML, tensorflow,

Front end - standard application

• Source: https://code.gramener.com/cto/gramex/issues/356
• Author: sundeep.mallu
• Created: Wed 14 Mar 2018 17:06

Summary

See https://developers.google.com/analytics/devguides/collection/protocol/v1/email

Benefits

• Developers can quickly introduce mail tracking

• Users can see which emails are being opened, perhaps by whwom

• Source: https://code.gramener.com/cto/gramex/issues/347

• Author: s.anand

• Created: Sat 10 Mar 2018 01:48

Feature request from Sarad G

Can we have a diagnostic page on top of gramex logs which gives info on some of the basic questions below

• Top 10 pages with high average response time
• Top 10 users with high average response time and what specific pages for these users
• Also can the response time be split ( data processing ,visual rendering ,etc)?

• Source: https://code.gramener.com/cto/gramex/issues/464
• Author: sundeep.mallu
• Created: Thu 12 Jul 2018 11:50

We want to migrate our code into Github.

pip install gramex installs the open source version of Gramex.

gramex install auth installs additional authentication plugins from Gitlab for authenticated users.

• Source: https://code.gramener.com/cto/gramex/issues/593
• Author: s.anand
• Created: Thu 17 Jan 2019 05:16

I propose Gramex 2.0 as a:

• upgrade of Gramex 1.x
• but does not support Python 2.7 (only 3.7+)
• uses modern features like native coroutines, type annotations
• migrates to Python 3.x only libraries (e.g. Tornado 6)
• and removes deprecated features (promotes good code, reduces code size)

Pandas and Numpy have already dropped support. Tornado drops Python 2 support in July 2019.

We will target March 2019 for this release.

• Source: https://code.gramener.com/cto/gramex/issues/605
• Author: s.anand
• Created: Sat 26 Jan 2019 12:10

Create a component that lets users check if links are working.

• This helps QA teams check if all links are working fine across runs -- enabling regression testing
• This helps clients monitor applications to see if the links are functioning well

This component is deployed at https://learn.gramener.com/guide/linkcheck/ and needs to be componentised

• Source: https://code.gramener.com/cto/gramex/issues/486
• Author: s.anand
• Created: Wed 29 Aug 2018 01:14

Ability to generate static file from templates driven by data

for a given row

Pick template from templates/{row.country}/index.html pass {{ row }} data to generate stories/countries/{row.country}/index.html

Add more details.

Alerts use email service, extend to use file writer.

• Source: https://code.gramener.com/cto/gramex/issues/572
• Author: pratap.vardhan
• Created: Fri 14 Dec 2018 04:23

• Report on application errors, like Rollbar

• Report on request performance statistics (highlight slow requests, etc)

• Source: https://code.gramener.com/cto/gramex/issues/114

• Author: s.anand

• Created: Sat 05 Aug 2017 02:03