Git Product home page Git Product logo

datastreams's People

Contributors

evan-kim2028 avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar

datastreams's Issues

Expand Functionality of `datastreamer.py`

datastreamer currently runs as a (limited) proof of concept data flow path. There is still a lot of friction that remains between datastreamer and streamer. A refactor/enchancement of datastreamer is needed to expand functionality

Refactor `streamer.py`

streamer.py has a lot of unused comments. Needs a refactor as well as design considerations with respect to datastreams.py

Schema query does not always query every single column

https://gist.github.com/Evan-Kim2028/df57d5528c7a35fb40cc308eba3dfead

The schema query for sub.Query.liquidityPools has more than 16 available columns to query from, but only shows 16 by default. For example, if inputTokenBalances is not explicitly mentioned, then it will not appear in the query results.

Current proposed idea for this solution is to make a FieldPath + string function that binds string columns to an existing fieldpath.

Example:
FieldPath = sub.Query.liquidityPools
String = 'inputTokenBalances`

Function Output: --> sub.Query.liquidityPools.inputTokenBalances as a single FieldPath object

Streamer crashes when empty queries are retrieved

File ~/.local/lib/python3.10/site-packages/requests/models.py:971, in Response.json(self, **kwargs)
    970 try:
--> 971     return complexjson.loads(self.text, **kwargs)
    972 except JSONDecodeError as e:
    973     # Catch JSON-related errors and raise as requests.JSONDecodeError
    974     # This aliases json.JSONDecodeError and simplejson.JSONDecodeError

File ~/.local/lib/python3.10/site-packages/simplejson/__init__.py:525, in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, use_decimal, **kw)
    521 if (cls is None and encoding is None and object_hook is None and
    522         parse_int is None and parse_float is None and
    523         parse_constant is None and object_pairs_hook is None
    524         and not use_decimal and not kw):
--> 525     return _default_decoder.decode(s)
    526 if cls is None:

File ~/.local/lib/python3.10/site-packages/simplejson/decoder.py:370, in JSONDecoder.decode(self, s, _w, _PY3)
    369     s = str(s, self.encoding)
--> 370 obj, end = self.raw_decode(s)
    371 end = _w(s, end).end()

File ~/.local/lib/python3.10/site-packages/simplejson/decoder.py:400, in JSONDecoder.raw_decode(self, s, idx, _w, _PY3)
    399         idx += 3
--> 400 return self.scan_once(s, idx=_w(s, idx).end())
...
---> 89       raise PaginationError(exn.args[0], strategy)
     91   return data
     93 except SkipPagination:

PaginationError: Expecting value: line 1 column 1 (char 0)```

Make Streamer Class more composable

Problem: Streamer is becoming too monolithic to update over time. There are too many moving pieces to get a concise understanding of how all of the functions work together.

Solution: Break Streamer up into multiple classes. Separate query functionality from schema information. This will more clearly delineate functional dependencies that aren't so clear in Streamer right now.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.