dtaivpp / community-pulse Goto Github PK
View Code? Open in Web Editor NEWThis is a system that will allow any person to start monitoring their communities activity.
License: Other
This is a system that will allow any person to start monitoring their communities activity.
License: Other
If you use capital letter in a job name this will silently cause the index to fail to create. Two parts are needed to fix this:
Get marker currently returns the last scraped tweet ID. The issue with this is when that tweet is older than seven days old passing it to twitter as the since_id causes the twitter automation to fail.
Therefor the get_marker function from twitter.py should only return an ID if the tweets created at is less than 7 days old.
At the moment every job will execute in series which could cause issues as one job failing would take down the pipeline and most jobs (of different types) could be run in parallel.
community-pulse should have a way to ensure that jobs of a similar type (eg. twitter) are executed in series so that they do not exhaust their API limits but jobs of dissimilar types are executed in parallel such that they do not block each other.
At the moment get_data from twitter.py is tightly coupled making it difficult to troubleshoot. This should be separated into two different functions. One that retrieves the data from twitter and another that enriches it for storage in OpenSearch.
There should be a CLI that allows users to pass in configuration parameters for running. These parameters include config file location or which specific jobs should be run.
Examples:
pulse --config=/etc/pulse/ --job=twitter
pulse --job twitter discourse
Multi Job Support is as follows. Users should be able to name their jobs whatever they wish in the config file. These names should be carried over into the index names. There should be a default dashboard deployed by type which is then filterable by job name.
Sub tasks:
At the moment community pulse only supports twitter but the goal is to eventually support all the places communities meet and discuss information.
Some ideas are:
Find New Articles:
Async Discussions:
Sync Discussions:
Social Media:
At the moment, the query to return the most recent tweet ID are only returning the latest tweet from the previous day. Need to debug to determine the cause.
There is currently a bug where duplicate tweets are being introduced.
Users should be presented default dashboards for all the data in a particular job type. These dashboards should be generically applicable and versioned so that they can be updated at a later date.
SubTasks:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.