Git Product home page Git Product logo

data-feed's Introduction

Data feed

A company is building a system for managing data feeds inputs from external sources.
Data feed is a collection of data that has keys and values ordered according to specific structure.
Each source has a different way to login and extract the relevant data feed.
User can filter data using display options sidebar or update field value with new value.

Design

The user should be logged into the system to view the data feed.

Image

It means that the user can see only one feed at a time.

User's table:
user_id (Primary key)
username
pwd - password (in order to enable login functionality)
email
created_on - creation time of the user
last_login (might be helpful in future feature, for example - token refresh)

Note:
in the future, there will be secured login using token. This change will required adding token to user's table. Token will be generated as hash of email & password. The pwd (user's password) that is saved in DB will be encrypted. It means that in login implemention, the server should decrypt the saved password from DB in order to compare it to the sent password by client. If passwords are equal, server returns success, otherwise - failure. On login implemention we also need to consider the case where email & password exist in DB but the token expired, then server need to generate new one and user should be able to login with the new token. (client should support updating token on his side).

Feed's table:
feed_id (Primary key)
user_id (foriegn key, references user table)
updated_on - timestamp that represents the last time the feed was updated
feed_data - json that represnts the data of the feed

Image

Workflow

Image

Prerequisites

The following technologies should be installed globally

  • Node (preferred version / project version - 14.15.3)
  • ts-node
  • TypeScript (latest)
  • Nodemon
  • PostgreSQL

Get a key from this site (or use the one I supplied)

https://aviationstack.com/

Installation

Before running this project install node modules with this command:

npm install

API

1) Get data feed
Retrieves feed from an external source.
POST - http://localhost:9000/api/feed

Image

Try using curl:

curl --location --request POST 'http://localhost:9000/api/feed' \
--header 'Content-Type: application/json' \
--header 'Cookie: connect.sid=s%3ATgv3GUfYzw7-EcNOhnaJVjy8NVcg5aWH.ZlJHeqmwkwnVIWzj0tENzz61oy2%2FiLbLnv0NRvC6CiI' \
--data-raw '{
    "source": "http://api.aviationstack.com/v1/flights?access_key=${API_KEY}&limit=5"
}'

In case the source is not supplied by the client, the server returns an error.

Image

2) Filter feed
Filters are based on a specific filter.
Supported filters:

  • Limit results - choose the number of rows you want to see in the feed.
  • Show specified columns - choose which columns you want to see in the feed

    Note:
  • You can choose one of the following filters or both.
  • Filters like contains, greater than, lower than, will be implemented in the future.

POST - http://localhost:9000/api/feed/filter

An example of both filters applied: Image

Try using curl:

curl --location --request POST 'http://localhost:9000/api/feed/filter' \
--header 'Content-Type: application/json' \
--header 'Cookie: connect.sid=s%3AcwpJbF2WrTLlpin0cGFs_9d2fsyQAW2D.a%2FMWgd3wloyb7TWyrBnCSEkhwroRWd1bQbJ4dR2hUcM' \
--data-raw '{
    "filters": {
        "columns": ["flight"],
        "limit": 1
    }
}'

In case no column is sent, the filter(s) is ignored since there is no way to filter an empty set of data โ€“ this scenario is equal to not sending a request at all, and in a real-world system, the client should prevent it from going out. It is added here as a safeguard If the body of the request is empty, the server returns an error.

Image

If the body of the request is plain, server returns error
Image

Show only flight date & flight status columns:

Image

3) Update field
You can update field by passing the field name you want to alter & the new value.
Note: currently this feature updating row & multiple rows of specific column.

POST - http://localhost:9000/api/feed/updateField

Let's say we want to update the first row (first record) with a new value for the field "live".
Original field equals null as you can see:

Image

After sending the request to the server, you will notice the value is altered to 5.

Image

Try using curl:

curl --location --request POST 'http://localhost:9000/api/feed/updateField' \
--header 'Content-Type: application/json' \
--header 'Cookie: connect.sid=s%3AwIcv2CWOo4MfACEOzmeyAcOHIOjz0JI-.NVuIKBNKBIAh1%2F7CVQD9d4khNV6pFOqZBSlfCdJGGK0' \
--data-raw '{
    "data": {
        "rowId": 1,
        "fieldToUpdate": "live",
        "newValue": "5"
    }
}''

All data fields should be supplied. Otherwise, the server returns an error. Image

data-feed's People

Contributors

maia14 avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.