Description

Similar to Airline names we must add Bank names from the data dictionary as a part of validly-server.

Task

• Column mapping should include a new column which will incorporate bank name columns
• Then the validation rule will be added similar to how state validation is present

Introduction

In Validly Server - meta-facts different modification functions are written to update expectation key word arguments combine them to single modification function

• Add common modifier function in Validly Server - meta-facts

Description
Similar to States we must add PSU names from the data dictionary as a part of validly-server.

Task
Column mapping should include a new column that will incorporate PSU name columns
Then the validation rule will be added similar to how state validation is present

Is there a way that we would be able to update csv and dictionary files dynamically rather than making them only static files.

For every change made in the data dictionary , we have to made the same change and deploy the whole repo again with a new tag and deploy it.

Task : Create a new endpoint series which would:

• Take sheets from the Google sheet and create csv file accordingly for some name.
• would able to delete a csv file which contains data (in case if we would like to delete something)

Add new custom expectation to check if time periods has no breakage overall, mean if there no Month missing on between on year missing in between

1. Candidate column is treated as 'Date', so validations failed. The regex pattern to identify 'Date' columns should be modified.
2. We don't have any regex pattern to identify 'Currency' columns.

Validations and metadata is not generated for few csv processed files that are not in CSV - (utf-8) format

Below is the error log while uploading a non utf-8 file with validly server

INFO:fastapi:Created Minio Client
INFO:fastapi:Folder name :bbfa2a4f-353f-4698-b455-e70fff120822
INFO:fastapi:File name : output.csv
INFO:fastapi:Uploaded output.csv to minio
INFO:fastapi:Error reading Dataset from : bbfa2a4f-353f-4698-b455-e70fff120822/output.csv: 'utf-8' codec can't decode byte 0xa0 in position 794: invalid start byte
INFO:     10.40.3.10:50116 - "POST /expectation/datasets/?format=json HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 376, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 208, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/cors.py", line 92, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/usr/local/lib/python3.9/site-packages/starlette/middleware/cors.py", line 147, in simple_response
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 226, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 159, in run_endpoint_function
    return await dependant.call(**values)
  File "/app/./app/api/api_v1/routers/dataset.py", line 96, in execute_dataset_expectation_post
    expectations = await datasets_expectation(s3_files_key, result_type)
  File "/app/./app/utils/dataset.py", line 60, in datasets_expectation
    expectations = await asyncio.gather(
  File "/app/./app/utils/dataset.py", line 24, in dataset_expectation
    dataset = await read_dataset(dataset_path, s3_client, bucket_name)
  File "/app/./app/utils/common.py", line 39, in read_dataset
    return dataset
UnboundLocalError: local variable 'dataset' referenced before assignment
ERROR:uvicorn.error:Exception in ASGI application

Use json-to-pydantic to get response models and the pass it as part of routes

Description

For the first version of validly , endpoints are coupled with the Jinja Templates, thus , there are various instances if HTML-templates, CSS and other unnecessary stuff that is clutter to the repository

Tasks

Make a list of all such files (HTML/CSS) that are not used in newer versions of validly and remove them

validly(Dataset check)

• Datetime column lengths(4-10)
• - numbers in text fields
• - Minimum no. of rows or size
• - latest data on top
• - update data dictionaries like countries and districts
• - Detecting special characters
• - Flag Data-Time empty cells
• - column_names in lower case
• - Display local path for Directory
• Numeric values format(roundoff to 2 decimal digits)
• Identify Negative Values(Flag)

metafacts(Generating Metadata from Dataset)

• adding temporal coverage for datasets with dates
• Duplicate units
• Granularity (airline, language, districts, crops, gender)
• Separate granularity for Fiscal Years
• remove formats_available, is_public
• Display local path for Directory

validly metafacts(Metadata check)

• length of title (factly_dataset_name) < 200
• Spell Check
• to check whether file_path in s3
• Columns order
• length of description to be > 50

All endpoints for columns-mapped, datetime, geography, unit, notes, etc should me modified to post requests

Description

Ambiguity between existence of datasets with less than 10 rows, thus we must raise error if they are less.

Introduction

Instead of reading the standard names from local file read the standard names from google sheet will be more easy for reading updates from NR team so we are changing way of reading the standard name

• Make necessary changes to read standard names from google sheet.

Description

While creating metadata , there are certain error that could be automatically check. Thus we must create validation rules for meta-data in the similar fashion it is present for Datasets validation.

Consider below mention rules to start with :

• Set of values for : Sectors, Organizations, Short-form, Frequency of update
• Proper pattern for : File path, Data next update date
• Range of values : Time Saved in hours

Task

• Endpoint route that take an upload file object and does validations on those.
• Endpoint route that accepts Google Sheet ID and worksheet name where meta-data is stored

Description

Cross-check if missing values are present due to Cleaning issue or due to source only.
We must flag any null row with missing value so that user can check if it is because the raw file is empty or because of some error made while cleaning the datasets

Need documentation related to following topics :

• Set up validly project locally and setting up it for development
• How to add and new expectation for a column type
• How to create custom expectation and add it to expectation suite

Description

There should be validation within time series datasets that checks if a portion within time-series is left out or not.
For example : If a monthly granular file from 2010-2022 dataset if 2014 is missing(or any month) of it , then it should throw an error.

Unable to retrieve metadata from S3 buckets for projects containing large files.

• Districts
• Insurance company names
• PSU Names

Description
Validly raises an error if Multiple sector values are present in a single cell.

Task
Identity all the sector names in a single cell and check if all sector names are proper or not.