biglocalnews / bln-etl Goto Github PK
View Code? Open in Web Editor NEWUtilities for BLN data gathering pipelines.
License: Other
Utilities for BLN data gathering pipelines.
License: Other
Basic wrapper class that cribs most frequently used bits from the BLN sdk, and provides a simple Project and file-oriented interface for managing files on the platform.
Project.update
should enable updating of optional metadata such as project description.
Should create proper docstrings and update API usage docs to list all options.
pip install
instructionsFile.download
needs to be implemented.
Using @pytest.mark.vcr()
to record web interactions for testing produces the below error related to creating file upload URIs. We've sidestepped by marking such tests as a "webtest" and allowing them to hit the live platform, but it's worth investigating more deeply for a potential fix that avoids hitting the live platform.
def test_project_upload_files():
# NOTE: This is a live webtest to sidestep headaches
# related to handling errors raised during the process
# of generating file upload URIs (KeyError: 'createFileUploadUri')
kwargs = {
'is_open': False ,
'description': 'This is a test project.'
}
uuid = '<FAKE_UUID>'
project = Project.get(uuid, TOKEN)
to_upload = [
fixture_path('test.csv'),
fixture_path('test2.csv')
]
project.upload_files(to_upload)
expected = ['test.csv', 'test2.csv']
actual = [f.name for f in project.files]
> assert project.files == expected
Basic wrapper class that shells out to git cli utilities. At outset, just need the basics to clone and update:
Repository('/local/path/to/repo')
Repository.clone
Repository.pull
It seems like a good thing to do long term, and might be required by tickets like this.
The current webtests assume the existence of (and access to) a Test project
created by me. This is gross, since other folks can't use it without being added to the project as admins and the tests will break if the project gets deleted (its UUID is hard-coded in several tests).
Update the test setup with a module-scope fixture that bootstrap and tear down a new platform project on each run of webtests.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.