Comments (1)
So after examining this a bit, it seems the breakdown is this:
- Reading the data file into a dictionary: 58% of the time
- Creating the Datapath object (from_dict): 42% of the time
By simply not adding in the raw data to the object, we can reduce the "Creating the Datapath object" by about 78%, AKA reducing the total load time by 31%. Source: loading the structured big test file (250MB) on laptop repeatedly
This simple fix might also take care of the memory issues.
I'm not sure the best way to programmatically not load the raw file from disk into dictionary; one way around it would be changing the defaults so that raw data is not saved unless specified, then the current BEEPDatapath code would automatically ignore the data and the run would be marked as "legacy". I'll update my PR with some timing results.
Edit:
Decided to just not save raw data when using CLI by default. It's the easiest and cleanest solution, requiring no changes to as/from_dict. See #250
from beep.
Related Issues (20)
- [Feature Request] Documentation on How to Contribute to BEEP Codebase
- Calculate Diagnostic Features at Every Diagnostic Cycle HOT 2
- [Feature Request] Integrate with amplabs.ai API Read/GET endpoints
- How to load Biologic raw data file in beep ? [Question] HOT 2
- [Feature Request] Novonix Datapath HOT 2
- Add Novonix data requirements to the documentation
- Featurize the Structured Data @ardunn HOT 2
- Create ability for datapaths to produce plots
- Noxonix protocol generation HOT 1
- [Feature Request] Usability of working with data
- Incorporate unifying scheme for featurization
- [Feature Request] Decouple the modelling from the data
- Include `step_type_name` in structured data
- [Bug] Allow for structured summary to account gracefully for steps without cv segment HOT 1
- New Arbin Log HOT 2
- Struggling with featurization HOT 2
- [Question]
- [Bug] pip install crashes HOT 1
- `indeterminate_step_charge_default` needs test
- multiple steps with same step-index are collated together within a single cycle
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from beep.