Comments (12)
patrickherring-TRI
commented on August 19, 2024
1
That's a good point.
For us, data is typically in cloud storage, and I imagine for many users the data is in some shared drive. So being able to point to the data directory makes a lot of sense.
But in order to try out the package, pointing to the beep installation directory is probably better since it's less likely that there will be permission issues.
from beep.
tinosulzer
commented on August 19, 2024
Ok, I did export AWS_DEFAULT_REGION=us-west-2 and the region error was fixed but now I am getting a similar NoCredentialsError.
from beep.
patrickherring-TRI
commented on August 19, 2024
Hi Valentin!
Thanks for trying out our package!
Could you try setting the environment variable BEEP_ENV to 'dev'?
export BEEP_ENV='dev'
I think that should solve the problem, but let me know if you are still seeing errors when running tests.
Thanks,
Patrick
from beep.
tinosulzer
commented on August 19, 2024
Thanks @patrickherring-TRI , that got me a bit further but I'm now getting errors like
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPConnection object at 0x13cfde990>: Failed to establish a new connection: [Errno 64] Host is down
and the tests result is
----------------------------------------------------------------------
Ran 46 tests in 8.678s
FAILED (SKIP=1, errors=33)
I also tried to run collate but got the error
FileNotFoundError: [Errno 2] No such file or directory: 'data-share/raw_cycler_files'
Should I download data-share separately? Or adapt to use my own data?
And running anything else (e.g. run-model) raised more credential errors:
Credential retry:0
Credential retry:1
Credential retry:2
Credential retry:3
Credential retry:4
Credential retry:5
Credential retry:6
Credential retry:7
Credential retry:8
Credential retry:9
Credential retry:10
Credential retry:11
Traceback (most recent call last):
File "/Users/vsulzer/Documents/Energy_storage/BEEP/venv/bin/run_model", line 11, in <module>
load_entry_point('beep', 'console_scripts', 'run_model')()
File "/Users/vsulzer/Documents/Energy_storage/BEEP/beep/run_model.py", line 606, in main
raise e
File "/Users/vsulzer/Documents/Energy_storage/BEEP/beep/run_model.py", line 603, in main
print(process_file_list_from_json(input_json, model_dir=MODEL_DIR), end="")
File "/Users/vsulzer/Documents/Energy_storage/BEEP/beep/run_model.py", line 503, in process_file_list_from_json
events = KinesisEvents(service='DataAnalyzer', mode=file_list_data['mode'])
File "/Users/vsulzer/Documents/Energy_storage/BEEP/beep/utils/events.py", line 80, in __init__
raise NoCredentialsError("Unable to get credentials in specified number of retries")
TypeError: __init__() takes 1 positional argument but 2 were given
from beep.
patrickherring-TRI
commented on August 19, 2024
Does something like this work?
$ structure '{
"mode": "events_off",
"run_list": [1],
"validity": ["valid"],
"file_list": ["/Users/vsulzer/Documents/Energy_storage/BEEP/beep/tests/test_files/FastCharge_000000_CH29.csv"]}'
from beep.
tinosulzer
commented on August 19, 2024
Works for a bit then fails:
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 188/188 [00:15<00:00, 12.40it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 188/188 [00:16<00:00, 11.15it/s]
Traceback (most recent call last):
File "/Users/vsulzer/Documents/Energy_storage/BEEP/venv/bin/structure", line 11, in <module>
load_entry_point('beep', 'console_scripts', 'structure')()
File "/Users/vsulzer/Documents/Energy_storage/BEEP/beep/structure.py", line 1594, in main
raise e
File "/Users/vsulzer/Documents/Energy_storage/BEEP/beep/structure.py", line 1591, in main
print(process_file_list_from_json(input_json))
File "/Users/vsulzer/Documents/Energy_storage/BEEP/beep/structure.py", line 1556, in process_file_list_from_json
dumpfn(processed_cycler_run, processed_cycler_run_loc)
File "/Users/vsulzer/Documents/Energy_storage/BEEP/venv/lib/python3.7/site-packages/monty/serialization.py", line 131, in dumpfn
with zopen(fn, "wt") as fp:
File "/Users/vsulzer/Documents/Energy_storage/BEEP/venv/lib/python3.7/site-packages/monty/io.py", line 43, in zopen
return io.open(filename, *args, **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: '/data-share/structure/FastCharge_000000_CH29_structure.json'
from beep.
patrickherring-TRI
commented on August 19, 2024
Quick fix would be to create a directory /data-share/structure on your system, I will also look at creating a better solution right now.
If you create that directory, does the previous command work?
from beep.
tinosulzer
commented on August 19, 2024
I couldn't create a directory in / but after creating data-share/structure inside beep and changing the BEEP_ROOT it worked:
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 188/188 [00:14<00:00, 12.99it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 188/188 [00:16<00:00, 11.35it/s]
{"file_list": ["/Users/vsulzer/Documents/Energy_storage/BEEP/data-share/structure/FastCharge_000000_CH29_structure.json"], "run_list": [1], "result_list": ["success"], "message_list": [{"comment": "", "error": ""}], "invalid_file_list": []}
from beep.
patrickherring-TRI
commented on August 19, 2024
Great! Sorry for the issue, I think I understand what the problem is now. I am going to put together a PR to change the test setup and folder structure creation.
from beep.
tinosulzer
commented on August 19, 2024
No problem, I'll try again when the PR is merged.
By the way, is the idea that BEEP_ROOT should always point to the beep installation? Or do you usually point BEEP_ROOT to somewhere else (e.g. some cloud storage)? If the former, you could add a function like
def root_dir():
""" return the root directory of the BEEP install directory """
return str(pathlib.Path(beep.__path__[0]).parent)which always points to where beep has been installed
from beep.
tinosulzer
commented on August 19, 2024
I set up my AWS credentials and changed back to BEEP_ENV="local" and tried running the tests again, it does a bit better (Ran 57, FAILED (SKIP=2, errors=18, failures=1)) but now most of the errors are like this:
======================================================================
ERROR: test_serialized_prediction (test_models.TestRunModel)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/vsulzer/Documents/Energy_storage/BEEP/beep/tests/test_models.py", line 103, in test_serialized_prediction
processed_dir=TEST_FILE_DIR)
File "/Users/vsulzer/Documents/Energy_storage/BEEP/beep/run_model.py", line 503, in process_file_list_from_json
events = KinesisEvents(service='DataAnalyzer', mode=file_list_data['mode'])
File "/Users/vsulzer/Documents/Energy_storage/BEEP/beep/utils/events.py", line 85, in __init__
self.stream = get_secret(config['test']['kinesis']['stream'])['streamName']
File "/Users/vsulzer/Documents/Energy_storage/BEEP/beep/utils/secrets_manager.py", line 58, in get_secret
raise e
File "/Users/vsulzer/Documents/Energy_storage/BEEP/beep/utils/secrets_manager.py", line 55, in get_secret
SecretId=secret_name
File "/Users/vsulzer/Documents/Energy_storage/BEEP/venv/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/vsulzer/Documents/Energy_storage/BEEP/venv/lib/python3.7/site-packages/botocore/client.py", line 626, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.ResourceNotFoundException: An error occurred (ResourceNotFoundException) when calling the GetSecretValue operation: Secrets Manager can't find the specified secret.
-------------------- >> begin captured logging << --------------------
botocore.hooks: DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x10b5147a0>
botocore.hooks: DEBUG: Event creating-client-class.kinesis: calling handler <function add_generate_presigned_url at 0x10b4e53b0>
botocore.endpoint: DEBUG: Setting kinesis timeout as (60, 60)
botocore.client: DEBUG: Registering retry handlers for service: kinesis
botocore.hooks: DEBUG: Event before-parameter-build.kinesis.ListStreams: calling handler <function generate_idempotent_uuid at 0x10b546dd0>
botocore.hooks: DEBUG: Event before-call.kinesis.ListStreams: calling handler <function inject_api_version_header_if_needed at 0x10b54f8c0>
botocore.endpoint: DEBUG: Making request for OperationModel(name=ListStreams) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'Kinesis_20131202.ListStreams', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.12.4 Python/3.7.7 Darwin/19.4.0 Botocore/1.15.4'}, 'body': b'{}', 'url': 'https://kinesis.us-east-2.amazonaws.com/', 'context': {'client_region': 'us-east-2', 'client_config': <botocore.config.Config object at 0x13086e1d0>, 'has_streaming_input': False, 'auth_type': None}}
botocore.hooks: DEBUG: Event request-created.kinesis.ListStreams: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x130880f50>>
botocore.hooks: DEBUG: Event choose-signer.kinesis.ListStreams: calling handler <function set_operation_specific_signer at 0x10b546320>
botocore.auth: DEBUG: Calculating signature using v4 auth.
botocore.auth: DEBUG: CanonicalRequest:
POST
/
content-type:application/x-amz-json-1.1
host:kinesis.us-east-2.amazonaws.com
x-amz-date:20200521T230557Z
x-amz-target:Kinesis_20131202.ListStreams
content-type;host;x-amz-date;x-amz-target
44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
botocore.auth: DEBUG: StringToSign:
AWS4-HMAC-SHA256
20200521T230557Z
20200521/us-east-2/kinesis/aws4_request
719feeb6c07c2d8eaf634b66bb7006e78f3839c8e2da29e6c55ff0fc023eaed3
botocore.auth: DEBUG: Signature:
ddd7ee240f8ff04793c1056ed2f96801d6f1a7827440cf5ff8069acb2a6e4316
botocore.endpoint: DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://kinesis.us-east-2.amazonaws.com/, headers={'X-Amz-Target': b'Kinesis_20131202.ListStreams', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'Boto3/1.12.4 Python/3.7.7 Darwin/19.4.0 Botocore/1.15.4', 'X-Amz-Date': b'20200521T230557Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIARMITENQXCCNA6LEU/20200521/us-east-2/kinesis/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=ddd7ee240f8ff04793c1056ed2f96801d6f1a7827440cf5ff8069acb2a6e4316', 'Content-Length': '2'}>
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): kinesis.us-east-2.amazonaws.com:443
urllib3.connectionpool: DEBUG: https://kinesis.us-east-2.amazonaws.com:443 "POST / HTTP/1.1" 200 41
botocore.parsers: DEBUG: Response headers: {'x-amzn-RequestId': 'e7b2999f-43d6-240e-b0f3-920972a71bac', 'x-amz-id-2': 'AIVqt5N8kDf94dcIkVg+fXxSZY8DsQmZfyNLsGZHWZxyZjyLyoywm7W9Q1fY+BgR3bU8FMqEOwBJoNCUhBPyjsaWb7LmM9xi', 'Date': 'Thu, 21 May 2020 23:05:58 GMT', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '41'}
botocore.parsers: DEBUG: Response body:
b'{"HasMoreStreams":false,"StreamNames":[]}'
botocore.hooks: DEBUG: Event needs-retry.kinesis.ListStreams: calling handler <botocore.retryhandler.RetryHandler object at 0x130895d50>
botocore.retryhandler: DEBUG: No retry needed.
botocore.hooks: DEBUG: Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
botocore.hooks: DEBUG: Changing event name from before-call.apigateway to before-call.api-gateway
botocore.hooks: DEBUG: Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
botocore.hooks: DEBUG: Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
botocore.hooks: DEBUG: Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
botocore.hooks: DEBUG: Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
botocore.hooks: DEBUG: Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
botocore.hooks: DEBUG: Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
botocore.hooks: DEBUG: Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
botocore.hooks: DEBUG: Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
botocore.hooks: DEBUG: Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
botocore.credentials: DEBUG: Looking for credentials via: env
botocore.credentials: DEBUG: Looking for credentials via: assume-role
botocore.credentials: DEBUG: Looking for credentials via: assume-role-with-web-identity
botocore.credentials: DEBUG: Looking for credentials via: shared-credentials-file
botocore.credentials: INFO: Found credentials in shared credentials file: ~/.aws/credentials
botocore.loaders: DEBUG: Loading JSON file: /Users/vsulzer/Documents/Energy_storage/BEEP/venv/lib/python3.7/site-packages/botocore/data/endpoints.json
botocore.hooks: DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x10b5147a0>
botocore.loaders: DEBUG: Loading JSON file: /Users/vsulzer/Documents/Energy_storage/BEEP/venv/lib/python3.7/site-packages/botocore/data/secretsmanager/2017-10-17/service-2.json
botocore.loaders: DEBUG: Loading JSON file: /Users/vsulzer/Documents/Energy_storage/BEEP/venv/lib/python3.7/site-packages/botocore/data/secretsmanager/2017-10-17/service-2.sdk-extras.json
botocore.hooks: DEBUG: Event creating-client-class.secrets-manager: calling handler <function add_generate_presigned_url at 0x10b4e53b0>
botocore.endpoint: DEBUG: Setting secretsmanager timeout as (60, 60)
botocore.loaders: DEBUG: Loading JSON file: /Users/vsulzer/Documents/Energy_storage/BEEP/venv/lib/python3.7/site-packages/botocore/data/_retry.json
botocore.client: DEBUG: Registering retry handlers for service: secretsmanager
botocore.hooks: DEBUG: Event before-parameter-build.secrets-manager.GetSecretValue: calling handler <function generate_idempotent_uuid at 0x10b546dd0>
botocore.hooks: DEBUG: Event before-call.secrets-manager.GetSecretValue: calling handler <function inject_api_version_header_if_needed at 0x10b54f8c0>
botocore.endpoint: DEBUG: Making request for OperationModel(name=GetSecretValue) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'secretsmanager.GetSecretValue', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.12.4 Python/3.7.7 Darwin/19.4.0 Botocore/1.15.4'}, 'body': b'{"SecretId": "local/beep/eventstream"}', 'url': 'https://secretsmanager.us-west-2.amazonaws.com/', 'context': {'client_region': 'us-west-2', 'client_config': <botocore.config.Config object at 0x1308870d0>, 'has_streaming_input': False, 'auth_type': None}}
botocore.hooks: DEBUG: Event request-created.secrets-manager.GetSecretValue: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x130887050>>
botocore.hooks: DEBUG: Event choose-signer.secrets-manager.GetSecretValue: calling handler <function set_operation_specific_signer at 0x10b546320>
botocore.auth: DEBUG: Calculating signature using v4 auth.
botocore.auth: DEBUG: CanonicalRequest:
POST
/
content-type:application/x-amz-json-1.1
host:secretsmanager.us-west-2.amazonaws.com
x-amz-date:20200521T230558Z
x-amz-target:secretsmanager.GetSecretValue
content-type;host;x-amz-date;x-amz-target
c5763faee03d0d8a3d28e22207a85ed8cd6db3208b6b5007b772485712bb1914
botocore.auth: DEBUG: StringToSign:
AWS4-HMAC-SHA256
20200521T230558Z
20200521/us-west-2/secretsmanager/aws4_request
c17eeae423cc2d24ece9cd8f7f9c92ea45b22b0463e545784e8473446e4cd25f
botocore.auth: DEBUG: Signature:
f18f1db7b52fa2308415f44d7da697bcabde280be4ca6629967f6c6248597c68
botocore.endpoint: DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://secretsmanager.us-west-2.amazonaws.com/, headers={'X-Amz-Target': b'secretsmanager.GetSecretValue', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'Boto3/1.12.4 Python/3.7.7 Darwin/19.4.0 Botocore/1.15.4', 'X-Amz-Date': b'20200521T230558Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIARMITENQXCCNA6LEU/20200521/us-west-2/secretsmanager/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=f18f1db7b52fa2308415f44d7da697bcabde280be4ca6629967f6c6248597c68', 'Content-Length': '38'}>
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): secretsmanager.us-west-2.amazonaws.com:443
urllib3.connectionpool: DEBUG: https://secretsmanager.us-west-2.amazonaws.com:443 "POST / HTTP/1.1" 400 99
botocore.parsers: DEBUG: Response headers: {'Date': 'Thu, 21 May 2020 23:05:58 GMT', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '99', 'Connection': 'keep-alive', 'x-amzn-RequestId': '2a61612b-1bc4-4253-b4b0-8da03865589f'}
botocore.parsers: DEBUG: Response body:
b'{"__type":"ResourceNotFoundException","Message":"Secrets Manager can\'t find the specified secret."}'
botocore.hooks: DEBUG: Event needs-retry.secrets-manager.GetSecretValue: calling handler <botocore.retryhandler.RetryHandler object at 0x130887590>
botocore.retryhandler: DEBUG: No retry needed.
--------------------- >> end captured logging << ---------------------
from beep.
patrickherring-TRI
commented on August 19, 2024
Yes, the associated PR is going to change to a broad exception catch so that whenever it cannot access AWS for any reason it will fall back to processing using local resources only.
from beep.
Related Issues (20)
- [Feature Request] Documentation on How to Contribute to BEEP Codebase
- Calculate Diagnostic Features at Every Diagnostic Cycle HOT 2
- [Feature Request] Integrate with amplabs.ai API Read/GET endpoints
- How to load Biologic raw data file in beep ? [Question] HOT 2
- [Feature Request] Novonix Datapath HOT 2
- Add Novonix data requirements to the documentation
- Featurize the Structured Data @ardunn HOT 2
- Create ability for datapaths to produce plots
- Noxonix protocol generation HOT 1
- [Feature Request] Usability of working with data
- Incorporate unifying scheme for featurization
- [Feature Request] Decouple the modelling from the data
- Include `step_type_name` in structured data
- [Bug] Allow for structured summary to account gracefully for steps without cv segment HOT 1
- New Arbin Log HOT 2
- Struggling with featurization HOT 2
- [Question]
- [Bug] pip install crashes HOT 1
- `indeterminate_step_charge_default` needs test
- multiple steps with same step-index are collated together within a single cycle
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from beep.