microsoft / batch-processing-kit Goto Github PK

Generic batch processing framework for managing the orchestration, dispatch, fault tolerance, and monitoring of arbitrary work items against many endpoints. Extensible via dependency injection. Includes examples against Cognitive Service containers for ML eval workloads.

License: MIT License

Python 99.37% Shell 0.35% Dockerfile 0.28%

batch-processing-kit's Introduction

Introduction

Generic batch processing framework for managing the orchestration, dispatch, fault tolerance, and monitoring of arbitrary work items against many endpoints. Extensible via dependency injection. Worker endpoints can be local, remote, containers, cloud APIs, different processes, or even just different listener sockets in the same process.

Includes examples against Azure Cognitive Service containers for ML eval workloads.

Consuming

The framework can be built on via template method pattern and dependency injection. One simply needs to provide concrete implementation for the following types:

WorkItemRequest: Encapsulates all the details needed by the WorkItemProcessor to process a work item.

WorkItemResult: Representation of the outcome of an attempt to process a WorkItemRequest.

WorkItemProcessor: Provides implementation on how to process a WorkItemRequest against an endpoint.

BatchRequest: Represents a batch of work items to do. Produces a collection of WorkItemRequests.

BatchConfig: Details needed for a BatchRequest to produce the collection of WorkItemRequests.

BatchRunSummarizer: Implements a near-real-time status updater based on WorkItemResults as the batch progresses.

EndpointStatusChecker: Specifies how to determine whether an endpoint is healthy and ready to take on work from a WorkItemProcessor.

The Speech Batch Kit is currently our prime example for consuming the framework.

The batchkit package is available as an ordinary pypi package. See versions here: https://pypi.org/project/batchkit

Dev Environment

This project is developed for and consumed in Linux environments. Consumers also use WSL2, and other POSIX platforms may be compatible but are untested. For development and deployment outside of a container, we recommend using a Python virtual environment to install the requirements.txt. The Speech Batch Kit example builds a container.

Tests

This project uses both unit tests run-tests and stress tests run-stress-tests for functional verification.

Building

There are currently 3 artifacts:

The pypi library of the batchkit framework as a library.
The pypi library of the batchkit-examples-speechsdk.
Docker container image for speech-batch-kit.

Examples

Speech Batch Kit

The Speech Batch Kit (batchkit_examples/speech_sdk) uses the framework to produce a tool that can be used for transcription of very large numbers of audio files against Azure Cognitive Service Speech containers or cloud endpoints.

For introduction, see the Azure Cognitive Services page.

For detailed information, see the Speech Batch Kit's README.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repositories using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

batch-processing-kit's People

Contributors

Stargazers

Watchers

Forkers

anwald aahill taffywrinkle claudiusgonzo sverigesradio danielfalksr global19 global-localhost global19-atlassian-net tryweirder standardgalactic arturosorio test-mass-forker-org-1 isabella232 abossard adam-s-tech gougouccnu mahya8585

batch-processing-kit's Issues

/submit request with REST mode not working

I get this error :
Request body argument 'type' not found in syspath: None

My request :
{"type": "TBatchConfig","files": ["D:\Projects\Call-33.mp3", "D:\Projects\Call-34.mp3"],"some_int": 1,"some_bool": "True"}

BatchKit Processing in Docker fails on APISERVER submit

I am trying to run the batch container along with a speech container locally on Windows 10 using WSL2 docker.

When I run the Daemon mode on the batch container, I am able to see the transcripts being processed and generated successfully.

When I run the APISERVER I am getting a blocking log that never seems to go away:

2022-10-28 21:01:04.528:DEBUG:MainProcess:SpeechContainer1: {
  "concurrency": 1,
  "host": "SpeechToText",
  "isCloudService": false,
  "isSecure": false,
  "language": "en-US",
  "port": 5000,
  "rtf": 1,
  "startConcurrency": 1
}
2022-10-28 21:01:04.530:INFO:MainProcess:Set new EndpointManagers after hot-swap: {'SpeechContainer1': {'concurrency': 1, 'host': 'SpeechToText', 'port': 5000, 'rtf': 1, 'language': 'en-US', 'isCloudService': False, 'isSecure': False, 'startConcurrency': 1}}
2022-10-28 21:01:04.530:INFO:MainProcess:Orchestrator: Starting batch 294213034283874517753283500456277750068
2022-10-28 21:01:04.552:DEBUG:MainProcess:Prioritizing work item: /my_nfs/audio_files/2897306.30s.wav at priority: 62528
2022-10-28 21:01:04.553:INFO:MainProcess:write_json_file_atomic():  Atomically wrote file /tmp/tmpcwdp0jp3/batch_294213034283874517753283500456277750068/run_summary.json
2022-10-28 21:01:04.556:DEBUG:MainProcess:EndpointManager name: HotswapGen1_SpeechContainer1  will try to steal work.
2022-10-28 21:01:04.556:DEBUG:MainProcess:Prioritizing work item: /my_nfs/audio_files/2897306.30s.wav at priority: 62528
2022-10-28 21:01:04.556:INFO:MainProcess:EndpointManager name: HotswapGen1_SpeechContainer1  was requested to stop while stealing work.

I dug through the project a bit and think it has to do with line 174 in endpoint_manager.py. I don't really understand yet what might be causing the file to be set to "STOP" though.

I was worried I was providing the wrong file path so I did double check that the file path I provided to the submit endpoint is the same as the one that is generated when the batch endpoint works.

An extra white space in the -log_folder parameter makes the program exit with a FailedRecognitionError

The below command doesn't work as there is an additional space before "/my_nfs/logs" for the -log_folder parameter.

docker run --rm -ti -v /home/ml/my_nfs:/my_nfs docker.io/batchkit/speech-batch-kit:latest -config /my_nfs/config.yaml -input_folder /my_nfs/audio_files -output_folder /my_nfs/transcriptions -log_folder /my_nfs/logs -file_log_level DEBUG -nbest 1 -m ONESHOT -diarization None -profanity Raw -language en-US -console_log_level DEBUG

The output we get is only console based as the logs also doesn't get saved in this scenario.

Please find the logs below:

2020-10-27 19:30:43.472:INFO:LoggingListenerProcess:Log folder \udcc2\udca0/my_nfs/logs created successfully
2020-10-27 19:30:43.477:INFO:MainProcess:Starting a watch on file: /my_nfs/config.yaml
2020-10-27 19:30:43.477:INFO:MainProcess:Orchestrator created by process: 1
2020-10-27 19:30:43.477:INFO:MainProcess:client.py: run(): Running:  OneShotClient  with settings: Settings(input_folder='/my_nfs/audio_files', input_list=None, output_folder='/my_nfs/transcriptions', apiserver_port=5000, scratch_folder='/my_nfs/transcriptions/.scratch', log_folder='\udcc2\udca0/my_nfs/logs', store_combined_json=False, config_file='/my_nfs/config.yaml', strict_configuration_validation=False),  and speech config: SpeechSDKBatchConfig({'combine_results': False, 'language': 'en-US', 'nbest': 1, 'diarization': 'None', 'profanity': 'Raw', 'sentiment': False, 'allow_resume': False})
2020-10-27 19:30:43.869:DEBUG:MainProcess:BatchRequest subtypes on sys.path identified: [<class 'batchkit_examples.speech_sdk.batch_request.SpeechSDKBatchRequest'>]
2020-10-27 19:30:43.900:DEBUG:MainProcess:BatchRequest subtypes on sys.path identified: [<class 'batchkit_examples.speech_sdk.batch_request.SpeechSDKBatchRequest'>]
2020-10-27 19:30:43.900:INFO:MainProcess:OneShotClient:  Submitting SpeechSDKBatchRequest with 2 files.
2020-10-27 19:30:43.901:INFO:MainProcess:ApiServer: submit() request:  {"py/object": "batchkit_examples.speech_sdk.batch_request.SpeechSDKBatchRequest", "batch_id": 331792578788503786707517342978000830611, "files": ["/my_nfs/audio_files/jurrasic1_16k_mono.wav", "/my_nfs/audio_files/sotl1.wav"], "combine_results": false, "language": "en-US", "diarization": "None", "nbest": 1, "profanity": "Raw", "allow_resume": false, "enable_sentiment": false}
2020-10-27 19:30:43.902:INFO:MainProcess:OneShotClient:  Waiting on batch id: 331792578788503786707517342978000830611, in status: BatchStatusEnum.waiting
2020-10-27 19:30:43.902:INFO:MainProcess:ApiServer: watch() request:  batch_id: 331792578788503786707517342978000830611, target_state: BatchStatusEnum.done, timeout: 30
2020-10-27 19:30:43.921:DEBUG:MainProcess:Original configuration:
2020-10-27 19:30:43.921:DEBUG:MainProcess:MyContainer: {
  "concurrency": 2,
  "host": "10.100.169.79",
  "port": 5000
}
2020-10-27 19:30:43.921:DEBUG:MainProcess:Normalized configuration:
2020-10-27 19:30:43.921:DEBUG:MainProcess:MyContainer: {
  "concurrency": 2,
  "host": "10.100.169.79",
  "isCloudService": false,
  "isSecure": false,
  "language": "en-US",
  "port": 5000,
  "rtf": 3.0,
  "startConcurrency": 2
}
2020-10-27 19:30:43.928:INFO:MainProcess:Set new EndpointManagers after hot-swap: {'MyContainer': {'concurrency': 2, 'host': '10.100.169.79', 'port': 5000, 'rtf': 3.0, 'language': 'en-US', 'isCloudService': False, 'isSecure': False, 'startConcurrency': 2}}
2020-10-27 19:30:43.929:INFO:MainProcess:Orchestrator: Starting batch 331792578788503786707517342978000830611
2020-10-27 19:30:43.933:DEBUG:MainProcess:Updating singleton run_summary: /my_nfs/transcriptions/run_summary.json
2020-10-27 19:30:43.934:INFO:MainProcess:write_json_file_atomic():  Atomically wrote file /my_nfs/transcriptions/run_summary.json
2020-10-27 19:30:43.936:DEBUG:MainProcess:EndpointManager name: HotswapGen0_MyContainer  will try to steal work
2020-10-27 19:30:43.936:DEBUG:MainProcess:EndpointManager name: HotswapGen0_MyContainer  stole work and will delegate to a worker.
2020-10-27 19:30:43.936:DEBUG:MainProcess:EndpointManager name: HotswapGen0_MyContainer  will try to steal work
2020-10-27 19:30:43.936:DEBUG:MainProcess:EndpointManager name: HotswapGen0_MyContainer  stole work and will delegate to a worker.
2020-10-27 19:30:43.937:INFO:HotswapGen0_MyContainer_PoolWorker-2:Processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, with process: HotswapGen0_MyContainer_PoolWorker-2
2020-10-27 19:30:43.937:INFO:HotswapGen0_MyContainer_PoolWorker-3:Processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, with process: HotswapGen0_MyContainer_PoolWorker-3
2020-10-27 19:30:43.940:DEBUG:HotswapGen0_MyContainer_PoolWorker-3:Starting FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-3__SDKRequestChildProc
2020-10-27 19:30:43.940:DEBUG:HotswapGen0_MyContainer_PoolWorker-2:Starting FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-2__SDKRequestChildProc
2020-10-27 19:30:44.130:DEBUG:HotswapGen0_MyContainer_PoolWorker-3:Finished FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-3__SDKRequestChildProc
2020-10-27 19:30:44.130:DEBUG:HotswapGen0_MyContainer_PoolWorker-3:Exception details: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 143, in recognize
    audio_duration = self.__recognize_in_subproc(self._filepath, json_data, cancellation_token)
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 231, in __recognize_in_subproc
    raise FailedRecognitionError(err_msg)
batchkit.utils.FailedRecognitionError: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-3__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-3

2020-10-27 19:30:44.130:WARNING:HotswapGen0_MyContainer_PoolWorker-3:FailedRecognitionError happened, must terminate the recognition.Details: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-3__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-3
2020-10-27 19:30:44.130:DEBUG:HotswapGen0_MyContainer_PoolWorker-2:Finished FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-2__SDKRequestChildProc
2020-10-27 19:30:44.131:INFO:HotswapGen0_MyContainer_PoolWorker-3:Processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, with process: HotswapGen0_MyContainer_PoolWorker-3
2020-10-27 19:30:44.131:DEBUG:HotswapGen0_MyContainer_PoolWorker-2:Exception details: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 143, in recognize
    audio_duration = self.__recognize_in_subproc(self._filepath, json_data, cancellation_token)
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 231, in __recognize_in_subproc
    raise FailedRecognitionError(err_msg)
batchkit.utils.FailedRecognitionError: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-2__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-2

2020-10-27 19:30:44.131:WARNING:HotswapGen0_MyContainer_PoolWorker-2:FailedRecognitionError happened, must terminate the recognition.Details: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-2__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-2
2020-10-27 19:30:44.131:INFO:HotswapGen0_MyContainer_PoolWorker-2:Processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, with process: HotswapGen0_MyContainer_PoolWorker-2
2020-10-27 19:30:44.132:DEBUG:HotswapGen0_MyContainer_PoolWorker-3:Starting FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-3__SDKRequestChildProc
2020-10-27 19:30:44.133:DEBUG:HotswapGen0_MyContainer_PoolWorker-2:Starting FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-2__SDKRequestChildProc
2020-10-27 19:30:44.211:DEBUG:HotswapGen0_MyContainer_PoolWorker-3:Finished FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-3__SDKRequestChildProc
2020-10-27 19:30:44.211:DEBUG:HotswapGen0_MyContainer_PoolWorker-2:Finished FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-2__SDKRequestChildProc
2020-10-27 19:30:44.211:DEBUG:HotswapGen0_MyContainer_PoolWorker-2:Exception details: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 143, in recognize
    audio_duration = self.__recognize_in_subproc(self._filepath, json_data, cancellation_token)
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 231, in __recognize_in_subproc
    raise FailedRecognitionError(err_msg)
batchkit.utils.FailedRecognitionError: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-2__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-2

2020-10-27 19:30:44.211:WARNING:HotswapGen0_MyContainer_PoolWorker-2:FailedRecognitionError happened, must terminate the recognition.Details: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-2__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-2
2020-10-27 19:30:44.211:DEBUG:HotswapGen0_MyContainer_PoolWorker-3:Exception details: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 143, in recognize
    audio_duration = self.__recognize_in_subproc(self._filepath, json_data, cancellation_token)
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 231, in __recognize_in_subproc
    raise FailedRecognitionError(err_msg)
batchkit.utils.FailedRecognitionError: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-3__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-3

2020-10-27 19:30:44.211:WARNING:HotswapGen0_MyContainer_PoolWorker-3:FailedRecognitionError happened, must terminate the recognition.Details: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-3__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-3
2020-10-27 19:30:44.212:DEBUG:MainProcess:EndpointManager name: HotswapGen0_MyContainer  will try to steal work
2020-10-27 19:30:44.212:DEBUG:MainProcess:EndpointManager name: HotswapGen0_MyContainer  stole work and will delegate to a worker.
2020-10-27 19:30:44.212:DEBUG:MainProcess:EndpointManager name: HotswapGen0_MyContainer  will try to steal work
2020-10-27 19:30:44.212:DEBUG:MainProcess:EndpointManager name: HotswapGen0_MyContainer  stole work and will delegate to a worker.
2020-10-27 19:30:44.213:INFO:HotswapGen0_MyContainer_PoolWorker-4:Processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, with process: HotswapGen0_MyContainer_PoolWorker-4
2020-10-27 19:30:44.213:INFO:HotswapGen0_MyContainer_PoolWorker-5:Processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, with process: HotswapGen0_MyContainer_PoolWorker-5
2020-10-27 19:30:44.216:DEBUG:HotswapGen0_MyContainer_PoolWorker-4:Starting FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-4__SDKRequestChildProc
2020-10-27 19:30:44.216:DEBUG:HotswapGen0_MyContainer_PoolWorker-5:Starting FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-5__SDKRequestChildProc
2020-10-27 19:30:44.296:DEBUG:HotswapGen0_MyContainer_PoolWorker-5:Finished FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-5__SDKRequestChildProc
2020-10-27 19:30:44.296:DEBUG:HotswapGen0_MyContainer_PoolWorker-4:Finished FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-4__SDKRequestChildProc
2020-10-27 19:30:44.296:DEBUG:HotswapGen0_MyContainer_PoolWorker-4:Exception details: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 143, in recognize
    audio_duration = self.__recognize_in_subproc(self._filepath, json_data, cancellation_token)
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 231, in __recognize_in_subproc
    raise FailedRecognitionError(err_msg)
batchkit.utils.FailedRecognitionError: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-4__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-4

2020-10-27 19:30:44.296:DEBUG:HotswapGen0_MyContainer_PoolWorker-5:Exception details: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 143, in recognize
    audio_duration = self.__recognize_in_subproc(self._filepath, json_data, cancellation_token)
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 231, in __recognize_in_subproc
    raise FailedRecognitionError(err_msg)
batchkit.utils.FailedRecognitionError: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-5__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-5

2020-10-27 19:30:44.297:WARNING:HotswapGen0_MyContainer_PoolWorker-4:FailedRecognitionError happened, must terminate the recognition.Details: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-4__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-4
2020-10-27 19:30:44.297:WARNING:HotswapGen0_MyContainer_PoolWorker-5:FailedRecognitionError happened, must terminate the recognition.Details: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-5__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-5
2020-10-27 19:30:44.297:INFO:HotswapGen0_MyContainer_PoolWorker-4:Processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, with process: HotswapGen0_MyContainer_PoolWorker-4
2020-10-27 19:30:44.297:INFO:HotswapGen0_MyContainer_PoolWorker-5:Processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, with process: HotswapGen0_MyContainer_PoolWorker-5
2020-10-27 19:30:44.299:DEBUG:HotswapGen0_MyContainer_PoolWorker-5:Starting FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-5__SDKRequestChildProc
2020-10-27 19:30:44.299:DEBUG:HotswapGen0_MyContainer_PoolWorker-4:Starting FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-4__SDKRequestChildProc
2020-10-27 19:30:44.378:DEBUG:HotswapGen0_MyContainer_PoolWorker-5:Finished FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-5__SDKRequestChildProc
2020-10-27 19:30:44.378:DEBUG:HotswapGen0_MyContainer_PoolWorker-4:Finished FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-4__SDKRequestChildProc
2020-10-27 19:30:44.379:DEBUG:HotswapGen0_MyContainer_PoolWorker-5:Exception details: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 143, in recognize
    audio_duration = self.__recognize_in_subproc(self._filepath, json_data, cancellation_token)
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 231, in __recognize_in_subproc
    raise FailedRecognitionError(err_msg)
batchkit.utils.FailedRecognitionError: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-5__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-5

2020-10-27 19:30:44.379:DEBUG:HotswapGen0_MyContainer_PoolWorker-4:Exception details: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 143, in recognize
    audio_duration = self.__recognize_in_subproc(self._filepath, json_data, cancellation_token)
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 231, in __recognize_in_subproc
    raise FailedRecognitionError(err_msg)
batchkit.utils.FailedRecognitionError: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-4__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-4

2020-10-27 19:30:44.379:WARNING:HotswapGen0_MyContainer_PoolWorker-5:FailedRecognitionError happened, must terminate the recognition.Details: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-5__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-5
2020-10-27 19:30:44.379:WARNING:HotswapGen0_MyContainer_PoolWorker-4:FailedRecognitionError happened, must terminate the recognition.Details: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-4__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-4
2020-10-27 19:30:44.379:DEBUG:MainProcess:EndpointManager name: HotswapGen0_MyContainer  will try to steal work
2020-10-27 19:30:44.379:DEBUG:MainProcess:EndpointManager name: HotswapGen0_MyContainer  stole work and will delegate to a worker.
2020-10-27 19:30:44.379:DEBUG:MainProcess:EndpointManager name: HotswapGen0_MyContainer  will try to steal work
2020-10-27 19:30:44.379:CRITICAL:MainProcess:Endpoint manager HotswapGen0_MyContainer has failed 4 consecutive recognitions on endpoint 10.100.169.79:5000
2020-10-27 19:30:44.379:INFO:HotswapGen0_MyContainer_PoolWorker-2:Processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, with process: HotswapGen0_MyContainer_PoolWorker-2
2020-10-27 19:30:44.380:DEBUG:MainProcess:EndpointManager name: HotswapGen0_MyContainer  stole work and will delegate to a worker.
2020-10-27 19:30:44.380:INFO:HotswapGen0_MyContainer_PoolWorker-3:Processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, with process: HotswapGen0_MyContainer_PoolWorker-3
2020-10-27 19:30:44.381:DEBUG:HotswapGen0_MyContainer_PoolWorker-2:Starting FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-2__SDKRequestChildProc
2020-10-27 19:30:44.382:DEBUG:HotswapGen0_MyContainer_PoolWorker-3:Starting FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-3__SDKRequestChildProc
2020-10-27 19:30:44.461:DEBUG:HotswapGen0_MyContainer_PoolWorker-3:Finished FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-3__SDKRequestChildProc
2020-10-27 19:30:44.461:DEBUG:HotswapGen0_MyContainer_PoolWorker-2:Finished FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-2__SDKRequestChildProc
2020-10-27 19:30:44.461:DEBUG:HotswapGen0_MyContainer_PoolWorker-3:Exception details: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 143, in recognize
    audio_duration = self.__recognize_in_subproc(self._filepath, json_data, cancellation_token)
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 231, in __recognize_in_subproc
    raise FailedRecognitionError(err_msg)
batchkit.utils.FailedRecognitionError: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-3__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-3

2020-10-27 19:30:44.461:DEBUG:HotswapGen0_MyContainer_PoolWorker-2:Exception details: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 143, in recognize
    audio_duration = self.__recognize_in_subproc(self._filepath, json_data, cancellation_token)
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 231, in __recognize_in_subproc
    raise FailedRecognitionError(err_msg)
batchkit.utils.FailedRecognitionError: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-2__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-2

2020-10-27 19:30:44.461:WARNING:HotswapGen0_MyContainer_PoolWorker-3:FailedRecognitionError happened, must terminate the recognition.Details: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-3__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-3
2020-10-27 19:30:44.461:WARNING:HotswapGen0_MyContainer_PoolWorker-2:FailedRecognitionError happened, must terminate the recognition.Details: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-2__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-2
2020-10-27 19:30:44.461:INFO:HotswapGen0_MyContainer_PoolWorker-3:Processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, with process: HotswapGen0_MyContainer_PoolWorker-3
2020-10-27 19:30:44.461:INFO:HotswapGen0_MyContainer_PoolWorker-2:Processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, with process: HotswapGen0_MyContainer_PoolWorker-2
2020-10-27 19:30:44.463:DEBUG:HotswapGen0_MyContainer_PoolWorker-2:Starting FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-2__SDKRequestChildProc
2020-10-27 19:30:44.463:DEBUG:HotswapGen0_MyContainer_PoolWorker-3:Starting FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-3__SDKRequestChildProc
2020-10-27 19:30:44.543:DEBUG:HotswapGen0_MyContainer_PoolWorker-3:Finished FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-3__SDKRequestChildProc
2020-10-27 19:30:44.543:DEBUG:HotswapGen0_MyContainer_PoolWorker-2:Finished FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-2__SDKRequestChildProc
2020-10-27 19:30:44.543:DEBUG:HotswapGen0_MyContainer_PoolWorker-3:Exception details: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 143, in recognize
    audio_duration = self.__recognize_in_subproc(self._filepath, json_data, cancellation_token)
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 231, in __recognize_in_subproc
    raise FailedRecognitionError(err_msg)
batchkit.utils.FailedRecognitionError: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-3__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-3

2020-10-27 19:30:44.543:DEBUG:HotswapGen0_MyContainer_PoolWorker-2:Exception details: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 143, in recognize
    audio_duration = self.__recognize_in_subproc(self._filepath, json_data, cancellation_token)
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 231, in __recognize_in_subproc
    raise FailedRecognitionError(err_msg)
batchkit.utils.FailedRecognitionError: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-2__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-2

2020-10-27 19:30:44.543:WARNING:HotswapGen0_MyContainer_PoolWorker-3:FailedRecognitionError happened, must terminate the recognition.Details: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-3__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-3
2020-10-27 19:30:44.543:WARNING:HotswapGen0_MyContainer_PoolWorker-2:FailedRecognitionError happened, must terminate the recognition.Details: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-2__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-2
2020-10-27 19:30:44.543:DEBUG:MainProcess:EndpointManager name: HotswapGen0_MyContainer  will try to steal work
2020-10-27 19:30:44.543:CRITICAL:MainProcess:Endpoint manager HotswapGen0_MyContainer has failed 5 consecutive recognitions on endpoint 10.100.169.79:5000
2020-10-27 19:30:44.544:CRITICAL:MainProcess:Endpoint manager HotswapGen0_MyContainer has failed 6 consecutive recognitions on endpoint 10.100.169.79:5000
2020-10-27 19:30:44.544:DEBUG:MainProcess:EndpointManager name: HotswapGen0_MyContainer  stole work and will delegate to a worker.
2020-10-27 19:30:44.544:DEBUG:MainProcess:EndpointManager name: HotswapGen0_MyContainer  will try to steal work
2020-10-27 19:30:44.544:DEBUG:MainProcess:EndpointManager name: HotswapGen0_MyContainer  stole work and will delegate to a worker.
2020-10-27 19:30:44.544:INFO:HotswapGen0_MyContainer_PoolWorker-4:Processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, with process: HotswapGen0_MyContainer_PoolWorker-4
2020-10-27 19:30:44.544:INFO:HotswapGen0_MyContainer_PoolWorker-5:Processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, with process: HotswapGen0_MyContainer_PoolWorker-5
2020-10-27 19:30:44.546:DEBUG:HotswapGen0_MyContainer_PoolWorker-5:Starting FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-5__SDKRequestChildProc
2020-10-27 19:30:44.546:DEBUG:HotswapGen0_MyContainer_PoolWorker-4:Starting FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-4__SDKRequestChildProc
2020-10-27 19:30:44.625:DEBUG:HotswapGen0_MyContainer_PoolWorker-4:Finished FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-4__SDKRequestChildProc
2020-10-27 19:30:44.625:DEBUG:HotswapGen0_MyContainer_PoolWorker-5:Finished FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-5__SDKRequestChildProc
2020-10-27 19:30:44.625:DEBUG:HotswapGen0_MyContainer_PoolWorker-4:Exception details: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 143, in recognize
    audio_duration = self.__recognize_in_subproc(self._filepath, json_data, cancellation_token)
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 231, in __recognize_in_subproc
    raise FailedRecognitionError(err_msg)
batchkit.utils.FailedRecognitionError: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-4__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-4

2020-10-27 19:30:44.625:DEBUG:HotswapGen0_MyContainer_PoolWorker-5:Exception details: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 143, in recognize
    audio_duration = self.__recognize_in_subproc(self._filepath, json_data, cancellation_token)
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 231, in __recognize_in_subproc
    raise FailedRecognitionError(err_msg)
batchkit.utils.FailedRecognitionError: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-5__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-5

2020-10-27 19:30:44.625:WARNING:HotswapGen0_MyContainer_PoolWorker-5:FailedRecognitionError happened, must terminate the recognition.Details: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-5__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-5
2020-10-27 19:30:44.625:WARNING:HotswapGen0_MyContainer_PoolWorker-4:FailedRecognitionError happened, must terminate the recognition.Details: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-4__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-4
2020-10-27 19:30:44.626:INFO:HotswapGen0_MyContainer_PoolWorker-5:Processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, with process: HotswapGen0_MyContainer_PoolWorker-5
2020-10-27 19:30:44.626:INFO:HotswapGen0_MyContainer_PoolWorker-4:Processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, with process: HotswapGen0_MyContainer_PoolWorker-4
2020-10-27 19:30:44.627:DEBUG:HotswapGen0_MyContainer_PoolWorker-4:Starting FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-4__SDKRequestChildProc
2020-10-27 19:30:44.627:DEBUG:HotswapGen0_MyContainer_PoolWorker-5:Starting FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-5__SDKRequestChildProc
2020-10-27 19:30:44.705:DEBUG:HotswapGen0_MyContainer_PoolWorker-4:Finished FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-4__SDKRequestChildProc
2020-10-27 19:30:44.705:DEBUG:HotswapGen0_MyContainer_PoolWorker-5:Finished FileRecognizer.__recognize() in subproc: HotswapGen0_MyContainer_PoolWorker-5__SDKRequestChildProc
2020-10-27 19:30:44.706:DEBUG:HotswapGen0_MyContainer_PoolWorker-5:Exception details: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 143, in recognize
    audio_duration = self.__recognize_in_subproc(self._filepath, json_data, cancellation_token)
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 231, in __recognize_in_subproc
    raise FailedRecognitionError(err_msg)
batchkit.utils.FailedRecognitionError: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-5__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-5

2020-10-27 19:30:44.706:DEBUG:HotswapGen0_MyContainer_PoolWorker-4:Exception details: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 143, in recognize
    audio_duration = self.__recognize_in_subproc(self._filepath, json_data, cancellation_token)
  File "/usr/local/lib/python3.6/dist-packages/batchkit_examples/speech_sdk/recognize.py", line 231, in __recognize_in_subproc
    raise FailedRecognitionError(err_msg)
batchkit.utils.FailedRecognitionError: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-4__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-4

2020-10-27 19:30:44.706:WARNING:HotswapGen0_MyContainer_PoolWorker-5:FailedRecognitionError happened, must terminate the recognition.Details: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-5__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-5
2020-10-27 19:30:44.706:WARNING:HotswapGen0_MyContainer_PoolWorker-4:FailedRecognitionError happened, must terminate the recognition.Details: Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-4__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-4
2020-10-27 19:30:44.706:DEBUG:MainProcess:EndpointManager name: HotswapGen0_MyContainer  will try to steal work
2020-10-27 19:30:44.706:CRITICAL:MainProcess:Endpoint manager HotswapGen0_MyContainer has failed 7 consecutive recognitions on endpoint 10.100.169.79:5000
2020-10-27 19:30:44.706:INFO:MainProcess:Orchestrator: Completed batch 331792578788503786707517342978000830611
2020-10-27 19:30:44.706:CRITICAL:MainProcess:Endpoint manager HotswapGen0_MyContainer has failed 8 consecutive recognitions on endpoint 10.100.169.79:5000
2020-10-27 19:30:46.938:DEBUG:MainProcess:Updating singleton run_summary: /my_nfs/transcriptions/run_summary.json
2020-10-27 19:30:46.939:INFO:MainProcess:write_json_file_atomic():  Atomically wrote file /my_nfs/transcriptions/run_summary.json
2020-10-27 19:30:49.944:INFO:MainProcess:Orchestrator: Updated batch status to Done: 331792578788503786707517342978000830611
2020-10-27 19:30:49.946:INFO:MainProcess:OneShotClient:  Moving files (intermediate): /my_nfs/transcriptions/.scratch/batch_331792578788503786707517342978000830611 -> /my_nfs/transcriptions
2020-10-27 19:30:49.947:INFO:MainProcess:OneShotClient:  Moving files (final): /my_nfs/transcriptions/.scratch/batch_331792578788503786707517342978000830611 -> /my_nfs/transcriptions
2020-10-27 19:30:49.947:INFO:MainProcess:OneShotClient: Batch id: 331792578788503786707517342978000830611 is finished.
2020-10-27 19:30:49.947:INFO:MainProcess:OneShotClient:  Cleaning up scratch directory for single batch: 331792578788503786707517342978000830611
2020-10-27 19:30:49.947:INFO:MainProcess:BatchStatusProvider: rm_batch(): Removing batch_id: 331792578788503786707517342978000830611
2020-10-27 19:30:49.948:INFO:MainProcess:OneShotClient:  Requesting Orchestrator to stop.
2020-10-27 19:30:49.950:INFO:MainProcess:EndpointManager name: HotswapGen0_MyContainer  was requested to stop while stealing work.
2020-10-27 19:30:49.959:DEBUG:MainProcess:Original configuration:
2020-10-27 19:30:49.959:DEBUG:MainProcess:MyContainer: {
  "concurrency": 2,
  "host": "10.100.169.79",
  "port": 5000
}
2020-10-27 19:30:49.959:DEBUG:MainProcess:Normalized configuration:
2020-10-27 19:30:49.959:DEBUG:MainProcess:MyContainer: {
  "concurrency": 2,
  "host": "10.100.169.79",
  "isCloudService": false,
  "isSecure": false,
  "language": "en-US",
  "port": 5000,
  "rtf": 3.0,
  "startConcurrency": 2
}
2020-10-27 19:30:49.960:INFO:MainProcess:Output list:
2020-10-27 19:30:49.960:INFO:MainProcess:Result for: /my_nfs/audio_files/jurrasic1_16k_mono.wav: {
  "audio_duration": null,
  "cached": false,
  "can_retry": true,
  "endpoint": "ws://10.100.169.79:5000",
  "error_type": "FailedRecognitionError",
  "failed_reason": "Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-4__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/jurrasic1_16k_mono.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-4",
  "filepath": "/my_nfs/audio_files/jurrasic1_16k_mono.wav",
  "latency": 0.08022069931030273,
  "passed": false,
  "retries": 11,
  "thread": "HotswapGen0_MyContainer_PoolWorker-4"
}
2020-10-27 19:30:49.960:INFO:MainProcess:Result for: /my_nfs/audio_files/sotl1.wav: {
  "audio_duration": null,
  "cached": false,
  "can_retry": true,
  "endpoint": "ws://10.100.169.79:5000",
  "error_type": "FailedRecognitionError",
  "failed_reason": "Terminating signum: 11 was received in sdk subproc: HotswapGen0_MyContainer_PoolWorker-5__SDKRequestChildProcwhile processing file: /my_nfs/audio_files/sotl1.wav on endpoint: ws://10.100.169.79:5000, caught by supervisor: HotswapGen0_MyContainer_PoolWorker-5",
  "filepath": "/my_nfs/audio_files/sotl1.wav",
  "latency": 0.0802311897277832,
  "passed": false,
  "retries": 11,
  "thread": "HotswapGen0_MyContainer_PoolWorker-5"
}
2020-10-27 19:30:49.960:INFO:MainProcess:Decoded 0.00 seconds in 6.48 seconds on 1 decoders for a ratio of 0.00
2020-10-27 19:30:49.960:INFO:MainProcess:Processed 2/2 files, 0 passed, 0 fetched from cache and 2 failed.
2020-10-27 19:30:49.960:INFO:MainProcess:Total retries: 22
2020-10-27 19:30:49.960:INFO:MainProcess:
Failed recognitions (2):

2020-10-27 19:30:49.960:INFO:MainProcess: END POINT                     AUDIO FILE
2020-10-27 19:30:49.960:INFO:MainProcess: ws://10.100.169.79:5000       /my_nfs/audio_files/jurrasic1_16k_mono.wav
2020-10-27 19:30:49.961:INFO:MainProcess: ws://10.100.169.79:5000       /my_nfs/audio_files/sotl1.wav
2020-10-27 19:30:49.961:WARNING:MainProcess:Overall # files failed: 2
2020-10-27 19:30:49.961:INFO:MainProcess:client.py: run(): Finished!
Killed:  18
Killed:  19
Killed:  20
Killed:  21

Consider changing the name of the repo to something that identifies that it's related to Azure Cognitive Service Speech

batch processing kit is very broad in the context of Microsoft.

Add output-folder functionality also to APISERVER mode

BatchKit currently only has output-folder implemented when used in Oneshot or Daemon mode, but not in APISERVER mode.

Are there any plans to also add to the rest mode?

Would a PR for that topic be welcomed?

Cheers

Tasks

Beta Give feedback

No tasks being tracked yet.

Options

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.