Git Product home page Git Product logo

abhisrn1986 / twitterdatapipeline Goto Github PK

View Code? Open in Web Editor NEW
1.0 1.0 0.0 56 KB

The project comprises a real-time tweets data pipeline, a sentimental analysis of the tweets module, and a Slack bot to post the tweets' sentiments. The project uses SentimentIntensityAnalyzer from the VaderSentiment library. The analyzer gives positive, negative, and compound scores for small texts (such as tweets in this case). The real-time data pipeline flow is as follows: 1.Tweets are collected and stored in a database. 2.The sentiment of the tweets is analyzed. 3.The tweet sentiment is posted on a Slack channel using a Slack bot.

Dockerfile 3.16% Python 89.01% Shell 7.83%
data-pipeline data-science docker docker-compose machine-learning-engineering machine-learning-operations mongodb pipeline python replicaset

twitterdatapipeline's Introduction

Twitter Data Pipeline

Overview

The project comprises a real-time tweets data pipeline, a tweets sentiment analyzer module, and a Slack bot to post the tweets' sentiments. The project uses SentimentIntensityAnalyzer from the VaderSentiment library. The analyzer gives positive, negative, and compound scores for small texts (such as tweets in this case).

The real-time data pipeline flow is as follows:

  1. Tweets are collected and stored in a database.
  2. The sentiment of the tweets is analyzed.
  3. The tweet sentiment is posted on a Slack channel using a Slack bot.

Docker container collects tweets in real-time based on a particular query using the streaming functionality in Tweepy API. This docker container stores the tweets in MongoDB. As soon as the program inserts new tweets in the database, it sends these tweets to another docker container. This docker container performs a sentimental analysis of the tweets and posts the results in a slack channel using the Slack bot. The tweets collection and storage are independent of the sentimental analysis and Slack posting.

Features

  1. Streams tweets based on search queries using the Tweepy API.
  2. Stores tweets dictionary in a MongoDB database.
  3. Tweets' insertion in the tweets storage database is detected using MongoDB replica sets, which provide real-time sentiment analysis and posting of tweets in slack.
  4. Two docker containers keep tweets collection and storage independent of the tweet's sentiment analysis and slack posting job.
  5. Tweets' sentiments are analyzed using the Vader sentiment analysis.
  6. Sentiments of the tweets' are posted in a Slack channel using a Slack bot.
  7. Multiple Docker containers are defined and run using Docker compose tool.

Technology Stack

How to Run Locally

I have tested the project on Ubuntu 20.04 machines (one with 2GB and another with 8GB RAM) with the default settings by folllowing the instructions given below. For running on other Ubuntu versions with docker installed (or Debian OSs), use more or less the below instructions.

  1. git clone https://github.com/abhisrn1986/TwitterDataPipeline.git.

  2. Install docker from instructions here.

  3. To avoid prefacing the docker command with sudo, follow the instructions here.

  4. Install docker-compose executing the command sudo apt install docker-compose in a shell.

  5. Sign up for a Twitter developer account with elevated access (maximum limit of 2 million tweet pulls per month) if you don't have one. It is mandatory to run this project (as Tweepy API Stream functionality is accessible with elevated access account). Here is a link on how to get started with Twitter API.

  6. Create a slack bot to post tweets in a slack channel (see the section below for instructions)

  7. Create a .env file in the root directory (TwitterDataPipeline) consisting of the credentials of Twitter and slack channel webhook in the following format:

    CUSTOMER_KEY=key1
CUSTOMER_SECRET_KEY=key2
ACCESS_TOKEN=token1
ACCESS_TOKEN_SECRET=token2
BEARER_TOKEN=token3
SLACK_WEBHOOK=url

    Note: Replace the text after = in each line with appropriate values (such as the customer keys, access tokens, slack webhook URL, and the query) and remember there shouldn't be any space after =.

  8. Build all the docker containers by running docker-compose build in the terminal within the directory TwitterDataPipeline.

  9. To run the pipeline execute bash script run_pipeline.sh as run_pipeline.sh -q "query1;query2". -q option is mandatory to provide queries. Users can provide as many queries as possible with queries separated by semicolons. For instance, to stream all tweets related to China and Germany, run the command run_pipeline.sh -q "China;Germany" and if the configuration is correct, this should post tweets with sentiment score in the slack channel used.

How to build a slack bot

  1. Login and go to Your Apps.
  2. Choose to Create New App.
  3. Choose the option From scratch.
  4. Fill in a name and choose your slack workspace as Development Slack Workspace.
  5. Press Create App.
  6. Under "Add features and functionality" click on "Incoming Webhooks".
  7. Activate incoming webhooks by clicking on the switch.
  8. Click on Add new webhook to the workspace at the bottom of the page.
  9. Select a channel where you want to post messages and click on Allow.
  10. Scroll down and copy the Webhook URL into the code.

Note: More info can be found here

twitterdatapipeline's People

Contributors

abhisrn1986 avatar pawlodkowski avatar spicedacademy avatar

Stargazers

avatar

Watchers

avatar

twitterdatapipeline's Issues

Complete tweet text is not extracted

Sometimes the complete text of the tweet is not extracted. For instance tweet text comprises of ... in the end if the text is too long.

For instance:
Run! The door of China is closing/The government blocks the outflow of capital...

Run error asks how to solve

root@instance-20220525-0117:~/TwitterDataPipeline# bash run_pipeline.sh -q "China;Germany"
Creating twitterdatapipeline_mongodb_1 ... done
Creating twitterdatapipeline_tweet_collector_1 ... done
Creating twitterdatapipeline_etl_job_1 ... done
Attaching to twitterdatapipeline_mongodb_1, twitterdatapipeline_tweet_collector_1, twitterdatapipeline_etl_job_1
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:42.665+00:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"-","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:42.672+00:00"},"s":"I", "c":"NETWORK", "id":4915701, "ctx":"-","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":13},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":13},"outgoing":{"minWireVersion":0,"maxWireVersion":13},"isInternalClient":true}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:42.677+00:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:42.677+00:00"},"s":"I", "c":"NETWORK", "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:42.686+00:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:42.687+00:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:42.688+00:00"},"s":"I", "c":"REPL", "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","ns":"config.tenantMigrationDonors"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:42.688+00:00"},"s":"I", "c":"REPL", "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","ns":"config.tenantMigrationRecipients"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:42.688+00:00"},"s":"I", "c":"CONTROL", "id":5945603, "ctx":"main","msg":"Multi threading initialized"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:42.689+00:00"},"s":"I", "c":"CONTROL", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"ee808c9c60e1"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:42.689+00:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"5.0.9","gitVersion":"6f7dae919422dcd7f4892c10ff20cdc721ad00e6","openSSLVersion":"OpenSSL 1.1.1f 31 Mar 2020","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2004","distarch":"aarch64","target_arch":"aarch64"}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:42.689+00:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:42.689+00:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"},"replication":{"replSet":"dbrs"}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:42.701+00:00"},"s":"I", "c":"STORAGE", "id":22297, "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:42.701+00:00"},"s":"I", "c":"STORAGE", "id":22315, "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=11482M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.355+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1659056983:355752][1:0xffff803ef980], txn-recover: [WT_VERB_RECOVERY_ALL] Set global recovery timestamp: (0, 0)"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.355+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1659056983:355816][1:0xffff803ef980], txn-recover: [WT_VERB_RECOVERY_ALL] Set global oldest timestamp: (0, 0)"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.365+00:00"},"s":"I", "c":"STORAGE", "id":4795906, "ctx":"initandlisten","msg":"WiredTiger opened","attr":{"durationMillis":664}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.365+00:00"},"s":"I", "c":"RECOVERY", "id":23987, "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","attr":{"recoveryTimestamp":{"$timestamp":{"t":0,"i":0}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.383+00:00"},"s":"I", "c":"STORAGE", "id":4366408, "ctx":"initandlisten","msg":"No table logging settings modifications are required for existing WiredTiger tables","attr":{"loggingEnabled":false}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.384+00:00"},"s":"I", "c":"STORAGE", "id":22262, "ctx":"initandlisten","msg":"Timestamp monitor starting"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.389+00:00"},"s":"W", "c":"CONTROL", "id":22120, "ctx":"initandlisten","msg":"Access control is not enabled for the database. Read and write access to data and configuration is unrestricted","tags":["startupWarnings"]}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.389+00:00"},"s":"W", "c":"CONTROL", "id":22138, "ctx":"initandlisten","msg":"You are running this process as the root user, which is not recommended","tags":["startupWarnings"]}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.402+00:00"},"s":"I", "c":"STORAGE", "id":5071100, "ctx":"initandlisten","msg":"Clearing temp directory"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.402+00:00"},"s":"I", "c":"CONTROL", "id":20536, "ctx":"initandlisten","msg":"Flow Control is enabled on this deployment"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.413+00:00"},"s":"I", "c":"SHARDING", "id":20997, "ctx":"initandlisten","msg":"Refreshed RWC defaults","attr":{"newDefaults":{}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.414+00:00"},"s":"I", "c":"FTDC", "id":20625, "ctx":"initandlisten","msg":"Initializing full-time diagnostic data capture","attr":{"dataDirectory":"/data/db/diagnostic.data"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.414+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"initandlisten","msg":"createCollection","attr":{"namespace":"local.startup_log","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"8055d675-b916-4dec-8c56-9648ff33830e"}},"options":{"capped":true,"size":10485760}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.430+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"initandlisten","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"local.startup_log","index":"id","commitTimestamp":null}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.432+00:00"},"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ReadConcernMajorityNotAvailableYet: Read concern majority reads are currently not possible.","nextWakeupMillis":200}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.433+00:00"},"s":"I", "c":"REPL", "id":6015317, "ctx":"initandlisten","msg":"Setting new configuration state","attr":{"newState":"ConfigStartingUp","oldState":"ConfigPreStart"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.435+00:00"},"s":"I", "c":"REPL", "id":4280500, "ctx":"initandlisten","msg":"Attempting to create internal replication collections"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.436+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"initandlisten","msg":"createCollection","attr":{"namespace":"local.replset.oplogTruncateAfterPoint","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"7bba52e2-de5b-4d32-97f4-bfa1c7dd335f"}},"options":{}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.465+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"initandlisten","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"local.replset.oplogTruncateAfterPoint","index":"id","commitTimestamp":null}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.465+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"initandlisten","msg":"createCollection","attr":{"namespace":"local.replset.minvalid","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"4fb77728-b5a9-404e-9597-3bf424818b18"}},"options":{}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.477+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"initandlisten","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"local.replset.minvalid","index":"id","commitTimestamp":null}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.480+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"initandlisten","msg":"createCollection","attr":{"namespace":"local.replset.election","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"ce555ed9-1c10-4f8a-ad6e-7a8591a923b3"}},"options":{}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.493+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"initandlisten","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"local.replset.election","index":"id","commitTimestamp":null}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.493+00:00"},"s":"I", "c":"REPL", "id":4280501, "ctx":"initandlisten","msg":"Attempting to load local voted for document"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.493+00:00"},"s":"I", "c":"REPL", "id":21311, "ctx":"initandlisten","msg":"Did not find local initialized voted for document at startup"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.493+00:00"},"s":"I", "c":"REPL", "id":4280502, "ctx":"initandlisten","msg":"Searching for local Rollback ID document"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.493+00:00"},"s":"I", "c":"REPL", "id":21312, "ctx":"initandlisten","msg":"Did not find local Rollback ID document at startup. Creating one"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.493+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"initandlisten","msg":"createCollection","attr":{"namespace":"local.system.rollback.id","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"11b753c6-33d5-4c27-b95c-d2b975382fe0"}},"options":{}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.506+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"initandlisten","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"local.system.rollback.id","index":"id","commitTimestamp":null}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.506+00:00"},"s":"I", "c":"REPL", "id":21531, "ctx":"initandlisten","msg":"Initialized the rollback ID","attr":{"rbid":1}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.506+00:00"},"s":"I", "c":"REPL", "id":21313, "ctx":"initandlisten","msg":"Did not find local replica set configuration document at startup","attr":{"error":{"code":47,"codeName":"NoMatchingDocument","errmsg":"Did not find replica set configuration document in local.system.replset"}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.506+00:00"},"s":"I", "c":"REPL", "id":6015317, "ctx":"initandlisten","msg":"Setting new configuration state","attr":{"newState":"ConfigUninitialized","oldState":"ConfigStartingUp"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.507+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"initandlisten","msg":"createCollection","attr":{"namespace":"local.system.views","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"cd8215e2-bc33-4123-b284-fd33fd2904b9"}},"options":{}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.519+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"initandlisten","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"local.system.views","index":"id","commitTimestamp":null}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.523+00:00"},"s":"I", "c":"CONTROL", "id":20714, "ctx":"LogicalSessionCacheRefresh","msg":"Failed to refresh session cache, will try again at the next refresh interval","attr":{"error":"NotYetInitialized: Replication has not yet been configured"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.523+00:00"},"s":"I", "c":"REPL", "id":40440, "ctx":"initandlisten","msg":"Starting the TopologyVersionObserver"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.524+00:00"},"s":"I", "c":"REPL", "id":40445, "ctx":"TopologyVersionObserver","msg":"Started TopologyVersionObserver"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.524+00:00"},"s":"I", "c":"CONTROL", "id":20712, "ctx":"LogicalSessionCacheReap","msg":"Sessions collection is not set up; waiting until next sessions reap interval","attr":{"error":"NamespaceNotFound: config.system.sessions does not exist"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.525+00:00"},"s":"I", "c":"NETWORK", "id":23015, "ctx":"listener","msg":"Listening on","attr":{"address":"/tmp/mongodb-27017.sock"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.525+00:00"},"s":"I", "c":"NETWORK", "id":23015, "ctx":"listener","msg":"Listening on","attr":{"address":"0.0.0.0"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.525+00:00"},"s":"I", "c":"NETWORK", "id":23016, "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.632+00:00"},"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"NotYetInitialized: Cannot use non-local read concern until replica set is finished initializing.","nextWakeupMillis":400}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.974+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.18.0.4:38638","uuid":"5c9b9211-6cdc-43db-933d-01ce307381c7","connectionId":1,"connectionCount":1}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.975+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn1","msg":"client metadata","attr":{"remote":"172.18.0.4:38638","client":"conn1","doc":{"driver":{"name":"PyMongo","version":"4.1.1"},"os":{"type":"Linux","name":"Linux","architecture":"aarch64","version":"5.15.0-1013-oracle"},"platform":"CPython 3.8.13.final.0"}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.976+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.18.0.4:38640","uuid":"e632c54f-612a-4932-8041-ae8c5cde94c7","connectionId":2,"connectionCount":2}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:43.977+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2","msg":"client metadata","attr":{"remote":"172.18.0.4:38640","client":"conn2","doc":{"driver":{"name":"PyMongo","version":"4.1.1"},"os":{"type":"Linux","name":"Linux","architecture":"aarch64","version":"5.15.0-1013-oracle"},"platform":"CPython 3.8.13.final.0"}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:44.033+00:00"},"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"NotYetInitialized: Cannot use non-local read concern until replica set is finished initializing.","nextWakeupMillis":600}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:44.633+00:00"},"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"NotYetInitialized: Cannot use non-local read concern until replica set is finished initializing.","nextWakeupMillis":800}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:45.434+00:00"},"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"NotYetInitialized: Cannot use non-local read concern until replica set is finished initializing.","nextWakeupMillis":1000}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:46.435+00:00"},"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"NotYetInitialized: Cannot use non-local read concern until replica set is finished initializing.","nextWakeupMillis":1200}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:47.637+00:00"},"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"NotYetInitialized: Cannot use non-local read concern until replica set is finished initializing.","nextWakeupMillis":1400}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:49.039+00:00"},"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"NotYetInitialized: Cannot use non-local read concern until replica set is finished initializing.","nextWakeupMillis":1600}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:50.641+00:00"},"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"NotYetInitialized: Cannot use non-local read concern until replica set is finished initializing.","nextWakeupMillis":1800}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:52.441+00:00"},"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"NotYetInitialized: Cannot use non-local read concern until replica set is finished initializing.","nextWakeupMillis":2000}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.413+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.18.0.3:57924","uuid":"93c2aa2f-4fca-45f8-8ffe-ec8fa2a40734","connectionId":3,"connectionCount":3}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.414+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn3","msg":"client metadata","attr":{"remote":"172.18.0.3:57924","client":"conn3","doc":{"driver":{"name":"PyMongo","version":"4.1.1"},"os":{"type":"Linux","name":"Linux","architecture":"aarch64","version":"5.15.0-1013-oracle"},"platform":"CPython 3.8.13.final.0"}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.416+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.18.0.3:57926","uuid":"02f8c689-6e78-4336-9269-2a070ac2b3e4","connectionId":4,"connectionCount":4}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.416+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.18.0.3:57928","uuid":"af2490d6-ffe5-4c1c-b62c-1bdbfe9d356c","connectionId":5,"connectionCount":5}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.417+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn4","msg":"client metadata","attr":{"remote":"172.18.0.3:57926","client":"conn4","doc":{"driver":{"name":"PyMongo","version":"4.1.1"},"os":{"type":"Linux","name":"Linux","architecture":"aarch64","version":"5.15.0-1013-oracle"},"platform":"CPython 3.8.13.final.0"}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.417+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn5","msg":"client metadata","attr":{"remote":"172.18.0.3:57928","client":"conn5","doc":{"driver":{"name":"PyMongo","version":"4.1.1"},"os":{"type":"Linux","name":"Linux","architecture":"aarch64","version":"5.15.0-1013-oracle"},"platform":"CPython 3.8.13.final.0"}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.419+00:00"},"s":"I", "c":"REPL", "id":21356, "ctx":"conn5","msg":"replSetInitiate admin command received from client"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.419+00:00"},"s":"I", "c":"REPL", "id":6015317, "ctx":"conn5","msg":"Setting new configuration state","attr":{"newState":"ConfigInitiating","oldState":"ConfigUninitialized"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.419+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"conn5","msg":"createCollection","attr":{"namespace":"admin.system.version","uuidDisposition":"provided","uuid":{"uuid":{"$uuid":"4e8db18f-f43a-48f4-814b-b76a4a6bac38"}},"options":{"uuid":{"$uuid":"4e8db18f-f43a-48f4-814b-b76a4a6bac38"}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.430+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"conn5","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"admin.system.version","index":"id","commitTimestamp":null}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.430+00:00"},"s":"I", "c":"REPL", "id":20459, "ctx":"conn5","msg":"Setting featureCompatibilityVersion","attr":{"newVersion":"5.0"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.430+00:00"},"s":"I", "c":"NETWORK", "id":4915702, "ctx":"conn5","msg":"Updated wire specification","attr":{"oldSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":13},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":13},"outgoing":{"minWireVersion":0,"maxWireVersion":13},"isInternalClient":true},"newSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":13},"incomingInternalClient":{"minWireVersion":13,"maxWireVersion":13},"outgoing":{"minWireVersion":13,"maxWireVersion":13},"isInternalClient":true}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.430+00:00"},"s":"I", "c":"NETWORK", "id":22991, "ctx":"conn5","msg":"Skip closing connection for connection","attr":{"connectionId":5}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.430+00:00"},"s":"I", "c":"NETWORK", "id":22991, "ctx":"conn5","msg":"Skip closing connection for connection","attr":{"connectionId":4}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.430+00:00"},"s":"I", "c":"NETWORK", "id":22991, "ctx":"conn5","msg":"Skip closing connection for connection","attr":{"connectionId":3}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.430+00:00"},"s":"I", "c":"NETWORK", "id":22991, "ctx":"conn5","msg":"Skip closing connection for connection","attr":{"connectionId":2}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.430+00:00"},"s":"I", "c":"NETWORK", "id":22991, "ctx":"conn5","msg":"Skip closing connection for connection","attr":{"connectionId":1}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.439+00:00"},"s":"I", "c":"REPL", "id":21357, "ctx":"conn5","msg":"replSetInitiate config object parses ok","attr":{"numMembers":1}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.439+00:00"},"s":"I", "c":"REPL", "id":21251, "ctx":"conn5","msg":"Creating replication oplog","attr":{"oplogSizeMB":2169}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.439+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"conn5","msg":"createCollection","attr":{"namespace":"local.oplog.rs","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"1d1c843f-12dc-4756-8d14-944b1c1b883c"}},"options":{"capped":true,"size":2275157811,"autoIndexId":false}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.446+00:00"},"s":"I", "c":"STORAGE", "id":22383, "ctx":"conn5","msg":"The size storer reports that the oplog contains","attr":{"numRecords":0,"dataSize":0}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.446+00:00"},"s":"I", "c":"STORAGE", "id":22382, "ctx":"conn5","msg":"WiredTiger record store oplog processing finished","attr":{"durationMillis":0}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.449+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"conn5","msg":"WiredTiger message","attr":{"message":"[1659056993:449901][1:0xffff67fb9d00], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 68, snapshot max: 68 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 1"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.485+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"conn5","msg":"createCollection","attr":{"namespace":"local.system.replset","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"a9bb0921-cab5-42a5-af88-a761c9cb3e34"}},"options":{}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.496+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"conn5","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"local.system.replset","index":"id","commitTimestamp":{"$timestamp":{"t":1659056993,"i":1}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.497+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"conn5","msg":"WiredTiger message","attr":{"message":"[1659056993:497798][1:0xffff67fb9d00], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 77, snapshot max: 77 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 1"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.512+00:00"},"s":"I", "c":"REPL", "id":5872101, "ctx":"conn5","msg":"Taking a stable checkpoint for replSetInitiate"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.512+00:00"},"s":"I", "c":"REPL", "id":5872100, "ctx":"conn5","msg":"Updating commit point for initiate","attr":{"_lastCommittedOpTimeAndWallTime":"{ ts: Timestamp(1659056993, 1), t: -1 }, 2022-07-29T01:09:53.496+00:00"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.512+00:00"},"s":"I", "c":"STORAGE", "id":22310, "ctx":"conn5","msg":"Triggering the first stable checkpoint","attr":{"initialDataTimestamp":{"$timestamp":{"t":1659056993,"i":1}},"prevStableTimestamp":{"$timestamp":{"t":0,"i":0}},"currStableTimestamp":{"$timestamp":{"t":1659056993,"i":1}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.512+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"conn5","msg":"WiredTiger message","attr":{"message":"[1659056993:512607][1:0xffff67fb9d00], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 79, snapshot max: 79 snapshot count: 0, oldest timestamp: (1659056993, 1) , meta checkpoint timestamp: (1659056993, 1) base write gen: 1"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.519+00:00"},"s":"I", "c":"REPL", "id":6015317, "ctx":"conn5","msg":"Setting new configuration state","attr":{"newState":"ConfigSteady","oldState":"ConfigInitiating"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.519+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":"[1659056993:519878][1:0xffff77bdcd00], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 81, snapshot max: 81 snapshot count: 0, oldest timestamp: (1659056993, 1) , meta checkpoint timestamp: (1659056993, 1) base write gen: 1"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.519+00:00"},"s":"I", "c":"REPL", "id":21392, "ctx":"conn5","msg":"New replica set config in use","attr":{"config":{"_id":"dbrs","version":1,"term":0,"members":[{"_id":0,"host":"mongodb:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"tags":{},"secondaryDelaySecs":0,"votes":1}],"protocolVersion":1,"writeConcernMajorityJournalDefault":true,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"catchUpTakeoverDelayMillis":30000,"getLastErrorModes":{},"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":{"$oid":"62e33361d4cecc23691951e4"}}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.519+00:00"},"s":"I", "c":"REPL", "id":21393, "ctx":"conn5","msg":"Found self in config","attr":{"hostAndPort":"mongodb:27017"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.520+00:00"},"s":"I", "c":"REPL", "id":21358, "ctx":"conn5","msg":"Replica set state transition","attr":{"newState":"STARTUP2","oldState":"STARTUP"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.520+00:00"},"s":"I", "c":"REPL", "id":21306, "ctx":"conn5","msg":"Starting replication storage threads"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.520+00:00"},"s":"I", "c":"REPL", "id":4280512, "ctx":"conn5","msg":"No initial sync required. Attempting to begin steady replication"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.520+00:00"},"s":"I", "c":"REPL", "id":21358, "ctx":"conn5","msg":"Replica set state transition","attr":{"newState":"RECOVERING","oldState":"STARTUP2"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.520+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"conn5","msg":"createCollection","attr":{"namespace":"local.replset.initialSyncId","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"64aad0d3-1978-431c-9fce-1ced5d8d5f4b"}},"options":{}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.533+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"conn5","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"local.replset.initialSyncId","index":"id","commitTimestamp":null}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.533+00:00"},"s":"I", "c":"REPL", "id":21299, "ctx":"conn5","msg":"Starting replication fetcher thread"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.533+00:00"},"s":"I", "c":"REPL", "id":21300, "ctx":"conn5","msg":"Starting replication applier thread"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.534+00:00"},"s":"I", "c":"REPL", "id":21301, "ctx":"conn5","msg":"Starting replication reporter thread"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.534+00:00"},"s":"I", "c":"REPL", "id":21224, "ctx":"OplogApplier-0","msg":"Starting oplog application"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.534+00:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn5","msg":"Slow query","attr":{"type":"command","ns":"local.system.replset","command":{"replSetInitiate":{"_id":"dbrs","members":[{"_id":0,"host":"mongodb:27017"}]},"$db":"admin","$readPreference":{"mode":"primaryPreferred"}},"numYields":0,"reslen":38,"locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":15}},"ReplicationStateTransition":{"acquireCount":{"w":16}},"Global":{"acquireCount":{"r":8,"w":6,"W":2}},"Database":{"acquireCount":{"r":6,"w":6,"R":1}},"Collection":{"acquireCount":{"r":2,"w":5}},"Mutex":{"acquireCount":{"r":12}},"oplog":{"acquireCount":{"w":1}}},"flowControl":{"acquireCount":5,"timeAcquiringMicros":6},"storage":{},"remote":"172.18.0.3:57928","protocol":"op_msg","durationMillis":115}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.534+00:00"},"s":"I", "c":"REPL", "id":21358, "ctx":"OplogApplier-0","msg":"Replica set state transition","attr":{"newState":"SECONDARY","oldState":"RECOVERING"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.534+00:00"},"s":"I", "c":"ELECTION", "id":4615652, "ctx":"OplogApplier-0","msg":"Starting an election, since we've seen no PRIMARY in election timeout period","attr":{"electionTimeoutPeriodMillis":10000}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.535+00:00"},"s":"I", "c":"ELECTION", "id":21438, "ctx":"OplogApplier-0","msg":"Conducting a dry run election to see if we could be elected","attr":{"currentTerm":0}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.536+00:00"},"s":"I", "c":"ELECTION", "id":21444, "ctx":"ReplCoord-0","msg":"Dry election run succeeded, running for election","attr":{"newTerm":1}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.536+00:00"},"s":"I", "c":"ELECTION", "id":6015300, "ctx":"ReplCoord-1","msg":"Storing last vote document in local storage for my election","attr":{"lastVote":{"term":1,"candidateIndex":0}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.537+00:00"},"s":"I", "c":"ELECTION", "id":21450, "ctx":"ReplCoord-1","msg":"Election succeeded, assuming primary role","attr":{"term":1}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.537+00:00"},"s":"I", "c":"REPL", "id":21358, "ctx":"ReplCoord-1","msg":"Replica set state transition","attr":{"newState":"PRIMARY","oldState":"SECONDARY"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.537+00:00"},"s":"I", "c":"REPL", "id":21106, "ctx":"ReplCoord-1","msg":"Resetting sync source to empty","attr":{"previousSyncSource":":27017"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.537+00:00"},"s":"I", "c":"REPL", "id":21359, "ctx":"ReplCoord-1","msg":"Entering primary catch-up mode"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.537+00:00"},"s":"I", "c":"REPL", "id":6015304, "ctx":"ReplCoord-1","msg":"Skipping primary catchup since we are the only node in the replica set."}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.537+00:00"},"s":"I", "c":"REPL", "id":21363, "ctx":"ReplCoord-1","msg":"Exited primary catch-up mode"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.537+00:00"},"s":"I", "c":"REPL", "id":21107, "ctx":"ReplCoord-1","msg":"Stopping replication producer"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.537+00:00"},"s":"I", "c":"REPL", "id":21239, "ctx":"ReplBatcher","msg":"Oplog buffer has been drained","attr":{"term":1}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.538+00:00"},"s":"I", "c":"REPL", "id":21343, "ctx":"RstlKillOpThread","msg":"Starting to kill user operations"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.538+00:00"},"s":"I", "c":"REPL", "id":21344, "ctx":"RstlKillOpThread","msg":"Stopped killing user operations"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.538+00:00"},"s":"I", "c":"REPL", "id":21340, "ctx":"RstlKillOpThread","msg":"State transition ops metrics","attr":{"metrics":{"lastStateTransition":"stepUp","userOpsKilled":0,"userOpsRunning":2}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.539+00:00"},"s":"I", "c":"REPL", "id":4508103, "ctx":"OplogApplier-0","msg":"Increment the config term via reconfig"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.539+00:00"},"s":"I", "c":"REPL", "id":6015313, "ctx":"OplogApplier-0","msg":"Replication config state is Steady, starting reconfig"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.539+00:00"},"s":"I", "c":"REPL", "id":6015317, "ctx":"OplogApplier-0","msg":"Setting new configuration state","attr":{"newState":"ConfigReconfiguring","oldState":"ConfigSteady"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.539+00:00"},"s":"I", "c":"REPL", "id":21353, "ctx":"OplogApplier-0","msg":"replSetReconfig config object parses ok","attr":{"numMembers":1}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.539+00:00"},"s":"I", "c":"REPL", "id":51814, "ctx":"OplogApplier-0","msg":"Persisting new config to disk"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.540+00:00"},"s":"I", "c":"REPL", "id":6015315, "ctx":"OplogApplier-0","msg":"Persisted new config to disk"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.540+00:00"},"s":"I", "c":"REPL", "id":6015317, "ctx":"OplogApplier-0","msg":"Setting new configuration state","attr":{"newState":"ConfigSteady","oldState":"ConfigReconfiguring"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.540+00:00"},"s":"I", "c":"REPL", "id":21392, "ctx":"OplogApplier-0","msg":"New replica set config in use","attr":{"config":{"_id":"dbrs","version":1,"term":1,"members":[{"_id":0,"host":"mongodb:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"tags":{},"secondaryDelaySecs":0,"votes":1}],"protocolVersion":1,"writeConcernMajorityJournalDefault":true,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"catchUpTakeoverDelayMillis":30000,"getLastErrorModes":{},"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":{"$oid":"62e33361d4cecc23691951e4"}}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.540+00:00"},"s":"I", "c":"REPL", "id":21393, "ctx":"OplogApplier-0","msg":"Found self in config","attr":{"hostAndPort":"mongodb:27017"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.540+00:00"},"s":"I", "c":"REPL", "id":6015310, "ctx":"OplogApplier-0","msg":"Starting to transition to primary."}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.540+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"OplogApplier-0","msg":"createCollection","attr":{"namespace":"config.transactions","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"04b128b0-c970-4ec6-8e49-754a8d2c29e5"}},"options":{}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.552+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"OplogApplier-0","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"config.transactions","index":"id","commitTimestamp":{"$timestamp":{"t":1659056993,"i":3}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.552+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"OplogApplier-0","msg":"createCollection","attr":{"namespace":"config.image_collection","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"5458432b-a9f7-4dc7-9f41-7b0f4a364e8d"}},"options":{}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.563+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"OplogApplier-0","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"config.image_collection","index":"id","commitTimestamp":{"$timestamp":{"t":1659056993,"i":4}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.564+00:00"},"s":"I", "c":"REPL", "id":6015309, "ctx":"OplogApplier-0","msg":"Logging transition to primary to oplog on stepup"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.564+00:00"},"s":"I", "c":"STORAGE", "id":20657, "ctx":"OplogApplier-0","msg":"IndexBuildsCoordinator::onStepUp - this node is stepping up to primary"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.564+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"OplogApplier-0","msg":"createCollection","attr":{"namespace":"config.system.indexBuilds","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"4f9d5031-c889-464e-af8e-05236a57fe63"}},"options":{}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.575+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"OplogApplier-0","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"config.system.indexBuilds","index":"id","commitTimestamp":{"$timestamp":{"t":1659056993,"i":6}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.576+00:00"},"s":"I", "c":"REPL", "id":21331, "ctx":"OplogApplier-0","msg":"Transition to primary complete; database writes are now permitted"}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.577+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"monitoring-keys-for-HMAC","msg":"createCollection","attr":{"namespace":"admin.system.keys","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"2ded9046-d9d5-44ea-9e71-6b586e5660cc"}},"options":{}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.578+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.18.0.4:38642","uuid":"4ebe1a97-7966-4506-8887-ec52de9d3786","connectionId":6,"connectionCount":6}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.579+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"TenantMigrationRecipientService-0","msg":"createCollection","attr":{"namespace":"config.tenantMigrationRecipients","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"3d3039f5-6b26-47bb-95ec-4282e309ddf1"}},"options":{}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.579+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"TenantMigrationDonorService-0","msg":"createCollection","attr":{"namespace":"config.tenantMigrationDonors","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"4a9832f6-132b-474e-bd93-6c49ee8d695f"}},"options":{}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.580+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn6","msg":"client metadata","attr":{"remote":"172.18.0.4:38642","client":"conn6","doc":{"driver":{"name":"PyMongo","version":"4.1.1"},"os":{"type":"Linux","name":"Linux","architecture":"aarch64","version":"5.15.0-1013-oracle"},"platform":"CPython 3.8.13.final.0"}}}
tweet_collector_1 | Stream encountered HTTP error: 403
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.600+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"monitoring-keys-for-HMAC","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"admin.system.keys","index":"id","commitTimestamp":{"$timestamp":{"t":1659056993,"i":7}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.616+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"TenantMigrationRecipientService-0","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"config.tenantMigrationRecipients","index":"id","commitTimestamp":{"$timestamp":{"t":1659056993,"i":11}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.616+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"TenantMigrationRecipientService-0","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"config.tenantMigrationRecipients","index":"TenantMigrationRecipientTTLIndex","commitTimestamp":{"$timestamp":{"t":1659056993,"i":11}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.616+00:00"},"s":"I", "c":"REPL", "id":5123005, "ctx":"TenantMigrationRecipientService-1","msg":"Rebuilding PrimaryOnlyService due to stepUp","attr":{"service":"TenantMigrationRecipientService"}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.622+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"TenantMigrationDonorService-0","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"config.tenantMigrationDonors","index":"id","commitTimestamp":{"$timestamp":{"t":1659056993,"i":12}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.622+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"TenantMigrationDonorService-0","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"config.tenantMigrationDonors","index":"TenantMigrationDonorTTLIndex","commitTimestamp":{"$timestamp":{"t":1659056993,"i":12}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.623+00:00"},"s":"I", "c":"STORAGE", "id":20320, "ctx":"TenantMigrationDonorService-0","msg":"createCollection","attr":{"namespace":"config.external_validation_keys","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"4c4a71d7-4d23-4752-bdbf-ac8bc814effa"}},"options":{}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.638+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"TenantMigrationDonorService-0","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"config.external_validation_keys","index":"id","commitTimestamp":{"$timestamp":{"t":1659056993,"i":15}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.639+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"TenantMigrationDonorService-0","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"config.external_validation_keys","index":"ExternalKeysTTLIndex","commitTimestamp":{"$timestamp":{"t":1659056993,"i":15}}}}
mongodb_1 | {"t":{"$date":"2022-07-29T01:09:53.639+00:00"},"s":"I", "c":"REPL", "id":5123005, "ctx":"TenantMigrationDonorService-0","msg":"Rebuilding PrimaryOnlyService due to stepUp","attr":{"service":"TenantMigrationDonorService"}}
tweet_collector_1 | Stream encountered HTTP error: 403
tweet_collector_1 | Stream encountered HTTP error: 403

Create a streamlit application

Create a stream application with all the parameters for queries, tweets real-time display, and other optional parameters.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.