Comments (4)
I started the migration process at 10:03.
10:03
Create a MongoDB Atlas account with the Metakgp Gmail account.- Created a 3-node 512MB cluster
mftp-cluster
in the AWS Singapore cluster after verifying that the size of total notices in MFTP is way less than 512 MB.
rs-ds059694:PRIMARY> db.stats(1024*1024)
{
"db" : "heroku_15dt55xd",
"collections" : 4,
"views" : 0,
"objects" : 15233,
"avgObjSize" : 1093.25070570472,
"dataSize" : 15.882003784179688, // NOTE THIS, 15.88 MB
"storageSize" : 21.640625,
"numExtents" : 12,
"indexes" : 2,
"indexSize" : 0.4756317138671875,
"fileSize" : 48,
"nsSizeMB" : 1,
"extentFreeList" : {
"num" : 0,
"totalSize" : 0
},
"dataFileVersion" : {
"major" : 4,
"minor" : 22
},
"fsUsedSize" : 7830.7421875,
"fsTotalSize" : 30109.99609375,
"ok" : 1,
"operationTime" : Timestamp(1604853115, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1604853115, 1),
"signature" : {
"hash" : BinData(0,"qMgKv+wnPyUnqMVKE8cJtP9MisE="),
"keyId" : NumberLong("6862529136737386497")
}
}
}
- Configured Atlas instance to allow access from any IP as long as the connection string is correct. Add credentials to the Metakgp passwords sheet.
- Connected to both - the old instance and the new instance - from the mongo shell to monitor the process.
- Running the migration script to copy notices from old URI to new URI.
5.1 Faced an issue with the version ofpymongo
- we are using3.1.1
while recommended is 3.4 or later. Upgradedpymongo
to 3.4 (in bothrequirements.txt
andPipfile
) to keep at minimal changes ✔️ - will testpymongo
3.4 with running MFTP in a later step (done in 6).
5.2 Migration failed, realized that password had@
and other special characters which are not allowed - switched to a simpler password and updated in the password sheet ✔️ .
5.3 Ran migration using the following command withpymongo
3.4 and python 2.7 - took around 20 minutes to copy all 931 notices -python export_database.py --source "SOURCE_MONGO_URI" --target "DEST_MONGO_URI"
✔️
5.4 Confirmed that the last 10 notices (db.notices.find().sort({$natural: -1}).limit(10)
) on both the databases are the same ✔️ - Replicated production environment on my system
6.1 Send no mail to me for the last 10 notices as they are in the database - no emails received ✔️
6.2 Removed last 2 notices from the database and re-ran the script - received two emails ✔️ - Deploying to Heroku instances:
7.1 Enabled maintenance mode- DOWNTIME BEGINS AT 20 November 2020 11:20 PM IST ⬇️
7.2 Sent a mail to the MFTP google group regarding downtime ✔️
7.3 Confirm that the last two notices in both the old DB and new DB are the same so that we do not miss sending any important update.
7.4 Update environment variableMONGODB_URI
in Heroku app configuration.
7.5 Merge #36 with the required minimalpymongo
version changes. Push to Github repo.
7.6 Faced a minor build issue with Pipenv lock being out of sync (check figure 1) - fixed, force-pushed, and redeployed - successfully built and deployed ✔️
7.7 Removed mLab extension - should have been a step before redeploying and right after changing the environment variable ❗
7.8 Removed maintenance mode - logs fine, notices not sent as all in database ✔️
7.9 DOWNTIME OVER AT 20 November 2020 11:34 PM IST 🆙 - Atlas provides a nice GUI where we can check the health of the application (see figure 2, the mountain is running @iakshat 's migration script xD), it'll be handy in the future.
Screenshots
from mftp.
@mukul-mehta @sriyash421 @iakshat Can we please find some time before November 10 to fix this issue? It may lead to MFTP becoming non-functional and that will cause great harm to people sitting for placements.
from mftp.
cc/ @icyflame @amrav @hargup @kaustubhhiware
from mftp.
@icyflame performed the same for leprechaun. Please refer to metakgp/leprechaun#14 for an interesting read.
from mftp.
Related Issues (20)
- Repeated email sent for some Notices HOT 5
- Change subject from "Notice: CV submission" to "Placement: CV submission" HOT 2
- Update Vulnerable Dependency HOT 7
- Understand why bs4 recursion depth exceeded happens HOT 2
- Add attachments to the emails HOT 3
- Switch From Ping Mechanism To Heroku Scheduler HOT 15
- [IMPROVEMENT] Failure Monitoring HOT 4
- README update HOT 1
- Duplicate Mails HOT 1
- Project dependencies may have API risk issues HOT 1
- Move from Python 2 to 3 HOT 10
- MFTP Revamp HOT 1
- Personally tweaked features HOT 2
- Cronjob as alternative to service HOT 5
- Dockerise HOT 3
- MFTP doctor HOT 2
- Logs rotation HOT 7
- Documentation Update for MFTP as a cronjob
- Google Group Limitations HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mftp.