Comments (10)
Fixed tnx it was disk space.
On May 26, 2014 4:03 PM, "stewart long" [email protected] wrote:
[image: screen shot 2014-05-26 at 1 01 03 pm]https://cloud.githubusercontent.com/assets/1051020/3085409/d73706d8-e510-11e3-884e-c51a6752bf06.png
[image: screen shot 2014-05-26 at 1 00 41 pm]https://cloud.githubusercontent.com/assets/1051020/3085410/d7476942-e510-11e3-94fd-f5766f64361a.png—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3
.
from mapknitter.
Stu, I'm not sure how there could have been image placement work lost; perhaps you had been working on the map for 2.5 hours while the server was already down, but you hadn't refreshed your local browser window? Have you seen this happen again? If you do, could you create a new issue for it?
I'm sorry you lost 2.5 hours of work, that sucks. I've been inundated with feature requests, bug reports, etc on Infragram and PublicLab.org, while also trying to get this big tag migration going and a new test server. I've blocked out some time right now to make MapKnitter more efficient on disk space usage, but it is so huge right now that we're going to have to move it onto a larger disk very soon.
from mapknitter.
taking notes here; going through https://github.com/publiclab/mapknitter/blob/master/app/models/warpable.rb#L119 to find temp files from the export we can delete:
- believe everything in -working/ can be deleted; this is just so we can use the files locally outside of s3
- believe everything -masked.png can be deleted
- believe everything -mask.png can be deleted
- everything -geo WITH AN ID could be deleted, but there is a feature request to preserve these by Don, though I can't find it and it might've been solved a different way
- actually, everything ending in .png can be deleted -- this is just masks and local warped & masked files
This should win us a lot of space.
- first i'm going to try manually deleting them on a test map
- then will attempt to hardcode auto-cleanup in
- then run it on the test map
from mapknitter.
Would it be worth trying to offload the temporary files to an S3 bucket?
storage on S3 is cheap. 5 GiB ~= 2 cents.
We could create a new bucket for a new task, name it based on the task, and
then delete the bucket when the temporary utility has concluded.
We'll never run out of disk space, but you might get charged a few quarters
or something for each export.
-Bryan
On Thu, May 29, 2014 at 11:48 AM, Jeffrey Warren
[email protected]:
taking notes here; going through
https://github.com/publiclab/mapknitter/blob/master/app/models/warpable.rb#L119to find temp files from the export we can delete:
- believe everything in -working/ can be deleted; this is just so we
can use the files locally outside of s3
- believe everything -masked.png can be deleted
- believe everything -mask.png can be deleted
- everything -geo WITH AN ID could be deleted, but there is a feature
request to preserve these by Don, though I can't find it and it might've
been solved a different wayThis should win us a lot of space.
- first i'm going to try manually deleting them on a test map
- then will attempt to hardcode auto-cleanup in
- then run it on the test map
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-44569185
.
from mapknitter.
The source files are already stored in s3 - actually the first operation is to copy them to a local folder so they can be manipulated. I think we can be very aggressive here without losing anything. like rm warps/*/*.png
should be fine. also rm -r warps/*-working/
since there are no maps which end in -working
from mapknitter.
I'm going to start with deleting all -working directories and pngs, and leave the geo.tiff files, since they're used later in the export process. We can take this by stages.
from mapknitter.
tested and running... doing bulk deletions now
148gb available after deleting pngs
from mapknitter.
warren@tycho:/sites/mapknitter.org$ du --max-depth 1 -h public/warps/mestia//sites/mapknitter.org$ du --max-depth 1 -h public/warps/space-test/
2.7G public/warps/mestia/
warren@tycho:
7.0M public/warps/space-test/
So huge maps like http//mapknitter.org/map/view/mestia still have 2.7gb of geotiffs, but we've won a lot of space overall.
from mapknitter.
after the working directories were deleted, we now have 162gb of space. And the exports should now clean up after themselves.
from mapknitter.
If we need more later, we can delete all /public/warps/<mapname>/<warpable_id>-geo.tif
otherwise closing for now!
from mapknitter.
Related Issues (20)
- Update Staging urls HOT 3
- why didn't a yarn error trigger a failure in GitHub Actions? node version mismatch HOT 6
- FTO: Fix typos and update config.yml HOT 4
- AWS credentials issue on stable HOT 7
- CodeCov not registering recent test reports HOT 3
- Outreachy: "MapKnitter.org Spam Management System" Project Planning HOT 16
- Outreachy: "MapKnitter.org Spam Management System" Project Q&A HOT 19
- Create status columns on maps and users tables
- Create batch-spam and single spam endpoints
- Create batch-publish and single publish endpoints
- Improve "card" section of the maps show page HOT 13
- Add first-timers-only issue template HOT 11
- Create batch-delete endpoint
- Adjust existing fetch map(s) and user(s) endpoints to fetch based on status
- building in stable - notes HOT 5
- Create batch-ban and single ban endpoints
- Create batch-unban and single unban endpoints
- Create endpoints for fetching maps on the spam dashboard page
- Create endpoints for fetching users on the spam dashboard page
- Edit login process to allow only active users HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mapknitter.