Comments (13)
Hi @pedropalb
It seems like the new Docker for windows, solved the need for a specific volume for the mongodb docker (which was the reason to map the parent /data
folder and not the /data/db
and /data/configdb
). The docker compose for windows is now updated with similar fix.
Let me know if the issue consists.
from clearml-server.
Hi @pedropalb,
The services' containers indeed have their data folders mapped to the host file system... This might have something to do with an issue during the Elasticsearch container initialization - can you please share the container's log? You can get the log using the following command:
$ docker logs trains-elastic
from clearml-server.
It seems to be a problem with the free disk space. But I believe that 23.2 GB (the current free space in my disk) would be enough. The used space in c:\opt\trains is only 250 MB. The available space for docker disk image is 16 GB (from which only 3.1 GB). It follows the logs file:
from clearml-server.
Hi @pedropalb,
By default, the high watermark is 90% (see here), so the question is not how much free space you have on your disk, but what is the percentage of used space - try clearing up some space to see if it helps.
Alternatively, you can also configure the elasticsearch container using a different watermark by setting the value to a different percentage or a hard-coded number of bytes - simply edit your docker compose file and add a new line under the services / elasticsearch / environment
section:
services:
...
elasticsearch:
...
environment:
...
cluster.routing.allocation.disk.watermark.high: "15gb"
In the following example, elasticsearch will hit the high watermark only when you have less than 15 gigabytes free on your disk.
Please let me know if that works for you :)
from clearml-server.
Oh I see! I managed to free some space and it seems to have fixed the elastic search issue. But still, all my data has gone after the docker-compose down and up.
Every time we restart the server we have to create a new user and credentials? Can't I recover my data anymore?
from clearml-server.
Hi @pedropalb,
Every time we restart the server we have to create a new user and credentials? Can't I recover my data anymore?
The user and credentials are stored in the configuration files, not in the Elasticsearch data - did you lose those as well?
Regarding Elasticsearch, the data should still be there - can you find and send the directory contents of the nodes
folder inside the Elasticsearch data folder? It should be located in c:/opt/trains/data/elastic/nodes
or thereabouts.
from clearml-server.
It seems the Elasticsearch data is still there in the path you said. But the MongoDB is almost empty. I tried to query tasks, projects, users, etc. Everything is empty but the user collection that has only the newest user I created. What does go to MongoDB and what does go to Elasticsearch?
I didn't lose the configuration file but it has only the old credential. I didn't specify a user in the config file. I did that through the Web UI. So after the restart, I had to create new credentials and replace them in the config file. With the old credentials I couldn't even use the APIClient().
from clearml-server.
Hi @pedropalb,
You are correct in assuming that tasks, projects etc. (including user credentials) are stored in mongodb.
I now realize that your mongodb data has somehow disappeared, which is very strange - I previously thought it was only Elastic-related.
Can you please share the trains-mongo
docker container log?
Also, can you see if there's anything in your C:\Users\Public\Documents\Hyper-V\Virtual hard disks
folder? Maybe a file or sub-folder named mongodata
?
A few other thoughts:
- Is it possible that your docker-compose file was somehow changed and the mount point for the mongodb data folder was changed? Did you update the docker-compose file or download a new one?
- Did you upgrade your Docker Desktop? In order to use Linux containers, you usually need to add mapped drives to the Shared Drives list in the Docker Desktop Settings. However, Docker Desktop seems to have an inconsistency in this feature since I can't find this setting any more in the Docker Desktop Settings page, but their troubleshoot page still says it's required (see Troubleshoot, under VOLUME MOUNTING REQUIRES SHARED DRIVES FOR LINUX CONTAINERS - the link there now points to nowhere...) In case of the mount silently failing (it shouldn't, but still), what you're seeing now is probably the result of the mongo data folder not being mapped outside of the docker container, in which case mongo simply creates a new empty database inside the docker that has nothing to do with the outside world.
from clearml-server.
Here is the MongoDB log:
trains-mongo-logs.txt
There is nothing in C:\Users\Public\Documents\Hyper-V\Virtual hard disks
.
-
I did
docker-compose down
, downloaded a newer docker-compose file and diddocker-compose up
. But, now, without changing the docker-compose file, every time I restart the server, all the data is gone (I tested creating a new project and restarting the server). -
I upgraded the docker. I'm using Docker Desktop 2.2.0.4 with docker engine 19.03.8. I believe that this volume mapping is the source of my problems. I will check it.
Thanks!
from clearml-server.
Hi @bmartinn!
The issue is in the volume mapping in the docker-compose.yaml for the MongoDB service. By default, MongoDB writes the data in /data/db
. But in the docker-compose file, we have mongodata:/data
. But /data
has no data at all and, consequently, neither does the volume trains_mongodata
. When the container dies, all the mongo data in /data/db
dies with it.
So, the solution - I hope - is to change the mapping from mongodata:/data
to mongodata:/data/db
so the MongoDB's records are mapped to the volume trains_mongodata
.
I noted that docker-compose.win10.yml has this issue but the docker-compose.yml has not. The latter has no volume named trains_mongodata. It maps /data/db
directly in a host directory as it is done with the other data sources.
Would this change impact other TRAINS' functionalities?
Thanks!
from clearml-server.
Hi @pedropalb,
I just found where the shared drives feature was moved to in the new Docker Desktop: please go to Settings / Resources / File Sharing
and make sure your drive (c
drive, I assume) is marked for file sharing - let me know if that changes anything.
from clearml-server.
Hi @bmartinn,
I figured out what was the problem and reported the solution above in my previous comment.
Thanks.
from clearml-server.
Hi,
I noticed that in the docker-compose-win10.yml file, the volume mapping of the mongo container is still using /data
. Have you tested this on windows? As I reported above (#37 (comment)), it only worked for me changing - mongodata:/data
to - mongodata:/data/db
.
https://github.com/allegroai/trains-server/blob/3bf5126d84a67e7dec193bec0f6eff165e25665f/docker-compose-win10.yml#L90
Is there any other place where you change the mongo default data directory to /data
?
from clearml-server.
Related Issues (20)
- Could not find host server definition HOT 5
- Feature Request: Get server configuration parameters from AWS Secrets Manager [security]
- [Customising web-ui] - Projects are loading tasks in web ui of self hosting server but i want them to show datasets HOT 3
- generating clearml-reports HOT 13
- How to write artifacts to S3 from server side? HOT 1
- Nginx Not Loading Plotly.js Resource: ClearML Self-Hosted Docker HOT 7
- Failed Navigate From Overview to Experiments Details HOT 4
- Async Delete Always Failed when Removing Experiments (using Minio)
- nginx 0.6.x < 1.20.1 1-Byte Memory Overwrite RCE vulnerability HOT 2
- ElasticSearch UI and Redis UI? HOT 2
- The problem with scalars HOT 12
- Curl 7.69 < 8.4.0 Heap Buffer Overflow vulnerability HOT 2
- OpenSSL 1.1.1 < 1.1.1x Vulnerability HOT 1
- Elasticsearch image tag 7.17 does not exist HOT 4
- Git package is not installed by default in node:20-bookworm-slim HOT 1
- SERVER UNAVAILABLE HOT 4
- APP Credentials disapper in webapp HOT 20
- Scalar graphs legend is too narrow for experiments with long names HOT 7
- Update from 1.14.1 to 1.15.0 leads to several fatal issues when booting HOT 3
- AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clearml-server.