Pycroft is the future user management system of the AG DSN student network. It is based on Flask and expects a postgres database making use of the SQLAlchemy ORM.
A basic understanding of git is advisable.
The first step should be to clone this repository via git clone <url>
, using what clone url shows you above
this very readme
An easy way of doing the setup is by using docker-compose.
Follow the
guides here
and here.
You will need a docker engine of 1.10.0+
and a docker compose
version of 1.8.0+
(I am not sure which exact version introduced
version 2 of the config)
Also, note that you might have to add your user to the docker
group
for running docker as a non-root:
sudo usermod -aG docker $(whoami)
su $(whoami) # to re-login as yourself to use the new groups
Note that if you start e.g. pycharm without logging out and in again, the docker group will not have been applied yet.
You should now be able to run docker-compose config
and see the
current configuration.
There are multiple setups defined in multiple files:
docker-compose.yml
: A minimal setup consisting of a python and a postgresql container for running the web server. This has everything you need initially.docker-compose.testing.yml
: A setup with an ldap container, a rabbitmq container and a celery worker (mocking the Hades RPC interface) used for integration testsdocker-compose.travis.yml
: Basically the same astesting
, but without volumes (i.e. mounting of your source directory into the relevant containers)
Different docker-compose setups defined in different files can be
accessed using the -f
switch like docker-compose -f docker-compose.testing.yml ps
. The default file, i.e. the one used
when omitting the -f
switch, is docker-compose.yml
. This is the
only one you need in order to set up an example pycroft instance.
Said default setup provides:
- web: The container running the webservice. This includes all python
libraries as well as bower for the installation of the js
dependencies. Instead of using the project directory from inside
the container, however, your project directory (
./
) will be mounted there, so you don't have to rebuild the container's image after every tiny change. As the server is started with--debug
, it will automatically reload when it detects a change to the project files. - db: A postgres database to be used for the pycroft database and the legacy cache.
In order to integrate the setup into pycharm, make sure you updated, there were important bugfixes.
The source code provides two running configurations, one for running the test instance, and one for unittesting. In order for them to work correctly, we need to add two project interpreters, which will represent the corresponding docker-compose setup.
Go to “Settings” → “Project interpreters” → Gear icon → “Add remote” → “Docker Compose”.
Create a new server (use the default settings for that) if noone
exists yet. Add the config file compose/default/docker-compose.yml
.
Select service: web
and python interpreter: python3
.
Repeat the same thing for compose/testing/docker-compose.yml
.
Save, and make sure the correct interpreter (/default/
, not
/testing/
) is selected as default for the project (“Project
settings” → “Project interpreter”). As a proof of concept, you can
run a python console.
Open the database window and open the preferences. Install any missing
driver files. Although the password should be set and remembered, for
some reason pycharm may ask you anyway. The password is password
.
Just as in the manual setup described below, you will need to import the legacy dump so the test instance has a data foundation.
After having installed docker and docker-compose, you can now start
all of the services (the -d
stands for “detached”). This pulls the
postgres image and builds the one for pycroft. Since a whole
production system is going to be set up, this may take a few minutes,
so grab a cup of tea and relax.
docker-compose up -d
Every container should now be marked as UP
if you take a look at
docker-compose ps
. There you see which port forwardings have been
set up (remember the port web
has been exposed!)
Because you started them in detached mode, you will not see what they print to stdout. You can inspect the output like this:
docker-compose logs # for all services
docker-compose logs web # for one service
docker-compose logs -f --tail=50 web # Print the last 50 entries and follow the logs
The last command should tell you that the server spawned an instance
at 0.0.0.0:5000 from inside the container. Due to the port
forwarding, you can take a peek at a working UI by accessing
0.0.0.0:5001
from your browser.
But don't be too excited, pycroft will fail after the login – we have to set up the database.
For this section, double check that every container is up and running
via docker-compose ps
, and if necessary run docker-compose up -d
again.
Pycroft needs a database in the backend. Because currently, the
schema isn't fixed and can change regularly (it actually does), it is
easier to provide a dump of data to feed into the legacy importer
(legacy/import_legacy.py
). The legacy importer translates data from
a cache database into the pycroft schema. The cached data is
retrieved using the legacy cacher, which accumulates data from our
ldap, mysql and postgres databases into one big cache database, a
regular dump of which is published in our
internal gitlab.
Once you got that dump (if you cloned the repo, pull the latest
changes), make sure it is placed somewhere in your source directory.
Let's assume this location is /example/legacy.sql
. Because the
whole source directory is mounted into the db
container to
/pycroft
, we can import it from inside:
docker-compose exec --user postgres db psql -f /pycroft/example/legacy.sql
You successfully included your legacy dump into the docker-container!
Now, you need to convert it into a pycroft database with the pycroft
importer. This is done by running /legacy/import_legacy.py
in a
separate container:
docker-compose stop web # this container blocks the /postgres db
docker-compose run --rm web python3 -m legacy.import_legacy
docker-compose start web
This will take some time as well, as you will be translating more than 400k financial records using an abstracted ORM backend.
Before we congratulate ourselves, let's dump the contents of the
pycroft
database just to be sure we don't have to run the importer
again (except someone changes the schema, then you'll have to do it
regardless). This is more tricky since the postgres
user from
inside the container does not have write permissions on the mounted
directory by default:
docker-compose exec --user postgres db pg_dump --create --clean --if-exists postgres:///pycroft -f /tmp/pycroft.sql
docker cp pycroft_db_1:/tmp/pycroft.sql ./example/pycroft_$(date +%Y-%m-%d).sql
docker-compose exec db rm /tmp/pycroft.sql
The “weird” container name in the second command arises from the need
to talk to containers directly via docker instead of operating on
the service (here: db
) using docker-compose. You will find the
current container name to a running service with docker-compose ps
.
If you want to re-import the dump, import it like the legacy database just with
docker-compose exec --user postgres db psql -f /pycroft/example/pycroft_2017-05-03_no_pw.sql
After all that, you should be able to log in into your pycroft
instance at 0.0.0.0:5001
! Congratulations!
For the testing setup, there exists a separate docker-compose file:
# get the stack up and running
docker-compose -f docker-compose.testing.yml up -d
# run all the tests
docker-compose -f docker-compose.testing.yml run --rm web nosetests -v
# run only the frontend tests
docker-compose -f docker-compose.testing.yml run --rm web nosetests -v tests.frontend
This is not the recommended way anymore, except you know what you do! (even if you know what you do, most of the integration tests' backends (ldap, rabbitmq) haven't been backported to the vagrant setup.)
A vagrant box using a docker container has been pre-configured. Once
having installed vagrant and docker, vagrant up
starts the
container. This does not start the webserver yet! It has to be
manually invoked, either using pycharm and selecting the Web Postgres
run configuration, or by manually invoking the start script.
For the latter, see the section below. In any case, the server should
be accessible on 0.0.0.0:5000
(127.0.0.1
works as well for the
server itself, you can't use the builtin debugger however because
authentication somehow isn't passed that way.)
However, pycroft expects being bound to a database, which we haven't set up yet. The easiest way to do this is being an active member and using the importer. Because Pycroft is still in development, the database schema changes a lot, and thus checking in and maintaining example data has proved unfeasible.
Since pycroft uses an importer to cache and then import data from the legacy system, a dump of said cached data is being maintained in our internal gitlab.
In the container, there are three databases in use (although perhaps not yet created):
pycroft
: This is where the web app expects its actual data.pycroft_test
: This is where the unittests (if using postgres) expectlegacy
: This is wherelegacy/cache_legacy.py
deposits legacy data from the legacy backends (netusers(mysql), userman(postgres), ldap(planned/in progress)), and wherelegacy/import_legacy.py
reads the data from.
The latter is where the sql dump should be inserted.
Let's start it:
$ vagrant up
# …
$ vagrant ssh
The first command started the container, which starts services such as
sshd. The second presents you a shell. Under /vagrant
, the
directory where you cloned pycroft into is mounted
(bi-directionally!). You can thus place the dump e.g. in a folder called example
.
$ vagrant ssh
$ cd /vagrant
$ psql < example/legacy.sql
# psql resposnses…
$ cd legacy
$ cp conn.py.example conn.py
# because you're just importing a dump, you don't need to edit the conn.py!
$ python import_legacy.py
Note that importing takes an enormous amount of time. There are
hundreds of thousands of financial records which have to be processed!
Grab a cup of coffee bottle of Mate and relax.
If that's done, you can start the web server.
You should immediately dump the pycroft database using pgdump! Because the postgres instance runs in the docker container, which is considered stateless and thus may lose changes to the file system in many circumstances, halting vagrant and reupping it loses the data you just so hardly imported.
For the next time you start pycroft, it is easier to just import your
freshly dumped pycroft.sql
into the pycroft
database.
The web server can be started in a custom manner by starting python /vagrant/server_run.py --debug --exposed
in the vagrant ssh session,
given that PYCROFT_DB_URI
is set in the environment. The value
should be
postgresql+psycopg2:///pycroft?host=/postgresql&client_encoding=utf8"
.
The --exposed
is necessary to not restrict the web server for
container-internal access only and be able to let vagrant pass port
5000 to the host machine.