sci-oer / automated-builder Goto Github PK
View Code? Open in Web Editor NEWAn automated build tool to generate custom sci-oer docker images
Home Page: https://pypi.org/project/scioer-builder/
License: GNU Affero General Public License v3.0
An automated build tool to generate custom sci-oer docker images
Home Page: https://pypi.org/project/scioer-builder/
License: GNU Affero General Public License v3.0
Allow an environment variable like SSH_KEY_FILE
to be used to specify the location of the ssh private key file instead of using the cli flag.
We need a way to stream the video lectures. Wiki.js currently doesn't support that natively.
Is your feature request related to a problem? Please describe.
Having to pass all the configuration through as cli flags or envirionment variables can be frustrating if the same configuration is used for a number of different course configurations.
Describe the solution you'd like
This should load the configuration parameters from configuration files.
A hierarchy (higher takes priority):
~/.scioer.yml
~/.config/scioer/config.yml
$XDG_CONFIG_HOME/scioer/config.yml
/etc/scioer/config.yml
A specific configuration file can be specified with the --config <file>
or ignored with the --no-config
options.
Describe alternatives you've considered
This should tie into the configuration order of precedence with #43, #44, and #45.
once the container has started, is it possible to su to the student user?
The wiki links to practice problems are set up so that the path looks like:
/course/practiceProblems/dataStructures/hashMap
.
The current builder makes paths that look like: /course/oo-oer-examples/practiceProblems/dataStructures/hashMap
I can rename the oo-oer-examples repo to practiceProblems if that would make it easier. Or the builder should take the contents of the repo and put it into the practiceProblems subdir.
Documentation must be usable by instructors unfamiliar with docker.
Allow the ssh key to be passed in directly as a string using the --ssh-key
flag.
This flag is probably not the best option to load the the key as it will be visible through the command history.
In the INTED paper we mentioned that some of the course content would only be accessible directly from the file system. We do not currently have any way to load this kind of seeded content into the image.
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates are currently rate-limited. Click on a checkbox below to force their creation now.
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
Dockerfile
python 3
builder/data/Dockerfile
.github/workflows/create-release.yml
actions/checkout v3
actions/cache v3
heinrichreimer/github-changelog-generator-action v2.3
release-drafter/release-drafter v5
.github/workflows/deployment.yml
actions/setup-python v4
pypa/gh-action-pypi-publish v1.8.8
docker/metadata-action v4
docker/setup-qemu-action v2
docker/setup-buildx-action v2
docker/login-action v2
docker/build-push-action v4
.github/workflows/verification.yml
actions/checkout v3
actions/checkout v3
actions/setup-python v4
pyproject.toml
setuptools >=61.0.0
setuptools_scm >= 6.2
requirements.txt
colorlog ==6.7.0
docker ==6.1.3
docopt ==0.6.2
GitPython ==3.1.32
requests ==2.31.0
tqdm ==4.65.0
pyreadline3 ==3.4.1
Describe the bug
the ssh key is not used for any repo other than the wiki
To Reproduce
Use a git repo that requires authentication for the examples
The home directory for the student user should be /course or should have an alias to /course in it.
newer versions of wikijs don't use sqlite. We will need to sort that out. It might be possible to migrate to a sql server and preload the wiki using sql?
Since this project contains more code than the oo-resources project a linter should be added to help ensure that the code maintains readable and maintainable.
for your first container please use the flag that keeps the git repos intact so that you can work in the repos and push back to the main repo on gitlab easily.
Allow the ssh private key to be specified directly using an environment variable like SSH_KEY
instead of using the cli flag.
Automated tests are a good way to ensure that the module works as intended and that each portion can be re-factored in the future while ensuring the behaviour is the same.
Pytest is a testing framework for python that will allow this to be accomplished in a fairly straight forward way.
Is your feature request related to a problem? Please describe.
When cloning the git repository for the wiki mount the ssh private key in the container so that it does not need to be sent in the api call.
Describe the solution you'd like
This should mount the key in read only mode from the host into the initialization container.
Currently the automated builder must be run from within a docker container and interacts with the docker daemon on the host machine. It would be beneficial if the builder can detect when it is not running in a docker container and can change the network mode that it uses to communicate with the bare wiki image that it is customizing.
if one clicks "open in browser" on docker desktop it goes to port 2222. Is it possible to change that so it goes to the wiki port?
The /course directory should have the following subdirectories:
coursework
jupyter
lectures
logs
practiceProblems
wiki
The /course/jupyter folder should put the pre-written tutorials in a /tutorials folder instead of /builtin
In addition to allowing the ssh key to be loaded using various methods a set of prescience rule should be set and documented to state which will be used if multiple are specified.
Possible ordering, the highest number takes priority:
SSH_KEY
envirionment variableSSH_KEY_FILE
environment variable--ssh-key
or --key-file
cli flags, mutually exclusiveIs your feature request related to a problem? Please describe.
Some of the configuration options that can be passed through cli flags should also be configured to be passable through envirionment variables.
Describe the solution you'd like
Optionally load the configuration values from envirionment variables.
The envirionment variables can also be loaded from a .env file.
Describe alternatives you've considered
This is one of several methods that is being proposed to configure the builder
When getting @NLaundry, @judiMc and @CameronNorrie setup with this project a number of issues came up multiple times, having a FAQ section in the readme would make fixing those challenges easier.
these are the sections:
So I'm trying to enable the git synchronization in wiki.js and getting the errors noted below (and it isn't synching)
a casual google points to a versioning problem maybe? https://stackoverflow.com/questions/40435315/knex-timeout-acquiring-a-connection-the-pool-is-probably-full-are-you-missing
2022-04-07T19:17:10.301Z [MASTER] error: Syncing with storage target git: [ FAILED ]
2022-04-07T19:17:10.302Z [MASTER] error: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
2022-04-07T19:17:29.955Z [MASTER] warn: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
2022-04-07T19:18:10.237Z [MASTER] warn: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
2022-04-07T19:19:29.821Z [MASTER] warn: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
root@2d9bb7c29bde:/course/logs#
The readme file currently only contains the default content that github put in the file.
Need to make instructions for grabbing the container (assuming its been uploaded to the socs dockerhub) and for setting it up to be run from docker desktop. Instructions need to work on os/x as well as windows.
Using the --key-file
flag tell the build script where to load the the ssh key used for git authentication from.
There are labels present for the oo-resources custom Dockerfile but not for the automated builder file
A Changelog should be created for this project that gets updated every time a feature is added or a change is made
Currently the pre-seed wiki posts are created manually by directly interacting with the graphql endpoint.
This will not permit the upload of files, only text posts. This is not sufficient for use in this educational resources.
A better approach would be to use the graphql api to configure the git storage backed and trigger an import. This will allow all wiki content to be loaded. However I currently do not know enough about how the wikijs api works or how graphql works to be able to implement this.
Moved from sci-oer/java-resource#10
The wiki can be configured to import the posts from a git repository, however this needs to happen while the application is running.
Currently any seed content must be added manually, this is not conducive to automatically creating the image.
An image creation script should be developed that will:
Is your feature request related to a problem? Please describe.
Readme is missing the instructions to build the autobuilder image
Is your feature request related to a problem? Please describe.
Allow the builder script to be installed as a normal pip package
Describe the solution you'd like
Installation of the sci-automoated builder should be installable by running pip3 install sci-builder
argparse
or docopt
are some potential options.
This will allow the cli generator to be used independently of hardcoded options.
Currently the git repository that is being cloned to get the seed content for jupyter lab must not require authentication.
It should support optional username + password credentials as well as ssh key credentials
There will be at least 20 example projects included with the container, probably more. I'm concerned about the impact on container size if we include multiple sets of gradle wrapper files. thoughts?
If the automated builder container is running in a kuberneties cluster will the method of interacting with the docker daemon still work or will an alternative method be needed?
The intended deployment location for this service is on the school k8s cluster.
Use a similar ci pipeline as the oo-resources container to build and publish the image
Is your feature request related to a problem? Please describe.
Generating the docker image is only useful if it is in a form that can be accessed by students.
By generating a tar archive that can be downloaded and distributed to students is a way to achieve that.
Describe the solution you'd like
A cli flag that can be used to specify the name of the tar archive to produce in the output folder.
Describe alternatives you've considered
I have also considered being able to automatically push the generated image to a docker registry.
This should be an easier implementation than the full docker push implementation and could be supported as well.
Additional context
There may be issues with with naming conflicts of the generated tar archive.
This could be remedied by appending a timestamp the the file name that is generated and then printing the file name so it can be used by the caller.
the command prompt on the terminal window is not useful. Should be changed to some version of current directory.
Currently there are 3 spots where the version number must be updated for each release.
It would be easy to forget the version bump in one of these locations. The workflow should take care of automatically creating and bumping the version number whenever it is needed.
Some possible options:
Related: #43
Is your feature request related to a problem? Please describe.
Generating the docker image is only useful if it is in a form that can be accessed by students.
Uploading the image to a docker registry will make it easier for students to download the image.
Describe the solution you'd like
A set of cli flags can be used to specify the registry that the image gets pushed too.
--registry
, --registry-user
, --registry-password
, --registry-password-file
.
Describe alternatives you've considered
An alternative implementation is described in #43, both should be implemented.
As an alternative from keeping the lectures in a git repository allow the lectures to be imported from a local directory.
The method of loading the lecture content should be mutually exclusive.
The pip package should be kept up to date by deploying a new version every time a new tag is pushed with a version bump.
running git clone from within a container should not require user interaction
Wiki customization is a nice to have for individualized course content.
in addition to the wiki and the jupyter tutorials there is a set of practice problems that go, predictably, in the /course/practiceProblem folder. There will also be a set of video lectures that go in the /course/lectures folder
This should create a GH release very time a version tag is pushed to the repository.
This should include the build artifacts as well as the changelog.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.