openedx / edx-cookiecutters Goto Github PK
View Code? Open in Web Editor NEWOpen edx public templates for apps, libraries and services.
License: Apache License 2.0
Open edx public templates for apps, libraries and services.
License: Apache License 2.0
As we have been improving the native Dockerfile in all of our existing IDAs which are being used in devstack. So we should update the Dockerfile of Django IDA cookie-cutter so that all newly created IDAs should have this functionality out of the box.
The Dockerfile of the IDA should have the following things:
The file https://github.com/openedx/edx-cookiecutters/blob/master/python-template/%7B%7Bcookiecutter.placeholder_repo_name%7D%7D/.readthedocs.yml gets complaints from readthedocs.
I fixed using more of the template from edx-platform, but I'm not sure of all the parts we want: https://github.com/openedx/edx-platform/blob/master/.readthedocs.yaml. Note that the requirements path is different, so don't blindly copy-paste.
This is from the readthedocs documentation: https://docs.readthedocs.io/en/stable/config-file/v2.html
Lastly, the cookiecutter itself is missing this top-level file, which means its build would probably be broken. Whether as a separate ticket or this ticket, we should probably add that file and publish the cookiecutter docs.
The cookiecutter currently has a PyPI publish action that should be moved to the openedx/.github
repo's shared workflows.
Acceptance criteria:
pypa/gh-action-pypi-publish
to a specific release or commit; currently we're using master
, which the action is warning about (and is a bad idea in general for both security and reliability).master
version, since it's under our control)If you use cookiecutter-django-ida and complete all of the setup steps, then run make requirements docs
, the docs build fails. There are a number of things that need to be fixed before it succeeds.
Acceptance criteria:
make docs
is added to the tests for cookiecutter-django-ida. Currently this would need to be added as a new step in the ci.yml
GitHub workflow under the run_ida_tests
job, but it's possible that these tests will have been moved to the actual Python unit tests by the time this ticket has been picked up. The test should run make doc_requirements docs
or similar.Known issues that will need fixing:
cookiecutter-django-ida/hooks/post_gen_project.py
needs to include extra_context["requires_django"] = "yes"
just like the django-app cookiecutter does, otherwise there will be a failure when autodoc imports urls.py
test_settings
and that is hardcoded into python-template's docs/conf.py
, but django-ida will need to use {{ repo_name }}.settings.test
or similar.We have an opportunity to start out all new Django services with strict CSP, HSTS, and other good security defaults. These can then be changed as needed as the application grows and changes.
(Strict CSP is very easy to achieve in a brand-new application, and more painful to achieve in an existing application.)
Both for this repo, and for the repos that it creates.
The Makefile target piptools
, which installs pip-sync
, was removed from cookiecutter-django-ida (but not other cookiecutters) in [link to line]. This means that make requirements
in a clean virtualenv will fail for that cookiecutter's outputs.
The other cookiecutters vary in how they install the pip-tools
package.
Acceptance criteria:
pre-requirements
target (or similar) that installs pip and pip-tools pin files, as a dependency of the upgrade
and *requirements
targets.
make requirements
works in a fresh cookiecutter outputmake upgrade
is a separate post-setup manual step, so this work is deferred to #337To help with keeping repos up-to-date, the generated repo should have a clear indication of the commit of the cookiecutter used to make it, and the parameters that were used. This will help in the future when the repo maintainer wants to adopt changes that have since been made to the cookiecutter.
XBlocks made from the cookiecutter fail the codecov test. We should either fix it or remove it.
This ticket was spun out of #411
When using a cookiecutter, I'm presented with a lot of questions that I'm not really sure how to answer. For example, when using cookiecutter-django-app I'm asked to provide all three of repo_name
, app_name
, and project_name
. Problems I have with this:
It would be great if the prompts could include help text, but the cookiecutter library doesn't support this and probably won't any time soon.
Recommended work:
(This work can be done incrementally.)
If you add translations in the conf/locale
folder they are not rendered on the platform.
To make it work by default on the platform, the following changes can be made. These changes were successfully tested in this plugin
Move all files from conf/locale
to locale
.
Update MANIFEST.in
file to include the new locale
folder at the time of package installation.
recursive-include {{cookiecutter.sub_dir_name}}/locale *
These changes will allow translations included in the locale
folder to be automatically recognized by the platform. In addition, the make extract_translations
and make compile_translations
commands should not be modified, as they work with these changes.
Acceptance criteria:
In #355 we discovered that the "logo_only": True
present in the python-template's docs/conf.py was causing doc builds to fail. Specifically, using cookiecutter-python-library and then running make docs
in the output results in the following error:
Warning, treated as error:
unsupported theme option 'logo_only' given
This is contrary to https://docs.openedx.org/en/latest/developers/how-tos/switch-to-the-sphinx-book-theme.html which gives a configuration that includes the setting. It's possible that the how-to is incorrect, or that it was correct at the time but sphinx-book-theme (now at 1.0.1) no longer supports it. In fact, only 4 repos seem to use this option, and 3 of them are on pre 1.0 versions of the theme.
Update: See openedx/docs.openedx.org#345 in which there are changes to the guide, including removal of logo_only.
The action is currently at v1, should be at v3
There are manual steps required for configuring a repo based on the cookiecutter to make the set up the new repo consistently. For example, making some checks required. These steps should be added to the README so someone knows exactly what to do.
TODO: Update this issue (or the README itself) with the actual list of missing steps.
The xblock and library cookiecutters should cause python-template to add tags metadata to openedx.yaml, as specified in OEP 2.
edx-platform has a Make target upgrade-package
that upgrades a single Python dependency while keeping everything else as constant as possible. It also includes a GitHub workflow that calls upgrade-package, which makes a PR against a specified branch with the results of the upgrade.
We might like to include this functionality in all new repos.
Acceptance criteria:
upgrade-package
target added to Makefile of cookiecutter templatesupgrade-one-python-dependency.yml
workflow extracted to openedx/.github repo as a reusable workflowTrying to run the edx-cookiecutter tests locally:
% tox
py38 installed: accessible-pygments==0.0.4,alabaster==0.7.13,appdirs==1.4.4,arrow==1.2.3,asgiref==3.7.2,astroid==2.15.6,Babel==2.12.1,beautifulsoup4==4.12.2,binaryornot==0.4.4,bleach==6.0.0,build==0.10.0,certifi==2023.7.22,cffi==1.15.1,chardet==5.1.0,charset-normalizer==3.2.0,click==8.1.6,click-log==0.4.0,code-annotations==1.5.0,cookiecutter==2.2.3,coreapi==2.3.3,coreschema==0.0.4,coverage==7.2.7,cryptography==41.0.3,defusedxml==0.7.1,dill==0.3.7,distlib==0.3.7,Django==3.2.20,django-appconf==1.0.5,django-cors-headers==4.2.0,django-crum==0.7.9,django-dynamic-fixture==3.1.2,django-extensions==3.2.3,django-model-utils==4.3.1,django-rest-swagger==2.2.0,django-statici18n==2.4.0,django-waffle==4.0.0,djangorestframework==3.14.0,doc8==0.11.2,docutils==0.18.1,drf-jwt==1.19.2,edx-auth-backends==4.2.0,-e git+ssh://[email protected]/edx/edx-cookiecutters.git@a70363c0d44a79d2dc6cd11a1650f6f571187c89#egg=edx_cookiecutter_lib&subdirectory=lib,edx-django-release-util==1.3.0,edx-django-utils==5.6.0,edx-drf-extensions==8.8.0,edx-i18n-tools==1.1.0,edx-lint==5.3.4,edx-opaque-keys==2.3.0,edx-rest-api-client==5.6.0,exceptiongroup==1.1.2,filelock==3.12.2,fs==2.4.16,idna==3.4,imagesize==1.4.1,importlib-metadata==6.8.0,importlib-resources==6.0.0,iniconfig==2.0.0,isort==5.12.0,itypes==1.2.0,jaraco.classes==3.3.0,jeepney==0.8.0,Jinja2==3.1.2,keyring==24.2.0,lazy-object-proxy==1.9.0,lxml==4.9.3,Mako==1.2.4,markdown-it-py==3.0.0,MarkupSafe==2.1.3,mccabe==0.7.0,mdurl==0.1.2,more-itertools==9.1.0,mysqlclient==2.2.0,newrelic==8.9.0,oauthlib==3.2.2,openapi-codec==1.3.2,packaging==23.1,path==16.7.1,pbr==5.11.1,pip-tools==7.2.0,pkginfo==1.9.6,platformdirs==3.10.0,pluggy==1.2.0,polib==1.2.0,psutil==5.9.5,py==1.11.0,pycodestyle==2.10.0,pycparser==2.21,pydata-sphinx-theme==0.13.3,pydocstyle==6.3.0,Pygments==2.15.1,PyJWT==2.8.0,pylint==2.17.5,pylint-celery==0.3,pylint-django==2.5.3,pylint-plugin-utils==0.8.2,pymongo==3.13.0,PyNaCl==1.5.0,pyproject_hooks==1.0.0,pytest==7.4.0,pytest-cookies==0.7.0,pytest-cov==4.1.0,pytest-django==4.5.2,python-dateutil==2.8.2,python-slugify==8.0.1,python3-openid==3.2.0,pytz==2023.3,PyYAML==6.0.1,readme-renderer==40.0,requests==2.31.0,requests-oauthlib==1.3.1,requests-toolbelt==1.0.0,restructuredtext-lint==1.4.0,rfc3986==2.0.0,rich==13.4.2,SecretStorage==3.3.3,semantic-version==2.10.0,sh==2.0.4,simplejson==3.19.1,six==1.16.0,slumber==0.7.1,snowballstemmer==2.2.0,social-auth-app-django==5.2.0,social-auth-core==4.4.2,soupsieve==2.4.1,Sphinx==6.2.1,sphinx-book-theme==1.0.1,sphinx-rtd-theme==1.2.2,sphinxcontrib-applehelp==1.0.4,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.1,sphinxcontrib-jquery==4.1,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,sqlparse==0.4.4,stevedore==5.1.0,text-unidecode==1.3,tomli==2.0.1,tomlkit==0.12.1,tox==3.28.0,twine==4.0.2,typing_extensions==4.7.1,uritemplate==4.1.1,urllib3==2.0.4,virtualenv==20.24.2,web-fragments==2.1.0,webencodings==0.5.1,WebOb==1.8.7,wrapt==1.15.0,XBlock==1.7.0,xblock-utils==3.3.0,zipp==3.16.1
py38 run-test-pre: PYTHONHASHSEED='4160413870'
py38 run-test: commands[0] | pytest
===================================================================== test session starts =====================================================================
platform darwin -- Python 3.8.17, pytest-7.4.0, pluggy-1.2.0
cachedir: .tox/py38/.pytest_cache
rootdir: /System/Volumes/Data/root/src/edx/src/edx-cookiecutters
configfile: tox.ini
plugins: cookies-0.7.0, cov-4.1.0, django-4.5.2
collected 0 items / 1 error
=========================================================================== ERRORS ============================================================================
________________________________________________________________ ERROR collecting test session ________________________________________________________________
.tox/py38/lib/python3.8/site-packages/django_dynamic_fixture/global_settings.py:61: in get_data_fixture
if hasattr(settings, 'DDF_DEFAULT_DATA_FIXTURE'):
.tox/py38/lib/python3.8/site-packages/django/conf/__init__.py:82: in __getattr__
self._setup(name)
.tox/py38/lib/python3.8/site-packages/django/conf/__init__.py:63: in _setup
raise ImproperlyConfigured(
E django.core.exceptions.ImproperlyConfigured: Requested setting DDF_DEFAULT_DATA_FIXTURE, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
During handling of the above exception, another exception occurred:
.tox/py38/lib/python3.8/site-packages/_pytest/config/__init__.py:642: in _importconftest
mod = import_path(conftestpath, mode=importmode, root=rootpath)
.tox/py38/lib/python3.8/site-packages/_pytest/pathlib.py:565: in import_path
importlib.import_module(module_name)
/usr/local/pyenv/pyenv/versions/3.8.17/lib/python3.8/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1014: in _gcd_import
???
<frozen importlib._bootstrap>:991: in _find_and_load
???
<frozen importlib._bootstrap>:961: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
???
<frozen importlib._bootstrap>:1014: in _gcd_import
???
<frozen importlib._bootstrap>:991: in _find_and_load
???
<frozen importlib._bootstrap>:961: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
???
<frozen importlib._bootstrap>:1014: in _gcd_import
???
<frozen importlib._bootstrap>:991: in _find_and_load
???
<frozen importlib._bootstrap>:975: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:671: in _load_unlocked
???
<frozen importlib._bootstrap_external>:843: in exec_module
???
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
???
.tox/py38/lib/python3.8/site-packages/django_dynamic_fixture/__init__.py:17: in <module>
from django_dynamic_fixture.global_settings import DDF_DEFAULT_DATA_FIXTURE, DDF_FILL_NULLABLE_FIELDS, DDF_FK_MIN_DEPTH, \
.tox/py38/lib/python3.8/site-packages/django_dynamic_fixture/global_settings.py:76: in <module>
DDF_DEFAULT_DATA_FIXTURE = get_data_fixture(default='sequential')
.tox/py38/lib/python3.8/site-packages/django_dynamic_fixture/global_settings.py:73: in get_data_fixture
six.reraise(DDFImproperlyConfigured, DDFImproperlyConfigured("DDF_DEFAULT_DATA_FIXTURE (%s) must be 'sequential', 'static_sequential', 'global_sequential', 'random' or 'path.to.CustomDataFixtureClass'." % settings.DDF_DEFAULT_DATA_FIXTURE), sys.exc_info()[2])
.tox/py38/lib/python3.8/site-packages/django/conf/__init__.py:82: in __getattr__
self._setup(name)
.tox/py38/lib/python3.8/site-packages/django/conf/__init__.py:63: in _setup
raise ImproperlyConfigured(
E django.core.exceptions.ImproperlyConfigured: Requested setting DDF_DEFAULT_DATA_FIXTURE, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
=================================================================== short test summary info ===================================================================
ERROR - django.core.exceptions.ImproperlyConfigured: Requested setting DDF_DEFAULT_DATA_FIXTURE, but settings are not configured. You must either define the envir...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================================================== 1 error in 3.71s =======================================================================
ERROR: InvocationError for command /System/Volumes/Data/root/src/edx/src/edx-cookiecutters/.tox/py38/bin/pytest (exited with code 2)
___________________________________________________________________________ summary ___________________________________________________________________________
ERROR: py38: commands failed
tox -r
to recreate the environment works fine.
2023-12-19
2023-01-10
Redwood 2023-04
Codecov does not work OOTB for most repositories created using this cookiecutter and most people remove it once their repository has been created. There are known alternatives that do not require additional steps for setup.
We have written up a HowTo on using the GitHub Action python-coverage-comment-action.
Should not be needed
Should not be needed
No response
A/C
The django-ida cookiecutter includes a prod_requirements
Makefile target, but the configuration repo expects production-requirements
: https://github.com/openedx/configuration/blob/354249b7b0ef7e5c74656e340449fa1e38156658/playbooks/roles/edx_django_service/tasks/main.yml#L168
I don't think we had a particular principled reason to use prod_requirements
, and the string doesn't appear anywhere other than in Makefiles, probably generated by the cookiecutter: https://github.com/search?type=code&q=(org%3Aopenedx+OR+org%3Aedx)+prod_requirements+NOT+path%3AMakefile (or possibly in whatever Makefile was abstracted from to make the django-ida cookiecutter)
We should probably change this in the cookiecutter and update repos.
Hi,
I recently created an openedx xblock using the edx-cookiecutter template but turns out the github actions ci.yml is failing at this stage
- name: Create Build
run: |
rm -rf /tmp/myxblock-xblock
XBLOCK=$(pwd) && cd /tmp/ && echo -e '\n\n\n\n\n' | cookiecutter $XBLOCK
cd /tmp/myxblock-xblock && make help && pip install -e .
cd /tmp/myxblock-xblock && make dev.build
Error:
-> Run rm -rf /tmp/myxblock-xblock
/home/runner/work/_temp/54fcbeee-2c42-4109-b7e4-76c22b1b61fa.sh: line 2: cookiecutter: command not found
Error: Process completed with exit code 127.
I also installed the cookiecutter in the ci but the build still failed with the following error:
A valid repository for "/home/runner/work/openedx-cmi5-xblock/openedx-cmi5-xblock" could not be found in the following locations:
/home/runner/work/openedx-cmi5-xblock/openedx-cmi5-xblock
/home/runner/work/openedx-cmi5-xblock/openedx-cmi5-xblock
Error: Process completed with exit code 1.
This is happening from the very first commit since the template xblock got created and make upgrade didn’t help either.
Once you eliminate this Build step and perform certain necessary linting corrections in the template xblock, the workflow passes all the checks.
You can also discuss this issue here
The django-ida and xblock cookiecutters seem to still be configured for the v2 Transifex API, at least based on how the .tx/config files look. (They use headers like [edx-platform.xxx]
.) We should update these so that new repos created from these will work with the Transifex v3 API.
New django apps should be created with a 0001_purpose_of_this_app.rst in the app sub directory that is just a stub pointing to 0001_purpose_of_this_repo.rst in the top-level docs.
The README of each cookiecutter lists make upgrade
as a separate post-run setup step. However, this could be done automatically by the post-gen hooks.
make upgrade
in a post-gen hook
make upgrade
inmake upgrade
make requirements
A/C
I created https://github.com/edx/flashcards for a hackathon, and discovered several respects in which the IDA cookiecutter output was broken or just not ideal. Capturing here for now, although several of these should probably be broken out into separate issues and/or PRs for implementation.
make help
Please include mentions of:
Before #249 the xblock-cookiecutter was not running any of the standard cookiecutter tests. When we made that fix, we hand to not fix the pydocstyle check in the tests/test_cookiecutter_xblock.py::test_quality
test.
tox tests/test_cookiecutter_xblock.py::test_quality
OEP-32 standardizes the lms_user_id
as the canonical identifier of a user in Open edX.
However, the lms_user_id
is currently inaccessible from the default edx-cookiecutter IDA.
This issue is to add something like edx/program-intent-engagement@4de0f0d to edx-cookiecutter so JWT_PAYLOAD_USER_ATTRIBUTE_MAPPING
loads the lms_user_id
of incoming users into the Django User
model of new IDAs built from edx-cookiecutters.
lms_user_id
of incoming users.See edx/edx-arch-experiments#349
Affected file: cookiecutter-django-ida/{{cookiecutter.repo_name}}/Dockerfile
Our repos often have a "how to develop on this repo" section in their README that describes the use of virtualenvs, git, and Open edX-specific workflows (such as make requirements
). As a result, we have a large variety of READMEs with often outdated or incorrect information in this section.
A/C:
Notes:
A/C
The cookiecutters currently say things like "Contributions welcome" and "email [email protected]". Repos made in the edx GitHub organization need different details.
Possible implementation:
Have orgs add a cookiecutter file to .github that has the information that should go in this part of the repo
A/C:
python-template
cookiecutter includes the actionA declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.