Git Product home page Git Product logo

codeql-learninglab-actions's Introduction

Actions for Learning Lab CodeQL Courses

This repository provides Docker images and GitHub Actions for use in CodeQL courses on Learning Lab.

These actions allow you to specify workflows that can check that course participants' queries are correct, by running their queries against a well-known CodeQL database, and checking the results are as expected. Whatever the outcome, the action will post a comment on the commit which was pushed to add the queries.

When a user's results are incorrect, the comment will include details of which results are missing, and which are superfluous, including links to the lines of source code on GitHub when possible:

Screenshot:

Table of Contents

Creating your own course

There are two main components to any Learning Lab course for CodeQL that uses the components in this repository:

  • Query Checking Action:

    Each course has its own GitHub Action that is designed to be used in workflows that run when a course participant pushes new commits to their repo. The action will check which queries have changed in the push, and run the queries that it recognizes as part of the course (based on the filename).

    After running the queries, the action will check the results against a CSV file of expected results. It will then post a comment on the commit, detailing whether each query produced the correct results or not. And if not, it will include details of which results are missing, and which results are unexpected.

    These actions are bundled using Docker, and made available using GitHub Packages.

  • Learning Lab Course:

    This is the course itself. It creates the initial repository the participant will use for their course, posts instructions as GitHub issues, and listens for comments posted by the GitHub action to know when the user has completed the current task correctly, and is ready to advance to the next one.

Creating the Query Checking Action

(for an example of a working action, see courses/cpp/ctf-segv).

Course actions consist of an action.yml file, and docker image built from the base image codeql-learninglab-check.

The base image expects course images built on-top of it to add the file /home/codeql/config/config.json, which details the configuration for the course.

The file should look something like this:

{
  "databasePath": "<path-to-database-directory>",
  "locationPaths": "https://github.com/<owner>/<repo>/blob/<sha>{path}#L{line-start}-L{line-end}",
  "expectedResults": {
    "step-01.ql": "step-01.csv",
    "step-02.ql": "step-02.csv",
    "step-03.ql": false,
  }
}

In addition to the config.json file above, a course image needs to also add the snapshot directory that queries should be run against, and csv files for the expected results.

  • databasePath should be a directory in the docker image, relative to the config.json file, that contains the extracted CodeQL database that queries will be run against. If you are using the template below, it will usually be the name of the only top-level directory from inside the database zip file.
  • locationPaths is an optional template string that can be used to enable source links in comments, when participants have written queries that output unexpected rows, or are missing results. <owner>, <repo> and <sha> should be replace as appropriate, the placeholders {path}, {line-start} and {line-end} are used by the checker, and should be left as-is.
  • expectedResults is an object that maps expected query filenames to a csv file detailing what the expected results for this query should be. Only the first expression for each row in the query results is checked. If instead of a CSV filename, false is used, then the checker will assume that the CSV file has simply never been generated, and will print out the resulting output from the query for you to copy into a new file.

To simplify course creation, we recommend structuring your course folder like so:

├── answers               <─── Model Answers
│   ├── qlpack.yml
│   ├── step-01.ql        <─┬─ Answers with expected paths
│   ├── step-02.ql        <─┤  (relative to answers/)
│   └── ...               <─┘  as specified in config.json
├── image
│   ├── config
│   │   ├── config.json   <─── Main course configuration
│   │   ├── step-01.csv
│   │   ├── step-02.csv
│   │   └── ...
│   └── Dockerfile
└── action.yml

(For your convinience, we've created a template course that uses this file-structure in the folder templates/action. You can simply copy the folder, and follow the instructions in the template README for what things to replace).

action.yml should look something like this:

name: 'Check queries'
description: 'Check that the queries that have been pushed (as part of the lesson) produce the correct results'
author: 'GitHub <[email protected]>'
runs:
  using: 'docker'
  image: 'docker://docker.pkg.github.com/<owner>/<repo>/<package>'
branding:
  icon: 'check-circle'
  color: 'purple'

and Dockerfile should look something like:

FROM docker.pkg.github.com/github/codeql-learninglab-actions/codeql-learninglab-check:<version>

## Add course config
COPY --chown=codeql:codeql config /home/codeql/config
WORKDIR /home/codeql/config
# Download, unzip and then delete the zip file in one step to reduce image size
RUN wget --quiet <url-for-snapshot-zip> -O database.zip && unzip -qq database.zip && rm -rf database.zip

Note that we download, unzip and then delete the zip file of the snapshot in a single step here. This helps us reduce the size of the image, as separate steps would result in intermediate image layers that are built on-top of one another.

Testing the action

You can test the action either locally or on GitHub actions.

Locally:

To test a course locally, from the course directory, run either of these scripts:

  • scripts/test-course-actual.sh: which will download and use the specific version of codeql-learninglab-check that is specified in Dockerfile
  • scripts/test-course-latest.sh: Which will also build the codeql-learninglab-check image locally, and tag it with the expected base image of the course, allowing you to test how changes to the codeql-learninglab-check affect this specific course, without publishing any new images.

Both scripts take as argument an optional regexp string. If this string is passed, only the queries with names matching the regexp will be run. Otherwise all queries are run.

In GitHub Actions:

If adding a course to this repository, extend the workflow file .github/workflows/ci.yml to include your new course. Any subsequent pushes to any branch should trigger an Action to run that will succeed only when all the expected queries produce the right results.

If you are creating a course in another repository, you can copy the scripts/test-course-actual.sh and scripts/test-course-latest.sh files into that repository, and add a similar workflow file to the one mentioned above.

Adding new queries & calculating the contents for the CSV files

When testing the action (as detailed above), when a query that is run produces unexpected results, or it is specified as false in config.yml instead of listing a CSV filename, the actual results that it produces are printed out in the console. You can then store this output as the relevant CSV file.

So the workflow for adding a new query and CSV file looks like:

  • add the query (.ql file) to answers/.
  • add the query to the expectedResults property in config.json, with a starting value of false.
  • Test the action (whichever method you prefer).
  • Copy the CSV output to the appropriate file in image/config/.
  • Re-test the action to ensure it marks the query as producing the correct results.

Publishing your action

The main thing you need to do here is publish your Docker image somewhere, and ensure that action.yml referrs to a tag that is downloadable.

We recommend setting up a GitHub Actions Workflow to automatically publish your docker image with the version latest to docker.pkg.github.com whenever you get a new push to master. This is what we do in .github/workflows/publish.yml.

Any courses that are added to this repository need to be published in this manner.

Contributing your GitHub Action to this repository

If you want to add a course to this repository, ensure that:

Creating the Learning Lab Course

If you have not created a Learning Lab course before, it is recommended to take the course on creating a course!

There are core repositories that need to be created as part of any learning-lab course:

  • The course repository: All the course configuration, instructions etc...
  • The template repository: The initial contents that populate the repository created on behalf of the course participant. (All courses are taken with respect to it's own repository)

We've created two template directories that you can use as a starting point for your own CodeQL Learning Lab Course:

Simply copy the contents of these templates into their own repositories, and follow the template instructions to get started.

(Remember that you need to create 2 separate repositories for your Learning Lab course, they can't be directories in an existing repo).

Example Courses

Feel free to add your own courses to this list! See CONTRIBUTING.md.

Contributing

We welcome contributions, both for new courses, and improvements to existing courses ot the codeql-learninglab-check docker image.

Releasing new versions or updating dependencies

See: Updating and Releasing

License

The code in this repository is licensed under MIT (see LICENSE.md). However as it makes use of the CodeQL CLI, you must also abide by the GitHub CodeQL Terms and Conditions, whenever your usage involves the CodeQL CLI.

In particular, you are not permitted to use these docker images or actions to create CodeQL databases using the CLI in CI/CD, as per the terms & conditions:

the Software cannot be used [...] To generate CodeQL databases for or during automated analysis, continuous integration or continuous delivery, whether as part of normal software engineering processes or otherwise.

codeql-learninglab-actions's People

Contributors

adityasharad avatar ashikpaul avatar dependabot[bot] avatar rvermeulen avatar s0 avatar xcorail avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

codeql-learninglab-actions's Issues

Use GitHub Packages directly in action.yml files

Docker images published to http://docker.pkg.github.com/ can't directly be used in action.yml files via docker://, for a couple of reasons:

  • the docker package registry currently requires HTTP auth, including for public repositories
  • the actions docker daemon doesn't authenticate with docker.pkg.github.com by default

Workaround

To work around this, course templates need to currently specify their own action locally, that builds a Dockerfile locally that references the course image, and the workflow files need to add an additional step to authenticate with docker before running the course action.

Once Fixed

Once this is no longer the case, and actions.yml files can directly reference docker images on docker.pkg.github.com, the following changes need to be made in this repository:

  • The action .github.to.move/workflows/action/ directory should be deleted
    from templates/learninglab/course-template
  • The workflow file templates/learninglab/course-template/.github.to.move/workflows/check-queries.yml should be updated to address the TODO comments
  • Instructions should be updated in templates/learninglab/README.md to instruct course creators to modify the uses property in .github.to.move/workflows/check-queries.yml instead of updating the Dockerfile.

Archive me

Learning lab is shut down. Please archive this repository.

Mismatch between suggested and actual alert counts

The CSVs of expected results seems to include all raw results of the query (including those not present within the database source archive root). In the VS Code IDE, these correspond to the #select table in the results view.
However, since the suggested template query has the problem metadata, the user is likely to see results in the alerts view, which filters out results in files outside the source archive root.

This leads to a mismatch in numbers, e.g. step 09 of the segv CTF has 70 results in the alerts view and 98 results in the #select view, and if you submit an incorrect solution it tells you it expects 98.

This suggests we need to either:

  • use only the interpreted alerts within the source archive in the expected results files, or
  • better explain the result counts.

First comment is attached to the latest commit of the course template

When we are executing the very first check answers action in a course, a welcome comment from the Bot is written:

comment += `\n Hey, I am the CodeQL Learning Lab bot :robot:`

But this comment is also attached to the latest commit, and then notify also the author of the course, which is annoying.

It makes sense that this comment is posted on the first issue of the course (it's not atm), but not attached to the commit.
Even better: The course could wait for this comment, to make sure that the workflow is set up, before asking the learner to close the issue and navigate to the next one

Course templates should directly create Actions Workflows

Learning Lab can't currently use course templates that directly create GitHub workflow files in .github/workflows. This is due to an access restriction. As CodeQL courses rely on being able to run actions to check participants's answers, we need to work around this.

Workaround

We create the actions workflows as needed, and then rename .github to .github.to.move.
We then instruct course creators to include instructions asking users

Once Fixed

  • Rename .github.to.move to .github in templates/learninglab/course-templates
  • Remove any step + check in the course asking the learner to do that and checking it
  • Update the instructions in templates/learninglab/README.md as appropriate.
  • All CodeQL courses should be updated to do the same thing.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.