This is a prototype of scheduled task, a service aiming at scheduling data analysis, simulation and AI model training with MATLAB.
Automate live script execution overnight
Under the hood: Github Action implementation
The basic example will be checking Bitcoin prices daily, and storing it into a new row of an existing csv file.
mkdir Tasks
mkdir Data
First iteration of the task executed locally and interactively:
jason = webread("https://api.coinbase.com/v2/prices/BTC-USD/spot")
jason = struct with fields:
data: [1x1 struct]
T = struct2table(jason.data)
amount | base | currency | |
---|---|---|---|
1 | 63300.09 | BTC | USD |
T.time = datetime("now")
amount | base | currency | time | |
---|---|---|---|---|
1 | 63300.09 | BTC | USD | 26-Apr-2024 12:08:59 |
writetable(T,'Data/btc.csv')
Second iteration: Simply read the existing dataset as a timetable and add a row
% Read input data
T = readtable("Data/btc.csv");
jason = webread("https://api.coinbase.com/v2/prices/BTC-USD/spot");
Tnew = struct2table(jason.data);
Tnew.time = datetime("now");
% Transform data to fit the input format
Tnew.amount = double(string(Tnew.amount));
Tnew.base = cellstr(Tnew.base);
Tnew.currency = cellstr(Tnew.currency);
T = [T;Tnew] % adding a row
amount | base | currency | time | |
---|---|---|---|---|
1 | 6.3300e+04 | 'BTC' | 'USD' | 26-Apr-2024 12:08:59 |
2 | 6.3404e+04 | 'BTC' | 'USD' | 26-Apr-2024 12:09:48 |
% Write output data
writetable(T,'Data/btc.csv')
Save this calculation into a script to run daily:
edit Tasks/task1.m
% Read input data
T = readtable("Data/btc.csv");
jason = webread("https://api.coinbase.com/v2/prices/BTC-USD/spot");
Tnew = struct2table(jason.data);
Tnew.time = datetime("now");
% Transform data to fit the input format
Tnew.amount = double(string(Tnew.amount));
Tnew.base = cellstr(Tnew.base);
Tnew.currency = cellstr(Tnew.currency);
T = [T;Tnew]; % adding a row
% Write output data
writetable(T,'Data/btc.csv')
addpath("code/");
filepath = 'Tasks/task1.m';
schedule = "*/5 * * * *"; % every 5 minutes
% output = "Data/"; % optionally specify output to zip and save
task = scheduleTask(filepath, schedule)
This task is going to generate a report in different formats (md, html, ipynb, docx, pdf).
The results can be served up:
- here as HTML: https://yanndebray.github.io/scheduled-task/Reports/task2
- or here as pdf: https://yanndebray.github.io/scheduled-task/Reports/task2.pdf
Create the mlx file to execute:
edit Tasks/task2.mlx
Schedule the execution:
filepath = 'Tasks/task2.mlx';
schedule = "0 0 * * *"; % every night at midnight
output = "Reports/"; % optionally specify output to zip and save
task = scheduleTask(filepath, schedule,output)
In the .github/workflows/
directory, create a new file called task.yml
and add the following code.
edit .github/workflows/task.yml
name: MATLAB task
run-name: ${{ github.actor }} is scheduling a MATLAB task
on:
schedule:
- cron: "*/5 * * * *"
workflow_dispatch: {}
jobs:
task:
runs-on: ubuntu-latest
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@v4
# Sets up MATLAB on a GitHub-hosted runner
- name: Set up MATLAB
uses: matlab-actions/setup-matlab@v2
# You can use "run-command" to execute custom MATLAB scripts, functions, or statements
- name: Run custom testing procedure
uses: matlab-actions/run-command@v2
with:
command: disp('Running my task!'); addpath('Tasks'); task1;
# Commit and push the result of the MATLAB task
- name: Commit and push changes
run: |
date > last-run.txt
git config user.name github-actions
git config user.email [email protected]
git add .
git commit -m "generated"
git push
# Save the result as artifact
- name: Archive output data
uses: actions/upload-artifact@v4
with:
name: bitcoin-price-history
path: Data/btc.csv
This will run the task1.m on a specific schedule, here every 5 minutes
This follows the cron syntax:
# โโโโโโโโโโโโโโ minute (0โ59)
# โ โโโโโโโโโโโโโโ hour (0โ23)
# โ โ โโโโโโโโโโโโโโ day of the month (1โ31)
# โ โ โ โโโโโโโโโโโโโโ month (1โ12)
# โ โ โ โ โโโโโโโโโโโโโโ day of the week (0โ6) (Sunday to Saturday;
# โ โ โ โ โ 7 is also Sunday on some systems)
# โ โ โ โ โ
# โ โ โ โ โ
# * * * * * <command to execute>
We are also adding a workflow_dispatch to enable a manual execution of the workflow in the actions tab:
This implementation saves the result of the task by pushing back the changes. For this ensure that you give Read and write permissions to the workflow in Settings/Actions
export("GettingStarted.mlx", "README.md");