We present Ogma, the first approach, which provides a systematic test framework for machine-learning systems that accepts grammar-based inputs. See the paper Grammar Based Directed Testing of Machine Learning Systems for more details
- Python 2.7.15
- numpy 1.14.5
- scipy 1.1.0
- scikit-learn 1.19.0
The authors used Pycharm CE 2017.2.3 as the development IDE.
The Ogma approach is encapsulated in GramFuzz*.py. To run these files, you'll need a Context Free Grammar, a Jaccard Threshold and an API for your NLP service. You can look at the sample APIs in aylien_API.py, rosette_API.py and uclassify_API.py. Please run these with an example to see the type of inputs required of the output.
This folder consists of the APIs of the software used in the evaluationg of Ogma.
We have specifically written the APIs such that the output of the get_label
function
for aylien_API.py
, rosette_API.py
and uclassify_API.py
are standard.
Please create a file called API_KEYS.py
and add your keys in the folowing format:
TEXTRAZOR_API_KEY = "<insert key here>"
UCLASSIFY_API_KEY_READ = "<insert key here>"
UCLASSIFY_API_KEY_WRITE = "<insert key here>"
AYLIEN_KEY = "<insert key here>"
AYLIEN_APP_ID = "<insert key here>"
ROSETTE_API_KEY = "<insert key here>"
This folder contains .txt
files of all the Grammars we have used in
our experimentation.
The main files containing all the data for our experiments.
These folders are contain tests such that the first inputs is an error state
These folders are contain tests such that the first inputs is not an error state
These folders are contain tests corresponding to varying the Jaccard Threshold as seen in RQ4 in the paper
These folders contains folders in the format <Classifier 1> <Classifier2> <Grammar>
. Each folder has three files containing
data of the experiments.
ErrorDataDirected
- Inputs generated by OgmaErrorDataRandom
- Inputs generated RandomlyErrorDataNoBacktrack
- Inputs generated by Ogma without backtracking
This folder consists of scripts to reproduce the figures seen in the appendix of the paper.
There are two scripts in the folder, namely PlotData.py
and PlotTime.py
.
These will consolidate the data into image form in the Plots
folder. The
PlotData.py
script will plot the number of errors and the PlotTime.py
script will plot the time taken.
python <filename>
eg. python GramFuzz_Rosette_uClassify.py
- Please contact [email protected] for any comments/questions