We present Ogma, the first approach, which provides a systematic test framework for machine-learning systems that accepts grammar-based inputs. See the paper Grammar Based Directed Testing of Machine Learning Systems for more details

• Python 2.7.15
• numpy 1.14.5
• scipy 1.1.0
• scikit-learn 1.19.0

The authors used Pycharm CE 2017.2.3 as the development IDE.

The Ogma approach is encapsulated in GramFuzz*.py. To run these files, you'll need a Context Free Grammar, a Jaccard Threshold and an API for your NLP service. You can look at the sample APIs in aylien_API.py, rosette_API.py and uclassify_API.py. Please run these with an example to see the type of inputs required of the output.

This folder consists of the APIs of the software used in the evaluationg of Ogma. We have specifically written the APIs such that the output of the get_label function for aylien_API.py, rosette_API.py and uclassify_API.py are standard.

Please create a file called API_KEYS.py and add your keys in the folowing format:

TEXTRAZOR_API_KEY = "<insert key here>"
UCLASSIFY_API_KEY_READ = "<insert key here>"
UCLASSIFY_API_KEY_WRITE = "<insert key here>"
AYLIEN_KEY = "<insert key here>"
AYLIEN_APP_ID = "<insert key here>"
ROSETTE_API_KEY = "<insert key here>"

This folder contains .txt files of all the Grammars we have used in our experimentation.

The main files containing all the data for our experiments.

These folders are contain tests such that the first inputs is an error state

These folders are contain tests such that the first inputs is not an error state

These folders are contain tests corresponding to varying the Jaccard Threshold as seen in RQ4 in the paper

These folders contains folders in the format <Classifier 1> <Classifier2> <Grammar> . Each folder has three files containing data of the experiments.

• ErrorDataDirected - Inputs generated by Ogma
• ErrorDataRandom - Inputs generated Randomly
• ErrorDataNoBacktrack - Inputs generated by Ogma without backtracking

This folder consists of scripts to reproduce the figures seen in the appendix of the paper.

There are two scripts in the folder, namely PlotData.py and PlotTime.py. These will consolidate the data into image form in the Plots folder. The PlotData.py script will plot the number of errors and the PlotTime.py script will plot the time taken.

python <filename>

eg. python GramFuzz_Rosette_uClassify.py

• Please contact [email protected] for any comments/questions