yetanalytics / project-persephone Goto Github PK

View Code? Open in Web Editor NEW

7.0 3.0 1.0 2.93 MB

Library for validating Statements against Profiles

Home Page: https://clojars.org/com.yetanalytics/project-persephone

License: Apache License 2.0

Clojure 99.39% Makefile 0.36% Shell 0.09% Dockerfile 0.15%

xapi xapi-profiles validation

project-persephone's Introduction

project-persephone

Only Persephone, daughter of Zeus and wife of Hades, could travel between the Underworld and the world of the living. Project Persephone is the liaison between our physical world and the world of the Semantic Web.

A Clojure library for validating xAPI Statements against xAPI Profiles, featuring interactive CLI and webserver applications.

Index

Library: How to use the library/API functions
CLI: How to run the validate and match commands
Webserver: How to start up and run a webserver

Installation

Add the following to the :deps map in your deps.edn file:

com.yetanalytics/project-persephone {:mvn/version "0.9.1"}

Alternatively, to run the CLI or server as an application, you can pull a Docker image from DockerHub:

docker pull yetanalytics/persephone:latest

How It Works

Persephone performs two main tasks on xAPI Statements: Statement Template validation and Pattern matching. The former checks if a Statement follows the specifications of one or more Statement Templates in xAPI Profiles, while the latter performs regex-like matching against Profile Patterns. Persephone does so following the validation and matching guidelines of the xAPI Profile specification.

For example, suppose that you have the following Statement:

{
  "id": "00000000-4000-8000-0000-000000000000",
  "timestamp": "2022-06-27T10:10:10.000Z",
  "actor": {
    "name": "My Name",        
    "mbox": "mailto:[email protected]",
    "objectType": "Agent"
  },
  "verb": {
    "id": "https://xapinet.org/xapi/yet/calibration/v1/concepts#did",
    "display": {"en-US": "Did"}
  },
  "object": {
    "id": "https://xapinet.org/xapi/yet/calibration/v1/concepts#activity-1",
    "objectType": "Activity",
    "definition": {
      "name": {"en-US": "Activity 1"},
      "description": {"en-US": "The first Activity"}
    }
  },
  "context": {
    "contextActivities": {
      "category": [
        {
          "id": "https://xapinet.org/xapi/yet/calibration/v1",
       	  "objectType": "Activity"
        }
      ]
    }
  }
}

The Statement will match against the following Statement Template:

{
  "id" : "https://xapinet.org/xapi/yet/calibration/v1/templates#activity-1",
  "inScheme" : "https://xapinet.org/xapi/yet/calibration/v1",
  "prefLabel" : {
    "en" : "Activity Template 1"
  },
  "definition" : {
    "en" : "The statement template and rules associated with Activity 1 getting done."
  },
  "type" : "StatementTemplate",
  "verb" : "https://xapinet.org/xapi/yet/calibration/v1/concepts#did",
  "rules" : [
    {
      "location" : "$.id",
      "presence" : "included"
    },
    {
      "location" : "$.timestamp",
      "presence" : "included"
    },
    {
      "any" : ["https://xapinet.org/xapi/yet/calibration/v1/concepts#activity-1"],
      "location" : "$.object.id",
      "presence" : "included"
    },
    {
      "any" : ["Activity 1"],
      "location" : "$.object.definition.name.en-US",
      "presence" : "included"
    },
    {
      "any" : ["The first Activity"],
      "location" : "$.object.definition.description.en-US",
      "presence" : "included"
    },
    {
      "any" : ["https://xapinet.org/xapi/yet/calibration/v1"],
      "location" : "$.context.contextActivities.category[0].id",
      "presence" : "included"
    }
  ]
}

because the verb in the Statement is the same as the verb in the Statement Template, and because the Statement's id, name, description, and category.id property values are included in the values specified by the Template's rules, which are matched using the location JSONPath strings.

Now suppose that the Statement is the first in a sequence of Statements. This Statement will match against the following Patterns:

{
  "definition" : {
    "en" : "Pattern 1"
  },
  "primary" : true,
  "prefLabel" : {
    "en" : "Learning Pattern 1"
  },
  "type" : "Pattern",
  "inScheme" : "https://xapinet.org/xapi/yet/calibration/v1",
  "id" : "https://xapinet.org/xapi/yet/calibration/v1/patterns#pattern-1",
  "sequence" : [
    "https://xapinet.org/xapi/yet/calibration/v1/patterns#pattern-2",
    "https://xapinet.org/xapi/yet/calibration/v1/patterns#pattern-3"
  ]
},
{
  "definition" : {
    "en" : "Pattern 2"
  },
  "primary" : false,
  "prefLabel" : {
    "en" : "Learning Pattern 2"
  },
  "type" : "Pattern",
  "inScheme" : "https://xapinet.org/xapi/yet/calibration/v1",
  "id" : "https://xapinet.org/xapi/yet/calibration/v1/patterns#pattern-2",
  "optional" : "https://xapinet.org/xapi/yet/calibration/v1/templates#activity-1"
}

since Pattern 1 (which is a primary Pattern, indicating that it is the starting point for Pattern matching) specifies a sequence of Patterns 2 and 3, and Pattern 2 indicates that the Statement can optionally match against the aforementioned Statement Template, which the Statement indeed does.

License

Distributed under the Apache License version 2.0.

The Persephone logo is based off of Proserpine by Hiram Powers.

project-persephone's People

Contributors

Stargazers

Watchers

Forkers

mrbillmcdonald

project-persephone's Issues

Incompatibility with project-pan

How to reproduce:

Uncomment the template-valid function in template-validation.clj
Run make ci in the command line
You should get an error along the lines of:

Syntax error compiling at (ubergraph/core.clj:881:44).
No such var: d/save!

The cause of this is that Ubergraph (a dependency of project-pan, which in turn is a dependency of project-persephone) uses the 0.06 version of Dorothy, while project-persephone uses the 0.07 version. Rolling back the version number for project-persphone fixes this, but causes other issues, e.g. dorothy/jvm is no longer available.

The long-term solution is to migrate from Ubergraph to Loom for project-pan; the latter doesn't use Dorothy to visualize, and should be done for cljc compatibility anyways.

Distinguish between "unmatched" and "invalid" template validation results

Currently, template validation fails if the Determining Properties and rules fail, and succeeds otherwise; this is a binary system. However, the Profile server spec specifies a ternary logic system, where if the Determining Properties fail, the Statement is unmatched against the Template(s), while if the rules fail, the Statement is invalid.

PERS-8 JSONPath fails to recognize string-valued keys

Issue: The current JSONPath functionality does not accept strings with string-valued keys, such as:

$. context.extensions['https://w3id.org/xapi/cmi5/context/extensions/moveon']

from the cmi5 profile or

$.context.contextActivities.category['https://w3id.org/xapi/catch/v1']

from Will Hoyt's CATCH profile (found in the profile-tools lib).

Possible fix: Replace the current JSONPath library (which does searching on Clojure maps) with a more stable one, such as the Jayway Java library. This change has been suggested since the beginning, but this issue gives more reason to do so.

Return failure location when matching against Patterns

According to the Profile server spec, when Statements fail to match against a Pattern, descriptive information on what part of the Pattern was violated should be returned.

[PERS-17] Incorrect Determining Properties logic

Currently, Persephone treats Determining Properties as special cases of all rules. However, this is incorrect, as the pseudocode provided with the xAPI Profile spec shows.

Determining Properties:

function matches_determining_properties(statement, template):
    for each specified Determining Property in the template:
        if the corresponding Statement locations equal or are a superset of that Determining Properties values:
            continue
        else:
            return false
    return true

all rule:

if rule.all is specified:
    if strict or values is not empty:
        if values contains UNMATCHABLE:
            return false
        for value in values:
            if rule.all does not contain value:
                return false

In other words, for Determining Properties, the found values are a superset of the rule values, whereas for all rules, the found values are a subset of the rule values.

Pattern validation is still buggy

Problem: Despite extensive testing of the finite state machine (FSM) functionality on basic predicates, our library does not seem to correctly validate Patterns of Statement Templates.

Reproduction: Compile the toplevel Pattern from the cmi5 profile (the very last entry of the Pattern array) and try to validate the following sequence of statements:

;; ex-statement is a generic Statement
;; cmi-profile is the cmi5 profile in JSON-LD format

(def cmi-fsm (first (compile-profile cmi-profile)))
(->> {}
    (read-next-statement
      cmi-fsm
      (-> example-statement
            (assoc-in [:verb :id] "http://adlnet.gov/expapi/verbs/waived")
            (update :result dissoc :score)
            (assoc-in [:result :success] true)
            (assoc-in [:result :completion] true)
            (assoc-in [:result "https://w3id.org/xapi/cmi5/result/extensions/reason"] "string")
            (assoc-in [:context :contextActivities :category]
                             [{:id "https://w3id.org/xapi/cmi5/context/categories/moveon"}])))
    (read-next-statement
      cmi-fsm
      (-> example-statement
            (assoc-in [:verb :id] "http://adlnet.gov/expapi/verbs/launched")
            (update :result dissoc :score)
            (update :result dissoc :success)
            (update :result dissoc :completion)
            (assoc-in [:result "https://w3id.org/xapi/cmi5/result/extensions/launchmode"] "Normal")
            (assoc-in [:result "https://w3id.org/xapi/cmi5/result/extensions/launchurl"] "https://example.org/foo")
            (assoc-in [:result "https://w3id.org/xapi/cmi5/result/extensions/moveon"] "Passed")
            (assoc-in [:result "https://w3id.org/xapi/cmi5/result/extensions/launchparameters"] {})))
    (read-next-statement
      cmi-fsm
      (-> example-statement
            (assoc-in [:verb :id] "http://adlnet.gov/expapi/verbs/initialized")
            (update :result dissoc :score)
            (update :result dissoc :success)
            (update :result dissoc :completion)))
    (read-next-statement
      cmi-fsm
      (-> example-statement
            (assoc-in [:verb :id] "http://adlnet.gov/expapi/verbs/passed")
            (assoc-in [:result :success] true)
            (update :result dissoc :completion)
            (assoc-in [:result :duration] "PT4H35M69.14S")
            (assoc-in [:context :contextActivities :category]
                             [{:id "https://w3id.org/xapi/cmi5/context/categories/moveon"}])))

The Pattern should validate the sequence of Statements (as indicated by :rejected-last being false), but it fails to validate the passed pattern.

Suggestion: use ubergraph.core/viz in order to visualize the FSM structure.

JSONPath does not return unmatchable values

Description: The current JSONPath implementation filters out unmatchable values, instead of returning them as nil. For example, given the following JSON object and JSON path string:

(def json-object
    {"book"
     [{"category" "reference"
       "author"   "Nigel Rees"
       "title"    "Sayings of the Century"
       "price"    8.95}
      {"category" "fiction"
       "author"   "Evelyn Waugh"
       "title"    "Sword of Honour"
       "price"    12.99}
      {"category" "fiction"
       "author"   "Herman Melville"
       "title"    "Moby Dick"
       "isbn"     "0-553-21311-3"
       "price"    8.99}
      {"category" "fiction"
       "author"   "J.R.R. Tolkien"
       "title"    "The Lord of the Rings"
       "isbn"     "0-395-19395-8"
       "price"    22.99}]})

(def json-path "$.book[*].isbn")

it should return:

[nil nil 0-553-21311-3 0-395-19395-8]

but instead returns:

[0-553-21311-3 0-395-19395-8]

Consequences: Certain Statement Template rule properties require that no unmatchable values be returned by a given JSONPath string, even if matchable values exist. For example, a Statement:

MUST NOT include any unmatchable values if presence is included

so if the above JSON were a Statement, it would have passed the rule even though it should not have.

Solution: For Java, the Jayway implementation has a DEFAULT_PATH_LEAF_TO_NULL option which, if set, returns null for unmatchable values. However, there does not seem to be a similar option for any JavaScript implementation, which may force us to go back to using in-house solutions.

Statement Template validation does not support keys that are IRIs

Description: Statement Templates with rules that reference keys that are IRIs will not properly validate Statements, as the validation library cannot extract values properly.

Example: The following Statement (generated by DATASIM):

{
        "actor": {
            "mbox": "mailto:[email protected]",
            "name": "Bob Fakename"
        },
        "context": {
            "contextActivities": {
                "category": [
                    {
                        "id": "https://w3id.org/xapi/cmi5/v1.0"
                    }
                ]
            },
            "extensions": {
                "https://w3id.org/xapi/cmi5/context/extensions/launchmode": [
                    "Review",
                    "Normal",
                    "Browse"
                ],
                "https://w3id.org/xapi/cmi5/context/extensions/launchparameters": "UVzahSV5qzm5DI1IZbyFJ9g1OT2W7",
                "https://w3id.org/xapi/cmi5/context/extensions/launchurl": "8S6Z7rLwyOJ4pwBvq",
                "https://w3id.org/xapi/cmi5/context/extensions/moveon": [
                    "NotApplicable",
                    "CompletedOrPassed",
                    "Completed",
                    "CompletedAndPassed",
                    "Passed"
                ]
            },
            "registration": "f851859f-b0fe-4b36-9939-4276b96d302d"
        },
        "id": "8dfe2e4f-5644-4486-ba16-a3289a507593",
        "object": {
            "definition": {
                "type": "https://w3id.org/xapi/cmi5/activitytype/block"
            },
            "id": "https://example.org/block/1587280280"
        },
        "timestamp": "2019-11-18T13:05:34.279Z",
        "verb": {
            "id": "http://adlnet.gov/expapi/verbs/launched"
        }
    }

should conform to the following Statement Template:

{
            "id": "https://w3id.org/xapi/cmi5#launched",
            "type": "StatementTemplate",
            "prefLabel": {
                "en": "Launched"
            },
            "definition": {
                "en": "Launched some event"
            },
            "inScheme": "https://w3id.org/xapi/cmi5/v1.0",
            "verb": "http://adlnet.gov/expapi/verbs/launched",
            "rules": [
                {
                    "location": "$.result.score",
                    "presence": "excluded"
                },
                {
                    "location": "$.result.success",
                    "presence": "excluded"
                },
                {
                    "location": "$.result.completion",
                    "presence": "excluded"
                },
                {
                    "location": "$.context.contextActivities.category[*].id",
                    "none": ["https://w3id.org/xapi/cmi5/context/categories/moveon"]
                },
                {
                    "location": "$.context.extensions['https://w3id.org/xapi/cmi5/context/extensions/launchmode']",
                    "presence": "included",
                    "all": ["Normal", "Browse", "Review"]
                },
                {
                    "location": "$.context.extensions['https://w3id.org/xapi/cmi5/context/extensions/launchurl']",
                    "presence": "included",
                    "scopeNote": {"en": "Insert scope note here."}
                },
                {
                    "location": "$.context.extensions['https://w3id.org/xapi/cmi5/context/extensions/moveon']",
                    "presence": "included",
                    "all": ["Passed", "Completed", "CompletedAndPassed", "CompletedOrPassed", "NotApplicable"]
                },
                {
                    "location": "$.context.extensions['https://w3id.org/xapi/cmi5/context/extensions/launchparameters']",
                    "presence": "included"
                }
            ]
        }

but it will report errors stating that the rules involving $.context.extensions['...'] locations are not being followed.

Solution: Use a different JSONPath library, such as Pathetic (which appears to support IRI keys).

yetanalytics/persephone:latest - repository does not exist

Could you please publish yetanalytics/persephone to docker hub?

Thank you

PERS-9 Does not support patterns w/ IRI sub-pattern identifiers

Description: Persephone currently only supports optional, zeroOrMore, and oneOrMore Patterns that contain the following key-value pair:

"optional" : {"id" : "https://foo.org/pattern"}

instead of the following:

"optional": "https://foo.org/pattern"

Note that it is the second format that conforms to the xAPI Profile spec.

Because of this incompatibility, Profiles containing these Patterns will not be able to fully compile, which will lead to errors such as being unable to read Statements.