Can you provide further insight to what is the command line arguments specifically revolving around the schema. For example, if I have a json of FTP results. Here is what I have tried with Python errors revolving around the schema name:

zschema elasticsearch zgrab2-ftp ftp.json

Any further insight would be helpful. Thank you.

The required flag is ignored during Record/SubRecord validation.

If a Record or SubRecord has a field with required=True, validation should fail if that field is not present in the value being validated.

We should allow a schema file to indicate the minimum version of zschema it needs to be compiled correctly.

This code:

extension = SubRecord({
  "critical": Boolean(),
  "oid": String(),
  "value": Binary(),
}) 
relativeDistinguishedName = SubRecord({
  "type": String(),
  "value": String(),
})
certificate = SubRecord({
  "extensions": ListOf(extension, required=False),
  "raw_subject": ListOf(relativeDistinguishedName, required=True)
})

gives a runtime error TypeError: __init__() got an unexpected keyword argument 'required'.

I looked the codes here:
https://github.com/zmap/zschema/blob/master/zschema/__init__.py

the version is 0.0.27

But the one on pypi is 0.0.32.
https://pypi.python.org/pypi/zschema

I have installed through pypi.

import zschema
print(zschema.version)

It showed 0.0.27. But the site-packages is named of 0.0.32. So I think you might forget to change that version scope.

B.T.W.
How to use zschema to upload date into ElasticSearch? I haven't understood from here

Thanks!

Right now, the json command outputs an odd subset of information about a schema---probably what we knew about when we first implemented it. We should go back and make sure that it has everything.

h/t @cdzombak for reporting

https://travis-ci.org/zmap/zschema

Example:

"certificate_policies": ListOf(CertificatePoliciesData(), category="Certificate Policies", validation_policy="warn", exclude=["bigquery"]),

no hypehs

The allow_unknown feature is currently broken (it will lead to a KeyError when it tries to index theSubRecord[theUnknownKey]).

So, just remove it, since it's not needed.

https://en.wikipedia.org/wiki/Apache_Avro#Avro_Object_Container_File.5B2.5D

If I receive the following validation error:

class mismatch for id: expected [<type 'str'>, <type 'unicode'>], 1 has class int

I'm not sure where in my document its referring to the id field. It'd be great if the output indicated, say, the full JSON path of the field (e.g., certificate.parsed.certificate_policies.id)

I could be using the tool incorrectly, so bear with me 😄

I have a file that failed validation when updating to BigQuery because a non-leaf node was set to null, rather than omitted. zschema validate does not currently catch this, and passes the file with no errors.

E.g., when running validate and compiling to big query, no times should be past 9999-12-31. https://cloud.google.com/bigquery/docs/reference/standard-sql/migrating-from-legacy-sql#timestamp_differences

There is a bug in zschema when using the elasticsearch option:

zschema --module zgrab2_schemas.zgrab2 elasticsearch zgrab2-ftp /mnt/hgfs/Shared/ftp_dod_cidr.json
Traceback (most recent call last):
  File "/usr/local/bin/zschema", line 11, in <module>
    load_entry_point('zschema==0.10.2', 'console_scripts', 'zschema')()
  File "/home/juser/.local/lib/python2.7/site-packages/zschema/__main__.py", line 87, in main
    print json.dumps(record.to_es(recname))
UnboundLocalError: local variable 'recname' referenced before assignment

Removing recname from the to_es() call seems to fix this bug. You should probably check the other to_X() calls, as using recname is not correct most of the time.