For avro/protobuf to be extended

Create higher level objects to be used in templates to easily maintain "consistency" in data created:

name, surname, gender, email, company, etc.

At the moment it's needed to create consistency directly in the template code, see email crreation in [1] for example, which could become cumbersome in more complex scenarios

[1]

or stderr may be intertwined on the screen with stdout

% jr run -k '{{key "KEY" 20}}' -f 1s -d 10s net-device | jq

{
"VLAN": "ALPHA",
Elapsed time: 10s
"IPV4_SRC_ADDR": "10.1.78.31",
Data Generated (Objects): 10
Data Generated (bytes): 3469
"IPV4_DST_ADDR": "10.1.226.185",
"IN_BYTES": 1788,
Throughput (bytes per second): 346

"FIRST_SWITCHED": 1677849577,
"LAST_SWITCHED": 1679728004,
"L4_SRC_PORT": 80,
"L4_DST_PORT": 22,
"TCP_FLAGS": 0,
"PROTOCOL": 3,
"SRC_TOS": 201,
"SRC_AS": 4,
"DST_AS": 0,
"L7_PROTO": 443,
"L7_PROTO_NAME": "ICMP",
"L7_PROTO_CATEGORY": "Application"
}

• buildargs (version, time, who, as in the Makefile)
• document how to use it (local kafka and registry props file mounted, local templates, etc)
• multiarch build to have linux/amd and linux/arm in the same image

See for example:

https://fakerjs.dev/api/image.html

Add mongodb producer

Have in the template context the possibility to use pre-built, coherent and consistent hilevel data like:

user
orders
products
etc

Create JR website (on netlify with hugo)

Generic way to have kind of polymorphism for data:

names function should generate random names typical for US or IT or other locales

https://fakerjs.dev/api/date.html

Without -f these two options don't have any meaning: -d should be parsed only in presence of -f

Add version and --version option

see for example: https://fakerjs.dev/api/finance.html

Use logging instead of fmt.Print

-k and -v

possibility to use different templates for both.

-k could be null by default

Script [1] is failing with "Resource not accessible by integration".

[1] https://github.com/ugol/jr/blob/main/.github/workflows/go-linux-release.yml

To simplify the command line, user should be able to add topic name and/or schema enabling on the _meta part in the schema itself

hifly@localhost ~/Projects/jr (main)$ make test
go test ./...

jr/storage/redis

package jr/storage/redis
imports jr/storage: import cycle not allowed in test
FAIL jr/storage/redis [setup failed]

Most of these:

buildingNumber
cardinalDirection
county
direction
latitude
longitude
nearbyGPSCoordinate
ordinalDirection
secondaryAddress
street
streetAddress
streetName
streetPrefix
streetSuffix
timeZone

Install jr with brew install jr

More info:

• https://docs.brew.sh/How-to-Create-and-Maintain-a-Tap
• https://docs.brew.sh/Adding-Software-to-Homebrew

Create a function to generate tags, paragraphs, words from a list of words on a file (separated by newlines)
The function should be cached if executed (do not read again from file when invoked again) and accept functions to operate on the words with basic functionality:

choose a random word from the list
shuffle the array and get the first n
use the words to create combinations
markov chains

then higher level functions like name, city, etc would done just pointing at a specific path. This will also enable localization

random function to create blobs of given size and with different compressibility (uncompressible, compressible)

Generate a string from a regex, something like: {{regex "[A-Z][1-9]{8,8}" }} -> D8939482

Install jr in bin directory and templates in home directory (~.jr/templates)

Output design for jr to write directly to different stores (redis, etc), integrating redis code

Separating K and V (see issue #21 ) makes harder to | to kcat.

A nice and simple solution could be to use nested templates:

kcat_net-device:
{{random KEY1|KEY2|KEY3|KEY4}},{{template "net-device"}}

ability to run:

jr run user orders

this is needed for #60

A docker image with JR and kcat to make it easy to use in containerized environments. Ideally FROM SCRATCH or FROM ALPINE to minimize disk usage.

To be added to DockerHub too

Implement ability to write directly to Kafka

when closing a report should be generated:

• data created (number, MBs, throughput, etc)
• ideally also the effectiveness of the kafka compression, if present

Something sasl related is missing in Alpine image:

/usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/librdkafka_vendor/librdkafka_musl_linux_arm64.a(rdkafka_sasl_cyrus.o): in function `rd_kafka_sasl_cyrus_close':
#15 12.65 (.text+0xb4): undefined reference to `sasl_dispose'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/librdkafka_vendor/librdkafka_musl_linux_arm64.a(rdkafka_sasl_cyrus.o): in function `rd_kafka_sasl_cyrus_recv':
#15 12.65 (.text+0x1a0): undefined reference to `sasl_client_step'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0x1c8): undefined reference to `sasl_errdetail'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0x35c): undefined reference to `sasl_getprop'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0x38c): undefined reference to `sasl_getprop'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0x3ac): undefined reference to `sasl_getprop'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/librdkafka_vendor/librdkafka_musl_linux_arm64.a(rdkafka_sasl_cyrus.o): in function `rd_kafka_sasl_cyrus_client_new':
#15 12.65 (.text+0xf74): undefined reference to `sasl_client_new'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0xfd4): undefined reference to `sasl_client_start'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0xff4): undefined reference to `sasl_errdetail'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0x110c): undefined reference to `sasl_listmech'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0x1180): undefined reference to `sasl_errstring'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/librdkafka_vendor/librdkafka_musl_linux_arm64.a(rdkafka_sasl_cyrus.o): in function `rd_kafka_sasl_cyrus_global_init':
#15 12.65 (.text+0x16dc): undefined reference to `sasl_client_init'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0x170c): undefined reference to `sasl_errstring'

% jr man name

Description
name is a function bla bla

Usage:
...
"Name:" {{name}}
...

Notes:
bla bla

See issue #34

JR should be shut down waiting for the last cycle of generation (not in the middle of producing a message)

Templates are not limited to JSON, so it must be renamed to .tpl or something

with Description, Category, Usage, etc

Types only exist for net-device and user example templates at the moment. So you can't use schema registry with those templates

Avro-specific: avro-generic is already supported.

schema registry support

use the right permissions/user

A page describing how to use JR to implement an E2E complete use case

email -> from domain of the company
name, surname -> easily repeatable in different sections of the template (to create a realistic email for example)

Set librdkafka properties on the command line, like kcat

Expose REST Apis to consume random data

Create man page for JR with documentation:

man jr

https://pkg.go.dev/github.com/ugol/jr#section-readme

To serialize with avro according to the input, the types are generated at build time by gogen-avro starting from an avsc schema with go:generate. But then the serde wants an instance to serialize on, see [1] and [2]: there is a megaswitch to choose the right instance, and if one adds his own template, he then has to necessarily add a case to it and recompile (if he wants schema registry support) which is ugly.

[1]

Implement relationships in _metadata associated with a template

customer
order
product

see #59 too