To serialize with avro according to the input, the types are generated at build time by gogen-avro starting from an avsc schema with go:generate. But then the serde wants an instance to serialize on, see [1] and [2]: there is a megaswitch to choose the right instance, and if one adds his own template, he then has to necessarily add a case to it and recompile (if he wants schema registry support) which is ugly.

[1]

Types only exist for net-device and user example templates at the moment. So you can't use schema registry with those templates

or stderr may be intertwined on the screen with stdout

% jr run -k '{{key "KEY" 20}}' -f 1s -d 10s net-device | jq

{
"VLAN": "ALPHA",
Elapsed time: 10s
"IPV4_SRC_ADDR": "10.1.78.31",
Data Generated (Objects): 10
Data Generated (bytes): 3469
"IPV4_DST_ADDR": "10.1.226.185",
"IN_BYTES": 1788,
Throughput (bytes per second): 346

"FIRST_SWITCHED": 1677849577,
"LAST_SWITCHED": 1679728004,
"L4_SRC_PORT": 80,
"L4_DST_PORT": 22,
"TCP_FLAGS": 0,
"PROTOCOL": 3,
"SRC_TOS": 201,
"SRC_AS": 4,
"DST_AS": 0,
"L7_PROTO": 443,
"L7_PROTO_NAME": "ICMP",
"L7_PROTO_CATEGORY": "Application"
}

Create man page for JR with documentation:

man jr

Most of these:

buildingNumber
cardinalDirection
county
direction
latitude
longitude
nearbyGPSCoordinate
ordinalDirection
secondaryAddress
street
streetAddress
streetName
streetPrefix
streetSuffix
timeZone

Set librdkafka properties on the command line, like kcat

Avro-specific: avro-generic is already supported.

schema registry support

Implement ability to write directly to Kafka

https://pkg.go.dev/github.com/ugol/jr#section-readme

when closing a report should be generated:

• data created (number, MBs, throughput, etc)
• ideally also the effectiveness of the kafka compression, if present

Generate a string from a regex, something like: {{regex "[A-Z][1-9]{8,8}" }} -> D8939482

Generic way to have kind of polymorphism for data:

names function should generate random names typical for US or IT or other locales

Implement relationships in _metadata associated with a template

customer
order
product

see #59 too

Output design for jr to write directly to different stores (redis, etc), integrating redis code

Add version and --version option

email -> from domain of the company
name, surname -> easily repeatable in different sections of the template (to create a realistic email for example)

random function to create blobs of given size and with different compressibility (uncompressible, compressible)

Add mongodb producer

with Description, Category, Usage, etc

• buildargs (version, time, who, as in the Makefile)
• document how to use it (local kafka and registry props file mounted, local templates, etc)
• multiarch build to have linux/amd and linux/arm in the same image

Install jr with brew install jr

More info:

• https://docs.brew.sh/How-to-Create-and-Maintain-a-Tap
• https://docs.brew.sh/Adding-Software-to-Homebrew

Have in the template context the possibility to use pre-built, coherent and consistent hilevel data like:

user
orders
products
etc

Separating K and V (see issue #21 ) makes harder to | to kcat.

A nice and simple solution could be to use nested templates:

kcat_net-device:
{{random KEY1|KEY2|KEY3|KEY4}},{{template "net-device"}}

% jr man name

Description
name is a function bla bla

Usage:
...
"Name:" {{name}}
...

Notes:
bla bla

See issue #34

use the right permissions/user

A page describing how to use JR to implement an E2E complete use case

-k and -v

possibility to use different templates for both.

-k could be null by default

To simplify the command line, user should be able to add topic name and/or schema enabling on the _meta part in the schema itself

ability to run:

jr run user orders

this is needed for #60

JR should be shut down waiting for the last cycle of generation (not in the middle of producing a message)

Expose REST Apis to consume random data

A docker image with JR and kcat to make it easy to use in containerized environments. Ideally FROM SCRATCH or FROM ALPINE to minimize disk usage.

To be added to DockerHub too

hifly@localhost ~/Projects/jr (main)$ make test
go test ./...

jr/storage/redis

package jr/storage/redis
imports jr/storage: import cycle not allowed in test
FAIL jr/storage/redis [setup failed]

Install jr in bin directory and templates in home directory (~.jr/templates)

Script [1] is failing with "Resource not accessible by integration".

[1] https://github.com/ugol/jr/blob/main/.github/workflows/go-linux-release.yml

See for example:

https://fakerjs.dev/api/image.html

Create a function to generate tags, paragraphs, words from a list of words on a file (separated by newlines)
The function should be cached if executed (do not read again from file when invoked again) and accept functions to operate on the words with basic functionality:

choose a random word from the list
shuffle the array and get the first n
use the words to create combinations
markov chains

then higher level functions like name, city, etc would done just pointing at a specific path. This will also enable localization

https://fakerjs.dev/api/date.html

Something sasl related is missing in Alpine image:

/usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/librdkafka_vendor/librdkafka_musl_linux_arm64.a(rdkafka_sasl_cyrus.o): in function `rd_kafka_sasl_cyrus_close':
#15 12.65 (.text+0xb4): undefined reference to `sasl_dispose'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/librdkafka_vendor/librdkafka_musl_linux_arm64.a(rdkafka_sasl_cyrus.o): in function `rd_kafka_sasl_cyrus_recv':
#15 12.65 (.text+0x1a0): undefined reference to `sasl_client_step'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0x1c8): undefined reference to `sasl_errdetail'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0x35c): undefined reference to `sasl_getprop'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0x38c): undefined reference to `sasl_getprop'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0x3ac): undefined reference to `sasl_getprop'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/librdkafka_vendor/librdkafka_musl_linux_arm64.a(rdkafka_sasl_cyrus.o): in function `rd_kafka_sasl_cyrus_client_new':
#15 12.65 (.text+0xf74): undefined reference to `sasl_client_new'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0xfd4): undefined reference to `sasl_client_start'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0xff4): undefined reference to `sasl_errdetail'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0x110c): undefined reference to `sasl_listmech'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0x1180): undefined reference to `sasl_errstring'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/confluentinc/confluent-kafka-go/[email protected]/kafka/librdkafka_vendor/librdkafka_musl_linux_arm64.a(rdkafka_sasl_cyrus.o): in function `rd_kafka_sasl_cyrus_global_init':
#15 12.65 (.text+0x16dc): undefined reference to `sasl_client_init'
#15 12.65 /usr/lib/gcc/aarch64-alpine-linux-musl/12.2.1/../../../../aarch64-alpine-linux-musl/bin/ld: (.text+0x170c): undefined reference to `sasl_errstring'

Create higher level objects to be used in templates to easily maintain "consistency" in data created:

name, surname, gender, email, company, etc.

At the moment it's needed to create consistency directly in the template code, see email crreation in [1] for example, which could become cumbersome in more complex scenarios

[1]

For avro/protobuf to be extended

Templates are not limited to JSON, so it must be renamed to .tpl or something

Use logging instead of fmt.Print

see for example: https://fakerjs.dev/api/finance.html

Create JR website (on netlify with hugo)

Without -f these two options don't have any meaning: -d should be parsed only in presence of -f