markus-wa / cq Goto Github PK

Clojure Query: A Command-line Data Processor for JSON, YAML, EDN, XML and more

License: Eclipse Public License 2.0

Clojure 94.93% Shell 3.46% Batchfile 1.38% Emacs Lisp 0.23%

clojure yaml json edn csv command-line cli msgpack hacktoberfest xml

cq's Introduction

Hi there, I'm Markus! 👋

I'm a Freelance Backend & Cloud Engineer from Switzerland 🇨🇭, currently based in the UK 🇬🇧 (Bristol).
During the day I run Saikō Technology (saiko.tech / @saiko-tech), a software consultancy focused on eSports & backend systems.
At night I like to tinker with various things like audio-visuals and building open source tooling.

🔭 I’m currently working on @esportalgroup Infrastructure, API & Backend
💖 I love Clojure - and functional programming in general
👯 I’m looking to collaborate on game-data visualisation (frontend)
💬 Ask me about backend systems, data pipelines & eSports - or anything else, really!
📰 Check out my previous work on my resume
📫 How to reach me: [email protected]

cq's People

Contributors

Stargazers

Watchers

Forkers

fossabot se-neaxplore borkdude byronclark macroz saguywalker reedho pedromanoel

cq's Issues

--color=auto should detect if output supports colors

see #15 (comment)

pretty was released with automatic logic for enabling colors. Seems pretty straightforward: clj-commons/pretty@7cb0c16

transit+msgpack only works on the JVM

Version: 2022.01.26-15.35.00
Platform: macOS 12.5

Encoding to or decoding from transit+msgpack fails on the GraalVM binary but works on the JVM.

JVM

❯ echo '{"something": 42}' | java -jar ~/Downloads/cq.jar -i json -o transit --transit-format-out msgpack | xxd
00000000: 81ab 7e3a 736f 6d65 7468 696e 672a       ..~:something*

GraalVM binary

❯ echo '{"something": 42}' | ~/Downloads/cq-native-macos -i json -o transit --transit-format-out msgpack
Aug 17, 2022 4:15:01 PM org.msgpack.template.builder.TemplateBuilderChain createForceTemplateBuilder
WARNING: Failed to create a TemplateBuilder reflectively
java.lang.ClassNotFoundException: org.msgpack.template.builder.JavassistTemplateBuilder
	at com.oracle.svm.core.hub.ClassForNameSupport.forName(ClassForNameSupport.java:60)
	at java.lang.Class.forName(DynamicHub.java:1247)
	at org.msgpack.template.builder.TemplateBuilderChain.createForceTemplateBuilder(TemplateBuilderChain.java:84)
	at org.msgpack.template.builder.TemplateBuilderChain.reset(TemplateBuilderChain.java:73)
	at org.msgpack.template.builder.TemplateBuilderChain.<init>(TemplateBuilderChain.java:54)
	at org.msgpack.template.builder.TemplateBuilderChain.<init>(TemplateBuilderChain.java:49)
	at org.msgpack.template.TemplateRegistry.createTemplateBuilderChain(TemplateRegistry.java:102)
	at org.msgpack.template.TemplateRegistry.<init>(TemplateRegistry.java:78)
	at org.msgpack.template.TemplateRegistry.<init>(TemplateRegistry.java:93)
	at org.msgpack.MessagePack.<init>(MessagePack.java:60)
	at com.cognitect.transit.impl.WriterFactory.getMsgpackInstance(WriterFactory.java:77)
	at com.cognitect.transit.TransitFactory.writer(TransitFactory.java:95)
	at cognitect.transit$writer.invokeStatic(transit.clj:161)
	at cognitect.transit$writer.invoke(transit.clj:139)
	at cognitect.transit$writer.invokeStatic(transit.clj:155)
	at cognitect.transit$writer.invoke(transit.clj:139)
	at cq.formats$__GT_transit_writer$fn__14686.invoke(formats.clj:154)
	at cq.formats$format__GT_writer$fn__14701.invoke(formats.clj:199)
	at cq.core$run.invokeStatic(core.clj:97)
	at cq.core$run.invoke(core.clj:93)
	at cq.main$main.invokeStatic(main.clj:109)
	at cq.main$main.invoke(main.clj:99)
	at cq.main$_main.invokeStatic(main.clj:114)
	at cq.main$_main.doInvoke(main.clj:111)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at cq.main.main(Unknown Source)

#error {
 :cause org.msgpack.template.ListTemplate.<init>(org.msgpack.template.Template)
 :via
 [{:type java.lang.RuntimeException
   :message java.lang.IllegalArgumentException: java.lang.NoSuchMethodException: org.msgpack.template.ListTemplate.<init>(org.msgpack.template.Template)
   :at [com.cognitect.transit.TransitFactory writer TransitFactory.java 104]}
  {:type java.lang.IllegalArgumentException
   :message java.lang.NoSuchMethodException: org.msgpack.template.ListTemplate.<init>(org.msgpack.template.Template)
   :at [org.msgpack.template.GenericCollectionTemplate <init> GenericCollectionTemplate.java 33]}
  {:type java.lang.NoSuchMethodException
   :message org.msgpack.template.ListTemplate.<init>(org.msgpack.template.Template)
   :at [java.lang.Class getConstructor0 DynamicHub.java 3349]}]
 :trace
 [[java.lang.Class getConstructor0 DynamicHub.java 3349]
  [java.lang.Class getConstructor DynamicHub.java 2151]
  [org.msgpack.template.GenericCollectionTemplate <init> GenericCollectionTemplate.java 30]
  [org.msgpack.template.TemplateRegistry registerTemplatesWhichRefersRegistry TemplateRegistry.java 151]
  [org.msgpack.template.TemplateRegistry registerTemplates TemplateRegistry.java 140]
  [org.msgpack.template.TemplateRegistry <init> TemplateRegistry.java 81]
  [org.msgpack.template.TemplateRegistry <init> TemplateRegistry.java 93]
  [org.msgpack.MessagePack <init> MessagePack.java 60]
  [com.cognitect.transit.impl.WriterFactory getMsgpackInstance WriterFactory.java 77]
  [com.cognitect.transit.TransitFactory writer TransitFactory.java 95]
  [cognitect.transit$writer invokeStatic transit.clj 161]
  [cognitect.transit$writer invoke transit.clj 139]
  [cognitect.transit$writer invokeStatic transit.clj 155]
  [cognitect.transit$writer invoke transit.clj 139]
  [cq.formats$__GT_transit_writer$fn__14686 invoke formats.clj 154]
  [cq.formats$format__GT_writer$fn__14701 invoke formats.clj 199]
  [cq.core$run invokeStatic core.clj 97]
  [cq.core$run invoke core.clj 93]
  [cq.main$main invokeStatic main.clj 109]
  [cq.main$main invoke main.clj 99]
  [cq.main$_main invokeStatic main.clj 114]
  [cq.main$_main doInvoke main.clj 111]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [cq.main main nil -1]]}

make `clojure.instant`, or some other string ⇔ date/time library, available (or provide examples.)

example:

> cq ":docs first :faction_presence (map :updated_at) first"
"2023-03-11T03:01:11.000Z"

Easily handled by clojure.instant/read-instant-timestamp, but making that available in a cq one-liner is ... more challenging than I can manage at this hour. :)

I'd like to handle those fields as dates, so I can sensibly do things like "filter everything less than three days old" from the source material.

Provide binaries for various architectures

Might be a good idea to ship binaries for ARM (new M1 macbook) and other non x86_64 archs.

Create a homebrew-brew repository

Please create a repository such as this one https://github.com/pedromanoel/homebrew-brew so people can install your awesome tool using homebrew!

The install process becomes brew install markus-wa/brew/cq

Pretty indendation and coloring should be separate options

Currently the only options are to either indent and colorize, or not do either. Colors make it difficult to work with files, or really anything that cannot handle the terminal/ANSI color codes. Would be nice to have command line options to have the pretty indentation without colors.

Or perhaps logic that can see if stdout is attached to a TTY, but looks like that can get pretty hairy in JVM. I guess even more so with graalvm.

Add colored output for edn

Would you be open to include https://github.com/greglook/puget for colored edn output?

I patched it into my local copy and the output looks like this:

`-i json` fails with given examples

Something I noticed is that if you take one of the examples (from README, or CLI help) such as

echo '{a: {b: 2, c: 3}}' | cq ':a  (m/map-kv #f [%2 %1])'
{2 :b, 3 :c}

then it works, but if you specify the input format -i json it doesn't. With quoted "a" etc. it will work.

echo '{a: {b: 2, c: 3}}' | cq -i json ':a  (m/map-kv #f [%2 %1])'
#error {
 :cause JSON error (unexpected character): a
 :via
 [{:type java.lang.Exception
   :message JSON error (unexpected character): a
   :at [clojure.data.json$_read invokeStatic json.clj 230]}]
 :trace
 [[clojure.data.json$_read invokeStatic json.clj 230]
  [clojure.data.json$_read invoke json.clj 181]
  [clojure.data.json$read_object invokeStatic json.clj 97]
  [clojure.data.json$read_object invoke json.clj 74]
  [clojure.data.json$_read invokeStatic json.clj 224]
  [clojure.data.json$_read invoke json.clj 181]
  [clojure.data.json$read invokeStatic json.clj 276]
  [clojure.data.json$read doInvoke json.clj 232]
  [clojure.lang.RestFn invoke RestFn.java 439]
  [cq.formats$__GT_json_reader$fn__15904 invoke formats.clj 19]

 ...

Should this happen?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.