I made ppx_tools build with 4.05 in this branch:

https://github.com/diml/ppx_tools/tree/4.05

You might want to import this as a 4.05 branch

Right now the type errors are confusing:

File "_none_", line 1:
Error: This pattern matches values of type Parsetree.expression
       but a pattern was expected which matches values of type
         Parsetree.expression_desc

Currently the version on opam does not work with 4.08.0. I believe a tag needs to be created.

On a 4.08.0+beta1 switch:

File "ast_convenience.ml", line 1:
Error: The implementation ast_convenience.ml
       does not match the interface ast_convenience.cmi:
       Values do not match:
         val has_attr : 'a -> ('a Location.loc * 'b) list -> bool
       is not included in
         val has_attr : string -> Parsetree.attributes -> bool
       File "ast_convenience.mli", line 108, characters 0-42:
         Expected declaration
       File "ast_convenience.ml", line 121, characters 4-12:
         Actual declaration

problem

i'm trying out ocaml 5, but ppx_tools is not yet compatible

discussion

no discussion, really. just a tracking thread :)

Hi,
I tried to install ppx_tools.5.0+4.02.3 with opam (which is requiried by js_of_ocaml 2.8) and I got the following error.

#=== ERROR while installing ppx_tools.5.0+4.02.3 ==============================#
# opam-version 1.2.2
# os           linux
# command      make all
# path         /home/julien/.opam/4.02.3/build/ppx_tools.5.0+4.02.3
# compiler     4.02.3
# exit-code    2
# env-file     /home/julien/.opam/4.02.3/build/ppx_tools.5.0+4.02.3/ppx_tools-3392-a00279.env
# stdout-file  /home/julien/.opam/4.02.3/build/ppx_tools.5.0+4.02.3/ppx_tools-3392-a00279.out
# stderr-file  /home/julien/.opam/4.02.3/build/ppx_tools.5.0+4.02.3/ppx_tools-3392-a00279.err
### stdout ###
# ocamlc -bin-annot -w +A-4-17-44-45 -I +compiler-libs -safe-string -c ast_convenience.mli
# ocamlc -bin-annot -w +A-4-17-44-45 -I +compiler-libs -safe-string -c ast_convenience.ml
# ocamlc -bin-annot -w +A-4-17-44-45 -I +compiler-libs -safe-string -c ast_mapper_class.mli
# ocamlc -bin-annot -w +A-4-17-44-45 -I +compiler-libs -safe-string -c ast_mapper_class.ml
# Makefile:69 : la recette pour la cible « ast_mapper_class.cmo » a échouée
### stderr ###
# File "ast_mapper_class.ml", line 337, characters 6-17:
# Error: This variant pattern is expected to have type Parsetree.pattern_desc
#        The constructor Ppat_effect does not belong to type Parsetree.pattern_desc
# make: *** [ast_mapper_class.cmo] Erreur 2

I think most packages install .mli files in lib/pkg, for documentation or something. People complain about this.

I think
ocamlfind -c -package ppx_tools.metaquot my_ppx_code.ml
should be
ocamlfind c -c -package ppx_tools.metaquot my_ppx_code.ml

ie:

    | Ptyp_arrow (lab, t1, t2) ->
        arrow ~loc ~attrs lab (sub # typ t1) (sub # typ t2)

should be written

    | Ptyp_arrow (lab, t1, t2) ->
        let t1' = sub # typ t1 in
        let t2' = sub # typ t2 in
        arrow ~loc ~attrs lab t1' t2'

I have quite a few hacks in my code that could be removed after 9b8bdd9 and 3e733c7 make it into a release; the rewriter tool is also useful enough, I think.

Is it just missing or is there any difficulty to have it?

It would be great if I could do e.g. [%expr Some [%e mk_reader field [%expr reader]]],
so ppx_metaquot should probably expand [%expr]s inside antiquotations too.

OCaml 4.10 has been branched off the main development branch (trunk) a few days ago. We would like to start preparing a beta version for users to try and check their own software. ppx_tools currently (at least versions of opam) does not support the current 4.10. Would it be possible to have a release that is compatible with the current 4.10 branch? (we don't expect the parsetree to change before the release, although that's a possibility given feedback from the ppx toolchain)

As ocamlfind 1.5.1 and base-bytes are finally merged, I think it's time to make a release.

It seems the rewriter does not include the source file in location information for some reason.

When I use ocamlc to spit out new source and print a location (in my AST mapper) I get:

File "examples/test.ml", line 3, characters 0-166

When I use the rewriter I get:

File "", line 3, characters 0-166

I just started looking at the opam-builder reports for 4.05.0+beta2, and the fact that no 4.05-ready release of ppx_tools exists is blocking a lot of packages. This is a testament to the usefulness of the package, of course, but it would be very nice if we could get a 4.05.0-ready release very soon, to evaluate the remaining transition hurdles for the part of the ecosystem that depends on it.

The proper META syntax for "an executable locally installed in this package" is ppx = "ppx_metaquot", not ./ppx_metaquot. Currently the findlib integration doesn't work.

This is the dual issue of #75: pexp_loc_stack, ptyp_loc_stack, ppat_loc_stack need to be present code generated by the ppx, but aren't. For example:

let x = [%expr "test"]

fails at compile-time with the error Some record fields are undefined: pexp_loc_stack.

The main point of breakage for ppxs in 4.03 is going to be the new type for labeled arguments. I think we could ease a bit the transition with some additions to ppx_tools, in both the 4.02.3 versions and the 4.03 version:

type label = 
  string (* in 4.02 *)
  Asttypes.arg_label (* in 4.03 *)

type label_desc = (* = Asttypes.arg_label for 4.03 only *)
    Nolabel
  | Labelled of string 
  | Optional of string

val explode : label -> label_desc

val nolabel : label
val labelled : string -> label
val optional : string -> label

By using only these functions, it should be possible to do almost all manipulations on functions with 4.02 and 4.03 compatibility.

@alainfrisch @diml opinions ?

I believe metaquot is currently broken on 4.08.
One should ignore pexp_loc_stack, ptyp_loc_stack, ppat_loc_stack when generating patterns on the ast. see what's done for location and attributes: https://github.com/ocaml-ppx/ppx_tools/blob/master/ppx_metaquot.ml#L175

Also see the patch in ppx_tools_versionned https://github.com/ocaml-ppx/ppx_tools_versioned/pull/22/files#diff-7ab097557bb7183454f885e4595acc38R180

I would be willing to submit PRs for this, but want to discuss first.

Since 4.04 has Ppat_open, and 4.03 and 4.02 do not, would it be reasonable to add two functions to Ast_convenience:

1. Some kind of higher-order pattern deconstructor, that hides the pattern matching on Parsetree.pattern_desc inside it. The implemention for < 4.04 would simply never call the closure corresponding to Ppat_open.
2. A wrapper around Ast_helper.Pat.open_. The implementation for < 4.04 would fail at run time. Perhaps we could mark that version with @@ocaml.deprecated to warn about using it.

This kind of thing would help with aantron/bisect_ppx#113, which concerns, in part, pattern matching on patter_desc in 4.04 while preserving compatibility with 4.03 and 4.02.

cc @rleonid

I realize this is not the right place to ask this question, but unfortunately I don't know what the right place is. We're considering moving from camlp4 to ppx for the frenetic project. Currently we use camlp4 to embed our DSL (namely NetKAT) into OCaml, so we can write things like

let p1 = <:netkat< port:=2 >>
let p2 = <:netkat< ip4Dst:=10.0.0.1 >>
let p = <:netkat< switch=1; $p1$ + switch=2; $p2$  >>

Camlp4 will parse the DSL at compile time and replace the code snippets with their less pretty AST versions:

let p1 = Filter(Port, 2)
let p2 = Filter(Ip4Dst, 167772161L)
let p = Union (Seq (Filter(Switch,1), p1), Seq (Filter(Switch,2), p2))

Is ppx powerful enough to support this kind of use case? In particular, can ppx support antiquotations, as in the definition of p above? I have seen that with ppx we could have something like

let%netkat p = ip4Dst=10.0.0.1 

but I couldn't find anything on antiquotations.

Sorry for asking this question here, and thanks in advance for any help or pointers you might be able to provide.

It seems rewriter exits with status 0 and an empty output file even when the ppx command fails

$ ocamlfind ppx_tools/rewriter -ppx whatever -impl strange_input.ml -o strange_input.out.ml
File "", line 5, characters 15-17:
Error: Syntax error

Even it says Error, it exits with 0, which makes it hard to use in a build system...

ppx_tools fails to compile with 4.04.0+trunk as of last night with this error:

ocamlc -bin-annot -annot -w +A-4-17-44-45 -I +compiler-libs -safe-string -c ast_mapper_class.ml
ocamlc -bin-annot -annot -a -o ppx_tools.cma ast_convenience.cmo ast_mapper_class.cmo
ocamlc -bin-annot -annot -w +A-4-17-44-45 -I +compiler-libs -safe-string -c genlifter.ml
ocamlc -bin-annot -annot -w +A-4-17-44-45 -I +compiler-libs -safe-string -o genlifter ocamlcommon.cma ppx_tools.cma genlifter.cmo
./genlifter -I +compiler-libs Parsetree.expression > ast_lifter.ml || rm -rf ast_lifter.ml
ocamlc -bin-annot -annot -w +A-4-17-44-45 -I +compiler-libs -safe-string -c ast_lifter.ml
ocamlc -bin-annot -annot -w +A-4-17-44-45 -I +compiler-libs -safe-string -c dumpast.ml
ocamlc -bin-annot -annot -w +A-4-17-44-45 -I +compiler-libs -safe-string -o dumpast ocamlcommon.cma ocamlbytecomp.cma ast_lifter.cmo dumpast.cmo
ocamlc -bin-annot -annot -w +A-4-17-44-45 -I +compiler-libs -safe-string -c ppx_metaquot.ml
ocamlc -bin-annot -annot -w +A-4-17-44-45 -I +compiler-libs -safe-string -o ppx_metaquot ocamlcommon.cma ppx_tools.cma ast_lifter.cmo ppx_metaquot.cmo
ocamlc -bin-annot -annot -w +A-4-17-44-45 -I +compiler-libs -safe-string -c rewriter.ml
File "rewriter.ml", line 95, characters 16-44:
Error: This expression has type string but an expression was expected of type
         'a Pparse.ast_kind

Would be useful to have support to unblock testing 4.04 features (e.g. cstruct depends on this)

The payload of Pconst_* nodes in the Parsetree is changing on a regular basis: Pconst_int changed, now it is the turn of Pconst_string, etc. Having support for literal quotation would be nice as it would let us write version-robust code to create literals or pattern-match on them, in the simple case where we don't need all the data/knobs of the full datatype. (Typically to parse attribute information encoded in literals.)

( @thierry-martinez and myself were discussing this is ocaml-ppx/ppx_deriving_protobuf#36 (comment) )

I would propose the following interface: [%lit.{integer,int,int32,in64,nativeint,float,char,string} <payload>], available in both expressions and patterns (with a question mark). This is one quotation per constructor in the Ast_helper.Const module (suggestion from @thierry-martinez), and I would propose to use the same payload type as the non-default parameter of the constructor (in particular, string for %lit.integer and int for %lit.int).

If presented with an empty expression list, it should either assert or return the function. I vote for the latter, as it's very convenient for partial applications with unknown number of arguments.

When I tried installing ppx_tools using OPAM on UBUNTU 14.04 I get the above error message. Is ppx_tools not compatible with newer Ubuntu version or do I need to install something else beforehand?

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04 LTS
Release: 14.04
Codename: trusty

.# To complete the configuration of OPAM, you need to run:
eval opam config env --root=/usr/.opam
$ eval opam config env --root=/usr/.opam
$ sudo opam install --root=/usr/.opam ppx_tools
[ERROR] "ppx_tools" is not available for your compiler or your OS.
'opam install --root=/usr/.opam ppx_tools' failed.

It would be very nice to be able to write

$ ocamlfind query ppx_tools/rewriter -ppx ppx_deriving.show foo.ml`

rather than

$ ocamlfind ppx_tools/rewriter \
   -ppx '`ocamlfind query ppx_deriving`/ppx_deriving
   `ocamlfind query ppx_deriving`/ppx_deriving_show.cma' \
   foo.ml

Hello,

https://github.com/alainfrisch/ppx_tools/blob/master/ppx_metaquot.ml refers to metaquot_test.ml which does not seem to be in the repository.

ppx_tools.5.0 upstream url is pointing to a non-stable GitHub url:

$ opam info ppx_tools.5.0 -f upstream-url
https://github.com/ocaml-ppx/ppx_tools/tarball/f70ca1d07d565989a0041e312e8efc00ff2ad87e

that archive is re-generated by GitHub regularly so its md5 is regularly changing. Would it be possible to use a proper release instead? The current 5.0 release doesn't point to the same SHA1 (not sure why) but it could be nice if it would -- it's seem that GitHub cache for release tarball is much more long-lived in practice (we don't see many md5 changes). It is probably better to upload an archive to that release to be sure that the md5 will never change anymore. topkg publish could help to do this automatically.

There are a few problems with OCaml 4.11. I added the following very hacky patch to Fedora. Use with caution!
https://src.fedoraproject.org/rpms/ocaml-ppx-tools/blob/master/f/ppx_tools-6.1-ocaml-4.11.patch

It would be useful to have antiquotations [%int ...], [%string ...] and so on instead of just [%e ...]. This would simplify

let s' = Ast_convenience.str s in [%expr Foo [%e s']]

to just

[%expr Foo [%string s]]

currently ppx_metaquot.ml is hard coded with Ast_mapper.run_main which makes it can not be reused as a library.

the only way to use ppx_metaquot is ocamlc.opt -ppx ppx_metaquot (it will imply type checking which is a bit annoying)

My use case is that I want to have a command line utilty to do pure source transformation,
so I can check in generated code without imposing dependencies to my users, are you interested in such use case? thanks

The current ppx_tools package is not yet available for OCaml 4.07, and some existing PPX drivers are not yet using ppx_tools_versioned, so I thought it would be helpful to create an issue report to cover the task of adding support for OCaml 4.07.

I have a test file, test.ml:

(* test.ml *)
let x = [%expr 2 + 2]

When I try to build it with ocamlbuild, I see an error:

anand@localhost> ocamlbuild  -package ppx_tools -package ppx_tools.metaquot test.byte

+ /home/anand/.opam/4.04.0/bin/ocamlc.opt -c -I /home/anand/.opam/4.04.0/lib/ocaml/compiler-libs -I /home/anand/.opam/4.04.0/lib/ppx_tools -o test.cmo test.ml
File "test.ml", line 1, characters 10-14:
Error: Uninterpreted extension 'expr'.
Command exited with code 2.

I have ppx_tools 5.0, ocamlbuild 0.11.0 and ocaml 4.04.0. Thanks in advance!

With the following example:

type t1 = int[@@deriving yojson]
type t2 = string[@@deriving yojson]

type ('a, 'b) msg = [ `A of 'a | `B of 'b][@@deriving yojson]

type t = (t1, t2) msg [@@deriving yojson]

The generated msg_to_yojson function is typed as:

val msg_to_yojson :
  ('a -> Yojson.Safe.json) ->
  ('a -> Yojson.Safe.json) -> [< `A of 'a | `B of 'a ] -> Yojson.Safe.json

Thus the the generated function t_of_yojson cannot be typed, as the type of msg_to_yojson should rather be

val msg_to_yojson :
  ('a -> Yojson.Safe.json) ->
  ('b -> Yojson.Safe.json) -> [< `A of 'a | `B of 'b ] -> Yojson.Safe.json

It means we can't put restriction over the ppx_tools versions. The versioning should use be something like VERSION+COMPILER instead.

ocamlc -w +A-4-17-44-45 -I +compiler-libs -safe-string -c ast_lifter.ml
File "ast_lifter.ml", line 44, characters 18-21:
Error: Syntax error: operator expected.

Seems to be a bug in Pprintast, but I may be wrong.

I've recently gotten started using PPX and metaquot so I apologize in advance if I'm just making a stupid mistake, but it seems to me like the [%%s ... ] antiquotation isn't begin expanded.

I'm using OCaml 4.03.0 and ppx_tools 5.0+4.03.0.

I tried to produce a fairly small example of the problem. I have my ppx file:

> open Ast_mapper
> open Parsetree
> 
> let test_mapper argv =
>   { default_mapper with
>     structure = fun mapper strl ->  
>       let rec stri_replace acc str =
>         match str with
>         | { pstr_desc = Pstr_extension (({txt = "test"},_ ),_) } -> 
>           let x = [%stri type x = string] in
>           [%str [%%s [x]]] @ acc
>         | x -> default_mapper.structure_item mapper x :: acc
>       in
>       List.rev (List.fold_left stri_replace [] strl)
>   }
> 
> let () = register "test" test_mapper

And when I run the ppx_rewriter on my test file:
[%%test ]

using:

ocamlfind ppx_tools/rewriter ./ppx_forest.native examples/test.ml

I get:
[%%s [x]]
instead of:
type x = string

which is what I would have expected. Am I just doing something stupid or is it not working right?