jcburley / joker Goto Github PK

From @jcburley on November 8, 2018 6:13

When converting named types such as structs, a single function should be emitted to perform any potentially length conversion, and the wrapper functions changed to call that single function.

Consider this code:

func interfaceByIndex(index int) Object {
	res1, res2 := net.InterfaceByIndex(index)
	res := EmptyVector
	map1 := EmptyArrayMap()
	map1.Add(MakeKeyword("Index"), MakeInt((*res1).Index))
	map1.Add(MakeKeyword("MTU"), MakeInt((*res1).MTU))
	map1.Add(MakeKeyword("Name"), MakeString((*res1).Name))
	vec2 := EmptyVector
	for _, el2 := range (*res1).HardwareAddr {
		vec2 = vec2.Conjoin(MakeInt(el2))
	}
	map1.Add(MakeKeyword("HardwareAddr"), vec2)
	map1.Add(MakeKeyword("Flags"), MakeInt((*res1).Flags))
	res.Conjoin(map1)
	res.Conjoin(string(res2))
	return res
}

The block beginning with map1 appears several times in different wrapper functions. It should be implemented in a single function, e.g. func interface(res Interface) (map Object) { ... }, and called by the above code, e.g. map1 := interface(el1). That would probably (unless Go optimizes this away itself) reduce code size noticeably, as more APIs are implemented.

Copied from original issue: jcburley/gostd2joker#5

The new gotypes.go module should be a good replacement for a number of ad-hoc mechanisms in the code, besides the new stuff supporting constructors, which could also stand some refactoring (see the code near the end of codegen.go).

While at it, maybe get rid of the dead code in pre.go, post.go, etc.

From @jcburley on November 6, 2018 14:46

E.g.:

type A struct {
  B *B
}

type B struct {
  A *A
}

An example of this is found in http.Request and http.Response, which refer to each other (via pointers/references).

Copied from original issue: jcburley/gostd2joker#1

to make the quickstart work I had to

1. set export GO111MODULE=off

2. replace git co gostd with git checkout gostd

3. then I get:

./run.sh
# github.com/candid82/joker/core
./g_goswitch.go:8:20: undefined: GoTypeInfo
core/object.go:3: running "go": exit status

Originally described here:

https://stackoverflow.com/questions/61342000/why-might-exec-command-start-hang-on-darwin

Having isolated what seems to be the offending commit in Golang's repo, and (more recently) the specific package (plugin) that gostd includes (but official Joker does not), it seems the most straightforward workaround for now is to simply exclude plugin when building on the Darwin OS.

Assuming the bug is someday fixed upstream (in Go and/or, conceivably, Darwin itself), plugin inclusion can then be restored in the gostd version of Joker.

It could be helpful, for those exploring the Go API offerings of Joker, to say more than Constructor for .... E.g. list the order and types of elements for a vector (and, someday when supported, keys for a map).

From @jcburley on November 6, 2018 16:42

No processing/tracking of import statements (in the Go code being processed) is done.

Therefore, unqualified references to types, functions, etc., are implicitly qualified based on the package statement for the file (since that's how the code-reading API being used organizes things). Thus, top-level imports (e.g. import ( _ "some-package" )) won't be searched.

Further, qualified references are treated as referring to the concrete packages to which they'd belong, in the absence of any import that renames them.

Copied from original issue: jcburley/gostd2joker#4

trying out std.net and ways to debug if things go wrong

user=> (require '[go.std.net :as n])
nil
user=> (def conn (n/Dial "tcp" "localhost:8000"))
#'user/conn
user=> (println (conn "GET / HTTP/1.0\r\n\r\n"))
<repl>:6:10: Eval error: Key must be integer

this comes vector.go, not sure how to proceed from there

Unsure of the proper name for it, but let's say (go-convert g) would, given (type g) => GoObject[...], return a canonical Clojure representation, such as an Int, Number, vector, map, etc.

E.g.:

user=> (use 'go.std.net)
nil
user=> (def d (Dial "tcp" "localhost:8000"))  ; assumes local port 8000 is accepting connections (e.g. HTTP server)
#'user/d
user=> (nil? (d 1))  ; else something went wrong
true
user=> (def c (d 0))
#'user/c
user=> (.SetKeepAlive c true)  ; invokes a method defined on the concrete type go.std.net/TCPConn
nil
user=> (.Close c)  ; invokes a receiver defined on another concrete type that implements interface/abstract type go.std.net/Conn
<file>:0:0: Eval error: No such member (field) GoObject[*net.TCPConn]/Close
user=>

Here, gostd is not noticing that TCPConn's struct embeds Conn, and thus does not generate info indicating that any methods (receivers) defined for interface type Conn should also be considered when looking up a method/receiver to invoke.

I believe this is a format error in the readme

Dash docset: dash-feed://https%3A%2F%2Fraw.githubusercontent.com%2Fcandid82%2Fjoker%2Fmaster%2Fdocs%2Fjoker.xml

Though plenty of progress has been made, there's still not nearly good-enough coverage (autowrapping) of the functions available in Go's std library.

Looking at amd64-linux and amd64-darwin, of the total functions available for wrapping (as they're public), only about 20% are non-methods (do not have receivers) and so are currently eligible for autowrapping.

Of that 20%, between 20% and 30% are successfully converted (exact numbers depend on target system).

That's around 4-6% (let's say 5%) of the total public functions being converted.

The main obstacles to further wrapping now appears to be:

• No support for native user-defined (std-defined) Go types in Joker, which blocks support for methods and for many functions that take such types as input arguments and/or return them among results. This is Issue #2 .
• Imports aren't processed properly so a reference to e.g. x.y is not handled. This is Issue #4 .
• Star types aren't supported correctly, so *Foo types for input arguments or returned results are not supported. This is Issue #3 .
• Mutual type reference is not supported. It's not clear this happens often enough to worry about yet (it might be in the "noise" of the above issues), but it would appear to block some useful capabilities being exposed to Joker. This is Issue #1 .

The next course of action would appear to be to define deftype and friends for Joker, including both Joker-defined and Go (host) types, so Clojure-style special operators (such as .) would work on them. The gostd tool would then autogenerate the appropriate code and data structures for std. It's "just" a matter of deciding what that stuff should look like.

In the meantime, I can work on the Imports issue, as it's somewhat orthagonal to the other issues -- only "somewhat", though, because it's unlikely to be of much (if any) benefit until the Types issue is addressed.

Star types and mutual type reference need to be studied further, probably informed by the direction Joker goes in terms of exposing Go types (and objects) to Joker code.

Once those four issues are resolved, the coverage of STD could easily exceed 90%, and the remaining issues preventing full coverage should comprise some combination of simpler things (like supporting [][]byte as an input-argument type) and things we just don't see the need to support.

Some types aren't in the "table" generated by gostd and so don't yield proper names. Perhaps fall back to using reflect?

Also, don't put the . before a typename if there's no "package" name. E.g. GoVar[.byte] should be GoVar[byte].

From @jcburley on November 6, 2018 14:55

E.g. net.MX is converted to its underlying type, so the information on MX itself is lost to the generated Joker code.

So, instead of getting back an array of (pointers/references to) MX objects, Joker code sees an array of maps with the underlying (struct) type's field names, converted to keywords, as keys.

For quick and dirty low-level conversion, this might be okay at first, but we should think carefully about the long term here to ensure that, if Joker ever does support proper types/classes/objects/etc., either existing Joker code (that references returned objects without regard to formal types) will continue to work, or we'll be okay with "breaking" such code.

(If we wrap the code people care about in higher-level Joker std wrappers, and support only those, we can change those implementations if and when the underlying interfaces change, and avoid breaking user expectations.)

Copied from original issue: jcburley/gostd2joker#2

Given e.g. (go.std.os/LinkError. ["one" "two"]), this diagnostic seems insufficient:

<repl>:1:1: Eval error: Index 2 is out of bounds [0..1]

It should specify how many elements are required in the vector.

Further, a type mismatch in one of the elements fails to specify which one has the problem:

user=> (go.std.os/LinkError. ["one" "two" :hey "there"])
<repl>:6:1: Eval error: Expected String, got Keyword
user=>

A suffix such as (element 2) would help.

From @jcburley on November 6, 2018 14:58

E.g. a return type of *int suggests that the underlying value could be changed by Joker code, but currently the generated Joker code basically ignores the * and returns the underlying type (int in this case).

In some cases (e.g. the return type for net.LookupMX()), this is probably okay, as the underlying API is using a pointer just to save some time/space and doesn't seem to offer any utility to its caller changing the copy.

Copied from original issue: jcburley/gostd2joker#3

(new typename ...) should be the same as (typename. ...).

goobject? would be something like class?. See:

user=> (doc class?)
-------------------------
clojure.core/class?
([x])
  Returns true if x is an instance of Class
nil
user=>

Per Clojure:

user=> (doc .)
-------------------------
.
  (.instanceMember instance args*)
  (.instanceMember Classname args*)
  (Classname/staticMethod args*)
  Classname/staticField
Special Form
  The instance member form works for both fields and methods.
  They all expand into calls to the dot operator at macroexpansion time.

Oddly, the above does not actually document the . special form itself, which is explained in detail here:

https://clojure.org/reference/java_interop

Per subject.

So, given a var g that is a GoObject[Some-go-type], one should be able to do:

user=> (.Method-name g args...)

This would invoke func (t Some-go-type) Method-name(args...), return the result, etc.

The current workflow is for gostd to be run when Joker is built, and for it to read, parse, and generate code to wrap only those Go APIs that are supported for the current architecture (see the GOARCH and GOOS variables as shown by go env ...).

This lengthens build times for everyone who builds Joker is somewhat intrusive on the Joker code base (it updates at least two source files).

It might be better to redesign gostd to be run just before a release. Instead of focusing on only one target (the current target), it would read, parse, and generate code to wrap all the Go APIs, by outputting suitably conditionally-compiled code (and automatically git add whatever new files it creates?).

For example, instead of std/go/std/os/os_native.go being generated, it might generate files corresponding to exactly what it finds in $GOROOT/src/os/, with the same build (and version?) restrictions, if any, to contain wrappers for types and functions defined in the corresponding Go source file.

That would leave std/go/std/os.joke to either be split out into different files, or perhaps each defn would be tagged with suitable info, such that different std/go/std/os/a_*.go files would be generated with appropriate restrictions.

This approach would probably result in some duplication of code (wrappers, conversions, constructors, etc.), if different implementations in the Go source tree shared the same API, such that a single wrapper would do, but would be duplicated into each *.go file corresponding to the restrictions for each defining file in the Go source tree. But that code duplication would not result in the Joker itself getting any larger (since it would include only one instance of each duplicated wrapper) -- just the source distribution.