Git Product home page Git Product logo

grumpy's Introduction

Grumpy: Go running Python

Build Status Join the chat at https://gitter.im/grumpy-devel/Lobby

Overview

Grumpy is a Python to Go source code transcompiler and runtime that is intended to be a near drop-in replacement for CPython 2.7. The key difference is that it compiles Python source code to Go source code which is then compiled to native code, rather than to bytecode. This means that Grumpy has no VM. The compiled Go source code is a series of calls to the Grumpy runtime, a Go library serving a similar purpose to the Python C API (although the API is incompatible with CPython's).

Limitations

Things that will probably never be supported by Grumpy

  1. exec, eval and compile: These dynamic features of CPython are not supported by Grumpy because Grumpy modules consist of statically-compiled Go code. Supporting dynamic execution would require bundling Grumpy programs with the compilation toolchain, which would be unwieldy and impractically slow.

  2. C extension modules: Grumpy has a different API and object layout than CPython and so supporting C extensions would be difficult. In principle it's possible to support them via an API bridge layer like the one that JyNI provides for Jython, but it would be hard to maintain and would add significant overhead when calling into and out of extension modules.

Things that Grumpy will support but doesn't yet

There are three basic categories of incomplete functionality:

  1. Language features: Most language features are implemented with the notable exception of old-style classes. There are also a handful of operators that aren't yet supported.

  2. Builtin functions and types: There are a number of missing functions and types in __builtins__ that have not yet been implemented. There are also a lot of methods on builtin types that are missing.

  3. Standard library: The Python standard library is very large and much of it is pure Python, so as the language features and builtins get filled out, many modules will just work. But there are also a number of libraries in CPython that are C extension modules which will need to be rewritten.

  4. C locale support: Go doesn't support locales in the same way that C does. As such, some functionality that is locale-dependent may not currently work the same as in CPython.

Running Grumpy Programs

Method 1: make run:

The simplest way to execute a Grumpy program is to use make run, which wraps a shell script called grumprun that takes Python code on stdin and builds and runs the code under Grumpy. All of the commands below are assumed to be run from the root directory of the Grumpy source code distribution:

echo "print 'hello, world'" | make run

Method 2: grumpc and grumprun:

For more complicated programs, you'll want to compile your Python source code to Go using grumpc (the Grumpy compiler) and then build the Go code using go build. Since Grumpy programs are statically linked, all the modules in a program must be findable by the Grumpy toolchain on the GOPATH. Grumpy looks for Go packages corresponding to Python modules in the __python__ subdirectory of the GOPATH. By convention, this subdirectory is also used for staging Python source code, making it similar to the PYTHONPATH.

The first step is to set up the shell so that the Grumpy toolchain and libraries can be found. From the root directory of the Grumpy source distribution run:

make
export PATH=$PWD/build/bin:$PATH
export GOPATH=$PWD/build
export PYTHONPATH=$PWD/build/lib/python2.7/site-packages

You will know things are working if you see the expected output from this command:

echo 'import sys; print sys.version' | grumprun

Next, we will write our simple Python module into the __python__ directory:

echo 'def hello(): print "hello, world"' > $GOPATH/src/__python__/hello.py

To build a Go package from our Python script, run the following:

mkdir -p $GOPATH/src/__python__/hello
grumpc -modname=hello $GOPATH/src/__python__/hello.py > \
    $GOPATH/src/__python__/hello/module.go

You should now be able to build a Go program that imports the package "__python__/hello". We can also import this module into Python programs that are built using grumprun:

echo 'from hello import hello; hello()' | grumprun

grumprun is doing a few things under the hood here:

  1. Compiles the given Python code to a dummy Go package, the same way we produced __python__/hello/module.go above
  2. Produces a main Go package that imports the Go package from step 1. and executes it as our __main__ Python package
  3. Executes go run on the main package generated in step 2.

Developing Grumpy

There are three main components and depending on what kind of feature you're writing, you may need to change one or more of these.

grumpc

Grumpy converts Python programs into Go programs and grumpc is the tool responsible for parsing Python code and generating Go code from it. grumpc is written in Python and uses the pythonparser module to accomplish parsing.

The grumpc script itself lives at tools/grumpc. It is supported by a number of Python modules in the compiler subdir.

Grumpy Runtime

The Go code generated by grumpc performs operations on data structures that represent Python objects in running Grumpy programs. These data structures and operations are defined in the grumpy Go library (source is in the runtime subdir of the source distribution). This runtime is analogous to the Python C API and many of the structures and operations defined by grumpy have counterparts in CPython.

Grumpy Standard Library

Much of the Python standard library is written in Python and thus "just works" in Grumpy. These parts of the standard library are copied from CPython 2.7 (possibly with light modifications). For licensing reasons, these files are kept in the third_party subdir.

The parts of the standard library that cannot be written in pure Python, e.g. file and directory operations, are kept in the lib subdir. In CPython these kinds of modules are written as C extensions. In Grumpy they are written in Python but they use native Go extensions to access facilities not otherwise available in Python.

Source Code Overview

  • compiler: Python package implementating Python -> Go transcompilation logic.
  • lib: Grumpy-specific Python standard library implementation.
  • runtime: Go source code for the Grumpy runtime library.
  • third_party/ouroboros: Pure Python standard libraries copied from the Ouroboros project.
  • third_party/pypy: Pure Python standard libraries copied from PyPy.
  • third_party/stdlib: Pure Python standard libraries copied from CPython.
  • tools: Transcompilation and utility binaries.

Contact

Questions? Comments? Drop us a line at [email protected] or join our Gitter channel

grumpy's People

Contributors

aisk avatar alanjds avatar alexekoren avatar brettcannon avatar choleraehyq avatar corona10 avatar feilengcui008 avatar florianludwig avatar garburator avatar jbremer avatar johnlunney avatar lalaithion avatar leekchan avatar m4ns0ur avatar meadori avatar miguelammatos avatar mirkodziadzka avatar nairb774 avatar ns-cweber avatar paulsmith avatar ryankennedy avatar s-you avatar stevenmaude avatar trotterdylan avatar wuttem avatar xyproto avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

grumpy's Issues

how to run make

how to run 'make run' in Mac os?

My steps:

  1. clone grumpy into local directory, for example ~/test
  2. cd test
  3. echo "print 'hello, world'" | make run

failed:

sxy@Mac ~/test/grumpy$ echo 'print "hello world"' | make run
build/src/grumpy/lib/errno/module.go:174: undefined: syscall.EADV
build/src/grumpy/lib/errno/module.go:190: undefined: syscall.EBADE
build/src/grumpy/lib/errno/module.go:198: undefined: syscall.EBADFD
build/src/grumpy/lib/errno/module.go:206: undefined: syscall.EBADR
build/src/grumpy/lib/errno/module.go:210: undefined: syscall.EBADRQC
build/src/grumpy/lib/errno/module.go:214: undefined: syscall.EBADSLT
build/src/grumpy/lib/errno/module.go:218: undefined: syscall.EBFONT
build/src/grumpy/lib/errno/module.go:234: undefined: syscall.ECHRNG
build/src/grumpy/lib/errno/module.go:238: undefined: syscall.ECOMM
build/src/grumpy/lib/errno/module.go:258: undefined: syscall.EDEADLOCK
build/src/grumpy/lib/errno/module.go:258: too many errors
make: *** [build/pkg/darwin_amd64/grumpy/lib/errno.a] Error 1

Python 3 compatibility?

Alright, I'll straight up ask: What's the status of Python 3 compatibility for this runtime? Never/Maybe someday/Working on it?

Running when Python 3 is the default

It doesn't take much to make grumpy work under a non-default installation of 2.7, so I thought it could be somehow supported or at least described in the README.

I have Python 3 as my default (as in, python or /usr/bin/env python will invoke Python 3), but I do have python2.7 installed as well. I presume this is a setup common for a lot of people, so I thought grumpy could be supported on system configurations like this.

There are only two changes needed:

  • in the Makefile, all calls to @python need to be @python2.7 - but to make this universal, the invocation of python here could be parametrised by using a variable, so that one could set the python version of choice in the Makefile header.
  • tools/* use #!/usr/bin/env python, which will point to Python 3, so I had to change this to /usr/bin/env python2.7 and I'm not quite sure what's the best way to make this parametrised. Perhaps using the aforementioned variable and using find/replace when copying the tool scripts to the build folder

I hardcoded these two changes and tested it out, worked like a charm.

Fields for pointer-to-struct objects should be accessible

For example, http.Response object fields like Body should be accessible from *Response objects. The following should work:

from __go__.net.http import Get
from __go__.io import Copy
from __go__.os import Stdout
rsp, _ = Get("http://example.com/")
Copy(Stdout, rsp.Body)

Use python function as go function

I'm trying to implement a simple HTTP server, but it seems I can't send a python function as a go function to HandleFunc.

from __go__.net.http import ListenAndServe
from __go__.githubcom.gorilla.mux import NewRouter
from __go__.fmt import Fprintf

def HomeHandler(w, r):
    Fprintf(w, "Hello world")

r = NewRouter()
r.HandleFunc("/", HomeHandler)
ListenAndServe(":8000", r)

I'm receiving this error:

$ cat web.py | make run
TypeError: cannot convert grumpy.Object to func(http.ResponseWriter, *http.Request)
exit status 1
make: *** [run] Error 1

Support tuple args in lambda

Per justinsaccount on HN:

$ make run
c = {}
top = sorted(c.items(), key=lambda (k,v): v)
^D
Traceback (most recent call last):
File "./tools/grumpc", line 102, in
sys.exit(main(parser.parse_args()))
File "./tools/grumpc", line 60, in main
visitor.visit(mod)
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 302, in visit_Module
self._visit_each(node.body)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 632, in _visit_each
self.visit(node)
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stin visit_Assign
with self.expr_visitor.visit(node.value) as value:
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 101, in visit_Call
values.append((util.go_str(k.arg), self.visit(k.value)))
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 246, in visit_Lambda
return self.visit_function_inline(func_node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 388, in visit_function_inline
func_visitor = block.FunctionBlockVisitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/block.py", line 432, in init
args = [a.id for a in node_args.args]
AttributeError: 'Tuple' object has no attribute 'id'

cant find import "grumpy/lib/_collections"

jyf@pc:~/repo/git/grumpy$ make
build/src/grumpy/lib/itertools/module.go:4: can't find import: "grumpy/lib/_collections"
Makefile:192: recipe for target 'build/pkg/linux_amd64/grumpy/lib/itertools.a' failed
make: *** [build/pkg/linux_amd64/grumpy/lib/itertools.a] Error 1

jyf@pc:~/repo/git/grumpy$ git log -n 1
commit 61bd06c7f51fcbb0668ea32dddb71fc41e3c612e
Author: Alex Koren <[email protected]>
Date:   Wed Jan 4 17:47:52 2017 -0800

    more itertools defined (#15)
    
    * added compress, cycle, dropwhile, from_iterable, ifilter, ifilterfalse, and takewhile in itertools
    * added itertools tests for cycle, dropwhile, from_iterable, ifilter, ifilterfalse, takewhile

jyf@pc:~/repo/git/grumpy$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.1 LTS
Release:        16.04
Codename:       xenial

jyf@pc:~/repo/git/grumpy$ go version
go version go1.6.2 linux/amd64

cannot find package "grumpy/lib/io"

I am using the python io libraries to read and write files in py python script and I'm getting an erorr trying to import io. Below is a simplified version that demonstrates the error. Is the python io library supported?

$ echo "import io" | make run
/tmp/tmpdlxl4K.go:4:2: cannot find package "grumpy/lib/io" in any of:
/usr/local/go/src/grumpy/lib/io (from $GOROOT)
/opt/home/daden/dev/grumpy/build/src/grumpy/lib/io (from $GOPATH)
Makefile:78: recipe for target 'run' failed
make: *** [run] Error 1

grumpy docker image

Not a issue, but just sharing my grumpy docker image

https://hub.docker.com/r/syou/alpine-grumpy/

Usage

quick run by using grumprun

echo 'print "hello, world"' | docker run -i --rm \
	syou/alpine-grumpy grumprun
hello, world

run grumpy inside container

docker run -it --rm syou/alpine-grumpy sh

  • echo 'print "hello, world"' > hello.py
  • grumpc hello.py > hello.go
  • go run hello.go

hello, world

run directly by mounting host folder

cd /tmp/
echo 'print "hello, world"' > hello.py
docker run -it --rm -v $PWD:/src/app syou/alpine-grumpy \
	grumpc hello.py > hello.go
docker run -it --rm -v $PWD:/src/app syou/alpine-grumpy \
	go run hello.go
hello, world
docker run -it --rm -v $PWD:/src/app syou/alpine-grumpy \
	go build -ldflags "-extldflags -static" \
	-o hello hello.go
./hello
hello, world

Note that $PWD should be writable or use /tmp folder to write file from docker,
since final image not using root user, but you can specify --user 0 if you insist, but not recommended.

how to import github Go packages

we can import standard go packages, but how to add a package from a github reference ?
For example for github.com/gorilla/mux

I don't think the syntax is supported in grumpc

The following gets parsed but is wrong:

from __go__.net.http import ListenAndServe, RedirectHandler
from __go__.github.com.gorilla.mux import NewRouter

it leads to:

import (
        Ο€_githubΞ“comΞ“gorillaΞ“mux "github/com/gorilla/mux"

namedtuple excluded from collections module

I read that you don't want to have things execing. Here's a rough sketch of what a namedtuple might look like without exec.

class NamedTuple(tuple):
    'tuple with named attribute aliases for elements'

    def __new__(cls, *args, **kwds):
        'Create new instance of {typename}'
        excess = set(kwds) - set(cls._fields)
        if excess:
            raise TypeError('Unexpected keyword arguments %r' % excess)
        kwds = sorted(kwds.items(), key=lambda pair: cls._fields.index(pair[0]))
        return tuple.__new__(cls, args + tuple(v for k, v in kwds))

    @classmethod
    def _make(cls, iterable, new=tuple.__new__, len=len):
        'Make a new {typename} object from a sequence or iterable'
        n = len(cls._fields)
        result = new(cls, iterable)
        if len(result) != n:
            raise TypeError('Expected %d arguments, got %d' % (n, len(result)))
        return result

    def __repr__(self):
        'Return a nicely formatted representation string'
        names = self._fields
        values = [getattr(self, name) for name in self._fields]
        attrstring = ', '.join('%s=%r' % (k, v) for k, v in zip(names, values))
        return '%s(%s)' % (type(self).__name__, attrstring)

    def _asdict(self):
        'Return a new OrderedDict which maps field names to their values'
        return OrderedDict(zip(self._fields, self))

    def _replace(_self, **kwds):
        'Return a new {typename} object replacing specified fields with new values'
        result = _self._make(map(kwds.pop, _self._fields, _self))
        if kwds:
            raise ValueError('Got unexpected field names: %r' % kwds.keys())
        return result

    def __getnewargs__(self):
        'Return self as a plain tuple.  Used by copy and pickle.'
        return tuple(self)

    __dict__ = property(_asdict)

    def __getstate__(self):
        'Exclude the OrderedDict from pickling'
        pass



def namedtuple(typename, field_names, verbose=False, rename=False):
    """Returns a new subclass of tuple with named fields.

    >>> Point = namedtuple('Point', ['x', 'y'])
    >>> Point.__doc__                   # docstring for the new class
    'Point(x, y)'
    >>> p = Point(11, y=22)             # instantiate with positional args or keywords
    >>> p[0] + p[1]                     # indexable like a plain tuple
    33
    >>> x, y = p                        # unpack like a regular tuple
    >>> x, y
    (11, 22)
    >>> p.x + p.y                       # fields also accessible by name
    33
    >>> d = p._asdict()                 # convert to a dictionary
    >>> d['x']
    11
    >>> Point(**d)                      # convert from a dictionary
    Point(x=11, y=22)
    >>> p._replace(x=100)               # _replace() is like str.replace() but targets named fields
    Point(x=100, y=22)

    """

    # Validate the field names.  At the user's option, either generate an error
    # message or automatically replace the field name with a valid name.
    if isinstance(field_names, basestring):
        field_names = field_names.replace(',', ' ').split()
    field_names = map(str, field_names)
    typename = str(typename)
    if rename:
        seen = set()
        for index, name in enumerate(field_names):
            if (not all(c.isalnum() or c=='_' for c in name)
                or iskeyword(name)
                or not name
                or name[0].isdigit()
                or name.startswith('_')
                or name in seen):
                field_names[index] = '_%d' % index
            seen.add(name)
    for name in [typename] + field_names:
        if type(name) != str:
            raise TypeError('Type names and field names must be strings')
        if not all(c.isalnum() or c=='_' for c in name):
            raise ValueError('Type names and field names can only contain '
                             'alphanumeric characters and underscores: %r' % name)
        if iskeyword(name):
            raise ValueError('Type names and field names cannot be a '
                             'keyword: %r' % name)
        if name[0].isdigit():
            raise ValueError('Type names and field names cannot start with '
                             'a number: %r' % name)
    seen = set()
    for name in field_names:
        if name.startswith('_') and not rename:
            raise ValueError('Field names cannot start with an underscore: '
                             '%r' % name)
        if name in seen:
            raise ValueError('Encountered duplicate field name: %r' % name)
        seen.add(name)

    if verbose:
        raise NotImplementedError('verbose mode irrelevant to this implementation')

    bases = (NamedTuple,)
    field = lambda i: property(itemgetter(i), doc='Alias for field number %d' % i)
    attributes = {name: field(i) for i, name in enumerate(field_names)}
    attributes['__slots__'] = ()
    attributes['_fields'] = tuple(field_names)
    attributes['__doc__'] = '%s(%s)' % (typename, str(field_names).replace("'", '')[1:-1])
    result = type(typename, bases, attributes)

    return result

I copy-pasted from the CPython stdlib, deleted the exec and fiddled until it seemed to work.

Mypy integration for type inference and combined 2+3 codebase

This looks fantastic.

@JukkaL on the mypy team is looking at a static compile capability using mypy here : python/mypy#1862

Referencing pyir ideas here: https://docs.google.com/document/d/1jGksgI96LdYQODa9Fca7EttFEGQfNODphVmbCX0DD1k/edit

Perhaps these efforts could be combined so that a standard mypy/python typed IR can be emmited that grumpy can consume?

This would have the following advantages:

  1. Integration with type hints for large codebases
  2. improved type inference for fast code
  3. Potentially making 2+3 support easier
  4. development synergy with dropbox (and maybe other players below)

I might be dreaming but perhaps Pyjion (@brettcannon et al) and Numba (@seibert, @pzwang et al) can get in on this as well. I think it would be mutually beneficial for python and it's users.

binary op not implemented: Pow

how to reproduce

try to compile

MAX_TCP_PORT = (2 ** 16) - 1

expected result

it compiles

actual result

line 3: binary op not implemented: Pow

Make Grumpy compatible with standard python projects

There wasn't an explicit ticket for this, so here goes πŸ˜€

It would be great if Grumpy could digest a python project and spit out a Go project.
I think this would make Grumpy a lot more usable in practice on existing codebases.

This means a couple things, off the top of my head:

  • Compatibility with PYTHONPATH / VIRTUAL_ENV environment variables
  • Walking a project directory and subfolders
  • Understanding python modules and __init__.py
  • Compiling dependant libraries that are imported
  • Outputting a reasonable folder structure

Might be more, let me know what you think.

It seems to me that a ticket like this could use a concrete example and target. How about PyMySQL? PyMySQL is a popular, small (5.6k LOC) project that is explicitly pure-python. I had a gander through their import statements and everything seemed pretty reasonable. Notably, there are a few instances where they snag some small things from __future__, so that might need some implementation work. I only saw print_function and absolute_import which sounds straightforward.

I think this ticket would be satisfied by something like this:

git clone https://github.com/PyMySQL/PyMySQL

git clone https://github.com/google/grumpy
( cd grumpy; make )

grumpy/tools/grumpc --read PyMySQL --save $GOPATH/src/github.com/PyMySQL/PyMySQL

cd $GOPATH/src/github.com/PyMySQL/PyMySQL
go build

And now you have a Go library that you could import and use (it doesn't look like PyMySQL has a CLI).
This is just an example though, maybe there's a better litmus test to work towards.

Relates #5, #11, #23, possibly others.

Thoughts? πŸ˜„

NewFileFromFD did not call fileInit which did not initialize file.reader causing fp.read(num) crash

when we open a file to read by fd, the NewFileFromFD in runtime/file.go will return a new File object file, but it did not call fileInit, due to this, the file.reader will be still nil, but then if we call file.read(num) to read num bytes, the in fileRead it will call file.reader.Read, and this will cause nil pointer dereference.

poc:

add a test method in sys module:

def test():
    # take stdin as an example
    fpd = NewFileFromFD(0)
    data = fpd.read(10)
    print data

then use grumpy run this:

import sys
sys.test()

got this error:

2017-01-05 13 41 43

silent assertion error

I tried running the following python code:

print("before assert")
assert False
print("after assert")

Output:

mths@MTHS-T450s:~/go/grumpy$ ./test
before assert
mths@MTHS-T450s:~/go/grumpy$ echo $?
0

Assert exited the program silently and the return code was 0.
If i changed the code to the following it works like i would expect it.

print("before assert")
try:
    assert False
except:
    raise AssertionError
print("after assert")

Output:

mths@MTHS-T450s:~/go/grumpy$ ./test
before assert
AssertionError
mths@MTHS-T450s:~/go/grumpy$ echo $?
1

Shouldn't this also happen in the first example? Am i missing something?

sys.{stdin,stdout,stderr} missing

$ make run
import sys
sys.stdout.write("hi")
^D
AttributeError: 'module' object has no attribute 'stdout'
exit status 1
make: *** [run] Error 1

How to compile with dependencies?

Great innovative tool.

Most python code uses 3rd party modules. How to compile those?

Example: I have a tool that imports configobj, after using grumpc on my tools .py file the go build returns:

mytool.go:4:2: cannot find package "grumpy/lib/configobj" in any of:
	/usr/local/Cellar/go/1.7.3/libexec/src/grumpy/lib/configobj (from $GOROOT)
	/Users/rtoma/projects/grumpy/build/src/grumpy/lib/configobj (from $GOPATH)

Can't access Go struct fields from Python

The following code fails with the error message: AttributeError: '*Response' object has no attribute 'Body'

from __go__.os import Stdout as stdout
from __go__.net.http import DefaultClient as client

rsp, err = client.Get("http://www.google.com")
if err is not None:
    raise Exception(err.Error())

_, err = Copy(stdout, rsp.Body)
if err is not None:
    raise Exception(err.Error())

grumpc cannot compile grumpc

Grumpy doesn't seem to be able to generate a working grumpyc in go.

$ git clone https://github.com/google/grumpy && cd grumpy
$ make
$ export GOPATH=$PWD/build
$ export PYTHONPATH=$PWD/build/lib/python2.7/site-packages
$ tools/grumpyc tools/grumypc > grumpyc.go && go build -o gorumpyc grumpyc.go 
grumpc.go:4:2: cannot find package "grumpy/lib/argparse" in any of:
	/usr/lib/go/src/grumpy/lib/argparse (from $GOROOT)
	/home/omeid/go/src/github.com/google/grumpy/build/src/grumpy/lib/argparse (from $GOPATH)
grumpc.go:5:2: cannot find package "grumpy/lib/ast" in any of:
	/usr/lib/go/src/grumpy/lib/ast (from $GOROOT)
	/home/omeid/go/src/github.com/google/grumpy/build/src/grumpy/lib/ast (from $GOPATH)
grumpc.go:6:2: cannot find package "grumpy/lib/grumpy" in any of:
	/usr/lib/go/src/grumpy/lib/grumpy (from $GOROOT)
	/home/omeid/go/src/github.com/google/grumpy/build/src/grumpy/lib/grumpy (from $GOPATH)
grumpc.go:7:2: cannot find package "grumpy/lib/grumpy/compiler" in any of:
	/usr/lib/go/src/grumpy/lib/grumpy/compiler (from $GOROOT)
	/home/omeid/go/src/github.com/google/grumpy/build/src/grumpy/lib/grumpy/compiler (from $GOPATH)
grumpc.go:8:2: cannot find package "grumpy/lib/grumpy/compiler/block" in any of:
	/usr/lib/go/src/grumpy/lib/grumpy/compiler/block (from $GOROOT)
	/home/omeid/go/src/github.com/google/grumpy/build/src/grumpy/lib/grumpy/compiler/block (from $GOPATH)
grumpc.go:9:2: cannot find package "grumpy/lib/grumpy/compiler/stmt" in any of:
	/usr/lib/go/src/grumpy/lib/grumpy/compiler/stmt (from $GOROOT)
	/home/omeid/go/src/github.com/google/grumpy/build/src/grumpy/lib/grumpy/compiler/stmt (from $GOPATH)
grumpc.go:10:2: cannot find package "grumpy/lib/grumpy/compiler/util" in any of:
	/usr/lib/go/src/grumpy/lib/grumpy/compiler/util (from $GOROOT)
	/home/omeid/go/src/github.com/google/grumpy/build/src/grumpy/lib/grumpy/compiler/util (from $GOPATH)
grumpc.go:12:2: cannot find package "grumpy/lib/textwrap" in any of:
	/usr/lib/go/src/grumpy/lib/textwrap (from $GOROOT)
	/home/omeid/go/src/github.com/google/grumpy/build/src/grumpy/lib/textwrap (from $GOPATH)

is this because third_party/stdlib uses exec and eval? If that is the case, it would be nice if tools/grumpyc actually give error or at least warnings when generating the go file.

Clarify the state of the project in the README

Going through the README it seems like the project is in a more-or-less complete state. However, this blog post indicates that isn't the case:

There are still holes to fill β€” many built-in types are missing methods and attributes, built-in functions are absent and the standard library is virtually empty.

It would be good to clarify that in the README to avoid confusion.

__builtins__ not defined

I'm aware that not all builtins have been written yet. I tried to take a look at what has been written so far:

$ echo 'dir(__builtins__)' | make run
NameError: name '__builtins__' is not defined
exit status 1
make: *** [run] Error 1

I think there's some low-hanging fruit in writing plain Python implementations for a few builtins like sum, map, etc. Where's the right place to write those?

Allow for builtin slots to be overriden

Consider the following program:

class T(object):
   def __repr__(self):
      return "__repr__" 

def foo(self):
   return "foo"

a = T()

print a.__repr__
print repr(a)

T.__repr__ = foo

print a.__repr__
print repr(a)

And the following execution of CPython and Grumpy:

$ python --version
Python 2.7.10
$ python bar.py   
<bound method T.__repr__ of __repr__>
__repr__
<bound method T.foo of foo>
foo
$ ./tools/grumpc bar.py > bar.go 
$ go run bar.go 
<bound method T.__repr__ of __repr__>
__repr__
<bound method T.foo of __repr__>
__repr__

CPython allows for the slot method to be overriden at runtime. For the builtins where this applies (which can be determined from Objects/typeobject.c in the CPython source), Grumpy should lookup the method by name.

That being said, I don't know how common this is in real code. I noticed it reading the CPython source while implementing __cmp__.

Runtime unit tests failing on master

Is this expected? Are these tests not run on Travis for some reason?

Exception: foo
--- FAIL: TestNativeFuncName (0.00s)
        native_test.go:106: func1(<func(*T) github.com/google/grumpy/runtime.TestNativeFuncName at 0xc4204dd440>,) = 'runtime.TestNativeFuncName', want 'grumpy.TestNativeFuncName'
--- FAIL: TestNativeFuncStrRepr (0.00s)
        native_test.go:128: str(<func(*T) github.com/google/grumpy/runtime.TestNativeFuncStrRepr at 0xc4203f02d0>) = '<func(*T) github.com/google/grumpy/runtime.TestNativeFuncStrRepr at 0xc4203f02d0>', want <func\(\*T\) .*grumpy\.TestNativeFuncStrRepr at 0x[a-f0-9]+>
        native_test.go:135: repr(<func(*T) github.com/google/grumpy/runtime.TestNativeFuncStrRepr at 0xc4203f02d0>) = '<func(*T) github.com/google/grumpy/runtime.TestNativeFuncStrRepr at 0xc4203f02d0>', want <func\(\*T\) .*grumpy\.TestNativeFuncStrRepr at 0x[a-f0-9]+>
        native_test.go:128: str(<func() github.com/google/grumpy/runtime.TestNativeFuncStrRepr.func1 at 0xc4203f03f0>) = '<func() github.com/google/grumpy/runtime.TestNativeFuncStrRepr.func1 at 0xc4203f03f0>', want <func\(\) .*grumpy\.TestNativeFuncStrRepr\.\w+ at 0x[a-f0-9]+>
        native_test.go:135: repr(<func() github.com/google/grumpy/runtime.TestNativeFuncStrRepr.func1 at 0xc4203f03f0>) = '<func() github.com/google/grumpy/runtime.TestNativeFuncStrRepr.func1 at 0xc4203f03f0>', want <func\(\) .*grumpy\.TestNativeFuncStrRepr\.\w+ at 0x[a-f0-9]+>
        native_test.go:128: str(<func(*Frame, *Object) (*Str, *BaseException) github.com/google/grumpy/runtime.Repr at 0xc4203f0450>) = '<func(*Frame, *Object) (*Str, *BaseException) github.com/google/grumpy/runtime.Repr at 0xc4203f0450>', want <func\(\*Frame, \*Object\) .*grumpy\.Repr at 0x[a-f0-9]+>
        native_test.go:135: repr(<func(*Frame, *Object) (*Str, *BaseException) github.com/google/grumpy/runtime.Repr at 0xc4203f0450>) = '<func(*Frame, *Object) (*Str, *BaseException) github.com/google/grumpy/runtime.Repr at 0xc4203f0450>', want <func\(\*Frame, \*Object\) .*grumpy\.Repr at 0x[a-f0-9]+>
RuntimeError: foo
FAIL
FAIL    github.com/google/grumpy/runtime        1.149s

Use PyPy for pure python code

On the current Readme it states:
"But there are also a number of libraries in CPython that are C extension modules that need to be rewritten. This includes re, itertools and others."

I think that at least itertools is implemented in app level python on PyPy, why not just use them instead of rewriting modules?

Strange,Some characters make the terminal crash

my test code for grumpy , test.py:

#--coding:utf-8--
#import urllib
import sys

print "++++++++++++++++++++++++++++++++++++"
#ζ΅‹θ―•θΎ“ε‡Ίε―ζ‰§θ‘Œη¨‹εΊθ‡ͺθΊ«
with open(sys.argv[0],'r') as f:
for line in f.readlines():
print line

print "++++++++++++++++++++++++++++++++++++"
#尝试输出Htmlθ§£ζžηš„Header倴俑息
'''
url='http://baidu.com'
request=urllib.urlopen(url=url)
print request.info()
'''

Everthing seems to be ok before I run the target go exe file,When I run it in my terminal ,it just print out a lot of unreadable characters,and the history command is also unreadable,my OS is mac OS 10.12.2 . The output looks like below:
β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_610β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_62β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_63β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_64β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_65β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_66β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_67β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_68β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_69β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_70β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_71β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_72β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_73β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_74β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_75β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_76β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_77β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_79β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_80β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_81β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_82β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_83β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_84β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_85β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_86β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_87β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_88β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_89β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_90β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_91β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_92β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_93β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_94β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_95β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_96β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_97β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_98β”€β”Όβ‹βŒβŽΊββŠ.βŽ½β”œβ–’β”œβ‹βŒβ”œβ””βŽ»_99β”€β”Όβ‹βŒβŽΊββŠ.β”œβŽΊβ”€β”Όβ‹βŒβŽΊββŠ/β”€β”œΒ°8.D␊␌⎺␍␊Lβ–’βŽ½β”œRβ”€β”ΌβŠβ”€β”Όβ‹βŒβŽΊββŠ/β”€β”œΒ°8.D␊␌⎺␍␊Lβ–’βŽ½β”œRβ”€β”ΌβŠIβ”ΌSβ”œβŽΌβ‹β”ΌΒ±β”€β”Όβ‹βŒβŽΊββŠ/β”€β”œΒ°8.D␊␌⎺␍␊Rβ”€β”ΌβŠβ”€β”Όβ‹βŒβŽΊββŠ/β”€β”œΒ°8.D␊␌⎺␍␊Rβ”€β”ΌβŠβ”ΌSβ”œβŽΌβ‹β”ΌΒ±β”€β”Όβ‹βŒβŽΊββŠ/β”€β”œΒ°8.Eβ”ΌβŒβŽΊββŠRβ”€β”ΌβŠβ”€β”Όβ‹βŒβŽΊββŠ/β”€β”œΒ°8.Fβ”€β”Œβ”ŒRβ”€β”ΌβŠβ”€β”Όβ‹βŒβŽΊββŠ/β”€β”œΒ°8.Rβ”€β”ΌβŠCβŽΊβ”€β”Όβ”œβ”€β”Όβ‹βŒβŽΊββŠβ”€β”œΒ°8.Rβ”€β”ΌβŠCβŽΊβ”€β”Όβ”œIβ”ΌSβ”œβŽΌβ‹β”ΌΒ±β”€β”Όβ‹βŒβŽΊββŠ/β”€β”œΒ°8.β–’βŒβŒβŠβŽ»β”œRβ–’β”ΌΒ±βŠβŽ½β”€β”Όβ‹βŒβŽΊββŠ/β”€β”œΒ°8.Β°β‹βŽΌβŽ½β”œ_└▒␋┼
++++++++++++++++++++++++++++++++++++
BβŠβ”ΌD␊⎼Pβ–’β”ΌββŠMβ–’βŒBβŽΊβŽΊβ”-P⎼⎺:P│┐C⎺␍␊ BβŠβ”ΌD␊⎼Pβ–’β”Ό$ 1;2␌1;2␌1;2␌1;2␌
-β‰β–’βŽ½β€: 1: βŒβŽΊβ””β””β–’β”Όβ β”ΌβŽΊβ”œ Β°βŽΊβ”€β”Όβ
-β‰β–’βŽ½β€: 2␌1: βŒβŽΊβ””β””β–’β”Όβ β”ΌβŽΊβ”œ Β°βŽΊβ”€β”Όβ
-β‰β–’βŽ½β€: 2␌1: βŒβŽΊβ””β””β–’β”Όβ β”ΌβŽΊβ”œ Β°βŽΊβ”€β”Όβ
-β‰β–’βŽ½β€: 2␌1: βŒβŽΊβ””β””β–’β”Όβ β”ΌβŽΊβ”œ Β°βŽΊβ”€β”Όβ
-β‰β–’βŽ½β€: 2␌: βŒβŽΊβ””β””β–’β”Όβ β”ΌβŽΊβ”œ Β°βŽΊβ”€β”Όβ
BβŠβ”ΌD␊⎼Pβ–’β”ΌββŠMβ–’βŒBβŽΊβŽΊβ”-P⎼⎺:P│┐C⎺␍␊ BβŠβ”ΌD␊⎼Pβ–’β”Ό$
BβŠβ”ΌD␊⎼Pβ–’β”ΌββŠMβ–’βŒBβŽΊβŽΊβ”-P⎼⎺:P│┐C⎺␍␊ BβŠβ”ΌD␊⎼Pβ–’β”Ό$
BβŠβ”ΌD␊⎼Pβ–’β”ΌββŠMβ–’βŒBβŽΊβŽΊβ”-P⎼⎺:P│┐C⎺␍␊ BβŠβ”ΌD␊⎼Pβ–’β”Ό$
BβŠβ”ΌD␊⎼Pβ–’β”ΌββŠMβ–’βŒBβŽΊβŽΊβ”-P⎼⎺:P│┐C⎺␍␊ BβŠβ”ΌD␊⎼Pβ–’β”Ό$

I thought it's caused by special characters ,but when I try to make it all ASCII,it's also output something strange ,but better .and this is a part of the output:
mp_604unicode.statictmp_605unicode.statictmp_606unicode.statictmp_607unicode.statictmp_608unicode.statictmp_609unicode.statictmp_61unicode.statictmp_610unicode.statictmp_62unicode.statictmp_63unicode.statictmp_64unicode.statictmp_65unicode.statictmp_66unicode.statictmp_67unicode.statictmp_68unicode.statictmp_69unicode.statictmp_70unicode.statictmp_71unicode.statictmp_72unicode.statictmp_73unicode.statictmp_74unicode.statictmp_75unicode.statictmp_76unicode.statictmp_77unicode.statictmp_79unicode.statictmp_80unicode.statictmp_81unicode.statictmp_82unicode.statictmp_83unicode.statictmp_84unicode.statictmp_85unicode.statictmp_86unicode.statictmp_87unicode.statictmp_88unicode.statictmp_89unicode.statictmp_90unicode.statictmp_91unicode.statictmp_92unicode.statictmp_93unicode.statictmp_94unicode.statictmp_95unicode.statictmp_96unicode.statictmp_97unicode.statictmp_98unicode.statictmp_99unicode.tounicode/utf8.DecodeLastRuneunicode/utf8.DecodeLastRuneInStringunicode/utf8.DecodeRuneunicode/utf8.DecodeRuneInStringunicode/utf8.EncodeRuneunicode/utf8.FullRuneunicode/utf8.RuneCountunicode/utf8.RuneCountInStringunicode/utf8.acceptRangesunicode/utf8.first_main
BenDerPandeMacBook-Pro:test BenDerPan$ 1;2c1;2c1;2c1;2c

Can't import Go libraries with names containing [^A-Za-z0-9_]

How to import libraries when its path contains "." ?

For example, I can import "github.com/labstack/echo" in pure go by

import "github.com/labstack/echo"

In grumpy, I can import it like

from __go__.github.com.labstack.echo import New

But it will be compiled to something like this

import "github/com/labstack/echo"

Anyway to correct this issue?

Relative imports not supported: `from . import xxx`

File "/Users/xxx/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 302, in visit_Module
self._visit_each(node.body)
File "/Users/xxx/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 632, in _visit_each
self.visit(node)
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/Users/xxx/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 270, in visit_ImportFrom
if node.module.startswith(_NATIVE_MODULE_PREFIX):
AttributeError: 'NoneType' object has no attribute 'startswith'

TypeError: cannot convert int to io.Writer (ioutil.Discard)

Trying to pass ioutil.Discard as an io.Writer argument raises a TypeError. Here's a simple example program:

from __go__.io.ioutil import Discard as devnull
from __go__.fmt import Fprintln as fprintln

fprintln(devnull, "Hello, world")

The definition for ioutil.Discard is var Discard io.Writer = devNull(0)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.