grumpyhome / grumpy Goto Github PK

View Code? Open in Web Editor NEW

This project forked from alanjds/grumpy

392.0 9.0 19.0 1.99 MB

Grumpy is a Python to Go source code transcompiler and runtime.

License: Apache License 2.0

Makefile 0.97% Python 26.05% Go 72.98%

hacktoberfest python golang transpiler

grumpy's Introduction

Grumpy: Go running Python

Overview

Grumpy is a Python to Go source code transcompiler and runtime that is intended to be a near drop-in replacement for CPython 2.7. Grumpy has no VM, because it compiles Python source code to Go source code which is then compiled to native code (rather than to bytecode). The compiled Go source code uses Go library called Grumpy runtime, which replaces Python C API for system calls (although the API is incompatible with CPython's).

Limitations

Things that will probably never be supported by Grumpy

exec, eval and compile: These dynamic features of CPython are not supported by Grumpy because Grumpy modules consist of statically-compiled Go code. Supporting dynamic execution would require bundling Grumpy programs with the compilation toolchain, which would be unwieldy and impractically slow.
C extension modules: Grumpy has a different API and object layout than CPython and so supporting C extensions would be difficult. In principle it's possible to support them via an API bridge layer like the one that JyNI provides for Jython, but it would be hard to maintain and would add significant overhead when calling into and out of extension modules.

Things that Grumpy will support but doesn't yet

There are three basic categories of incomplete functionality:

Language features: Most language features are implemented with the notable exception of old-style classes. There are also a handful of operators that aren't yet supported.
Builtin functions and types: There are a number of missing functions and types in __builtins__ that have not yet been implemented. There are also a lot of methods on builtin types that are missing.
Standard library: The Python standard library is very large and much of it is pure Python, so as the language features and builtins get filled out, many modules will just work. But there are also a number of libraries in CPython that are C extension modules which will need to be rewritten.
C locale support: Go doesn't support locales in the same way that C does. As such, some functionality that is locale-dependent may not currently work the same as in CPython.

Running Grumpy

Pre-requisites

Python 2.7 pip Go 1.10+

Mac Python 2.7

OSX users can go to https://www.python.org/downloads/release/python-2718/

Once you've succesfull installed Python2.7 please ensure that pip is also installed on your system by running the command "python -m pip --version". If pip is not installed please follow the instructions found here https://pip.pypa.io/en/stable/installing/#

Once pip has succesfully been installed please run the following command

python -m pip install --upgrade pip setuptools

Linux Python 2.7

For Debian based distos you can install Python 2 with.

sudo apt install python2

Next ensure that pip is installed by running "python -m pip --version" or, "python2 -m pip version". If it returns pip [version number] then pip is already installed. If it doesn't run the command "sudo apt install python-pip". Note: Again, replace apt with the appropriate package tool.

Go Compiler

Go can be downloaded here: https://golang.org/dl/

Please see the official go installation documentation to install Go: https://golang.org/doc/install

If you wish to build Go from source see the documentation here: https://golang.org/doc/install/source

Installing Grumpy

The commands ahead assumes that you have Golang installed and a recent version of Python 2 and pip.

Method 0: binary package

For convenience, a Python package is provided from the PyPI. During install, many Grumpy will be compiled and stored inside your Python installation.

You need Golang preinstalled anyway for the installation to be successful.

pip2 install -U grumpy-runtime -I --no-cache
(wait about 5 minutes)
echo "print 'hello, world'" | grumpy run

Method 1: make run:

The simplest way to execute a Grumpy program is to use make run, which wraps a shell script called grumprun that takes Python code on stdin and builds and runs the code under Grumpy:

cd grumpy-tools-src
python2 setup.py develop
cd ../grumpy-runtime-src
echo "print 'hello, world'" | make run

Method 2: grumpc and grumprun:

For more complicated programs, you'll want to compile your Python source code to Go using grumpc (the Grumpy compiler) and then build the Go code using go build. Since Grumpy programs are statically linked, all the modules in a program must be findable by the Grumpy toolchain on the GOPATH. Grumpy looks for Go packages corresponding to Python modules in the __python__ subdirectory of the GOPATH. By convention, this subdirectory is also used for staging Python source code, making it similar to the PYTHONPATH.

The first step is to set up the shell so that the Grumpy toolchain and libraries can be found. From the root directory of the Grumpy source distribution run:

cd grumpy-tools-src
python2 setup.py develop
cd ../grumpy-runtime-src
make
export PATH=$PWD/build/bin:$PATH
export GOPATH=$PWD/build
export PYTHONPATH=$PWD/build/lib/python2.7/site-packages

You will know things are working if you see the expected output from this command:

cd grumpy-runtime-src
echo 'import sys; print sys.version' | grumprun

Next, we will write our simple Python module into the __python__ directory:

cd grumpy-runtime-src
echo 'def hello(): print "hello, world"' > $GOPATH/src/__python__/hello.py

To build a Go package from our Python script, run the following:

cd grumpy-runtime-src
mkdir -p $GOPATH/src/__python__/hello
grumpc -modname=hello $GOPATH/src/__python__/hello.py > \
    $GOPATH/src/__python__/hello/module.go

You should now be able to build a Go program that imports the package "__python__/hello". We can also import this module into Python programs that are built using grumprun:

cd grumpy-runtime-src
echo 'from hello import hello; hello()' | grumprun

grumprun is doing a few things under the hood here:

Compiles the given Python code to a dummy Go package, the same way we produced __python__/hello/module.go above
Produces a main Go package that imports the Go package from step 1. and executes it as our __main__ Python package
Executes go run on the main package generated in step 2.

Developing Grumpy

There are three main components and depending on what kind of feature you're writing, you may need to change one or more of these.

Grumpy Tools

Grumpy converts Python programs into Go programs and grumpy transpile is the CLI tool responsible for parsing Python code and generating Go code from it. grumpy transpile is written in Python and uses the pythonparser module to accomplish parsing.

The CLI main entrypoint lives at grumpy-tools-src/grumpy_tools/cli.py. It is supported by a number of Python modules in the grumpy-tools-src/grumpy_tools/compiler subdir.

Grumpy Runtime

The Go code generated by grumpy transpile performs operations on data structures that represent Python objects in running Grumpy programs. These data structures and operations are defined in the grumpy Go library (source is in the grumpy-runtime-src/runtime subdir of the source distribution). This runtime is analogous to the Python C API and many of the structures and operations defined by grumpy have counterparts in CPython.

Grumpy Standard Library

Much of the Python standard library is written in Python and thus "just works" in Grumpy. These parts of the standard library are copied from CPython 2.7 (possibly with light modifications). For licensing reasons, these files are kept in the grumpy-runtime-src/third_party subdir.

The parts of the standard library that cannot be written in pure Python, e.g. file and directory operations, are kept in the grumpy-runtime-src/lib subdir. In CPython these kinds of modules are written as C extensions. In Grumpy they are written in Python but they use native Go extensions to access facilities not otherwise available in Python.

Source Code Overview

grumpy-tools-src/grumpy_tools/compiler: Python package implementating Python -> Go transcompilation logic.
grumpy-runtime-src/lib: Grumpy-specific Python standard library implementation.
grumpy-runtime-src/runtime: Go source code for the Grumpy runtime library.
grumpy-runtime-src/third_party/ouroboros: Pure Python standard libraries copied from the Ouroboros project.
grumpy-runtime-src/third_party/pypy: Pure Python standard libraries copied from PyPy.
grumpy-runtime-src/third_party/stdlib: Pure Python standard libraries copied from CPython.
grumpy-tools-src/grumpy_tools/: Transcompilation and utility CLI.

Contact

Questions? Comments? Drop us a line at [email protected] or join our Gitter channel

grumpy's People

Contributors

Stargazers

Watchers

Forkers

corona10 templeblock kavanista mewbak tenta-browser cgl funny-falcon ssteo benlubar-pr andgrew ctimmerman astrolox abitrolly srinivas11789 shifu-engineer renesugar priestd09 aminya emacsway

grumpy's Issues

Failed to import some python std libs(e.g. json) in python code and then called in go code

Issue from google#408

def feature1(inputJsonString):
    print inputJsonString
    import json
    import sys
    print json.dumps(1)
    return sys.version

feature1({'s': 3})

Expected:

{'s': 3}
1

Got:

{'s': 3}
Traceback (most recent call last):
  File "/Users/grumpy.git/__main__.py", line 8, in <module>
  File "/Users/grumpy.git/__main__.py", line 3, in feature1
  File "/Users/grumpy.git/grumpy-runtime-src/build/src/__python__/json/__init__.py", line 110, in <module>
  File "/Users/grumpy.git/grumpy-runtime-src/build/src/__python__/json/decoder.py", line 25, in <module>
  File "/Users/grumpy.git/grumpy-runtime-src/build/src/__python__/json/decoder.py", line 23, in _floatconstants
TypeError: bad operand type for unary -: 'tuple'
exit status 1

Build fails under pyenv: build/bin/grumpc: line 18: A Python -> Go transcompiler.: command not found

I cloned grumpy and run method 1, but it cause error

➜  grumpy git:(master) echo "print 'hello, world'" | make run
build/src/grumpy/lib/__builtin__/module.go:1: syntax error: package statement must be first
make: *** [build/pkg/darwin_amd64/grumpy/lib/__builtin__.a] Error 1

'print' statement ignores >> operator

google#289 opened by @alanjds on 22 Apr 2017

The print statement is ignoring the >> keyword:

This test script outputs AssertionError: 0 chars printed, instead of 3, after printing 'foo' in the stdout.

import StringIO

fake_stdout = StringIO.StringIO()

print >> fake_stdout, 'foo',
chars = fake_stdout.tell()
assert chars == 3, '%s chars printed, instead of 3' % chars

Use python function as go function

google#14 opened by @teodor-pripoae on 4 Jan 2017

I'm trying to implement a simple HTTP server, but it seems I can't send a python function as a go function to HandleFunc.

from __go__.net.http import ListenAndServe
from __go__.githubcom.gorilla.mux import NewRouter
from __go__.fmt import Fprintf

def HomeHandler(w, r):
    Fprintf(w, "Hello world")

r = NewRouter()
r.HandleFunc("/", HomeHandler)
ListenAndServe(":8000", r)

I'm receiving this error:

$ cat web.py | make run
TypeError: cannot convert grumpy.Object to func(http.ResponseWriter, *http.Request)
exit status 1
make: *** [run] Error 1

%g: NotImplementedError

google#306 opened by @navytux on May 11, 2017

Spawning from google#305 (comment) %g formatting is not yet implemented:

(grumpy) kirr@deco:~/src/tools/go/grumpy$ cat g.py 
#!/usr/bin/env python

print '%g' % 1.0
(grumpy) kirr@deco:~/src/tools/go/grumpy$ python2 g.py 
1

(grumpy) kirr@deco:~/src/tools/go/grumpy$ grumprun <g.py 
NotImplementedError: conversion type not yet supported: g
exit status 1

Support creating, getting and putting to native Go channels

google#96 opened by @mattseh on 12 Jan 2017

Hi,

Very cool project.

I grepped the code, but couldn't find a mention of channels or goroutines.

Are they available, either directly or via the python stdlib?

Thanks

Change the Gitter away from grumpy-devel ?

Should we change the Gitter away, creating a new room?

cc: @rezaghanbari

no wildcard support

google#337 opened by @piyu7274 on Jun 28, 2017

traceback (most recent call last):
  File "tools/grumpc", line 118, in <module>
    sys.exit(main(parser.parse_args()))
  File "tools/grumpc", line 76, in main
    visitor.visit(mod)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 32, in _visit_one
    return getattr(self, visit_attr)(node)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 320, in visit_Module
    self._visit_each(node.body)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 734, in _visit_each
    self.visit(node)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 32, in _visit_one
    return getattr(self, visit_attr)(node)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 296, in visit_ImportFrom
    for imp in self.block.root.importer.visit(node):
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 32, in _visit_one
    return getattr(self, visit_attr)(node)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/imputil.py", line 102, in visit_ImportFrom
    raise util.ImportError(node, msg)
grumpy.compiler.util.ImportError: line 1: wildcard member import is not implemented: from ExtractMsg import *

Implement str.format()

google#171 opened by @trotterdylan commented on 19 Jan 2017

There's a lot stdlib code (especially in the unittest library) that uses str.format() so this will become very important soon.

Modules that have names that are also Go keywords don't work

google#108 opened by @trotterdylan on 13 Jan 2017

E.g. the struct module has to declare package struct at the top of the file, which is a compilation error. The obvious options seem to be:

Always prefix Grumpy package names so that there is never a keyword collision
Have special handling in grumpc that changes the way imports work for modules that are named the same as a Go keyword.

pythonparser is not handling ``print >> filelike`` correctly

google#297 opened by @alanjds on 24 Apr 2017

The new AST parser acts strange on print on some case (not all):

# tst.py
import StringIO
sio = StringIO.StringIO()

print 'foo'
print >> sio, 'foo'
print >>sio, 'foo'
print >> sio, 'foo',
print >>sio, 'foo',
print
print >> sio
print >>sio

CPython reference:

$ cat tst.py | python -
foo

$

Before change to pythonparser

$ git checkout c7f97e72452ff91b906c9edabdec0024e9078470
$ cat tst.py | make run
foo
foo
foo
foo foo

$

After moving pythonparser into grumpy

$ git checkout 0def2c25b977bbc200f5797f67ec8ca328893735
$ cat tst.py | make run
<unknown>:10:13-11:1: fatal: unexpected newline: expected !=, %, &, (, *, **, +, -, ., /, //, <, <<, <=, <>, ==, >, >=, >>, @, [, ^, and, if, in, is, not, or or |
print >> sio
            ^
Traceback (most recent call last):
  File "/Users/alanjds/src/git/grumpy/build/bin/grumpc", line 118, in <module>
    sys.exit(main(parser.parse_args()))
  File "/Users/alanjds/src/git/grumpy/build/bin/grumpc", line 53, in main
    mod = pythonparser.parse(py_contents)
  File "/Users/alanjds/src/git/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/__init__.py", line 59, in parse
    mode, flags, version, engine)
  File "/Users/alanjds/src/git/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/__init__.py", line 29, in parse_buffer
    return parser.file_input(), lexer.comments
  File "/Users/alanjds/src/git/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/parser.py", line 89, in outer_rule
    result = inner_rule(parser)
  File "/Users/alanjds/src/git/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/parser.py", line 151, in rule
    parser.diagnostic_engine.process(error)
  File "/Users/alanjds/src/git/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/diagnostic.py", line 165, in process
    raise Error(diagnostic)
grumpy.pythonparser.diagnostic.Error: <unknown>:10:13-11:1: fatal: unexpected newline: expected !=, %, &, (, *, **, +, -, ., /, //, <, <<, <=, <>, ==, >, >=, >>, @, [, ^, and, if, in, is, not, or or |
print >> sio
            ^
make: *** [run] Error 1

Build fails under pyenv: build/bin/grumpc: line 18: A Python -> Go transcompiler.: command not found

google#104 opened by @toughrogrammer on 13 Jan 2017

I cloned grumpy and run method 1, but it cause error

➜  grumpy git:(master) echo "print 'hello, world'" | make run
build/src/grumpy/lib/__builtin__/module.go:1: syntax error: package statement must be first
make: *** [build/pkg/darwin_amd64/grumpy/lib/__builtin__.a] Error 1

Will sys.getsizeof() be implemented?

google#298 opened by @corona10 commented on 25 Apr 2017

This issue is related with .__sizeof__() slots for python.
In PyPy implementation, it is not explicitly implemented.
But CPython has implemented this feature.
If we implemented this feature, the result of this function could be different.

Mypy integration for type inference and combined 2+3 codebase

google#8 opened by @datnamer on 4 Jan 2017

This looks fantastic.

@JukkaL on the mypy team is looking at a static compile capability using mypy here : python/mypy#1862

Referencing pyir ideas here: https://docs.google.com/document/d/1jGksgI96LdYQODa9Fca7EttFEGQfNODphVmbCX0DD1k/edit

Perhaps these efforts could be combined so that a standard mypy/python typed IR can be emmited that grumpy can consume?

This would have the following advantages:

Integration with type hints for large codebases
improved type inference for fast code
Potentially making 2+3 support easier
development synergy with dropbox (and maybe other players below)

I might be dreaming but perhaps Pyjion (@brettcannon et al) and Numba (@seibert, @pzwang et al) can get in on this as well. I think it would be mutually beneficial for python and it's users.

Support tuple args in lambda

google#9 opened by @trotterdylan on 4 Jan 2017
I will not cite @trotterdylan on its issues anymore. Lets not spam the dude 👍

Per justinsaccount on HN:

$ make run
c = {}
top = sorted(c.items(), key=lambda (k,v): v)
^D
Traceback (most recent call last):
File "./tools/grumpc", line 102, in
sys.exit(main(parser.parse_args()))
File "./tools/grumpc", line 60, in main
visitor.visit(mod)
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 302, in visit_Module
self._visit_each(node.body)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 632, in _visit_each
self.visit(node)
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stin visit_Assign
with self.expr_visitor.visit(node.value) as value:
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 101, in visit_Call
values.append((util.go_str(k.arg), self.visit(k.value)))
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 246, in visit_Lambda
return self.visit_function_inline(func_node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/expr_visitor.py", line 388, in visit_function_inline
func_visitor = block.FunctionBlockVisitor(node)
File "/Users/foo/src/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/block.py", line 432, in init
args = [a.id for a in node_args.args]
AttributeError: 'Tuple' object has no attribute 'id'

Go 32-bit compilation fails when grumpc run on a 64-bit system

google#165 opened by @luca76 on 19 Jan 2017

See this error:

  [grumpy] $ make

build/src/grumpy/lib/time_test/module.go:59: constant 3000000000 overflows int
Makefile:212: recipe for target 'build/pkg/linux_386/grumpy/lib/time_test.a' failed
make: *** [build/pkg/linux_386/grumpy/lib/time_test.a] Error 1

How can I fix it?

builtins not defined

google#50 opened @selik commented on 5 Jan 2017

I'm aware that not all builtins have been written yet. I tried to take a look at what has been written so far:

$ echo 'dir(__builtins__)' | make run
NameError: name '__builtins__' is not defined
exit status 1
make: *** [run] Error 1

I think there's some low-hanging fruit in writing plain Python implementations for a few builtins like sum, map, etc. Where's the right place to write those?

namedtuple excluded from collections module

google#29 opened by @selik on 5 Jan 2017

I read that you don't want to have things execing. Here's a rough sketch of what a namedtuple might look like without exec.

class NamedTuple(tuple):
    'tuple with named attribute aliases for elements'

    def __new__(cls, *args, **kwds):
        'Create new instance of {typename}'
        excess = set(kwds) - set(cls._fields)
        if excess:
            raise TypeError('Unexpected keyword arguments %r' % excess)
        kwds = sorted(kwds.items(), key=lambda pair: cls._fields.index(pair[0]))
        return tuple.__new__(cls, args + tuple(v for k, v in kwds))

    @classmethod
    def _make(cls, iterable, new=tuple.__new__, len=len):
        'Make a new {typename} object from a sequence or iterable'
        n = len(cls._fields)
        result = new(cls, iterable)
        if len(result) != n:
            raise TypeError('Expected %d arguments, got %d' % (n, len(result)))
        return result

    def __repr__(self):
        'Return a nicely formatted representation string'
        names = self._fields
        values = [getattr(self, name) for name in self._fields]
        attrstring = ', '.join('%s=%r' % (k, v) for k, v in zip(names, values))
        return '%s(%s)' % (type(self).__name__, attrstring)

    def _asdict(self):
        'Return a new OrderedDict which maps field names to their values'
        return OrderedDict(zip(self._fields, self))

    def _replace(_self, **kwds):
        'Return a new {typename} object replacing specified fields with new values'
        result = _self._make(map(kwds.pop, _self._fields, _self))
        if kwds:
            raise ValueError('Got unexpected field names: %r' % kwds.keys())
        return result

    def __getnewargs__(self):
        'Return self as a plain tuple.  Used by copy and pickle.'
        return tuple(self)

    __dict__ = property(_asdict)

    def __getstate__(self):
        'Exclude the OrderedDict from pickling'
        pass



def namedtuple(typename, field_names, verbose=False, rename=False):
    """Returns a new subclass of tuple with named fields.

    >>> Point = namedtuple('Point', ['x', 'y'])
    >>> Point.__doc__                   # docstring for the new class
    'Point(x, y)'
    >>> p = Point(11, y=22)             # instantiate with positional args or keywords
    >>> p[0] + p[1]                     # indexable like a plain tuple
    33
    >>> x, y = p                        # unpack like a regular tuple
    >>> x, y
    (11, 22)
    >>> p.x + p.y                       # fields also accessible by name
    33
    >>> d = p._asdict()                 # convert to a dictionary
    >>> d['x']
    11
    >>> Point(**d)                      # convert from a dictionary
    Point(x=11, y=22)
    >>> p._replace(x=100)               # _replace() is like str.replace() but targets named fields
    Point(x=100, y=22)

    """

    # Validate the field names.  At the user's option, either generate an error
    # message or automatically replace the field name with a valid name.
    if isinstance(field_names, basestring):
        field_names = field_names.replace(',', ' ').split()
    field_names = map(str, field_names)
    typename = str(typename)
    if rename:
        seen = set()
        for index, name in enumerate(field_names):
            if (not all(c.isalnum() or c=='_' for c in name)
                or iskeyword(name)
                or not name
                or name[0].isdigit()
                or name.startswith('_')
                or name in seen):
                field_names[index] = '_%d' % index
            seen.add(name)
    for name in [typename] + field_names:
        if type(name) != str:
            raise TypeError('Type names and field names must be strings')
        if not all(c.isalnum() or c=='_' for c in name):
            raise ValueError('Type names and field names can only contain '
                             'alphanumeric characters and underscores: %r' % name)
        if iskeyword(name):
            raise ValueError('Type names and field names cannot be a '
                             'keyword: %r' % name)
        if name[0].isdigit():
            raise ValueError('Type names and field names cannot start with '
                             'a number: %r' % name)
    seen = set()
    for name in field_names:
        if name.startswith('_') and not rename:
            raise ValueError('Field names cannot start with an underscore: '
                             '%r' % name)
        if name in seen:
            raise ValueError('Encountered duplicate field name: %r' % name)
        seen.add(name)

    if verbose:
        raise NotImplementedError('verbose mode irrelevant to this implementation')

    bases = (NamedTuple,)
    field = lambda i: property(itemgetter(i), doc='Alias for field number %d' % i)
    attributes = {name: field(i) for i, name in enumerate(field_names)}
    attributes['__slots__'] = ()
    attributes['_fields'] = tuple(field_names)
    attributes['__doc__'] = '%s(%s)' % (typename, str(field_names).replace("'", '')[1:-1])
    result = type(typename, bases, attributes)

    return result

I copy-pasted from the CPython stdlib, deleted the exec and fiddled until it seemed to work.

grumpy docker image

google#37 opened by @S-YOU on 5 Jan 2017

Not a issue, but just sharing my grumpy docker image

https://hub.docker.com/r/syou/alpine-grumpy/

Usage

quick run by using grumprun

echo 'print "hello, world"' | docker run -i --rm \
	syou/alpine-grumpy grumprun
hello, world

run grumpy inside container

docker run -it --rm syou/alpine-grumpy sh

echo 'print "hello, world"' > hello.py
grumpc hello.py > hello.go
go run hello.go

hello, world

run directly by mounting host folder

cd /tmp/
echo 'print "hello, world"' > hello.py

docker run -it --rm -v $PWD:/src/app syou/alpine-grumpy \
	grumpc hello.py > hello.go

docker run -it --rm -v $PWD:/src/app syou/alpine-grumpy \
	go run hello.go
hello, world

docker run -it --rm -v $PWD:/src/app syou/alpine-grumpy \
	go build -ldflags "-extldflags -static" \
	-o hello hello.go

./hello
hello, world

Note that $PWD should be writable or use /tmp folder to write file from docker,
since final image not using root user, but you can specify --user 0 if you insist, but not recommended.

numpy

google#330 opened by @aep on Jun 15, 2017

continuing the issue started with google#112

numpy is currently the most relevant data science library, but it uses c extensions which grumpy doesn't support.

Maybe there's a way to bring numpy over to grumpy

Support Windows platform

google#311 opened by @retsyo on May 18, 2017

I am using mingw32 with msys2 on windows
So, any way to let grumpy support windows? Thanks

USR@USR-PC MINGW32 /r/grumpy-master
$ make
build/src/__python__/os/module.go:257: undefined: syscall.SYS_FCNTL
build/src/__python__/os/module.go:265: undefined: syscall.F_GETFD
make: *** [build/stdlib.mk:1453: build/pkg/windows_amd64/__python__/os.a] Error 1

ABCMeta and @abstractmethod suppose to work?

google#199 opened by @S-YOU on 23 Jan 2017

I don't get OrderedDict working, because I am getting error on

self.__update(*args, **kwds)
TypeError: unbound method update() must be called with MutableMapping instance as first argument (got list instance instead)

self.__update is assigned to MutableMapping.update at class level.

It is not possible to get instance of MutableMapping, because there is abstractmethods on it.

TypeError: Can't instantiate abstract class MutableMapping with abstract methods __delitem__, __getitem__, __iter__, __len__, __setitem__

Is there a way to work-around those?

Grumpy panics when it can't fulfill an allocation

google#166 opened by @S-YOU on 19 Jan 2017

This python code

x = [True] * sys.maxint

cause

resultElems := make([]*Object, newNumElems)

in seq.go to be panicked. I can't think of a way to handle that.

wrong issue

[CLOSED] Implementation of raw_input()

I implemented a simplified version of raw_input().

Please review on focusing these.
First, It might be not 100% compatible with CPython and pypy's raw_input().
and also need more implementation to handle error cases.

Second, I need some guide for writing unit test codes to check it is well working for raw_input().
I don't know how to test user's input from keyboard.

Third, prompt printing is slower when running with go run *.go so if you have solutions then please give me a feedback.

Related PR= google#247

Guidelines to install grumpy & test

google#53 opened @KukkadapuSushma commented on 6 Jan 2017

It would be great if someone can write up the installation steps and how to use/test grumpy!

Trailing space in output of print with comma

google#223 opened by @S-YOU on 27 Jan 2017

Minor issue, but reporting what I noticed. For given code.

for i in range(2):
  for j in range(2):
    print j,
  print

% python hello.py
0 1
0 1
% python hello.py | hexdump
0000000 30 20 31 0a 30 20 31 0a                        
0000008
% cat hello.py | make run | hexdump
0000000 30 20 31 20 0a 30 20 31 20 0a                  
000000a

find: compiler: No such file or directory

Since compiler directory move to https://github.com/grumpyhome/grumpy/tree/master/grumpy-tools-src/grumpy_tools/compiler

We have to update the Makefile of https://github.com/grumpyhome/grumpy/blob/master/grumpy-runtime-src/Makefile
Also, it should make to enable to run unit tests also.

I think that @alanjds is a proper person to do it. I don't know the history which is related with why the compiler directory is moved.

Spaces in path names cause makefile to fail

google#236 opened by @drgrib on 4 Feb 2017

If there is a space in the path to grumpy, there will be a long chain of errors of the type found here on make:

IOError: [Errno 2] No such file or directory: 'grumpyTest/grumpy/third_party/stdlib:/Users/credford/Dropbox/Years/__2017/__Scripts/Professional/8'
build/src/grumpy/lib/itertools/module.go:4: can't find import: "grumpy/lib/_collections"
make: *** [build/pkg/darwin_amd64/grumpy/lib/itertools.a] Error 1

Specifically, the makefile is transforming the path from

/Users/credford/Dropbox/Years/__2017/__Scripts/Professional/8 grumpyTest/grumpy/third_party/stdlib

(notice the space after the 8) to

grumpyTest/grumpy/third_party/stdlib:/Users/credford/Dropbox/Years/__2017/__Scripts/Professional/8

Changing the name of the offending folder from 8 grumpyTest to 8-grumpyTest relieved the error. You may want to make your makefile more robust for spaces in paths, though.

Homebrew does not have a formula for Grumpy

google#204 opened by @cclauss on 24 Jan 2017

Homebrew/homebrew-core#9213

Does anyone know how the homebrew installer for Mac OSX works? I spent an few hours trying to build a formula to install Grumpy based on these examples but I failed.

Remove unnecessary branches

@alanjds
There looks like the bunch of branches. Can we remove unnecessary things?

Please publish comparative benchmarks

google#6 opened on Jan 4, 2017 by @jkugler

It would be nice if the REAME (or another file in the repo) contained comparative benchmarks, for informational purposes.

ImportError: no such module: socket

google#299 opened by @ch3ck commented on 25 Apr 2017

How do I get past this error:

Traceback (most recent call last):
  File "./build/bin/grumpc", line 118, in <module>
    sys.exit(main(parser.parse_args()))
  File "./build/bin/grumpc", line 76, in main
    visitor.visit(mod)
  File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 32, in _visit_one
    return getattr(self, visit_attr)(node)
  File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 320, in visit_Module
    self._visit_each(node.body)
  File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 734, in _visit_each
    self.visit(node)
  File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 32, in _visit_one
    return getattr(self, visit_attr)(node)
  File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 287, in visit_Import
    for imp in self.block.root.importer.visit(node):
  File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 32, in _visit_one
    return getattr(self, visit_attr)(node)
  File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/imputil.py", line 88, in visit_Import
    imp = self._resolve_import(node, alias.name)
  File "/home/localhost/Documents/Dev/play/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/imputil.py", line 153, in _resolve_import
    raise util.ImportError(node, 'no such module: {}'.format(modname))
grumpy.compiler.util.ImportError: line 9: no such module: socket

I"m trying to transpile python to Go code on Fedora 25

TestRawInput() failed frequently.

google#282 opened by @corona10 on 5 Apr 2017

TestRawInput looks like have a significant issue.
Sometimes I can reproduce on my local environment.

--- FAIL: TestRawInput (0.00s)
	builtin_types_test.go:522: func1('HelloGrumpy\n', '') raised EOFError('EOF when reading a line',), want <nil>
Exception: foo

No support for syslog module

google#300 opened by @ch3ck on 26 Apr 2017

Grumpy currently lacks support for the syslog module

Asking

From grumpy created by zantdev : google#403

Can django run on grumpy? Any tester here?

Use PyPy for pure python code

google#60 opened by @santagada on 6 Jan 2017

On the current Readme it states:
"But there are also a number of libraries in CPython that are C extension modules that need to be rewritten. This includes re, itertools and others."

I think that at least itertools is implemented in app level python on PyPy, why not just use them instead of rewriting modules?

Applying AST optimizer.

google#295 opened by @corona10 on 23 Apr 2017

CPython project has the long-term plan to applying AST optimizer in their interpreter.

I think that we could apply this one for our project.
https://pypi.python.org/pypi/astoptimizer

Migrate google/grumpy issues to grumpyhome/grumpy

I had look at some import tools. Some import only one issue, some import many at once. But no one import pull-requests as pull-requests.

The best so far is https://github.com/IQAndreas/github-issues-import . But it imports PRs as issues. May be possible to convert such issues to PRs later.

As a sample, I imported an Issue (alanjds#1) and a PR (alanjds#2) to my own fork.

Are the results acceptable? Should I proceed with a full import?

TypeError: class must have base classes

google#341 opened by @m4ns0ur on Jul 1, 2017

any transcompiled simple (empty) python class, raising same error:

TypeError: class must have base classes

to reproduce the issue, create a python file with below code, then transcompile, and run:

class c:
    pass

it is happening because of a sanity check here, anyway it needs more investigation to see why base class is empty there.

Allow for builtin slots to be overriden

google#71 opened by @meadori on 9 Jan 2017

Consider the following program:

class T(object):
   def __repr__(self):
      return "__repr__" 

def foo(self):
   return "foo"

a = T()

print a.__repr__
print repr(a)

T.__repr__ = foo

print a.__repr__
print repr(a)

And the following execution of CPython and Grumpy:

$ python --version
Python 2.7.10
$ python bar.py   
<bound method T.__repr__ of __repr__>
__repr__
<bound method T.foo of foo>
foo
$ ./tools/grumpc bar.py > bar.go 
$ go run bar.go 
<bound method T.__repr__ of __repr__>
__repr__
<bound method T.foo of __repr__>
__repr__

CPython allows for the slot method to be overriden at runtime. For the builtins where this applies (which can be determined from Objects/typeobject.c in the CPython source), Grumpy should lookup the method by name.

That being said, I don't know how common this is in real code. I noticed it reading the CPython source while implementing __cmp__.

cannot find package "grumpy/lib/io"

google#81 opened by @bigcmos on 9 Jan 2017

I am using the python io libraries to read and write files in py python script and I'm getting an erorr trying to import io. Below is a simplified version that demonstrates the error. Is the python io library supported?

$ echo "import io" | make run
/tmp/tmpdlxl4K.go:4:2: cannot find package "grumpy/lib/io" in any of:
/usr/local/go/src/grumpy/lib/io (from $GOROOT)
/opt/home/daden/dev/grumpy/build/src/grumpy/lib/io (from $GOPATH)
Makefile:78: recipe for target 'run' failed
make: *** [run] Error 1

Behavior difference of super with CPython

google#217 opened by @S-YOU on 26 Jan 2017

For the given code:

class A(tuple):
  def __init__(self, arg=None):
    print super(A, self)
    print super(A, self).__init__
    super(A, self).__init__(arg)

print A([1,2,3])

---	CPython	Grumpy
`super(A, self)`	`<super: <class 'A'>, <A object>>`	`<super object at 0xc4203b8d80>`
`super(A, self).__init__`	`<method-wrapper '__init__' of A object at 0x1024d1110>`	`<bound method super.__init__ of <super object at 0xc4203b8e10>>`
`super(A, self).__init__(arg)`	`(1, 2, 3)`	`TypeError: '__init__' requires 2 arguments`

Support future import division not implemented

google#285 opened by @trotterdylan on 7 Apr 2017

Here's the spec: https://www.python.org/dev/peps/pep-0238/

No support for requests, struct, cjson, boto and pika modules

google#301 opened by @ch3ck on Apr 26, 2017

Grumpy currently lacks support for the following python modules:

'print' statement ignores 'sys.stdout'

google#290 opened by @alanjds on 22 Apr 2017

Replacing sys.stdout have no effect over print statement.

This test script outputs AssertionError: 0 chars printed, instead of 3, after printing 'foo' in the stdout.

import StringIO
import sys

sio = StringIO()
stdout = sys.stdout
sys.stdout = sio

print 'foo'

sys.stdout = stdout

chars = sio.tell()
assert chars == 3, '%s chars printed, instead of 3' % chars

email-parser

google#338 opened by @piyu7274 on Jun 28, 2017

Traceback (most recent call last):
  File "tools/grumpc", line 118, in <module>
    sys.exit(main(parser.parse_args()))
  File "tools/grumpc", line 76, in main
    visitor.visit(mod)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 32, in _visit_one
    return getattr(self, visit_attr)(node)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 320, in visit_Module
    self._visit_each(node.body)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 734, in _visit_each
    self.visit(node)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 32, in _visit_one
    return getattr(self, visit_attr)(node)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/stmt.py", line 296, in visit_ImportFrom
    for imp in self.block.root.importer.visit(node):
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/pythonparser/algorithm.py", line 32, in _visit_one
    return getattr(self, visit_attr)(node)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/imputil.py", line 137, in visit_ImportFrom
    member_imp = resolver(node, node.module)
  File "/home/avdhesh/grumpy/build/lib/python2.7/site-packages/grumpy/compiler/imputil.py", line 153, in _resolve_import
    raise util.ImportError(node, 'no such module: {}'.format(modname))
grumpy.compiler.util.ImportError: line 35: no such module: email.parser

Can't import Go libraries with names containing [^A-Za-z0-9_]

google#11 opened by @waitingkuo on 4 Jan 2017

How to import libraries when its path contains "." ?

For example, I can import "github.com/labstack/echo" in pure go by

import "github.com/labstack/echo"

In grumpy, I can import it like

from __go__.github.com.labstack.echo import New

But it will be compiled to something like this

import "github/com/labstack/echo"

Anyway to correct this issue?

Make Grumpy compatible with standard python projects

google#43 opened @kofalt commented on 5 Jan 2017

There wasn't an explicit ticket for this, so here goes 😀

It would be great if Grumpy could digest a python project and spit out a Go project.
I think this would make Grumpy a lot more usable in practice on existing codebases.

This means a couple things, off the top of my head:

Compatibility with PYTHONPATH / VIRTUAL_ENV environment variables
Walking a project directory and subfolders
Understanding python modules and __init__.py
Compiling dependant libraries that are imported
Outputting a reasonable folder structure

Might be more, let me know what you think.

It seems to me that a ticket like this could use a concrete example and target. How about PyMySQL? PyMySQL is a popular, small (5.6k LOC) project that is explicitly pure-python. I had a gander through their import statements and everything seemed pretty reasonable. Notably, there are a few instances where they snag some small things from __future__, so that might need some implementation work. I only saw print_function and absolute_import which sounds straightforward.

I think this ticket would be satisfied by something like this:

git clone https://github.com/PyMySQL/PyMySQL

git clone https://github.com/google/grumpy
( cd grumpy; make )

grumpy/tools/grumpc --read PyMySQL --save $GOPATH/src/github.com/PyMySQL/PyMySQL

cd $GOPATH/src/github.com/PyMySQL/PyMySQL
go build

And now you have a Go library that you could import and use (it doesn't look like PyMySQL has a CLI).
This is just an example though, maybe there's a better litmus test to work towards.

Relates google#5, #9, google#23, possibly others.

Thoughts? 😄

Interactive shell

google#9 opened by @waitingkuo on 4 Jan 2017

Is it possible to launch a python interactive shell?