almenon / arepl-backend Goto Github PK

In my projects that use arepl-backend I'm redefining the type myself - I should be able to export it from arepl-backend.

see Almenon/AREPL-electron#73

see Almenon/AREPL-vscode#91

see Almenon/AREPL-vscode#82

in https://github.com/Almenon/AREPL-backend/blob/master/python/customHandlers.py I have handlers for some objects like datetime and regex. When JSONpickle encodes these objects it uses the custom handlers. The end result is a different set of properties display, so you can provide a more human-friendly representation of the object.

The downside to this is that there are literally infinite possible variations of objects, and I can only cover the most common ones. Dynamically importing a python file of the user's choice would allow them to add custom handlers for objects of their choosing.

The alternative is that they have to submit a pull request to this repo. It's a lot easier on my end to merge a pull request than do all that dev work so until (if) arepl becomes popular this is going straight to the trashbin. I mean, the backlog.

https://github.com/Microsoft/dts-gen

or refactor your code into typescript and use

tsc --declaration js.ts

porting this issue over from AREPL

In AREPL json.decode fails to decode a simple pickled enum. In Pycharm the decoding happens without any error. The reason behind this is that AREPL uses exec to execute the user's code. In the example below you can see the exact same code that works outside of exec() fails inside exec().

from enum import Enum
from jsonpickle import encode, decode


class C(Enum):
    VALUE = 1

my_list_encode = encode(C.VALUE)
decode(my_list_encode) # no error

execCode = """ # exact same code as above follows!
from enum import Enum
from jsonpickle import encode, decode


class C(Enum):
    VALUE = 1

my_list_encode = encode(C.VALUE)
decode(my_list_encode)
"""

execLocals = {}
exec(execCode,execLocals) # error!

Traceback (most recent call last):
File "C:/dev/random python scripts/misc.py", line 24, in
exec(execCode,execLocals)
File "", line 10, in
File "C:\dev\AREPL\src\python\jsonpickle_init_.py", line 152, in decode
return unpickler.decode(string, backend=backend, keys=keys)
File "C:\dev\AREPL\src\python\jsonpickle\unpickler.py", line 27, in decode
return context.restore(backend.decode(string), reset=reset)
File "C:\dev\AREPL\src\python\jsonpickle\unpickler.py", line 120, in restore
value = self._restore(obj)
File "C:\dev\AREPL\src\python\jsonpickle\unpickler.py", line 162, in _restore
return restore(obj)
File "C:\dev\AREPL\src\python\jsonpickle\unpickler.py", line 183, in _restore_reduce
if f == tags.NEWOBJ or f.__name__ == '__newobj__':
AttributeError: 'dict' object has no attribute '__name__'
"""

from time import sleep
from datetime import datetime

x=0

while(x<2):
    x = x+1
    sleep(2)
    print(datetime.now())    
      

prints should come in with a gap of 2 seconds but instead they come in all at once at the end

from p5 import *

def setup():
    size(640, 360)
    no_stroke()
    background(204)

def draw():
    if mouse_is_pressed:
        fill(random_uniform(255), random_uniform(127), random_uniform(51), 127)
    else:
        fill(255, 15)

    circle_size = random_uniform(low=10, high=80)

    circle((mouse_x, mouse_y), circle_size)

def key_pressed(event):
    background(204)

run()

It should bring up a canvas that you can use your mouse to draw on.

Instead, the canvas is brought up but clicking on it does nothing.

Very bizare bug - there should be no difference between running the code inside exec and outside exec. And if there is a difference it should bring up an error instead of just not letting me draw on the canvas. Without an error I don't have much to go off of.

I'll have to debug the p5 code - maybe something in there would give me a clue as to what is going wrong? Otherwise I'll have look deeper into exec() and what the function does exactly.

Without numpy support the following code will fail

import matplotlib.pyplot as plt
fig = plt.figure(figsize=(1,1))
ax = fig.add_axes([0,0,0,1])

numpy support is very simple:

import jsonpickle.ext.numpy as jsonpickle_numpy
jsonpickle_numpy.register_handlers()

The only catch is that i need to only register the numpy handler if the user has numpy installed.

def foo():
    raise Exception

try:
    foo()
except Exception as e:
    fah

results in:

Traceback (most recent call last):
Exception

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\almenon\.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\pythonEvaluator.py", line 177, in exec_input
    exec(codeToExec, evalLocals)
  line 7, in <module>
NameError: name 'fah' is not defined

Internal error with travis. I'll have to wait for them to release a fix.

See travis-ci/travis-ci#9815

see Almenon/AREPL-vscode#86

from p5 import *

def setup():
    size(640, 360)
    no_stroke()
    background(204)

def draw():
    if mouse_is_pressed:
        fill(random_uniform(255), random_uniform(127), random_uniform(51), 127)
    else:
        fill(255, 15)

    circle_size = random_uniform(low=10, high=80)

    circle((mouse_x, mouse_y), circle_size)

def key_pressed(event):
    background(204)

run()

Result:

Sorry, AREPL has ran into an error

Traceback (most recent call last):
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\pythonEvaluator.py", line 205, in
returnInfo = exec_input(data.evalCode, data.savedCode)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\pythonEvaluator.py", line 183, in exec_input
userVariables = pickle_user_vars(evalLocals)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\pythonEvaluator.py", line 138, in pickle_user_vars
return jsonpickle.encode(userVariables, max_depth=100) # any depth above 245 resuls in error and anything above 100 takes too long to process
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle__init__.py", line 132, in encode
numeric_keys=numeric_keys)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 43, in encode
return backend.encode(context.flatten(value, reset=reset))
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 156, in flatten
return self._flatten(obj)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 160, in _flatten
return self._pop(self._flatten_obj(obj))
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 176, in _flatten_obj
return flatten_func(obj)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 433, in _flatten_dict_obj
flatten(k, v, data)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 508, in _flatten_key_value_pair
data[k] = self._flatten(v)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 160, in _flatten
return self._pop(self._flatten_obj(obj))
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 176, in _flatten_obj
return flatten_func(obj)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 234, in _ref_obj_instance
return self._flatten_obj_instance(obj)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 404, in _flatten_obj_instance
return self._flatten_dict_obj(obj.__dict__, data)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 433, in _flatten_dict_obj
flatten(k, v, data)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 508, in _flatten_key_value_pair
data[k] = self._flatten(v)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 160, in _flatten
return self._pop(self._flatten_obj(obj))
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 176, in _flatten_obj
return flatten_func(obj)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 234, in _ref_obj_instance
return self._flatten_obj_instance(obj)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 404, in _flatten_obj_instance
return self.flatten_dict_obj(obj._dict__, data)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 433, in _flatten_dict_obj
flatten(k, v, data)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 508, in _flatten_key_value_pair
data[k] = self._flatten(v)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 160, in _flatten
return self._pop(self._flatten_obj(obj))
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 176, in _flatten_obj
return flatten_func(obj)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 234, in _ref_obj_instance
return self._flatten_obj_instance(obj)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 290, in _flatten_obj_instance
reduce_val = obj.__reduce__()
ValueError: ctypes objects containing pointers cannot be pickled

On a scale from 1-10, 1 being lowest, this has to be near a 1 on the severity list.

Still, it would be nice. This worked:

# in arepl backend
import sys, codecs
print(sys.stdout.encoding) # cp1252
w = codecs.getwriter('utf-8')
sys.stdout = w(sys.stdout.buffer)

import emoji
print(emoji.emojize(':eggplant:'))

but then certain accents (like á) started printing malformed: á

I really hate encodings. Why can't everyone just talk in esperanto???

file should be set to the user input file, not arepl file

not sure how I can do this exactly.

porting from Almenon/AREPL-electron#62

The restart method currently nukes the entire python process. Instead of doing that I could having the python exec run in a seperate thread and i could just send a nice signal to python asking it to stop the thread.

I'm currently running tests on 3.5 but 3.4 is used by ~20% of users according to
https://semaphoreci.com/blog/2017/10/18/python-versions-used-in-commercial-projects-in-2017.html

"PythonEvaluator Tests returns result" and "PythonEvaluator Tests dump returns result" are both flaky and should pass consistently.

Typing was a good improvement to javascript, it should be good for python too. I can add typing with pyannotate - I saw a cool talk about it at PyGotham.

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper App’s white list on Github. You'll find this list on your repo or organization’s settings page, under Installed GitHub Apps.

The context should be totally erased with each AREPL run and begun anew. However running an async program will somehow affect the next run if you close the event loop. To fix this I need to reset the event loop beforehand.

asyncio.set_event_loop(asyncio.new_event_loop()) 

Example program:

import asyncio
import time
import random

# asyncio.set_event_loop(asyncio.new_event_loop()) 
# without above fix the second run will fail

# run the_sync_run()
async def async_run(name):
    random_int = random.randint(50000, 54304231)
    start_time = time.time()
    for i in range(0, random_int): # webpage
        pass
    ellapsed_time = time.time() - start_time
    msg = "elapsed time: " + str(ellapsed_time) 
    print(msg)
    return msg

def compile_async_tasks():
    tasks = []

    for i in range(0,4):
        name = "Async Iteration " + str(i)
        tasks.append(
            asyncio.ensure_future(async_run(name))
        )
    print(tasks)
    return tasks

tasks = compile_async_tasks()

start_time = time.time()
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(*tasks))
loop.close()

ellapsed_time = time.time() - start_time
print("Async looop took " + str(ellapsed_time))

Error:

Traceback (most recent call last):
line 29, in
line 24, in compile_async_tasks
File "C:\Program Files\Python36\lib\asyncio\tasks.py", line 512, in ensure_future
task = loop.create_task(coro_or_future)
File "C:\Program Files\Python36\lib\asyncio\base_events.py", line 282, in create_task
self._check_closed()
File "C:\Program Files\Python36\lib\asyncio\base_events.py", line 357, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

Thanks to @HQupgradeHQ for the example program. I found the issue out while investigating Almenon/AREPL-vscode#60

Would have found this if travis fixed #39.... darnit

I forgot to update some code in stdlib_list

checkSyntax fails if the string is too long. I need to create a temp file and use that instead.

jsonPickle should be 'fail-safe' in that if there is a failure when pickling a object it would skip the object (or even better, put a "could not pickle object" message in its place).

should resolve a promise when it's done or call a callback or something.

I tried this but the timing on the method is a nightmare. Change a single thing and it stops working.

copy_saved_imports_to_exec, get_eval_locals_from_saved, exec_saved, and get_imports could probably be refactored into a seperate file

although refactoring the backend to work with arepl and arepl-vscode makes sense from a software engineering perspective (stay DRY, don't repeat yourself!) it's a bit annoying having to publish changes.

Now that howdoi has caching it would be cool if I could integrate it into arepl.

from howdoi import howdoi
howdoi._enable_cache()
parser = howdoi.get_parser()
args = vars(parser.parse_args('eat a apple'.split(' ')))
print(howdoi.howdoi(args))
print(howdoi.howdoi(args))

sys.argv should be set to the name of the file they are running, or '' if the file is not saved.

Currently it is set to the arepl file.

AREPL currently does not show stderr, just stdout.

This is a problem when it comes to logs - the user can't see the output of such logs, as it logs to stderr by default

import logging

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

logger.info("yo")
logger.warning("warn")
logger.error("error")
logger.critical("oh noes!")

Not sure why because it's working on my machine.

FAIL: test_userVarImportDeleted (main.TestPythonEvaluator)

Traceback (most recent call last):
File "python/pythonEvaluatorTest.py", line 218, in test_userVarImportDeleted
assert jsonpickle.decode(returnInfo.userVariables)['myVar'] == 3
AssertionError

Ran 15 tests in 0.032s
FAILED (failures=1)

The dependency python-shell was updated from 1.0.4 to 1.0.5.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

python-shell is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

In same folder:

file1:

def foo():
  print('hello world')

file2:

from file1 import foo
foo()

If you open AREPL on file2 AREPL will be unable to find file1 because it is executing the code in a different directory.

UserVariables used to be an empty string at first, but somewhere along the line I changed it to a object while forgetting to update the reference in the result handler

no point in showing functions. Generally speaking people define functions and dont change them, so showing them is pointless. It also clutters up the view.

It would be great if I could have something similar to linqpad's dump function. That way users could dump local variables that don't show up in the top-level global frame. It would be a simple python package users could import. Or better yet - I could automatically install it then place it in first line when AREPL starts.

It should be pretty simple to implement:

import jsonPickle
userDumpSignal = "6eq3o11"
def dump(var):
  print(userDumpSignal + jsonPickle.pickleToJson(var))
  # i could get fancy and print first 3 and last calls of same object of same func if in loop?

I could also print function caller's name w/ inspect: https://stackoverflow.com/questions/900392/getting-the-caller-function-name-inside-another-function-in-python?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa

porting this issue over from AREPL

does not work:

d = {'a':5}
x = ((k,v) for (k,v) in d)

does work:

d = {'a':5}
x = ((k,v) for (k,v) in d.items())
x = [(k,v) for k,v in d.items()]
x = {k:v for k,v in d.items()}
x = ((k) for k in d)

⚠ There has been a error when trying to display your variables. Sorry :(

Traceback (most recent call last):
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\pythonEvaluator.py", line 77, in exec_input
returnInfo['userVariables'] = jsonpickle.encode(userVariables, max_depth=100) # any depth above 245 resuls in error and anything above 100 takes too long to process
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle_init_.py", line 135, in encode
numeric_keys=numeric_keys)
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 44, in encode
return backend.encode(context.flatten(value, reset=reset))
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 157, in flatten
return self._flatten(obj)
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 161, in _flatten
return self._pop(self._flatten_obj(obj))
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 177, in _flatten_obj
return flatten_func(obj)
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 429, in _flatten_dict_obj
flatten(k, v, data)
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 498, in _flatten_key_value_pair
data[k] = self._flatten(v)
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 161, in _flatten
return self._pop(self._flatten_obj(obj))
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 177, in _flatten_obj
return flatten_func(obj)
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 235, in _ref_obj_instance
return self._flatten_obj_instance(obj)
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 367, in _flatten_obj_instance
data[tags.ITERATOR] = list(map(self._flatten, islice(obj, self._max_iter)))
File "", line 2, in
ValueError: not enough values to unpack (expected 2, got 1)

I don't see any benefit in showing functions to the user and if there's a lot of functions it just takes up space

callbacks although simple are somewhat limited and promises are all the rage now.

Frame does not display any useful info:

"f": -{
    "py/object": "__builtin__.frame"
},

Override it to display the below:

'f_back', 'f_builtins', 'f_code', 'f_globals', 'f_lasti', 'f_lineno', 'f_locals', 'f_trace'

Code also does not display any useful info:

"f": -{
    "py/object": "__builtin__.code"
},

Override it to display the below:

'co_argcount', 'co_cellvars', 'co_code', 'co_consts', 'co_filename', 'co_firstlineno', 'co_flags', 'co_freevars', 'co_kwonlyargcount', 'co_lnotab', 'co_name', 'co_names', 'co_nlocals', 'co_stacksize', 'co_varnames'

I don't know why anyone would raise BaseException or KeyboardInterrupt, but might as well catch it to prevent that possibility.

see Almenon/AREPL-vscode#127

I need to upgrade my jsonpickle version - the latest version fixes this.

see Almenon/AREPL-vscode#120

porting this issue over from AREPL

x = [[i for i in range(10000000)]]

Timings in ms:

I'd be okay with a doubling or even tripling of the execution time but this is clearly rediculous. Only ~2% of the execution time is actually spent executing the user's code. God knows what the computer is doing the rest of the time. Sipping martinis on the beach? 🏖️ 🍸

Example error:

Traceback (most recent call last):
exec(codeToExec, evalLocals) <- this line should NOT be here!
line 24, in
TypeError: print() argument after * must be an iterable, not NoneType

This happens with any error

null's in python are "None" and should be displayed as such.