almenon / arepl-backend Goto Github PK
View Code? Open in Web Editor NEWTS interface to python evaluator for AREPL
License: MIT License
TS interface to python evaluator for AREPL
License: MIT License
In my projects that use arepl-backend I'm redefining the type myself - I should be able to export it from arepl-backend.
in https://github.com/Almenon/AREPL-backend/blob/master/python/customHandlers.py I have handlers for some objects like datetime and regex. When JSONpickle encodes these objects it uses the custom handlers. The end result is a different set of properties display, so you can provide a more human-friendly representation of the object.
The downside to this is that there are literally infinite possible variations of objects, and I can only cover the most common ones. Dynamically importing a python file of the user's choice would allow them to add custom handlers for objects of their choosing.
The alternative is that they have to submit a pull request to this repo. It's a lot easier on my end to merge a pull request than do all that dev work so until (if) arepl becomes popular this is going straight to the trashbin. I mean, the backlog.
https://github.com/Microsoft/dts-gen
or refactor your code into typescript and use
tsc --declaration js.ts
porting this issue over from AREPL
In AREPL json.decode fails to decode a simple pickled enum. In Pycharm the decoding happens without any error. The reason behind this is that AREPL uses exec to execute the user's code. In the example below you can see the exact same code that works outside of exec() fails inside exec().
from enum import Enum
from jsonpickle import encode, decode
class C(Enum):
VALUE = 1
my_list_encode = encode(C.VALUE)
decode(my_list_encode) # no error
execCode = """ # exact same code as above follows!
from enum import Enum
from jsonpickle import encode, decode
class C(Enum):
VALUE = 1
my_list_encode = encode(C.VALUE)
decode(my_list_encode)
"""
execLocals = {}
exec(execCode,execLocals) # error!
Traceback (most recent call last):
File "C:/dev/random python scripts/misc.py", line 24, in
exec(execCode,execLocals)
File "", line 10, in
File "C:\dev\AREPL\src\python\jsonpickle_init_.py", line 152, in decode
return unpickler.decode(string, backend=backend, keys=keys)
File "C:\dev\AREPL\src\python\jsonpickle\unpickler.py", line 27, in decode
return context.restore(backend.decode(string), reset=reset)
File "C:\dev\AREPL\src\python\jsonpickle\unpickler.py", line 120, in restore
value = self._restore(obj)
File "C:\dev\AREPL\src\python\jsonpickle\unpickler.py", line 162, in _restore
return restore(obj)
File "C:\dev\AREPL\src\python\jsonpickle\unpickler.py", line 183, in _restore_reduce
if f == tags.NEWOBJ or f.__name__ == '__newobj__':
AttributeError: 'dict' object has no attribute '__name__'
"""
from time import sleep
from datetime import datetime
x=0
while(x<2):
x = x+1
sleep(2)
print(datetime.now())
prints should come in with a gap of 2 seconds but instead they come in all at once at the end
from p5 import *
def setup():
size(640, 360)
no_stroke()
background(204)
def draw():
if mouse_is_pressed:
fill(random_uniform(255), random_uniform(127), random_uniform(51), 127)
else:
fill(255, 15)
circle_size = random_uniform(low=10, high=80)
circle((mouse_x, mouse_y), circle_size)
def key_pressed(event):
background(204)
run()
It should bring up a canvas that you can use your mouse to draw on.
Instead, the canvas is brought up but clicking on it does nothing.
Very bizare bug - there should be no difference between running the code inside exec and outside exec. And if there is a difference it should bring up an error instead of just not letting me draw on the canvas. Without an error I don't have much to go off of.
I'll have to debug the p5 code - maybe something in there would give me a clue as to what is going wrong? Otherwise I'll have look deeper into exec() and what the function does exactly.
Without numpy support the following code will fail
import matplotlib.pyplot as plt
fig = plt.figure(figsize=(1,1))
ax = fig.add_axes([0,0,0,1])
numpy support is very simple:
import jsonpickle.ext.numpy as jsonpickle_numpy
jsonpickle_numpy.register_handlers()
The only catch is that i need to only register the numpy handler if the user has numpy installed.
def foo():
raise Exception
try:
foo()
except Exception as e:
fah
results in:
Traceback (most recent call last):
Exception
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\almenon\.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\pythonEvaluator.py", line 177, in exec_input
exec(codeToExec, evalLocals)
line 7, in <module>
NameError: name 'fah' is not defined
Internal error with travis. I'll have to wait for them to release a fix.
from p5 import *
def setup():
size(640, 360)
no_stroke()
background(204)
def draw():
if mouse_is_pressed:
fill(random_uniform(255), random_uniform(127), random_uniform(51), 127)
else:
fill(255, 15)
circle_size = random_uniform(low=10, high=80)
circle((mouse_x, mouse_y), circle_size)
def key_pressed(event):
background(204)
run()
Result:
Sorry, AREPL has ran into an error
Traceback (most recent call last):
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\pythonEvaluator.py", line 205, in
returnInfo = exec_input(data.evalCode, data.savedCode)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\pythonEvaluator.py", line 183, in exec_input
userVariables = pickle_user_vars(evalLocals)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\pythonEvaluator.py", line 138, in pickle_user_vars
return jsonpickle.encode(userVariables, max_depth=100) # any depth above 245 resuls in error and anything above 100 takes too long to process
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle__init__.py", line 132, in encode
numeric_keys=numeric_keys)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 43, in encode
return backend.encode(context.flatten(value, reset=reset))
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 156, in flatten
return self._flatten(obj)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 160, in _flatten
return self._pop(self._flatten_obj(obj))
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 176, in _flatten_obj
return flatten_func(obj)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 433, in _flatten_dict_obj
flatten(k, v, data)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 508, in _flatten_key_value_pair
data[k] = self._flatten(v)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 160, in _flatten
return self._pop(self._flatten_obj(obj))
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 176, in _flatten_obj
return flatten_func(obj)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 234, in _ref_obj_instance
return self._flatten_obj_instance(obj)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 404, in _flatten_obj_instance
return self._flatten_dict_obj(obj.__dict__, data)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 433, in _flatten_dict_obj
flatten(k, v, data)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 508, in _flatten_key_value_pair
data[k] = self._flatten(v)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 160, in _flatten
return self._pop(self._flatten_obj(obj))
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 176, in _flatten_obj
return flatten_func(obj)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 234, in _ref_obj_instance
return self._flatten_obj_instance(obj)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 404, in _flatten_obj_instance
return self.flatten_dict_obj(obj._dict__, data)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 433, in _flatten_dict_obj
flatten(k, v, data)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 508, in _flatten_key_value_pair
data[k] = self._flatten(v)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 160, in _flatten
return self._pop(self._flatten_obj(obj))
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 176, in _flatten_obj
return flatten_func(obj)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 234, in _ref_obj_instance
return self._flatten_obj_instance(obj)
File "C:\Users\almenon.vscode\extensions\almenon.arepl-0.0.9\node_modules\arepl-backend\python\jsonpickle\pickler.py", line 290, in _flatten_obj_instance
reduce_val = obj.__reduce__()
ValueError: ctypes objects containing pointers cannot be pickled
On a scale from 1-10, 1 being lowest, this has to be near a 1 on the severity list.
Still, it would be nice. This worked:
# in arepl backend
import sys, codecs
print(sys.stdout.encoding) # cp1252
w = codecs.getwriter('utf-8')
sys.stdout = w(sys.stdout.buffer)
import emoji
print(emoji.emojize(':eggplant:'))
but then certain accents (like á) started printing malformed: á
I really hate encodings. Why can't everyone just talk in esperanto???
file should be set to the user input file, not arepl file
not sure how I can do this exactly.
porting from Almenon/AREPL-electron#62
The restart method currently nukes the entire python process. Instead of doing that I could having the python exec run in a seperate thread and i could just send a nice signal to python asking it to stop the thread.
I'm currently running tests on 3.5 but 3.4 is used by ~20% of users according to
https://semaphoreci.com/blog/2017/10/18/python-versions-used-in-commercial-projects-in-2017.html
"PythonEvaluator Tests returns result" and "PythonEvaluator Tests dump returns result" are both flaky and should pass consistently.
Typing was a good improvement to javascript, it should be good for python too. I can add typing with pyannotate - I saw a cool talk about it at PyGotham.
🚨 You need to enable Continuous Integration on all branches of this repository. 🚨
To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.
Since we didn’t receive a CI status on the greenkeeper/initial
branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.
If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/
.
Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please delete the greenkeeper/initial
branch in this repository, and then remove and re-add this repository to the Greenkeeper App’s white list on Github. You'll find this list on your repo or organization’s settings page, under Installed GitHub Apps.
The context should be totally erased with each AREPL run and begun anew. However running an async program will somehow affect the next run if you close the event loop. To fix this I need to reset the event loop beforehand.
asyncio.set_event_loop(asyncio.new_event_loop())
Example program:
import asyncio
import time
import random
# asyncio.set_event_loop(asyncio.new_event_loop())
# without above fix the second run will fail
# run the_sync_run()
async def async_run(name):
random_int = random.randint(50000, 54304231)
start_time = time.time()
for i in range(0, random_int): # webpage
pass
ellapsed_time = time.time() - start_time
msg = "elapsed time: " + str(ellapsed_time)
print(msg)
return msg
def compile_async_tasks():
tasks = []
for i in range(0,4):
name = "Async Iteration " + str(i)
tasks.append(
asyncio.ensure_future(async_run(name))
)
print(tasks)
return tasks
tasks = compile_async_tasks()
start_time = time.time()
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(*tasks))
loop.close()
ellapsed_time = time.time() - start_time
print("Async looop took " + str(ellapsed_time))
Error:
Traceback (most recent call last):
line 29, in
line 24, in compile_async_tasks
File "C:\Program Files\Python36\lib\asyncio\tasks.py", line 512, in ensure_future
task = loop.create_task(coro_or_future)
File "C:\Program Files\Python36\lib\asyncio\base_events.py", line 282, in create_task
self._check_closed()
File "C:\Program Files\Python36\lib\asyncio\base_events.py", line 357, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Thanks to @HQupgradeHQ for the example program. I found the issue out while investigating Almenon/AREPL-vscode#60
Would have found this if travis fixed #39.... darnit
I forgot to update some code in stdlib_list
checkSyntax fails if the string is too long. I need to create a temp file and use that instead.
jsonPickle should be 'fail-safe' in that if there is a failure when pickling a object it would skip the object (or even better, put a "could not pickle object" message in its place).
should resolve a promise when it's done or call a callback or something.
I tried this but the timing on the method is a nightmare. Change a single thing and it stops working.
copy_saved_imports_to_exec, get_eval_locals_from_saved, exec_saved, and get_imports could probably be refactored into a seperate file
although refactoring the backend to work with arepl and arepl-vscode makes sense from a software engineering perspective (stay DRY, don't repeat yourself!) it's a bit annoying having to publish changes.
Now that howdoi has caching it would be cool if I could integrate it into arepl.
from howdoi import howdoi
howdoi._enable_cache()
parser = howdoi.get_parser()
args = vars(parser.parse_args('eat a apple'.split(' ')))
print(howdoi.howdoi(args))
print(howdoi.howdoi(args))
sys.argv should be set to the name of the file they are running, or '' if the file is not saved.
Currently it is set to the arepl file.
AREPL currently does not show stderr, just stdout.
This is a problem when it comes to logs - the user can't see the output of such logs, as it logs to stderr by default
import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
logger.info("yo")
logger.warning("warn")
logger.error("error")
logger.critical("oh noes!")
Not sure why because it's working on my machine.
FAIL: test_userVarImportDeleted (main.TestPythonEvaluator)
Traceback (most recent call last):
File "python/pythonEvaluatorTest.py", line 218, in test_userVarImportDeleted
assert jsonpickle.decode(returnInfo.userVariables)['myVar'] == 3
AssertionError
Ran 15 tests in 0.032s
FAILED (failures=1)
1.0.4
to 1.0.5
.This version is covered by your current version range and after updating it in your project the build failed.
python-shell is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.
There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.
Your Greenkeeper Bot 🌴
In same folder:
file1:
def foo():
print('hello world')
file2:
from file1 import foo
foo()
If you open AREPL on file2 AREPL will be unable to find file1 because it is executing the code in a different directory.
UserVariables used to be an empty string at first, but somewhere along the line I changed it to a object while forgetting to update the reference in the result handler
Line 175 in 388cba5
no point in showing functions. Generally speaking people define functions and dont change them, so showing them is pointless. It also clutters up the view.
It would be great if I could have something similar to linqpad's dump function. That way users could dump local variables that don't show up in the top-level global frame. It would be a simple python package users could import. Or better yet - I could automatically install it then place it in first line when AREPL starts.
It should be pretty simple to implement:
import jsonPickle
userDumpSignal = "6eq3o11"
def dump(var):
print(userDumpSignal + jsonPickle.pickleToJson(var))
# i could get fancy and print first 3 and last calls of same object of same func if in loop?
I could also print function caller's name w/ inspect: https://stackoverflow.com/questions/900392/getting-the-caller-function-name-inside-another-function-in-python?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa
porting this issue over from AREPL
does not work:
d = {'a':5}
x = ((k,v) for (k,v) in d)
does work:
d = {'a':5}
x = ((k,v) for (k,v) in d.items())
x = [(k,v) for k,v in d.items()]
x = {k:v for k,v in d.items()}
x = ((k) for k in d)
⚠ There has been a error when trying to display your variables. Sorry :(
Traceback (most recent call last):
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\pythonEvaluator.py", line 77, in exec_input
returnInfo['userVariables'] = jsonpickle.encode(userVariables, max_depth=100) # any depth above 245 resuls in error and anything above 100 takes too long to process
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle_init_.py", line 135, in encode
numeric_keys=numeric_keys)
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 44, in encode
return backend.encode(context.flatten(value, reset=reset))
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 157, in flatten
return self._flatten(obj)
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 161, in _flatten
return self._pop(self._flatten_obj(obj))
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 177, in _flatten_obj
return flatten_func(obj)
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 429, in _flatten_dict_obj
flatten(k, v, data)
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 498, in _flatten_key_value_pair
data[k] = self._flatten(v)
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 161, in _flatten
return self._pop(self._flatten_obj(obj))
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 177, in _flatten_obj
return flatten_func(obj)
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 235, in _ref_obj_instance
return self._flatten_obj_instance(obj)
File "C:\Users\anon\Downloads\arepl-win32-x64\arepl-win32-x64\resources\app\src\python\jsonpickle\pickler.py", line 367, in _flatten_obj_instance
data[tags.ITERATOR] = list(map(self._flatten, islice(obj, self._max_iter)))
File "", line 2, in
ValueError: not enough values to unpack (expected 2, got 1)
I don't see any benefit in showing functions to the user and if there's a lot of functions it just takes up space
callbacks although simple are somewhat limited and promises are all the rage now.
Frame does not display any useful info:
"f": -{
"py/object": "__builtin__.frame"
},
Override it to display the below:
'f_back', 'f_builtins', 'f_code', 'f_globals', 'f_lasti', 'f_lineno', 'f_locals', 'f_trace'
Code also does not display any useful info:
"f": -{
"py/object": "__builtin__.code"
},
Override it to display the below:
'co_argcount', 'co_cellvars', 'co_code', 'co_consts', 'co_filename', 'co_firstlineno', 'co_flags', 'co_freevars', 'co_kwonlyargcount', 'co_lnotab', 'co_name', 'co_names', 'co_nlocals', 'co_stacksize', 'co_varnames'
I don't know why anyone would raise BaseException or KeyboardInterrupt, but might as well catch it to prevent that possibility.
I need to upgrade my jsonpickle version - the latest version fixes this.
porting this issue over from AREPL
x = [[i for i in range(10000000)]]
Timings in ms:
List Size | Python Time | Total Time (py & JS) |
---|---|---|
10^4 | 0 | 23 |
10^5 | 2.98 | 225 |
10^6 | 48.1 | 2363 |
10^7 | 489 | 24337 |
I'd be okay with a doubling or even tripling of the execution time but this is clearly rediculous. Only ~2% of the execution time is actually spent executing the user's code. God knows what the computer is doing the rest of the time. Sipping martinis on the beach? 🏖️ 🍸
Example error:
Traceback (most recent call last):
exec(codeToExec, evalLocals) <- this line should NOT be here!
line 24, in
TypeError: print() argument after * must be an iterable, not NoneType
This happens with any error
null's in python are "None" and should be displayed as such.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.