seattletestbed / portability Goto Github PK

It would be very helpful if repyhelper would look for files in the python path the same way that doing an import in python looks in the path for the module to import. This would make repyhelper.translate_and_import() a closer relative of python's import from the developer's perspective.

The important ability that this change would allow is to make it so that all of the repy files don't have to be copied to the same directory that the code using it lives in. They could live in their own directory which is in the python path.

Down the road, we could ultimately avoid having duplicate copies of these files around on the production servers, if desired. (That's a separate issue to consider, not part of this ticket.)

Attached is my attempt at this, which seems to work for me but I haven't run any tests. I don't know if if this plays nicely with #291/r1935 (and I think I may disagree with #291 and see this as a replacement, in case it turns out that they don't play well together and nobody is making use of the change from #291).

If you use repyportability or repyhelper, all restrictions are irrevokably turned off. It should be possible to reenable restrictions. This impacts ticket #586.

Currently we are not able to use dy_import_module_symbols() to import repy modules into python files even if the modules are in the pythonpath. In RepyV1, repyhelper was able to translate and import all repy files into a python file as long as the repy file was in the pythonpath. It seems we lost this feature in Repy V2 when using repyportability + dylink. This feature would be very nice for projects such as AFFIX.

As repyhelper is deprecated with repyV2, we should come up with an alternative strategy to address repyportability's use of repyhelper. It is being used to import dylink, the new method of importing repy modules.

https://seattle.poly.edu/browser/seattle/branches/repy_v2/portability/repyportability.py#L226

Dynamic linking is a repy file that wraps around a normal repy application allowing us to use dynamic importing in repy V2. However if we want to run repy files without restrictions (like in python), this cannot be accomplished. We need to be able to run a repy file with dylink in it in python so it is unrestricted.

One of the global variables _context (which is the usercontext) is not exposed in repyportability. This is used quite often in some of the repyV2 applications/libraries.

When a python file is run from crontab where the file is using repyhelper, then we get a ValueError where the file we are trying to import is not found.

Example:
I have an integrationtest (test_time_servers_running.py) that is running on a crontab. When i copy the line from the crontab and run it manually, everything works fine. But when the crontab actually runs the file I get an error. Here is the traceback:

Traceback (most recent call last):
  File "/home/integrationtester/cron_tests/timeserver_tests/test_time_servers_running.py", line 21, in <module>
    repyhelper.translate_and_import('advertise.repy')
  File "/home/integrationtester/cron_tests/timeserver_tests/repyhelper.py", line 290, in translate_and_import
    modulename = translate(filename, shared_mycontext, callfunc, callargs, force_overwrite)
  File "/home/integrationtester/cron_tests/timeserver_tests/repyhelper.py", line 233, in translate
    raise ValueError("File " + filename + " does not exist")
ValueError: File advertise.repy does not exist

The repyportability tests are not ported to utf. This shouldn't be hard to do, but needs to be done.

The calls "open" and "file" are not mapped into the repy program correctly. The built-in version of these routines is used instead. This is not a major issue, but should be corrected.

I noticed some buggy behavior when trying to use httpserver and httpretrieve in a pure Python script using repyhelper.

There seems to be an issue with the callback function given to httpserver_registercallback() not having access certain code imported using repyhelper's translate_and_import() function. In the attached file, I try to run a simple webpage on localhost:1234 using httpserver_registercallback(). Running it with the version of httpserver.repy currently in trunk results in nothing being retrieved upon request. However, changing the break statement on line 216 of httpserver.repy to a raise causes the following exception to be thrown:

Exception (with type 'exceptions.NameError'): global name '_httpretrieve_parse_responseheaders' is not defined

For some reason, methods within httpretrieve.repy aren't accessible to the callback function.

Trivial emulfile bug detected with tests from r2502.

See attached IRC conversation for details on the problems it caused for the new time.repy file.

The repyhelper module is extremely useful in eliminating the need for pesky mix files, but there is no good document/tutorial on how to use it effectively.

When the repy code contains advertise.repy (and possibly other socket related code), dylink_portability throws an error because some of the repy functionality has not been ported in repyportability.

fobj.read() doesn't follow semantics expressed in https://seattle.cs.washington.edu/wiki/FileApiSemantics as demonstrated with repy/tests/z_testfilereadclosedfile.py.

Something in emulfile needs (minor) fixing.

When running a SeattleGENI installation with Apache, I got the following error:

Cannot open file for translation '/home/sportzer/geni_dev/seattle/rsa.repy': 13 Permission denied: '/home/sportzer/geni_dev/seattle/rsa_repy.py'

Apache is running under it's own user, so unless it has permission to overwrite the *_repy.py files, repyhelper imports will fail. Running chmod a+w *_repy.py to change the permissions works as a temporary fix.

Here's a list:

• Put all imports in one place, and then decide for the type of import.
• Deduplicate code. Reason why which copy should stay in. (Also note that __builtin__ and __builtins__ are related but different things.)
• Deduplicate some more, this time from dylink.r2py.
• Check constructs that point at issues that have been fixed.
• Verify that funny namespace overrides are still necessary in the RepyV2 API. (The Repy-file / Python-file name clash only existed in V1.)

If you give repyhelper an importcache directory you don't have permissions to write to, you get a nonsensical error that looks like this:

Traceback (most recent call last):
  File "nmmain.py", line 55, in <module>
    import nmconnectionmanager
  File "C:\Documents and Settings\kimbrl\test\nmconnectionmanager.py", line 49, in <module>
    import nmrequesthandler
  File "C:\Documents and Settings\kimbrl\test\nmrequesthandler.py", line 22, in <module>
    repyhelper.translate_and_import("signeddata.repy")
  File "C:\Documents and Settings\kimbrl\test\repyhelper.py", line 414, in translate_and_import
    modulename = translate(filename, shared_mycontext, callfunc, callargs, force_overwrite)
  File "C:\Documents and Settings\kimbrl\test\repyhelper.py", line 365, in translate
    _generate_python_file_from_repy(filenamewithpath, generatedfilenamewithpath, shared_mycontext, callfunc, callargs)
  File "C:\Documents and Settings\kimbrl\test\repyhelper.py", line 230, in _generate_python_file_from_repy
    fh.close()
UnboundLocalError: local variable 'fh' referenced before assignment

repyportability exists so that Python code can make use of Repy modules. Therefore, the portability module must make the Repy API functions available in the global context. However, it is not necessary at this point to ensure performance isolation (i.e. non nannying) nor namespace separation (so that what would usually be sandboxed code cannot access the internals of the sandbox).

The current repyportability module does the former (see code), but has issues with the latter (see more code). In particular, lines 204-211 seem to allow all the builtins, but the program flow in safe.py indicates that builtins are replaced preferrably even if they are considered OK.

Patching this behavior should be as simple as

@@ -205,6 +205,8 @@ def initialize_safe_module():
     for builtin_type in dir(__builtins__):
       if builtin_type not in safe._BUILTIN_OK:
         safe._BUILTIN_OK.append(builtin_type)
+      if builtin_type in safe._BUILTIN_REPLACE:
+        del safe._BUILTIN_REPLACE[builtin_type]

A nodemanager log from a Linux MIPS box (likely an OpenWrt router) shows a weird syntax error or claims a function is missing when trying to import the preprocessed dylink. The service vessel is extremely small (20kB of disk space), meaning that there is very little storage space on the device overall that Seattle could use.

I conjecture that when repyportability calls into repyhelper to make dylink.r2py importable as a Python module, repyhelper runs out of disk space while generating the .py version of dylink; subsequently trying to import that will likely fail.

Obviously, this comment is right about the state of affairs.

See also #26, and aaaaalbert/portability@274c758 for an initial take at a solution.

Using repyhelper.translate_and_import will throw a !TranslationError (see below) if the *_repy.py file of a module it is trying to import appears to not be auto-generated. It does this to prevent the chance of clobbering a file that isn't actually a _repy.py file generated from a .repy file.

However, if for any reason one of the repy files that the software updater depends on doesn't appear to be auto-generated, it will crash when it starts due to a !TranslationError.

So, the concern is along the lines of: what if the file should actually be overwritten because it's bogus? Maybe the user's disk was full the last time it tried to be auto-generated and an empty file was written, or the system crashed or process was killed while it was generating the file?

There is also risk of programmer error. What if the tag line used to identify auto-generated files was changed and all of the testing was done with clean installers rather than updates? Seems unlikely not to be noticed, but if it happened it could knock out every node that updated. (Similarly, had the _repy.py files accidentally pushed with 0.1j (#529) not had the correct tag line, it would have knocked out every node that updated due to the !TranslationError.)

So, there's probably risk of accidental clobbering, but it might be worth considering removing the auto-generated check from repyhelper. For that matter, it might be worth considering allowing use of a _repy.py file even if there is no corresponding .repy rather than throw a !TranslationError in that case, as well.

It's worth noting that it's not just a software updater issue. This same issue would cause the nodemanager to refuse to start. However, that seems less risky than if the software updater refused to start because the software updater not running would make it impossible to correct the issue.

~/downloads/seattle_repy$ python softwareupdater.py                                                  
Traceback (most recent call last):                                                                                
  File "softwareupdater.py", line 41, in <module>                                                                 
    repyhelper.translate_and_import("signeddata.repy")                                                            
  File "/home/justin/downloads/seattle_repy/repyhelper.py", line 291, in translate_and_import                     
    _import_file_contents_to_caller_namespace(modulename, preserve_globals)                                       
  File "/home/justin/downloads/seattle_repy/repyhelper.py", line 321, in _import_file_contents_to_caller_namespace
    import_module = __import__(modulename)                                                                        
  File "/home/justin/downloads/seattle_repy/signeddata_repy.py", line 50, in <module>                             
    repyhelper.translate_and_import('rsa.repy')                                                                   
  File "/home/justin/downloads/seattle_repy/repyhelper.py", line 291, in translate_and_import                     
    _import_file_contents_to_caller_namespace(modulename, preserve_globals)                                       
  File "/home/justin/downloads/seattle_repy/repyhelper.py", line 321, in _import_file_contents_to_caller_namespace
    import_module = __import__(modulename)                                                                        
  File "/home/justin/downloads/seattle_repy/rsa_repy.py", line 41, in <module>                                    
    repyhelper.translate_and_import('pycryptorsa.repy')
  File "/home/justin/downloads/seattle_repy/repyhelper.py", line 290, in translate_and_import
    modulename = translate(filename, shared_mycontext, callfunc, callargs, force_overwrite)
  File "/home/justin/downloads/seattle_repy/repyhelper.py", line 241, in translate
    if force_overwrite or _translation_is_needed(filename, generatedfilename):
  File "/home/justin/downloads/seattle_repy/repyhelper.py", line 110, in _translation_is_needed
    raise TranslationError("File name exists but wasn't automatically generated: " + generatedfile)
repyhelper.TranslationError: File name exists but wasn't automatically generated: pycryptorsa_repy.py

A trivial bug picked up with tests added in #372.

When importing a file in the current directory, repyhelper now has an exception. This is almost certainly the result of the fix for #799 (r3329).

>>> import repyhelper
>>> repyhelper.translate_and_import('centralizedadvertise.repy')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "repyhelper.py", line 436, in translate_and_import
    modulename = translate(filename, shared_mycontext, callfunc, callargs, force_overwrite)
  File "repyhelper.py", line 387, in translate
    _generate_python_file_from_repy(filenamewithpath, generatedfilenamewithpath, shared_mycontext, callfunc, callargs)
  File "repyhelper.py", line 224, in _generate_python_file_from_repy
    os.makedirs(os.path.dirname(generatedfilename))
  File "/usr/lib64/python2.5/os.py", line 171, in makedirs
    mkdir(name, mode)
OSError: [2](Errno) No such file or directory: ''

When doing from repyportability import * and add_dy_support(locals()) to add Repy and dylink calls to a Python program, dylink_import_module_symbols no longer respects the usual overriding conventions. Specifically,

# Import a variable ``foo''
dy_import_module_symbols("examplelib.repy")

# Override it locally
foo = "local override"

# Importing it again should reset foo to the library's version
dy_import_module_symbols("examplelib.repy")

assert(foo != "local override")
# XXX Failed to reset!

I'll add six unit tests that document the Python / Python-with-repyportability / Repy behavior for "import x" / "from x import *" (and the corresponding dylink calls).

Something in emulfile needs minor fixing.

dylink caches modules that have already been imported in (the same or another) source file so that ciruclar imports don't become a problem, see SeattleTestbed/seattlelib_v2#127. This works as expected when running Repy, but fails under repyportability.
Here's a test case that shows what happens. (I'll make a unit test out of it too). testlib1 just defines an error class, which is imported and raised in a function of testlib2, which in turn is called by the main program, where we try to catch that exception.

# This is testlib1.r2py
class Lib1Error(Exception):
  """The exception we are trying to catch."""

##################################
# This is testlib2.r2py
lib1 = dy_import_module("testlib1.r2py")

def raise_lib1error():
  raise lib1.Lib1Error

################################
# This is the main program
lib1 = dy_import_module("testlib1.r2py") # defines the exception
lib2 = dy_import_module("testlib2.r2py") # raises it

try:
  lib2.raise_lib1error()
except lib1.Lib1Error:
  # Branch taken in Rep
  log("Could catch!\n")
except Exception, e:
  # Branch taken under Repyportability
  log("Nargh, caught", repr(e), "and not", repr(lib1.Lib1Error()))

To check the repyportability behavior, add this to the main program:

from repyportability import *
add_dy_support(locals())

The issue is related to #27 and possibly Seattle's #1422, SeattleTestbed/seattlelib_v2#125, SeattleTestbed/repy_v2#22 (why's that a RepyV2 issue btw?), and SeattleTestbed/repy_v2#55.

I've found a node (planetlab1.williams.edu) where the node manager (and software updater !?!) repeatedly refuse to start with a translation error. Surprisingly, the sockettimeout.repy file seems normal, but the sockettimeout_repy.py files in the caches are both screwed up.

SyntaxError: EOF while scanning triple-quoted string
Traceback (most recent call last):
  File "nmmain.py", line 71, in <module>
    import nmconnectionmanager
  File "/home/uw_seattle/seattle_repy/nmconnectionmanager.py", line 49, in <module>
    import nmrequesthandler
  File "/home/uw_seattle/seattle_repy/nmrequesthandler.py", line 22, in <module>
    repyhelper.translate_and_import("signeddata.repy")
  File "/home/uw_seattle/seattle_repy/repyhelper.py", line 421, in translate_and_import
    _import_file_contents_to_caller_namespace(modulename, preserve_globals)
  File "/home/uw_seattle/seattle_repy/repyhelper.py", line 451, in _import_file_contents_to_caller_namespace
    import_module = __import__(modulename)
  File "nodemanager.repyhelpercache/signeddata_repy.py", line 51, in <module>
    repyhelper.translate_and_import('time.repy')
  File "/home/uw_seattle/seattle_repy/repyhelper.py", line 421, in translate_and_import
    _import_file_contents_to_caller_namespace(modulename, preserve_globals)
  File "/home/uw_seattle/seattle_repy/repyhelper.py", line 451, in _import_file_contents_to_caller_namespace
    import_module = __import__(modulename)
  File "nodemanager.repyhelpercache/time_repy.py", line 36, in <module>
    repyhelper.translate_and_import('tcp_time.repy')
  File "/home/uw_seattle/seattle_repy/repyhelper.py", line 421, in translate_and_import
    _import_file_contents_to_caller_namespace(modulename, preserve_globals)
  File "/home/uw_seattle/seattle_repy/repyhelper.py", line 451, in _import_file_contents_to_caller_namespace
    import_module = __import__(modulename)
  File "nodemanager.repyhelpercache/tcp_time_repy.py", line 39, in <module>
    repyhelper.translate_and_import('advertise.repy')
  File "/home/uw_seattle/seattle_repy/repyhelper.py", line 421, in translate_and_import
    _import_file_contents_to_caller_namespace(modulename, preserve_globals)
  File "/home/uw_seattle/seattle_repy/repyhelper.py", line 451, in _import_file_contents_to_caller_namespace
    import_module = __import__(modulename)
  File "nodemanager.repyhelpercache/advertise_repy.py", line 32, in <module>
    repyhelper.translate_and_import('centralizedadvertise.repy')
  File "/home/uw_seattle/seattle_repy/repyhelper.py", line 421, in translate_and_import
    _import_file_contents_to_caller_namespace(modulename, preserve_globals)
  File "/home/uw_seattle/seattle_repy/repyhelper.py", line 451, in _import_file_contents_to_caller_namespace
    import_module = __import__(modulename)
  File "nodemanager.repyhelpercache/centralizedadvertise_repy.py", line 29, in <module>
    repyhelper.translate_and_import('sockettimeout.repy')
  File "/home/uw_seattle/seattle_repy/repyhelper.py", line 421, in translate_and_import
    _import_file_contents_to_caller_namespace(modulename, preserve_globals)
  File "/home/uw_seattle/seattle_repy/repyhelper.py", line 451, in _import_file_contents_to_caller_namespace
    import_module = __import__(modulename)
  File "nodemanager.repyhelpercache/sockettimeout_repy.py", line 80
    Wrapper for Repy like socket
                               ^
SyntaxError: EOF while scanning triple-quoted string

On SeattleOnAndroid it happens that pycryptorsa_repy.py and sockettimeout_repy.py have zero bytes, crashing the software updater (see below). I speculate this happens due to a race condition of the softwareupdater starting, building the cache (slowly, it's running on a phone!), but not appearing active to our GUI, which then restarts the softwareupdater --- bang!

(Judging from #617, this might be worth investigating on slow/low-end/highly loaded machines in general.)

I suggest to add a clause checking for empty (but properly named) files, recreating them as required. Checking for the suspected race in the SeattleOnAndroid GUI is also required.

localhost seattle_repy # python softwareupdater.py
dlopen libpython2.6.so
dlopen /system/lib/libc.so
Traceback (most recent call last):
  File "softwareupdater.py", line 64, in <module>
    repyhelper.translate_and_import("signeddata.repy")
  File "/mnt/sdcard/sl4a/seattle/seattle_repy/repyhelper.py", line 458, in translate_and_import
    _import_file_contents_to_caller_namespace(modulename, preserve_globals)
  File "/mnt/sdcard/sl4a/seattle/seattle_repy/repyhelper.py", line 491, in _import_file_contents_to_caller_namespace
    import_module = __import__(modulename)
  File "softwareupdater.repyhelpercache/signeddata_repy.py", line 51, in <module>
    repyhelper.translate_and_import('rsa.repy')
  File "/mnt/sdcard/sl4a/seattle/seattle_repy/repyhelper.py", line 458, in translate_and_import
    _import_file_contents_to_caller_namespace(modulename, preserve_globals)
  File "/mnt/sdcard/sl4a/seattle/seattle_repy/repyhelper.py", line 491, in _import_file_contents_to_caller_namespace
    import_module = __import__(modulename)
  File "softwareupdater.repyhelpercache/rsa_repy.py", line 42, in <module>
    repyhelper.translate_and_import('pycryptorsa.repy')
  File "/mnt/sdcard/sl4a/seattle/seattle_repy/repyhelper.py", line 457, in translate_and_import
    modulename = translate(filename, shared_mycontext, callfunc, callargs, force_overwrite)
  File "/mnt/sdcard/sl4a/seattle/seattle_repy/repyhelper.py", line 407, in translate
    if force_overwrite or _translation_is_needed(filenamewithpath, generatedfilenamewithpath):
  File "/mnt/sdcard/sl4a/seattle/seattle_repy/repyhelper.py", line 198, in _translation_is_needed
    raise TranslationError("File name exists but wasn't automatically generated: " + generatedfile)
repyhelper.TranslationError: File name exists but wasn't automatically generated: softwareupdater.repyhelpercache/pycryptorsa_repy.py

localhost seattle_repy #

The following call does not work:

repyhelper.translate_and_import("./rsa.repy")

However, this works just fine:
repyhelper.translate_and_import("rsa.repy")

It seems likely that repyhelper should accept files in any directory. You can use os.basename to get the filename to figure out what a good temporary name for the file is (taking care to handle multiple files of the same name that were imported from different directories).

On my Vista desktop, it failed with the following output:

tagline detection should have passed, but failed
file modification time tests failed!

Oddly enough, when run on my Vista laptop, the first error line never showed, and the "file modification time tests failed!" line inconsistently showed.