plasma-umass / doppio Goto Github PK

Currently we can only deal with ASCII strings.

When following the instructions get the submodules after cloning the git repo, I get this error:

fatal: reference is not a tree: 5dc5b1d169549e476b76d75c2ca1e15aac66c6f7
Unable to checkout '5dc5b1d169549e476b76d75c2ca1e15aac66c6f7' in submodule path 'third_party/ace'

It seems like a commit was not pushed to the repo, http://stackoverflow.com/questions/2155887/git-submodule-head-reference-is-not-a-tree-error

Currently we store everything as a JS (640bit floating point) number.

The two commented-out lines in Strings.java trigger failures. The valueOf() error seems to be thread-related.

After implementing reflection, our current tests seem to pass even when all the Thread* method traps are commented out. Are there other cases that I am missing, or can we remove them?

Right now we preload a bunch of classes that are the union of those used by almost all our demo programs. This is somewhat slow and will only get slower if we add rhino to the mix. We should preload a simple core set of classes, and preload extra tarballs -- customized to each program -- when the user decides to run them.

b5a9f40 is the first bad commit, at least for Rhino.

General idea:

• Replace opcodes with expression 'templates'. So iadd will be defined like this: in: [1,1], out: [1], exp: '$0 + $1', which means that it takes two 1-byte values from the stack and pushes a one-byte value. We can re-generate our current set of execute functions from these templates, but we could also string multiple opcodes together into one larger function.
• When compiling, we can evaluate the stack symbolically. So when iadd is called we might have symbols (strings) 'a' and 'b' on the stack, and when we are done we have the string 'c' on the stack. We can then produce code like c = a + b. Not absolutely clear on how to analyze jumps and exception handling at the moment. See Harissa for more details.
• Reduce virtual method lookup overhead by having compiled classes contain references to inherited functions.
• Reconstruct 'normal' control flow code via decompilation. See Harissa, or Emscripten.

I vote for doppio -- a fancy name for a double espresso. Punning on twice the amount of coffee, etc.

Bit.js looks like a decent choice.

Current untarring time with the handrolled implementation is ~37 seconds.

Instead of the correct exception message, we see [object Object] on the browser's jQuery console, due to a mysterious YieldIOException that manages to get out of the internals of the JVM.

At least in the non-release mode.

The com.sun.tools.javac package is deprecated and is not present on Linux platforms. You use this in test/special/ZipRead.java. javax.tools.JavaCompiler is the replacement, and does not appear to have a zip subpackage.

Source:

• https://forums.oracle.com/forums/thread.jspa?threadID=1075102

What is the condense() function in constant_pool.coffee supposed to do? Could you clarify the comment about fixing array references?

By the way, it's not necessary to initialize arrays using (new X for _ in [0...count]). Arrays in JS are hash tables, so we can simply assign without initialization. The length property will be automatically taken care of.

Nice work so far.

We have these left to test:

daload
dastore
dcmpg
dcmpl
dreturn
dstore_0
dstore_2
dup2
dup2_x1
dup2_x2
dup_x2
faload
fastore
fconst_2
fload
freturn
fstore
fstore_0
goto_w
jsr_w
laload
lastore
lushr
nop
saload
sastore
swap

Try to make test cases that cover them, and update this issue when you do!

One of the goals we've had for a long time is the ability to use Doppio as a drop-in replacement for the native Java applet runner plugin.

Here's a sketch of how I envision it working, from a user perspective:

• Navigate to a page with a Java applet embedded
• Click a bookmarklet that runs the applet with Doppio

The bookmarklet would:

• scrape the page source for the <applet> tag
• load Doppio
• replace the <applet> with a <canvas> of that same dimensions
• run the applet in Doppio

The hardest part is the last step, of course, because we'd need to emulate java.awt.* on the HTML canvas, and (naturally) AWT uses a metric ton of native function calls. Just like we did with Jazzlib for java.util.zip, we could use a mostly pure-Java implementation of AWT: the PJA Toolkit. Sadly, the project is a few major JRE versions out of date, and it would need a some significant effort to get in shape for Java 1.6. Alternatively, we can bite the bullet and implement all the natives ourselves. There's a test case for the most basic AWT usage in test/special/Gfx.java.

The other non-trivial piece of code is the applet harness itself. This would be responsible for creating the Applet object and calling its lifecycle methods init, start, stop, and destroy. We don't yet have a test case for this.

It seems to me that we would need to account for the fields in the superclasses as well when generating the slot numbers. I attempted an implementation of this but it didn't seem to work very well, so I haven't checked it in.

e.g. twitter, personal blog, etc. I'm thinking of putting it at the bottom of the screen, along with our github usernames.

See test in commit 59451c0.

Also means that loading in java/lang/System.class dies painfully.

Breaks disas and runtime.

The current implementation uses parse_descriptor to generate a dict structure that represents the type. It's awkward to convert that back to the raw form, which is necessary for reflection. We should investigate whether there's any use for the dict form. I actually implemented the parsing function early on in the project before I knew whether there was a good use case for it -- I still haven't found one but I don't know if anyone else has.

Instead of trying to fit everything into LocalStorage or memory, we could enable users to read/write files from a cloud storage service, providing more of a true filesystem.

We could use any or all of:

• Google Drive SDK
• Dropbox API
• Box API

Those seem to be the most friendly and widely used, at least.

Even those that don't use System.in are affected by the latest change.

See test case ObjectWithLoop.java

It looks like it might be an issue with the way we parse the StackMapTable.

We don't give a non-null return value to the command handlers javac, java, javap, or rhino (all located in browser/frontend.coffee). This causes the jQuery console to throw an exception whenever any of these commands is run. It's not visible unless you're looking at the developer console, but it's not good.

Unfortunately, the simple fix of returning true from each of these functions has the following result:

doppio > java Fib
fib(20) = 6765doppio > 

doppio >  

We need a fix for the exception that also preserves the correct reprompt behavior.

The browser frontend (on the github page) works for javap Fib (without flags), but then:

doppio > java Main -c Fib
Compiled from "Fib.java"
public class test.Fib extends java.lang.Object{
public test.Fib();
  Code:
   0:   Exception in thread "main" java.lang.NoSuchFieldError: No such field found in sun/tools/javap/Tables: opcNamesTab::[Ljava/lang/String;
    at sun.tools.javap.Tables.<clinit>(Tables.java:36)
    at sun.tools.javap.JavapPrinter.printInstr(JavapPrinter.java:352)
    at sun.tools.javap.JavapPrinter.printcodeSequence(JavapPrinter.java:312)
    at sun.tools.javap.JavapPrinter.printMethods(JavapPrinter.java:199)
    at sun.tools.javap.JavapPrinter.print(JavapPrinter.java:40)
    at sun.tools.javap.Main.displayResults(Main.java:202)
    at sun.tools.javap.Main.perform(Main.java:61)
    at sun.tools.javap.Main.entry(Main.java:49)
    at sun.tools.javap.Main.main(Main.java:34)

If you have classes like so:

class AParent {
    public int a;
    public int b;
}

class BChild extends AParent {
    public int a;
}

AParent.a and BChild.a are distinct fields. Currently, Doppio aliases the two to the same field.

CJ suggested that I change field lookup to use class/fieldname, which is the format to the putfield and getfield bytecodes. Unfortunately, that does not work correctly... I discovered (the hard way) that the classname in those bytecodes is not the class that owns the field, but the type of the object that is trying to access that field. Thus, if I request BChild.b, javac will emit a bytecode of getfield with class type BChild.

Thus, a getfield or putfield request will need to navigate the class hierarchy if it cannot find a field in the effective class of the current object.

Fixing this will also require you to fix how you access fields in native methods, and when you "cheat" to construct objects like Strings by directly assigning their fields. You may want to add a level of indirection to when you get / put an object's field.

Running make -j4 seems to call run_one_test.rb on all the requisite files, but it stops after printing out maybe half the errors. The .result files never get deleted, so typing make -j4 a second time runs the cat and rm -f instructions but skips run_one_test.rb. I think it might be an issue with our programs exiting with a nonzero return value, but I'm not sure... I've tried marking run_one_test in the Makefile with - to tell Make to continue despite errors, but to no avail.

It appears to be the norm for coffeescript code. And I have a slight preference for spaces over tabs.

From Reddit:

• I don't have discount installed, but markdown
• the gnu cpp requires -traditional-cpp to work while being raped in such a way.
• I have no clue what it put into third_party/classes. If you don't want to put that into you git (what i can understand) please add a script that downloads it. That helps alot. And if you don't tell me what i should put in there. (Java 4/5/6? does gnu classpath work? download link? What kind of file hierarchy is expected? Should they match the host's JVM or your JVM?)
• make fails, due to the lack of third_party/classes i guess. but surprisingly some stuff compiles. test/special/ZipRead.java:5: error: package com.sun.tools.javac.zip does not exist
• Also Coffeescript: In src/constant_pool.coffee, Parse error on line 157: Unexpected 'INDENT' In src/opcodes.coffee, Parse error on line 470: Unexpected 'FORIN'
• Also this strage 404: localhost - - [01/May/2012 01:18:27] "GET /test/special/Fib.java HTTP/1.1" 404

It appears to be the norm for coffeescript code. And I have a slight preference for spaces over tabs.

Ideas to speed it up:

• Pre-load as many classes as possible on startup by scanning the constant pool for class references.
• Load these classes asynchronously (and change the server to a concurrent one -- Node seems a natural choice here)

http://ai-programming.com/Java%20AI%20Projects/chatterbot10.htm

I got it to print out its first message, but I haven't gotten it to parse my responses. Seems like a problem with I/O -- I'm getting a null reference with readBytes(). Seems doable, and would make a nice demo if we got it working.

It appears to be the norm for coffeescript code. And I have a slight preference for spaces over tabs.

We're using the node.js filesystem module currently; we'll probably need to write a browser-based version (or find one to plug in).

We could also plug in another server-side method of getting things.

We can run the 'new' opcode, but it currently just makes a dummy object. Objects should live on the heap, with some sort of reference to them on the stack.

See commit acefd9c

Repro:

1. Go to http://int3.github.com/doppio/

2. Notice the error:

   GET http://int3.github.com/doppio/ace.js 404 (Not Found)

I think having a static html version of doppio that interprets java is far more impressive.

Having a server kind of defeats the purpose of a js jvm since you can easily write a web front-end interpreting java code with a java server backend.

Edit: Although this is already plenty impressive and is a feat of engineering that I myself cannot accomplish

I mentioned this in the bug about untarring, but I thought I'd break it out -- we should avoid using splice to parse byte arrays, but keep a counter instead. It doesn't allow us to use typed arrays, and I think V8 recopies the whole array when we do this -- notice how untarring gets faster as we reach the end of the array. I'm thinking of creating a wrapper object on which you can call something like next_uint8(), and have it manage a position index internally.

I guess it has something to do with how PrintStream.write is getting trapped. Confusingly, switching to Node's process.stdout.write (which doesn't insert automatic newlines) fixes print but causes println to not print newlines as well.

I've noticed that the commits post September 6th (211dbb7) have an issue in run_class where the done_cb callback fires almost immediately, rather than at the end of the run as was the case in older builds of Doppio. Looking at your commits it appears to be about the time you introduced thread support (which would make sense).

You can replicate this by running something like

for (int i = 0; i < 800; i++) System.out.println("abc ");

you'll see that the reprompt happens after the first iteration or so, i.e. the callback has fired (early).

It doesn't break the demo site too badly - you just end up with a prompt in the middle of your output - but for my use case I need to be able to call something on "program end" reliably.

I found this implementation of java.util.zip in pure Java: http://jazzlib.sourceforge.net/

As a quick test, I did s/net.sf.jazzlib/java.util.zip/, dropped the files in third_party/classes/java/util/zip, compiled, and voila! We no longer need any of that brutal ZipFile mocking. It's also about 10x faster (for the ZipFile initialization, at least).

Pros:

• we'll be able to support loading of arbitrary JARs and zip files (including the rest of the classpath/bootclasspath)
• we'll eliminate a large set of bugs (if we tried to emulate ZipFile ourselves)

Cons:

• we won't be able to just unzip a classes.jar into third_party/classes anymore (though if we include the patched .class files from jazzlib it won't be too hard to get things set up)
• I haven't looked into the license on jazzlib yet, it might not be compatible
• we'd need to include the sun.boot.class.path jars for the browser version
• breaks both javacs and rhino, because we don't have the classpaths set up correctly

Thoughts?

If any Java program tries to access a file not in the localStorage cache, the lookup will fail. This is expected behavior in general, because we don't want to give away access to arbitrary files, but some programs (notably javac and rhino) need to access properties files from $JAVA_HOME or the JCL.

Specifically: trying to run javac in the browser will fail because it can't find the file third_party/classes/sun/tools/javac/resources/javac.properties.

It would be convenient for hosting e.g. on localhost. Also, I'd like to add some caching in to speed things up.

I have tested the nifty demo site in several browsers on WIndows, Linux, Mac and on a Linux VM running on Mac. Some things work, some don't: To summarize, mostly the newest versions of browsers work, older ones not so much.

On an Ubuntu 10.10 32bit VM (hosted on Mac) running in Chromium 8.0.552.224 the initial console load produces:

Enter 'help' for full a list of commands. Ctrl-D is EOF.
doppio >
  
Could not load 'DiffPrint.class'.
Could not load 'Chatterbot.class'.
Could not load 'Lzw.class'.
Could not load 'RegexTestHarness.class'.

list_cache produced lines like this at the end:

-third_party/classes/!javac.class
-!javac.class

Trying to compile:

doppio > javac Fib.java
 > 
ReferenceError: Float32Array is not defined

In Chrome 18.0.1025.168 on OS X lion. I can't find anything in the list_cache output in third_part/classes/sun/tools. There are lots of other sun packages there, but not that one. Attempting to compile Fib.java there produces "Exception in thread "main"" and it hangs.

On a 64bit Ubuntu (not a VM) running Firefox 12.0 it works. Chromium 14.0.835.202 also works on that box.

WIndows 7 18.0.1025.168 m chrome, fine.
Windows 7 Firefox 3.6.2 hangs after Done!, console doesn't respond to input
Windows 7 Firefox 12.0 works fine, though running javac gets warning Note: sun.tools.javac.Main has been deprecated.

IE Hangs on loading:

Webpage error details
 > 
User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET CLR 1.1.4322)
Timestamp: Sat, 12 May 2012 15:24:48 UTC
 > 
 > 
Message: Exception thrown and not caught
Line: 11
Char: 25598
Code: 0
URI: http://int3.github.com/doppio/compressed.js
 > 
 > 
Message: Exception thrown and not caught
Line: 11
Char: 25598
Code: 0
URI: http://int3.github.com/doppio/compressed.js
 > 
 > 
Message: 'null' is null or not an object
Line: 1
Char: 3198
Code: 0
URI: http://int3.github.com/doppio/ace.js
 > 
 > 
Message: Exception thrown and not caught
Line: 11
Char: 25598
Code: 0

Opcodes like iastore do not verify that they are writing within the bounds of the given array. The expected behavior is to throw an ArrayIndexOutOfBoundsException.

I have not personally seen this cause a bug in an actual program, but I can easily see a Java programmer being 'clever' and writing into an array until this exception is triggered rather than explicitly checking array.length. :)

Look at the disassembly for this case:

class Test {
public static void main(String[] args) {
int a = 3-14;
}
}

And look at the arg to bipush. In the console version, it's correct (-11), but in the browser it's very wrong (63221).

Right now the -c flag tells run_tests.rb to run only a specific test. Could we hook this up with Make so as to get tests to run in parallel using -j4? I'd do this myself but I think @perimosocordiae knows how to write better Makefiles.

1. test/Println.* does not exist
2. when running make release, the following error message is displayed:
   ...
   make: *** No rule to make target 'third_party/ace/build/src/ace.js', needed by 'build/ace.js'. Stop.
   ...

What I've done so far: packages are now represented as a nested series of dicts. Also, in the method descriptor, if a method is under the package java/lang/foo/Bar, !/!/!/AnotherBar will expand to java/lang/foo/AnotherBar. The o function was used because I initially thought I might want to annotate each method with some extra properties -- I'm not so sure about this now but I'm going to leave it there for the time being.

Next up: Since we know the type signature of the native method, we can automate the dereferencing of the function's arguments in the local variable table, and pass it to the function as normal JS arguments. We could automatically push the function's return values on the caller stack too.