TorqueScript has suboptimal performance about torque3d HOT 86 CLOSED

2 options. 1) Optimize the compiler with V-tune, fix things, and test the heck out of the engine. 2) Ditch Torque Script.

from torque3d.

I would prefer that we try to fix features before tossing them out as garbage, especially such a critical part that is the glue of the game engine that TorqueScript is. So, lets go with option 1 for the time being. ;)

from torque3d.

I would like to see TorqueScript completely replaced with Lua, personally. As far as scripting languages go, it's doesn't get much faster than Lua.

from torque3d.

I would recommend using JavaScript, for the following reasons:

1. Should someday someone want to add the option to run Torque3D games as HTML5, compiled to JavaScript, this would make it much much more feasible. (I actually work on that area myself, have ported another 3D game engine to HTML5 - https://developer.mozilla.org/demos/detail/bananabread - and might be curious to try the same with Torque3D now that it is open source). The web is becoming a very important compilation target since everyone has a browser, and browsers are just now becoming powerful enough to run full 3D games.
2. Modern JS engines are very fast. Comparisons to Lua(JIT) are difficult because it's hard to find apples-to-apples comparisons, but from my knowledge of the internals of their implementations, they should be on par with each other.
3. There are 3 modern and fast open source implementations of JavaScript: Google's V8, Firefox's SpiderMonkey and WebKit's JSC.
4. JavaScript is likely the most familiar programming language because it is used on the web (and now also Node.js etc.).

from torque3d.

We are implementing V8 on a service project that we are working on right now. Very nice.

from torque3d.

Or turn TS into a JIT compiler...
Eric,
I was thinking about modifying the macros used to create TS commands and variables. Would it be feasible to modify those to automagically create bindings into another VM? I was thinking of looking into this with Python. Have the macros generate the Python function calls and globals in parallel to TS commands.

Also, I know .NET code can be compiled from Python so I am guessing Mono does this too. That could be another avenue is to switch to .NET for the VM. You may even be able to target the .NET VM via Torque Script. Ooh, that is a good option. Target a VM that has already got a large code base like .NET. Then you can program in many languages. I wonder if there is a Lua translator for .NET?

from torque3d.

I for some reason always cringe when I hear of ECMAScript in game engines. It's kind of a strange thing because I actually like the language (or at least a subset of the language commonly called "The Good Parts" thanks to Douglas Crockford.) Some people don't realize that ECMAScript and Lua actually have a fair number of things in common if you disregard syntax. I think the reason I tend to like Lua in engines better is because it's actually quite similar to "The Good Parts" of ECMAScript without the rest that makes ECMAScript so treacherous for new devs. For instance, many people are surprised by the following:

if ( "0" == false ) { console.log("Are they equal?"); }

Now, that said. I'd personally like to see more than one scripting language implemented in addition to TS, perhaps in a sort of plugin nature. Getting rid of TS would be a mistake IMO since it will break lots of existing code (and in this case I think that's unacceptable since it's an unneeded break.) I'd personally prefer it not to be rebuilt on something like the CLR (.NET for those who don't know what the VM for it is) or the JVM. Again, though, if this ends up being the way it goes, I hope it's as an optional plugin. I'm going to say this just to stress the point: TS should stay around to keep legacy code working

from torque3d.

i don't like the idea of replacing a scripting language by another, firstly it will take a long time to replace it and i'm pretty sure that there are many ppl that have learned to use TS. must improve first and then maybe someday we might re-discuss this.

from torque3d.

Can someone explain to me what TS really is? I know there is a compiler, but the actual language calls are just a set of console callbacks. The callbacks are called from an interface called the console. This interface is where most of the slow down comes from because it is a string lookup. So if the calls were exported to say Python calls then we would gain the advantage of the Python interpreter being more efficient. It should effectively allow more than one language to exist in the same engine. The same thing could be done for Lua or any other language. That would certainly make it modular.

I will have to rethink my interface with Python. If I can tell the engine to create Python function calls when the T3D console calls are created by modifying the macro then I can gain automatic export of all console calls. Which would still allow someone to write standard console functions as those would be exported as well. Right now I can call console functions, but having the calls directly tied to Python calls would make it faster and not have the lookup issue. Currently the console can call Python as well and it is slightly faster due to the design.

After that it should be trivial to convert the TS code to Python code or leave as is. It won't matter. It will just be faster to be in Python. Since I am using SWIG it may be a portable concept to use Lua or any other SWIG supported language.

@setlec,
It took me 3 months to write my Python interface for T3D part time. It is in the resources at GG. I will be updating it soon to reflect a licensing change to MIT as well as adding support for T2D. It did not take a long time. What I am talking about here is making it closer to the metal than it is now. Python is also an easier language to grasp than TS and has thousands of professionally designed packages already ported to use with it.

from torque3d.

@Demolishun TorqueScript itself is the bulk of what is in console combined with the core stuff like as StringTable. The interpreter itself is compiledEval & StringStack.

There appears to be a half-completed rewrite of the Console binding system in engineApi.h which everything using "DefineEngine..." uses, but as it's incomplete an a lot of the code still uses the "Console*" macros it's mostly just a fancy newer way of exposing console functions and objects. There is also a Javascript binding API in the web plugin code which uses Console binding system to bind methods from Javascript. Come to think of it several years ago there was even a resource for a python binding system back in the TGE 1.x days which I recall was used in Minions of Mirth.

On a general note...

If anyone is going the "Ditch torque script route", keep in mind the need to work with existing code unless you fancy rewriting all the script code (core, editor, ...). Also consider the possible need to run on mobile devices which are more of a prevalent platform nowadays.

It seems that everyones favourite language is being put forward as a replacement, maybe it is wiser to finish the engineAPI stuff and stick on a pluggable scripting system?

As for "Optimizing the TorqueScript compiler", I think that's a good step in any case. For a start there's clearly something wrong with function calls currently which I think can be solved with a little refactoring of how function parameters are passed.

from torque3d.

I agree with James and Demolishun ('zat you, Frank?). Ditching TS would break lots of things, but at the same time, I'd really like to see a more modern and non-Torque-specific scripting language available. Making scripting languages a plugin sort of deal would be fantastic.

from torque3d.

@eightyeight,
Yep, zat me.
http://www.garagegames.com/community/resources/view/21706
This is a new Python binding to T3D using SWIG. It is working now. Now, I want to know more about how this scripting plugin is supposed to work. If that is possible and it would be fast enough I would update my resource to use that. I already have a working framework even if it is not perfect. It is based upon SWIG so theoretically it could support any of the SWIG supported scripting languages. I have kept the code separated so Python stuff is in one file and generic SWIG stuff is in another.

from torque3d.

While I agree ditching Torque Script would alienate anyone with a sizable preexisting code base, in it's current form it's just not a viable language for any sort of demanding work.

Plugging a different language in (Lua or JS, I'd avoid python) is certainly a solution and would broaden the user base significantly. As it stands, Torque Script is sort of the bastard child of Javascript and Perl. It's not particularly easy to learn and there are a number of gotchas and inconsistencies with the language syntax. It's easily my most despised feature of Torque.

As much as I dislike Torque Script, I propose a middle ground. Plug in the Lua or V8 VM and compile down Torque Script to the chosen VM's byte code. If done properly, you even have the option of using a mix of languages during development Torque Script, Lua/Moonscript or Javascript/Coffeescript.

As for the reason I'd steer clear of Python. I'll be honest, I love Python. However the standard library is huge and there are, as mentioned, thousands of preexisting professional quality libs. This is great for the developers but can be terrifying when looked at from a security perspective. Sandboxing the V8 or Lua VM's is easy. Sandboxing Python, not so much. This is particularly scary when downloading levels/missions from an untrusted source when joining a server.

Python is not a bad option, but if chosen, we would need to go over it's implementation very carefully.

from torque3d.

I've looked at this project at-a-glance, so far. I've compared it to Second Life. When the VM bit was mentioned, that got me interested. I thought about the Torque3D version of SL. One thing I would do differently than SL is separate the VM and 3D into separate processes. That works good with high frame rates (tested SL with ReSTful queues).

I wouldn't bother with any 3D SDK if it doesn't aim to work in VMX mode instead as an SDK.

from torque3d.

I am pretty sure Lua could coexist with Python at the same time. I agree on the sandboxing issue with Python. That has always been an issue due to the introspection of the language. Lua could handle sandboxed code, TS could handle legacy code, and Python (or insert your favored language here) could handle everything else. Also, levels and the like should not be downloaded in script form. They should be XML and so they can be parsed. Levels as script is not a smart thing to begin with.

What I am really concerned about here is there are lots of "I would want this", but no action. I am willing to put forth action, but I will be supporting Python and other requested SWIG languages. The reason I am using SWIG is because it is something I understand and has multiple language bindings, including .NET. I also know how to write libraries for Python. From what I have seen of Lua it would be even simpler to support. If someone wants to work on a complete replacement of the VM I will help, but unless I see action I am going to take the route I prefer. If we end up with competing choices that would not hurt my feelings one bit. It is actually going to make the engine that much better by having choices. I fully expect the .NET version already in the works to be used by a lot of people as well.

To be honest about TS. I don't like it. It tries to be C/C++, but it is not. I have started a rewrite of the VM to use functions for each op code rather than a switch statement. That was my first step to optimizing for eventual JIT implementation. The function code takes advantage of tail call optimization to reduce return calls for each opcode. When I compared startup times it was about the same. So at least it has not hurt performance. I was hoping to lay the ground work for a faster version. So if any VM experts would like to get a hold of my work I can make it available. Oh and we should point out that missions as script are certainly NOT sandboxed 'as is'.

"Python is not a bad option, but if chosen, we would need to go over it's implementation very carefully."
From what I have read is "Don't use Python for untrusted code." So supporting a sandboxable language in addition to anything else would be a good idea. So I like your Lua idea for that type of application. Also, we can design the interface to make it so you can comment out any language a person does not want. If someone only wants Lua, we can oblige.

from torque3d.

@Demolishun
My hesitation stems from Garage Games apparently already implementing V8 for some of their service work. I'm fine with V8, I'm just waiting word if it's going to make it back down the pipeline to the public repo here. If it's not, then yes, I'd very much be interested in working on integrating Lua. It's small, it's fast, it works on every platform I can think of, it's easy to learn and use, and I have a decent amount of experience integrating it into projects.

However, I'd like to make it clear that my preference would be to rip Torque Script out entirely. It's a horrendous language and adding further abstraction to support both TS and Lua will only slow both implementations down. I understand that this would alienate anyone with prior code bases. Considering Torque has just moved to be open source under the MIT, it might be best to treat this beast as a completely new engine.

from torque3d.

Virtual machine extensions allows us to skip the sandbox analog. Anything graphic intensive should be written with VMX. On machine that don't support VMX, you need the sandbox analogy of VMs, which is access control on the host. In guest mode you don't need that overhead, as the guest OS can deny all device access except to cache and display.

Past 3D engines redo their VM and become web browsers (with V8), as that is common development phase. I rather see you use VMX, and move forward.

from torque3d.

@Dzonatas
Maybe I'm getting my terms mixed up, but I think we're talking about a very different type of virtual machine. We're talking about VM's that interpret bytecode (CLR, JVM, etc.), not hardware sandboxing and segmentation.. The two are not analogous in this context.

In addition, VMX tech is not supported on any console or embedded device that I'm aware of (iOS, Android, Xbox 360, PS3, Wii, etc.)

from torque3d.

@Demolishun I think it is a great idea to implement the SWIG interface!
Just looking at the languages on the compatibility page it would give support for java, ruby, python, perl and seems like it can even handle c#!
But as you already have it running with Python, do you have any idea of how much faster it is?
I'm with Demolishun on this one.
On a sidenote how much work is it to make it work with new languages?

Btw SWIG compatibility page: http://www.swig.org/compat.html

from torque3d.

@Dzonatas,
Like BrentTaylor pointed out. VMX is not applicable here. Nobody wants to work below the OS. Unless I am missing something is there VMs for scripting languages using the VMX interfaces?

"adding further abstraction to support both TS and Lua will only slow both implementations down"
I don't think it will. When you create a TS console command you are basically writing a C++ function that automagically creates a TS command. This same macro could create any number of bindings to other scripting languages. The call from the scripting language to the actual function would be direct with nothing in between. So it would be just as fast.

Throwing out TS is not a good idea. There is too much existing code showing how to use the engine. Perhaps a couple years down the road as people start using other interfaces. Or even rewriting the TS compiler to target another VM would be an option.

from torque3d.

There is another problem about using another scripting language. The C++ in many places calls TS functions. That would have to be reconciled if the scripting language gets replaces. Even SimObject creation is called using a script function from a C++ call.

from torque3d.

@lukaspj,
Those are not automatic. It still requires work to add each language. I don't know if the interface is faster than TS. There is a Python class abstraction in between. I tried to get rid of that and it would not compile. The goal was not to replace TS at the time. I have done some novel things that do speed up this interface. For instance you can literally create callbacks in either direction. You can also import Python objects into TS. That way TS can use those objects as if they are TS objects.

For speed I would need to re-evaluate the calling mechanism, I think. That is what I was referring to with the console command macros. You should be able to create a Python command that calls the C++ code created when you create a console function. Object lookup would have to be re-evaluated as well.

There are ways to speed up the interface and SWIG has support for getting rid of the class abstraction layer. I tried to have it produce an interface with the abstraction class turned off and I got all sorts of errors. So I decided I wanted to finish the interface as is so I could continue working on my game. I mostly wanted all the libs Python offers. It is currently not an ideal substitute for TS, but it works.

When the Python code calls TS commands it does so through Con::exec. I believe this is pretty close to the metal, but there might be faster ways. I am certain there are better ways to interface the engine for speed.

from torque3d.

@Demolishun
Ah, this is where we have a methodology disagreement. I greatly dislike using Macro's for things like this. They tend to be overused and make debugging painful. Using Macro's, I agree, would not slow the system down one bit. Doing it properly, certainly would. However, let's hold off on the discussion of how the "proper" way to do this is. Let's get some of the bigger issues settled. What course of action do we take? Fix TS in it's current state? Rewrite it to compile down to the bytecode of a proper and well maintained VM? Rip it out and replace it with another language? Or do we write a plugin system for it?

I fully understand the arguments for wanting to keep TS as an option, but as I've mentioned, I really do believe that we should be treating the new release of Torque as a completely new engine. I see little reason to hinder it's direction at this early stage.

As for TS being so fully ingrained in the C++ source code, I'm very much of the opinion that this is part of the problem with TS as a whole. Regardless of whether we keep TS or replace it, this needs to be cleaned up dramatically.

As for a plugin system, here are my thoughts on the subject:
On the surface, I love the idea. However I do feel the cons are rather significant. Primarily that you'll be segmenting the community. Resources will be in a number of languages, languages that not everyone is likely to know. Unity suffers from this as do frameworks like Leadwerks. A certain amount of uniformity is a Good Thing(TM).

EDIT: @'d the wrong person. My bad.

from torque3d.

@brenttaylor I read the new XBox has VMX, and last I read Atom processors do not. I am aware of game loops and console-only modes.

@Demolishun Anything that wants full screen mode will want to be friendly with the hypervisor, which runs with VMX. One simple setup of the system would be Torque3D, ASP.NET/XSP, and linux with no other packages. Absolute minimal before you rid of the kernel itself. As said, you don't need to go that far, but along the way any system under VMX doesn't have to deal with all the BS from the other device drivers and window managers. We can deal with it from the HTTP server direct to device driver.

I have an agent-region in mind that detaches from the user. Games usually deal with inventory, and we need regions, not sandboxes, where that inventory can be as active as the frame rate. Skew is the common reason for buffer bloat, and we could get mesh optimization into the kernel or its module. The user mode programmer sees this connection between graphics and regions usually different than what is optimal for meshes, which obviously need the physics processor. In other words, after minimal, we need the 3D server to interop or co-op with the physics server.

There may be many, and at that point the co-op methods scales. I like avatar to have its own region if it wants to compete. I want each avatar with its own specialized VM. The server needs to be agnostic, and thus the choice in high-level language is irrelevant to the VM.

from torque3d.

@Dzonatas
Again, we're talking about two completely different things here. Even so, this is a very bad idea. On consoles and handheld devices we're resource constrained as is. And again, most of these devices don't support VMX. If the new XBox in development supports it, I'd love a citation about it. None of the other consoles or handhelds support it.

from torque3d.

@brenttaylor The ideal script language you want is COLLADA. Realize that when any process serves those files that other devices only need to request resources that relates to that process or more optimally embeds it for other processes, devices, and cores. The search engine has your citation: "new xbox vmx", came up top.

It was not any bad idea when it was hard work to get it this far towards open standards.

from torque3d.

@Dzonatas,
You are speaking completely different language. I don't understand any part of what you are saying. I did a search and found 0 libraries or implementations referring to VMX. The onus will be on you to educate us and create/find a reference application to show how this should be done. Your text makes me think you are either using a translator or we are talking to a bot. For ex: "Skew is the common reason for buffer bloat, and we could get mesh optimization into the kernel or its module." This makes no sense in any context.

COLLADA eh. Maybe this should be embedded in the Neko VM usign ASP tech. Perhaps we can use SSE3 and VW instructions to optimize the bit rate flow for pixel spark reduction? Should we be using a Wankel Rotary Microprocessor like the Propeller too?

from torque3d.

@Demolishun I'm glad I'm not the only one who thought he sounded like a markov bot. It could just be a very bad translator, but that's not the vibe I'm getting.

from torque3d.

@Demolishun I don't judge, and maybe that is why it didn't make sense if you expected to be judgmental. I wanted to meet the people that are involved this project, and this discussion seemed like it was one good start. In games, it is always easy to find disagreement and difference, or follow any other trend.

VMX already exists in many stationary computers. It makes sense in that context, so no complaints. I think you can answer your own questions.

VMX is hardware instructions, so it is not some library. It sounds like it is new tech to you.

@brenttaylor That attitude is why projects fail. Good luck.

from torque3d.

@Dzonatas,
If you want to pursue helping with the VMX just show us by writing some code. I will do the same with my approach. I saw on your page you have worked on .NET/Mono. Maybe you can bring that to the table? I just don't think it is necessary to work at the hardware level. It is hard to take someone serious when they mention Collada as a scripting language. Collada is a graphics interchange format. It has not flow control constructs useful to writing game logic.

from torque3d.

@brenttaylor,
You have some good ideas that I will think about. It would be very nice to have a language like Lua for sandboxing. I will be modifying/redoing the ScriptT3D project in the near future for T2D. At that time I will evaluate the effort to include Lua. It won't be an ideal interface unless I see a simple way to use the macros as I mentioned before. I would like to understand how the modular code is supposed to work as well. I may have a look at that too. There are still issues with the macro approach as it does not completely address dynamic function and variable creation in TS. I may have to find a mechanism to make that transparent to any scripting languages attached. Right now I have solved that through console variable lookup, but that is a string lookup and just as slow as TS variable lookup. It is a similar story with SimObjects. Although with those I can import them as objects into Python, but methods still have to be looked up.

from torque3d.

@Demolishun Anything in XML is like scripts to me, as it is simply not code. It works with canoncalization, however.

As an interchange format, COLLADA needs its asset server, but I wrote HTTP server, above. The format does have sections for shader languages and object definitions, so it doesn't forbid it. Plus, it is more dynamic than that. People tried to go direct from COLLADA format to render, and they missed the whole point of the exchange features.

There is no need to be the most abstract while being agnostic about any new code except that there are truly genuine asset that are quite unique and available under open standards. What's impossible is for anybody to replicate that unique data, and COLLADA does not define how to access unique asset data... only the most common in the format mentioned.

Scripts and shapes, the two things that categorize 3D worlds. You know how to transform XML into XHTML with ECMA. JPEG2000 allows us to embed XML, so already every face on every object can have its own script when extracted from the JP2 format based on modern standards.

from torque3d.

I'm sure we can talk all day about whose scripting language or upcoming technology is the best replacement, but my intention with this Issue is to suggest a realistic solution to the problem which I and many other Torque developers have faced: at the moment TorqueScript is fine for small-scale coding glue, but once you go beyond that you start to run into issues with performance.

I've experimented with a few more extreme methods myself without much success, including even changing the function parameter passing so it passes native types instead of just strings (sadly this introduces too many other problems without much speed benefit).

Looking in the profiler, besides the interpreter i'm seeing a lot of time being spent in:

• Dictionary::lookup
• dSprintf (called from Con::getData, StringStack::setFloatValue, Con::setIntVariable, Con::setFloatVariable)
• dAtof called from Dictionary::Entry::setStringValue and StringStack::getFloatValue
• dAtoi (similar to dAtof)
• Dictionary::reset

Looking at this I'm pretty sure at least variable conversion can be optimized as a start.

from torque3d.

@james,
You mentioned the engineAPI.h. What do you see needing to be completed? I am kind of clueless as to how that actually works. I just don't understand why that is better than the old console command binding methods. I do like the idea of a pluggable scripting system, but I don't understand how it works. Is there a whitepaper somewhere in the game industry about that type of system?

After doing the work to convert the VM to function calls versus a switch statement I lost interest in spending time writing a VM. There are way more talented people than I with already fast and proven VMs. So I would like to avoid trying to bring TS up to speed. I am more in favor of grafting a VM in some way onto T3D. It may not be pretty, it may not be good code, but I can use it now rather than 1 to 2 years from now. So some sort of TS and cyborg is better than just TS now.

from torque3d.

@Demolishun engineAPI.h itself isn't a pluggable scripting system, it just provides an abstraction so you can define engine objects and methods using the DefineEngine* macros, which allows you to define functions using native types (as opposed to an argv[] string array). From what I can tell when you call these methods from TS, it uses the normal Console API but automatically converts all the string parameters to native types. If you look in engineDoc.cpp and engineXMLExport.cpp, there are a few examples of how to enumerate this information. Looks a bit incomplete but maybe with a little work it could be good, which is why I mentioned it.

from torque3d.

Oh, okay. I had no idea that worked. I saw the consoleXMLExport was all defined out and when I tried to enable that it did not work. The only issue would be commands that take variable amount of arguments, but that could be handled with the standard console export. This could indeed make things much simpler. With SWIG I can predefine data type conversion as well. This would make my current interface better if it works.

from torque3d.

I just tried using exportEngineAPIToXML() and it creates an object with no data. So this is not working at the moment. If it can be made to work then this will make creating interfaces to any language simpler. Now I see why you were so interested in this. You help my mind do that "mental click". Thanks!

I put in a counter and it never finds any engine exports. So to make this work I need to understand what it missing then.

from torque3d.

@Demolishun I was going to try to take this message off issue but couldn't find an email or other method to contact you directly (since I hate adding to an already long issue).

Some examples of C++ plugin systems within game engines would be Ogre3D (albeit not for scripting, but it would be the same general idea), PixelLight IIRC actually has scripting as plugins (though, it makes heavy use of the RTTI throughout the framework and I think uses this to help creating some of the bindings.) Careful not to use any code from PixelLight though, it's LGPL, Ogre3D is fine if you want to use their plugin loading mechanism. The actually "plugin" doesn't have to be dynamically loaded (which will actually require platform specific code like dlopen), but could be done within the compiled engine and called/instantiated based on a defined interface. (I kind of think this discussion should be taken off issue onto the forum.)

from torque3d.

@losinggeneration
I'd have to disagree. This is exactly where this kind of thing should be discussed. It doesn't become an actual ticket until someone tags it and this is a more open place for that discussion. I'd think it's best to keep it all in one place.

from torque3d.

@brenttaylor Except the majority of the discussion would be better suited in a different issue at the very least (since replacing TorqueScript isn't really relevant to this issue.) So I stand by that this isn't the correct place to be talking about possible plugin systems, if TorqueScript should be removed, adding other scripting, etc.

from torque3d.

This was Eric Preisz's suggestion:
"options. 1) Optimize the compiler with V-tune, fix things, and test the heck out of the engine. 2) Ditch Torque Script."
I think we have been talking about option 2.

from torque3d.

@Demolishun Heh, I had actually forgot about that specific comment. I stand corrected.

from torque3d.

So, while we're on the subject, coming from a position of relative ignorance (I've never installed a scripting language in an engine, always just used whatever was there)... can someone lay out some of the actual steps necessary for anyone who was going to either replace Torque Script entirely, or else (preferably) add another scripting language as an additional option to work side by side with Torque Script? How and where do you have to start cutting to actually do that, and how does the job vary depending on which language you choose?

Also, is there a reason when people are casting around for a good choice, that they don't just go straight to Mono/C#? Given the power, dependability, similarity in syntax to C-family languages, cross platform compatibility (for Mono anyway) and widespread industry use (eg Unity), it would seem like the obvious starting place to me... unless hooking it up is an order of magnitude more difficult than some of the other options.

Feedback? Very curious on this front... Just had to fix a set of totally random script errors that popped up while porting to the MIT T3D build. The code in question looked completely perfect, but the common thread turned out to be the presence of preceding comments that had three slashes at the beginning of the line, rather than two, ie:

///
/// comments...
///
function myFunc(%arg) {...}

If I changed those to look like this:
//
// comments
//
function myFunc(%arg) {..}

.. then they compiled just fine. I for one am more than ready to start looking at well supported third party scripting languages.

from torque3d.

And since we're on the subject, shall we start compiling a list of competitive GUI options as well? Qt, anyone?

from torque3d.

@ChrisCalef
While there's nothing wrong with Mono/.NET, it's a pain to embed compared to options such as Python, Lua or any number of others.

Unfortunately ripping out Torque Script and replacing it is quite the undertaking. It's embedded throughout the engine itself. Adding another language alongside it isn't terribly difficult, it just doesn't really solve the problem which is speed.

As for GUI options, this isn't really the discussion for that.

from torque3d.

Any specifics as to exactly what makes it more of a pain than the other ones? (Just curious, because I'm very tempted to go that way, but don't want to walk blindly into another WOOOORRRMMMMHHOOOOOLLLE if you know what I mean... )

Re: running alongside, it would solve the speed problem at least for any new code that gets added, and it might pay off in the short term to convert only the existing Torque Scripts that are creating the biggest lags, while avoiding the nastiness of converting everything and ripping out all the hooks in the engine.

from torque3d.

@ChrisCalef
So much of the engine actually calls TS functions and does a lot of it's work in TS that running another language alongside it isn't going to solve a lot of the problem. Any new code you write, will in theory be faster, certainly. The internals of the engine however are still going to chug along.

As for why Mono is more of a pain...
Lua, for example, has a very simple API designed specifically for embedding itself in a code base. It's trivial to work with. It really is a few function calls and exposing functionality to Lua. Calling Lua is also very simple.

Mono on the other hand is actually very simple to initially set up and call an assembly to start working. Exposing functionality to C# is also fairly trivial. The hard part is calling into C# from C++. It's technically trivial to do, it's just not a small amount of code. You however do need to be very careful when keeping references and pointers around. You can get very subtle bugs. But again, I'm very much looking at it from the perspective of replacing Torque, not embedding a language to work alongside it. For information on embedding Mono, take a look here:
http://www.mono-project.com/Embedding_Mono

from torque3d.

There is already a .NET script language in the works and it will be a produce of WinterLeaf which you can purchase. They have already done the hard work of integrating and to me it would be worth the price.

@brenttaylor,
I don't think there will be a speed issue if the added scripting language calls the C++ call directly. The console actually calls the C++ directly, but it has to do string lookups and string parameter parsing and those are slow. I think James pointed out the issues above. With a language such as Python or Lua you don't have that slowdown. So if you can keep the scripting language from depending on the console and export functions and objects directly then it will actually be faster. The problem is exporting the information. As I pointed out above the function that is supposed to provide the map to this is not currently working.

Now, there is a compromise that could be made. That is have the scripting language lookup the function and objects then create a reference to those so the next time they are accessed it is faster as it would be direct access to the object. This is something I have partially implemented. This does not require the binding code to work.

I am starting to think that we should not abandon TS right now. However, if we did graft on a VM that worked fast by calling functions directly then it would be a little more effort to have the TS compiled for that VM. Then the TS would benefit from that speedup. So you don't lose the code and you get the speed. So people would not even need to program in a different language to still get the speed advantage.

One place to start is the code James pointed out. That is the code that supposed to export the binding points. It is literally a mapping of all the commands that are exposed to script in the compiled DLL. So if that were working a mapper function could be written for any scripting language. So that should be made to work to reduce the hacks needed to connect to the engine now. I know a lot of the hacks because I came up with a bunch!

Do we want to take a stab at this for a few weeks and see where we get?

from torque3d.

@Demolishun
Having whatever language we tack on call C++ functions directly, as mentioned, only solves half the problem. That part I understand and it's relatively trivial to do. It's been done before, as far back as the old TGE days with Minions of Mirth and their Python resource. :) But again, that only solves half the problem.

As you mentioned before: "There is another problem about using another scripting language. The C++ in many places calls TS functions. That would have to be reconciled if the scripting language gets replaces. Even SimObject creation is called using a script function from a C++ call."

If your intention is only to speed up new code, then what you propose will work just fine. The engine itself should see a rather impressive speed boost however, if we solve the real problem. As I proposed before, we can either replace Torque Script with another language or we can keep the TS language syntax but compile it down to the Lua (or V8, or what have you) VM's byte code.

The second option solves both the speed issue and allows everyone to keep their old code bases. The downside is that the interface to TS internal to the engine is likely to change. So we'd have to do quite a bit of cleanup. I think it's worth the effort. I think it's the overall best solution to keep everyone happy and get the speed benefits which are the real issue.

However, I'd still prefer to just remove Torque Script all together. I just doubt the community, as it is today, will really want it. People looking to use the engine now that it's under MIT...that's another matter. I highly expect anyone new to the engine to be highly put off by TS.

from torque3d.

You are right and those calls to the console would have to be intercepted. That should not be difficult.

I actually ported the original PyTorque to T3D and created a resource. That was my first attempt. However, it had problems. One thing is it did not release the GIL properly. This made it slow and did not thread properly. This was fixed in that first resource. I am guessing PrairieGames fixed this in their internal version for their game later. I also changed the callback mechanism when I rewrote my own resource using SWIG. The original PyTorque used a hack to get the namespace. I made a new console function type that included namespace information.

I don't really understand it, but I want to see if we can get the engineAPI stuff working. I hope there is someone hanging around that knows what that was supposed to do. That would help solve the issue with connecting to another VM.

I expect to have some competing implementations. When the dust settles we will have choices. What path were you thinking of taking?

from torque3d.

What path? For me, that depends entirely on if Garage Games is releasing their V8 implementation or not. If they are, then I'll likely not bother with the effort. It also depends on what the committee decides to do about it. If they do have a specific fix they have in mind, I very likely will help in that effort. As I said, a certain amount of uniformity isn't a bad thing.

Otherwise, I'm debating between Lua and C#. Both are battling in my mind and I'm unsure who is winning.

from torque3d.

The intermediate layer is more important than choice of script.

"The client's script is the client's script."

WIth Debian's dependency boot, T3D team should focus only on the Debian package and no more. Debian users need someone that supports the package online, which means commitment to update the package such that it compiles clean, standalone. That can be achieved through deb-src selections instead of binary selections. This allows us to template the installation more by client's choice of front-end scripting language.

Instead of binaries and interop, we only the need the assembly list by unique identifiers (i.e. IPv6). T3D only needs to be compatible with wine-lib. That saves you all other maintenance and development required to port any Windows source to Linux or other flavor. Any backward compatibility with T3D is still non-free, like Wine, so derive from it's Debian package. You really want to aim any game engine for its own IPv6 on Wine. Interop is then as simple as bridge on demand, UPnP.

from torque3d.

No matter what language anyone chooses there will always be others who believe language "this-other-one" is superior to language "that-one". Personally I would favor C# > Java > Lua > Python out of the possibilities already presented.

I would think, and hope, that for general use in the main stable branch that the Steering Committee would favor any optimizations and fixes that could improve Torque Script performance over blanket removal of it. To appease those who would argue which language was better a plugin system would not be amiss. This would allow greater flexibility for everyone, and not just users who favor scripting language "whatever". In personal branches it would not matter.

from torque3d.

Well, I have decided to be a participant in this development community. If people want a voice I suggest backing it up by effort unless they want to remain a spectator. I will favor improvements to TS, but I am not going to wait a year to get it. I will be putting pull requests for whatever I come up with. I would rather have code in hand and a finished product than cool ideas.

from torque3d.

Hmm. Well there's a lot to think about here, but personally I'm still leaning hard toward Mono. I'm in a different position from many people here, because my primary use of Torque at this point is Ecstasy Motion, for which I'm looking more at how I can make things comfortable for my customers than I am looking at what language I personally want to use. For this, due to its use in Unity and general flexibility and power, Mono/C# seems to be the winner.

I'd be very interested in backporting much of the existing Torque script code over time, but my first priority would really be making C# (and possibly other .net languages) available as an eval option for Ecstasy Motion users to write their own code, so I don't have to go out in the big wide world with my app and tell people they have to learn Torque Script to get anything done.

Count me in if a Mono interest group develops, otherwise I will probably put my fork out there as a test case.

from torque3d.

Let me rant...

Cool ideas to me work in co-operative mode in Win32/Wine32. Don't even bother with 64bit except for IPv6 and (future) SSE/APU.

Make embedded systems the plugin-system, as it works at all levels of cooperation, like OpenGL ES. Some people use persistent streams too simulate congestion. OpenStack sounds like an appropriate abstraction for cloud performance of the service stack. You could add C# linqs with ServiceStack on github.

Maybe Torque Script can be converted to JSIL# as it's back-end, and you won't have to worry about JS maintenance from the auto-generated aspect.

I know, NDA may make someone state if they understood. Speed improvement by more separation of script context, like 4k memory games to 9GB HD transformation in printed media... meshes. The file owner can indicate apartment partitions in shared memory, and those partitions can be given IP, with name that symbolically links to the IPv6 device drive: /dev/[IPv6] or sh //[email protected]/resource/uuid.sh.xml or mailto:json.agent@domain.... with private TLDs. COLLADA helps keep the content consistent while the context is wild, yet we use unix to keep uncooperative systems tame. Easier if you state your license for single core CPUs or more. I mean, like your intent, so nobody steps on each others toes after we fork. It would be nice if we could upload XML readable files that allows us to tag and abstract as an alternative to fork with the same flow. At least create the "exclude list" of proprietary files that is off-topic for discussion on any thread here. Printed Torque meshes printed from COLLADA format may help us visualize what we "don't understand."

Oprofile is nice. I haven't seen "oprofile#" innovated, yet any transient scripts can use rsh() or NaCl.

Mono helps you stay with the MIT license, and the Apache license doesn't become the client's dependency. The common denominator is X on XEN, which would be your headless mode. For example, X disables hardware access except to subsystems. Shell scripts still..., rock on... help us with #!/usr/bin/t3d.exe and #!/usr/bin/games/t3d.exe. Thanks; we need this difference across unix, like standard, or it won't be considered stable, as it becomes too expensive for each of us.

Start with the solution, use .net.exes from Visual Studio, and set it to 3.5 for standard and 5.0 for experimental. If I fork it, I would submit the 3.5 standard ECMA compliant build system with regression tests for Mono. You would only need to make sure the regression tests are portable.

"You," rant over... it doesn't have to be you, yet this is my contribution here before there is any we-fork-ng. I rather not lose my focus from the OpenJPEG solution, so my obvious source contribution is the wrapper. No promises at this internet level.

from torque3d.

Hey guys, the V8 work I was referring to is actually server side script for a project we are working on. The client side (where we are using Toqrue) uses Awesomium so we didn't need to implement V8 since Awesomium already does. It's not an implementation of V8 in Torque. But we are at least well versed in integrating V8.

I personally (which wasn't the original point of the thread) would prefer to work with C# just because I like it as a language, but don't you think Mono would only benefit a smaller audience since it's LGPL? LGPL would prevent Torque from running on iOS. The only way around that is to buy a license which costs $400 for personal use. Unity purchased(s) a reseller license (I don't know the details or terms) to distribute Mono across multiple platforms. No such license could exist in an open source engine. Also, I've heard a lot of complaints about the memory footprint of Mono on smaller devices. Someone correct me if I'm wrong...it's been well over a year since I last spoke with the Xamarin guys (they built and then bought Mono after Novell dumped it when Novell was purchased)

I guess it all depends on your goal.

I also wouldn't mind seeing Javascript. While I think it's a horrible language to develop in (IMO it worse than TS for OOP), there are certainly many, many people who already know it. Plus, it's the only scripting language that will be usable for HTML5. I think it would be nice to some day export a T3D level into a WebGL engine and simply port over the game logic without needing to implement a cross-compiler.

If performance really is the issue (at least it was the start of the thread), I'm confident that there's a lot that can be done with Torque Script. Optimizing TS would be the fastest path to that solution and I'm certain there's a lot that can be done to modernize the work underneath the hood. When Torque Script was implemented, machine bottlenecks were very different than they are today.

from torque3d.

@ EricPreiszGarageGames,
If performance was the only issue I would agree. Code maintainability is one. External resources is another. I really think getting the engineAPI interface working to allow attaching of scripting languages to be automated would win lots of developers over. If that were to happen and work were done to optimize TS then you would satisfy just about everyone. Those that want another language would have the interface for it. Those that wanted a faster TS would have it as well. There would be no need to pull TS out. The small issue of the C++ using TS to create objects or for callbacks could be addressed/intercepted if needed.

I am starting to agree with James. It is really pointless debating languages until the infrastructure (engineAPI and TS) is addressed. TS is not going away any time soon.

I have the current VM converted to function calls for each opcode. There are some minor bugs, but the engine does run and the normal script startup works correctly. I did this in a stock version of 1.2. The reason I pursued this was to try and take advantage of Tail Call Optimization. Also, I believe that in order for TS to be converted to JIT the opcodes needed to be turned into discrete functions. Perhaps I misread that, but that was part of the motivation. Do you guys want a copy of that to help with optimizing TS?

Edit:
I want to help out, but after looking at engineAPI I got lost as to what needed to be done. If someone who knows that code can point me in the right direction I would gladly look at this. I don't expect GG to do all this. I want a part of this.

from torque3d.

Wow. I haven't seen a TorqueScript vs. the world discussion this lively for some time. It is great to see this level of interest in the guts of Torque 3D again.

If I remember my history correctly, TorqueScript's (the .cs extenstion is for C Script) purpose in life was to perform data definition (eg: Datablocks), glue compiled source code together, and allow designers to implement some game logic without needing the help of a source code programmer. If a critical path through script was too slow, then it would be rewritten in compiled code. Real games have actually been shipped using TorqueScript.

Today, we tend to abuse TorqueScript and expect it to operate like the higher level languages we're used to. Right now, TorqueScript is heavily integrated in Torque 3D. All of the game code and tools rely upon it. To do a total conversion to a different scripting language is a big job in itself, and then you have to rebuild the user facing components. I'm not against this so long as those working on and waiting for it know that it would be a long term project.

Personally, I would have to agree with James U. I would rather make the TorqueScript system the best that it could be. This work could be done with an eye towards being able to replace TorqueScript in a pluggable way -- I know that C# has been done in the recent past. Of course, plugging in other scripting languages can fragment the community into now needing to know multiple languages to implement different resources. I feel for those who are not programmers when that happens.

It will be interesting to see what comes out of the Steering Committee meetings. :)

from torque3d.

@ericp, thanks for the info re: licensing restrictions on Mono, did not know that, how unfortunate.

Taking that into account, with an eye toward maintaining cross platform compatibility and MIT licensing, brings me back around to square one, I guess. I'll wait and see what the great minds in the Steering Committee come up with...

from torque3d.

@DavidWyand-GG That makes sense. So a guess a branch of this discussion should be: "Do we want (insert scripting language here) to be a language for developing complete game logic or just for small stuff like datablocks and small amount of gamelogic"
Personally this is one of the topics i briefly cover in the guides i recently wrote for beginners to T3D.
And I am using it for prototyping AI, and stuff like spellcasting. Functions that is only run around 1 time per second maximum.

from torque3d.

The argument I share is that we know games existed on the market that was processed by T3D. If we think of those builds as real estate then, we do not need the script engine/VM in the same package that is more for graphic processes. This is where developers, like Unity, decided there is no marketability for games in Linux. Note where I asked to help us in the difference between /usr/bin and /usr/games/bin, as games in general are not professional enough for /usr/bin. If we run /usr/bin/t3d.exe and get some VM prompt (that should be in /bin like /bin/sh) then, it confuses the application layer.

UNIX organization improves rapid development. Gates only hid the shell prompt, so that /bin/win.exe (instead of /bin/sh) would be the default environment and IDE for anything in /usr/games/bin. UNIX comes with its own script interpreter, so to leave Torque Script as-is and avoid further VM maintenance is probably your ideal path of less resistance. This organization doesn't make sense under DOS (or cloud containers), so Microsoft didn't need to explain /usr/* to anybody, like AT&T and SCO did. There still is this minimal difference due to cooperation that still lets both OSs to run under one single core, concurrently.

It's easier for us to discuss speed optimizations with real-time protocols, like RTP. If anybody (re)writes the plug-in system then, it should be based on RTP, and I have seen people that have written their own licensed signals, like SL. http://www.ietf.org/rfc/rfc1889.txt

from torque3d.

@DavidWyand-GG
Keep TS and make it better and faster, possibly even JIT compiled. Finish the engineAPI interface (I will help). This gives programmers simpler access to support their preferred language. For instance Lua would be an excellent resource for sandboxing user code, Python would be excellent for bringing in 10s of thousands of external packages that are already proven and well maintained, .NET gives you MS tech for those that need that or any of the thousands of packages it provides, JS for web tech, PHP for servers, etc, etc, etc.

Edit:
Where is the ignore button? Maybe I can Code Monkey one in.
Done:
demolishun.net/code/http://demolishun.net/code/GitHubIgnoreUserComments.user.js

from torque3d.

Well I fixed the 'consoleExportXML' TS command. It was having issues with enumTable generation. Turns out I could have commented that out because it did not return any to begin with. Does anyone know how usable that function is? Can I use it to capture all the interfaces to the DLL?

Edit:
Awe crap. It does export function information, but the only data it has is function name and min and max args. So basically it is going to take a string for those inputs and it will have to do string processing on the arg no matter what. It may be faster than going through the VM, but you still end up with string processing of inputs. So, yes it can be used to interface via the DLL it looks like to me. However, it will not be as close to the metal as it could be.

So the console commands need to be reworked to have typed parameters if we want this to be faster for external interfaces. It will still be of benefit to use that commands as is though for external interfaces. I am going to look into this for my ScriptT3D. It may be worlds faster than having to lookup console commands. Can't help pure TS commands, but the C++ ones can be faster. It will also give me some introspection!

from torque3d.

Mono would be just fine for use with T3D. You have to read the fine print.
Mono Licensing

from torque3d.

Yeah, seems pretty legit to me... what would be the particular problem with iOS?

from torque3d.

Great discussion here!

I just want to reiterate: I'd really like to see Lua replacing TS at some point. I've researched game scripting languages quite a bit (with an eye to replacing TS), and Lua consistently came out on top. The language syntax itself isn't my favorite, but I like it better than TS, and the overall combination of features, including performance, code libraries, and general ease of integration, trump the other scripting language options out there. It's also been used with success in AAA games like World of Warcraft and Farcry, which says something.

So... Yes, we should keep TS in the interim, and yes, it'll be some work to fully integrate Lua into T3D, but I think it would be well worth it in the long run.

from torque3d.

The key problem would be completely rewriting MonoTouch or Mono for Android from the ground up if we did not want to make people purchase a $400 Xamarin license per platform.

Sent from my iPhone

On Sep 24, 2012, at 5:09 PM, Chris Calef [email protected] wrote:

Yeah, seems pretty legit to me... what would be the particular problem with iOS?

—
Reply to this email directly or view it on GitHub.

from torque3d.

Ah, I get it now...

"We only require licensing for uses of Mono and Moonlight on embedded systems, or systems where you are unable to fulfill the obligations of the GNU LGPL. "

But still, might be worth a community Mono fork at least, could be highly useful for anyone using T3D for a non-mobile application, or anyone who is aiming for mobile but willing to pay the $400. That's less than half what T3D used to cost, after all... :-)

from torque3d.

Okay, I read up on V8. BSD license, faster than most likely anything else out there, huge code base, natural extension to web code, etc. This would be a really good replacement for TS. I am also guessing it would require fewer code changes to TS thus maintaining the codebase to some degree. GG is already up to speed on it and that makes it more attractive.

Even with this change, I am still going to vote for a component system for adding custom scripting languages. Best of both worlds. You can use V8 and add on any specific purpose language you might need.

from torque3d.

@Demolishun
So, where are we at on this thread then? Are we working to refactor and optimize TS or are we working to replace it with JS or are we working on engineAPI? I would like to know how I can help but I do not really know what the goals or milestones are or what iterations there will be.

from torque3d.

I've been experimenting a bit with optimizing the type conversion in the interpreter but so far I've not had much luck. The trouble is there's so many type conversions going on in places it's hard to nail everything down. I'm beginning to suspect a more fundamental revamp of the bytecode generator is needed.

Since some of you are interested, I cooked up an example on how to use engineAPI: https://gist.github.com/3780691 - technically speaking considering all the information is there, you should be able to call any engine function and create any engine object using native types. You could very easily create a Python binding for the engine DLL for example.

from torque3d.

@jamesu Oh thank you! I will get on that soon. I just needed a nudge and that should do it.
@dasper Well, I am ping ponging i know. After reading up on V8 I don't think replacing TS with V8 would be out of line. The syntax is similar and the VM is highly optimized with lots of money behind it. It is here for the long term. My short term goal would be to improve performance and API for external VMs. Long term I think V8 is a good thing.

In the next couple of months I can look at the engineAPI code and see if I can improve my existing interface and potentially expand to include other VMs or at least provide a ref implementation. I am eyeing T2D to get the interface as well. I kind of want to see what GG is doing with T2D though. I know I can now rip the libraries needed from T3D to keep the code bases the same. I had decided to work on the components of T3D so I will pursue the engineAPI as part of that. I feel pretty good about the interface, but if it can be made faster that is a good thing. Better to be on a stronger foundation.

The reason I am looking at T2D is because people were asking about for it. The more users using it the faster issues will be fleshed out and the better the code will be. It is just a lot easier now with T3D going MIT.

from torque3d.

@Demolishun Sounds like you have a pretty good idea of the direction you'd like to head. I know V8 is a serious consideration because, like has been said, GG is already familiar with it, and ECMAscript is known by a very large group. Another VM worth looking at that tends to be very fast is LuaJIT. What I'd actually suggest looking into is seeing which VM (V8 or LuaJIT) has an internal representation that would be easiest to compile down to, get the VM worked into the project with the original language (ECMAScript & Lua respectively), and then it can be looked into if TS can be compiled directly down to byte code for that VM. I don't think TS would need to be removed while getting the new VM & scripting language in. Then if it could be compiled down to that VM, it should actually speed up TS as well.

from torque3d.

The runtime libraries are under the GNU Library GPL 2.0 (LGPL 2.0).

It's this section that makes it difficult. Many companies won't use LGPL because it's a particularly vague license that hasn't been tested in court. To paraphrase, LGPL states that you can use the library (usually a .dll) as long as you give someone the right to replace that .dll with their own version of the .dll. On a Windows/Linux/OSX machine that's no problem. On iOS or any other closed system, you violate that persons ability to do the switch so you need the commercial license. In addition, Monotouch and Monodroid have a lot of functionality that would need to be rebuilt or licensed from Xamarin. Btw...Just to make a note...I think they Xamarin guys are really cool...they are super talented guys and if you like what they are doing, I can recommend their services. I don't want to make it sound like I'm trashing those guys.

from torque3d.

@EricPreiszGarageGames

Second Life switched from GPL to LGPL in the client source, so there is some experience. I rather have it as the GPL version because then we know the source code is more truth than documentation. People often get confused how to share the source. The difference is BSD/MIT are permissive in the source but not in the documentation. GPL is permissive in the documentation but not in the source. It is simpler when we want to know the flow of truths and intents of the code. LGPL tries to balance the two licenses for those that need context removed.

LGPL code is like documentation and samples for GPL code. What doesn't continue to work is trademarked language (any) as we go from GPL to BSD/MIT. It's not intuitive or automated in anyway to make that happen easy. The debate over trademarked language and any intuitive automation (of it) is endless. The effort behind every possible interop is best spent with conversion to ReSTful interfaces than predict every aspect, and skip the entire license issue.

Script performance is greatly improved by ReSTful interfaces. The engineAPI could be converted to more ReSTful context; otherwise, it's like reverse polarity, which is not always so obvious in code. I rather suggest that the UDP version of engineAPI() may be easier to mess with than any license issues of interop. UDP is good and lets us keep context separate. The published scheme of the data goes much further than any API and its wrapper. This keeps us friendly.

from torque3d.

@Demolishun The value of Quaternions in that code monkey script, it creates the basic hypercard with the before and after state, like the web-browsers back button. Maybe that was an example of newer Torque Script and how context changes.

from torque3d.

GG may want to look at http://schema.org/.

If that is of no interest (to GG) then, I could blog about T3D's schema with AdSense enabled. I knew there was some good use for AdWords, like an API. Oprofile# on IntelliSense reveals much.

from torque3d.

If anyone wants to move the interfacing discussion to the GG forums I am game. Like I said I will be pursuing this to understand how it works and see if it improves what I already have. I am really impressed by what V8 is. It can only get better.

I am thinking it is enough of an offshoot of this discussion that it warrants moving. I also am wanting to reduce the signal to noise ratio of the Markovian factorizations being implemented in the chaotic regions of spacial repository conversation. If you catch my drift...

from torque3d.

I would agree. On a side note, I've done work on a JIT VM for torque, and it's not a small piece of work but in the end it's been worth it. The biggest performance bottlenecks I've identified with TS are the reliance on the string table / dictionary and all it's lookups and all of the type conversions on the fly. I would much rather see us involve LLVM into this as it would solve a lot of problems with an intermediate assembly language, allow us to bind the T3D engine to the VM the way we see fit, and keep it manageable. Plus IT'S MIT Licensed already! The only caveat to this is windows because of the lack of a linker outside of Visual Studio. Though if we are only looking for scripts, we don't need a real linker for the scripts since they won't be put to a binary executable, they will just stay as the intermediate binary code. This would allow us to create an actual VM, create a binding layer and optimize the hell out of the scripts in any language we want a front end for. We can add compiler optimizations very easily using the LLVM model as well. I would much rather go this route but it's not without work. Either way, if we choose V8 or something else too, it's still about rewriting language bindings for that specific VM and I would much rather we kept control of that ourselves.

Edit: Mispoke on the license. It's University of Illinois/NCSA Open Source License, a BSD derived license
http://opensource.org/licenses/UoI-NCSA.php
Though, that doesn't change it's usability or commercial viability for anyone, at least I can't imagine how it would.

from torque3d.

@Smally With V8 I was thinking wholesale replacement of TS to JS. The syntax is close enough to TS that the conversion would not be difficult and we would no longer be maintaining a VM.

Of course if there is a way to modularize that interface we might be able to componentize the VM insertion and allow for swapping in anything. Hmmm...that would be interesting. Start with V8 and shoe horn in while modularizing the interface. Wow, that could be a kick in the pants feature wise. This really sounds like it could be fun.

from torque3d.

I just didn't think a javascript vm would be appropriate and lacks a lot of flexability, where as with the LLVM route, you have all the flexability you would ever need and then some. We could also pull in the TorqueScript AST and have a frontend for torquescript rather quickly, which is why I suggested that route.

from torque3d.

I've never implemented LLVM, but I've always been a huge fan. I've always assumed that LLVM was how Marmalade was doing their cross-compiling (would love a confirmation on that). Are we going to move this mega thread over to GG.com?

from torque3d.

It sounds like there are three main points emerging from those who've been profiling TorqueScript:

• Type conversions
• String lookup for variable names and so on
• Function calls

Aside from replacing TS, what can we do to improve these three factors of the language? I'm not going to profess any knowledge of the console process here, but I'm willing to learn.

From a user's point of view, I think the type conversions issue is one we could pick on. Everything being a string feels really dodgy, even if that's mostly hidden from the user. There are times when I look at a script and see something like %var = "1.0"; and cringe a little.

from torque3d.

@eightyeight - Unless someone has ruled out memory (cache misses) as a problem, it probably is. #2 in your list is probably the killer here.

I'm trying to push out a deliverable I'm working on, but when I'm done, I'll break out VTune and see if I can get to the details. I'm friends with the optimization guys at Intel and I bet they would lend a hand if I'm willing to put together a case study/presentation on how we made Torque Script faster.

from torque3d.

Ok, created a forum topic to continue this conversation there: http://www.garagegames.com/community/forums/viewthread/131792

This should give us the playground for the topic ;)

In regards to cache misses, Yes, it's a problem.

from torque3d.