rsmenon / matlink Goto Github PK

Since the MATLAB COM server needs to be registered anyway on Windows, theoretically it's possible to auto-detect the location of the required dlls (libmx.dll and libeng.dll)

This would remove the requirement to add MATLAB's bin\win64 directory to the PATH on Windows and would avoid some user confusion.

The steps required are these:

1. Read the location of MATLAB from the registry. Throw a user-friendly error if not found.
2. Modify mengine to use delayed library loading
3. at the start of WinMain(), set the correct DLL search path

I am not going to work on this right now and because of lack of time and because this will become irrelevant when we switch to the MEX API. However, if anyone would be willing to contribute these changes, contact me and I'll walk you through the steps. It'd take a day or so if, like me, you're not too familiar with the Win32 API.

MSet["x", symbol], where symbol is a symbol with no ownvalue, gives a MATLink::noerr message. It should give a MATLink::errx message and indicate that symbol is the culprit. Compare MSet["x", {symbol}], which report the error correctly.

In the long term, we'll need to support transferring Inf and NaN from MATLAB to Mathematica.

At the moment these quantities show up as Inf`` and Nan`` in Mathematica and they are unusable (their behaviour is unpredictable even under simple operations such as ==).

Example of a function that typically generate useful Inf values: voronoin.

I think this can wait until the next big release though.

Notes: Because of the unreliability of Inf`` and NaN`` in Mathematica, this will need to be implemented on the C++ side. Arrays should be scanned for these values, and their indices transferred separately.

Add error checking to option values.

I accidentally used MFunction["eig", "Output" -> 2] instead of MFunction["eig", "OutptutArguments" -> 2]. There was no indication that this was incorrect.

Empty strings are returned as {}. This is because size('') is [0 0] and all empty arrays are returned as {}, regardless of their type.

Fixing needs to be done in C++.

Originally, using checkcode() was introduced to work around the problem of engEvaluate hanging with incomplete input. Currently MEvaluate executes commands by writing them to a script, so checkcode is not needed for this purpose any more.

Proposal:

1. remove checkcode from MEvaluate to reduce overhead
2. add checkcode to MScript and MFunction: it should be run when a script is created
3. If possible, make MFunction verify that the file it creates is a true function and not just a script (is there a MATLAB function for this)?

Test case:

Right after startup, do

In[18]:= MEvaluate["pwd"]

Out[18]= "
ans =

C:\\Program Files\\MATLAB\\R2013a

"

Add license description to the package.
Need to decide if license should go in every source file or only in an auxiliary file.
License needs to be updated on the webpage.

A cell of empty matrices is incorrectly returned as an empty list.

In[44]:= MEvaluate["x=cell(3,4)"]

Out[44]= ">> 
x = 

    []    []    []    []
    []    []    []    []
    []    []    []    []

"

In[45]:= MGet["x"]

Out[45]= {}

structs have the same problem:

In[60]:= MEvaluate["s=struct(); s = [s s s; s s s]"]

Out[60]= ">> 
s = 

2x3 struct array with no fields.

"

In[61]:= MGet["s"]

Out[61]= {}

MEvaluate doesn't use the JIT compiler on Windows

Test case:

MEvaluate["
  x=zeros(1e7,1);
  tic;
  for i=1:1e7
  x(i)=rand();
  end
  toc"]

Takes 11 seconds on Windows, 0.3 seconds on OS X. (For comparison, it take 16 seconds on OS X without JIT, run on the same hardware.)

There may not be a solution for this for as long as we use engEvalString() to submit commands. Note that the MATLAB Engine implementations on Windows and Linux/OSX are completely different.

MFunction checks if engineOpenQ 5 times per call. While it is negligible for a one time use, it is noticeable if the function is called repeatedly (in a loop).

Currently MATLinkmEngine`` is added to $ContextPath because `BeginPackage[]` (not `Begin[]`) is used in `template.c`.

We need to figure out how to change this to Begin[] and keep the package working.

Also need to ensure that engEngine[] returns type wrapper symbols in the MATLinkmEngine`` context.

Before the beta (not final) release, the following parts of the documentation need to be completed:

• Reference section that lists all user-visible functions (Names["MATLink*"]`)
• ???

MEvaluate["disp('hello\nworld')"] (or MScript) gives a "A quoted string is unterminated." error, because the \n in the input is interpreted as a new line when writing the script, which leads to a MATLAB syntax error.

MSet allows creating structs with invalid field names

In[13]:= MSet["s", {"1" -> 1}]

In[14]:= MEvaluate["s"]

Out[14]= ">> 
s = 

    1: 1

"

This is really a MATLAB problem and it should be low priority.

One instance where this may cause trouble is when setting a struct that came from an unusual source (e.g. it may have come from Import[..., "JSON"] in Mathematica, and the user may not have noticed that some of the field names are not MATLAB compatible)

Reproducible crash on Win64:

Implement multi-element struct handling in C++

Creating an MScript and overwriting it right away fails sometimes. Unfortunately this is not consistently reproducible, and seems to be timing related. Paste the code in a notebook and use Evaluation -> Evaluate notebook to reproduce the issue.

Note that the output should be Goodbye!.

Here's the plain text code:

<< MATLink`

OpenMATLAB[]

(*create a script*)
hello = MScript["hello", "disp('Hello world!')"];
MEvaluate[hello]

(*overwrite and test again*)
hello = 
 MScript["hello", "disp('Goodbye!')", "Overwrite" -> True]
MEvaluate[hello]

MScript[] will often, but not always, fail on Windows:

In[29]:= script = 
 MScript["testscript", "disp(123)", "Overwrite" -> True]

In[30]:= MEvaluate[script]

Out[30]= "Undefined function or variable 'testscript'.

"

$sessionID should include a random string. The link names are not totally random. I often get warnings that the session directory already exists when I start MATLink twice within a minute.

If MATLAB crashes because of an MEvaluate, it triggers StringFreeQ::strse in MATLink

I don't consider this a MATLink bug because MATLAB just shouldn't crash (reported to MW), but it's something worth documenting.

The following input crashes MATLAB R2013a on OS X (but it leaves mengine running)

obj = DelaunayTri(rand(10,2))
struct(obj)

We can use this to test how MATLink would cope with a crashing MATLAB.

MScript and MFunction should warn if attempting to create a script that conflicts with the name of en existing command.

Perhaps use exist()

Support MSet[]ting arbitrary character arrays such as {{"a", "b"}, {"c", "d"}}

It would be very useful to have a test suite that tests both basic functionality and edge cases.

Ideally there would be one test quite for the eng* functions and one for the M* functions, but since we're short on time, let's do the M* ones only for now.

This is important to ensure that unsafe changes (especially to the C++ code) do not break anything, or ensuring that porting to a new platform (e.g. 32 bit) has worked.

We haven't thought of the possibility of using 32-bit MATLAB with 64-bit Mathematica or vice versa. Both of these are possible and easy: we just need to launch the version of mengine compatible with MATLAB (the "bitness" of Mathematica doesn't matter)

This is important because on Windows (and Linux for MATLAB < R2013a??) the student version of MATLAB is 32-bit only (see here)

Currently MATLink checks the "bitness" of Mathematica and launches the corresponding mengine.

I propose adding a "Force32Bit" option to ConnectEngine and to OpenMATLAB. (This is also what InstallNET has in NETLink.) If there's demand, we can also add a "Force64Bit" option, but this is not likely to be needed.

Notice the "!" where we should see "中". When using 2012b on OS X I get the correct "中".

Needs["MATLink`"] gives no problems, however OpenMATLAB[] fails with the following error message :
OpenMATLAB::noopen: Could not open a connection to MATLAB. >>

This is what I get in my console when starting Mathematica
$ mathematica > /tmp/firstrun.txt
set.cpp: In function ‘void eng_make_String(const short unsigned int*, int, int)’:
set.cpp:159:31: warning: narrowing conversion of ‘len’ from ‘int’ to ‘mwSize {aka long unsigned int}’ inside { } is ill-formed in C++11 [-Wnarrowing]
set.cpp: In function ‘void eng_make_Struct()’:
set.cpp:191:5: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]

---Start firstrun.txt
/opt/Wolfram/Mathematica/9.0.1/Executables/../SystemFiles/Links/MathLink/DeveloperKit/Linux-x86-64/CompilerAdditions/mprep -prototypes mengine.tm > menginetm.cpp
g++ -O -Wall -I/opt/Wolfram/Mathematica/9.0.1/Executables/../SystemFiles/Links/MathLink/DeveloperKit/Linux-x86-64/CompilerAdditions -I/opt/matlab/R2013a/bin/../extern/include -L/opt/Wolfram/Mathematica/9.0.1/Executables/../SystemFiles/Links/MathLink/DeveloperKit/Linux-x86-64/CompilerAdditions -L/opt/matlab/R2013a/bin/../bin/glnxa64 menginetm.cpp mengine.cpp set.cpp get.cpp main.cpp -Wl,-rpath-link,/opt/matlab/R2013a/bin/../bin/glnxa64/ -lML64i3 -lmx -leng -o mengine
---End firstrun.txt

Getting an array of size 300,000,000 fails on Win64
See the LargeArray.mt test file

Test system: Windows 7 64 bit, Mathematica 9.0.1, MATLAB R2012b

This array has less than 2^31 - 1 elements, but it takes more than 2 GB to store.

Consider this mistake of omitting a comma:

MSet["x" 1]

MSet will simply not evaluate in this case, and it will not show any errors. This is something confusing to debug if the MSet call is part of a larger function.

MSet should issue an error at least in the case when it's called with only one argument. I think the mistake of calling it with more than two arguments is not as important. The error can be the built-in General::argx, issued as Message[MSet::argx]

MSet[] should report the cause of failure whenever possible to make it easier to find why it failed with a complex object (e.g. a struct). If should report the offending sub-element in the error message.

If should indicate if the error is coming from the MATLAB Engine instead of the Mathematica layer.

MEvaluate is too slow because MScript, which it relies on, need to use rehash every time. This is only required on Windows, to pick up newly created scripts. It should be removed on other platforms and preferably replaced by something faster on Windows.

The MATLAB engine function engEvalString() and it's mEngine equivalent engCmd[] hang when given incomplete MATLAB commands such as "x=[1 2" or "for i=1:10".

We need to implement a wrapper which calls eval or evalc in MATLAB, detects if the input is incomplete or has a syntax error, and reports accordingly.

This problem is not caused by d5fe543

Logical vectors are returned as a column-matrix, not as a vector. They should be returned as a vector, just like numbers.

x = {True, True, False, False, True, False};
MSet["x", x];
MGet["x"]

    (* ==> {{True}, {True}, {False}, {False}, {True}, {False}} *)

Allow the user to set a custom MATLink working directory instead of a temporary one each session, so that they can reuse MScripts and custom MFunctions.

The following is a potential, long term improvement to ConnectEngine[]. We can discuss it here.

If for some reason mengine cannot be launched on OS X/Linux, OpenMATLAB[] will appear to hang. The reason for this is that we're launching mengine.sh, not mengine, therefore the process launching itself will always succeed. After Install has launched the process, it'll create a link and wait for an incoming connection, which may never happen. The end result is that OpenMATLAB[] appears to hang.

The typical reason why mengine can't be launched would be:

• On OS X, the user hasn't set the path to MATLAB in mengine.sh, and auto-detection has failed. mengine can't launch because it can't find the shared libraries from MATLAB that it needs.
• On Linux, this is much less likely to happen: if MATLAB can't be found, the point of failure will be auto-compilation, not launching. If auto-compilation fails, MATLink will issue an informative message.
• On Windows the typical cause of failure is that MATLAB is not added to the path. In this case a window will come up saying that mengine.exe can't find some DLLs. So once again, there's an informative message.

It's only OS X where OpenMATLAB[] is likely to hang with no message at all that could indicate the cause of failure.

Proposed fix: make Install time out by wrapping it in TimeConstrained. A default timeout could be 10 seconds. This should give plenty of time on slow systems which might be swapping at the same time. If Install times out, and the OS is OS X, MATLink should suggest to the user to edit mengine.sh and set the path to MATLAB (detailed instructions are included in mengine.sh as a comment)

This is not critical, but it may break things if we make a mistake, so it can be a post release improvement.

This is just a reminder, and does not need to be fixed now.

Certain data structures, such as empty arrays of any class (including empty strings), do not go through post processing functions such as matArray[], matString, etc.

This is not a problem now, but it'll be relevant if we try to implement a $ShowStructure flag which would insert MCell, MStruct, etc. heads in MGet's result.

OpenMATLAB[] should check if opening was successful using engOpenQ[] and report back if it wasn't

MFunction[] should check if MSeting its arguments was successful:

What currently happens is that it gives an error about an undefined MATLAB variable. This is confusing to a user who doesn't know how MFunction works internally.

It would be preferable not to leave a lot of junk in the temporary directory of the system.

I see two ways for now:

1. Either not create temporary files (probably unavoidable)
2. Clean them up on exit. This is problematic again because creating a kernel-exit hook is possible, but messy and error prone

Since changing MEvaluate to use evalc, the help command returns HTML.

Example:

In[15]:= MEvaluate["help pi"]

Out[15]= " <strong>pi</strong>     3.1415926535897....
    <strong>pi</strong> = 4*atan(1) = imag(log(-1)) = 3.1415926535897....

    Reference page in Help browser
       <a href=\"matlab:doc pi\">doc pi</a>

"

This can be reproduced in MATLAB by running it as matlab -nodesktop then comparing help pi with evalc('help pi')

Reminder to re-build binaries before release: OS X, Win64 & Win32

Long MATLAB error get truncated when they're reported in Mathematica.

Example, executed in MATLAB:

>> voronoin([-1 -1; -1 1; 1 -1; 1 1])
Error using qhullmx
QH6019 qhull input error: can not scale last coordinate.  Input is
cocircular
   or cospherical.   Use option 'Qz' to add a point at infinity.

While executing:  | qhull v Qbb
Options selected for Qhull 2010.1 2010/01/14:
  run-id 850880052  voronoi  Qbbound-last  _pre-merge  _zero-centrum  Pgood


Error in voronoin (line 107)
[v, c] = qhullmx(x', 'v ', opt);

Reminder: Run test suite on all platforms before release

MSet[] won't handle SparseArray containing rationals

MSet["x", SparseArray[Range[10]/3]]

We should check how much work it takes to get it to work on 32-bit systems.
If it's not a lot of work, we should make sure it works on 32-bit, provide Linux makefiles and Windows binaries. OS X is not relevant because we don't support lower than 10.6 anyway (minimum R2011b requirement), and the number of 32 bit 10.6 Macs must be low.

Linux netbooks will typically be 32-bit.
Windows XP will typically be 32-bit as well.

ShowMATLAB[] and HideMATLAB[] functions for showing/hiding the command window on Windows systems.

We need to ensure that no unnecessary unpacking occurs during MGet/MSet.