See this Zulip chat thread.

Errors in e.g. module execution do not appear as popups in napari; they are only printed on the console. They should appear in napari as well.

See this zulip thread for context.

We should be able to support common types that are enums or enum-like, including:

• StructuringElement, used in the labeling.cca Op
• OutOfBoundsFactory, of which many implementations could be reduced to an enum.

Running Frangi_Vesselness on the Embryos sample image from Fiji takes several seconds on my machine. During that time the napari UI blocks and is unresponsive.

Questions/thoughts:

• Do plugins need to be run on the graphical dispatch thread in napari for any reason?
• Does napari already have a progress/work indicator mechanism?
• Are there any barriers to launching plugins in a separate thread and popping a progress bar?

So that you can do a bunch of ImageJ stuff at once, in its native UI.

@ctrueden said:

4. Pressing the down arrow from search bar doesn't drop down into search results lists. Pressing tab first, then using the arrow keys, works.
5. Select filter.gauss(image, image, sigmas) op.
6. Pressing enter does not run the op. Neither does double clicking it. Neither does the Run button nor Widget button do anything. The reason is that a Java exception is thrown (see following posts). But unfortunately, we are not dumping the Java exception stack trace to the console. We should do that using scyjava's jstacktrace function.
7. Then try filter.gauss(image, image, sigmas, outOfBounds) instead. Pressing enter does not run it, instead dumping a stack trace again. But double clicking does run it, with the magicgui dialog appearing. Which op does the enter key actually try to run here?

napari_imagej._module_utils._add_param_metadata is broken.

This line is the source of the failure; ij is a function that returns the imagej instance used by the plugin, thus it must be called.

This slipped by as the try/except surrounding this call catches all Exceptions, while ij().py.from_java() can only throw TypeErrors.

Steps to fix this:

• Change ij to ij() in that function
• Change the except Exception to except TypeError.
• Add tests for this functionality to ensure this is fixed.

Here we have threshold.apply - I think this is mapping to this op whose parameters a) not images, I think?, as reported by the napari-imagej ui - and also if they were images they are not being represented properly in the widget.

If I actually try to run this I get an exception:

I'm not sure this how this Op would actually be runnable from napari-imagej. Are the search results already restricted to image-applicable results? and is this just a case of misinterpreting the params as images?

These are indistinguishable, flood the results, and will be confusing to users who see no reason to choose one over the other.

Napari has a Tracks layer that cannot currently be converted into something usable by ImageJ. This would be a problem save for the fact that nothing in PyImageJ really uses tracks/tracking data.

While there is certainly tracking functionality in the ImageJ ecosystem e.g. Trackmate, Mastodon, that functionality is not available without Fiji; we don't expose Fiji in this plugin. Furthermore, there is no abstract tracking data structure in Imglib/Scijava, which would be the prerequisite for conversion.

@tinevez, @ctrueden and I decided to postpone this conversion until we have a use case and available SciJava/Imglib data structures.

It would be convenient if:

• Double-clicking a search result created the widget for it
• Right-clicking a search-result opened a context menu with the same "staging" options (Run, Help, Source, Batch)

The offending commits:

• 4bb94c3
• 24c2ba9
• 1bdc8cc
• eb2203a
• 7e8f579

Thanks for @ctrueden for noticing!

When running a SciJava module with a BOTH (i.e. input+output) item, right now, a new layer is created. Instead, napari-imagej should skip creating a layer in favor of refreshing the existing input image layer, since it was mutated inplace.

@ctrueden says (in #14):

I'd much rather create a napari-trackmate plugin or so instead of trying to shoehorn that functionality into this plugin.

That is fair, and I think I agree. But I do think you should be able to, with napari-imagej and no other napari plugins installed, initialize your ImageJ2 with whatever init string you want (including a locally wrapped ImageJ2 installation if desired). That way, people can invoke Fiji plugins with the search bar, for example—they just won't magically gain additional data structure conversions beyond those fundamentally supported. But for the vast majority of such plugins, they won't need anything extra.

Hmm, how do we add this functionality without it becoming an annoyance? If I just want stock ImageJ would I want to click through a startup screen every time I want to start the plugin? I'd prefer that this was some setting you could specify.

Maybe we could have an "napari-imagej-startup" plugin that allows you to modify where your imagej comes from? 😆

SciJava features the ColorRGB class, which does not work in napari-imagej. This was made clear by #60, where I had to comment out the ColorRGB parameter.

I'm not sure that we can support this yet, as magicgui has no support for a color type, so entering a color would be hack-y and would have to rely on a Line edit.

Is there even really a demand to support ColorRGB? @ctrueden

A peculiar set of circumstances cause segmentation faults when running the plugin.

1. The JVM must be running
2. A Callable with a faulty signature must be passed to magicgui.

This is not specific to napari-imagej; see gselzer/napari-foobar@993f3019298bb744e0c139afde8c5612dae9ad81for a reproducible example. It is, however, going to affect napari-imagej; if anyone has napari-imagej running, and tries to create any widget that is improperly written (likely due to incorrect param_options), napari will silently crash.

Under the hood, napari is calling VerboseTB to display the error arising from the call to magicgui. VerboseTB tries to access the locals at each frame in the error traceback, causing a segfault when looking at the _function_gui frame. The VerboseTB docs suggest it prints all variables in the stack; if this is actually what happens, then it is probably trying to call data managed by jpype. JPype seems to still be running by the time we reach the segfault.

See https://napari.zulipchat.com/#narrow/stream/309872-plugins/topic/Passing.20parameter.20options.20to.20magicgui/near/273885825

In a zulip chat with @ctrueden we talked about how napari-imagej will overwrite outputs of functional Ops. This is because napari-imagej returns its data layers via LayerDataTuples,, and napari uses the passed name to denote layer uniqueness.

We want layers to be overwritten when they are the output of Computers (i.e. we currently get the intended behavior). We don't want this when the layers are the output of Functions.

Thus, when the output is pure, we should try to work with napari's functionality for unique names. We'd want to use the name given to the output (usually out, but who knows), with napari's standard suffixes in the case of a duplicate layer name.

To reproduce this image:

1. search for AddDimension
2. select View: AddDimension
3. search for frangi

At this point it looks like frangi is selected, but we actually still have AddDimension selected.

Also both the Ops and modules panes seem to track selection separately:

I think:

• selections in one pane should clear selection in the other
• searching should clear the "staged" selection for creating widgets

Thanks to @ctrueden for finding this.

The bug is here

@ctrueden thinks that ij.py.from_java is recursive, so we probably don't even need this check.

I blame @elevans for this bug as he told me it wasn't recursive 😜

The Manual Threshold... .js script creeps into the search results and its ImagePlus parameter doesn't map appropriately. This will have to be handled appropriately in conjunction with imagej/imagej-legacy#264

The Widgets.js script template exercises a lot of parameter types. Let's make sure it fully works in napari-imagej. (Right now, it crashes on the non-integer step size parameter.)

At the end of the day, TypeMappings seems to be promising more than it actually does. One might think, reading that documentation, that all keys and values in that dict are concrete types. This is not the case, however; many Python types are implied as forward references. Even if we resolved these forward references, things like napari.types.ImageData aren't types, so we couldn't use TypeMappings.ptypes values in e.g. isinstance.

We probably have to rename napari_imagej._module_utils.python_type_of to be clearer about when it should be used.

• When typing in the search bar, if you press enter, it should do "Run" on the first module result
• When pressing the arrow keys down, you should be able to go down into those result lists and select things (and enter to run)
• When a result is run, the keyboard focus should jump to the first keyboardable component of the created widget. For widgets with no inputs, that might just be that widget's Run button.
• When pressing enter from any component of a module widget, it should do Run.
• Other nice things that occur to us along these lines.

There has been a lot of discussion within napari/napari#4579 about whether the parameter harvesting/initialization widgets should be persistent or modal. Let's hash it out in our own issue here.

This discussion was started by @ctrueden:

@gselzer I am starting to wonder, now that you have modal dialogs working, whether we should just always use them, in the same way that Fiji does. It is much simpler when hitting enter to run a command in the search bar, to have the dialog pop up, which you can quickly fill in or just press enter again if the settings are already correct, to continue with the execution. Whereas popping the new widget down into the panel clutters things up, as we are discussing here. To what gain? Why do we need this persistent widget with inputs? I prefer modal dialogs.

If modal dialogs are considered unnaparic (wink) then we can stick with persistent widgets as we have now, but then of course the behavior of napari needs to be better when there are many of them, as discussed here.

The discussion continues after the linked comment.

I don't think this button should be called "Run":

It's adding a widget for the selected command right? So, let's say something to that effect (with a verbose explanation in a tooltip)

Having two widgets that both contain a "Run" button, where only the latter truly "applies" the plugin results, makes for a confusing user experience.

If it's trying to run the Op I think it should not, and just always make a widget. If it's trying to make a widget and hitting an exception then that is a separate issue.

Edit: this was a separate issue

The napari assistant is extensible. We could expose the functions to that interface so that they can be included as part of assistant workflows, undoable/redoable, etc., as the napari assistant continues to develop.

Our current search is synchronous, which slows us down. @ctrueden informed me that the SearchService can do this asynchronously for us!

Recently, the Source and Javadoc action buttons disappeared. On my system at least, these buttons were working, and I would prefer they be restored, since they provide more transparency to how the algorithms function. We could rework them to take up less screen real estate (little buttons!).

As you add widgets from ops/modules they stack endlessly on top of each other. This also increases the minimum height of the napari window which leads to other display bugs. It would be safest to just make the widgets scrollable.

This seems reproducible. Here are the steps.

1. Make sure that you have the work of #64.
2. Open napari, boot up napari-imagej. Initialize an Op that takes a preallocated output. I use filter.gauss(image, image, sigmas)
3. Run the Op, does not matter what the inputs are so long as inputs exist (i.e. you need inputs, but the inputs don't have to make sense). The important part is that the module executes; throwing an error during that execution due to invalid inputs is fine.
4. Close napari
5. Open napari again, boot up napari-imagej again.
6. Initialize filter.gauss(image, image, sigmas)
7. See this error:

File ~/code/imagej/napari-imagej/src/napari_imagej/_module_utils.py:667, in functionify_module_execution(viewer=Viewer(axes=Axes(visible=False, labels=True, col...ings._transform_active_layer at 0x7f5f8654bb50>}), module=<java object 'org.scijava.command.CommandModule'>, info=<java object 'org.scijava.command.CommandInfo'>)
    665 # Run preprocessors until we hit input harvesting
    666 input_harvesters: List["jc.PreprocessorPlugin"]
--> 667 input_harvesters = _preprocess_to_harvester(module)
        module = <java object 'org.scijava.command.CommandModule'>
    669 # Determine which inputs must be resolved by the user
    670 unresolved_inputs = _filter_unresolved_inputs(module, info.inputs())

File ~/code/imagej/napari-imagej/src/napari_imagej/_module_utils.py:190, in _preprocess_to_harvester(module=<java object 'org.scijava.command.CommandModule'>)
    188     return list(preprocessors)[i:]
    189 # preprocess
--> 190 preprocessor.process(module)
        preprocessor = <java object 'org.scijava.module.process.LoadInputsPreprocessor'>
        module = <java object 'org.scijava.command.CommandModule'>

java.lang.IllegalArgumentException: java.lang.IllegalArgumentException

The (temporary) fix, suggested by @ctrueden.

1. Open up the console from within napari, either by clicking the lower-left console button or pressing ctrl+C
2. enter from imagej import ij; ij.prefs().clearAll()
3. Initialize filter.gauss(image, image, sigmas) again, everything works fine.

More digging needed to determine why the LoadInputsPreprocessor throws the exception. Maybe because it cannot find the Numpy arrays that were passed in?

Can anyone reproduce? cc @ctrueden @hinerm

@gselzer made a good point in his code review.

If a Module has multiple outputs, we should return it in a dict, or in some other collection. This is possible

napari-imagej requires a mapping of python types (or type hints) to "equivalent" java types. This mapping is not designed to be exhaustive; instead, it often uses interfaces (e.g. net.imglib2.RandomAccessibleInterval) to cover large amounts of types. It then relies on methods like Class.isAssignableFrom and ij.convert().supports to determine whether a subtype should use a type in the mapping.

Unfortunately, there are many cases where, for a given java type, multiple mappings might apply. This is particularly an issue for ij.convert().supports, where the DefaultConverter seems eager to suggest a confusing type. For example, when testing python_type_of I found that DoubleArray inputs convert to a python int, since DefaultConverter knows how to convert a java Integer into a DoubleArray. The sequence of actions leading to this is:

1. python_types_of calls canConvertChecker. since DoubleArray is an input, it checks to find a tuple (jtype, ptype) in TypeMappings such that the conversion ptype -> jtype -> DoubleArray is valid. If it can find such a tuple, it knows that a magicgui widget providing an instance of ptype can be converted into a DoubleArray through pyimagej and the ConvertService.
2. It iterates across these tuples in the order they are declared within TypeMappings, and returns the first ptype that satisfies the chain. The tuple (java.lang.Integer, int) satisfies first because DoubleArray has a constructor that takes an integer, meaning the DefaultConverter can convert an Integer into a DoubleArray 😦

To avoid a large deal of work, I simply rewrote TypeMappings.ptypes as an OrderedDict here: f0ba973

This commit allows us to pursue order in the dictionary as a form of priority, but it is obviously not good enough.

We might want to instead iterate through all (ptype, jtype) tuples, finding the highest-priority handler of all tuples that could satisfy a given java_type, and use that ptype. It should be doable.

Commands like Blue that call ApplyLookupTable do not work. They convert the Image layer into a ImgView, and then pass it into the ImageDisplay parameter, which naturally throws a NPE when the Command tries to access nonexistent fields of the ImgView.

These commands would be nice to have, if we can map LUTs to napari's ColorMaps.

We can remove the banner once we feel like napari-imagej has reached a state where most things work as expected.

SciJava module items have various metadata, including:

• name
• label
• type
• min/max values
• default values
• widget styles

We should map as many of these as possible to their magicgui equivalents.

For required=false parameters, we may want to use Optional[...] so that they are allowed to be unset.

On main, starting up napari, starting up napari-imagej, and then closing napari results in the following exception:

Exception during shutdown callback: java.lang.UnsupportedOperationException: java.lang.NullPointerException

This also happens in CI pytest runs. I do not see it consistently running pytest locally.

It does not seem to affect the running of napari-imagej in any way.

As per The Hitchhiker's Guide to Python, proper repository setup should not include a src file. napari also follows this approach.

src/napari_imagej/_tests/test_ntypes.py currently tests the conversion of Labeling objects to Labels layers.

We do not test the conversion capabilities of ImgLabelings to Labels layers. There is thus functionality (caught here) that is untested.

Thanks to napari/napari#4831, we no longer need the return type logic of _module_untils._widget_return_type and can instead directly state the return type List[Layer] here.

This wouldn't actually introduce any new functionality, it'd just make the code cleaner 😄

But we will have to wait for a new release of napari for this.

MutableOutputWidget graphically shows a layer selection on the left, and a New image button on the right. However, in widget lists, the button comes first.

This leads to unexpected behavior when e.g. delegating focus to subwidgets, elaborated on here.

Ideally, left-to-right should be the same order as it shows up in the list of parent's subwidgets.

It would be nice to test that the conda environment builds in different environments just like we do with pip.

For some reason, though, we cannot build the environment on CI. I have had a great deal of trouble getting conda-incubator/setup-miniconda to work on different environments.

Using mamba, I get this failure. This issue looks something like mamba-org/mamba#78, but the remedy for that issue seemed inapplicable for our Github action.
Using conda, I get this failure.

I cannot replicate any of these issues on my linux machine, or on my windows machine. Furthermore, the particular conflict within pyimagej does not seem to be an issue within the pyimagej Github Actions 😕. For now, I'm just going to continue testing conda as I had in the past; if I cannot replicate the issue locally, I'll hope that it is restricted to CI and users will not have this issue either.

It might be worth filing an issue in the conda-incubator/setup-miniconda repo, although I'm not sure how much they'd be able to help.

PRs #11, #12, #13, and #15 all build with passing tests and are strict improvements, but that does not mean that they don't need review.

I (am going to) review each in its own PR, but I want to make the changes in separate branches atop all of them to minimize the need for rebasing. This will also make things more convenient for @ctrueden.

The following PRs have had all review addressed:

• #11
• #12
• #13
• #15

Right now, all menu items appear directly under Plugins > napari-imagej, due to limitations in how the napari_experimental_provide_function hook works. We'll need to work with the napari core devs to figure out the best way to render the whole collection of SciJava modules according to their preferred (nested) positions in the menu.

E.g. "Plugins > napari-imagej > Plugins > Debug > System Information" rather than "Plugins > napari-imagej > Plugins Debug System Information"

To start ImageJ, we currently use multithreading (here). I'm starting to think that multiprocessing might be the better play. It might also make the lazy initialization of ImageJ easier to handle.

There isn't really a good way for us to generate preallocated outputs in napari. napari doesn't really seem to provide an easy way to make a blank image layer, making preallocated image outputs especially tricky.

We should pass add_legacy=False to imagej.init for now, so that no-legacy commands appear in the search results. Otherwise, the functionality usable from napari-imagej is greatly reduced.

Until imagej/pyimagej#197 is resolved, we cannot support Mac for this plugin. The clash between imagej, which needs to run headless on mac, and napari, which makes very little sense running headless. As there seems to be an incomplete separation between the two on Mac, we will not support that OS until the issues with it are resolved.

Running the math.add(arg, value) op results in this exception when generating the widget panel:

See #48 (comment)