Emscripten compilation runs into undeclared `Tm` when importing `std/times` about nimskull HOT 12 OPEN

In order to have a productive discussion I think the first bit would be to gather/summarize what's provided/required in a wasm environment, as opposed to something like wasi or whatever distinctions that might exist, at which point we can figure out what sort of approach to take.

Generally speak reviews and some guidance aren't an issue. There are two things that become costly:

1. If one or more of us need to spend time developing significant wasm/wasi expertise on an ad-hoc basis. It's a lot of context, unstructured fact finding, etc.
2. Testing, not just writing them but good reliable harnesses, clear and organized description of specifications and requirements, and finally minimal examples

In some ways the above are the same thing, the first more theory and docs, while the second is more practice and code.

from nimskull.

After discussing further with @zerbina here in matrix, a revised approach:

• setup the following --os:wasi and --cc:empscripten
  • testament can do this if we have a wasi target, needs to be added
  • os:wasi implies using the wasi interface, not emscripten's own thing
• make a trivial test(s) for emscripten work with testament, the bigger lift here will be CI
• incrementally add support for the wasi target to various stdlib tests, until the category can be enabled
  • use wasi defines to appropriately enable disable various API
  • the wasm distinction can be added later

from nimskull.

Sounds like a good plan. Thanks @saem . I'll start taking a look at testament sometime next week

from nimskull.

My understanding is that it should only be defined for Windows targets

The Tm is a type imported from the posix <time.h> header, but when building for non-Windows targets, the type is provided by the posix module.

Currently, the times module doesn't account for --os:any, and thus you're seeing "undeclared identifier" errors when importing it. What should happen instead is that either:

1. the procedures requiring some form of OS interaction or system headers are not defined at all when using --os:any
2. said procedures are stubs that report an error at run-time

If you want to use the times module, I think at the moment your best bet is to build with --os:linux, though there might still be the problem that emscripten doesn't implement all necessary functions from the type.h header.

from nimskull.

Okay. For what I've currently got --os:linux seems to work fine. I remember running into problems with that in the past, which is why I switched to --os:any. But for my current project, it seems to be working fine so far. Thanks

from nimskull.

Similarly, things fail when the os module is imported (when compiling for wasm).

Error:

/Users/adam/.asdf/installs/nimskull/0.1.0-dev.21252/lib/pure/os.nim(3514, 15) Error: undeclared identifier: 'paramCount'
candidates (edit distance, scope distance); see '--spellSuggest':
 (5, 7): 'FTW_MOUNT' [var declared in /Users/adam/.asdf/installs/nimskull/0.1.0-dev.21252/lib/posix/posix_other_consts.nim(187, 5)]
 (5, 7): 'count' [func declared in /Users/adam/.asdf/installs/nimskull/0.1.0-dev.21252/lib/pure/strutils.nim(1978, 6)]
 (5, 7): 'count' [func declared in /Users/adam/.asdf/installs/nimskull/0.1.0-dev.21252/lib/pure/strutils.nim(1998, 6)]
 (5, 7): 'count' [func declared in /Users/adam/.asdf/installs/nimskull/0.1.0-dev.21252/lib/pure/strutils.nim(1987, 6)]
 (5, 7): 'parseInt' [func declared in /Users/adam/.asdf/installs/nimskull/0.1.0-dev.21252/lib/pure/strutils.nim(1064, 6)]
 (5, 7): 'parseOctInt' [func declared in /Users/adam/.asdf/installs/nimskull/0.1.0-dev.21252/lib/pure/strutils.nim(1135, 6)]
 (5, 7): 'parseUInt' [func declared in /Users/adam/.asdf/installs/nimskull/0.1.0-dev.21252/lib/pure/strutils.nim(1085, 6)]
 (5, 7): 'pathconf' [proc declared in /Users/adam/.asdf/installs/nimskull/0.1.0-dev.21252/lib/posix/posix.nim(525, 6)] [rsemUndeclaredIdentifier]
lookups.nim(791, 35) compiler msg instantiated here [MsgOrigin]
sempass2.nim(1449, 16) compiler report submitted here [MsgOrigin]
No stack traceback available
To create a stacktrace, rerun compilation with './koch temp $1 <file>'

So I'm starting to wonder if we should have a larger discussion around wasm support because I think there's going to be a lot of stuff like this and I think we should get on the same page as to how much work it's going to be (both initial implementation and for maintenance). Of course, I'm willing to contribute but it would still require some efforts from the core team to guide me and review PRs, etc. So it may be worth estimating how much labor that would be.

from nimskull.

In order to have a productive discussion I think the first bit would be to gather/summarize what's provided/required in a wasm environment, as opposed to something like wasi or whatever distinctions that might exist, at which point we can figure out what sort of approach to take.

To be clear, I think wasi would be the best option. The specification appears to be evolving, but it's the best option for something like stability. And there's a browser shim available. Mostly what I'm thinking is just incrementally fixing modules so that nimskull can output code that emscripten can turn into wasi. The extent of my understanding right now is that there are some modules in the standard library that cause the compiler to error when attempting to compile nim->c with the flags that would make the c consumable by emscripten. And all I can speak to at the moment is the situation on macos. It may be better or worse on windows or linux. Providing functionality that emscripten already provides is a non-goal.

One potential mismatch is that emscripten seems to like when nimskull compiles with --os:any or --os:linux but macos may not. I'm not sure how much that's going to matter in general.

If one or more of us need to spend time developing significant wasm/wasi expertise on an ad-hoc basis. It's a lot of context, unstructured fact finding, etc.

At the moment, I seem to be the only one with much interest. So I wouldn't think anybody from the core team would need to push this effort or champion it to the point of adding features/extending coverage. I know the core team is prioritizing correctness and simplicity at the moment, and I don't won't to undermine that.

Testing, not just writing them but good reliable harnesses, clear and organized description of specifications and requirements, and finally minimal examples

I don't know much about what the nimskull testing approach looks like, yet. So from my perspective, this feels like the biggest effort and possible the area where I will need to most guidance to make sure that the test harness is something that fits into the general workflow.

What I don't want to do is start bringing in PRs and adding to the core team's maintenance burden. I don't have the bandwidth to go whole hog, but I'd like to incrementally fix modules as I need them and make sure there's a test harness in place to make it easy-ish to maintain. But I don't want this to become something the core team has to continually work around if I'm no longer able to contribute in 3 months or something.

from nimskull.

Reading over what you wrote, @theSherwood here are a few things I noticed, which are basically what I identified in my previous message, just restated:

1. we need a map: there is wasm, wasi, and emscripten, that have partially overlapping impact across runtime, os, and codegen aspects
2. we don't have a way to test these changes

I think both these things need to be sorted out in order for any significant changes to stdlib, and possibly codegen.

from nimskull.

FWIW, my current thinking is:

• create a backend called empscripten
  • initially just an alias for c; later maybe extccomp makes it more turnkey
  • backend in testament is target, setup an emscripten target, to run it across various OSes
• make a trivial test(s) for emscripten work with testament, the bigger lift here will be CI
• incrementally add support for the emscripten target to various stdlib tests, until the category can be enabled
  • use wasm and wasi defines to appropriately enable disable various API
  • assume wasm if empscripten, and user can define wasi to open up greater APIs
    • if wasm isn't a strict subset of wasi, then we need to revise this appproach

from nimskull.

Could you tell me more about what you mean about the differences between wasm, wasi, and emscripten? I'm definitely a novice with this stuff, but my experience with wasm over the course of the past few weeks is that wasm is the binary format, wasi is a standardization effort for wasm module interfaces (such as wasi_snapshot_preview1 and the in-progress wasi_snapshot_preview2), and emscripten is a compiler for wasm that historically has used its own module interface but now supports the wasi_snapshot_preview1 interface with an Emscripten compilation flag (PURE_WASI).

My understanding is that the only wasm interface standards are wasi and Emscripten's. I don't know whether there are any particular features of a C program that would make Emscripten unable to compile to the wasi standard, but there may be.

When you talk about a wasm define in contrast to a wasi define, do you mean throwing compile errors on anything that would require any wasi imports? I would think that for an MVP, that could just be a user problem. To my mind, a separate wasm target seems unnecessary. That should be handled with compiler options passed to Emscripten (or clang or wasi-sdk, etc.). Maybe I'm misunderstanding what you're saying. If I have any obvious gaps in my understanding, please point them out.

Another thing to consider is wasm32 vs wasm64. I don't think 64-bit wasm is really a thing yet, but I think it's in active development.

Edit: spelling and punctuation

from nimskull.

Could you tell me more about what you mean about the differences between wasm, wasi, and emscripten? I'm definitely a novice with this stuff, but my experience with wasm over the course of the past few weeks is that wasm is the binary format, wasi is a standardization effort for wasm module interfaces (such as wasi_snapshot_preview1 and the in-progress wasi_snapshot_preview2), and emscripten is a compiler for wasm that historically has used its own module interface but now supports the wasi_snapshot_preview1 interface with an Emscripten compilation flag (PURE_WASI).

• wasm is a binary format, but it also implies some basics for browsers can load/work with wasm modules, IIRC
• wasi is an interface
• emscripten is both a compiler toolchain and defines at least two interfaces, it's own and wasi

When you talk about a wasm define in contrast to a wasi define, do you mean throwing compile errors on anything that would require any wasi imports? I would think that for an MVP, that could just be a user problem. To my mind, a separate wasm target seems unnecessary. That should be handled with compiler options passed to Emscripten (or clang or wasi-sdk, etc.). Maybe I'm misunderstanding what you're saying. If I have any obvious gaps in my understanding, please point them out.

The defines are for the stdlib, these are compile time errors, not compiler errors. Anyhow, if this stuff is ambiguous that raises core dev workload because now we need to know lots of intricacies in order to reason about whether a define is ambiguous/sloppy or whether it's sensible.

Another thing to consider is wasm32 vs wasm64. I don't think 64-bit wasm is really a thing yet, but I think it's in active development.

All the above is stuff you should research and summarize, so we have a map we can discuss, but if I, or another compiler dev, are doing that, then we're back to the significantly increased workload.

from nimskull.

So, to me it sounds like there is:

• emscripten, the compiler
• emscripten, the os/system interface
• wasi, the os/system interface

Notably, the compiler called emscripten supports the system interfaces called emscripten and wasi?

from nimskull.