mgree / smoosh Goto Github PK

Consider the following snippet:

FOO="\\a"
echo ${FOO#*\\}

What I expect this to output is a (and this is what dash and bash does), but smoosh outputs \a.

Another issue:

FOO="a?b"
echo ${FOO#*"?"}

This should output b (as it does on dash and bash), but with smoosh is outputs ?b, because it does not consider the question mark as escaped.

It seems that when try_match_substring receives the pattern, it is already fully expanded, with quotes removed, despite the code assuming that it still has quotes and escaping backslashes in it.

A similar pattern to the first example is used in the Modernish test suite to iterate over a string, this bug causes it to be stuck in an endless loop forever.

Shtepper is too verbose.

Hello,

First off, thank you for your impressive project.
I was looking for various POSIX shell implementations and found your project. The goal of this project seems to be to formalize the POSIX shell standard, but I have determined that you are also developing an executable POSIX shell.

I was assuming that smoosh would actually work, but the following code terminated unexpectedly.

set -e
putsn() { echo "$@"; }
false && true
putsn "It should be executed" # Not executed by smoosh.

I'm developing ShellSpec, a unit testing framework that works with POSIX shells, and one of its goals is to support all POSIX shells. The goal is almost achieved, and it already works with almost all POSIX shells (even if there are some bugs). As part of this development, I am reporting a bug of the shell.

Due to the above bug, smoosh is not yet working with ShellSpec. If your project its (one of) goals is to develop an executable POSIX shell, I will gladly report back.

Regards,

version.lem just adds a timestamp but it doesn't tell you which branch/commit was built.

Should those 1s be 0s? We should test carefully (and add tests). Is this somehow not in the POSIX test suite?

There is duplication in shim.ml in smoosh and ast.ml in libdash. With recent work on PaSh helping to improve the libdash parsers, we should move Smoosh over to that infrastructure. There'll be some cost in copying (parse dash ast -> libdash ast -> smoosh ast), but it'll simplify things.

There's no way for us to represent a single word, leading to some minor inadequacies. There are a few places where there really only ever ought to be a single word, statically:

• assignments
• redirect targets

Changes here will be tricky, because they'll interfere with expansion. It may make sense to make these changes after accommodating multiple parsers #6.

With #36 (a fix for #21, among other things), escaping is explicit in words. It's still the case that expanded_words and intermediate_fields use backslashes to escape things. It would be good to have explicit escapes throughout, though getting all the details right will be tricky.

Have the shell test runner indicate which tests failed at the end (irrespective of TEST_DEBUG).

Assertion #379 (A): Test of GA11 - the utility recreates files with correct attributes.
Contents of _ga11_errout:
Testing GA11 for directory
Testing GA11 for regular file
getconf: Unrecognized variable `_POSIX_TIMESTAMP_RESOLUTION'
CheckFileTimes: could not obtain timestamp resolution -
continuing with assumed resolution of 1 second
Testing GA11 for FIFO
getconf: Unrecognized variable `_POSIX_TIMESTAMP_RESOLUTION'
CheckFileTimes: could not obtain timestamp resolution -
continuing with assumed resolution of 1 second
Failure: File ctime 1571676327.687798556 outside of acceptable range 1571676330.000000000 to 1571676330.843451261
Testing GA11 for a symbolic link to an existing file
getconf: Unrecognized variable `_POSIX_TIMESTAMP_RESOLUTION'
CheckFileTimes: could not obtain timestamp resolution -
continuing with assumed resolution of 1 second
Testing GA11 for a symbolic link to a non-existing file
getconf: Unrecognized variable `_POSIX_TIMESTAMP_RESOLUTION'
CheckFileTimes: could not obtain timestamp resolution -
continuing with assumed resolution of 1 second

Either my new laptop is too slow, Linux/Docker is misconfigured, or there's an actual bug.

Implement:

• Track history (can we integrate this with logging?)
• Sh_nolog
• HISTFILE
• builtin_fc

Once a utility has been searched for and found (either as a result of this specific search or as part of an unspecified shell start-up activity), an implementation may remember its location and need not search for the utility again unless the PATH variable has been the subject of an assignment. If the remembered location fails for a subsequent invocation, the shell shall repeat the search to find the new location for the utility, if any.

• Add hash to shell state (record hits, like bash!)
• Have checked_set_param clear hash table on $PATH update
• Update resolve_command_name to do memoization
• builtin_hash with -r flag (report hits)
• Implement -h/-o earlyhash (called hash-on-def in yash and hashall in bash)

Bash reports whether or not a command is hashed. Non-POSIX but semi-useful. Would need to update a few things to be able to report that.

• Measure difference in timing on POSIX suite.

I'm unable to even build the POSIX test suite when running in a Vagrant VM; the TET3.8 test-suite manager complains about getting spurious TSTP signals. No such issue on Docker; issue is present in v0.1.

0|3.8-lite 22:20:49 20200210|User: vagrant (1000) TCC Start, Command line: tcc -bp vsc posix_shell
5|Linux debian9.localdomain 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64|System Information
20|/home/vagrant/posix/tet3.8/vsc/tetbuild.cfg 0|Config Start
30||TET_OUTPUT_CAPTURE=False
30||TET_EXPAND_CONF_VARS=true
30||TET_BUILD_TOOL=sh
30||TET_BUILD_FILE=/home/vagrant/posix/tet3.8/vsc/Bin/buildtool
30||TET_API_COMPLIANT=True
30||TET_PASS_TC_NAME=False
30||TET_VERSION=3.8-lite
40||Config End
110|0 /tset/POSIX.shell/shell/sh_04.ex 22:20:49|Build Start, scenario ref 1-0
15|0 3.8 1|TCM Start
510|0|unexpected signal 20 received by TCM
510|0 0 1 22:20:49|IC Start
510|0|unexpected signal 20 received by TCM
510|0 1 22:20:49|TP Start
510|0|unexpected signal 20 received by TCM
510|0|unexpected signal 20 received by TCM
510|0 1 7 22:20:50|NORESULT
510|0|unexpected signal 20 received by TCM
510|0 0 1 22:20:50|IC End
510|0|Abandoning test case: caught unexpected signal 20
130|0 0 22:20:50|Build End, scenario ref 1-0
110|1 /tset/POSIX.shell/shell/sh_05.ex 22:20:50|Build Start, scenario ref 2-0
15|1 3.8 1|TCM Start
510|1|unexpected signal 20 received by TCM
510|1 0 1 22:20:50|IC Start
510|1|unexpected signal 20 received by TCM
510|1 1 22:20:50|TP Start
510|1|unexpected signal 20 received by TCM
510|1|unexpected signal 20 received by TCM
510|1 1 7 22:20:51|NORESULT
510|1|unexpected signal 20 received by TCM
510|1 0 1 22:20:51|IC End
510|1|Abandoning test case: caught unexpected signal 20
130|1 0 22:20:51|Build End, scenario ref 2-0
110|2 /tset/POSIX.shell/shell/sh_06.ex 22:20:51|Build Start, scenario ref 3-0
15|2 3.8 1|TCM Start
510|2|unexpected signal 20 received by TCM
510|2 0 1 22:20:51|IC Start
510|2|unexpected signal 20 received by TCM
510|2 1 22:20:51|TP Start
510|2|unexpected signal 20 received by TCM
510|2|unexpected signal 20 received by TCM
510|2 1 7 22:20:52|NORESULT
510|2|unexpected signal 20 received by TCM
510|2 0 1 22:20:52|IC End
510|2|Abandoning test case: caught unexpected signal 20
130|2 0 22:20:52|Build End, scenario ref 3-0
110|3 /tset/POSIX.shell/shell/sh_07.ex 22:20:52|Build Start, scenario ref 4-0
15|3 3.8 1|TCM Start
510|3|unexpected signal 20 received by TCM
510|3 0 1 22:20:52|IC Start
510|3|unexpected signal 20 received by TCM
510|3 1 22:20:52|TP Start
510|3|unexpected signal 20 received by TCM
510|3|unexpected signal 20 received by TCM
510|3 1 7 22:20:53|NORESULT
510|3|unexpected signal 20 received by TCM
510|3 0 1 22:20:53|IC End
510|3|Abandoning test case: caught unexpected signal 20
130|3 0 22:20:53|Build End, scenario ref 4-0
110|4 /tset/POSIX.shell/shell/sh_08.ex 22:20:53|Build Start, scenario ref 5-0
15|4 3.8 1|TCM Start
510|4|unexpected signal 20 received by TCM
510|4 0 1 22:20:53|IC Start
510|4|unexpected signal 20 received by TCM
510|4 1 22:20:54|TP Start
510|4|unexpected signal 20 received by TCM
510|4|unexpected signal 20 received by TCM
510|4 1 7 22:20:54|NORESULT
510|4|unexpected signal 20 received by TCM
510|4 0 1 22:20:54|IC End
510|4|Abandoning test case: caught unexpected signal 20
130|4 0 22:20:54|Build End, scenario ref 5-0
110|5 /tset/POSIX.shell/shell/sh_09.ex 22:20:54|Build Start, scenario ref 6-0
15|5 3.8 1|TCM Start
510|5|unexpected signal 20 received by TCM
510|5 0 1 22:20:54|IC Start
510|5|unexpected signal 20 received by TCM
510|5 1 22:20:55|TP Start
510|5|unexpected signal 20 received by TCM
510|5|unexpected signal 20 received by TCM
510|5 1 7 22:20:55|NORESULT
510|5|unexpected signal 20 received by TCM
510|5 0 1 22:20:55|IC End
510|5|Abandoning test case: caught unexpected signal 20
130|5 0 22:20:55|Build End, scenario ref 6-0
110|6 /tset/POSIX.shell/shell/sh_10.ex 22:20:55|Build Start, scenario ref 7-0
15|6 3.8 1|TCM Start
510|6|unexpected signal 20 received by TCM
510|6 0 1 22:20:55|IC Start
510|6|unexpected signal 20 received by TCM
510|6 1 22:20:56|TP Start
510|6|unexpected signal 20 received by TCM
510|6|unexpected signal 20 received by TCM
510|6 1 7 22:20:56|NORESULT
510|6|unexpected signal 20 received by TCM
510|6 0 1 22:20:56|IC End
510|6|Abandoning test case: caught unexpected signal 20
130|6 0 22:20:56|Build End, scenario ref 7-0
110|7 /tset/POSIX.shell/shell/sh_11.ex 22:20:56|Build Start, scenario ref 8-0
15|7 3.8 1|TCM Start
510|7|unexpected signal 20 received by TCM
510|7 0 1 22:20:56|IC Start
510|7|unexpected signal 20 received by TCM
510|7 1 22:20:57|TP Start
510|7|unexpected signal 20 received by TCM
510|7|unexpected signal 20 received by TCM
510|7 1 7 22:20:57|NORESULT
510|7|unexpected signal 20 received by TCM
510|7 0 1 22:20:57|IC End
510|7|Abandoning test case: caught unexpected signal 20
130|7 0 22:20:57|Build End, scenario ref 8-0
110|8 /tset/POSIX.shell/shell/sh_12.ex 22:20:57|Build Start, scenario ref 9-0
15|8 3.8 1|TCM Start
510|8|unexpected signal 20 received by TCM
510|8 0 1 22:20:57|IC Start
510|8|unexpected signal 20 received by TCM
510|8 1 22:20:58|TP Start
510|8|unexpected signal 20 received by TCM
510|8|unexpected signal 20 received by TCM
510|8 1 7 22:20:58|NORESULT
510|8|unexpected signal 20 received by TCM
510|8 0 1 22:20:58|IC End
510|8|Abandoning test case: caught unexpected signal 20
130|8 0 22:20:58|Build End, scenario ref 9-0
110|9 /tset/POSIX.shell/shell/sh_13.ex 22:20:58|Build Start, scenario ref 10-0
15|9 3.8 1|TCM Start
510|9|unexpected signal 20 received by TCM
510|9 0 1 22:20:58|IC Start
510|9|unexpected signal 20 received by TCM
510|9 1 22:20:59|TP Start
510|9|unexpected signal 20 received by TCM
510|9|unexpected signal 20 received by TCM
510|9 1 7 22:20:59|NORESULT
510|9|unexpected signal 20 received by TCM
510|9 0 1 22:20:59|IC End
510|9|Abandoning test case: caught unexpected signal 20
130|9 0 22:20:59|Build End, scenario ref 10-0
900|22:20:59|TCC End

If the TSTP signals are my fault, I suspect it's an issue with the monitoring/interactive work in system.ml. This seems unlikely because the build was working on Linux in October.

We currently track unspec/undef behavior on a per-shell basis.

• Find all mentions of unspec/undef in the POSIX spec: unspec.md and undef.md
• Log them in smoosh
• A way to log unspec/undef to an external location (to track in subshells); timestamped occurrences w/maximum context (PID, shell state, etc.)

Once this is done, try logging all unspec/undef in the various test suites.

Core problem: the symbolic process model is inadequate.

The variable $$ isn't currently installed in a symbolic shell, but PPID is. Trickiness: $$ is unchanged in subshells, which can signal the top-level. We need to carefully hold on to such signals.

It's not currently possible for symbolic shells to send signals to themselves---including SIGPIPE, which leads to a wrong execution of the following two programs:

while true; do echo 5; done | true
while true; do echo 5; done | { read x; echo $((x + 42)) ; }

The first should terminate immediately; the second should print 47 and terminate. Right now neither loop ever stops, because the first process in either pipeline never receives the SIGPIPE.

Plan (checked means done in local branch sigpipe):

• Check readers in symbolic_write_fd
• Add process entry for top-level shell
• Each shell process has an status : maybe nat (of a possible exit status) and pending : list signal and stepped : bool
• Stepping sets stepped
• Symbolic state has curpid representing current thread of execution
• Symbolic system calls require that curpid is a live thread, otherwise immediate abort of call
• Sending a signal mutates shell process state. Set status on, e.g., uncaught SIGPIPE
• Implement os_pending_signal for symbolic mode
• Tests for pending signals in symbolic mode
• Tracer looks up pid 0 each time and steps it (with a cascade of curpid updates)
• Tracer finds pids that weren't stepped and steps them
• Update visualization
  • Change shim.ml to send over the full OS state, including the proc list
  • Render all of the live procs side by side, collapsing all but the main shell and the active proc

Running ./install -s smoosh in Docker yields a parse abort. It's probably related to the other parse aborts we've seen, which I suspect is a mishandling of erroneous parsing in eval.

There's a bug on true Linux platforms that never showed up in OS X/Docker on OS X (though who could tell because of #5).

$ yes n | ./install.sh -s smoosh
Relaunching install.sh with /home/mgree/.local/bin/smoosh...
* Modernish version 0.15.2-dev, now running on /home/mgree/.local/bin/smoosh.
* This shell identifies itself as smoosh version 0.1.
  Modernish detected the following bugs, quirks and/or extra features on it:
34:ln=01: command not found
36:mh=00:pi=40: command not found
33:so=01: command not found
35:do=01: command not found
35:bd=40: command not found
33: command not found
01:cd=40: command not found
33: command not found
01:or=40: command not found
31: command not found
01:mi=00:su=37: command not found
41:sg=30: command not found
43:ca=30: command not found
41:tw=30: command not found
42:ow=34: command not found
42:st=37: command not found
44:ex=01: command not found
32:*.tar=01: command not found
31:*.tgz=01: command not found
31:*.arc=01: command not found
31:*.arj=01: command not found
31:*.taz=01: command not found
31:*.lha=01: command not found
31:*.lz4=01: command not found
31:*.lzh=01: command not found
31:*.lzma=01: command not found
31:*.tlz=01: command not found
31:*.txz=01: command not found
31:*.tzo=01: command not found
31:*.t7z=01: command not found
31:*.zip=01: command not found
31:*.z=01: command not found
31:*.Z=01: command not found
31:*.dz=01: command not found
31:*.gz=01: command not found
31:*.lrz=01: command not found
31:*.lz=01: command not found
31:*.lzo=01: command not found
31:*.xz=01: command not found
31:*.zst=01: command not found
31:*.tzst=01: command not found
31:*.bz2=01: command not found
31:*.bz=01: command not found
31:*.tbz=01: command not found
31:*.tbz2=01: command not found
31:*.tz=01: command not found
31:*.deb=01: command not found
31:*.rpm=01: command not found
31:*.jar=01: command not found
31:*.war=01: command not found
31:*.ear=01: command not found
31:*.sar=01: command not found
31:*.rar=01: command not found
31:*.alz=01: command not found
31:*.ace=01: command not found
31:*.zoo=01: command not found
31:*.cpio=01: command not found
31:*.7z=01: command not found
31:*.rz=01: command not found
31:*.cab=01: command not found
31:*.wim=01: command not found
31:*.swm=01: command not found
31:*.dwm=01: command not found
31:*.esd=01: command not found
31:*.jpg=01: command not found
35:*.jpeg=01: command not found
35:*.mjpg=01: command not found
35:*.mjpeg=01: command not found
35:*.gif=01: command not found
35:*.bmp=01: command not found
35:*.pbm=01: command not found
35:*.pgm=01: command not found
35:*.ppm=01: command not found
35:*.tga=01: command not found
35:*.xbm=01: command not found
35:*.xpm=01: command not found
35:*.tif=01: command not found
35:*.tiff=01: command not found
35:*.png=01: command not found
35:*.svg=01: command not found
35:*.svgz=01: command not found
35:*.mng=01: command not found
35:*.pcx=01: command not found
35:*.mov=01: command not found
35:*.mpg=01: command not found
35:*.mpeg=01: command not found
35:*.m2v=01: command not found
35:*.mkv=01: command not found
35:*.webm=01: command not found
35:*.ogm=01: command not found
35:*.mp4=01: command not found
35:*.m4v=01: command not found
35:*.mp4v=01: command not found
35:*.vob=01: command not found
35:*.qt=01: command not found
35:*.nuv=01: command not found
35:*.wmv=01: command not found
35:*.asf=01: command not found
35:*.rm=01: command not found
35:*.rmvb=01: command not found
35:*.flc=01: command not found
35:*.avi=01: command not found
35:*.fli=01: command not found
35:*.flv=01: command not found
35:*.gl=01: command not found
35:*.dl=01: command not found
35:*.xcf=01: command not found
35:*.xwd=01: command not found
35:*.yuv=01: command not found
35:*.cgm=01: command not found
35:*.emf=01: command not found
35:*.ogv=01: command not found
35:*.ogx=01: command not found
35:*.aac=00: command not found
36:*.au=00: command not found
36:*.flac=00: command not found
36:*.m4a=00: command not found
36:*.mid=00: command not found
36:*.midi=00: command not found
36:*.mka=00: command not found
36:*.mp3=00: command not found
36:*.mpc=00: command not found
36:*.ogg=00: command not found
36:*.ra=00: command not found
36:*.wav=00: command not found
36:*.oga=00: command not found
36:*.opus=00: command not found
36:*.spx=00: command not found
36:*.xspf=00: command not found
36:: command not found
   LOCALVARS TRAPPRSUBSH BUG_MULTIBYTE BUG_HDOCMASK
* Running modernish test suite on /home/mgree/.local/bin/smoosh ...
* lib/modernish/tst/@sanitychecks.t 
  002: ASCII chars and control char constants   - FAIL
* WARNING: modernish has some bug(s) in combination with this shell.
           Run 'modernish --test' after installation for more details.
Are you happy with /home/mgree/.local/bin/smoosh as the default shell? (y/n) install.sh: Aborting.

• Show positional parameter info whenever a call or return happens
• Show call stack somewhere

Real talk: we should get rid of the JS stateful morass and reimplement this all in Elm. (That's the part that's a "good student project".)

++ and -- aren't required, but we should have them.

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_04

TEST_UTIL and default.env are a weirdness. Can we simplify things?

Incoprorate some of Thelford's ideas from smoosh-fuzz?

Things are a disaster. I have to unset SDKROOT when on macOS to get a build to work. (But I need it set to get Haskell to work.)

Ideally we should have:

• opam package for libdash
• change .travis.yml to actually build things rather than use Docker
• build on multiple OSes
• make -j4 or something, i mean come on breaks lem and maybe us too?
• add back Modernish tests
• change to just depend on an opam package for lem
• build uses lem OPAM package, submodule removed
• opam package for smoosh
• build just uses these packages (with local pinning?)
• have a Travis build that actually tests Docker (and Vagrant, too?) best approach: use OPAM!
• log test output and post somewhere meaningful (run other shells, too? or is all this a separate repo?)

See http://austingroupbugs.net/view.php?id=789#c4102.

Add tests from #18?

I am working through build this on macOS. I have hit a bunch of error which are for the most part resolvable.

I am currently stuck at this error:

~/Github/smoosh/libdash 0.2-rc~16*
❯ sudo gmake install
Password:
Making install in src
gmake[1]: Entering directory '/Users/mandalarian/Github/smoosh/libdash/src'
gmake  install-am
gmake[2]: Entering directory '/Users/mandalarian/Github/smoosh/libdash/src'
gmake[3]: Entering directory '/Users/mandalarian/Github/smoosh/libdash/src'
 .././install-sh -c -d '/usr/lib/x86_64-linux-gnu'
mkdir: /usr/lib/x86_64-linux-gnu: Operation not permitted
gmake[3]: *** [Makefile:668: install-libLTLIBRARIES] Error 1
gmake[3]: Leaving directory '/Users/mandalarian/Github/smoosh/libdash/src'
gmake[2]: *** [Makefile:1881: install-am] Error 2
gmake[2]: Leaving directory '/Users/mandalarian/Github/smoosh/libdash/src'
gmake[1]: *** [Makefile:1875: install] Error 2
gmake[1]: Leaving directory '/Users/mandalarian/Github/smoosh/libdash/src'
gmake: *** [Makefile:397: install-recursive] Error 1

I am guessing I need to change .././install-sh -c -d '/usr/lib/x86_64-linux-gnu' to something else but what I dont know. Any guidance provided would be greatly appreciated. Thank you.

Generally, speaking, the install tools required in the Dockerfile are not the same tools to be installed on mac, apparently. For example, libtools-dev and autotools-dev dont seem to be required. I think those are Fedora specific.

Alternatively, how would i connect a docker build smoosh to my local machine?

Thanks for all the great work done here. Looking forward to typed shell commands!

There are some bugs in interactive job control.

• ab3aa4d#diff-597a09bc85e5022f70eb206c759b4b85 was accidentlly using job control in pipes, breaking, e.g., hostname | wc -l
• need to be careful update current job statuses on ECHLD
• Sh_notify should trigger ASAP on SIGCHLD

• fix PS1 and PS2 expansion
• "The shell shall replace each instance of the character '!' in PS1 with the history file number of the next command to be typed. Escaping the '!' with another '!' (that is, "!!" ) shall place the literal character '!' in the prompt. This volume of POSIX.1-2017 specifies the effects of the variable only for systems supporting the User Portability Utilities option."

functions needed:

• libdash instance

• support for EvalLoop

  • context (e.g., stackmark/parser state)
  • parse_next function

• set_ps1

• set_ps2

• morbig instance colis-anr/morbig#102

• Oil instance?

Our fork is currently several hundred commits behind. What new bugs await?!

Right now each builtin does its own ad hoc parsing in command.lem. We should separate the parsing descriptions from the builtin implementations. Not only will be able to give better help text, we'll be able to do more static reasoning, e.g., identifying unspec argument use early.

It would be interesting to follow the fish approach of parsing the man page directly.