From discussion here:

https://lobste.rs/s/mm99iz/beware_shell_globs

• If it's no, then a glob that results in an arg that starts with - would be disallowed
• Although maybe it should be turned off if the glob comes after --? That might be a little too smart? Is it possible that -- is an arg and not a flag? Like
  • mycommand --arg -- *

Maybe it has to be an unquoted -- to turn it off? So you could do mycommand --arg '--' and that doesn't count?

• Should it be off by default, but on in oil:basic (and oil:all ?)

Implements the dirs builtin, following the behavior of dirs in bash. This includes using the -c, -v, and -p flags. Two flags have not yet been implemented (-l and -N/+N).

I've been hit with some other work so I'll tackle the spec tests later. Wanted to get this merged anyway in case it takes me a while to get back to working on this.

@jyn514 I'm pretty sure this is the issue you just mentioned in #257. There are two separate bugs:

• If you keep typing past the end of the terminal, OSH has no idea (but zsh and fish do). This is #257.
• If your prompt contains a unicode character that is say 4 bytes long in utf-8, the code will currently move right FOUR cursor positions instead of ONE. I think that explains the screenshot, since you have a unicode character there.

I think we can just call the POSIX function wcwidth(), although I guess that involves some conversion to wchar_t, which is annoying:

http://man7.org/linux/man-pages/man3/wcwidth.3.html

There is a pure Python version of it, but I'm not sure if we should use it.

https://pypi.org/project/wcwidth/#files

Fixes #137; addresses part of #116.

It looks like my builtins test for set -o vi/emacs somehow didn't make it in as part of #120 (??), so that is included as well.

Hm this uses several (ugly) bash features not implemented in Oil:

We should definitely be able to parse it -- running it will take some work.

• [x] trap builtin
• [ ] compatibility issue: case "$(type -t "$hookName")" in fails in OSH because we respect errexit in command sub, but bash doesn't
• [ ] compatibility issue: for i in "${nativePkgs[@]}"; do -- OSH doesn't allow indexing empty var/string with [@]
• [x] type builtin (should be easy, only need -t)
• [x] read flags: -r, -n etc. (but lack of -r not implemented)
• [x] oldOps="$(shopt -po nounset)"; ...; eval "$oldOpts" (use case for "with" in Oil)
• [x] setup.sh has same issue as Gentoo in issue #19, trying to run [ with empty $PATH
  • [x] test -x
• [x] content="${content//"$pattern"/$replacement}"
• [x] ${FUNCNAME[@]}
• [x] ${!varRef} -- completion scripts also use this
• [x] shopt -s nullglob
• [x] local -a times=(...) (unparsed)
• [x] declare -a (unparsed)
• [x] append e.g. eval "$var"'+=("$pkg")' (unparsed)
• [x] might be tricky: if [[ -z "$makeFlags" && ! ( -n "$makefile" || -e Makefile || -e makefile || -e GNUmakefile[[ ) ]]

Deferred:

• [ ] named file descriptors: exec {fd}< "$fn" (unparsed)
• [ ] closing descriptor: exec {fd}<&- (unparsed)
• [x] printf builtin -- %q is used, but external command should be OK

https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh#L318

Forking from #290.

$ var=+-
$ echo "${var//['+-']}"
Line 16 of '/home/user/.config/oil/oshrc'                                                            
  echo "${var//['+-']}"
             ^
fatal: Error matching regex "(['+-'])": Invalid regex syntax (func_regex_first_group_match)

This is indeed a difference between OSH and bash/mksh. It has to do with whether the single quote and hyphen is a SHELL metcharacter or a REGEX metacharacter.

• Is it a character class containing two characters + and - ?
• Or does the char class contain a range of characters + through ' ? (which is a regex syntax error)

@Crestwave can you tell me why you have the single quotes? What are they supposed to do? How is it different than

  echo "${var//[+-]}"

This removes all + and - characters from the string in bash. I can't see what the extra single quotes do?

I'm not saying this is not a bug -- it's definitely a difference between bash and OSH. But I'm trying to understand the intention behind the code. Thanks.

Hi,

I am interested in run with ShellSpec.

ShellSpec is a testing framework for POSIX compliant shell script. One of the notable points is that most features are implemented by pure shell scripts instead of calling external commands.

It contains tests for ShellSpec itself. Therefore you can check if it works easily.

cd /shellspec/installed/directory
shellspec -s /path/to/osh

The currently code size is

• Total (includes comment and blank): about 7000
• NLOC (Non-comment Lines of Code): 5000 over
• LLOC (Logical Lines of Code): 4000 over

This is reported by ShellMetrics - Cyclomatic Complexity Analyzer. It is also implemented by shell script.

ShellSpec needs a requires correct implementation of errexit. (ref. #709) (Note: Workarounds have been implemented for some shell flaws.) So when the bosh (Schily Bourne Shell) broke, I could immediately notice it.

Unfortunately, ShellSpec is for the POSIX shell, so it contains almost no bash syntax. However, you can check if osh is POSIX compliant.

I'm sorry, but I'm a little busy right now, so I may not be able to help enough. but I think ShellSpec can be used to test run under osh.

I ran ShellSpec on osh 0.8.pre4 and it showed as below.

  [ ${BASH_VERSION+x} ] &&:
  ^
[ eval at line 34 of /home/koichi/workspace/shellspec/lib/general.sh ]:34: fatal: Undefined variable
  [ ${ZSH_VERSION+x} ] &&:
  ^
[ eval at line 34 of /home/koichi/workspace/shellspec/lib/general.sh ]:34: fatal: Undefined variable
  [ ${YASH_VERSION+x} ] &&:
  ^
[ eval at line 34 of /home/koichi/workspace/shellspec/lib/general.sh ]:34: fatal: Undefined variable
  [ ${POSH_VERSION+x} ] &&:
  ^
[ eval at line 34 of /home/koichi/workspace/shellspec/lib/general.sh ]:34: fatal: Undefined variable
  [ ${KSH_VERSION+x} ] &&:
  ^
[ eval at line 34 of /home/koichi/workspace/shellspec/lib/general.sh ]:34: fatal: Undefined variable
      [ -u "$1" ] && chmod 0700 "$1"
      ^
/home/koichi/workspace/shellspec/lib/libexec/runner.sh:11: fatal: Id.BoolUnary_u isn't implemented
    ( umask 0077; mkdir "$1"
    ^
/home/koichi/workspace/shellspec/lib/libexec/runner.sh:8: fatal: Exiting with status 1 (subshell invoked from PID 32443)

Regards,

This should help some of the error cases.

This looks like a good intro:

https://barro.github.io/2018/01/taking-a-look-at-python-afl/

I think the steps are:

• Create a main program like bin/fuzzable_parser.py.
  • Since a parser is basically a pure function, this should use one of the "fast modes".
  • That is, we shouldn't have to fork bin/oil.py every time, which would be really slow
  • Also, by fuzzing only the parser and not the runtime (basically bin/osh -n), then we avoid the problem of the fuzzer making syscalls like fork() and so forth.
• Write some shell scripts to automate it.

It would be nice to find a server to do this, but I have at least one at home laying around that could be used.

Hm this is because the ./configure step writes out the PREFIX to _tmp/detected-config.sh. If the file isn't installed to that place, it won't run.

The core issue is that a Unix executable doesn't know where it is run from! The executable has to be set as PYTHONPATH in order to find the bytecode.

https://stackoverflow.com/questions/4031672/without-access-to-argv0-how-do-i-get-the-program-name

$>   export OVM_VERBOSE=1
$>   oil.ovm 
ovm_path = ovm_path_buf (/usr/local/bin/oil.ovm)
ovm_path: /usr/local/bin/oil.ovm
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
oil.ovm: Modules/main.c:347: Ovm_Main: Assertion `sts != -1' failed.
Aborted (core dumped)

In the PKGBUILD, I do execute ./configure and make to compile OSH. I do not, however, invoke the install script that ships with OSH. This is because the PKGBUILD convention dictates that you shouldn’t rely on an external script to actually do the placement of binaries on the system.

The PKGBUILD convention prefers that any binaries that need to be installed on the system be placed within a specific subdirectory beneath wherever the PKGBUILD file is sitting (this is referred to as $pkgdir).

From @bar-g on #678

Reason: I think osh:strict oil:basic oil:all has more consistency.

strict:all was meant to imply the prefix of strict_errexit. However there is no oil_ prefix!

So the current scheme doesn't make that much sense.

• I think osh:strict makes more sense because you're taking the OSH language and subsetting it. It still runs under other shells.
• oil:basic makes sense because you're writing in the basic parts of the Oil language, without breaking things.
• oil:all is for new scripts. You have everything

https://github.com/oilshell/oil/wiki/OSH-versus-Oil

I'm wary of breaking things -- especially since I'm using strict:all myself !!!

However I think what we can do is:

1. Make osh:strict an alias for strict:all in the next release. Should be easy.
2. Change all the docs
3. Change the wiki.
4. Maybe even correct the blog
5. And then like 6 months from now, since we're pre-1.0, just drop strict:all altogether.

It seems feasible and probably worth it for such a prominent feature

It's one of the first things you hit when you get started

Also what's the difference between declare -A dict and declare -A dict=() in bash?

There are three different states for variables in Bash. In addition, the "set" state can be categorized into two sub-states:

1. Not declared (slot is not allocated)
2. Declared but unset (slot is allocated, but no value is set)
3. Set (slot is allocated, and a value is set)
   1. Empty
   2. Non-empty

See the following test case for the behavior differences:

# test5.sh

function is-declared() { declare -p "$1" &>/dev/null; }
function is-enumerated-by-declare() { declare | grep -q "^$1="; }
function is-enumerated-by-varname-expansion() { eval "printf '%s\n' \"\${!$1@}\"" | grep -q "^$1\$"; }
function is-enumerated-by-compgen () { compgen -v | grep -q "^$1\$"; }
function is-enumerated-by-compgen-arrayvar () { compgen -A arrayvar | grep -q "^$1\$"; }
function show-status {
  local -a msg=(yes)
  local f
  for f in $(compgen -A function -- is-); do
    "$f" "$1"
    echo "$f: ${msg[$?]:-no}"
  done
}

echo "[undeclared]"
show-status dict
echo "[declared]"
declare -A dict
show-status dict
echo "[set]"
dict=()
show-status dict

$ bash test5.sh
[undeclared]
is-declared: no
is-enumerated-by-compgen: no
is-enumerated-by-compgen-arrayvar: no
is-enumerated-by-declare: no
is-enumerated-by-varname-expansion: no
[declared]
is-declared: yes
is-enumerated-by-compgen: no
is-enumerated-by-compgen-arrayvar: no
is-enumerated-by-declare: no
is-enumerated-by-varname-expansion: no
[set]
is-declared: yes
is-enumerated-by-compgen: yes
is-enumerated-by-compgen-arrayvar: yes
is-enumerated-by-declare: yes
is-enumerated-by-varname-expansion: yes

Related to this, in the help-bash mailing list, I have heard about some script library that gives a default array content when the definition is not given by the application. I'm not sure if it is a good design, but in that library, the empty array is one of the possible configurations, so one needs to distinguish the unset state (declare -A dict) from the empty state (declare -A dict=()). Also, if the array is associative, the library wants to declare the associative array attribute in advance. Such a script can be achieved in the following way:

# library
declare -A dict
is-array-set() { compgen -A arrayvar -X "!$1" -- "$1" &>/dev/null; }

function initialize-lib {
  if ! is-array-set dict; then
    # default dictionary
    dict=([left]=right [up]=down)
  fi
}


# application

# ... set up dict here ...

initialize-lib

Originally posted by @akinomyoga in https://github.com/oilshell/oil/issues/653#issuecomment-599193601

consider the following script:

for (( n=0; n<(3-(1)); n++ )) ; do echo $n; done

which, with bash, runs to completion and produces the following output:

0
1

osh fails to correctly parse (or execute this) statement:

[user@localhost:~]$ osh --version
Oil version 0.12.9
Release Date: 2022-11-10 16:36:00+00:00
Arch: x86_64
OS: Linux
Platform: #1-NixOS SMP Wed Dec 21 16:36:38 UTC 2022
Compiler: GCC 11.3.0
Interpreter: OVM
Interpreter version: 2.7.13
Bytecode: bytecode-opy.zip
[user@localhost:~]$ osh bug.sh
  for (( n=0; n<(3-(1)); n++ )) ; do echo $n; done
                   ^
'bug.sh':1: Unexpected left paren (might need a space before it)

correcting that issue, osh -n now succeeds; but running the script, we get a new error:

[user@localhost:~]$ osh bug.sh
  for (( n=0; n<(3- (1)); n++ )) ; do echo $n; done
                    ^
'bug.sh':1: fatal: Unexpected typed args passed to external command '3-'
  for (( n=0; n<(3- (1)); n++ )) ; do echo $n; done
              ^
'bug.sh':1: fatal: Invalid integer constant 'n/dev/fd/3'

it looks like osh is still failing to parse the arithmetic context and doing a file redirection. if we add different whitespace, we can get it working:

--- a/bug.sh
+++ b/bug.sh
@@ -1 +1 @@
-for (( n=0; n<(3-(1)); n++ )) ; do echo $n; done
+for (( n=0; n< (3-(1)); n++ )) ; do echo $n; done

note, this appears present at least as far back as 0.8.12, but with a different error code:

[user@localhost:~]$ osh --version
Oil version 0.8.12
Release Date: 2021-07-06 06:38:25+00:00
Arch: x86_64
OS: Linux
Platform: #1-NixOS SMP Wed Dec 21 16:36:38 UTC 2022
Compiler: GCC 11.3.0
Interpreter: OVM
Interpreter version: 2.7.13
Bytecode: bytecode-opy.zip
[user@localhost:~]$ osh bug.sh
  for (( n=0; n<(3-(1)); n++ )) ; do echo $n; done
                    ^
bug.sh:1: Expected ) in function definition

also, this trivial example is a distillation of AAXtoMP3, and was discovered during nixos/nixpkgs#209620:

[user@localhost:~]$ curl -OL https://raw.githubusercontent.com/KrumpetPirate/AAXtoMP3/v1.3/AAXtoMP3
[user@localhost:~]$ nix shell nixpkgs#osh -- -n AAXtoMP3
    for (( n=0; n<(20-(percentage/5)); n++ )) ; do progressbar="$progressbar "; done
                      ^
AAXtoMP3:204: Unexpected left paren (might need a space before it)

as expected, we can apply the fix that we discovered above, but this still seems like a bash compatibility bug:

diff --git a/AAXtoMP3 b/AAXtoMP3
index 90566ad..71e94da 100755
--- a/AAXtoMP3
+++ b/AAXtoMP3
@@ -200,8 +200,8 @@ progressbar() {

   #draw progressbar with one # for every 5% and blank spaces for the missing part.
   progressbar=""
-  for (( n=0; n<(percentage/5); n++ )) ; do progressbar="$progressbar#"; done
-  for (( n=0; n<(20-(percentage/5)); n++ )) ; do progressbar="$progressbar "; done
+  for (( n=0; n< (percentage/5); n++ )) ; do progressbar="$progressbar#"; done
+  for (( n=0; n< (20-(percentage/5)); n++ )) ; do progressbar="$progressbar "; done

   #print progressbar
   echo -ne "Chapter splitting: |$progressbar| $print_percentage% ($part/$total chapters)\r"

This gives the "proper" big-O improvement to O(num live objects), not just O(num allocated objects)

https://oilshell.zulipchat.com/#narrow/stream/121539-oil-dev/topic/TODO.20on.20core.20GC/near/319627427

And note lazy free() was actually pretty significant on the benchmarks

Say I'm writing some scripts to bootstrap my OS. I may need to know what this shell script depends on(what environment variables are used, what binaries are expected on path, etc. I think this should be possible to implement as a plugin, but at least we need to have API to make it possible.

• Should only be in Oil mode ? Because therwise they will cause incompatibility
• Probably want to write these as shell, not in Python/C++
  • #750

See thread on #oil-help about "$NEWLINE$message$NEWLINE"

https://oilshell.zulipchat.com/#narrow/stream/264891-oil-help/topic/Surround.20string.20with.20.5Cn

https://oilshell.zulipchat.com/#narrow/stream/325160-oil-discuss-public/topic/Oil.20Changes.20to.20support.20String.2FData.20language.20Unification

• add \{var} interpolation to $''
• possibly add $"" as a JSON parser, for generating shell code from other languages?
  • then [$"foo", $"bar\n"] becomes a valid ARGV array I guess
  • differences from "": the characters $ and backtick are no longer special
  • possibly add '{var}` as well, for consistency
• ~~double quoted strings can support C-escapes like \x00 and \u{1234}, as long as strict_backslash~~
• ~~add $"" as a synonym to "", which also makes it consistent with $''~~ - NO, maintain this assymetry on purpose