We can't know how many bytes fgets() read when we reach EOF and fgets()
didn't see a newline; we can only assume that at least strlen(buf) bytes
were read. This is quite obnoxious if one wants to use NULs in raw
input, but at least we can make reading "a\0b\0c\0" with no newline
yield "a\0b\0c", losing only the final sequence of NULs.
We can't use getline() either, since it will want to allocate a buffer
big enough for an entire line, and we might not have any newlines in our
input. A complete fix will have to use getc() or read(), preferably the
latter.
When trying to use the new regex functions (match/test/sub/capture/etc)
in a JQ that was compiled without the ONIGURAMA regex library,
report an informative error message instead of a 'not defined' error.
Before:
$ echo '"foo"' | ./jq-old 'test("foo")'
jq: error: test/1 is not defined at <top-level>, line 1:
test("foo")
jq: 1 compile error
After:
$ echo '"foo"' | ./jq 'test("foo")'
jq: error: jq was compiled without ONIGURAMA regex libary. match/test/sub and related functions are not available.
When running `make` I ran into a couple of problems building the manual. While
I'm not entirely sure that this is the root cause, it appears to have been
related to the fact that ruby 2.0 dropped syck completely in favor of psych
(which was introduced in 1.9.2) for YAML processing. I'm currently using ruby
2.1.0p0.
I'm assuming that the fact that the YAML engine was explicitly set to syck in
the Rakefile was an attempt to work around some incompatibility between the two
libraries, so I looked into what would be necessary to get it to work with the
newer one. The changes to `manual.yml` ended up being pretty minor: I ran it
through `iconv` to convert some ISO-8859-1 characters to UTF-8 and added some
quotes in places (apparently you can't start a string value with '`').
When escaping string (e.g. for `@tsv` or `@csv` outputs),
escape NULs as '\0'.
Existing behaviour, unchanged by this patch:
$ echo '"a\u0000b"' | ./jq '.'
"a\u0000b"
$ echo '"a\u0000b"' | ./jq -r '.' | od -a
0000000 a nul b nl
0000004
When using `@tsv`, escape NUL to `\0`:
$ echo '"a\u0000b"' | ./jq -r '[.]|@tsv'
a\0b
$ echo '"a\u0000b"' | ./jq '[.]|@tsv'
"a\\0b"
When using '@tsv' output format with '-r' option,
escape \r, \n and \\ in addition to \t.
Example:
$ printf '{"id":"hello\\ttab\\nworld","x":43 }' | jq .
{
"id": "hello\ttab\nworld",
"x": 43
}
Before: newlines are not escaped, generating invalid TSV output:
$ printf '{"id":"hello\\ttab\\nworld","x":43 }' \
| ./jq-old -r '[.id,.x] | @tsv'
hello\ttab
world 43
After: newlines are properly escaped:
$ printf '{"id":"hello\\ttab\\nworld","x":43 }' \
| ./jq-new -r '[.id, .x] | @tsv'
hello\ttab\nworld 43
Before: backslashes themselves are not escaped, so there's no way to
distinguish between escaped characters and 'real' backslashes
(in the example below, there is should not be newline, despite the
output containing "\n".
$ printf '{"x":"hello\\ttab\\\\new world"}' \
| ./jq-old -r '[.x]|@tsv'
hello\ttab\new world
After: backslashes are escaped:
$ printf '{"x":"hello\\ttab\\\\new world"}' \
| ./jq-new -r '[.x]|@tsv'
hello\ttab\\new world
Detect output errors when the program exits.
Currently:
$ echo '{}' | jq . > /dev/full && echo ok
ok
with the patch:
$ echo '{}' | jq . > /dev/full && echo ok
Error: writing output failed: No space left on device
also apply to hardware/network/other I/O errors.
Signed-off-by: Nicolas Williams <nico@cryptonector.com>
Improve robustness in automated system when using exit code in shell scripts,
by exiting with code 2 if there was any input error (even overriding other
possible error exit codes).
Exit code 2 is already used to indicate system errors.
Without the patch:
$ jq . no-such-file ; echo $?
jq: no-such-file: No such file or directory
0
With the patch:
$ jq . no-such-file ; echo $?
jq: no-such-file: No such file or directory
2
Signed-off-by: Nicolas Williams <nico@cryptonector.com>
With this change, runtime exceptions are propagated to non-zero exit
code of 'jq', allow better scripting and automation. The new exit code
value is 5.
This allows using the shell's and/or operations ('&&' and '||') to
detect input runtime exceptions.
Before:
runtime exceptions are printed to STDERR, but not reported as non-zero exit-code:
$ echo '"hello"' | jq '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
0
After:
$ echo '"hello"' | ./jq '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
5
Note that there's a subtle interplay when using "-e" option.
The value of the non-zero exit code changes from 4 to 5, but it still
indicates a 'failure' or non-true value.
Before:
$ echo '"hello"' | jq -e '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
4
After:
$ echo '"hello"' | ./jq -e '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
5
Also flex is now optional.
The outputs of flex and bison are now committed. By default they get
built, but users who want to build from git can now
./configure --disable-maintainer-mode
to turn off the dependency on bison and flex.
Maintainers must, of course, commit the bison and/or flex outputs when
they make changes to parser.y and/or lexer.l, respectively.
Tests won't pass if built without Oniguruma. We don't have a way to
make a test optional yet. That will come later. For now the ability to
reduce build-time dependencies could really help some users.
