Detect output errors when the program exits.
Currently:
$ echo '{}' | jq . > /dev/full && echo ok
ok
with the patch:
$ echo '{}' | jq . > /dev/full && echo ok
Error: writing output failed: No space left on device
also apply to hardware/network/other I/O errors.
Signed-off-by: Nicolas Williams <nico@cryptonector.com>
Improve robustness in automated system when using exit code in shell scripts,
by exiting with code 2 if there was any input error (even overriding other
possible error exit codes).
Exit code 2 is already used to indicate system errors.
Without the patch:
$ jq . no-such-file ; echo $?
jq: no-such-file: No such file or directory
0
With the patch:
$ jq . no-such-file ; echo $?
jq: no-such-file: No such file or directory
2
Signed-off-by: Nicolas Williams <nico@cryptonector.com>
With this change, runtime exceptions are propagated to non-zero exit
code of 'jq', allow better scripting and automation. The new exit code
value is 5.
This allows using the shell's and/or operations ('&&' and '||') to
detect input runtime exceptions.
Before:
runtime exceptions are printed to STDERR, but not reported as non-zero exit-code:
$ echo '"hello"' | jq '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
0
After:
$ echo '"hello"' | ./jq '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
5
Note that there's a subtle interplay when using "-e" option.
The value of the non-zero exit code changes from 4 to 5, but it still
indicates a 'failure' or non-true value.
Before:
$ echo '"hello"' | jq -e '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
4
After:
$ echo '"hello"' | ./jq -e '.|tonumber' ; echo $?
jq: error: Invalid numeric literal at EOF at line 1, column 5 (while parsing 'hello')
5
Much of this could be in libjq. Eventually all of the work of reading
from files and looping over `jq_next()` should move into libjq, with
`main()` mostly doing all the command-line option processing.
To import a module now use:
# Import module.jq file:
import "relative/path/to/module" as foo;
# Use the module's defs as foo::<def-name>
To import a JSON file:
# Read file.json:
import "relative/path/to/file" as $foo;
#
# Use as $foo::foo
Using `-L` now drops the builtin library path and appends the requested
path to the empty array (or the result of an earlier `-L`).
Support for the `$JQ_LIBRARY_PATH` environment variable has been
removed.
This is to fix the problem where `break` is dynamic, not lexical.
With this it should be possible to do this sort of thing:
label $break | inputs | if ... then $break|error else . end
This is a backwards-incompatible change for master, but the previous
`break` hadn't shipped yet.
Still needed:
- testing
Streaming means that outputs are produced as soon as possible. With the
`foreach` syntax one can write programs which reduce portions of the
streaming parse of a large input (reduce into proper JSON values, for
example), and discard the rest, processing incrementally.
This:
$ jq -c --stream .
should produce the same output as this:
$ jq -c '. as $dot | path(..) as $p | $dot | getpath($p) | [$p,.]'
The output of `jq --stream .` should be a sequence of`[[<path>],<leaf>]`
and `[[<path>]]` values. The latter indicate that the array/object at
that path ended.
Scalars and empty arrays and objects are leaf values for this purpose.
For example, a truncated input produces a path as soon as possible, then
later the error:
$ printf '[0,\n'|./jq -c --stream .
[[0],0]
parse error: Unfinished JSON term at EOF at line 3, column 0
$
The search path listed in an import directive can now be an array. The
top-level search path is appended. Null and empty strings in the path
terminate any search. The "." in "." and "./*" is replaced with the
directory containing the file doing the import (for command-line
programs this is the current directory, though that may be a bad idea).
No version numbers or anything of the sort are gratuitously added to the
search paths.
All this makes external package managers possible by allowing
dependencies to be installed local to dependents.
This reverts commit 77936a594d.
There are too many odd bugs in this mode, and it turns out to be a bad
idea anyways. Instead, in the future a better option will be to pursue
alternative parsers, such as:
- streaming parser that outputs only when a new leaf value is added or
an array/object is opened/closed; options here include whether to
include a path in each output;
- parsers for binary JSON encodings (there's a variety of them).
Then one might run jq with a streaming parser and use `reduce` to
coalesce inputs from some depth down (instead of from one level down as
the reverted commit had intended).
Besides, a fully streaming parser is desirable in some cases, therefore
we should have such a thing as an option.
I've explored modifying the current parser to support a streaming
option, but it only makes the code very difficult to follow, which is
one reason that alternate parsers makes sense. At any rate, this is all
for the future. For now there's no streaming of individual texts, just
text sequences.
--unbuffered was only affecting the normal output case, not the --raw-output case. Make the two of them play together.
This also makes sure that the output is flushed *after* printing the newline, so a consumer doesn't lag a line behind.
