Streaming means that outputs are produced as soon as possible. With the
`foreach` syntax one can write programs which reduce portions of the
streaming parse of a large input (reduce into proper JSON values, for
example), and discard the rest, processing incrementally.
This:
$ jq -c --stream .
should produce the same output as this:
$ jq -c '. as $dot | path(..) as $p | $dot | getpath($p) | [$p,.]'
The output of `jq --stream .` should be a sequence of`[[<path>],<leaf>]`
and `[[<path>]]` values. The latter indicate that the array/object at
that path ended.
Scalars and empty arrays and objects are leaf values for this purpose.
For example, a truncated input produces a path as soon as possible, then
later the error:
$ printf '[0,\n'|./jq -c --stream .
[[0],0]
parse error: Unfinished JSON term at EOF at line 3, column 0
$
The search path listed in an import directive can now be an array. The
top-level search path is appended. Null and empty strings in the path
terminate any search. The "." in "." and "./*" is replaced with the
directory containing the file doing the import (for command-line
programs this is the current directory, though that may be a bad idea).
No version numbers or anything of the sort are gratuitously added to the
search paths.
All this makes external package managers possible by allowing
dependencies to be installed local to dependents.
Now we have the ability to define a generator in jq:
def for(cond; update):
def _for:
if cond then ., (update | _for) else . end;
_for;
for(. < 10; . + 1) # generates numbers between `.` and 10
Running this by hand with --debug-dump-disasm (with a fix for that
coming up next) we can see that the call to _for is optimized:
_for:0:
0000 DUP
0001 CALL_JQ cond:0^1
0005 JUMP_F 0022
0007 POP
0008 FORK 0012
0010 JUMP 0020
0012 CALL_JQ update:1^1
0016 TAIL_CALL_JQ _for:0^1
0020 JUMP 0023
0022 POP
0023 RET
And timing this with 1000, 10000, 100000 iterations shows that
indeed we must be applying TCO; otherwise, without TCO, this gets
very slow very quickly.
Close #446.
Currently tested by disassembling and tracing various recursive jq
programs by hand under valgrind. An improved test framework that
can test for errors and specific bytecode patterns is in
development.
Instead of checking for self-recursion check that the thing we're
calling is a function and not a closure, therefore the new frame will
have the same env as the current frame.
Make XPath-like `//a/b` recursive structure traversal easier in jq,
which then becomes:
..|.a?.b?
The `?` operator suppresses errors about . not being an array or object.
The `..` operator is equivalent to calling the new `recurse_down`
built-in, which in turn is equivalent to
recurse(.[]?)
Note that `..a` is not supported; neither is `...a`. That could be add
added, but it doesn't seem worth the trouble of saving the need to type
a '|'.
This lets valgrind find more bugs - if a field isn't given a
well-defined value valgrind will now find it instead of seeing it
set to zero with memset.
