mirror of
https://github.com/stedolan/jq.git
synced 2024-05-11 05:55:39 +00:00
1633 lines
59 KiB
YAML
1633 lines
59 KiB
YAML
headline: jq Manual
|
|
body: |
|
|
|
|
A jq program is a "filter": it takes an input, and produces an
|
|
output. There are a lot of builtin filters for extracting a
|
|
particular field of an object, or converting a number to a string,
|
|
or various other standard tasks.
|
|
|
|
Filters can be combined in various ways - you can pipe the output of
|
|
one filter into another filter, or collect the output of a filter
|
|
into an array.
|
|
|
|
Some filters produce multiple results, for instance there's one that
|
|
produces all the elements of its input array. Piping that filter
|
|
into a second runs the second filter for each element of the
|
|
array. Generally, things that would be done with loops and iteration
|
|
in other languages are just done by gluing filters together in jq.
|
|
|
|
It's important to remember that every filter has an input and an
|
|
output. Even literals like "hello" or 42 are filters - they take an
|
|
input but always produce the same literal as output. Operations that
|
|
combine two filters, like addition, generally feed the same input to
|
|
both and combine the results. So, you can implement an averaging
|
|
filter as `add / length` - feeding the input array both to the `add`
|
|
filter and the `length` filter and dividing the results.
|
|
|
|
But that's getting ahead of ourselves. :) Let's start with something
|
|
simpler:
|
|
|
|
manpage_intro: |
|
|
jq(1) -- Command-line JSON processor
|
|
====================================
|
|
|
|
## SYNOPSIS
|
|
|
|
`jq` [<options>...] <filter> [<files>...]
|
|
|
|
`jq` can transform JSON in various ways, by selecting, iterating,
|
|
reducing and otherwise mangling JSON documents. For instance,
|
|
running the command `jq 'map(.price) | add'` will take an array of
|
|
JSON objects as input and return the sum of their "price" fields.
|
|
|
|
By default, `jq` reads a stream of JSON objects (whitespace
|
|
separated) from `stdin`. One or more <files> may be specified, in
|
|
which case `jq` will read input from those instead.
|
|
|
|
The <options> are described in the [INVOKING JQ] section, they
|
|
mostly concern input and output formatting. The <filter> is written
|
|
in the jq language and specifies how to transform the input
|
|
document.
|
|
|
|
Note: it is important to mind the shell's quoting rules. As a
|
|
general rule it's best to always quote (with single-quote
|
|
characters) the jq program, as too many characters with special
|
|
meaning to jq are also shell meta-characters. For example, `jq
|
|
"foo"` will fail on most Unix shells because that will be the same
|
|
as `jq foo`, which will generally fail because `foo is not defined`.
|
|
|
|
## FILTERS
|
|
|
|
manpage_epilogue: |
|
|
## BUGS
|
|
|
|
Presumably. Report them or discuss them at:
|
|
|
|
https://github.com/stedolan/jq/issues
|
|
|
|
## AUTHOR
|
|
|
|
Stephen Dolan `<mu@netsoc.tcd.ie>`
|
|
|
|
sections:
|
|
- title: Invoking jq
|
|
body: |
|
|
|
|
jq filters run on a stream of JSON data. The input to jq is
|
|
parsed as a sequence of whitespace-separated JSON values which
|
|
are passed through the provided filter one at a time. The
|
|
output(s) of the filter are written to standard out, again as a
|
|
sequence of whitespace-separated JSON data.
|
|
|
|
You can affect how jq reads and writes its input and output
|
|
using some command-line options:
|
|
|
|
* `--version`:
|
|
|
|
Output the jq version and exit with zero.
|
|
|
|
* `--slurp`/`-s`:
|
|
|
|
Instead of running the filter for each JSON object in the
|
|
input, read the entire input stream into a large array and run
|
|
the filter just once.
|
|
|
|
* `--online-input`/`-I`:
|
|
|
|
When the top-level input value is an array produce its elements
|
|
instead of the array. This allows on-line processing of
|
|
potentially very large top-level arrays' elements.
|
|
|
|
* `--raw-input`/`-R`:
|
|
|
|
Don't parse the input as JSON. Instead, each line of text is
|
|
passed to the filter as a string. If combined with `--slurp`,
|
|
then the entire input is passed to the filter as a single long
|
|
string.
|
|
|
|
* `--null-input`/`-n`:
|
|
|
|
Don't read any input at all! Instead, the filter is run once
|
|
using `null` as the input. This is useful when using jq as a
|
|
simple calculator or to construct JSON data from scratch.
|
|
|
|
* `--compact-output` / `-c`:
|
|
|
|
By default, jq pretty-prints JSON output. Using this option
|
|
will result in more compact output by instead putting each
|
|
JSON object on a single line.
|
|
|
|
* `--colour-output` / `-C` and `--monochrome-output` / `-M`:
|
|
|
|
By default, jq outputs colored JSON if writing to a
|
|
terminal. You can force it to produce color even if writing to
|
|
a pipe or a file using `-C`, and disable color with `-M`.
|
|
|
|
* `--ascii-output` / `-a`:
|
|
|
|
jq usually outputs non-ASCII Unicode codepoints as UTF-8, even
|
|
if the input specified them as escape sequences (like
|
|
"\u03bc"). Using this option, you can force jq to produce pure
|
|
ASCII output with every non-ASCII character replaced with the
|
|
equivalent escape sequence.
|
|
|
|
* `--unbuffered`
|
|
|
|
Flush the output after each JSON object is printed (useful if
|
|
you're piping a slow data source into jq and piping jq's
|
|
output elsewhere).
|
|
|
|
* `--sort-keys` / `-S`:
|
|
|
|
Output the fields of each object with the keys in sorted order.
|
|
|
|
* `--raw-output` / `-r`:
|
|
|
|
With this option, if the filter's result is a string then it
|
|
will be written directly to standard output rather than being
|
|
formatted as a JSON string with quotes. This can be useful for
|
|
making jq filters talk to non-JSON-based systems.
|
|
|
|
* `-f filename` / `--from-file filename`:
|
|
|
|
Read filter from the file rather than from a command line, like
|
|
awk's -f option. You can also use '#' to make comments.
|
|
|
|
* `-e` / `--exit-status`:
|
|
|
|
Sets the exit status of jq to 0 if the last output values was
|
|
neither `false` nor `null`, 1 if the last output value was
|
|
either `false` or `null`, or 4 if no valid result was ever
|
|
produced. Normally jq exits with 2 if there was any usage
|
|
problem or system error, 3 if there was a jq program compile
|
|
error, or 0 if the jq program ran.
|
|
|
|
* `--arg name value`:
|
|
|
|
This option passes a value to the jq program as a predefined
|
|
variable. If you run jq with `--arg foo bar`, then `$foo` is
|
|
available in the program and has the value `"bar"`.
|
|
|
|
* `--argfile name filename`:
|
|
|
|
This option passes the first value from the named file as a
|
|
value to the jq program as a predefined variable. If you run jq
|
|
with `--argfile foo bar`, then `$foo` is available in the
|
|
program and has the value resulting from parsing the content of
|
|
the file named `bar`.
|
|
|
|
- title: Basic filters
|
|
entries:
|
|
- title: "`.`"
|
|
body: |
|
|
|
|
The absolute simplest (and least interesting) filter
|
|
is `.`. This is a filter that takes its input and
|
|
produces it unchanged as output.
|
|
|
|
Since jq by default pretty-prints all output, this trivial
|
|
program can be a useful way of formatting JSON output from,
|
|
say, `curl`.
|
|
|
|
examples:
|
|
- program: '.'
|
|
input: '"Hello, world!"'
|
|
output: ['"Hello, world!"']
|
|
|
|
- title: "`.foo`, `.foo.bar`"
|
|
body: |
|
|
|
|
The simplest *useful* filter is `.foo`. When given a
|
|
JSON object (aka dictionary or hash) as input, it produces
|
|
the value at the key "foo", or null if there's none present.
|
|
|
|
If the key contains special characters, you need to surround
|
|
it with double quotes like this: `."foo$"`.
|
|
|
|
A filter of the form `.foo.bar` is equivalent to `.foo|.bar`.
|
|
|
|
examples:
|
|
- program: '.foo'
|
|
input: '{"foo": 42, "bar": "less interesting data"}'
|
|
output: [42]
|
|
- program: '.foo'
|
|
input: '{"notfoo": true, "alsonotfoo": false}'
|
|
output: ['null']
|
|
- program: '.["foo"]'
|
|
input: '{"foo": 42}'
|
|
output: [42]
|
|
|
|
- title: "`.foo?`"
|
|
body: |
|
|
|
|
Just like `.foo`, but does not output even an error when `.`
|
|
is not an array or an object.
|
|
|
|
examples:
|
|
- program: '.foo?'
|
|
input: '{"foo": 42, "bar": "less interesting data"}'
|
|
output: [42]
|
|
- program: '.foo?'
|
|
input: '{"notfoo": true, "alsonotfoo": false}'
|
|
output: ['null']
|
|
- program: '.["foo"]?'
|
|
input: '{"foo": 42}'
|
|
output: [42]
|
|
- program: '[.foo?]'
|
|
input: '[1,2]'
|
|
output: ['[]']
|
|
|
|
- title: "`.[<string>]`, `.[2]`, `.[10:15]`"
|
|
body: |
|
|
|
|
You can also look up fields of an object using syntax like
|
|
`.["foo"]` (.foo above is a shorthand version of this). This
|
|
one works for arrays as well, if the key is an
|
|
integer. Arrays are zero-based (like javascript), so `.[2]`
|
|
returns the third element of the array.
|
|
|
|
The `.[10:15]` syntax can be used to return a subarray of an
|
|
array or substring of a string. The array returned by
|
|
`.[10:15]` will be of length 5, containing the elements from
|
|
index 10 (inclusive) to index 15 (exclusive). Either index may
|
|
be negative (in which case it counts backwards from the end of
|
|
the array), or omitted (in which case it refers to the start
|
|
or end of the array).
|
|
|
|
The `?` "operator" can also be used with the slice operator,
|
|
as in `.[10:15]?`, which outputs values where the inputs are
|
|
slice-able.
|
|
|
|
examples:
|
|
- program: '.[0]'
|
|
input: '[{"name":"JSON", "good":true}, {"name":"XML", "good":false}]'
|
|
output: ['{"name":"JSON", "good":true}']
|
|
|
|
- program: '.[2]'
|
|
input: '[{"name":"JSON", "good":true}, {"name":"XML", "good":false}]'
|
|
output: ['null']
|
|
|
|
- program: '.[2:4]'
|
|
input: '["a","b","c","d","e"]'
|
|
output: ['["c", "d"]']
|
|
|
|
- program: '.[2:4]'
|
|
input: '"abcdefghi"'
|
|
output: ['"cd"']
|
|
|
|
- program: '.[:3]'
|
|
input: '["a","b","c","d","e"]'
|
|
output: ['["a", "b", "c"]']
|
|
|
|
- program: '.[-2:]'
|
|
input: '["a","b","c","d","e"]'
|
|
output: ['["d", "e"]']
|
|
|
|
- title: "`.[]`"
|
|
body: |
|
|
|
|
If you use the `.[index]` syntax, but omit the index
|
|
entirely, it will return *all* of the elements of an
|
|
array. Running `.[]` with the input `[1,2,3]` will produce the
|
|
numbers as three separate results, rather than as a single
|
|
array.
|
|
|
|
You can also use this on an object, and it will return all
|
|
the values of the object.
|
|
|
|
examples:
|
|
- program: '.[]'
|
|
input: '[{"name":"JSON", "good":true}, {"name":"XML", "good":false}]'
|
|
output:
|
|
- '{"name":"JSON", "good":true}'
|
|
- '{"name":"XML", "good":false}'
|
|
|
|
- program: '.[]'
|
|
input: '[]'
|
|
output: []
|
|
|
|
- program: '.[]'
|
|
input: '{"a": 1, "b": 1}'
|
|
output: ['1', '1']
|
|
|
|
- title: "`.[]?`"
|
|
body: |
|
|
|
|
Like `.[]`, but no errors will be output if . is not an array
|
|
or object.
|
|
|
|
- title: "`,`"
|
|
body: |
|
|
|
|
If two filters are separated by a comma, then the
|
|
input will be fed into both and there will be multiple
|
|
outputs: first, all of the outputs produced by the left
|
|
expression, and then all of the outputs produced by the
|
|
right. For instance, filter `.foo, .bar`, produces
|
|
both the "foo" fields and "bar" fields as separate outputs.
|
|
|
|
examples:
|
|
- program: '.foo, .bar'
|
|
input: '{"foo": 42, "bar": "something else", "baz": true}'
|
|
output: ['42', '"something else"']
|
|
|
|
- program: ".user, .projects[]"
|
|
input: '{"user":"stedolan", "projects": ["jq", "wikiflow"]}'
|
|
output: ['"stedolan"', '"jq"', '"wikiflow"']
|
|
|
|
- program: '.[4,2]'
|
|
input: '["a","b","c","d","e"]'
|
|
output: ['"e"', '"c"']
|
|
|
|
- title: "`|`"
|
|
body: |
|
|
The | operator combines two filters by feeding the output(s) of
|
|
the one on the left into the input of the one on the right. It's
|
|
pretty much the same as the Unix shell's pipe, if you're used to
|
|
that.
|
|
|
|
If the one on the left produces multiple results, the one on
|
|
the right will be run for each of those results. So, the
|
|
expression `.[] | .foo` retrieves the "foo" field of each
|
|
element of the input array.
|
|
|
|
examples:
|
|
- program: '.[] | .name'
|
|
input: '[{"name":"JSON", "good":true}, {"name":"XML", "good":false}]'
|
|
output: ['"JSON"', '"XML"']
|
|
|
|
- title: Types and Values
|
|
body: |
|
|
|
|
jq supports the same set of datatypes as JSON - numbers,
|
|
strings, booleans, arrays, objects (which in JSON-speak are
|
|
hashes with only string keys), and "null".
|
|
|
|
Booleans, null, strings and numbers are written the same way as
|
|
in javascript. Just like everything else in jq, these simple
|
|
values take an input and produce an output - `42` is a valid jq
|
|
expression that takes an input, ignores it, and returns 42
|
|
instead.
|
|
|
|
entries:
|
|
- title: Array construction - `[]`
|
|
body: |
|
|
|
|
As in JSON, `[]` is used to construct arrays, as in
|
|
`[1,2,3]`. The elements of the arrays can be any jq
|
|
expression. All of the results produced by all of the
|
|
expressions are collected into one big array. You can use it
|
|
to construct an array out of a known quantity of values (as
|
|
in `[.foo, .bar, .baz]`) or to "collect" all the results of a
|
|
filter into an array (as in `[.items[].name]`)
|
|
|
|
Once you understand the "," operator, you can look at jq's array
|
|
syntax in a different light: the expression `[1,2,3]` is not using a
|
|
built-in syntax for comma-separated arrays, but is instead applying
|
|
the `[]` operator (collect results) to the expression 1,2,3 (which
|
|
produces three different results).
|
|
|
|
If you have a filter `X` that produces four results,
|
|
then the expression `[X]` will produce a single result, an
|
|
array of four elements.
|
|
|
|
examples:
|
|
- program: "[.user, .projects[]]"
|
|
input: '{"user":"stedolan", "projects": ["jq", "wikiflow"]}'
|
|
output: ['["stedolan", "jq", "wikiflow"]']
|
|
- title: Objects - `{}`
|
|
body: |
|
|
|
|
Like JSON, `{}` is for constructing objects (aka
|
|
dictionaries or hashes), as in: `{"a": 42, "b": 17}`.
|
|
|
|
If the keys are "sensible" (all alphabetic characters), then
|
|
the quotes can be left off. The value can be any expression
|
|
(although you may need to wrap it in parentheses if it's a
|
|
complicated one), which gets applied to the {} expression's
|
|
input (remember, all filters have an input and an
|
|
output).
|
|
|
|
{foo: .bar}
|
|
|
|
will produce the JSON object `{"foo": 42}` if given the JSON
|
|
object `{"bar":42, "baz":43}`. You can use this to select
|
|
particular fields of an object: if the input is an object
|
|
with "user", "title", "id", and "content" fields and you
|
|
just want "user" and "title", you can write
|
|
|
|
{user: .user, title: .title}
|
|
|
|
Because that's so common, there's a shortcut syntax: `{user, title}`.
|
|
|
|
If one of the expressions produces multiple results,
|
|
multiple dictionaries will be produced. If the input's
|
|
|
|
{"user":"stedolan","titles":["JQ Primer", "More JQ"]}
|
|
|
|
then the expression
|
|
|
|
{user, title: .titles[]}
|
|
|
|
will produce two outputs:
|
|
|
|
{"user":"stedolan", "title": "JQ Primer"}
|
|
{"user":"stedolan", "title": "More JQ"}
|
|
|
|
Putting parentheses around the key means it will be evaluated as an
|
|
expression. With the same input as above,
|
|
|
|
{(.user): .titles}
|
|
|
|
produces
|
|
|
|
{"stedolan": ["JQ Primer", "More JQ"]}
|
|
|
|
examples:
|
|
- program: '{user, title: .titles[]}'
|
|
input: '{"user":"stedolan","titles":["JQ Primer", "More JQ"]}'
|
|
output:
|
|
- '{"user":"stedolan", "title": "JQ Primer"}'
|
|
- '{"user":"stedolan", "title": "More JQ"}'
|
|
- program: '{(.user): .titles}'
|
|
input: '{"user":"stedolan","titles":["JQ Primer", "More JQ"]}'
|
|
output: ['{"stedolan": ["JQ Primer", "More JQ"]}']
|
|
|
|
- title: Builtin operators and functions
|
|
body: |
|
|
|
|
Some jq operator (for instance, `+`) do different things
|
|
depending on the type of their arguments (arrays, numbers,
|
|
etc.). However, jq never does implicit type conversions. If you
|
|
try to add a string to an object you'll get an error message and
|
|
no result.
|
|
|
|
entries:
|
|
- title: Addition - `+`
|
|
body: |
|
|
|
|
The operator `+` takes two filters, applies them both
|
|
to the same input, and adds the results together. What
|
|
"adding" means depends on the types involved:
|
|
|
|
- **Numbers** are added by normal arithmetic.
|
|
|
|
- **Arrays** are added by being concatenated into a larger array.
|
|
|
|
- **Strings** are added by being joined into a larger string.
|
|
|
|
- **Objects** are added by merging, that is, inserting all
|
|
the key-value pairs from both objects into a single
|
|
combined object. If both objects contain a value for the
|
|
same key, the object on the right of the `+` wins. (For
|
|
recursive merge use the `*` operator.)
|
|
|
|
`null` can be added to any value, and returns the other
|
|
value unchanged.
|
|
|
|
examples:
|
|
- program: '.a + 1'
|
|
input: '{"a": 7}'
|
|
output: ['8']
|
|
- program: '.a + .b'
|
|
input: '{"a": [1,2], "b": [3,4]}'
|
|
output: ['[1,2,3,4]']
|
|
- program: '.a + null'
|
|
input: '{"a": 1}'
|
|
output: ['1']
|
|
- program: '.a + 1'
|
|
input: '{}'
|
|
output: ['1']
|
|
- program: '{a: 1} + {b: 2} + {c: 3} + {a: 42}'
|
|
input: 'null'
|
|
output: ['{"a": 42, "b": 2, "c": 3}']
|
|
|
|
- title: Subtraction - `-`
|
|
body: |
|
|
|
|
As well as normal arithmetic subtraction on numbers, the `-`
|
|
operator can be used on arrays to remove all occurences of
|
|
the second array's elements from the first array.
|
|
|
|
examples:
|
|
- program: '4 - .a'
|
|
input: '{"a":3}'
|
|
output: ['1']
|
|
- program: . - ["xml", "yaml"]
|
|
input: '["xml", "yaml", "json"]'
|
|
output: ['["json"]']
|
|
|
|
- title: Multiplication, division, modulo - `*`, `/`, and `%`
|
|
body: |
|
|
|
|
These operators only work on numbers, and do the expected.
|
|
|
|
Multiplying a string by a number produces the concatenation of
|
|
that string that many times.
|
|
|
|
Dividing a string by another splits the first using the second
|
|
as separators.
|
|
|
|
Multiplying two objects will merge them recursively: this works
|
|
like addition but if both objects contain a value for the
|
|
same key, and the values are objects, the two are merged with
|
|
the same strategy.
|
|
|
|
examples:
|
|
- program: '10 / . * 3'
|
|
input: 5
|
|
output: [6]
|
|
- program: '. / ", "'
|
|
input: '"a, b,c,d, e"'
|
|
output: ['["a","b,c,d","e"]']
|
|
- program: '{"k": {"a": 1, "b": 2}} * {"k": {"a": 0,"c": 3}}'
|
|
input: 'null'
|
|
output: ['{"k": {"a": 0, "b": 2, "c": 3}}']
|
|
|
|
- title: `length`
|
|
body: |
|
|
|
|
The builtin function `length` gets the length of various
|
|
different types of value:
|
|
|
|
- The length of a **string** is the number of Unicode
|
|
codepoints it contains (which will be the same as its
|
|
JSON-encoded length in bytes if it's pure ASCII).
|
|
|
|
- The length of an **array** is the number of elements.
|
|
|
|
- The length of an **object** is the number of key-value pairs.
|
|
|
|
- The length of **null** is zero.
|
|
|
|
examples:
|
|
- program: '.[] | length'
|
|
input: '[[1,2], "string", {"a":2}, null]'
|
|
output: [2, 6, 1, 0]
|
|
|
|
- title: `keys`
|
|
body: |
|
|
|
|
The builtin function `keys`, when given an object, returns
|
|
its keys in an array.
|
|
|
|
The keys are sorted "alphabetically", by unicode codepoint
|
|
order. This is not an order that makes particular sense in
|
|
any particular language, but you can count on it being the
|
|
same for any two objects with the same set of keys,
|
|
regardless of locale settings.
|
|
|
|
When `keys` is given an array, it returns the valid indices
|
|
for that array: the integers from 0 to length-1.
|
|
|
|
examples:
|
|
- program: 'keys'
|
|
input: '{"abc": 1, "abcd": 2, "Foo": 3}'
|
|
output: ['["Foo", "abc", "abcd"]']
|
|
- program: 'keys'
|
|
input: '[42,3,35]'
|
|
output: ['[0,1,2]']
|
|
|
|
- title: `has`
|
|
body: |
|
|
|
|
The builtin function `has` returns whether the input object
|
|
has the given key, or the input array has an element at the
|
|
given index.
|
|
|
|
`has($key)` has the same effect as checking whether `$key`
|
|
is a member of the array returned by `keys`, although `has`
|
|
will be faster.
|
|
|
|
examples:
|
|
- program: 'map(has("foo"))'
|
|
input: '[{"foo": 42}, {}]'
|
|
output: ['[true, false]']
|
|
- program: 'map(has(2))'
|
|
input: '[[0,1], ["a","b","c"]]'
|
|
output: ['[false, true]']
|
|
|
|
- title: `del`
|
|
body: |
|
|
|
|
The builtin function `del` removes a key and its corresponding
|
|
value from an object.
|
|
|
|
examples:
|
|
- program: 'del(.foo)'
|
|
input: '[{"foo": 42, "bar": 9001, "baz": 42}]'
|
|
output: ['{"bar": 9001, "baz": 42}']
|
|
- program: 'del(.[1, 2])'
|
|
input: '[["foo", "bar", "baz"]]'
|
|
output: ['["foo"]']
|
|
|
|
- title: `to_entries`, `from_entries`, `with_entries`
|
|
body: |
|
|
|
|
These functions convert between an object and an array of
|
|
key-value pairs. If `to_entries` is passed an object, then
|
|
for each `k: v` entry in the input, the output array
|
|
includes `{"key": k, "value": v}`.
|
|
|
|
`from_entries` does the opposite conversion, and
|
|
`with_entries(foo)` is a shorthand for `to_entries |
|
|
map(foo) | from_entries`, useful for doing some operation to
|
|
all keys and values of an object.
|
|
|
|
examples:
|
|
- program: 'to_entries'
|
|
input: '{"a": 1, "b": 2}'
|
|
output: ['[{"key":"a", "value":1}, {"key":"b", "value":2}]']
|
|
- program: 'from_entries'
|
|
input: '[{"key":"a", "value":1}, {"key":"b", "value":2}]'
|
|
output: ['{"a": 1, "b": 2}']
|
|
- program: 'with_entries(.key |= "KEY_" + .)'
|
|
input: '{"a": 1, "b": 2}'
|
|
output: ['{"KEY_a": 1, "KEY_b": 2}']
|
|
|
|
|
|
- title: `select`
|
|
body: |
|
|
|
|
The function `select(foo)` produces its input unchanged if
|
|
`foo` returns true for that input, and produces no output
|
|
otherwise.
|
|
|
|
It's useful for filtering lists: `[1,2,3] | map(select(. >= 2))`
|
|
will give you `[2,3]`.
|
|
|
|
examples:
|
|
- program: 'map(select(. >= 2))'
|
|
input: '[1,5,3,0,7]'
|
|
output: ['[5,3,7]']
|
|
|
|
|
|
- title: `arrays`, `objects`, `iterables`, `booleans`, `numbers`, `strings`, `nulls`, `values`, `scalars`
|
|
body: |
|
|
|
|
These built-ins select only inputs that are arrays, objects,
|
|
iterables (arrays or objects), booleans, numbers, strings,
|
|
null, non-null values, and non-iterables, respectively.
|
|
|
|
examples:
|
|
- program: '.[]|numbers'
|
|
input: '[[],{},1,"foo",null,true,false]'
|
|
output: ['1']
|
|
|
|
- title: `empty`
|
|
body: |
|
|
|
|
`empty` returns no results. None at all. Not even `null`.
|
|
|
|
It's useful on occasion. You'll know if you need it :)
|
|
|
|
examples:
|
|
- program: '1, empty, 2'
|
|
input: 'null'
|
|
output: [1, 2]
|
|
- program: '[1,2,empty,3]'
|
|
input: 'null'
|
|
output: ['[1,2,3]']
|
|
|
|
- title: `map(x)`
|
|
body: |
|
|
|
|
For any filter `x`, `map(x)` will run that filter for each
|
|
element of the input array, and produce the outputs a new
|
|
array. `map(.+1)` will increment each element of an array of numbers.
|
|
|
|
`map(x)` is equivalent to `[.[] | x]`. In fact, this is how
|
|
it's defined.
|
|
|
|
examples:
|
|
- program: 'map(.+1)'
|
|
input: '[1,2,3]'
|
|
output: ['[2,3,4]']
|
|
|
|
- title: `paths`
|
|
body: |
|
|
|
|
Outputs the paths to all the elements in its input (except it
|
|
does not output the empty list, representing . itself).
|
|
|
|
`paths` is equivalent to
|
|
|
|
def paths: path(recurse(if (type|. == "array" or . == "object") then .[] else empty end))|select(length > 0);
|
|
|
|
examples:
|
|
- program: '[paths]'
|
|
input: '[1,[[],{"a":2}]]'
|
|
output: ['[[0],[1],[1,0],[1,1],[1,1,"a"]]']
|
|
|
|
- title: `leaf_paths`
|
|
body: |
|
|
|
|
Outputs the paths to all the leaves (non-array, non-object
|
|
elements) in its input.
|
|
|
|
`leaf_paths` is equivalent to
|
|
|
|
def leaf_paths: . as $dot|paths|select(. as $p|$dot|getpath($p)|type|. != "array" and . != "object");
|
|
|
|
examples:
|
|
- program: '[leaf_paths]'
|
|
input: '[1,[[],{"a":2}]]'
|
|
output: ['[[0],[1,1,"a"]]']
|
|
|
|
- title: `add`
|
|
body: |
|
|
|
|
The filter `add` takes as input an array, and produces as
|
|
output the elements of the array added together. This might
|
|
mean summed, concatenated or merged depending on the types
|
|
of the elements of the input array - the rules are the same
|
|
as those for the `+` operator (described above).
|
|
|
|
If the input is an empty array, `add` returns `null`.
|
|
|
|
examples:
|
|
- program: add
|
|
input: '["a","b","c"]'
|
|
output: ['"abc"']
|
|
- program: add
|
|
input: '[1, 2, 3]'
|
|
output: [6]
|
|
- program: add
|
|
input: '[]'
|
|
output: ["null"]
|
|
|
|
- title: `any`
|
|
body: |
|
|
|
|
The filter `any` takes as input an array of boolean values,
|
|
and produces `true` as output if any of the the elements of
|
|
the array is `true`.
|
|
|
|
If the input is an empty array, `any` returns `false`.
|
|
|
|
examples:
|
|
- program: any
|
|
input: '[true, false]'
|
|
output: ["true"]
|
|
- program: any
|
|
input: '[false, false]'
|
|
output: ["false"]
|
|
- program: any
|
|
input: '[]'
|
|
output: ["false"]
|
|
|
|
- title: `all`
|
|
body: |
|
|
|
|
The filter `all` takes as input an array of boolean values,
|
|
and produces `true` as output if all of the the elements of
|
|
the array are `true`.
|
|
|
|
If the input is an empty array, `all` returns `true`.
|
|
|
|
examples:
|
|
- program: all
|
|
input: '[true, false]'
|
|
output: ["false"]
|
|
- program: all
|
|
input: '[true, true]'
|
|
output: ["true"]
|
|
- program: all
|
|
input: '[]'
|
|
output: ["true"]
|
|
|
|
- title: `range`
|
|
body: |
|
|
|
|
The `range` function produces a range of numbers. `range(4;10)`
|
|
produces 6 numbers, from 4 (inclusive) to 10 (exclusive). The numbers
|
|
are produced as separate outputs. Use `[range(4;10)]` to get a range as
|
|
an array.
|
|
|
|
examples:
|
|
- program: 'range(2;4)'
|
|
input: 'null'
|
|
output: ['2', '3']
|
|
- program: '[range(2;4)]'
|
|
input: 'null'
|
|
output: ['[2,3]']
|
|
|
|
- title: `floor`
|
|
body: |
|
|
|
|
The `floor` function returns the floor of its numeric input.
|
|
|
|
examples:
|
|
- program: 'floor'
|
|
input: '3.14159'
|
|
output: ['3']
|
|
|
|
- title: `sqrt`
|
|
body: |
|
|
|
|
The `sqrt` function returns the square root of its numeric input.
|
|
|
|
examples:
|
|
- program: 'sqrt'
|
|
input: '9'
|
|
output: ['3']
|
|
|
|
- title: `tonumber`
|
|
body: |
|
|
|
|
The `tonumber` function parses its input as a number. It
|
|
will convert correctly-formatted strings to their numeric
|
|
equivalent, leave numbers alone, and give an error on all other input.
|
|
|
|
examples:
|
|
- program: '.[] | tonumber'
|
|
input: '[1, "1"]'
|
|
output: [1, 1]
|
|
|
|
- title: `tostring`
|
|
body: |
|
|
|
|
The `tostring` function prints its input as a
|
|
string. Strings are left unchanged, and all other values are
|
|
JSON-encoded.
|
|
|
|
examples:
|
|
- program: '.[] | tostring'
|
|
input: '[1, "1", [1]]'
|
|
output: ['"1"', '"1"', '"[1]"']
|
|
|
|
- title: `type`
|
|
body: |
|
|
|
|
The `type` function returns the type of its argument as a
|
|
string, which is one of null, boolean, number, string, array
|
|
or object.
|
|
|
|
examples:
|
|
- program: 'map(type)'
|
|
input: '[0, false, [], {}, null, "hello"]'
|
|
output: ['["number", "boolean", "array", "object", "null", "string"]']
|
|
|
|
- title: `sort, sort_by`
|
|
body: |
|
|
|
|
The `sort` functions sorts its input, which must be an
|
|
array. Values are sorted in the following order:
|
|
|
|
* `null`
|
|
* `false`
|
|
* `true`
|
|
* numbers
|
|
* strings, in alphabetical order (by unicode codepoint value)
|
|
* arrays, in lexical order
|
|
* objects
|
|
|
|
The ordering for objects is a little complex: first they're
|
|
compared by comparing their sets of keys (as arrays in
|
|
sorted order), and if their keys are equal then the values
|
|
are compared key by key.
|
|
|
|
`sort_by` may be used to sort by a particular field of an
|
|
object, or by applying any jq filter. `sort_by(foo)`
|
|
compares two elements by comparing the result of `foo` on
|
|
each element.
|
|
|
|
examples:
|
|
- program: 'sort'
|
|
input: '[8,3,null,6]'
|
|
output: ['[null,3,6,8]']
|
|
- program: 'sort_by(.foo)'
|
|
input: '[{"foo":4, "bar":10}, {"foo":3, "bar":100}, {"foo":2, "bar":1}]'
|
|
output: ['[{"foo":2, "bar":1}, {"foo":3, "bar":100}, {"foo":4, "bar":10}]']
|
|
|
|
- title: `group_by`
|
|
body: |
|
|
|
|
`group_by(.foo)` takes as input an array, groups the
|
|
elements having the same `.foo` field into separate arrays,
|
|
and produces all of these arrays as elements of a larger
|
|
array, sorted by the value of the `.foo` field.
|
|
|
|
Any jq expression, not just a field access, may be used in
|
|
place of `.foo`. The sorting order is the same as described
|
|
in the `sort` function above.
|
|
|
|
examples:
|
|
- program: 'group_by(.foo)'
|
|
input: '[{"foo":1, "bar":10}, {"foo":3, "bar":100}, {"foo":1, "bar":1}]'
|
|
output: ['[[{"foo":1, "bar":10}, {"foo":1, "bar":1}], [{"foo":3, "bar":100}]]']
|
|
|
|
- title: `min`, `max`, `min_by`, `max_by`
|
|
body: |
|
|
|
|
Find the minimum or maximum element of the input array. The
|
|
`_by` versions allow you to specify a particular field or
|
|
property to examine, e.g. `min_by(.foo)` finds the object
|
|
with the smallest `foo` field.
|
|
|
|
examples:
|
|
- program: 'min'
|
|
input: '[5,4,2,7]'
|
|
output: ['2']
|
|
- program: 'max_by(.foo)'
|
|
input: '[{"foo":1, "bar":14}, {"foo":2, "bar":3}]'
|
|
output: ['{"foo":2, "bar":3}']
|
|
|
|
- title: `unique`
|
|
body: |
|
|
|
|
The `unique` function takes as input an array and produces
|
|
an array of the same elements, in sorted order, with
|
|
duplicates removed.
|
|
|
|
examples:
|
|
- program: 'unique'
|
|
input: '[1,2,5,3,5,3,1,3]'
|
|
output: ['[1,2,3,5]']
|
|
|
|
- title: `unique_by`
|
|
body: |
|
|
|
|
The `unique_by(.foo)` function takes as input an array and produces
|
|
an array of the same elements, in sorted order, with
|
|
elqements with a duplicate `.foo` field removed. Think of it as making
|
|
an array by taking one element out of every group produced by
|
|
`group_by`.
|
|
|
|
examples:
|
|
- program: 'unique_by(.foo)'
|
|
input: '[{"foo": 1, "bar": 2}, {"foo": 1, "bar": 3}, {"foo": 4, "bar": 5}]'
|
|
output: ['[{"foo": 1, "bar": 2}, {"foo": 4, "bar": 5}]']
|
|
- program: 'unique_by(length)'
|
|
input: '["chunky", "bacon", "kitten", "cicada", "asparagus"]'
|
|
output: ['["chunky", "bacon", "asparagus"]']
|
|
|
|
|
|
- title: `reverse`
|
|
body: |
|
|
|
|
This function reverses an array.
|
|
|
|
examples:
|
|
- program: 'reverse'
|
|
input: '[1,2,3,4]'
|
|
output: ['[4,3,2,1]']
|
|
|
|
- title: `contains`
|
|
body: |
|
|
|
|
The filter `contains(b)` will produce true if b is
|
|
completely contained within the input. A string B is
|
|
contained in a string A if B is a substring of A. An array B
|
|
is contained in an array A is all elements in B are
|
|
contained in any element in A. An object B is contained in
|
|
object A if all of the values in B are contained in the
|
|
value in A with the same key. All other types are assumed to
|
|
be contained in each other if they are equal.
|
|
|
|
examples:
|
|
- program: 'contains("bar")'
|
|
input: '"foobar"'
|
|
output: ['true']
|
|
- program: 'contains(["baz", "bar"])'
|
|
input: '["foobar", "foobaz", "blarp"]'
|
|
output: ['true']
|
|
- program: 'contains(["bazzzzz", "bar"])'
|
|
input: '["foobar", "foobaz", "blarp"]'
|
|
output: ['false']
|
|
- program: 'contains({foo: 12, bar: [{barp: 12}]})'
|
|
input: '{"foo": 12, "bar":[1,2,{"barp":12, "blip":13}]}'
|
|
output: ['true']
|
|
- program: 'contains({foo: 12, bar: [{barp: 15}]})'
|
|
input: '{"foo": 12, "bar":[1,2,{"barp":12, "blip":13}]}'
|
|
output: ['false']
|
|
|
|
- title: `startswith`
|
|
body: |
|
|
|
|
Outputs `true` if . starts with the given string argument.
|
|
|
|
examples:
|
|
- program: '[.[]|startswith("foo")]'
|
|
input: '["fo", "foo", "barfoo", "foobar", "barfoob"]'
|
|
output: ['[false, true, false, true, false]']
|
|
|
|
- title: `endswith`
|
|
body: |
|
|
|
|
Outputs `true` if . ends with the given string argument.
|
|
|
|
examples:
|
|
- program: '[.[]|endswith("foo")]'
|
|
input: '["foobar", "barfoo"]'
|
|
output: ['[false, true, true, false, false]']
|
|
|
|
- title: `ltrimstr`
|
|
body: |
|
|
|
|
Outputs its input with the given prefix string removed, if it
|
|
starts with it.
|
|
|
|
examples:
|
|
- program: '[.[]|ltrimstr("foo")]'
|
|
input: '["fo", "foo", "barfoo", "foobar", "afoo"]'
|
|
output: ['["fo","","barfoo","bar","afoo"]']
|
|
|
|
- title: `rtrimstr`
|
|
body: |
|
|
|
|
Outputs its input with the given suffix string removed, if it
|
|
starts with it.
|
|
|
|
examples:
|
|
- program: '[.[]|rtrimstr("foo")]'
|
|
input: '["fo", "foo", "barfoo", "foobar", "foob"]'
|
|
output: ['["fo","","bar","foobar","foob"]']
|
|
|
|
- title: `explode`
|
|
body: |
|
|
|
|
Converts an input string into an array of the string's
|
|
codepoint numbers.
|
|
|
|
examples:
|
|
- program: 'explode'
|
|
input: '"foobar"'
|
|
output: ['[102,111,111,98,97,114]']
|
|
|
|
- title: `implode`
|
|
body: |
|
|
|
|
The inverse of explode.
|
|
|
|
examples:
|
|
- program: 'implode'
|
|
input: '[65, 66, 67]'
|
|
output: ['"ABC"']
|
|
|
|
- title: `split`
|
|
body: |
|
|
|
|
Splits an input string on the separator argument.
|
|
|
|
examples:
|
|
- program: 'split(", ")'
|
|
input: '"a, b,c,d, e"'
|
|
output: ['["a","b,c,d","e"]']
|
|
|
|
- title: `join`
|
|
body: |
|
|
|
|
Joins the array of elements given as input, using the
|
|
argument as separator. It is the inverse of `split`: that is,
|
|
running `split("foo") | join("foo")` over any input string
|
|
returns said input string.
|
|
|
|
examples:
|
|
- program: 'join(", ")'
|
|
input: '["a","b,c,d","e"]'
|
|
output: ['"a, b,c,d, e"']
|
|
|
|
|
|
- title: `recurse`
|
|
body: |
|
|
|
|
The `recurse` function allows you to search through a
|
|
recursive structure, and extract interesting data from all
|
|
levels. Suppose your input represents a filesystem:
|
|
|
|
{"name": "/", "children": [
|
|
{"name": "/bin", "children": [
|
|
{"name": "/bin/ls", "children": []},
|
|
{"name": "/bin/sh", "children": []}]},
|
|
{"name": "/home", "children": [
|
|
{"name": "/home/stephen", "children": [
|
|
{"name": "/home/stephen/jq", "children": []}]}]}]}
|
|
|
|
Now suppose you want to extract all of the filenames
|
|
present. You need to retrieve `.name`, `.children[].name`,
|
|
`.children[].children[].name`, and so on. You can do this
|
|
with:
|
|
|
|
recurse(.children[]) | .name
|
|
|
|
examples:
|
|
- program: 'recurse(.foo[])'
|
|
input: '{"foo":[{"foo": []}, {"foo":[{"foo":[]}]}]}'
|
|
output:
|
|
- '{"foo":[{"foo":[]},{"foo":[{"foo":[]}]}]}'
|
|
- '{"foo":[]}'
|
|
- '{"foo":[{"foo":[]}]}'
|
|
- '{"foo":[]}'
|
|
|
|
- title: `recurse_down`
|
|
body: |
|
|
|
|
A quieter version of `recurse(.[])`, equivalent to:
|
|
|
|
def recurse_down: recurse(.[]?);
|
|
|
|
- title: `..`
|
|
body: |
|
|
|
|
Short-hand for `recurse_down`. This is intended to resemble
|
|
the XPath `//` operator. Note that `..a` does not work; use
|
|
`..|a` instead.
|
|
|
|
examples:
|
|
- program: '..|.a?'
|
|
input: '[[{"a":1}]]'
|
|
output: ['1']
|
|
|
|
- title: "String interpolation - `\(foo)`"
|
|
body: |
|
|
|
|
Inside a string, you can put an expression inside parens
|
|
after a backslash. Whatever the expression returns will be
|
|
interpolated into the string.
|
|
|
|
examples:
|
|
- program: '"The input was \(.), which is one less than \(.+1)"'
|
|
input: '42'
|
|
output: ['"The input was 42, which is one less than 43"']
|
|
|
|
- title: "Convert to/from JSON"
|
|
body: |
|
|
|
|
The `tojson` and `fromjson` builtins dump values as JSON texts
|
|
or parse JSON texts into values, respectively. The tojson
|
|
builtin differs from tostring in that tostring returns strings
|
|
unmodified, while tojson encodes strings as JSON strings.
|
|
|
|
examples:
|
|
- program: '[.[]|tostring]'
|
|
input: '[1, "foo", ["foo"]]'
|
|
output: ['["1","foo","[\"foo\"]"]']
|
|
- program: '[.[]|tojson]'
|
|
input: '[1, "foo", ["foo"]]'
|
|
output: ['["1","\"foo\"","[\"foo\"]"]']
|
|
- program: '[.[]|tojson|fromjson]'
|
|
input: '[1, "foo", ["foo"]]'
|
|
output: ['[1,"foo",["foo"]]']
|
|
|
|
- title: "Format strings and escaping"
|
|
body: |
|
|
|
|
The `@foo` syntax is used to format and escape strings,
|
|
which is useful for building URLs, documents in a language
|
|
like HTML or XML, and so forth. `@foo` can be used as a
|
|
filter on its own, the possible escapings are:
|
|
|
|
* `@text`:
|
|
|
|
Calls `tostring`, see that function for details.
|
|
|
|
* `@json`:
|
|
|
|
Serialises the input as JSON.
|
|
|
|
* `@html`:
|
|
|
|
Applies HTML/XML escaping, by mapping the characters
|
|
`<>&'"` to their entity equivalents `<`, `>`,
|
|
`&`, `'`, `"`.
|
|
|
|
* `@uri`:
|
|
|
|
Applies percent-encoding, by mapping all reserved URI
|
|
characters to a `%xx` sequence.
|
|
|
|
* `@csv`:
|
|
|
|
The input must be an array, and it is rendered as CSV
|
|
with double quotes for strings, and quotes escaped by
|
|
repetition.
|
|
|
|
* `@sh`:
|
|
|
|
The input is escaped suitable for use in a command-line
|
|
for a POSIX shell. If the input is an array, the output
|
|
will be a series of space-separated strings.
|
|
|
|
* `@base64`:
|
|
|
|
The input is converted to base64 as specified by RFC 4648.
|
|
|
|
This syntax can be combined with string interpolation in a
|
|
useful way. You can follow a `@foo` token with a string
|
|
literal. The contents of the string literal will *not* be
|
|
escaped. However, all interpolations made inside that string
|
|
literal will be escaped. For instance,
|
|
|
|
@uri "http://www.google.com/search?q=\(.search)"
|
|
|
|
will produce the following output for the input
|
|
`{"search":"what is jq?"}`:
|
|
|
|
"http://www.google.com/search?q=what%20is%20jq%3f"
|
|
|
|
Note that the slashes, question mark, etc. in the URL are
|
|
not escaped, as they were part of the string literal.
|
|
|
|
examples:
|
|
- program: '@html'
|
|
input: '"This works if x < y"'
|
|
output: ['"This works if x < y"']
|
|
|
|
# - program: '@html "<span>Anonymous said: \(.)</span>"'
|
|
# input: '"<script>alert(\"lol hax\");</script>"'
|
|
# output: ["<span>Anonymous said: <script>alert("lol hax");</script></span>"]
|
|
|
|
- program: '@sh "echo \(.)"'
|
|
input: "\"O'Hara's Ale\""
|
|
output: ["\"echo 'O'\\\\''Hara'\\\\''s Ale'\""]
|
|
|
|
- title: Conditionals and Comparisons
|
|
entries:
|
|
- title: `==`, `!=`
|
|
body: |
|
|
|
|
The expression 'a == b' will produce 'true' if the result of a and b
|
|
are equal (that is, if they represent equivalent JSON documents) and
|
|
'false' otherwise. In particular, strings are never considered equal
|
|
to numbers. If you're coming from Javascript, jq's == is like
|
|
Javascript's === - considering values equal only when they have the
|
|
same type as well as the same value.
|
|
|
|
!= is "not equal", and 'a != b' returns the opposite value of 'a == b'
|
|
|
|
examples:
|
|
- program: '.[] == 1'
|
|
input: '[1, 1.0, "1", "banana"]'
|
|
output: ['true', 'true', 'false', 'false']
|
|
- title: if-then-else
|
|
body: |
|
|
|
|
`if A then B else C end` will act the same as `B` if `A`
|
|
produces a value other than false or null, but act the same
|
|
as `C` otherwise.
|
|
|
|
Checking for false or null is a simpler notion of
|
|
"truthiness" than is found in Javascript or Python, but it
|
|
means that you'll sometimes have to be more explicit about
|
|
the condition you want: you can't test whether, e.g. a
|
|
string is empty using `if .name then A else B end`, you'll
|
|
need something more like `if (.name | length) > 0 then A else
|
|
B end` instead.
|
|
|
|
If the condition A produces multiple results, it is
|
|
considered "true" if any of those results is not false or
|
|
null. If it produces zero results, it's considered false.
|
|
|
|
More cases can be added to an if using `elif A then B` syntax.
|
|
|
|
examples:
|
|
- program: |-
|
|
if . == 0 then
|
|
"zero"
|
|
elif . == 1 then
|
|
"one"
|
|
else
|
|
"many"
|
|
end
|
|
input: 2
|
|
output: ['"many"']
|
|
|
|
- title: `>, >=, <=, <`
|
|
body: |
|
|
|
|
The comparison operators `>`, `>=`, `<=`, `<` return whether
|
|
their left argument is greater than, greater than or equal
|
|
to, less than or equal to or less than their right argument
|
|
(respectively).
|
|
|
|
The ordering is the same as that described for `sort`, above.
|
|
|
|
examples:
|
|
- program: '. < 5'
|
|
input: 2
|
|
output: ['true']
|
|
|
|
- title: and/or/not
|
|
body: |
|
|
|
|
jq supports the normal Boolean operators and/or/not. They have the
|
|
same standard of truth as if expressions - false and null are
|
|
considered "false values", and anything else is a "true value".
|
|
|
|
If an operand of one of these operators produces multiple
|
|
results, the operator itself will produce a result for each input.
|
|
|
|
`not` is in fact a builtin function rather than an operator,
|
|
so it is called as a filter to which things can be piped
|
|
rather than with special syntax, as in `.foo and .bar |
|
|
not`.
|
|
|
|
These three only produce the values "true" and "false", and
|
|
so are only useful for genuine Boolean operations, rather
|
|
than the common Perl/Python/Ruby idiom of
|
|
"value_that_may_be_null or default". If you want to use this
|
|
form of "or", picking between two values rather than
|
|
evaluating a condition, see the "//" operator below.
|
|
|
|
examples:
|
|
- program: '42 and "a string"'
|
|
input: 'null'
|
|
output: ['true']
|
|
- program: '(true, false) or false'
|
|
input: 'null'
|
|
output: ['true', 'false']
|
|
# - program: '(true, false) and (true, false)'
|
|
# input: 'null'
|
|
# output: ['true', 'false', 'false', 'false']
|
|
- program: '(true, true) and (true, false)'
|
|
input: 'null'
|
|
output: ['true', 'false', 'true', 'false']
|
|
- program: '[true, false | not]'
|
|
input: 'null'
|
|
output: ['[false, true]']
|
|
|
|
- title: Alternative operator - `//`
|
|
body: |
|
|
|
|
A filter of the form `a // b` produces the same
|
|
results as `a`, if `a` produces results other than `false`
|
|
and `null`. Otherwise, `a // b` produces the same results as `b`.
|
|
|
|
This is useful for providing defaults: `.foo // 1` will
|
|
evaluate to `1` if there's no `.foo` element in the
|
|
input. It's similar to how `or` is sometimes used in Python
|
|
(jq's `or` operator is reserved for strictly Boolean
|
|
operations).
|
|
|
|
examples:
|
|
- program: '.foo // 42'
|
|
input: '{"foo": 19}'
|
|
output: [19]
|
|
- program: '.foo // 42'
|
|
input: '{}'
|
|
output: [42]
|
|
|
|
- title: Advanced features
|
|
body: |
|
|
Variables are an absolute necessity in most programming languages, but
|
|
they're relegated to an "advanced feature" in jq.
|
|
|
|
In most languages, variables are the only means of passing around
|
|
data. If you calculate a value, and you want to use it more than once,
|
|
you'll need to store it in a variable. To pass a value to another part
|
|
of the program, you'll need that part of the program to define a
|
|
variable (as a function parameter, object member, or whatever) in
|
|
which to place the data.
|
|
|
|
It is also possible to define functions in jq, although this is
|
|
is a feature whose biggest use is defining jq's standard library
|
|
(many jq functions such as `map` and `find` are in fact written
|
|
in jq).
|
|
|
|
Finally, jq has a `reduce` operation, which is very powerful but a
|
|
bit tricky. Again, it's mostly used internally, to define some
|
|
useful bits of jq's standard library.
|
|
|
|
entries:
|
|
- title: Variables
|
|
body: |
|
|
|
|
In jq, all filters have an input and an output, so manual
|
|
plumbing is not necessary to pass a value from one part of a program
|
|
to the next. Many expressions, for instance `a + b`, pass their input
|
|
to two distinct subexpressions (here `a` and `b` are both passed the
|
|
same input), so variables aren't usually necessary in order to use a
|
|
value twice.
|
|
|
|
For instance, calculating the average value of an array of numbers
|
|
requires a few variables in most languages - at least one to hold the
|
|
array, perhaps one for each element or for a loop counter. In jq, it's
|
|
simply `add / length` - the `add` expression is given the array and
|
|
produces its sum, and the `length` expression is given the array and
|
|
produces its length.
|
|
|
|
So, there's generally a cleaner way to solve most problems in jq than
|
|
defining variables. Still, sometimes they do make things easier, so jq
|
|
lets you define variables using `expression as $variable`. All
|
|
variable names start with `$`. Here's a slightly uglier version of the
|
|
array-averaging example:
|
|
|
|
length as $array_length | add / $array_length
|
|
|
|
We'll need a more complicated problem to find a situation where using
|
|
variables actually makes our lives easier.
|
|
|
|
|
|
Suppose we have an array of blog posts, with "author" and "title"
|
|
fields, and another object which is used to map author usernames to
|
|
real names. Our input looks like:
|
|
|
|
{"posts": [{"title": "Frist psot", "author": "anon"},
|
|
{"title": "A well-written article", "author": "person1"}],
|
|
"realnames": {"anon": "Anonymous Coward",
|
|
"person1": "Person McPherson"}}
|
|
|
|
We want to produce the posts with the author field containing a real
|
|
name, as in:
|
|
|
|
{"title": "Frist psot", "author": "Anonymous Coward"}
|
|
{"title": "A well-written article", "author": "Person McPherson"}
|
|
|
|
We use a variable, $names, to store the realnames object, so that we
|
|
can refer to it later when looking up author usernames:
|
|
|
|
.realnames as $names | .posts[] | {title, author: $names[.author]}
|
|
|
|
The expression `exp as $x | ...` means: for each value of expression
|
|
`exp`, run the rest of the pipeline with the entire original input, and
|
|
with `$x` set to that value. Thus `as` functions as something of a
|
|
foreach loop.
|
|
|
|
Variables are scoped over the rest of the expression that defines
|
|
them, so
|
|
|
|
.realnames as $names | (.posts[] | {title, author: $names[.author]})
|
|
|
|
will work, but
|
|
|
|
(.realnames as $names | .posts[]) | {title, author: $names[.author]}
|
|
|
|
won't.
|
|
|
|
examples:
|
|
- program: '.bar as $x | .foo | . + $x'
|
|
input: '{"foo":10, "bar":200}'
|
|
output: ['210']
|
|
|
|
- title: 'Defining Functions'
|
|
body: |
|
|
|
|
You can give a filter a name using "def" syntax:
|
|
|
|
def increment: . + 1;
|
|
|
|
From then on, `increment` is usable as a filter just like a
|
|
builtin function (in fact, this is how some of the builtins
|
|
are defined). A function may take arguments:
|
|
|
|
def map(f): [.[] | f];
|
|
|
|
Arguments are passed as filters, not as values. The
|
|
same argument may be referenced multiple times with
|
|
different inputs (here `f` is run for each element of the
|
|
input array). Arguments to a function work more like
|
|
callbacks than like value arguments.
|
|
|
|
If you want the value-argument behaviour for defining simple
|
|
functions, you can just use a variable:
|
|
|
|
def addvalue(f): f as $value | map(. + $value);
|
|
|
|
With that definition, `addvalue(.foo)` will add the current
|
|
input's `.foo` field to each element of the array.
|
|
|
|
examples:
|
|
- program: 'def addvalue(f): . + [f]; map(addvalue(.[0]))'
|
|
input: '[[1,2],[10,20]]'
|
|
output: ['[[1,2,1], [10,20,10]]']
|
|
- program: 'def addvalue(f): f as $x | map(. + $x); addvalue(.[0])'
|
|
input: '[[1,2],[10,20]]'
|
|
output: ['[[1,2,1,2], [10,20,1,2]]']
|
|
|
|
- title: Reduce
|
|
body: |
|
|
|
|
The `reduce` syntax in jq allows you to combine all of the
|
|
results of an expression by accumulating them into a single
|
|
answer. As an example, we'll pass `[3,2,1]` to this expression:
|
|
|
|
reduce .[] as $item (0; . + $item)
|
|
|
|
For each result that `.[]` produces, `. + $item` is run to
|
|
accumulate a running total, starting from 0. In this
|
|
example, `.[]` produces the results 3, 2, and 1, so the
|
|
effect is similar to running something like this:
|
|
|
|
0 | (3 as $item | . + $item) |
|
|
(2 as $item | . + $item) |
|
|
(1 as $item | . + $item)
|
|
|
|
examples:
|
|
- program: 'reduce .[] as $item (0; . + $item)'
|
|
input: '[10,2,5,3]'
|
|
output: ['20']
|
|
|
|
|
|
- title: Assignment
|
|
body: |
|
|
|
|
Assignment works a little differently in jq than in most
|
|
programming languages. jq doesn't distinguish between references
|
|
to and copies of something - two objects or arrays are either
|
|
equal or not equal, without any further notion of being "the
|
|
same object" or "not the same object".
|
|
|
|
If an object has two fields which are arrays, `.foo` and `.bar`,
|
|
and you append something to `.foo`, then `.bar` will not get
|
|
bigger. Even if you've just set `.bar = .foo`. If you're used to
|
|
programming in languages like Python, Java, Ruby, Javascript,
|
|
etc. then you can think of it as though jq does a full deep copy
|
|
of every object before it does the assignment (for performance,
|
|
it doesn't actually do that, but that's the general idea).
|
|
|
|
entries:
|
|
- title: "`=`"
|
|
body: |
|
|
|
|
The filter `.foo = 1` will take as input an object
|
|
and produce as output an object with the "foo" field set to
|
|
1. There is no notion of "modifying" or "changing" something
|
|
in jq - all jq values are immutable. For instance,
|
|
|
|
.foo = .bar | .foo.baz = 1
|
|
|
|
will not have the side-effect of setting .bar.baz to be set
|
|
to 1, as the similar-looking program in Javascript, Python,
|
|
Ruby or other languages would. Unlike these languages (but
|
|
like Haskell and some other functional languages), there is
|
|
no notion of two arrays or objects being "the same array" or
|
|
"the same object". They can be equal, or not equal, but if
|
|
we change one of them in no circumstances will the other
|
|
change behind our backs.
|
|
|
|
This means that it's impossible to build circular values in
|
|
jq (such as an array whose first element is itself). This is
|
|
quite intentional, and ensures that anything a jq program
|
|
can produce can be represented in JSON.
|
|
|
|
- title: "`|=`"
|
|
body: |
|
|
As well as the assignment operator '=', jq provides the "update"
|
|
operator '|=', which takes a filter on the right-hand side and
|
|
works out the new value for the property being assigned to by running
|
|
the old value through this expression. For instance, .foo |= .+1 will
|
|
build an object with the "foo" field set to the input's "foo" plus 1.
|
|
|
|
This example should show the difference between '=' and '|=':
|
|
|
|
Provide input '{"a": {"b": 10}, "b": 20}' to the programs:
|
|
|
|
.a = .b
|
|
.a |= .b
|
|
|
|
The former will set the "a" field of the input to the "b" field of the
|
|
input, and produce the output {"a": 20}. The latter will set the "a"
|
|
field of the input to the "a" field's "b" field, producing {"a": 10}.
|
|
|
|
- title: "`+=`, `-=`, `*=`, `/=`, `%=`, `//=`"
|
|
body: |
|
|
|
|
jq has a few operators of the form `a op= b`, which are all
|
|
equivalent to `a |= . op b`. So, `+= 1` can be used to increment values.
|
|
|
|
examples:
|
|
- program: .foo += 1
|
|
input: '{"foo": 42}'
|
|
output: ['{"foo": 43}']
|
|
|
|
- title: Complex assignments
|
|
body: |
|
|
Lots more things are allowed on the left-hand side of a jq assignment
|
|
than in most langauges. We've already seen simple field accesses on
|
|
the left hand side, and it's no surprise that array accesses work just
|
|
as well:
|
|
|
|
.posts[0].title = "JQ Manual"
|
|
|
|
What may come as a surprise is that the expression on the left may
|
|
produce multiple results, referring to different points in the input
|
|
document:
|
|
|
|
.posts[].comments |= . + ["this is great"]
|
|
|
|
That example appends the string "this is great" to the "comments"
|
|
array of each post in the input (where the input is an object with a
|
|
field "posts" which is an array of posts).
|
|
|
|
When jq encounters an assignment like 'a = b', it records the "path"
|
|
taken to select a part of the input document while executing a. This
|
|
path is then used to find which part of the input to change while
|
|
executing the assignment. Any filter may be used on the
|
|
left-hand side of an equals - whichever paths it selects from the
|
|
input will be where the assignment is performed.
|
|
|
|
This is a very powerful operation. Suppose we wanted to add a comment
|
|
to blog posts, using the same "blog" input above. This time, we only
|
|
want to comment on the posts written by "stedolan". We can find those
|
|
posts using the "select" function described earlier:
|
|
|
|
.posts[] | select(.author == "stedolan")
|
|
|
|
The paths provided by this operation point to each of the posts that
|
|
"stedolan" wrote, and we can comment on each of them in the same way
|
|
that we did before:
|
|
|
|
(.posts[] | select(.author == "stedolan") | .comments) |=
|
|
. + ["terrible."]
|