Commit Graph

1667 Commits

Author SHA1 Message Date
Emanuele Torre 71c2ab509a Merge pull request from GHSA-686w-5m7m-54vc
decNumberToString calls for a buffer that can hold a string of digits+14
characters, not a buffer of size digits+14.
We need to allocate an extra byte for the NUL byte.

-10E-1000010001, for example, will be stringified as -1.0E-1000010000
and decNumberToString will currently write an extra NUL byte after the
allocated buffer in the heap.

Originally reported by @SEU-SSL on GitHub.

Ref: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=64574

Fixes GHSA-686w-5m7m-54vc
jq-1.7.1
2023-12-13 20:20:22 +01:00
Emanuele Torre c9a5156521 Merge pull request from GHSA-7hmr-442f-qc8j
The unit allocated for decNumberCompare was accidentally removed by
commit 680baeffeb (PR #2804)

This caused a stack overflow when comparing a nan with a payload of 1000
or more.

This bug was found by OSS-fuzz.
Ref: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=64771
Fixes GHSA-7hmr-442f-qc8j

It also fixes 1e999999999 > 1e-1147483646   triggering UBSAN errors
Fixes #2968
2023-12-13 20:17:17 +01:00
Emanuele Torre c5fd64ba97 jv_parse: let decNumberFromString/strtod parse complex nans as a NaN
Before this patch (when using decNumber), "Nan123" was parsed as a NaN,
only if the first n was uppercase.
2023-12-13 20:14:31 +01:00
Emanuele Torre dcf9a8e580 NEWS.md: fix markdown syntax; mention more bug fixs; list cves (#2987) 2023-12-13 18:39:24 +01:00
Mattias Wadman c3b14967d3 Update NEWS.md with changes since 1.7 (#2948) 2023-12-13 15:38:59 +00:00
Emanuele Torre 5d95791a67 lexer: temporarily revert #\ patch; keep CR in comment bug fix
This commit temporarily reverts the commit that allows #\ "tcl-style"
comments everywhere and documents them, for the 1.7.1 patch release
cca1f7d18f.

\r is removed from the list of characters not allowed in a comment to
preserve that bugfix.
2023-12-13 16:38:18 +01:00
Emanuele Torre 511d50b15b ltrimstr/1+rtrimstr/1: don't leak on invalid input or arguments
ltrimstr/rtrimstr was ignoring and leaking the error returned by
f_startswith()/f_endswith().

This also means that they just let the input pass through for non-string
inputs or arguments.

Only fix the leak for now; in the next release, #2969 will make them
rethrow the error returned by startswith/endswith.

Ref: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=64946
2023-12-13 15:13:46 +01:00
Emanuele Torre 7298972551 Makefile.am: don't use -lshlwapi to build jq on WIN32
It does not need it, only libjq needs this.
2023-12-13 13:08:57 +01:00
Emanuele Torre 19453a60cf ci.yml: use sysctl -n hw.logicalcpu instead of nproc on macos
The "macos (arm64)" runner on github does not have nproc.

/Users/runner/work/_temp/f44f0d9f-19eb-4a23-860e-26533d7efdfa.sh: line 10: nproc: command not found
2023-12-11 22:02:42 +01:00
Emanuele Torre 527c5abab1 jv_mem_calloc(): always call with (nmemb, size)
It does not matter much since they most likely just get multiplied
together, but some compilers would complain about this if these were
calls to calloc.
2023-12-11 20:47:32 +01:00
Emanuele Torre c2db51eeb3 shtest+configure: remove uses of non-portable/non-standard commands
Replace deprecated test(1) parentheses and -a logical operator with two
tests command.

Replace deprecated  tail -1  with  tail -n1.

Replace non-standard egrep(1) command with  grep -E  ; this also
prevents obsolescence warnings on GNU systems.
2023-12-11 20:46:54 +01:00
Emanuele Torre 507eb8283c shtest: fix out-of-source tests
The locale test was using ./jq intead of $JQ.

I also removed the use of obsolete egrep instead of grep -E that
triggers warnings on GNU systems, and the use of deprecated head -1
instead of head -n1.

Also removed the unnecessary hiding of strptime/1 errors with
? // false.
2023-12-11 20:45:14 +01:00
Jesper Rønn-Jensen 2ccab044fe chore: Improve readability (for Alternative operator '//')
The description of the Alternative operator `//` was hard for me to grasp in its wording.

I suggest dividing the looong sentence into two parts. Since it is actually an alternative formulation, the sentence can be divided into the first formulation, and the alternative formulation.
2023-12-11 11:38:54 +00:00
Emanuele Torre 7e54d9646e jq.test: fix setpath PR number
It was #2970, not 2967.
2023-12-11 09:58:09 +01:00
dependabot[bot] 22f81401f8 Bump actions/setup-python from 4 to 5
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4 to 5.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-11 07:54:23 +01:00
Emanuele Torre 6b69ffe769 jv_setpath()+setpath/2: don't leak the input after an invalid get
Ref: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=64906
2023-12-10 19:40:29 +01:00
Emanuele Torre 80bc768c5d jq_fuzz_fixed: fix test cases that use string interpolation
"\(" is implementation-defined in C++, and both clang and gcc treat it
as equivalent to "(", not "\\(".
This patch replaces "\(" with "\\(" so that tests with string
interpolation actually use string interpolation.
2023-12-10 15:22:03 +01:00
David Korczynski 5029328d35 tests: add new fuzzer
Signed-off-by: David Korczynski <david@adalogics.com>
2023-12-07 12:57:54 +01:00
DavidKorczynski 20a403e97e jq_fuzz_execute.cpp: fix false assert
Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=64632 by following the suggestion here: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=64632#c2 -- the issue is fixed when this attribute is set.

I did not add any of the other suggestions in the comment: we haven't run into these issues in the fuzzer so far.
2023-12-03 17:54:25 +01:00
DavidKorczynski 693a7261e5 jq_fuzz_execute: fix memory leak
Signed-off-by: David Korczynski <david@adalogics.com>
2023-12-03 03:53:49 +01:00
David Korczynski bfb7fd570f jq_fuzz_execute: cleanup un-needed extern
Signed-off-by: David Korczynski <david@adalogics.com>
2023-11-30 14:40:36 +01:00
David Korczynski 252ab244ce Add fuzzer targeting jq_next
Signed-off-by: David Korczynski <david@adalogics.com>
2023-11-30 14:40:36 +01:00
David Korczynski 13353515bd jq_fuzz_compile: dump disassembly
Signed-off-by: David Korczynski <david@adalogics.com>
2023-11-30 14:40:36 +01:00
Mattias Wadman 98a206964d Convert decnum to binary64 (double) instead of decimal64
This is what the JSON spec suggests and will also be less confusing compared to other jq implementations and langauges.

Related to #2939
2023-11-29 09:36:33 +01:00
Emanuele Torre 1617091033 website: use https URLs instead of http URLs in download page
Also add markdown formatting for decNumber URL so it gets rendered as a
link in the html page.
2023-11-29 09:35:36 +01:00
itchyny d14393f552 Keep releasing executables with legacy names 2023-11-28 22:51:20 +01:00
Tyler Rockwood 6b82dd3c47 libjq: extern C for C++
If using libjq from C++ it would be nice to not need to do this at the
import site, so just extern "C" to the public headers for libjq

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
2023-11-28 22:50:25 +01:00
Emanuele Torre 460a5c12b4 jq_fuzz_parse_extended.c: don't jv_free() twice
jv_dump() frees its argument.

I missed this problem before merging #2952, whoops! =)

fixup from eb3b5654bb
2023-11-28 20:51:07 +01:00
David Korczynski 4570c23720 jv_parse: refactor jv_parse_sized
Signed-off-by: David Korczynski <david@adalogics.com>
2023-11-28 20:36:59 +01:00
David Korczynski eb3b5654bb test: add jv_dump to extended fuzzer
Signed-off-by: David Korczynski <david@adalogics.com>
2023-11-28 20:36:59 +01:00
David Korczynski 44300e4310 Extend fuzzing set up
Adds a parse function ins `jv_parse.c` that enables parsing using custom
flags for the parser. This is then used by two fuzzers added as well.

This is to make sure fuzzing hits various code parts currently not
fuzzed, e.g. `stream_token`:
https://storage.googleapis.com/oss-fuzz-coverage/jq/reports/20231125/linux/src/jq/src/jv_parse.c.html#L241

Signed-off-by: David Korczynski <david@adalogics.com>
2023-11-28 20:36:59 +01:00
Andreas Heiduk 88f01a741c simplify paths/0 and paths/1
`recurse/0` already handles traversing objects and arrays, so it is more
consistent to use that.
For `paths/1` it is easier to use the actual value returned by
`recurse` instead of querying that value with `getpath/1` afterwards.
2023-11-16 21:14:58 +01:00
Emanuele Torre 6c035133e8 README.md: tosed => to sed (#2944) 2023-11-06 23:59:22 +09:00
Emanuele Torre cca1f7d18f Comment bug fixes, and fully support Tcl-style multiline comments
* bugfix: comments were incorrectly being terminated by CR; for example
    jq -n $'1 #foo\r'
  fails to compile because the CR character terminates the comment, and
  CR is not a valid character in jq syntax.

* improvement: comments fully support Tcl-style line continuation.
  Previously this was only "supported" in `-f' scripts, whose first line
  starts with "#!", and second line starts with # and ends with \, only
  for the comment on the second line, only for one extra line.

* man: document comment syntax, which was previously undocumented.

* tests: add regression tests for the bugfix, and some tests for line
  continuation in comments.
2023-11-03 15:52:02 -05:00
Emanuele Torre f4929f3c19 src/builtin.c: remove unnecessary jv_copy-s in type_error/type_error2
Also fix doubled semicolon   jv ret;;   =>   jv ret;
2023-10-30 22:12:27 -05:00
Emanuele Torre d5590f90da jvp_object_contains: remove unnecessary jv_copy 2023-10-30 22:12:27 -05:00
Emanuele Torre e85e358233 Fix possible uninitialised value dereference if jq_init() fails
If jq_init() fails, goto out would try to free input_state which is
uninitialised. I initialised input_state to NULL to fix the problem.

Ref: https://github.com/jqlang/jq/pull/2934#discussion_r1367795641

Reported-By: Klemens Nanni <kn@openbsd.org>
2023-10-22 10:51:24 -05:00
Klemens Nanni 7ab117a483 Defer heap variable initialisation after pledge
Otherwise `AGRS` and `program_arguments` remain allocated/unfreed in the
early (extremely unlikely) pledge(2) failure case.

Move their allocation before jq_init(), the first case of jumping to
`out` where they are cleaned up, where it also seems to logically fit
better than above between function entry, locale setup and OpenBSD
specific pledge.
2023-10-21 23:30:36 -05:00
Klemens Nanni 77dcaf3fdc Remove unused mkstemp()
b82c231 "Remove -i option (#704)" removed its last usage in 2015.

Spotted while looking for code could potentially write/create/modify files.
2023-10-21 23:30:36 -05:00
Klemens Nanni f1bfd0c518 Restrict systems operations on OpenBSD
Use pledge(2)[0] to limit jq(1) to reading files.
It does not change files and only writes to standard output/error.
It never deals with TTY, network, process management or other subsystems.

This is to reduce jq's attack surface and potential damage.

OpenBSD is carrying a local patch[1] in its official jq port/package
since 2016.  An improved version:

- drop no longer needed "getpw" promise
  f1c4947 "Avoid getpwuid for static linking" removed getpwuid(3) usage
- pledge before jq_init() to simplify the error path
- use perror(3) to print errno(2)

No behaviour change in tests or real world usage observed on
OpenBSD/amd64 7.4.

0: https://man.openbsd.org/pledge.2
1: https://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/textproc/jq/patches/patch-main_c
2023-10-21 23:30:36 -05:00
Emanuele Torre 9de0e26ce6 Generate links in man page
Some [foo](bar) links were added to manual.yml without updating
build_manpage.py to make it generate roff for "a" tags.

Fixes #2930
2023-10-12 15:32:23 -05:00
Gaelan Steele c10cbbff34 Remove undefined behavior caught by LLVM 10 UBSAN. 2023-10-08 12:32:22 -05:00
Emanuele Torre 6b6feaffbb Revert "Revert "od -c => od -tc: od -c is an XSI extension..."
This reverts commit 0e70f7a57e.

There is no reason to revert this change.

In #2922, I only disagreed with the commit message suggesting that
  LC_CTYPE=C od -t c    is   equivalent to   od -c

The only documented differences are that -tc is required to be
influenced by -N and -j, while -c is not, and that -c is required to
only support a subset of the backslash sequences that -tc should
support.
2023-10-05 15:37:44 +02:00
Nicolas Williams 0e70f7a57e Revert "od -c => od -tc: od -c is an XSI extension equivalent to LC_CTYPE=C od -tc and not universally available"
This reverts commit 0bce9fb8ed.
2023-10-04 22:28:24 -05:00
наб 0bce9fb8ed od -c => od -tc: od -c is an XSI extension equivalent to LC_CTYPE=C od -tc and not universally available 2023-10-04 17:46:13 -05:00
Emanuele Torre 4ebd21e1eb Allow passing the inline jq script before --
jq previously only allowed passing the inline script before -- (as if
they were options) even though one would expect the inline script to be
a positional argument.

Since jq previously also refused to run with a usage error if the script
was passed after -- (It was not assuming  .  as script as it does when
no arguments are passed), and positional arguments are allowed before --
and even before other options, it should not be a breaking change to
change that weird behaviour, and allow the script to appear after --.

It also simplifies the option parsing code a bunch.

Fixes #2918
2023-10-04 00:00:46 -05:00
Thomas Bozeman th026106 7f547827e4 Simplify pick example
Old pick example included input array in command line, making `input` confusing
and redundant.
2023-10-03 22:48:12 +02:00
Emanuele Torre 6b5a18f036 Actually use number correctly casted from double to int as index
The code was using (int)jv_number_value(k) instead of (int)didx.

follow-up from 0e067ef936

Useless assignments to didx detected by clang-tidy.
2023-09-27 22:54:23 -05:00
Emanuele Torre 8206bc8fd2 main.c: Remove unused EXIT_STATUS_EXACT option
In process there is a suspicious  options |= EXIT_STATUS_EXACT  that
is run when the jq script is terminated by halt, or halt_error.

That line of code acutally does nothing because options is a local
argument variable, and is not passed as a pointer. It was probably meant
to be a   *options |= EXIT_STATUS_EXACT   with the options argument
passed as a int*.

In any case, we do not want to run the code in main() that was supposed
to run if EXIT_STATUS_EXACT is set (but didn't since it is never added
to options); as far as I can tell, we only want to run that code when
the --exit-status/-e option is passed.

So I removed EXIT_STATUS_EXACT completely, and the useless assignment,
instead of fixing it since it was not used for anything else.

Useless assignment detected by clang-tidy.
2023-09-27 21:52:23 -05:00
Emanuele Torre 623ace27e8 Remove a bunch of unused variables, and useless assignments
Detected by clang-tidy.
2023-09-27 21:50:44 -05:00