base64 gzip compression support for json_app_get (#14169)

* add lnms_return_optimizer

* add compression test using zfs-v1

* minor style fix

* save the original output if not json

* replace gzinflate with gzdecode as apparently that does not require yanking the header

* Minor comment cleanup. Also note it in the application notes as well.

* update docs on how it is called

* update the spelling of it in a few places

* and a few more

* dev docs updated a bit

* the suricata extend has native support for this now

* add exception handling for base64 and gzip decoding failure

* minor cleanup for new exceptions

* minor misc changes

* minor formatting fix

* more phpdoc tweaks

* minor formatting tweak

* remember to actually include the new exceptions

* more phpdoc tweaking

* correct name in JsonAppGzipDecodeException

* add debug and verbose output

* style fix

* not base64 is it starts with a line with only a integer

doc/Extensions/Applications.md

@@ -51,6 +51,84 @@ like that for proxmox:
 extend proxmox /usr/bin/sudo /usr/local/bin/proxmox
 ```
 
+### JSON Return Optimization Using librenms_return_optimizer
+
+While the json_app_get does allow for more complex and larger data
+to be easily returned by a extend and the data to then be worked
+with, this can also sometimes result in large returns that
+occasionally don't play nice with SNMP on some networks.
+
+`librenms_return_optimizer` fixes this via taking the extend output
+piped to it, gzipping it, and then converting it to base64. The
+later is needed as net-snmp does not play that nice with binary data,
+converting most of the non-printable characters to `.`. This does add
+a bit of additional overhead to the gzipped data, but still tends to
+be result in a return that is usually a third of the size for JSONs
+items.
+
+The change required is fairly simply. So for the portactivity example below...
+
+```
+extend portactivity /etc/snmp/extends/portactivity smtps,http,imap,imaps,postgresql,https,ldap,ldaps,nfsd,syslog-conn,ssh,matrix,gitea
+```
+
+Would become this...
+
+```
+extend portactivity /usr/local/bin/lnms_return_optimizer -- /etc/snmp/extends/portactivity smtps,http,imap,imaps,postgresql,https,ldap,ldaps,nfsd,syslog-conn,ssh,matrix,gitea
+```
+
+The requirements for this are Perl, MIME::Base64, and Gzip::Faster.
+
+Installing on FreeBSD...
+
+```
+pkg install p5-MIME-Base64 p5-Gzip-Faster wget
+wget https://raw.githubusercontent.com/librenms/librenms-agent/master/utils/librenms_return_optimizer -O /usr/local/bin/librenms_return_optimizer
+chmod +x /usr/local/bin/librenms_return_optimizer
+```
+
+Installing on Debian...
+
+```
+apt-get install zlib1g-dev cpanminus wget
+cpanm Gzip::Faster
+cpanm MIME::Base64
+wget https://raw.githubusercontent.com/librenms/librenms-agent/master/utils/librenms_return_optimizer -O /usr/local/bin/librenms_return_optimizer
+chmod +x /usr/local/bin/librenms_return_optimizer
+```
+
+Currently supported applications as are below.
+
+- backupninja
+- certificate
+- chronyd
+- dhcp-stats
+- docker
+- fail2ban
+- fbsd-nfs-client
+- fbsd-nfs-server
+- gpsd
+- mailcow-postfix
+- mdadm
+- ntp-client
+- ntp-server
+- portactivity
+- powerdns
+- powermon
+- puppet-agent
+- pureftpd
+- redis
+- seafile
+- supervisord
+- ups-apcups
+- zfs
+
+The following apps have extends that have native support for this,
+if congiured to do so.
+
+- suricata
+
 ## Enable the application discovery module
 
 1. Edit the device for which you want to add this support
@@ -2361,6 +2439,12 @@ cpanm Suricata::Monitoring
 extend suricata-stats /usr/bin/env PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin suricata_stat_check -c
 ```
 
+Or if you want to use try compressing the return via Base64+GZIP...
+
+```
+extend suricata-stats /usr/bin/env PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin suricata_stat_check -c -b
+```
+
 4. Restart snmpd on your system.
 
 You will want to make sure Suricata is set to output the stats