jq and sed are throwing warnings during support_parse.sh tech_check

• No such file
• Invalid file content

For example:

➜  42993 git:(master) ✗ support_parse.sh tech_check puppet_enterprise_support_42993_lxpr0721pv_20210202135301  > tech_check.puppet_enterprise_support_42993_lxpr0721pv_20210202135301.json 
jq: Bad JSON in --slurpfile infra puppet_enterprise_support_42993_lxpr0721pv_20210202135301/enterprise/pe_infra_status.json: Could not open puppet_enterprise_support_42993_lxpr0721pv_20210202135301/enterprise/pe_infra_status.json: No such file or directory
jq: Bad JSON in --slurpfile modules puppet_enterprise_support_42993_lxpr0721pv_20210202135301/enterprise/modules.json: Invalid numeric literal at line 1, column 10
sed: puppet_enterprise_support_42993_lxpr0721pv_20210202135301/enterprise/thundering_herd_query.txt: No such file or directory
sed: puppet_enterprise_support_42993_lxpr0721pv_20210202135301/enterprise/thundering_herd_query.txt: No such file or directory
sed: puppet_enterprise_support_42993_lxpr0721pv_20210202135301/enterprise/thundering_herd_query.txt: No such file or directory
sed: puppet_enterprise_support_42993_lxpr0721pv_20210202135301/enterprise/thundering_herd_query.txt: No such file or directory

Further investigation into the invalid modules.json reveals:

• invalid json
• an error message "Forbidden request: /puppet/v3/environment_modules (method :get). Please see the server logs for details."

➜  42993 git:(master) ✗ cat ./puppet_enterprise_support_42993_lxpr0721pv_20210202135301/enterprise/modules.json
Forbidden request: /puppet/v3/environment_modules (method :get). Please see the server logs for details.
➜  42993 git:(master) ✗ 

The puppetserver-access.log contains the following, which may or may not be related

10.188.25.23 - - [02/Feb/2021:13:48:36 +0100] "GET /puppet/v3/environment_modules HTTP/1.1" 200 47485 "-" "Apache-HttpAsyncClient/4.1.4 (Java/11.0.8-internal)" 2063 - 2002

The following error occurs when running against the exploded archive of the support script:

➜  42993 git:(master) ✗ support_parse.sh tech_check puppet_enterprise_support_42993_lxpr0728pv_20210202135239
zcat: can't stat: puppet_enterprise_support_42993_lxpr0728pv_20210202135239/logs/messages.gz (puppet_enterprise_support_42993_lxpr0728pv_20210202135239/logs/messages.gz.Z): No such file or directory
{
  "Server Version": "2019.8.1",
  "Infrastructure": [
    {

The file exists and running zcat on it produces the above error:

➜  42993 git:(master) ✗ ls -lrth ./puppet_enterprise_support_42993_lxpr0721pv_20210202135301/logs/messages.gz
-rw-r-----  1 gavin.didrichsen  staff    26K Feb  2 12:50 ./puppet_enterprise_support_42993_lxpr0721pv_20210202135301/logs/messages.gz
➜  42993 git:(master) ✗ zcat ./puppet_enterprise_support_42993_lxpr0721pv_20210202135301/logs/messages.gz
zcat: can't stat: ./puppet_enterprise_support_42993_lxpr0721pv_20210202135301/logs/messages.gz (./puppet_enterprise_support_42993_lxpr0721pv_20210202135301/logs/messages.gz.Z): No such file or directory
➜  42993 git:(master) ✗ 

As per Charlie Sharpsteen's suggestion on Mac changing zcat to gunzip -c makes the parse work:

diff --git a/bin/tech_check_parse.sh b/bin/tech_check_parse.sh
index 1a84be8..67cc836 100644
--- a/bin/tech_check_parse.sh
+++ b/bin/tech_check_parse.sh
@@ -24,7 +24,7 @@ _messages_parse() {
   # Would be nice if zcat didn't care if the file were compressed or not, like zgrep
   # Be more like zgrep, zcat
   if [[ $_messages_gz ]]; then
-    zcat "$_messages_gz" | gawk 'BEGIN{IGNORECASE=1} $5 ~ "puppet" { err = substr($0, index($0, $6)); if (err ~ /(error|failure|severe|exception)/) print err }'
+    gunzip -c "$_messages_gz" | gawk 'BEGIN{IGNORECASE=1} $5 ~ "puppet" { err = substr($0, index($0, $6)); if (err ~ /(error|failure|severe|exception)/) print err }'
   elif [[ $_messages ]]; then
     gawk 'BEGIN{IGNORECASE=1} $5 ~ "puppet" { err = substr($0, index($0, $6)); if (err ~ /(error|fail|severe|exception)/) print err }' "$_messages"
   else

I noticed the following error on a tech_check script run:

joel:puppet_enterprise_support__prodpuppet11lsat_20191101152122 joel.weierman$ ~/support-tooling-master/support_parse.sh tech_check ~/Downloads/puppet_enterprise_support__prodpuppet1**********.tar.gz

jq: Bad JSON in --slurpfile infra /var/folders/m4/bd2kkxy12g98h7wbn_p2dfr00000gq/T/tmp.I8bbizDJ/enterprise/pe_infra_status.json: Could not open /var/folders/m4/bd2kkxy12g98h7wbn_p2dfr00000gq/T/tmp.I8bbizDJ/enterprise/pe_infra_status.json: No such file or directory
awk: extra ] at source line 1
context is
$3 ~ /^-/ { m = match($11, >>> //[^/] <<<
awk: nonterminated character class /[^] at source line 1
context is
$3 ~ /^-/ { m = match($11, >>> //[^/] <<<
awk: nonterminated character class /[^
source line number 1
sed: /var/folders/m4/bd2kkxy12g98h7wbn_p2dfr00000gq/T/tmp.I8bbizDJ/enterprise/thundering_herd_query.txt: No such file or directory
sed: /var/folders/m4/bd2kkxy12g98h7wbn_p2dfr00000gq/T/tmp.I8bbizDJ/enterprise/thundering_herd_query.txt: No such file or directory
sed: /var/folders/m4/bd2kkxy12g98h7wbn_p2dfr00000gq/T/tmp.I8bbizDJ/enterprise/thundering_herd_query.txt: No such file or directory
sed: /var/folders/m4/bd2kkxy12g98h7wbn_p2dfr00000gq/T/tmp.I8bbizDJ/enterprise/thundering_herd_query.txt: No such file or directory
zcat: can't stat: /var/folders/m4/bd2kkxy12g98h7wbn_p2dfr00000gq/T/tmp.I8bbizDJ/logs/messages.gz (/var/folders/m4/bd2kkxy12g98h7wbn_p2dfr00000gq/T/tmp.I8bbizDJ/logs/messages.gz.Z): No such file or directory

The script does to appear to have completed with JSON output.

---- Metadata.json-----
more metadata.json
{
"version": "2.7.0",
"ticket": "",
"timestamp": "20191101152122"

When I first run this command convert.py:

support-tooling git:(master) ./support_parse.sh modules ~/Downloads/puppet_enterprise_support_42544_int-puppetmaster-02_20201229131528.tar.gz | ./convert.py
Traceback (most recent call last):
  File "./convert.py", line 3, in <module>
    from json2html import *
ImportError: No module named json2html
couldn't create json

sudo pip install json2html can resolve the issue, this should be mentioned as a requirement.

The above command will not work. Need to save the support_parse.sh output to a json file and then feeding it to convert.py. It is working the way as documented.