PKTC Config Hash about docsis HOT 24 OPEN

Hi @agentjdh,

I do not have the same device you have. Let me explain what is going on in the code:

That magical string you see in memcpy() is a TLV (Tag-Length-Value). For CableLabs, this is TLV 11, which is just a normal SNMP Object embedded in a config file.

So in order to do your new hash, you need to start with, followed by the total length of the TLV, which is the OID and the hash itself plus some glue bytes here and there. As you say, the encoded OID is
06 0C 01 03 06 01 02 01 81 0C 01 02 0B 00 ( but that is not ASN1 as you will see). The suffix of 0x04 0x20 is just to tell that the following field is bitsring, length of 20 bytes, which is the length of a SHA1 hash.

So if we put this all together, your magical string should be:

0x0b - the TLV type,
0x25 - the full length of everything following: 2+35=37=0x25
0x30 - The following stuff is an OID
0x23 - The full length of the OID and its value: 1+1+11+1+1+20 = 35 = 0x23
0x06 - This indicates that the following is an OID
0x0b - Length of the OID
0x2b 0x06 0x01 0x02 0x01 0x81 0x0C 0x01 0x02 0x0B 0x00 - ( encoded OID, see below) **
0x04 - What comes next is a bitstring
0x14 - That's 20 meaning that there are 20 bytes following because this is the length of a SHA1 hash

So the total number of bytes we have here is: 1+1+1+1+1+1+11+1+1=19

Se we can add a new hash, and the code should look like this:

if (hash == 3) 
 {
    memcpy (tlvbuf + tlvbuflen - 3, "\x0b\x25\x30\x23\x06\x0b\0x2b\0x06\0x01\0x02\0x01\0x81 \0x0C\0x01\0x02\0x0B\0x00\x04\x14", 19);
    tlvbuflen += 16;
  }

Now you will need to add the following code in main() in docsis.c to let you use this "special" encoding just after the "-eu" hash, so around line 358 in docsis.c

if (!strcmp (argv[0], "-spechash")) {
      if (hash) {
        usage();
      }
      hash = 3;
      continue;
    }

Then just run the same command with -spechash to try this out...

So if you make these changes, you should get closer to get this to work. I might be off in the lengths ( I have a problem adding 1 to things for some reason).

I don't have time to compile this right now, so let me know how this goes.

Cheers...

*Encoded OID...
For the encoded OID, it's not going to be what that site gave you. In their clever way, the ASN1/SNMP folks say that an OID is encoded by replacing the first 2 bytes with the following 40 Byte1 + Byte 2. So in this case, 40*1+3, that is 43, or 0x2b. Which means that your OID becomes:
0x2b 0x06 0x01 0x02 0x01 0x81 0x0C 0x01 0x02 0x0B 0x00

from docsis.

@agentjdh,

In the previous code, I inserted a space in the memcopy, please remove that in your code. This was a formatting mistake:

memcpy (tlvbuf + tlvbuflen - 3, "\x0b\x25\x30\x23\x06\x0b\0x2b\0x06\0x01\0x02\0x01\0x81\0x0C\0x01\0x02\0x0B\0x00\x04\x14", 19);

Thanks

from docsis.

@stephb9959

I'd come to the exact same numbers effectively from using xxd to take a hex dump of a working binary. So this part looks correct.

The actual Docsis tool, with your modifications, compiles OK, but using the -spechash option when converting the text to binary does not insert the Hash entry.

I got it (sort of) working by just modifying the existing -eu entry (hash ==2) - it works, but the generated binary is 1 byte larger than a working one, and the modem rejects the generated MTA file.

The working ones are created on a Windows PC using Thomson ConfigBldr.

from docsis.

from docsis.

mta_WORKING_999999.bin.gz
mta_NOT_WORKING.bin.gz

from docsis.

Hi @agentjdh,

I built a little app to parse the files in erlang and here is the output:

Values not found but int good file:
>>> OID:".1.3.6.1.2.1.140.1.2.11.0" VALUE:<<4,20,251,207,112,173,178,112,146,
                                            205,125,193,57,6,115,73,213,223,77,
                                            198,13,230>>
>>> OID:".1.3.6.1.4.1.4413.2.2.2.1.6.1.7.0" VALUE:<<2,1,2>>
>>> OID:".1.3.6.1.2.1.140.1.1.6.0" VALUE:<<2,1,1>>


Values not in good file but in bad:
>>> OID:".1.3.6.1.4.1.4491.2.2.1.1.1.7.0" VALUE:<<2,1,1>>
>>> OID:".1.3.6.1.4.1.406.20.2.10.1.2.0" VALUE:<<2,1,2>>
>>> OID:".1.3.6.1.2.1.140.1.2.11.0" VALUE:<<4,20,130,248,25,233,168,226,50,118,
                                            72,209,166,83,160,17,75,91,122,4,
                                            156,204>>

This shows that you have 2 mismatches OIDs. Forget the ".1.3.6.1.2.1.140.1.2.11.0" because that is the has and it should not match because the files do not match.

The top shows you what the good file has that the bad file does not. And the bottom shows you what the bad file contains that is not in the good file.

So I think you have OID problems.

Both files passed the MTA decoder so they are both valid.

Hope this helps.

from docsis.

Here is the proper docsis.c file that compiles and will do the spechash and still lets you pick from NA/EU too (I had forgotten one more location to enable to hash being appended, it's in now)...

docsis.zip

Let me know how it goes.

from docsis.

@stephb9959

I've replaced docsis.c with the one you provided and re-compiled. Looks like something is going wrong - generating a binary using

./docsis -dialplan -spechash -p mta_WORKING.txt mta_WORKING.bin

SnmpMibObject pktcMtaDevProvConfigHash.0 HexString 0x58df8e779cda95ca9a7ae0cb1589e775532ca691;
/* PC20 dialplan found, dialplan.txt file created. */

Error: bad type returned (31)
SnmpMibObject zeroDotZero.120.50.98.0.120.48.54.0.120.48 ��txDV Variable has bad type;
MtaConfigDelimiter 255;
}

Running the same command with -nohash or -eu switches works OK (but the modem rejects the binary as there is no/wrong has OID used).

from docsis.

I don't know how that code section got in above, ignore it.

from docsis.

@agentjdh

OK. I have it now. This was my mistake. Some debug stuff I added a while ago in docsis.c, now removed. Also generated some test to make sure the spechash could be encoded and decoded. Now that works. Also, my encoding was wrong with the memcpy(). Now fixed. ( It's amazing what happens when you actually compile and run tests! Who knew!!)

Let me know if this finally fixes your problem. If it does, then I can submit for a pull request.

This is the good file...

docsis.zip

Thanks

from docsis.

@stephb9959

I am not getting those errors now, but the generated file is still +1 byte, and is rejected by the modem.
files.tar.gz

I've attached the files, here is what I'm doing -

Decode the working binary (8480 bytes, not the TEST one), which generates the text configuration + dialplan text file

./docsis -d mta-working-2-lines--128_021.bin > mta-working-2-lines--128_021-TEST.txt

Encode using the following command line -

./docsis -dialplan -spechash -p mta-working-2-lines--128_021-TEST.txt mta-working-2-lines--128_021-TEST.bin

Resulting binary is 8481 and is rejected for -

error - 4000950906 (table linkages in the configuration file could not be resolved)
I get this twice, one for each line

Resulting in

critical - 4000950904 (mandatory parameter of the configuration file is missing)

The dialplan.txt generated by decoding working vs non-working is identical.

So is there something in the diaplan.txt file that is causing this? It could probably be simplified for a start, which I will work on this morning.

from docsis.

@agentjdh

There is a 1 byte difference but it's actually not that 5 bytes. The resulting file has 1 byte size difference but it's because the 4 mismatch OIDs have different sizes. You have 2 OIDs in the good file that are not in the non-working file, and you have 2 OIDs in the test file that are not in the good file. Here's that little analysis tool I ran on the working and non-working files:

Values in mta-working-2-lines--128_021.bin not in mta-working-2-lines--128_021-TEST.bin:
>>> OID:".1.3.6.1.2.1.140.1.2.11.0" VALUE:<<4,20,88,223,142,119,156,218,149,
                                            202,154,122,224,203,21,137,231,117,
                                            83,44,166,145>>
>>> OID:".1.3.6.1.4.1.4413.2.2.2.1.6.1.7.0" VALUE:<<2,1,2>>
>>> OID:".1.3.6.1.2.1.140.1.1.6.0" VALUE:<<2,1,1>>


Values in "mta-working-2-lines--128_021-TEST.bin" not in "mta-working-2-lines--128_021.bin":
>>> OID:".1.3.6.1.4.1.4491.2.2.1.1.1.7.0" VALUE:<<2,1,1>>
>>> OID:".1.3.6.1.4.1.406.20.2.10.1.2.0" VALUE:<<2,1,2>>
>>> OID:".1.3.6.1.2.1.140.1.2.11.0" VALUE:<<4,20,7,248,124,28,236,178,235,107,
                                            171,148,80,207,219,10,75,158,100,
                                            111,187,31>>

So I decided to run my own tests with the docsis.c file I gave you. All based on your working mta-working-2-lines--128_021.bin

docsis -d mta-working-2-lines--128_021.bin > good-bin.txt

Then I remove the last line at the end of good-bin.txt, "SnmpMibObject mib-2.140.1.2.11.0...". That's the hash OID and we do not need it. Then I ran:

docsis -dialplan -spechash -p good-bin.txt my-good-bin.bin

This gives me a file of 8480 bytes.

When I run my own tool to compare the 2 bins, I get something like this:

Values in mta-working-2-lines--128_021.bin not in my-good-bin.bin:
>>> OID:".1.3.6.1.2.1.140.1.2.11.0" VALUE:<<4,20,88,223,142,119,156,218,149,
                                            202,154,122,224,203,21,137,231,117,
                                            83,44,166,145>>


Values in "my-good-bin.bin" not in "mta-working-2-lines--128_021.bin":
>>> OID:".1.3.6.1.2.1.140.1.2.11.0" VALUE:<<4,20,157,12,6,164,190,146,223,49,
                                            23,4,12,252,156,39,95,180,174,203,
                                            255,240>>

Which tells me that the only difference is the hash. This is normal because the good file has the dialplan in the middle of the file, which the docsis tool puts the dialplan at the end of the file.

I am attaching the file I produced this way. Maybe you can try it and tell me if it works?

my-good-bin.zip

This is the good-bin.txt file I get too. Compare yours to this just to make sure that your setup is ok (i.e. you are not missing MIBs or you have old MIBS that would cause a problem.
good-bin-txt.zip

Other than that, this shows that the right has is getting in the file and that the docsis decode and reencodes to the same file.

Let me know how this works.

Thanks

from docsis.

@stephb9959

What you did there is exactly what I was doing - decoding a working binary to text + dialplan, deleting the hash line from the text file, then rebuilding the binary with exactly those switches. And the resulting binary did not match the original.

So where are these mismatched OIDs coming from? I am not editing the file apart from Hash removal.

Were you doing this with a release source code? I was using binary-master download, and replacing only docsis.c

I'll clean out the sources and start again.

from docsis.

from docsis.

@agentjdh

Just clone the project to get the master and replace with the new DOCSIS.c

from docsis.

@stephb9959

I've done exactly that, with the same results.

The bizarre thing is, once I have encoded the binary that is +1 byte ("wrong"), if I then decode this to a text file, and delete the hash, the resulting text file is identical to the one decode from the working binary.

I have a lot of mib files so I am never entirely sure that I have the correct ones.

Could you attach the mibs you have and/or your "working" binary?

from docsis.

from docsis.

Debian please

from docsis.

Here is a Debian build for you...

debian-docsis-buster.zip

from docsis.

@stephb9959

Thanks Stéphane, it looks like the file created is the same size (8481 bytes) as before. So I guess it is not the binary then.

Which pretty much leaves my mibs - I have a 3rd party mib folder, .snmp/mibs, with a lot of mibs I have collected over the last year or so.

If I move this folder, so the docsis application only uses the default installed mibs in /usr/share/snmp/mibs, I get another slightly different binary created this time 8483 bytes.

I've updated the system installed mibs with download-mibs. If I diff the text files decoded from the binaries it shows

7c7
< 	SnmpMibObject .1.3.6.1.4.1.4491.2.2.1.1.1.7.0 Integer 1;
---
> 	SnmpMibObject .1.3.6.1.2.1.140.1.1.6.0 Integer 1;

So something is still getting mixed up.

SnmpMibObject .1.3.6.1.4.1.4491.2.2.1.1.1.7.0 Integer 1;

Is not in the original binary - so how is it being inserted into the binary created by docsis tool?

Actually, it looks like it might be modifying an IETC MIB (which works) into the equivalent Cable Labs MIB - which is what I've been trying to avoid with the hash MIB.

I'll spend some more time on this tomorrow. Thanks for your help to date.

from docsis.

Turns out it was the decoding to text file that was using the "wrong" OID for pktcMtaDevEnabled

The particular modem I was using required the 1.3.6.1.2.1 OID from PKTC-IETF-MTA-MIB

It also obviously required the IETF hash OID as well, which is what triggered this whole scenario.

So just getting rid of the various MIBs - or using the -o option during decode the OIDs numerically worked.

Anyway working binaries are now generated from the decoded text output.

from docsis.

@agentjdh

That's very good news. I was just about to send you my MIBs...

So if you use the new executable with the -spechash all things work for you now?

Thanks

from docsis.

Yes, -spechash works properly. It is IETF spec, not NA or EU, to my knowledge.

This is the first device I have seen that uses IETF mibs.

from docsis.

In my case recompiling with -spechash or -eu it give me the same hexstring value
and normaly MTA reject it since it accepts only IETF hashed.

Installation was made from your site and also replaced docsis.c as you provide on the link.
-rw-r--r-- 1 root root 21572 Apr 30 13:04 docsis-backup.c
-rw-r--r-- 1 root root 22016 Apr 30 13:25 docsis.c

First I'm decoding the working one
docsis -o -d K_IP.BIN > K_IP.txt

Removing the last line from K_IP.txt and encoding again
docsis -o dialplan -spechash -p K_IP.txt K_IP_spechash.bin

Hexstring(1.3.6.1.2.1.140.1.2.11.0) are not matched from original one (K_IP.BIN_) and compiled one (K_IP_spechash.bin) and in this case config is rejected .

Tried with EU encoding i get same value for hexstring ( 1.3.6.1.4.1.7432.1.1.2.9.0 )
docsis -o -dialplan -eu -p K_IP.txt K_IP_eu.bin

I dont think i should get same encoding values with eu and spechash which was supposed to be IETF .

Any idea what is wrong with my compiler would be very helpful.

files1.tar.gz

from docsis.