`jq -s '.[] | unique_by(.program_name)' < program-list.json > sorted_duped.json` about diodb HOT 4 CLOSED

Hi @sickcodes,

I think this is a great idea! I also suggest to:

• To do so on every PR so the program-list.json is always sorted and de-duplicated. For example, we can add a workflow that checks for this.
• To de-duplicate entries based on "policy_url". Now we have quite a lot of duplicates (e.g. for Google VRP) and I can't find a reason why.

from diodb.

• To make a helper script that adds an entry to the program-list.json. Alike the npm init command (in a sense of interactive questionnaire). I think this will be useful for those who contribute for the first time to abstract away formatting issues from them and make contributing experience more smooth. The other way to achieve this is to add a git hook that will format before the commit automatically. I can help with this.

from diodb.

Hi @sickcodes,

I think this is a great idea! I also suggest to:

• To do so on every PR so the program-list.json is always sorted and de-duplicated. For example, we can add a workflow that checks for this.

• To de-duplicate entries based on "policy_url". Now we have quite a lot of duplicates (e.g. for Google VRP) and I can't find a reason why.

Workflow sounds good, but I do like the idea of having users de duplicate it themselves :)

Definitely can't de-dupe based on policy URL as multiple companies will have the same security "drop off point," and all blanks would disappear.

@yesnet0 has mentioned in the past program name is key (if I recall correctly)

It's also easy to de-dupe alphabetically for obvious reasons (eg. character case differences etc)

Instead of a git hook, I would go for a unit test of barebones alpine with jq to sort using the command in title.

Since maintainers can edit the forks, it reminds the pull requester to dedupe themselves, but collaborators can subsequently edit that individual's fork before commit, rather than automatically, just in case it creates errors.

However I do like the idea, and a little git hook sounds awesome.

apk add jq
jq -s '.[] | unique_by(.program_name)' < program-list.json > program-list.json.tmp
mv program-list.json.tmp program-list.json

That solves both Python scripts

from diodb.

I do like the idea of having users de duplicate it themselves :)

Me too because automatic de-duplication may lead to entry losses or other types of corruption.

Definitely can't de-dupe based on policy URL as multiple companies will have the same security "drop off point," and all blanks would disappear.

Makes sense, thank you!

from diodb.