Duplicates Issue about cent HOT 11 CLOSED

Glad I was able to help. :)

from cent.

Hi,
in the future I will implement new features to make cent better from this point of view.

from cent.

I think you should do something like this...

1. Delete the duplicates from the cent folder like same cves or try comparing the content there is a tool named fdupes that can help you with that.
2. Update the original nuclei templates and then install all this custom templates and then compare them, and delete the duplicates from the cent folder.

Reference:
https://www.tecmint.com/fdupes-find-and-delete-duplicate-files-in-linux/

*Also try other tools maybe you will find something better than this.

from cent.

Hi,
I really want to thank you, your reference helped me a lot.
I released version 1.0 of cent with template deduplication using fdupes.

Try it now and let me know if works

from cent.

How do you remove duplicates??
Do you compare the original templates and custom templates?

from cent.

And also remove this: https://github.com/pikpikcu/nuclei-templates
That user just forked the original templates...

from cent.

How do you remove duplicates??
Do you compare the original templates and custom templates?

Hi,
the dedupe of the templates is done through hashing thanks to the @hakluke pull request.
You can check the code here: RemoveDuplicates() function

...
func RemoveDuplicates(path string, console bool) {
	fmt.Println("Removing duplicate templates...")
	files := getFilePaths(path)
	hashes := make(map[string]string)

	// get file hashes
	for _, file := range files {
		hashes[file] = getFileHash(file)
	}
      ...
      ...
...

from cent.

And also remove this: https://github.com/pikpikcu/nuclei-templates
That user just forked the original templates...

I apologize for this, I have not updated the readme with all the links anymore, but if you check the .cent.yaml file you will see that it is not there and it has been updated with the new pikpikcu repo ( my-nuclei-templates )

from cent.

Hi!
Ok! But do you remove the templates which are already in pd's repo?
Because it will just burst with duplicates templates and disrupt the server with same requests twice...

from cent.

Yes, I recommend you to insert the main repo also in the .cent.yaml, in this way cent will also remove all the possible duplicates with respect to the main repo as well

...
community-templates:
  - https://github.com/projectdiscovery/nuclei-templates.git
...

from cent.

Oh Okay!
My bad, I didn't notice it. I will let you know if I have any other ideas or something, closing this issue for now.
Have a great day!

from cent.