aryan3212 / bdjobs-scraper Goto Github PK

Why?

• Jobs in BDjobs usually have a 30 day active period. We want to filter out duplicate results that we get onto our data. For now, we can just add the ID for the jobs.

How?

• Parse from the link or page and add to the dictionary that is generated.

Enhancements:

• Append unique jobs only to the CSV when running the script over days

What tests?

• Is the script properly parsing the details page.
• Is it getting all the links from a list.
• Is it getting all the jobs including the last page?

How?

• Fetch the html for a page from BDJobs, it should be both the list and details page.
• Grab those files from disk and run the parser on them.
• Write down the expected result(test fixtures) and use the logic from the script to see if the results match.

How are they gonna help?

• When BDJobs changes their page structure these tests will fail, then we can modify the code and retest. Tests would not and should not change in this case.
• Will help build the CI/CD workflow when making extra changes.