• The current codebase after adding web-crawler is missing some node dependencies. Resolve it so that it will work with npm i itself.
• Implement proper error handling for web-crawler part.
• Follow the response code convention like other APIs for this controller fully.
• Rename the controller file so as to have Controller as suffix - to maintain uniformity (see other controller files).
• Update the csvController.js accordingly after adding heading field. (Optionally, you can try to refactor the code so that any changes to contentModel doesn't require any changes to be made to this controller)
• Migrate the route to scrape.js file instead of new file. (i.e, scrape/web-crawler can be the endpoint)
• Return the number of Links scraped successfully along with failed links and any other such statistics that might be necessary as a part of successful response.

Issue: Setting up MongoDB and Creating Content Model

Task Description

In order to enhance the functionality of the IntranetSearch project, we need to perform the following tasks:

1. Setup MongoDB:

   • Configure MongoDB to store scraped content data.
   • Store the MongoDB database URL and other necessary configurations in a .env file inside the backend folder.

2. Ccomplete connectDB.js:

   • Write the code for MongoDB connection configuration in a connectDB.js file inside the configs folder in the backend.

3. Create Content Model:

   • Create a MongoDB model for storing scraped content.
   • This model should have the following fields:
     • content: To store the textual content of the scraped page.
     • url: To store the URL of the scraped page.
     • embeddings: To store a vector of 384 dimensions, which will be used for various content analysis tasks.
       (For now, store some dummy data inside these for testing purposes. embedding field can be made not required)

4. Name the Content Model File:

   • Name the content model file as contentModel.js and place it inside the models folder (to be created) in the backend.

5. Create a CSV Export Controller:

   • Create a controller inside the controllers/web-crawler directory as csvSaveController.js
   • This controller should be responsible for converting the documents stored in the content model to a CSV file.
   • The CSV file should have an appropriate header as given in Contentmodel
   • The name of the CSV file should be provided in req.body.
   • get the required fields to be included in the csv files also from req.body. It is basically a enum of [content,url,embeddings]
     (i.e. if the user specifies only content as required in req.body then only content column should be there in csv)
   • The generated CSV file has to be saved inside the data folder, which has already been created.

6. Update Routes:

   • Add appropriate routes and endpoints for the new functionalities in the routes folder.

Expected Outcome

Upon completion of these tasks, we will have MongoDB configured to store scraped content data, a content model for structured data storage, and the ability to convert content to CSV format for analysis. This will enhance our project's capabilities significantly.

Note: Please make sure to create separate commits for each task, and include relevant documentation and comments in your code. The necessary folders and file have been already created. Provide a postman collection for testing the same also.

• Write the Dockerfile and docker-compose.yml for backend part.
• Write docker-compose.yml for python endpoint also inside python folder.
• Might be needing to change the Dockerfile for including frontend part also in future.
• Dockerize the elasticsearch part also, make sure that it is secure (uses password and username)
  Note: The codebase should be able to run in the docker container. Test this locally.

Create dashing UI for the backend routes already created inside /backend folder

The current docker container in python folder uses lots of space due to sentence-transformers library.
Optimise this space if possible by making changes in Dockerfile.

• Create a working web crawler inside controllers/web-crawler directory.
• Store the scrapped data as content, url fields appropriately in the contentModel (created by Issue #6).
• Create a route for the same (/web-crawler) in routes folder. Name it as crawler.js.
• Use this route in app.js and create a PR after proper testing using postman.

Use admin bro or something similar to that to make an admin panel for managing scraped content and all other relavent functions properly.

Update Readme.md to show how to setup and requirements
Try to add comments

GitHub Actions Setup

Tasks

Setup github actions to:

1. Automatically add the hacktoberfest-accepted label to merged pull requests for an issue.
2. Automatically add the hacktoberfest and hacktoberfest2023 labels to newly created issues.
3. Comment assigned, tagging the user as soon as someone is assigned to an issue.

Context

GitHub Actions can help streamline our development and contribution processes by automating certain repetitive tasks. These tasks, as mentioned above, will be beneficial for tracking contributions related to Hacktoberfest.

• The cleaned content, especially headings have lot of newline and trailing and leading backspace, which might introduce unnecessary padding vectors in word embeddings. Hence clean them before saving it to the database.
• Additionally, scrape additional contents like some tags etc if they will be useful for searching in elasticsearch database.

Issue: Design User Interface for IntranetSearch Engine

Task Description

To improve the user experience of the IntranetSearch Engine, we need to design a clean and intuitive user interface. The UI should include a search bar and display search results with links and brief content, similar to popular search engines like Google.

UI Design Specifications

Design a UI that includes the following elements:

1. Search Bar:

   • A prominent search bar at the top of the page for users to input search queries.

2. Search Results:

   • Display search results in a structured and easy-to-read format.
   • Each search result should include:
     • Title or link to the intranet page.
     • A brief content snippet or description to provide context.

3. Pagination:

   • If there are multiple search results, include a pagination feature to navigate through pages of results.

4. Clean and Minimalist Design:

   • Use a clean and minimalist design with a focus on readability and usability.
   • Ensure responsive design for different screen sizes.

Provide Figma Design

Please create a Figma design that visually represents the proposed UI. The Figma design should include wireframes or mockups of the search results page, highlighting the search bar, search results, and any additional elements you think would improve the user experience.

Note: Please provide a link to the Figma design once it's ready, and ensure that the design is user-centric and visually appealing.

Issue: Setup Elasticsearch Locally in Windows and Test Connection

Task Description

Setting up an Elasticsearch server locally in Windows can sometimes be challenging for newcomers. To address this, we need to provide comprehensive instructions and a video tutorial demonstrating the setup process. Additionally, we will connect to Elasticsearch using a client (e.g., JavaScript) to ensure it's working correctly.

Detailed Instructions and Video Tutorial

1. Setup Elasticsearch Locally in Windows:

   • Provide detailed step-by-step instructions on how to download and install Elasticsearch on a Windows machine.
   • Mention any common pitfalls and troubleshooting tips.

2. Video Tutorial:

   • Create a video tutorial that complements the written instructions. make it short and crisp.
   • Demonstrate the entire setup process, from downloading the Elasticsearch package to running the server.

3. Connect to Elasticsearch Using a Client:

   • Utilize an Elasticsearch client, such as JavaScript (e.g., Elasticsearch.js library), to establish a connection to the local Elasticsearch server.
   • Create a simple script that performs a basic query or index operation to showcase the connection.

Note: Please make sure the video is short and crisp and include your documentation along with video in ELASTICSEARCHSETUP.md file in root directory.

Issue: Upload Data from CSV to Elasticsearch

Task Description

To enhance the functionality of the project, we need to create a route in the main.py file (located inside the python/app folder) for uploading data from a CSV file into Elasticsearch. We'll use the initialized Elasticsearch client for this purpose. Additionally, we should refer to the create-index route in the backend folder for the structure of the Elasticsearch index.

Detailed Task

1. Modify the store route in main.py:

• Inside python/app in main.py file store route, write the code for uploading this data from the df created into elasticsearch index using the intitialised elasticsearch client.
• Refer to create-index route in backend folder for the structure of index.

2. Refer to create-index Structure:
   • Refer to the structure of the Elasticsearch index used in the create-index route located in the backend folder.

• Index structure

embeddings: {
        type: "dense_vector",
        dims: req.body.dim ? req.body.dim : 384,
        index: true,
        similarity: "cosine",
      },
      content: { type: "text" },
      url: { type: "text" },

3. Hint - Bulk API for Data Upload:
   • Use the Elasticsearch Bulk API to efficiently upload data from the CSV file into the Elasticsearch index.

Write comments wherever possible and write clean and efficient codes.