This repository aims to provide in-depth insights into Webtoon comments using open-source Language Model Models (LLMs) and the Retrieval Augmented Generation (RAG) framework.

• Crawling & Parsing: Scrapes Webtoon comments using Selenium and parsed them accordingly
• LLM Loading: Loads LLM such as Mixtral and OpenChat
• RAG Setup: Sets up the RAG framework for generating responses
• App Building: Chat-based Streamlit app for analyzing comments

• Crawling and Parsing: Selenium and Pandas
• Backend (LLM serving): Ollama API
• RAG: Langchain
• Frontend (App): Streamlit

• Install GoogleChrome driver

  sudo apt install ./google-chrome-stable_current_amd64.deb'
  

• Create a Conda environment and install required packages:

  conda create --name <env> --file requirements.txt
  

• Step 0: Conda env activation

  conda activate <env>
  

• Step 1: Crawling & parsing

  python crawling_comments.py --title "lore-olympus" --title_id "1320" --start_ep 1 --end_ep 10 
  

• Step 2: Ollama serve

  CUDA_VISIBLE_DEVICES=0 ollama serve
  

• Step 3: Streamlit app

  • execution (Upload the csv file created in Step 1)

    streamlit run app.py --server.port 7087 --server.address 0.0.0.0
    

• More advanced features (within the RAG pipeline) will be introduced