This demo is built to showcase how you AI might assist you in troubleshooting network issues.

The components used by this demo are:

• Virtual IOS-XE devices running ISIS.
  • The CML Devnet sandbox was used to build the lab.
  • Sandbox DevBox VM 10.10.20.50, developer/C1sco12345)
• ncpeek. A python netconf client used for telegraf.
• TIG stack with docker 20.10+ 🐳
  • Telegraf uses ncpeek to pull telemetry data from network devices.
  • Grafana kicks a webhook when an alarm is detected. 🚨
• FastAPI.
  • Host the LLM.
  • Interacts with PyATS & Webex.
• PyATS. Provides a framework to interact with network devices. 🛠️
• Webex_bot use to interact with the LLM. 🤖
• OpenAI LLM. 🧠
  • gpt-4-turbo-preview was used. 🚀

For this demo one alarm was created.

if avgNeighbors(30sec) < avgNeighbors(30min) : send Alarm​

When the average number of ISIS neighbors in a lapse of 30 second is less than the average number of ISIS neighbors in a lapse of 30 minutes, the alarm will trigger a webhook for the LLM.

This signal that a stable ISIS neighbor that was working on the last 30 minutes was lost, and allows to work with N number of ISIS neighbors.

Environment variables are injected through the use of the Makefile on root of the project.

OPENAI_API_KEY=<YOUR_OPENAI_API_KEY>
WEBEX_TEAMS_ACCESS_TOKEN=<YOUR_TEAM_ACCESS_TOKEN>
WEBEX_APPROVED_USERS_MAIL=<MAILS_OF_USERS_APPROVED_SEPARATED_BY_COMMAS>
WEBEX_USERNAME=<YOUR_WEBEX_USERNAME>
WEBEX_ROOM_ID=<THE_WEBEX_ROOM_ID>

If you prefer to use another client, you need to:

• Modify the notify function to accomodate your client.
• Remove/comment the start of the webex bot
• Communicate with the LLM using REST API. See send_message_to_chat_api for an example. 📡

To get your webex token go to https://developer.webex.com/docs/bots and create a bot.

To get the WEBEX_ROOM_ID the easiest way is to open a room with your bot in the webex app. Once you have your room, you can get the WEBEX_ROOM_ID by using API list room, use your token created before.

For testing, you can use the GRAFANA_WEB_HOOK env var to send webhooks to other site, such as https://webhook.site/

If you have access to smith.langchain.com (recommended for view LLM operations) add your project ID and API key.

GRAFANA_WEB_HOOK=<WEB_HOOK_URL>
LANGCHAIN_PROJECT=<YOUR_LANGCHAIN_PROJECT_ID>
LANGCHAIN_API_KEY=<YOUR_LANGCHAIN_API_KEY>
LANGCHAIN_TRACING_V2=true
LANGCHAIN_ENDPOINT=https://api.smith.langchain.com

The .env.local file is used to define all variables used by the containers.

In a production environment, this file should be kept out of version control using the .gitignore file.

This demo uses a CML instance from the Cisco DevNet sandbox. You can also use a dedicated CML instance or a NSO sandbox. 🏖️

After acquiring your sandbox, stop the default topology and wipe it out. 🧹

Then, import the topology file used for this demo and start the lab.

The TIG stack requires Docker and IP reachability to the CML instance. For this demo, I used the sandbox VM 10.10.20.50.

First time, build the TIG stack.

make build-tig

Subsequent runs of the TIG stack you can run the containers.

make run-tig

Telegraf

• Logs: On 10.10.20.50 use docker exec -it telegraf bash then tail -F /tmp/telegraf-grpc.log.
  • See config defined here

Influxdb

• http://10.10.20.50:8086 with the credentials admin/admin123

Grafana

• http://10.10.20.50:3000/dashboards with the credentials admin/admin
• Navigate to General > Network Telemetry to see the grafana dashboard.

The llm_agent directory provides the entry point for the application, the app file

The llm container runs on the sandbox VM 10.10.20.50.

make run-llm

The demo involves shutting down one interface, causing an ISIS failure, and allowing the LLM to diagnose the issue and implement a fix.

In the images below, GigabitEthernet5 was shutting down on cat8000-v0 resulting in losing its ISIS adjacency with cat8000-v2

You can watch the recorded demo here

On Grafana, you can observe the ISIS count decreasing and triggering an alarm.

Next, you will receive a webex notification from grafana and the LLM will receive the webhook. The webhook triggers the LLM to start looking at what the issue is and how to resolve it.

• Tokens can run out easily with netconf, highly important to filter what is sent to the AI.
• Repeated alarms are suppresed by Grafana, this is controlled by the grafana policy file,
  • If you are testing continously, run make run-tig to destroy and create the TIG containers.
  • This isn't an ideal scenario, but a proper solution wasn't found within the given time.
• From time to time, the answers from the LLM are lost and not sent to webex. You can find them on the terminal output.
• This is the third iteration of this exercise. The first one was Cisco Live Amsterdam 2024
  • The main differences are the use of makefile, docker compose and the refactoring of the llm agent code for a better separation of concerns.