microsoft / ghcrawler-dashboard Goto Github PK

View Code? Open in Web Editor NEW

34.0 6.0 24.0 326 KB

Crawler Dashboard to control ghcrawler application.

License: MIT License

JavaScript 81.51% HTML 17.65% Dockerfile 0.84%

ghcrawler-dashboard's Introduction

Crawler Dashboard

Crawler Dashboard to control ghcrawler application.

Displays queued active messages.
Displays message rates.
Displays and allows to change configuration.
Allows to queue request.
Allows to recrete queues.
Allows to get and delete requests from a queue.

Usage

Configuration

Environment Variables

The bare minimum configuration is:

{
  "DEBUG_ALLOW_HTTP" : true,
  "CRAWLER_REDIS_URL": "",
  "CRAWLER_REDIS_ACCESS_KEY": ""
}

For more options see env/env-template.json file.

The variables may be set either individually as environment variables or in env.json file at the same level as the project (ghcrawler-dashboard/../env.json).

Install Node packages

$ npm install

Test

$ npm test

Run

DEBUG=appinsights npm start or

DEBUG=* npm start

Note: Local environment can be accessed at http://localhost:4000.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

ghcrawler-dashboard's People

Contributors

Stargazers

Watchers

Forkers

alexxnica kryndex craigez caweinshenker zhhb pombredanne sportsbitenews afcarl baggerspion rspier danisyellis bensbigolbeard womencancode modulexcite bhaskers-blu-org2 taffywrinkle claudiusgonzo dolphinux dearborn-open-ai

ghcrawler-dashboard's Issues

rename to ghcrawler-dashboard

We've settled on "ghcrawler" as the name for the system. rename this repo to line up

Dashboard API returns 500s if Crawler Service using self-signed certificate

If the endpoint at CRAWLER_SERVICE_URL is using a self-signed (or otherwise untrusted) certificate then the dashboard will fail to perform any actions that require talking to the Crawler Service. The dashboard should support passing a list of CA certificates to request like is done here: https://github.com/Microsoft/ghcrawler/blob/develop/providers/queuing/rabbitQueueManager.js#L48.

Deleting/Requeuing dealetters should not fail if one of requests fails

Warning messages should be added to response instead of either failing everything or completely swallowing the errors.

How do input to crawler to search for repos with specific contents.

unable to get dashboard working

So I have ghcrawler running and I'm able to manage it using command line.
However dashboard is not coming up.

I cloned this project locally then ran npm install and then npm start (unable to run DEBUG=* npm start)
however then when I go to localhost:4000 it doesn't load anything.

Not easy to configure "single-shot" running

The existing crawler is rather designed to be able to scale to crawling many hundreds or thousands of repositories; it's a daemon which carefully schedules its work, and a separate set of utilities to control that daemon and add to its queues and so on. This is great if you want to crawl thousands of repositories. It's pretty serious overkill if you want to crawl, say, three repositories though. Is there some way to run all this in "cut down" mode, where it wakes up and scans a repository and then shuts down, without needing lots of setup and a running background daemon and so on, so it's a bit less heavyweight? (The answer may be "no, that's not what it's designed for", which is fine, but if there's some way to spin it up in a lightweight way that would be useful to know about.)

Where does configuration need to go?

The README says that the bare minimum configuration is (some vars), but doesn't say where they should be put. env/env.json? env.json? config/env.json? Somewhere else?