Git Product home page Git Product logo

darksearch's Introduction

Build Status

###About Darksearch Darksearch allows you to query cached onion sites, irc chatrooms, various pdfs, game chats, blackhat forums etc...

API

Darksearch also has an API in the works. Currently you can't scrape specific data for your queries, but you can retrieve metadata on your searches by using a GET request on darksearch.io/api/YOUR_QUERY/PAGE_NUMBER

$ curl -XGET darksearch.com/api/spies/1

{
  "Duration": "0.026", 
  "Query": "spies", 
  "Results": "{'10': {'10': {'Host': u'http://ss2v6i44b3vy4tdf.onion//advisory-board/', 'Description': u\"live in hiding for a year and face arrest her partner was imprisoned twice for exposing the crimes of the <font color='red'><b>spies</b></font> she is the author of spies lies and whistleblowers mi5 and the david shayler affair she is now a\", 'Timestamp (scraped)': u'2016-04-24 08:02:57', 'Title': u'Advisory Board Courage Foundation', 'URL': u'ss2v6i44b3vy4tdf.onion/advisory-board.html', 'Tor2Web (.cab)': u'ss2v6i44b3vy4tdf.onion.cab/advisory-board.html', 'Tor2Web (.to)': u'ss2v6i44b3vy4tdf.onion.to/advisory-board.html', 'Size (bytes)': 39006}}, '1': {'1': {'Host': u'http://maskravvbmurcaiz.onion/', 'Description': u\"maskrabbit maskrabbit maskrabbit is an anonymous agency for real world operators we specialize in couriers thieves <font color='red'><b>spies</b></font> saboteurs hackers and goons maskrabbit only works with professional agents and serious clients to apply use the appropriate form to submit your needs\", 'Timestamp (scraped)': u'2016-04-21 09:53:20', 'Title': u'MaskRabbit', 'URL': u'maskravvbmurcaiz.onion/index.html', 'Tor2Web (.cab)': u'maskravvbmurcaiz.onion.cab/index.html', 'Tor2Web (.to)': u'maskravvbmurcaiz.onion.to/index.html', 'Size (bytes)': 1668}}, '3': {'3': {'Host': u'http://uksfvgmwpiww3n4s.onion/', 'Description': u\"when gov imprison bradly manning and torture him until he says he is guilty in the time when they setup <font color='red'><b>spies</b></font> and sexual crime to imprison julian assange do you really believe that gov will be stopped in legal way\", 'Timestamp (scraped)': u'2016-04-23 10:50:27', 'Title': u'Our reality', 'URL': u'uksfvgmwpiww3n4s.onion/index.html', 'Tor2Web (.cab)': u'uksfvgmwpiww3n4s.onion.cab/index.html', 'Tor2Web (.to)': u'uksfvgmwpiww3n4s.onion.to/index.html', 'Size (bytes)': 3495}}, '2': {'2': {'Host': u'http://ac4jrkjk4ialqkoh.onion/category/revealed-documents/', 'Description': u\"this gchq manual from 2008 explains how analysts would unscramble the video signals from israeli drones see the intercept article <font color='red'><b>spies</b></font> in the sky israeli drone feeds hacked by british and american intellience 29 january 2016 continue reading s455n israeli\", 'Timestamp (scraped)': u'2016-04-21 08:54:29', 'Title': u'Revealed documents Courage Snowden', 'URL': u'ac4jrkjk4ialqkoh.onion/category/revealed-documents.html', 'Tor2Web (.cab)': u'ac4jrkjk4ialqkoh.onion.cab/category/revealed-documents.html', 'Tor2Web (.to)': u'ac4jrkjk4ialqkoh.onion.to/category/revealed-documents.html', 'Size (bytes)': 36944}}, '4': {'4': {'Host': u'http://lcvkso2t5t3cmy3x.onion/', 'Description': u\"provide his blackberry phone password to canada border services agency cbsa officers at a halifax airport 2015 02 25 canadian <font color='red'><b>spies</b></font> collect domestic emails in secret security sweep the communications security establishment cse is covertly monitoring canadians emails 2015 01\", 'Timestamp (scraped)': u'2016-04-22 06:45:48', 'Title': u'Hack Canada', 'URL': u'lcvkso2t5t3cmy3x.onion/index.html', 'Tor2Web (.cab)': u'lcvkso2t5t3cmy3x.onion.cab/index.html', 'Tor2Web (.to)': u'lcvkso2t5t3cmy3x.onion.to/index.html', 'Size (bytes)': 9269}}, '6': {'6': {'Host': u'http://h2am5w5ufhvdifrs.onion', 'Description': u\"november 20 2013 2013 1612 htm bis end user certificates for china november 20 2013 2013 1611 pdf fisc usa <font color='red'><b>spies</b></font> deny release of doc to aclu nobember 19 2013 2013 1610 vid video nsa ddir inglis at ny law\", 'Timestamp (scraped)': u'2016-04-21 11:39:09', 'Title': u'Cryptome', 'URL': u'h2am5w5ufhvdifrs.onion/index.html', 'Tor2Web (.cab)': u'h2am5w5ufhvdifrs.onion.cab/index.html', 'Tor2Web (.to)': u'h2am5w5ufhvdifrs.onion.to/index.html', 'Size (bytes)': 66052}}, '9': {'9': {'Host': u'http://swehackmzys2gpmb.onion/./viewforum.php?f=21&amp;sid=51857442da28cbcb1e47336b287b2960', 'Description': u\"33 1 2 3 svar 27 27 svar 2842 visningar senaste inl gget av avlidienbrunn 18 apr 2016 13 14 <font color='red'><b>spies</b></font> in the skies artikel om fbi s flyg vervakning av chlo 13 apr 2016 03 50 svar 5 5\", 'Timestamp (scraped)': u'2016-04-22 18:45:05', 'Title': u'Lektyr och media swehack org', 'URL': u'swehackmzys2gpmb.onion/./viewforum.php?f=21&amp;sid=51857442da28cbcb1e47336b287b2960', 'Tor2Web (.cab)': u'swehackmzys2gpmb.onion.cab/./viewforum.php?f=21&amp;sid=51857442da28cbcb1e47336b287b2960', 'Tor2Web (.to)': u'swehackmzys2gpmb.onion.to/./viewforum.php?f=21&amp;sid=51857442da28cbcb1e47336b287b2960', 'Size (bytes)': 48200}}, '8': {'8': {'Host': u'http://swehackmzys2gpmb.onion/./viewforum.php?f=21&amp;sid=3941cd89256fc9b6c7561ccaa4d3a9a1', 'Description': u\"33 1 2 3 svar 27 27 svar 2769 visningar senaste inl gget av avlidienbrunn 18 apr 2016 13 14 <font color='red'><b>spies</b></font> in the skies artikel om fbi s flyg vervakning av chlo 13 apr 2016 03 50 svar 5 5\", 'Timestamp (scraped)': u'2016-04-21 07:22:11', 'Title': u'Lektyr och media swehack org', 'URL': u'swehackmzys2gpmb.onion/./viewforum.php?f=21&amp;sid=3941cd89256fc9b6c7561ccaa4d3a9a1', 'Tor2Web (.cab)': u'swehackmzys2gpmb.onion.cab/./viewforum.php?f=21&amp;sid=3941cd89256fc9b6c7561ccaa4d3a9a1', 'Tor2Web (.to)': u'swehackmzys2gpmb.onion.to/./viewforum.php?f=21&amp;sid=3941cd89256fc9b6c7561ccaa4d3a9a1', 'Size (bytes)': 48175}}}", 
  "Total Pages": "4", 
  "Total Results": "40"
}

The Darksearch index is growing as more scrapers get built...

darksearch's People

Contributors

vlall avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

darksearch's Issues

Routing Links

External onions and linked onion directories need to be scraped and presented in darksearch for each result. Reported at #2

Can't search for specific keywords

Hi Vlad,

Wishing you a very Happy New Year and hope the new year brings DarkSearch more success. I was just trying out your updated platform and I saw that while searches for generic keywords such as passwords, accounts worked fine, search for some keywords specific to a company name yielded no results. However, when searched for the same keyword on the sites like onion.city / onion.link yielded results.

Also, searches for phrases like two words matched only the first word in the phrase.

Regards,
Anish

Version tagging

In order to maintain consistency between versions of Darksearch that exist in Gitlab v Github v other branches, we should include version tagging to maintain consistency and organization.

Merge gitlab repo

Gitlab contains a private repo that's been hosting much of the Darksearch repo. This repo needs to replace the old content.

Dockerfile build issue

seems like there is no add-apt-repository in default ubuntu:14.04 docker image.

Status: Downloaded newer image for ubuntu:14.04
 ---> 14b59d36bae0
Step 2 : MAINTAINER Vishal Lall "[email protected]"
 ---> Running in 40dc37b7a1e0
 ---> 2374f7998cd2
Removing intermediate container 40dc37b7a1e0
Step 3 : RUN add-apt-repository ppa:webupd8team/java
 ---> Running in 93a52e34bf3f
/bin/sh: 1: add-apt-repository: not found

This can be fixed running:

apt-get install -y software-properties-common

before the add-apt-repository

Please verify your dockerfile builds correctly.

Improve automated testing

Right now Travis is not testing many parts of the pipeline... It's just making sure the /Site gets generated. The /Crawler process of scraping and adding to Elasticsearch is never tested, which is a bulk of the Darksearch functionality.

Update to newest flask-user

The newest flask-user is under development and could provide many solutions to pre-existing bugs I've run into in the past. Another part of this is actually reporting the issues I run into to the flask-user page.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.