Git Product home page Git Product logo

wikipedia-crawler's Introduction

wikipedia-crawler

This is a program to crawl entire Wikipedia using breath-first method and extract information from all the pages.

Summary

This is a Python program to crawl wikipedia pages and extract relevant inforation from the pages. It crawls in breath-first fashion and stores all the links that it has to crawl thus slowly covering all of the wikipedia. The program can be edited to extract customized information according to one's choice. MySQLdb library is then used to write the information into a MySQL table for further use.

Compatability

This program is written in Python 3.4 and can run on any version of Ptyhon (3.x). It is a download-and-run program with couple of changes according to user's requirements.

Status

This is first stable version of the program which is ready-to-run, but still under development. Many more features will be added to it shortly.

Disclaimer

This program lets you crawl the pages of wikipedia for information. It downloads then entire page of wikipedia that it is currently crawling. If this program is let to run for a long period of time, it can crawl the entire online database of wikipedia, which is highly discouraged!!!

Please use this program only for educational purpose by reviewing the content (in form of text, images, graphics or any other form) copyright and its licence. Also, crawling pages in a sequential manner puts a lot of pressure on Wikipedia servers. Please follow the Wikipedia robot.txt guidelines to understand thee restrictions on number of requests per second that can be made by an external robot. According to wikipedia friendly, low speed bots are allowed to crawl!

wikipedia-crawler's People

Contributors

hardikvasa avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wikipedia-crawler's Issues

Title output broken

Title outputs:

Link = https://en.wikipedia.org/wiki/Minkowski_diagram
Title = \n<html class="client-nojs" lang="en" dir="ltr">\n<head>\n<meta charset="utf-8"/>\n<title>minkowski diagram - wikipedia</title>\n<script>document.documentelement.classname=document.documentelement.classname.replace(/(^|\\s)client-nojs(\\s|$)/,"$1client-js$2");rlconf={"wgcanonicalnamespace":"","wgcanonicalspecialpagename":!1,"wgnamespacenumber":0,"wgpagename":"minkowski_diagram","wgtitle":"minkowski diagram","wgcurrevisionid":904360354,"wgrevisionid":904360354,"wgarticleid":11647860,"wgisarticle":!0,"wgisredirect":!1,"wgaction":"view","wgusername":null,"wgusergroups":["*"],"wgcategories":["commons category link is on wikidata","special relativity","geometry","diagrams"],"wgbreakframes":!1,"wgpagecontentlanguage":"en","wgpagecontentmodel":"wikitext","wgseparatortransformtable":["",""],"wgdigittransformtable":["",""],"wgdefaultdateformat":"dmy","wgmonthnames":["","january","february","march","april","may","june","july","august","september","october","november","december"],"wgmonthnamesshort":["","jan","feb","mar","apr","may","jun","jul","aug","sep","oct","nov","dec"],"wgrelevantpagename":"minkowski_diagram",\n"wgrelevantarticleid":11647860,"wgrequestid":"xuwfyapamfwaahiw-gaaaabw","wgcspnonce":!1,"wgisprobablyeditable":!0,"wgrelevantpageisprobablyeditable":!0,"wgrestrictionedit":[],"wgrestrictionmove":[],"wgmediavieweronclick":!0,"wgmediaviewerenabledbydefault":!0,"wgpopupsreferencepreviews":!1,"wgpopupsconflictswithnavpopupgadget":!1,"wgvisualeditor":{"pagelanguagecode":"en","pagelanguagedir":"ltr","pagevariantfallbacks":"en"},"wgmfdisplaywikibasedescriptions":{"search":!0,"nearby":!0,"watchlist":!0,"tagline":!1},"wgwmeschemaeditattemptstepoversample":!1,"wgpoweredbyhhvm":!0,"wgulscurrentautonym":"english","wgnoticeproject":"wikipedia","wgwikibaseitemid":"q177596","wgcentralauthmobiledomain":!1,"wgeditsubmitbuttonlabelpublish":!0};rlstate={"ext.gadget.charinsert-styles":"ready","ext.globalcssjs.user.styles":"ready","ext.globalcssjs.site.styles":"ready","site.styles":"ready","noscript":"ready","user.styles":"ready","ext.globalcssjs.user":"ready",\n"ext.globalcssjs.site":"ready","user":"ready","user.options":"ready","user.tokens":"loading","ext.cite.styles":"ready","ext.math.styles":"ready","mediawiki.legacy.shared":"ready","mediawiki.legacy.commonprint":"ready","mediawiki.toc.styles":"ready","wikibase.client.init":"ready","ext.visualeditor.desktoparticletarget.noscript":"ready","ext.uls.interlanguage":"ready","ext.wikimediabadges":"ready","ext.3d.styles":"ready","mediawiki.skinning.interface":"ready","skins.vector.styles":"ready"};rlpagemodules=["ext.cite.ux-enhancements","ext.math.scripts","site","mediawiki.page.startup","mediawiki.page.ready","mediawiki.toc","mediawiki.searchsuggest","ext.gadget.teahouse","ext.gadget.referencetooltips","ext.gadget.watchlist-notice","ext.gadget.drn-wizard","ext.gadget.charinsert","ext.gadget.reftoolbar","ext.gadget.extra-toolbar-buttons","ext.gadget.switcher","ext.centralauth.centralautologin","mmv.head","mmv.bootstrap.autostart","ext.popups","ext.visualeditor.desktoparticletarget.init",\n"ext.visualeditor.targetloader","ext.eventlogging","ext.wikimediaevents","ext.navigationtiming","ext.uls.compactlinks","ext.uls.interface","ext.cx.eventlogging.campaigns","ext.quicksurveys.init","ext.centralnotice.geoip","ext.centralnotice.startup","skins.vector.js"];</script>\n<script>(rlq=window.rlq||[]).push(function(){mw.loader.implement("user.tokens@0tffind",function($,jquery,require,module){/*@nomin*/mw.user.tokens.set({"edittoken":"+\\\\","patroltoken":"+\\\\","watchtoken":"+\\\\","csrftoken":"+\\\\"});\n});});</script>\n<link rel="stylesheet" href="/w/load.php?lang=en&amp;modules=ext.3d.styles%7cext.cite.styles%7cext.math.styles%7cext.uls.interlanguage%7cext.visualeditor.desktoparticletarget.noscript%7cext.wikimediabadges%7cmediawiki.legacy.commonprint%2cshared%7cmediawiki.skinning.interface%7cmediawiki.toc.styles%7cskins.vector.styles%7cwikibase.client.init&amp;only=styles&amp;skin=vector"/>\n<script async="" src="/w/load.php?lang=en&amp;modules=startup&amp;only=scripts&amp;raw=1&amp;skin=vector"></script>\n<meta name="resourceloaderdynamicstyles" content=""/>\n<link rel="stylesheet" href="/w/load.php?lang=en&amp;modules=ext.gadget.charinsert-styles&amp;only=styles&amp;skin=vector"/>\n<link rel="stylesheet" href="/w/load.php?lang=en&amp;modules=site.styles&amp;only=styles&amp;skin=vector"/>\n<meta name="generator" content="mediawiki 1.34.0-wmf.16"/>\n<meta name="referrer" content="origin"/>\n<meta name="referrer" content="origin-when-crossorigin"/>\n<meta name="referrer"
content="origin-when-cross-origin"/>\n<meta property="og:image" content="https://upload.wikimedia.org/wikipedia/commons/thumb/c/c2/minkowski_diagram_-_photon.svg/1200px-minkowski_diagram_-_photon.svg.png"/>\n<link rel="alternate" href="android-app://org.wikipedia/http/en.m.wikipedia.org/wiki/minkowski_diagram"/>\n<link rel="alternate" type="application/x-wiki" title="edit this page" href="/w/index.php?title=minkowski_diagram&amp;action=edit"/>\n<link rel="edit" title="edit this page" href="/w/index.php?title=minkowski_diagram&amp;action=edit"/>\n<link rel="apple-touch-icon" href="/static/apple-touch/wikipedia.png"/>\n<link rel="shortcut icon" href="/static/favicon/wikipedia.ico"/>\n<link rel="search" type="application/opensearchdescription+xml" href="/w/opensearch_desc.php" title="wikipedia (en)"/>\n<link rel="edituri" type="application/rsd+xml" href="//en.wikipedia.org/w/api.php?action=rsd"/>\n<link rel="license" href="//creativecommons.org/licenses/by-sa/3.0/"/>\n<link rel="canonical" href="https://en.wikipedia.org/wiki/minkowski_diagram"/>\n<link rel="dns-prefetch" href="//login.wikimedia.org"/>\n<link rel="dns-prefetch" href="//meta.wikimedia.org" />\n<!--[if lt ie 9]><script src="/w/resources/lib/html5shiv/html5shiv.js"></script><![endif]-->\n</head>\n<body class="mediawiki ltr sitedir-ltr mw-hide-empty-elt ns-0 ns-subject mw-editable page-minkowski_diagram rootpage-minkowski_diagram skin-vector action-view">\n<div id="mw-page-base" class="noprint"></div>\n<div id="mw-head-base" class="noprint"></div>\n<div id="content" class="mw-body" role="main">\n\t<a id="top"></a>\n\t<div id="sitenotice" class="mw-body-content"><!-- centralnotice --></div>\n\t<div class="mw-indicators mw-body-content">\n</div>\n\n\t<h1 id="firstheading" class="firstheading" lang="en">minkowski diagram</h1>\n\t\n\t<div id="bodycontent" class="mw-body-content">\n\t\t<div id="sitesub" class="noprint">from wikipedia, the free encyclopedia</div>\n\t\t<div id="contentsub"></div>\n\t\t\n\t\t\n\t\t\n\t\t<div id="jump-to-nav"></div>\n\t\t<a class="mw-jump-link" href="#mw-head">jump to navigation</a>\n\t\t<a class="mw-jump-link" href="#p-search">jump to search</a>\n\t\t<div id="mw-content-text" lang="en" dir="ltr" class="mw-content-ltr"><div class="mw-parser-output"><div class="thumb tright"><div class="thumbinner" style="width:258px;"><a href="/wiki/file:minkowski_diagram_-_photon.svg" class="image"><img alt="" src="//upload.wikimedia.org/wikipedia/commons/thumb/c/c2/minkowski_diagram_-_photon.svg/256px-minkowski_diagram_-_photon.svg.png" decoding="async" width="256" height="256" class="thumbimage" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/c/c2/minkowski_diagram_-_photon.svg/384px-minkowski_diagram_-_photon.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/c/c2/minkowski_diagram_-_photon.svg/512px-minkowski_diagram_-_photon.svg.png 2x" data-file-width="800" data-file-height="800" /></a>  <div class="thumbcaption"><div class="magnify"><a href="/wiki/file:minkowski_diagram_-_photon.svg" class="internal" title="enlarge"></a></div>the world line (yellow path) of a <a href="/wiki/photon" title="photon">photon</a>, which is at location <i>x</i> = 0 at time <i>ct</i> = 0.</div></div></div>\n<table class="vertical-navbox nowraplinks" style="float:right;clear:right;width:22.0em;margin:0 0 1.0em 1.0em;background:#f9f9f9;border:1px solid #aaa;padding:0.2em;border-spacing:0.4em 0;text-align:center;line-height:1.4em;font-size:88%"><tbody><tr><td style="padding-top:0.4em;line-height:1.2em">part of a series on</td></tr><tr><th style="padding:0.2em 0.4em 0.2em;padding-top:0;font-size:145%;line-height:1.2em"><a href="/wiki/spacetime" title="spacetime">spacetime</a></th></tr><tr><td style="padding:0.2em 0 0.4em"><a href="/wiki/file:gpb_circling_earth.jpg" class="image"><img alt="gpb circling earth.jpg" src="//upload.wikimedia.org/wikipedia/commons/thumb/d/d1/gpb_circling_earth.jpg/240px-gpb_circling_earth.jpg" decoding="async" width="240" height="180" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/d/d1/gpb_circling_earth.jpg/360px-gpb_circling_earth.jpg 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/d/d1/gpb_circling_earth.jpg/480px-gpb_circling_earth.jpg 2x" data-file-width="1200" data-file-height="900" /></a></td></tr><tr><td style="padding:0 0.1em 0.4em">\n<a href="/wiki/special_relativity" title="special relativity">special relativity</a><br /><a href="/wiki/general_relativity" title="general relativity">general relativity</a></td>\n</tr><tr><td style="padding:0 0.1em 0.4em">\n<div class="navframe collapsed" style="border:none;padding:0"><div class="navhead" style="font-size:105%;background:transparent;text-align:left">spacetime concepts</div><div class="navcontent" style="font-size:105%;padding:0.2em 0 0.4em;text-align:center"><a href="/wiki/spacetime" title="spacetime">spacetime manifold</a><br /><a href="/wiki/equivalence_principle" title="equivalence principle">equivalence
principle</a><br /><a href="/wiki/lorentz_transformation" title="lorentz transformation">lorentz transformations</a><br /><a href="/wiki/minkowski_space" title="minkowski space">minkowski space</a></div></div></td>\n</tr><tr><td style="padding:0 0.1em 0.4em">\n<div class="navframe collapsed" style="border:none;padding:0"><div class="navhead" style="font-size:105%;background:transparent;text-align:left">general relativity</div><div class="navcontent" style="font-size:105%;padding:0.2em 0 0.4em;text-align:center"><a href="/wiki/introduction_to_general_relativity" title="introduction to general relativity">introduction to general relativity</a><br /><a href="/wiki/introduction_to_the_mathematics_of_general_relativity" title="introduction to the mathematics of general relativity">mathematics of general relativity</a><br /><a href="/wiki/einstein_field_equations" title="einstein field equations">einstein field equations</a></div></div></td>\n</tr><tr><td style="padding:0 0.1em 0.4em">\n<div class="navframe collapsed" style="border:none;padding:0"><div class="navhead" style="font-size:105%;background:transparent;text-align:left">classical gravity</div><div class="navcontent" style="font-size:105%;padding:0.2em 0 0.4em;text-align:center"><a href="/wiki/gravity" title="gravity">introduction to gravitation</a><br /><a href="/wiki/newton%27s_law_of_universal_gravitation" title="newton&#39;s law of universal gravitation">newton\'s law of universal gravitation</a></div></div></td>\n</tr><tr><td style="padding:0 0.1em 0.4em">\n<div class="navframe collapsed" style="border:none;padding:0"><div class="navhead" style="font-size:105%;background:transparent;text-align:left">relevant mathematics</div><div class="navcontent" style="font-size:105%;padding:0.2em 0 0.4em;text-align:center"><a href="/wiki/four-vector" title="four-vector">four-vector</a><br /><a href="/wiki/derivations_of_the_lorentz_transformations" title="derivations of the lorentz transformations">derivations of relativity</a><br /><a class="mw-selflink selflink">spacetime diagrams</a><br /><a href="/wiki/differential_geometry" title="differential geometry">differential geometry</a><br /><a href="/wiki/curved_space" title="curved space">curved spacetime</a><br /><a href="/wiki/mathematics_of_general_relativity" title="mathematics of general relativity">mathematics of general relativity</a><br /><a href="/wiki/spacetime_topology" title="spacetime topology">spacetime topology</a></div></div></td>\n</tr><tr><td style="text-align:right;font-size:115%;padding-top: 0.6em;"><div class="plainlinks hlist navbar mini"><ul><li class="nv-view"><a href="/wiki/template:spacetime" title="template:spacetime"><abbr title="view this template">v</abbr></a></li><li class="nv-talk"><a href="/wiki/template_talk:spacetime" title="template talk:spacetime"><abbr title="discuss this template">t</abbr></a></li><li class="nv-edit"><a class="external text" href="https://en.wikipedia.org/w/index.php?title=template:spacetime&amp;action=edit"><abbr title="edit this template">e</abbr></a></li></ul></div></td></tr></tbody></table>\n<p>the <b>minkowski diagram</b>, also known as a <b>spacetime diagram</b>, was developed in 1908 by <a href="/wiki/hermann_minkowski" title="hermann minkowski">hermann minkowski</a> and provides an illustration of the properties of space and time in the <a href="/wiki/special_theory_of_relativity" class="mw-redirect" title="special theory of relativity">special theory of relativity</a>. it allows a qualitative understanding of the corresponding phenomena like <a href="/wiki/time_dilation" title="time dilation">time dilation</a> and <a href="/wiki/length_contraction" title="length contraction">length contraction</a> without mathematical equations.\n</p><p>minkowski diagrams are two-dimensional graphs that depict events as happening in
a <a href="/wiki/universe" title="universe">universe</a> consisting of one space dimension and one time dimension. unlike a
regular distance-time graph, the distance is displayed on the horizontal axis and time on the vertical axis. additionally, the time and space <a href="/wiki/units_of_measurement" class="mw-redirect" title="units of measurement">units of measurement</a> are chosen in such a way that an object moving at the speed of light is depicted as following a 45\xc2\xb0 angle to the diagram\'s axes.\n</p><p>in this way, each object, like an observer or a vehicle, traces a certain line in the diagram, which is called its <a href="/wiki/world_line" title="world line">world line</a>. also, each point in the diagram represents a
certain position in space and time, and is called an <b>event</b>, regardless of whether anything relevant happens there.\n</p>\n<div id="toc" class="toc"><input type="checkbox" role="button" id="toctogglecheckbox" class="toctogglecheckbox" style="display:none" /><div class="toctitle" lang="en" dir="ltr"><h2>contents</h2><span class="toctogglespan"><label class="toctogglelabel" for="toctogglecheckbox"></label>
Related Links = Minkowski space\nPenrose diagram\nRapidity\n\n
Introduction = The Minkowski diagram, also known as a spacetime diagram, was developed in 1908 by Hermann Minkowski and provides an illustration of the properties of space and time in the special theory of relativity. It allows a qualitative understanding of the corresponding phenomena like time dilation and length contraction without mathematical equations.\n

Expecting title to output:

minkowski diagram - wikipedia

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.