Git Product home page Git Product logo

nzedb / nzedb Goto Github PK

View Code? Open in Web Editor NEW
629.0 79.0 180.0 96.97 MB

nZEDb - a fork of nnplus(2011) | NNTP / Usenet / Newsgroup indexer.

Home Page: https://nzedb.github.io/

PHP 34.59% Shell 1.40% CSS 7.34% Smarty 26.04% AppleScript 0.03% C 0.09% Python 2.05% SQLPL 0.16% JavaScript 4.37% Makefile 0.01% Batchfile 0.02% PLpgSQL 0.01% Hack 0.01% TSQL 23.90%
nzedb nntp usenet newsgroups indexer php mariadb tmux sphinx nzb

nzedb's Introduction

Stable: Build Status Testing: Build Status Dev: Build Status

nZEDb automatically scans usenet, similar to the way web spiders scan the internet. It does this by collecting usenet headers and temporarily storing them in a database until they can be collated into posts/releases. It provides a web-based front-end providing search, browse, and programmable (API) functionality.

This project is a fork of the open source usenet indexer newznab plus: https://github.com/anth0/nnplus

nZEDb improves upon the original design, implementing several new features including:

  • Optional multi-threaded processing (header retrieval, release creation, post-processing etc)
  • Advanced search features (name, subject, category, post-date etc)
  • Intelligent local caching of metadata
  • Optional sharing of comments with other nZEDb sites
  • Optional tmux (terminal session multiplexing) engine that provides thread, database and performance monitoring
  • Image and video samples
  • SABnzbd/NZBGet integration (web, API and pause/resume)
  • CouchPotato integration (web and API)

Prerequisites

System Administration know-how. nZEDb is not plug-n-play software. Installation and operation requires a moderate amount of administration experience. nZEDb is designed and developed with GNU/Linux operating systems. Certain features are not available on other platforms. An experienced Windows administrator should be able to run nZEDb on a Windows OS.

Hardware

4GB RAM, 2 cores(threads) and 20GB disk space minimum.

If you wish to use more than 5 threads a quad core CPU is beneficial. While nZEDb can run with these minimums, you will have very limited performance.

The overall speed of nZEDb is largely governed by performance of the database. As many of the database tables should be held within system RAM as possible. See Database Section below.

Software

PHP (current actively supported versions and various modules)
MariaDB 10.0 (strongly preferred database choice)

The installation guides have more detailed software requirements.

Database

Most (if not all) distributions ship MariaDB with a default configuration that will perform well on a Raspberry Pi. If you wish to store more that 500K releases, these default settings will quickly lead to poor performance. Expect this.

As a general rule of thumb the database will need a minimum of 1-2G buffer RAM for every million releases you intend to store. That RAM should be assigned to either of these two parameters:

  • key_buffer_size (MyISAM)
  • innodb_buffer_pool_size (InnoDB)

Use mysqltuner.pl for recommendations for these and other important tuner parameters. Also refer to the project's wiki page: https://github.com/nZEDb/nZEDb/wiki/Database-tuning. This is particularly important before you start any large imports or backfills.

MariaDB is normally shipped using MyISAM tables by default. This is fine for running with one or a few threads and is a good way to start using nZEDb. You should migrate to the InnoDB table format if nZEDB is configured to use one of the following:

thread counts > 5
tmux mode

This conversion script is helpful:

misc/testing/DB/convert_mysql_tables.php

Before converting to InnoDB be sure to set:

innodb_file_per_table

Installation

Specific installation guides for common Operating Systems can be found on the nZEDb github wiki: https://github.com/nZEDb/nZEDb/wiki/Install-Guides

Getting Started

After you have installed nZEDb and gone throught the initial web-based installation steps (step1, step2 etc.), then review the settings on the Admin-Site-Edit page.

Most of the default settings are fine, however some will need changing.

The two amazon keys and the associate tag are needed to fetch anything from amazon. The trakt.tv key is optional, but it can help fetching extra information when tvrage and the NFO fails.

Setting the paths to unrar/ffmpeg/mediainfo is optional, but unrar is recommended for retrieving release names and finding passwords. It's best to get the very latest versions of these utilities, whatever comes as default with your distro is often not good enough.

If you have set the path to unrar, deep rar inspection is recommended.

Compressed headers are recommended if your provider supports XFeature gzip compression. (XFeature GZIP compression, originally by wafflehouse : link on pastebin was removed)

Once you have set all the options, you can enable one or two groups and start with the simple screen script running in single-threaded mode. Look in the misc/update directory; update_binaries.php downloads usenet articles into the local database; update_releases.php attempts to group these articles into releases and create NZB files.

Once you've become more familiar with the application, enable a few more groups and if needed enable multi-threading (with low thread counts i.e. < 5). We do not recommend enabling all the groups unless you have performant hardware and good database tuning knowledge.

If you want an automated way of doing this, you can use one of the scripts in the nix, or tmux folder. The Windows scripts may work.

To clean up the release names, check out fixReleaseNames.php in misc/testing.

For an overview of a complete process, look at the misc/update/nix/screen/sequential/threaded.sh script.

Advanced users may be interested to try the tmux version of nZEDb in either Complete Sequential, Sequential or non-Sequential (fully parallel) mode. Before starting, review the tmux settings in Site-Edit -> Tmux Settings.

Support

There is a web forum were you can search for issues previously encountered by others: https://forums.nzedb.com/

Also on IRC: irc.synirc.net #nZEDb

Note

The nZEDb team have no control over and are not responsible for what is posted on the usenet. Best efforts are made to avoid hazardous content (e.g. virii, malware etc) by nZEDb's automated processess. If you find any objectionable content, please direct any complaints to your usenet provider.

The Team

Kevin123, jonnyboy, Miatrix, zombu2, Codeslave, sinfuljosh, ugo, Whitelighter, Bart39, archer(niel), ThePeePs, ruhllatio, DariusIII

Donate

Licenses

nZEDb is GPL v3. See /docs/LICENSE.txt for the full license.

Other licenses by various software used by nZEDb:

Git.php => MIT and GPL v3

Net_NNTP => W3C

PHPMailer => GNU Lesser General Public License

forkdaemon-php => Barracuda Networks, Inc.

getid3 => GPL v3

password_compat => Anthony Ferrara

rarinfo => Modified BSD

smarty => GNU Lesser General Public v2.1

AmazonProductAPI.php => Sameer Borate

GiantBombAPI.php => MIT

TMDb PHP API class => BSD

TVDB PHP API => Moinax

TVMaze PHP API => JPinkney

Zip file creation class => No license specified.

simple_html_dom.php => MIT

All external libraries will have their full licenses in their respective folders.

Some licenses might have been missed in this document for various external software, they will be included in their respectful folders.

nzedb's People

Contributors

armsby avatar b3rs3rk avatar bart39 avatar bfincher avatar code-slave avatar dariusiii avatar devmem avatar enverex avatar fossil01 avatar greenbender avatar iamprestwood avatar iguyking avatar izolight avatar josh4trunks avatar junkman690 avatar kaibosh avatar kevinlekiller avatar king-cat avatar miatrix avatar mord0r avatar niel avatar nightah avatar sinfuljosh avatar taskym avatar testarea1 avatar thepeeps avatar torkiliuz avatar ugokanain avatar wally73 avatar zetas avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

nzedb's Issues

No book overview being pulled

Book over is being reported as Null at the moment. Also not sure what the worth of having the Amazon sales ranking is as a side note.

nzb downloads are invalid

www/pages/getnzb.php, line 90 includes an "echo $nzbpath" which makes the downloaded nzb files invalid.

86 if (isset($_GET["id"]))
87 {
88  $reldata = $rel->getByGuid($_GET["id"]);
89  $nzbpath = $nzb->getNZBPath($_GET["id"], $page->site->nzbpath, false, $page->site->nzbsplitlevel);
90  echo $nzbpath;
91  
92  if (!file_exists($nzbpath))
93      $page->show404();

Postprocess memory spike

Postprocess seems to be using lots of memory, can't pinpoint it but happening since updating to latest commit today (last updated yesterday). I have memory limit in php set to 512mb but have seen postprocess.php 1 spike to over 5 gb every time it runs.
I am using the tmux script

A few typos in install guide and on the options page

Instead of "sudo aptitude install software-properties-common" I needed to use "sudo apt-get install software-properties-common" in the Ubuntu 12.10 Install Guide.txt

wether should be whether on the site options page advanced section (x3)

Feature Request: Threaded backfill behavior

The threaded backfill behavior changed twice between 2013-05-01 and 2013-05-02 in backfill_threaded.py

2013-05-01 07:08:01 92ee9c5

-   cur.execute("SELECT name from groups where active = 1 ORDER BY first_record_postdate DESC")
+   cur.execute("SELECT name from groups where active = 1 ORDER BY first_record_postdate DESC limit 4")

and again

2013-05-02 05:40:11 (268bff0)

    cur.execute("SELECT name from groups where active = 1 ORDER BY first_record_postdate DESC limit %d" %(int(run_threads[0])))
    datas = cur.fetchall()
 else:
-   cur.execute("SELECT name from groups where active = 1 ORDER BY first_record_postdate DESC limit 4")
+   cur.execute("SELECT name from groups where active = 1 ORDER BY first_record_postdate DESC")

I use only the Backfill Intervals = False and Run sequential = False

With the version from 2013-05-01 I was able to have all groups on the same backfill date, because on every loop only the 4 groups with the newest first_postdate were backfilled.

Now again all groups were updated during 1 loop.

Maybe we can get a counter how many groups will be updated during 1 loop

Groups getting deactivated

I've been experiencing an issue where I have 29 groups activated and pulling in 120 days of backfill (so far) ... I leave and go to work. I come home 10 hours later and see that the TMUX screen isn't doing much anymore. The backfill pane has stopped. I check the active groups and all 29 have been deactivated. This has happened at least 3-5 times recently.

Not sure why it's happening.

update_binaries stall after one loop

Update binaries pane and update_binaries.php itself stall after one loop, has to be killed with ctrl-c to progress further. It started happening suddenly, with no changes except the git updates. I am on latest git update (master).

Book Categorie

Hi,
I would like a category for the books (like 70's, rock, hiphop in music). Is this possible with the amazon api?

Bug - Multiple Delete not working

Deleting multiple releases in List and Cover view doesn't seem to be working. You can only delete a release by using the delete button on the page for that release. Editing multiple releases in List and Cover view works fine, however.

column sort not sensitive to different formats so it sorts wrong

in /admin/group-list.php the first and last post columns are not sensitive to different ways of posting time so they sort wrong.
i.e if you sort descending, 1.1yrs is on top, followed by 1.1d when 1.1yrs should be around the other stuff in the 400 day range. This appears to sort correctly in other spots in the website.

MYSQL error

after running the install and didn't changed anythings exept for enabling tmux things in the admin menu. I'm getting these errors in tmux:

Naamloos

Sickbeard RSS

Latest git upll Sickbeard does not find episodes - but they are listed on the site.

Regexes

Hi,
I have almost every bigger serie many times in the tvrage list.
For every Series i have the same constelations.

Here for the mentalist:
euhd mentalist
ghost of usenet org The Mentalist
TOWN www town ag partner of www ssl news info The Mentalist
TOWNwww town ag partner of www ssl news info The Mentalist
Lords of Usenet Partner of SSL News info presents The Mentalist
The Mentalist
sof the mentalist
presents The Mentalist

It has only found the rage id for "The Mentalist" and "presents The Mentalist"

Its something with the regexes, but i cant change them in nzedb

Not an issue

Nice to see you're creating your own version. Will be following this and will be testing!

Good te see you're highly active ๐Ÿ‘

Feature Request - Script terminated

Hi,
I would like to set
The script will be terminated if no activity for xxx seconds in the postprocessing script for tmux manually in the tmux settings.
Is this possible?

Regex update

Hello,
I have many Releases like this:
presents Da Vincis Demons S01E04 Der Zauberer GERMAN DUBBED DL 720p WebHD x264 TVP

Original
(01/27) - [Lords-of-Usenet] <> presents Da.Vincis.Demons.S01E04.Der.Zauberer.GERMAN.DUBBED.DL.720p.WebHD.x264-TVP -"18465.par2" - 1,96 GB - yEnc

Can you please add the "presents" to the regexes or to the namefix script?

Tmux Monitor Undefined Variables

Getting these at the bottom of monitor.php after running git pull about 20 minutes ago:

Undefined variables: fix_timer (in monitor.php line 491), post_timer (in monitor.php line 504), seq_timer (in monitor.php line 534).

the full path to monitor.php on my install is:

/var/www/nZEDb/misc/update_scripts/nix_scripts/tmux/monitor.php

I'd copy and paste the actual errors but that doesn't appear to work.

Easy db upgrade script

Is there an option to include the newznab update db script from misc/update_scripts ?

It's faster for upgrading (I'm 10 upgrades behind lol)

Backfill to a date

Any chance can have in the tmux admin screen a date to backfill to.

Loved this in script for nn+ as easier to do backfill than adding a day to each group.

Or even a number of days to backfill too as a global variable on the tmux admin screen?

Many thanks.

if tmdb website is down catch the exception

If the api site is down the new tmdb throws and error and kills the processing

Content-Type: text/html; charset=utf-8
Date: Thu, 02 May 2013 19:43:40 GMT
Retry-After: 0
Server: nginx
Via: 1.1 varnish
X-Cache: MISS
X-Varnish: 3284417724
X-Varnish-Sid: tm-v-p-1100
Content-Length: 419
Connection: keep-alive

<title>503 Service Unavailable</title>

Error 503 Service Unavailable

Service Unavailable

Guru Meditation:

XID: 3284417724


Varnish cache server

' in /var/www/nZEDb/www/lib/TMDb.php:869

Typo and error under tmux settings for tcptrack

Says to You may need to run "sudo setcap -v cap_net_raw+ep /usr/bin/tcptrac", to be able to run as user. The pane for this can not be created after tmux starts.

That did not work for me.

I had to sudo su then setcap cap_net_raw=ep /usr/bin/tcptrack

Ubuntu 12.04

namefix script

hi,
in issue #306 i posted about the names who started with "presents".
it doesn't work.

i think the problem is, after the first run of the script the files changes from "blablabla presents goodname" to "presents goodname" and after this change the release is marked that this would not check again in the next run.

i think only when a release name doesn't have a change it should change the status to checked, because it needs sometime mor than one change.

Ebooks not being processed

Virtually no ebooks are being processed and made into releases.

Is this a known problem with small releases or is there a configuration change required?

Thanks

Fix release name

I have 1000 of this... Almost every 2nd german tv release or movie have one of those regex problems:

When this works correct, it should be works perfect with german releases:

Real name:
(01/11) - [Lords-of-Usenet] <> presents NCIS.Los.Angeles.S04E06.Quinn.GERMAN.DUBBED.WebHDRiP.XviD-SOF -"11843.par2" - 357,12 MB - yEnc

After the fix release name script:
presents NCIS Los Angeles S04E06 Quinn GERMAN DUBBED WebHDRiP XviD SOF

Should be:
NCIS Los Angeles S04E06 Quinn GERMAN DUBBED WebHDRiP XviD SOF

Real name:

GOU<< Secret.Service.Schutzobjekt.UN-Gebaeude.GERMAN.DOKU.AC3.HDTV.720p.x264-NVA >>www.SSL-News.info< - (01/33) - "nva-sese-schob-720p.par2" yEnc

Name in nzedb:
GOU Secret Service Schutzobjekt UN Gebaeude GERMAN DOKU AC3 HDTV 720p x264 NVA www SSL News info nva sese schob 720p

Should be:
Secret Service Schutzobjekt UN Gebaeude GERMAN DOKU AC3 HDTV 720p x264 NVA

Real name:
<www.town.ag > Russische.Frauen.Ihre.Maenner.ihre.Macht.German.DOKU.720p.HDTV.x264-UTOPiA [18/19] - "Russische.Frauen.Ihre.Maenner.ihre.Macht.German.DOKU.720p.HDTV.x264-UTOPiA.vol3+4.par2" - 735,53 MB - yEnc

Name in nzedb:
TOWNwww town ag partner of www ssl news info Russische Frauen Ihre Maenner ihre Macht German DOKU 720p HDTV x264 UTOPiA Russische Frauen Ihre Maenner ihre Macht German DOKU 720p HDTV x264 UTOPiA

Should be:
Russische Frauen Ihre Maenner ihre Macht German DOKU 720p HDTV x264 UTOPiA

Real name:

ghost-of-usenet.org<< Alias.Die.Agentin.S02E21.Zweites.Double.GERMAN.DL.DUBBED.1080p.WebHD.x264-TVP >>www.SSL-News.info> - "tvp-alias-s02e21-1080p.r36" yEnc

Name in nzedb:
ghost of usenet org Alias Die Agentin S02E21 Zweites Double GERMAN DL DUBBED 1080p WebHD x264 TVP www SSL News info tvp alias

Should be:
Alias Die Agentin S02E21 Zweites Double GERMAN DL DUBBED 1080p WebHD x264 TVP

New Category

I sit possible to add a new category WEB-DL in the TV directory.

And then fill this with just web-dl tv releases.

Works in NN+ Like this :

INSERT INTO category (ID, title, parentID, status) VALUES ('5010', 'WEB-DL', '5000', '1')

This inserts category in DB

And then in category.php

const CAT_TV_WEBDL = 5010;

public function isHDTV($releasename)
 {
     if (preg_match('/1080P.WEB.DL|720P.WEB.DL/i', $releasename))
            {
                    $this->tmpCat = Category::CAT_TV_WEBDL;
                    return true;
            }
    if (preg_match('/1080|720/i', $releasename))
     {
         $this->tmpCat = Category::CAT_TV_HD;
         return true;
     }

     return false;
 }

Apologise if not the place to post request, new to github

Cheers

Non-cross posted duplicate releases

I'm seeing a lot of duplicate releases in my db that appear to be individually posted to multiple groups, rather than cross-posted. Do you have any plans to weed these out?

Here's a query that returns all records which are duplicates:

SELECT t.,(select count() from releases as tt where tt.name=t.name) as count FROM releases as t where (select count(*) from releases as tt where tt.name=t.name) > 1 order by count desc

make who can see nzb download stats configureable

For small, personal nn installs, it would be great to have download stats configurable to either turn them off or make them per user. There's no need for a small group of people sharing an nn install to know what others have downloaded. (aka, I don't want to know what weird crap my room mates are into and my userbase isn't big enough to give them plausible deniability).

magazines and comics in the ebook section

In the latest round of updates I've suddenly started getting magazines in the ebook section. the magazine section only has two or three magazines in it. comics also appear to be getting dumped in the book section as well.

skip: is foreign

hi,
is it possible to make a checkbox to skip the check if foreign tv and check if foreign movie part in category.php.
i want to index the foreign tv shows and movies normal in sd, hd, like the english ones and now i have to edit the category.php after every update again.

If you can do this, i would be so happy ;P

Feature request - Blacklist catagories

Is it possible to add an option to remove catagories that are marked disabled or better yet not process them? There used to be a script floating around for newznab that would remove them after processing, but since has disappeared.

If meta lookups are disabled can't browse menu items

If meta lookups are disabled when you click on a menu item it should not display the meta search options it should just list the contents of the category. This is how TV menu is working or if you click on a category link.

Example, if Lookup Movies is set to No clicking the Movies menu should Browse the contents of the movie category and not display the meta search options. Similar for Lookup Audio and Lookup ebooks and so on you only get enhanced meta search options displaying.

For consistency perhaps if you have Lookup Movies set to Yes it should display the meta search options and the first page of browse releases underneath. If you have Lookup Movies set to No meta search options are not displayed but the browse releases is shown.

This means you are not relying on the meta data as currently it is quite difficult to view releases with the current setup.

Thanks

Select backfill groups

Might be useful, being able to enable/disable backfill selected groups.

Some pople, like me, may wish for a certain groups to be backfilled first, then others at a later date.

Cheers

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.