kiralycraft / filelistcrawlerserverversion Goto Github PK

A facut buba ... :)

[17:21] Clearing Torrent Cache ...
[17:21] Getting the 1st torrent page ...
[17:21] Response Code : 200
[17:21] Grabbing torrent data ...
[17:21] Analyzing results ...
[17:21] Asking Transmission how much free space there is.
[17:21] Got response: 255385600 bytes = 255 MB
[17:21] Asking Transmission how much free is required for current downloading torrents.
[17:21] Got response: 0 bytes = 0 MB
[17:21] ==========================================
[17:21] Gloria.1980.720p.BluRay.AAC.2.0.x264-SPEED - Checking requirements ...
[17:21] Not enough space to download Gloria.1980.720p.BluRay.AAC.2.0.x264-SPEED. It requires 10.99 GB
[17:21] Will ask Transmission to do a cleanup.
Removing torrent instance from Transmission
Removing associated files for:Hunter.Street.S01.1080p.NICK.WEBRip.AAC2.0.x264-TVSmash
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:657)
at java.util.ArrayList.get(ArrayList.java:433)
at com.google.gson.JsonArray.get(JsonArray.java:92)
at com.kiralycraft.tapidiy.Connection.getTorrentFiles(Connection.java:691)
at com.kiralycraft.tapidiy.Connection.cleanup(Connection.java:279)
at com.kiralycraft.fcsv.RunThread.parseTorrentData(RunThread.java:604)
at com.kiralycraft.fcsv.RunThread.run(RunThread.java:150)
[17:21] ERROR OCCURED! null
java.lang.NullPointerException
at com.kiralycraft.tapidiy.Connection.cleanup(Connection.java:280)
at com.kiralycraft.fcsv.RunThread.parseTorrentData(RunThread.java:604)
at com.kiralycraft.fcsv.RunThread.run(RunThread.java:150)

Retest, pentru siguranta

[17:33] Clearing Torrent Cache ...
[17:33] Getting the 1st torrent page ...
[17:33] Response Code : 200
[17:33] Grabbing torrent data ...
[17:33] Analyzing results ...
[17:33] Asking Transmission how much free space there is.
[17:33] Got response: 28562792448 bytes = 28562 MB
[17:33] Asking Transmission how much free is required for current downloading torrents.
[17:33] Got response: 0 bytes = 0 MB
[17:33] ==========================================
[17:33] Gotham.S04.720p.BluRay.DD5.1.x264-SbR - Checking requirements ...
[17:33] Not enough space to download Gotham.S04.720p.BluRay.DD5.1.x264-SbR. It requires 46.21 GB
[17:33] Will ask Transmission to do a cleanup.
Removing torrent instance from Transmission
Removing associated files for:Avast!.Internet.SecurityPremier.Antivirus.v19.1.2360.Build.19.1.4142.Avast!.Internet.SecurityPremier.Antivirus.v19.1.2360.Build.19.1.4142
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:657)
at java.util.ArrayList.get(ArrayList.java:433)
at com.google.gson.JsonArray.get(JsonArray.java:92)
at com.kiralycraft.tapidiy.Connection.getTorrentFiles(Connection.java:691)
at com.kiralycraft.tapidiy.Connection.cleanup(Connection.java:279)
at com.kiralycraft.fcsv.RunThread.parseTorrentData(RunThread.java:604)
at com.kiralycraft.fcsv.RunThread.run(RunThread.java:150)
[17:33] ERROR OCCURED! null
java.lang.NullPointerException
at com.kiralycraft.tapidiy.Connection.cleanup(Connection.java:280)
at com.kiralycraft.fcsv.RunThread.parseTorrentData(RunThread.java:604)
at com.kiralycraft.fcsv.RunThread.run(RunThread.java:150)

[13:58] Clearing Torrent Cache ...
[13:58] Getting the 1st torrent page ...
[13:58] Response Code : 302
[13:58]
[13:58] =====WARNING=====
[13:58] Response code was supposed to be 200, it was 302
[13:58] Check your Filelist.ro login data, it might not be correct
[13:58] =================

Azi, in timp ce aruncam un ochi peste codul sursa am observat o mica inadvertenta intre spatiul liber raportat de transmission si cel raportat de aplicatie.

• Sistem Operare: Windows 10
• Spatiu liber raportat in transmision: 622.4 Gb
• Spatiu dat de aplicatie:

[14:50:05] In MB
[14:50:05] /1024 = Got response: 622366121984 bytes = 593534 MB
[14:50:05] /1000 = Got response: 622366121984 bytes = 622366 MB
[14:50:05] In GB
[14:50:05] /1024 = Got response: 622366121984 bytes = 579 GB
[14:50:05] /1000 = Got response: 622366121984 bytes = 622 GB
[14:50:05] Sau utilizand o metoda pentru a replica suma din transmission
[14:50:05] Free space in download location: 622.4 GB

Cod utilizat pentru obtinerea datelor de mai

long freeSpaceOnTransmission = connection.getFreeSpaceAndDownDir().getKey();
log("In MB");
log("/1024 = Got response: "+freeSpaceOnTransmission+" bytes = "+freeSpaceOnTransmission/1024/1024+" MB");
log("/1000 = Got response: "+freeSpaceOnTransmission+" bytes = "+freeSpaceOnTransmission/1000/1000+" MB");
log("In GB");
log("/1024 = Got response: "+freeSpaceOnTransmission+" bytes = "+freeSpaceOnTransmission/1024/1024/1024+" GB");
log("/1000 = Got response: "+freeSpaceOnTransmission+" bytes = "+freeSpaceOnTransmission/1000/1000/1000+" GB");
log("Sau utilizand o metoda pentru a replica suma din transmission");
double freeSpace2 = round((((connection.getFreeSpaceAndDownDir().getKey()/1000.0)/1000.0)/1000.0),1);
log("Free space in download location: "+freeSpace2+" GB");

Metoda utilizata

	public static double round(double value, int places) {
	    if (places < 0) throw new IllegalArgumentException();
	    long factor = (long) Math.pow(10, places);
	    value = value * factor;
	    long tmp = Math.round(value);
	    return (double) tmp / factor;
	}

Dupa cum spuneam comportamentul a fost observat pe windows, nu am verificat pe linux :(

java -jar FilelistCrawlerServerVersion.jar
THIS ONLY WORKS IF PRESENT ON THE SAME PHYSICAL MACHINE AS TRANSMISSION!
You can type "help" to see various commands.
[01:21] Starting in progress
[01:21] Connecting to Transmission ...
[01:21] Connection to Transmission successfull
[01:21] Logging into Filelist.ro with username and password
[01:21] Acquiring CFDUID ...
[01:21] Error! null
[01:21] Clearing Torrent Cache ...
[01:21] Getting the 1st torrent page ...
[01:21] ERROR OCCURED! Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.doTunneling(Unknown Source)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source)
at com.kiralycraft.fcsv.RunThread.getBrowsePage(RunThread.java:408)
at com.kiralycraft.fcsv.RunThread.run(RunThread.java:147)

mentionez rulez prin run.bat specificat in tutorial si ca am acel .jar exact in directorul cu .bat

Fisierele descarcate de transmission au urmatoarele permisiuni

drwxr-sr-x  2 debian-transmission mosfet 4096 Jan  9 07:58 37.Days.S01.720p.BluRay.x264-SPLiTSViLLE
drwxr-sr-x  2 debian-transmission mosfet 4096 Jan  9 07:57 MAD.S02.720p.WEB-DL.AAC2.0.H264-NTb
drwxr-sr-x 25 debian-transmission mosfet 4096 Jan  8 23:27 Murder.She.Wrote.S01.1080p.AMZN.WEB-DL.DD2.0.H.264-MZABI
drwxr-sr-x  2 debian-transmission mosfet 4096 Jan  9 06:48 Riphagen.2017.720p.BluRay.DD2.0.x264-SbR

cu alte cuvinte :

• owner este : debian-transmission
• permisiunile pe fisiere sunt drwxr-sr-x , adica owner : RWX, grup R-X, other R-X
• scriptul este executat de userul normal sau root

Cred ca este evidenta problema si din pacate tine mai mult de transmission decat de aplicatia pentru seed.
Stergerea remote sau prin transmission merge fara probleme fiind realizata de userul debian-transmission.
Revin cu o solutie, in speranta ca voi gasi una.

1. Inadvertenta coment si cod
   if ((ti.getUploadRatio()>1.0f || ti.getTimeAddedSeconds()>172800)) //daca are 48 de ore si nu o mai facut upload de o zi
   codul verifica daca are ratia mai mare de 1 SAU timpul de seed de cel putin 48h - ceea ce este corect.

2. Ar trebui verificat si daca a mai facut in ultima ora, evitand astfel situatia cand se sterge un torrent care indeplineste conditiile dar care face upload in prezent.
   if ((ti.getUploadRatio()>1.0f || ti.getTimeAddedSeconds()>172800) && (ti.getLastActivitySeconds()>3600)) //daca are 48 de ore la seed sau ratie >1 si nu a mai facut trafic (upload) de o ora

ar mai fi o chestie sa verificam daca viteza medie de transfer in ultima ora depaseste X valoare, astfel se poate sterge un torrent care face uploade vu o viteza foarte mica.

Just my 2 cents

L.E.

Devin carcotas -- si vin cu o observatie fara a oferi solutia concreta.
Cand este evaluat un torent pentru stergere in ideea de a evita hit and run se verifica ratia si timpul trecut de la adaugarea lui (getTimeAddedSeconds).
Din ce stiu FL ia in considerare doar timpul de seed (dupa ce ai 100% din torrent).
Ca atare propun sa extragem si campul secondsSeeding pentru torentele din transmission, urmand ca valoare acestuia sa fie utilizata in cadrul evaluari pentru stergere.
Modificarea nu este una majora dar, din pacate, nu am timp sa o implementez acum.
Sper sa vin cu o propunere concreta maine.

P.S. Pana in prezent nu am asteptat 48h dupa nici un torent, toate au fost sterse (automat sau manual) fiind validata conditia de ratie >1.

Incercand sa contribui cu ceva cocncret :)

Pe scurt :

• Creat o metoda care calculeaza spatiul necesar pentru torentele ce nu s-au descarcat complet
  • Nume: getDownloadingTorrentsSpaceNeeded
  • Clasa: Connection
• Modificat torrent info pentru a include si cat a fost descarcat din torent
  • Nume parametru: haveValid
  • Clasa: Connection
• Modificat verificarile realizate la adaugarea unui torent ( spatiu raportat de transmission - spatiu necesar pentru torentele ce se descarca)
  • Clasa : RunThread

Fisierele pentru download :
Modifiedfiles.zip

Sau pastebin :

• Connection : https://pastebin.com/rJymuh8t
• RunThread : https://pastebin.com/THY95Bc6

L.E. clasa Runthread linia 605 trebuie sters long altfel da eroare de duplicate variables, SORRY

Multumiri Lorelai05 pentru debugging and feedback. O lista cu ce e de facut. Nu stiu exact cum merg issues, descoperim.

• Pe Linux, openjdk-8 nu include JavaFX. Folosit doar pentru Pair. Useless, facem ceva clasa noua.
• "Remote delete". Removed ca si workaround la problema de mai jos. Re-add cu un toggle switch la configurare.
• Transmission nu garanteaza ca torrentul a fost sters de pe disk in modul "remote delete". Fix, workaround?
• Nici workaround-ul nu merge mereu, bata-l vina.
• Nu se pot modifica setarile usor odata ce a fost configurat clientul. Ar fi frumos sa fie o optiune in meniu.
• La 3 IP-uri diferite, Filelist invalideaza cookie-urile de login. Asta e raportat ca si "wrong login data". Daca e identificabil comportamentul, add custom message strict pentru asta.
• Handle situatii de "Transmission not responding" (Windows only). Cand se intampla, raporteaza date aiurea (invalid storage path, free space = 0) . Executia continua de parca nu s-ar fi intamplat nimic.
• Daca nu e activat modul 2 de preallocation, Transmission raporteaza spatiul liber aiurea. Ca urmare, cum imparti 4 mere la 10 persoane? Problema majora.
• Adauga abilitatea de a redenumi un torrent, concatenand NODELETE oriunde in nume. Astfel, acel/acele torrent(e) nu vor fi sterse automat la cleanup.
• Handle situatii tip "Piece is corrupt" si "Unregistered torrent". Suntem asa rapizi la download ca unele torrente se sterg de pe site dar raman la descarcat, ca urmare sunt useless.
• Thx automat la torrente

Poți face un tutorial pentru ruTorrent? Văd că s-a adăugat și opțiunea pentru ruTorrent, dar nu știu să configurez exact ruTorrent-ul.

Precum stii filelist.ro nu mai este ci pe filelist.io. Am incercat sa schimb domeniul dar imi da eroare 302 ca response si nu ia torrentele deloc.

[17:58] The.Wire.S04.1080p.BluRay.DD5.1.x264-CtrlHD - Checking requirements ...
[17:58] With this torrent, out total used space would be: 1014.869728528 GB.
[17:58] Not enough space to download The.Wire.S04.1080p.BluRay.DD5.1.x264-CtrlHD. It requires 75.32 GB
[17:58] Will ask Transmission to do a cleanup.
Should be removed: Utopia.AU.S03.720p.HDTV.x264-BTN
Should be removed: Dr.Strangelove.1964.1080p.BluRay.AC3.x264-HDC
Should be removed: Grease 1978 720p Blu-Ray DTS x264-REPTiLE
Should be removed: First.Man.2018.REPACK.1080p.UHD.BluRay.DD+7.1.HDR.x265.DON.mkv
[17:59] ERROR OCCURED! Comparison method violates its general contract!
java.lang.IllegalArgumentException: Comparison method violates its general contract!
at java.util.TimSort.mergeLo(TimSort.java:777)
at java.util.TimSort.mergeAt(TimSort.java:514)
at java.util.TimSort.mergeCollapse(TimSort.java:441)
at java.util.TimSort.sort(TimSort.java:245)
at java.util.Arrays.sort(Arrays.java:1512)
at java.util.ArrayList.sort(ArrayList.java:1462)
at java.util.Collections.sort(Collections.java:175)
at com.kiralycraft.tapidiy.Connection.cleanup(Connection.java:267)
at com.kiralycraft.fcsv.torrentinterfaces.TransmissionInterface.cleanup(TransmissionInterface.java:60)
at com.kiralycraft.fcsv.RunThread.parseTorrentData(RunThread.java:635)
at com.kiralycraft.fcsv.RunThread.run(RunThread.java:163)

Dupa o perioada de vreo 3 zile non stop de folosire, a reusit sa imi umple un hdd de cativa TB.

Problema insa a aparut recent, cand spatiul disponibil este insuficient..

Deci dupa cum spuneam, ruleaza normal pana apare asta...

[00:11] Avg upload speed: 1.4739511 MB/s. Upload potential 23%
[00:12] Avg upload speed: 3.028912 MB/s. Upload potential 48%
[00:12] Avg upload speed: 2.4762194 MB/s. Upload potential 39%
[00:12] Clearing Torrent Cache ...
[00:12] Getting the 1st torrent page ...
[00:12] Response Code : 200
[00:12] Grabbing torrent data ...
[00:12] Analyzing results ...
[00:12] Asking Transmission how much free space there is.
[00:12] Got response: 2490368 bytes = 2 MB
[00:12] ==========================================
[00:12] The.Curse.1987.720p.BluRay.x264-CREEPSHOW - Checking requirements ...
[00:12] Not enough space to download The.Curse.1987.720p.BluRay.x264-CREEPSHOW. It requires 4.41 GB
[00:12] Will ask Transmission to do a cleanup.
Removing torrent instance from Transmission
Removing associated files for:UFC.The.Evolution.of.CM.Punk.S01.720p.HDTV.DD5.1.MPEG2-NTb
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(Unknown Source)
at java.util.ArrayList.get(Unknown Source)
at com.google.gson.JsonArray.get(JsonArray.java:92)
at com.kiralycraft.tapidiy.Connection.getTorrentFiles(Connection.java:613)
at com.kiralycraft.tapidiy.Connection.cleanup(Connection.java:215)
at com.kiralycraft.fcsv.RunThread.parseTorrentData(RunThread.java:589)
at com.kiralycraft.fcsv.RunThread.run(RunThread.java:150)
[00:12] ERROR OCCURED! null
java.lang.NullPointerException
at com.kiralycraft.tapidiy.Connection.cleanup(Connection.java:216)
at com.kiralycraft.fcsv.RunThread.parseTorrentData(RunThread.java:589)
at com.kiralycraft.fcsv.RunThread.run(RunThread.java:150)
[00:22] Clearing Torrent Cache ...
[00:22] Getting the 1st torrent page ...
[00:22] Response Code : 200
[00:22] Grabbing torrent data ...
[00:22] Analyzing results ...
[00:22] Asking Transmission how much free space there is.
[00:23] Got response: 2490368 bytes = 2 MB
[00:23] ==========================================
[00:23] The.Curse.1987.720p.BluRay.x264-CREEPSHOW - Checking requirements ...
[00:23] Not enough space to download The.Curse.1987.720p.BluRay.x264-CREEPSHOW. It requires 4.41 GB
[00:23] Will ask Transmission to do a cleanup.
Removing torrent instance from Transmission
Removing associated files for:Hospital.S01.720p.iP.WEBRip.AAC2.0.H.264-RTN
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(Unknown Source)
at java.util.ArrayList.get(Unknown Source)
at com.google.gson.JsonArray.get(JsonArray.java:92)
at com.kiralycraft.tapidiy.Connection.getTorrentFiles(Connection.java:613)
at com.kiralycraft.tapidiy.Connection.cleanup(Connection.java:215)
at com.kiralycraft.fcsv.RunThread.parseTorrentData(RunThread.java:589)