Git Product home page Git Product logo

mbruel / ngpost Goto Github PK

View Code? Open in Web Editor NEW
164.0 16.0 31.0 30.83 MB

Command Line (or minimalist GUI) usenet poster for binaries developped in C++/QT designed to be as fast as possible and offer all the main features to post data easily and safely. Releases for Linux, Windows and MacOS are available.

License: GNU General Public License v3.0

C++ 96.96% QMake 1.71% Shell 0.88% Dockerfile 0.17% C 0.28%
nzb usenet usenet-posters cmd gui obfuscation multithread nntp qt5 cpp11

ngpost's Introduction

ngPost

ngPost v4.16

Pour la version Française cliquez ici ;)

Here is an invite for the brand new discord server, feel free to join if you've questions or if interested in beta testing the new features before they're released ;)

Command Line and sexy GUI Usenet poster for binaries developped in C++11/Qt5
it is designed to be as fast as possible and offer ALL the main features to post data easily and safely.
it can compress (using your external rar binary) and generate the par2 before posting!
it has a posting queue to allow you to prepare several posts (especially using the GUI using Tabs).
it packs the next Post while uploading the current one (compression + par2 generation)
it can automate posts by scanning folder(s) and posting each file/folder individually after having them compressed (with a potential random archive name and password) and generated the par2!
it can monitor folder(s) to post each new file/folder individually after having them compressed.
it can auto delete files/folders once posted (only for monitoring with the HMI and with both --auto and --monitor in cmd)
it can execute a COMMAND or script after each Post (cf NZB_POST_CMD in the configuration file)
it can switch off the computer when all the posts are finished
it is translated in Chinese, Dutch, English, French, German, Portuguese and Spanish.

ngPost_v4.3

Releases are availables for: Linux 64bit, Windows (both 32bit and 64bit), MacOS and Raspbian (RPI 4). Soon for Android then iOS...

Here are the main features and advantages of ngPost:

  • full obfuscation of the Article Header : the Subject will be a UUID (as the msg-id) and a random Poster will be used. Be Careful, using this, you won't be able to find your post on Usenet (or any Indexers) if you lose the NZB file. But using this method is completely safe, no need to obfuscate your files or even tp use a password.
  • Posting Queue: you can prepare several posts while you're posting something. Of course you can cancel pending posts ;)
  • Post automation: scan a folder and post each file/folder after compression. (cf --auto option in CMD and the "Auto Posting" tab on the GUI)
  • compress using RAR or 7zip (external command) with random name obfuscation and password and generate par2 before posting
  • write history of posts in a csv file: so you can get the date, file name, size, upload speed but most important the archive name and password in simple excel spreadsheet ;)
  • par2cmdline is included in the package but you can use another tool (like Multipar) if you wish using the PAR2_CMD and PAR2_ARGS keywords in the config file
  • support multiple files and multiple folders
  • support several servers (using config file or the HMI) with each several connections (supporting ssl)
  • spread those connection on several threads. By default the number of cores of the station but you can set the number if you fancy.
  • prepare the Articles on the main Thread (yEnc encoding) so the upload threads are always ready to write when they can. (each connections has 2 Articles ready in advance)
  • use asynchronous sockets in the upload threads
  • limit the disk access to the minimum by opening the files to post only once in the main Thread and processing them sequentially one by one (no need to open several files at the same time, the Articles will be spread to all the connections)
  • generate a random poster (from) but can use a fixed one if desired
  • generate the nzb file
  • handle interruption : if a post is stopped, the nzb is generated with the files that have been posted and it will list the files that havn't been posted so you can repost only those ones and then manually concatenate the nzb files. (in CMD, the application would close properly)
  • add meta in the header of the nzb (typically for a password)
  • Retry to post Articles with a different UUID in case of error
  • Try to reconnect if there is an error on a socket (same Retry parameter than for the articles)
  • ...

How to build

Dependencies:

  • build-essential (C++ compiler, libstdc++, make,...)
  • qt5-default (Qt5 libraries and headers)
  • qt5-qmake (to generate the moc files and create the Makefile)
  • libssl (v1.0.2 or v1.1) but it should be already installed on your system

Build:

  • go to the src folder
  • qmake
  • make

Easy! it should have generate the executable ngPost
you can copy it somewhere in your PATH so it will be accessible from anywhere

As it is made in C++/QT, you can build it and run it on any OS (Linux / Windows / MacOS / Android)
releases have only been made for Linux x64 and Windows x64 (for 7 and above) and MacOS (
in order to build on other OS, the easiest way would be to install QT and load the project in QtCreator

How to use it in command line

Syntax: ngPost (options)* (-i  | --auto  | --monitor )+
	--help             : Help: display syntax
	-v or --version    : app version
	-c or --conf       : use configuration file (if not provided, we try to load $HOME/.ngPost)
	--disp_progress    : display cmd progressbar: NONE (default), BAR or FILES
	-d or --debug      : display extra information
	--fulldebug        : display full debug information
	-l or --lang       : application language
	--check            : check nzb file (if articles are available on Usenet) cf https://github.com/mbruel/nzbCheck
	-q or --quiet      : quiet mode (no output on stdout)

// automated posting (scanning and/or monitoring)
	--auto             : parse directory and post every file/folder separately. You must use --compress, should add --gen_par2, --gen_name and --gen_pass
	--monitor          : monitor directory and post every new file/folder. You must use --compress, should add --gen_par2, --gen_name and --gen_pass
	--rm_posted        : delete file/folder once posted. You must use --auto or --monitor with this option.

// quick posting (several files/folders)
	-i or --input      : input file to upload (single file or directory), you can use it multiple times
	-o or --output     : output file path (nzb)
	-x or --obfuscate  : obfuscate the subjects of the articles (CAREFUL you won't find your post if you lose the nzb file)
	-g or --groups     : newsgroups where to post the files (coma separated without space)
	-m or --meta       : extra meta data in header (typically "password=qwerty42")
	-f or --from       : poster email (random one if not provided)
	-a or --article_size: article size (default one: 716800)
	-z or --msg_id     : msg id signature, after the @ (default one: ngPost)
	-r or --retry      : number of time we retry to an Article that failed (default: 5)
	-t or --thread     : number of Threads (the connections will be distributed amongs them)
	--gen_from         : generate a new random email for each Post (--auto or --monitor)

// for compression and par2 support
	--tmp_dir          : temporary folder where the compressed files and par2 will be stored
	--rar_path         : RAR absolute file path (external application)
	--rar_size         : size in MB of the RAR volumes (0 by default meaning NO split)
	--rar_max          : maximum number of archive volumes
	--par2_pct         : par2 redundancy percentage (0 by default meaning NO par2 generation)
	--par2_path        : par2 absolute file path (in case of self compilation of ngPost)
	--auto_compress    : compress inputs with random name and password and generate par2 (equivalent of --compress --gen_name --gen_pass --gen_par2)
	--compress         : compress inputs using RAR or 7z
	--gen_par2         : generate par2 (to be used with --compress)
	--rar_name         : provide the RAR file name (to be used with --compress)
	--rar_pass         : provide the RAR password (to be used with --compress)
	--gen_name         : generate random RAR name (to be used with --compress)
	--gen_pass         : generate random RAR password (to be used with --compress)
	--length_name      : length of the random RAR name (to be used with --gen_name), default: 17
	--length_pass      : length of the random RAR password (to be used with --gen_pass), default: 13
	--rar_no_root_folder: Remove root (parent) folder when compressing Folders using RAR

// you can provide servers in one string using -S and/or split the parameters for ONE SINGLE server (this will overwrite the configuration file)
	-S or --server     : NNTP server following the format (<user>:<pass>@@@)?<host>:<port>:<nbCons>:(no)?ssl
	-h or --host       : NNTP server hostname (or IP)
	-P or --port       : NNTP server port
	-s or --ssl        : use SSL
	-u or --user       : NNTP server username
	-p or --pass       : NNTP server password
	-n or --connection : number of NNTP connections

Examples:
  - with monitoring: ngPost_v4.16_cmd-x86_64.AppImage --monitor /data/folder1 --monitor /data/folder2 --auto_compress --rm_posted --disp_progress files
  - with auto post: ngPost_v4.16_cmd-x86_64.AppImage --auto /data/folder1 --auto /data/folder2 --compress --gen_par2 --gen_name --gen_pass --rar_size 42 --disp_progress files
  - with compression, filename obfuscation, random password and par2: ngPost_v4.16_cmd-x86_64.AppImage -i /tmp/file1 -i /tmp/folder1 -o /nzb/myPost.nzb --compress --gen_name --gen_pass --gen_par2
  - with config file: ngPost_v4.16_cmd-x86_64.AppImage -c ~/.ngPost -m "password=qwerty42" -f [email protected] -i /tmp/file1 -i /tmp/file2 -i /tmp/folderToPost1 -i /tmp/folderToPost2
  - with all params:  ngPost_v4.16_cmd-x86_64.AppImage -t 1 -m "password=qwerty42" -m "metaKey=someValue" -h news.newshosting.com -P 443 -s -u user -p pass -n 30 -f [email protected]  -g "alt.binaries.test,alt.binaries.test2" -a 64000 -i /tmp/folderToPost -o /tmp/folderToPost.nzb

If you don't provide the output file (nzb file), we will create it in the nzbPath with the name of the first file or folder given in the command line.
so in the second example above, the nzb would be: /tmp/file1.nzb

Configuration file and keywords that are only in config

The default configuration file for Linux and Mac environment is: ~/.ngPost (no conf extension)
If you wish, you can use another one in command line with the -c option.
Here the example to follow.

Most configuration keywords can be used in command line but few of them, are only in the config file:

  • MONITOR_NZB_FOLDERS : each monitoring post would go in its own folder created in nzbPath (Req/Issue #15)
  • inputDir : Default folder to open to select files from the HMI (it can be updated if you change the Auto Dir in the HMI and the SAVE)
  • POST_HISTORY: csv file where all successful post will append the date, the file name, its size, the upload speed, the archive name and its password
  • GROUP_POLICY: to define the policy for posting when several Groups are provided (on all, one per post or one per file)
  • NZB_RM_ACCENTS: remove accents and special characters from the nzb file names
  • AUTO_CLOSE_TABS: close Quick Post Tabs when posted successfully (for the GUI)
  • RESUME_WAIT: Time to wait (seconds) before trying to resume a Post automatically in case of loss of Network (min: 30)
  • NO_RESUME_AUTO: stop a post when you loose the network
  • ** PREPARE_PACKING**: when several posts are queued, prepare the packing of the next post while uploading the current one

The following ones are for experimented posters:

  • RAR_EXTRA : to customize the rar command (no need to put the 'a', '-idp' or '-r'). No need to use it for 7-zip except if you wish to change the compession level.
  • PAR2_CMD : to change the par2 generator and be able to use Parpar or Multipar if you wish. (par2cmdline is the default embedded generator)
  • PAR2_ARGS : to customize the par2 command, especially if you choose to use another one than the default par2cmdline
  • TMP_RAM: use a temporary folder with size constraint, typically a tmpfs partition, for posting files that would fit (otherwise TMP_DIR is used)
  • TMP_RAM_RATIO: Ratio used on the source files size to compensate the par2 generation

Linux 64bit portable release (compiled with Qt v5.12.6)

if you don't want to build it and install the dependencies, you can also the portable release that includes everything.

  • download ngPost_v4.16-x86_64.AppImage or ngPost_v4.16_cmd-x86_64.AppImage for pure CMD (the first version doing both CMD and GUI)
  • chmod 755 ngPost_v4.16-x86_64.AppImage
  • launch it using the same syntax than describe in the section above
  • if you wish to keep the configuration file, edit the file ~/.ngPost using this model (don't put the .conf extension)

PS: for older system with GLIBC < 2.24, here is a version compiled on Debian8 with GLIBC 2.19 and Qt v5.8.0: ngPost_v4.16-x86_64_debian8.AppImage

Raspbian release (armhf for Raspberry PI)

  • download ngPost_v4.16-armhf.AppImage
  • chmod 755 ngPost_v4.16-armhf.AppImage
  • launch it using the same syntax than describe in the section above
  • if you wish to keep the configuration file, edit the file ~/.ngPost using this model (don't put the .conf extension)

As rar is not available on RPI, 7zip support has been added. Use RAR_PATH to point on the 7z and RAR_EXTRA to have 7zip options. You should have something like this:

## RAR or 7zip absolute file path (external application)
## /!\ The file MUST EXIST and BE EXECUTABLE /!\
## this is set for Linux environment, Windows users MUST change it
#RAR_PATH = /usr/bin/rar
RAR_PATH = /usr/bin/7z

## RAR EXTRA options (the first 'a' and '-idp' will be added automatically)
## -hp will be added if you use a password with --gen_pass, --rar_pass or using the HMI
## -v42m will be added with --rar_size or using the HMI
## you could change the compression level, lock the archive, add redundancy...
#RAR_EXTRA = -ep1 -m0 -k -rr5p
RAR_EXTRA = -mx0 -mhe=on

Windows installer

  • just use the packager ngPost_v4.16_x64_setup.exe or ngPost_v4.16_x86_setup.exe for the 32bit version
  • edit ngPost.conf (in the installation folder) to add your server settings (you can put several).
  • launch ngPost.exe (GUI version)
  • or you can use it with the command line: ngPost.exe -i "your file or directory"

if you prefer, you can give all the server parameters in the command line (cf the above section)
By default:

  • ngPost will load the configuration file 'ngPost.conf' that is in the directory
  • it will write the nzb file inside this directory too. (it's name will be the one of the latest input file in the command line)

Know Issue with non SSL support: you may need to install msvc2015 redistribuables as libssl depends on its APIs

MacOS release built on High Sierra (v10.13)

  • download ngPost_v4.16.1.dmg
  • launch it using the same syntax than describe in the section above
  • if you wish to keep the configuration file, edit the file ~/.ngPost using this model (don't put the .conf extension)

Alternatives

A list of Usenet posters from Nyuu github can be found here.

Licence

ngPost is published under licence GPL v3

//========================================================================
//
// Copyright (C) 2020 Matthieu Bruel 
// This file is a part of ngPost : https://github.com/mbruel/ngPost
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, version 3..
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see 
//
//========================================================================

Questions / Issues / Requests

  • if you've any troubles to build or run ngPost, feel free to drop me an email
  • if you've some comments on the code, any questions on the implementation or any proposal for improvements, I'll be happy to discuss it with you so idem, feel free to drop me an email
  • if you'd like some other features, same same (but different), drop me an email ;)

Here is my email: [email protected]

Supported Languages

For now, ngPost is translated in Chinese, English, French, German, Portuguese and Spanish.
If you'd like to translate ngPost in your language, it's easy to do (there is a nice GUI for that: QtLinguist), please get in touch with me ([email protected])

Thanks

  • Uukrull for his intensive testing and feedbacks and for building all the MacOS packages.
  • awsms for his testing on proper server with a 10Gb/s connection that helped to improve ngPost's upload speed and the multi-threading support
  • animetosho for having developped ParPar, the fasted par2 generator ever!
  • demanuel for the dev of NewsUP that was my first poster
  • noobcoder1983, tensai then yuppie for the German translation
  • tiriclote for the Spanish translation
  • hunesco for the Portuguese translation
  • Peng for the Chinese translation
  • all ngPost users ;)

Donations

I'm Freelance nowadays, working on several personal projects, so if you use the app and would like to contribute to the effort, feel free to donate what you can.

ex0days or in Bitcoin at this address: 3BGbnvnnBCCqrGuq1ytRqUMciAyMXjXAv6 ngPost_QR





















ngpost's People

Contributors

kcchouette avatar mbruel avatar pankkake avatar peskdale avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ngpost's Issues

split instead of rar?

If one has a big file (say a 40GB iso) one can create a par2 that can repair the whole file, split the file (using normal file system splitting tools) and just use a single par2 file (assuming no block errors) to reconstruct the file from all the splits.

now, this can be a big help on IO if one is trying to post the file.

Currently, one has to essentially do 3 passes over all the data

  1. rar to split it (and encrypt if so desired)
  2. par to create the pars on the rar split
  3. read to upload

If on the other hand, one just needed to split, one could redo this

  1. par to create pars on the big file
  2. read to upload. the read to upload will generate the split in real time

a 33% IO win.

yes, one doesn't get encryption this way, but that's the trade off. Some would be willing to make that tradeoff especially with obfuscated names.

one thing I might also add is perhaps in the context of split, one also has the ability to randomize the order of the splits (i.e. assuming a 40GB file split into 500MB chunks is going to have 80 parts, instead of uploading part1, 2, 3, it can be seek to part 75, upload that, seek to part 33, upload that and so on and so forth.

thoughts?

Auto Compress + top Archive password not working properly

Hello , there is small issues that Auto compress for quickposts and random archive name and par don't work so what is the reason for it if we select it ? without selecting all lower choices and tick each one of them

if i select a single file or 2 file parts with same names
aaa.rar
aaa.r00
or
2 different files
111.rar
222.rar

it will still pickup one name of the files name nothing random i can't make it to be random unless i click generate or write my own

same thing with nzb passwords it said fixed pass for all posts , this is not working for all posts it will only pick up a fixed password for quickposts no matter the archive password on top is ticked or not , if i tick the top archive password and have a fixed pass and put a fixed pass on the lower option the app will only pick up the lower option , the top pass only works with auto posting

i guess if you can fix the auto compress to work for the quickpost it will fix all this issues

also if the auto compress if enable could disable the option on the lower panel this will help without removing the lower option , just when you choose one option it disable the other option automatically till you change it again or do it manual when disable the auto compress

i dont know how to be more accurate explaining this but if you tried it while posting you will see , this is what i find so far .

thank you .

[Request] Add a --nfo option

Adding a --nfo paramater (--nfo file.nfo) so the .nfo file won't be splitted in the rest of the archive, and indexers will be able to identify it.

Memory leaks on the last version (2.3)?

No problem encountered during the archiving/par2 creating process, but as soon as the posting begins, the program is killed. Unfortunately, I don't have acces to the logs on the server I'm renting. Could you add a verbose parameter to the program so I could debug it?
I never had any issue on the 2.1 & 2.2 versions (using the AppImage on a server running Ubuntu).

Weird article ids?

Why article number has { and } ? how to remove them?

<segment bytes="blabla" number="blabla">{019b929b-blabla-438f-bab2-blabla}@blabla</segment>

!! Request !! Important features .

Please add an option for random poster email name on each upload / file or folder

add a default password on automatic option or a random rar names on the quick option

close each tab after its done correctly uploaded , ( green mark )

Option to control the cpu usage , such as if you are using auto mode and leave the pc on , it will not effect the other running app , the cpu for the par can take up to 67 and up while creating pars

thanks , hope to see this happened.

The remote host closed the connection( )Segmentation fault (core dumped)

Here is the output from the command

[16:06:34.237] Start posting: test.nzb
[16:06:34.238] Compressing files: /usr/bin/rar a -ep1 -m0 -idp -hpdummy -v400m /tmp/EZXLsJVU3F3koikT8/EZXLsJVU3F3koikT8.rar /root/test_ss/Screen Shot 2020-03-24 at 4.31.46 PM.png /root/test_ss/ngPost_v4.4-x86_64.AppImage /root/test_ss/test.nzb /root/test_ss/test_1.nzb /root/test_ss/test_2.nzb /root/test_ss/test_2_3.nzb

******************************************************

[16:06:34.651] Generating par2: /usr/bin/par2 c -l -m1024 -r20 /tmp/EZXLsJVU3F3koikT8/EZXLsJVU3F3koikT8.par2 /tmp/EZXLsJVU3F3koikT8/EZXLsJVU3F3koikT8*rar -q
******************************

[16:06:40.885] Number of available Nntp Connections: 30
Segmentation fault (core dumped)

Not clear why this is happening. I can post from the Usenetic app.

Here is the command

ngPost --disp_progress --debug --thread 1 -h <HOST> -P 563 -s -u <USER> -p <PASS> -n 30 -f [email protected] -g "alt.binaries.misc" -i /root/test_ss -a 313350 -o test.nzb --tmp_dir /tmp --compress --gen_name --gen_par2 --rar_path /usr/bin/rar --rar_size 400 --rar_pass dummy --par2_pct 20 --par2_path /usr/bin/par2

Packing from network location

When packing from a network location on Windows, RAR fails due to a command line error, example:

Start Post #1: xxx.nzb
Compressing files: C:\Program Files\WinRAR\Rar.exe a -ep1 -m0 -idp -hpXxx -v1024m F:/Temp/pack/xxx.rar -r //tower2/disk10/XXX/XXX.mkv
ERROR: Unknown option: /tower2/disk10/XXX/XXX.mkv
=> rar exit code: 7

[Bug] nzb password checkbox is ignored

Case:
When you have the nzb password text field filled with characters, uncheck the 'nzb password' and to post files, the program only checks if the text field is empty or not, then still will use the password(because the text field is just disabled but filled), I think that bug is on this line:

_ngPost->_rarName, _ngPost->_rarPass,
cause not have an if of verification with the checkbox, but just supposing.

Feature Request: Maximum upload speed

Feature Request:
To limit the maximum upload speed of ngPost to a specified value. If it's not achieved, it does not throttle, however, if the max upload speed is achievable, allow user to specify maximum upload speed.

Suggestion:

--max_speed : limit max data throughput of all combined connections/threads.
Set's maximum upload speed of all connections combined to X speed per second specified in KiB, (kilo-bytes per second),
e.g. 3500 or 10000
e.g. (--max_speed 3500)

Should use another connection to post if one fails

For the context, I don't think it happened under 4.6, but it might also be because one of the providers I use to post had issues just recently.
Anyway, I use 3 separate providers to post. When one provider has issues, I get stuff like that in the end:

[=================================================>] 99 % (76429 / 76441) avg. speed:  13.01 MB/s[Poster #6 {NntpCon #14}] Error Socket: The TLS/SSL connection has been closed                                   
Error: disconnected connection: #14                                                       
                                                                                          
[Poster #6 {NntpCon #13}] Error Socket: The TLS/SSL connection has been closed            
Error: disconnected connection: #13                                                       
                                                                                          
[Poster #7 {NntpCon #16}] Error Socket: The TLS/SSL connection has been closed            
Error: disconnected connection: #16                                                       
                                                                                          
[Poster #7 {NntpCon #15}] Error Socket: The TLS/SSL connection has been closed            
Error: disconnected connection: #15                                                       
                                                                                          
we lost all the connections...                                                            
Sleep for 30 sec before trying to reconnect                                               
Pause posting...                                                                          
[=================================================>] 99 % (76429 / 76441) avg. speed:  12.91 MB/sTry to resume posting
Resume posting...                                                                         
[=================================================>] 99 % (76440 / 76441) avg. speed:  12.82 MB/s[Poster #5 {NntpCon #12}] Socket Timeout (30000 ms)                                                              
[Poster #5 {NntpCon #12}] Connection lost, trying to reconnect! (nb disconnected: 1)      
[==================================================] 100 % (76441 / 76441) avg. speed:  12.82 MB/s
                                                                                          
[11:19:24.374] Upload size: 50.99 GB in 01:07:52.955 (4072 sec)                  => average speed:  12.82 MB/s (16 connections on 16 threads)                                                                     
4 / 76441 articles FAILED to be uploaded (even with 10 retries)...                    
[=================================================>] 99 % (76438 / 76441) avg. speed:  15.38 MB/s[Poster #2 {NntpCon #3}] Error Socket: The TLS/SSL connection has been closed
[Poster #2 {NntpCon #3}] Connection lost, trying to reconnect! (nb disconnected: 9)
[Poster #5 {NntpCon #6}] Error Socket: The TLS/SSL connection has been closed
[Poster #3 {NntpCon #4}] Error Socket: The TLS/SSL connection has been closed
[Poster #5 {NntpCon #6}] Connection lost, trying to reconnect! (nb disconnected: 9)
[Poster #3 {NntpCon #4}] Connection lost, trying to reconnect! (nb disconnected: 9)
[Poster #1 {NntpCon #2}] Error Socket: The TLS/SSL connection has been closed
[Poster #1 {NntpCon #2}] Connection lost, trying to reconnect! (nb disconnected: 9)
[Poster #4 {NntpCon #5}] Error Socket: The TLS/SSL connection has been closed
[Poster #4 {NntpCon #5}] Connection lost, trying to reconnect! (nb disconnected: 9)
[Poster #0 {NntpCon #1}] Error Socket: The TLS/SSL connection has been closed
[Poster #0 {NntpCon #1}] Connection lost, trying to reconnect! (nb disconnected: 9)
[=================================================>] 99 % (76438 / 76441) avg. speed:  15.37 MB/s[Poster #7 {NntpCon #8}] Error Socket: The TLS/SSL connection has been closed
[Poster #7 {NntpCon #8}] Connection lost, trying to reconnect! (nb disconnected: 9)
[=================================================>] 99 % (76438 / 76441) avg. speed:  14.13 MB/s[Poster #2 {NntpCon #3}] Error Socket: The TLS/SSL connection has been closed
[Poster #2 {NntpCon #3}] Connection lost, trying to reconnect! (nb disconnected: 10)
[Poster #3 {NntpCon #4}] Error Socket: The TLS/SSL connection has been closed
[Poster #3 {NntpCon #4}] Connection lost, trying to reconnect! (nb disconnected: 10)
[Poster #4 {NntpCon #5}] Error Socket: The TLS/SSL connection has been closed
[Poster #4 {NntpCon #5}] Connection lost, trying to reconnect! (nb disconnected: 10)
[Poster #5 {NntpCon #6}] Error Socket: The TLS/SSL connection has been closed
[Poster #5 {NntpCon #6}] Connection lost, trying to reconnect! (nb disconnected: 10)
[Poster #1 {NntpCon #2}] Error Socket: The TLS/SSL connection has been closed
[Poster #1 {NntpCon #2}] Connection lost, trying to reconnect! (nb disconnected: 10)
[Poster #0 {NntpCon #1}] Error Socket: The TLS/SSL connection has been closed
[Poster #0 {NntpCon #1}] Connection lost, trying to reconnect! (nb disconnected: 10)
[=================================================>] 99 % (76438 / 76441) avg. speed:  14.12 MB/s[Poster #7 {NntpCon #8}] Error Socket: The TLS/SSL connection has been closed
[Poster #7 {NntpCon #8}] Connection lost, trying to reconnect! (nb disconnected: 10)
[=================================================>] 99 % (76438 / 76441) avg. speed:  13.07 MB/s[Poster #4 {NntpCon #5}] Error Socket: The TLS/SSL connection has been closed
Error: disconnected connection: #5

[Poster #5 {NntpCon #6}] Error Socket: The TLS/SSL connection has been closed
Error: disconnected connection: #6

[Poster #0 {NntpCon #1}] Error Socket: The TLS/SSL connection has been closed
Error: disconnected connection: #1

[Poster #2 {NntpCon #3}] Error Socket: The TLS/SSL connection has been closed
Error: disconnected connection: #3

[Poster #1 {NntpCon #2}] Error Socket: The TLS/SSL connection has been closed
Error: disconnected connection: #2

[Poster #3 {NntpCon #4}] Error Socket: The TLS/SSL connection has been closed
Error: disconnected connection: #4


we lost all the connections...
Sleep for 30 sec before trying to reconnect
Pause posting...
[=================================================>] 99 % (76438 / 76441) avg. speed:  12.97 MB/sTry to resume posting
Resume posting...
[==================================================] 100 % (76441 / 76441) avg. speed:  12.96 MB/s

[15:26:16.665] Upload size: 50.99 GB in 01:07:07.503 (4027 sec)                  => average speed:  12.96 MB/s (8 connections on 16 threads)
1 / 76441 articles FAILED to be uploaded (even with 10 retries)...

The first try was with -x, and the nzb ended up having post IDs that were missing (I checked with nzbverify). Oddly, the second nzb without -x (and also less connections) had no missing post IDs (but maybe they were never written?).

Anyway, I think it would make sense to try with connections from another provider instead of retrying with the same.

[Request] Option to enable/disable servers without removing them from conf

Hello Matthieu :)

Is it possible to add the option to enable/disable servers. in the GUI ? current options is you have to use all of remove what you don't want to use.

I can disable it in the conf file by commenting them out, but when i open and change a setting in the GUI, it overwrites and removes the commented servers.

Create a .txt file

Is it possible to include the function that ngPost can create a txt file only with The following infos?

Filename,Header,Password,Group

By Multiple uploads
Filename,Header,Password,Group
Filename,Header,Password,Group
.......

I use the MacOS version

[Feature Request] Pause the posting if the disk is full

I realized I had a ton of defective nzbs, because when the disk is full, ngPost keeps posting and writting the .nzb on nothing....
I wish there was a prompt, like RAR and ParPar uses, Your disk is full, [R]esume [Q]uit in the CLI. Haven't tried the GUI yet but I guess it would be something similar?

Allow shutdown script to receive original path of upload and nzbfile.

for instance like sabnzbd does for its processing and preprocessing scripts $1 for the full path of the upload and $2 for the resulting nzb file $3 for the rarname and $4 for the password used. this would allow further postprocessing of the upload for archiving into a cloud for instance or gathering information on the nature of the upload like mediainfo and so on.

[issue] thread count is being ingored

I have this line uncommentated but is it still using the cpu core count, is this known ?

## uncomment the next line to limit the number of threads,  (by default it'll use the number of cores)
## all the connections are spread equally on those posting threads
thread  =  30

[Request] Button/Option for Upload speed

Could you please implement an option for changing the upload speed? For example in kb or percent so that not all upload speed will be used for ngPost.

Thanks!

[Request] Saving the Nzb to Sections and Dateddir

For my example i will take the following monitor folders as given:
/mnt/linux
/mnt/freebsd
These are the Folders to be monitored
The Base save path for nzbs will be
/mnt/nzbs
The feature i search for is quiete easy to implement
When a nzb is finished in monitor linux
the nzb should be saved to
/mnt/nzb/linux/2020-01-29/ubuntu.18.94.nzb
when a nzb is finished in freebsd it should be saved to
/mnt/nzb/freebsd/2020-01-29/freebsd.11.nzb
it would also be posstible to save it like this :
/mnt/nzbs/2020-01-29/freebsd/freebsd.11.nzb
/mnt/nzbs/2020-01-29/linux/ubuntu.18.04.nzb

so basically
NZBBASEPATH/SECTION/DATE/NZBNAME
or
NZBBASEPATH/DATE/SECTION/NZBNAME

Out of memory: killed process

Hello,

I am trying to upload a backup with 250 GB right now.
But after a couple of hours (around 70%), ngpost seems to use all the available RAM (the vm has 20 GB). When trying to login, the ngPost process gets killed, because the system has way too little RAM left.

What should I do?

[Bug report] First two jobs are rared/pared in parallel

If PREPARE_PACKING option is enabled and if the monitored folder is empty, when two folders/jobs are moved to it simultaneously, ngPost will start to rar/par both. This is not the intended behavior. At the same time only one post should be rared/pared. Rar/par of the second job should only start if first job is already uploading.

Regards, Tensai

New install overwrites ngPost.conf file and old 4.3 exe installed

Hello Matthieu,

I just downloaded the 4.4. release and installed it in the old directory.
It replaced ngPost.conf with a default one, can this not be the default action ? (luckily i made a back-up just before installing)

i donwloaded the the 64bit installer and the ngpost.exe file it installs is from feb 18 2020. seems to be missing all the modifications.

NZB overwritten on repost

1st concern
NZB is explicitly overwritten if performing a repost, recommend rename by date on --auto or --monitor, prompt if manual. Anything except just overwrite without any notifications.

2nd concern
Recommend not using 2 separate randomFrom functions to return different types, here is a fix:
file: hmi/MainWindow.cpp function: onGenPoster, use a single call to convert from stdString
_ui->fromEdit->setText(QString::fromStdString(_ngPost->_randomFrom()));

3rd is just a request, I've already done a rewrite on the randomFrom function in NgPost.cpp to make the entire from email random:

std::string NgPost::randomFrom() const
{
QString randomFrom, alphabet("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789
");
int nb = 21, nbLetters = alphabet.length();
for (int i = 0 ; i < nb ; ++i) {
if (i==10) { randomFrom += "@"; }
if (i==18) { randomFrom += "."; }
randomFrom.append(alphabet.at(std::rand()%nbLetters));
}
return randomFrom.toStdString();
}

This makes the from email a random(10) string at random(8) string dot random(3) string.

4th is a suggestion
Instead of requiring "--compress" for --monitor and --auto then failing if it's not included, I recommend a _log("Warning: --compress automatically selected for --auto operation"); then just enable compression for the user anyway, since it's required, then proceed without return false;

Apologies is toes are stepped on here, I like what you've put together thus far keep up the good work.

[REQUEST] Support for subdirectories

Love using ngPost, but I have one suggestion.

When posting some scene releases they have subfolders in their natural file structure (like Sample Folders and Subs Folders). These don't work when posting, I have to modify the file structure for that. So implementation of Sub-directory handling would be great.

Thanks in advance.

ngPost_v1.6-x86_64.AppImage: core dump when X display not present (aka: via a ssh session)

When I connect via ssh (so: no X settings), and start ngPost_v1.6-x86_64.AppImage without parameters, it gives a coredump.
I find that quite a hefty effect.
I would prefer ngPost_v1.6-x86_64.AppImage so say something nicely about missing X / GUI.

sander@brixit:~$ ./ngPost_v1.6-x86_64.AppImage 
qt.qpa.xcb: could not connect to display 
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: xcb.

Aborted (core dumped)

FWIW:

sander@brixit:~$ ./ngPost_v1.6-x86_64.AppImage --help
ngPost_v1.6 is a NNTP poster that can:
  - use multiple connections on multiple servers. (for multiple server, you've to use a config file)
  - spread those connections on several threads (by default the number of cores available)
  - post several files or directory (you can put several times the option -i)
  - generate the nzb
  - use SSL encryption if desired

Syntax: ngPost_v1.6-x86_64.AppImage (options)? (-i <file or directory to upload>)+
        --help             : Help: display syntax
        -v or --version    : app version
        -c or --conf       : use configuration file (if not provided, we try to load $HOME/.ngPost)

Option -o: double extension .nzb.nzb

When I specify -o /tmp/thisismyNZB.nzb as the intended NZB file, it results in a nzb file /tmp/thisismyNZB.nzb.nzb, so double extension.

Is that how it was intended?

$ ./ngPost_v1.6-x86_64.AppImage  -h upload.super.nl -P 443 -s -u babla -p mypassword -n 5  -g "alt.binaries.test,alt.binaries.test2" -i /tmp/folderToP
ost -o /tmp/thisismyNZB.nzb
The default config file doesn't exist:  "/home/sander/.ngPost"

Upload size: 10 MB in 00:00:02.134 (2 sec) => average speed:   4.69 MB/s (5 connections on 4 threads)
nzb file: /tmp/thisismyNZB.nzb.nzb

Quick Post pack multi files into a single post

Thanks for the update and YES the 4.5 version fixed all problem i have except this one or this might be made this way, im not sure.

Quick Post pack multi files from different folders into a single post

is there is anyway to have an option to pack each file into a single post .

or should we keep it that way so the quick post will pack all files into a single file and Auto Posting will pack each file or each folder into its own post ? however quick post split folder one for each post .

maybe be good if there is options to choose from for each type of post , thanks .. you are da man :)

[Request] Queueing - and save to a queue file

Hi!

I was thinking about a handy feature wherein ngPost creates a queue file from the quick jobs I have setup but not started posting yet, saving that queue file and then use that to trigger the posting from the commandline rather than the HMI/GUI.

I have no idea if this is possible, but it seemed like a good idea to me.

Edit: If you have ever used Handbrakes Queue feature and queue file feature, you'll know what I mean.

Adding support for UTF-8

Dear ngpost developer,

First of all, I want to thank you for creating this wonderful tool for us! It's very easy to use!

Can I request you make ngpost to support UTF-8? When I upload something with Chinese/Japanese characters as the folder name, the NZB that ngpost created only show the Latin characters. Is that possible the name of the NZB file is exactly the same as the folder/file's name, even the name contains Asian characters?

Thanks for your time, and stay safe and healthy!

2020-06-06_170144

Feature Request: Resume on stopped/broken/canceled uploads

Hi,

could you please implement a function etc. which allows to resume stopped/broken/canceled uploads?

For example when a few tmp folders are unprocessed that there is flag/function which allows to resume the upload and does not start a complete new upload of all the files for a example in one folder.

Thank you very much!

Regards noobcoder1983

[Bug] Same "random" Posting address for every posting job in "Monitor folder" mode

Hi,

when the automatic "Monitor folder" mode is used (which I love btw), the randomly generated Poster address is the same for all posting jobs.

Steps to reproduce:

  1. configure ngPost to use random Poster address
  2. start ngPost with "--monitor" option
  3. add several folders (or files) to the monitored folder
  4. for each file or folder a NZB file will be generetad showing the same Poster address

ngPost should however generate a true random Poster address for every different upload.

Thanks and best regards

[Request] Progress bar for Rar and PAR

Hello There,

the rar and par function currently shows the command line but no further output in the default state (unlike in -d, but that shows a lot of info, the rar info looks reasonable but the par shows 10 lines per %)

is it possible to add a progression bar or some indicator of how far it is with the rar or par operation ?

Feature suggestion: parallel rar/par and upload queue

Hi Matthieu

Have you ever thought of separating the rar/par process and and the upload process into separate queues so they can be done in parallel?
Especially when monitoring a folder with a lot of jobs, this could drastically improve the overall performance. E.g. while uploading the first job (after it has been initially rared and pared), the next job(s) could already be rared and pared in parallel and then put into the upload queue.
Programmatically this can be done easily and I used this approach in a private upload tool I wrote myself in node.js using nyuu as the actual upload program.

Would love to hear your thoughts about this.

Regards, Tensai

[request] add option to select "server doesn't require login"

hi

i upload via virgin media newsgroup server and a username + password is not required.

when leaving the username and password fields empty ( blank ) i get error authentication denied

is there an option to add where you can select "server doesn't require login" similar to camel powerpost or yEncBin

even though i am leaving the fields blank the virgin media server is not detecting it that way

Feature Request: Compress nzb files

Feature Request:
Compress nzb files to save space, I tested a single nzb file of 100 kb and the result was 20kb because the format of nzb that is text, this can save time and BIG space for archiving tasks.

Thanks.

Nzbname and output dir

Currently whats keeping me from going full auto with your build in autofunction ist the way nzbnaming is handled. in gui it removes the codec and grpname tag. atleast on a windows platform. i have a structured nzb output which is based on Sections for instance tv/2020-01-28/rlsname.nzb
Is there any chance that can be integrated via config file? dont want to use gui, currently im monitoring a base folder per script in that base folder are those sections as dirs. so my script greps the section from dirname of the section dir the upload comes from. basically it would be enough to put nzbs into a seperate folder named by the name of the monitor folder for instance a upload coming from monitored folder /mnt/incoming/linuxisos would be saved in /mnt/nzbs/linuxisos/DATE/ubuntu.1904.nzb

[Request] Progress timestamps

Hello There,

can you add some sort of timestamps to the command line outputs. I tried -d switch but it doesn't do any timestamps :)

I am sometimes running into the issue that a post is going fine but the speed suddenly drops and it stops progressing altogether, it stops sending packets but won't throw an error of a time-out or anything.

Only way to resolve is to stop the command line with a break (ctrl+c). preferably timestamp output when using the --disp_progress FILES.

For example the time it started processing a package and when it ended.

make: "nntp/NntpArticle.h:42:25: error: ‘StringFormat’ in ‘class QUuid’ does not name a type"

On Ubuntu 18.04 64-bit, I get the errors belows trying to make.

Tips?

sander@brixit:~/git/ngPost/src$ qmake
Info: creating stash file /home/sander/git/ngPost/src/.qmake.stash

sander@brixit:~/git/ngPost/src$ make
/usr/lib/qt5/bin/uic hmi/MainWindow.ui -o ui_MainWindow.h
g++ -c -pipe -O2 -std=gnu++1y -Wall -W -D_REENTRANT -fPIC -DAPP_VERSION="1.6" -D__USE_MUTEX__ -D__DISP_PROGRESS_BAR__ -DQT_NO_DEBUG_OUTPUT -DQT_DEPRECATED_WARNINGS -DQT_NO_DEBUG -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_NETWORK_LIB -DQT_CORE_LIB -I. -I. -isystem /usr/include/x86_64-linux-gnu/qt5 -isystem /usr/include/x86_64-linux-gnu/qt5/QtWidgets -isystem /usr/include/x86_64-linux-gnu/qt5/QtGui -isystem /usr/include/x86_64-linux-gnu/qt5/QtNetwork -isystem /usr/include/x86_64-linux-gnu/qt5/QtCore -I. -isystem /usr/include/libdrm -I. -I/usr/lib/x86_64-linux-gnu/qt5/mkspecs/linux-g++ -o NgPost.o NgPost.cpp
In file included from NgPost.cpp:25:0:
nntp/NntpArticle.h:42:25: error: ‘StringFormat’ in ‘class QUuid’ does not name a type
     static const QUuid::StringFormat sMsgIdFormat = QUuid::StringFormat::Id128;
                         ^~~~~~~~~~~~
nntp/NntpArticle.h: In member function ‘QString NntpArticle::id() const’:
nntp/NntpArticle.h:102:55: error: ‘sMsgIdFormat’ was not declared in this scope
 QString NntpArticle::id() const { return _id.toString(sMsgIdFormat); }
                                                       ^~~~~~~~~~~~
nntp/NntpArticle.h:102:55: note: suggested alternative: ‘QTextFormat’
 QString NntpArticle::id() const { return _id.toString(sMsgIdFormat); }
                                                       ^~~~~~~~~~~~
                                                       QTextFormat
Makefile:451: recipe for target 'NgPost.o' failed
make: *** [NgPost.o] Error 1
sander@brixit:~/git/ngPost/src$ 

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.