Progress bar with number of threads, bytes retrieved, bytes to retrieve, files retrieved, files to retrieve, (maybe ETA if single download).

Hi, you made a release but didn't tag it using git. Could you please do so? It would make my job as a packager a bit simpler.

Maybe it's already exists... Possible use to pre-generate cache.

Multithreading in this case really nice, wget too slow.

I tried to compile and test libmget. The following 3 tests fail on my Debian Stable system:

Test-iri
Test-iri-percent
Test-restrict-ascii

I haven't had the time to look into what exactly is going wrong.

Sometimes it might be nice to have that.
see https://savannah.gnu.org/bugs/index.php?40318

Debugging the current test suite is difficult. I couldn't find any cumulative output nor any log files to find raw information when the tests failed. As a result, in order to find which tests fail, I had to run each file manually.

Instead, how about using the Parallel Test Harness provided in automake? It should make life much easier.

I'm not very well versed with autotools and Makefiles, but I can try to shift this to the test harness from automake if others agree.

Some help would be very appreciated however. Especially since I don't understand, at first glance quite a few of the macros used in tests/Makefile.am

Hi
Please add .gitignore files that include all produced files, so git status will show nothings on clean unmodified repo. This is usefull for both humans and helper scripts.

Thanks for mget - it's just cool

Right now the HTTP authentication part of Mget ignores Authentication-Info Headers.
This leads to more request/response traffic than needed.

Namely support the extended filename*= stuff for file naming.

no comment

This is one of them:

mget -r -np http://www.gamewinners.com/C64/index.htm

I've tried both with release v0.1.5 and the latest master branch, i always get only 1 level of recursion.
With vanilla wget it works fine, but very slow...

These options are quite useful to reduce the download size of big sites:

‘-A acclist --accept acclist’
‘-R rejlist --reject rejlist’
Specify comma-separated lists of file name suffixes or patterns to accept or reject

wget documentation

Mget should download and respect /robots.txt "Robot Exclusion Standard" and <META name="robots" ...>.
More information here:
http://en.wikipedia.org/wiki/Robots_exclusion_standard

Thanks to the multithreading nature of Mget, this feature needs a kind of priorization / synchronisation of the downloader threads: robots.txt has to be downloaded and parsed before any other file of a domain...

When in recursive mode, following/testing directory listings would retrieve even more files than just scanning .html and .css files.

$ make
can't find hash.c.

make[3]: *** No rule to make target ../src/hash.c', needed bymget.pot-update'. Stop.

$ find . -name 'hash.c'
returns nothing...

Download sitemap urls from robots.txt (zipped and unzipped).
Parse these files with Mgets XML parser to fetch all urls.
Respect additional information/schemas from 'urlset', e.g.http://www.google.com/schemas/sitemap-image/1.1.

See http://www.sitemaps.org/protocol.html for more information.

Mget should be on this wikipedia table:

https://en.wikipedia.org/wiki/Metalink

Could you please implement support for multiple proxies.
I am curious to see implementation.

it shouldn't be too hard to implement such a feature

I haven't programmed with C for a while. It's interesting how it may deal with multithreading. How threads will divide proxies between them...

Hi,

Is there plan to support SOCKS proxies? I had to run polipo to make a HTTP proxy out of a TOR SOCKS proxy to try to use TOR, but I had troubles to resolve a hidden service.

Here is my polipo config:

$ cat /etc/polipo/config
daemonise=false
diskCacheRoot=/var/cache/polipo/
proxyAddress=127.0.0.1
proxyName=localhost
serverSlots=4
serverMaxSlots=8
cacheIsShared=true
allowedClients=127.0.0.1
socksParentProxy = localhost:9050

-D --domains Comma-separated list of domains to follow.
--exclude-domains Comma-separated list of domains NOT to follow.
Please add to a these options wildcard or regex support. It's needed for -r option, for ie
cdn1.domain.com
cdn2.domain.com
cdn3.domain.com
& etc.

I have googled it... No idea.

trying sudo apt-get install libz

it's probably me, not you :) could you point me to libz

Please add a proxy list support with mget.

To have more control over the GnuTLS SSL layer, it would be nice to an option to directly specify the so called 'priority string', e.g. 'SECURE128:-RSA'.

Putting it into /etc/mgetrc or ~/.mgetrc gives a default that can be overriden by --gnutls-ciphers.

It would not combine with --secure-protocol (mutual exclusive). Maybe it is a good idea to warn if both options are specified.

Hi,

with Ubuntu 14.04 I installed the prerequisites as follows:

apt-get install autoconf autogen automake libtool gtk-doc-tools xsltproc gettext zlib1g-dev liblzma-dev libbz2-dev libgnutls-dev libidn2-0-dev flex libssl-dev

The I did ./autogen.sh, ./configure, make, make check.

Make check did run successful except for test:

PASS: test
PASS: test-mget-1
PASS: test-restrict-ascii
PASS: test-i-http
FAIL: test-i-https
PASS: test-np
PASS: test--spider-r
PASS: test-meta-robots
PASS: test-idn-robots
PASS: test-idn-meta
PASS: test-idn-cmd
PASS: test-iri
PASS: test-iri-percent
PASS: test-iri-list
PASS: test-iri-forced-remote
PASS: test-auth-basic
PASS: test-parse-html
PASS: test-parse-rss
PASS: test--page-requisites
PASS: test--accept
PASS: test-k
make[4]: Entering directory `/tmp/mget/tests'
make[4]: Nothing to be done for `all'.
make[4]: Leaving directory `/tmp/mget/tests'
============================================================================
Testsuite summary for mget 0.1.8
============================================================================
# TOTAL: 21
# PASS:  20
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See tests/test-suite.log
Please report to [email protected]
============================================================================
make[3]: *** [test-suite.log] Error 1
make[3]: Leaving directory `/tmp/mget/tests'
make[2]: *** [check-TESTS] Error 2
make[2]: Leaving directory `/tmp/mget/tests'
make[1]: *** [check-am] Error 2
make[1]: Leaving directory `/tmp/mget/tests'
make: *** [check-recursive] Error 1

As the test only fails for https I think I might have installed a wrong prerequisite... what do you think?

Is it possible? is it already built in? Usually this way it requires much less time to download huge file. Under Windows I used Flashget to achieve it.
Flashget also gives nice graphic representation of download progress. It's possible to see how parts of file filling in same time...

Sorry if it's noob question.

Convert filenames to local encoding.
Non-printable (in the means of local encoding) characters should be escaped.
Right now everything >126, <32, and / is percent escaped.
Since we internally work with UTF-8, we just need a UTF-8 -> local_encoding encoder plus percent escaping to non-printable characters.

could you tell me what I'm doing wrong?

$ ./autogen.sh
./autogen.sh: line 3: gtkdocize: command not found
Copying file config.rpath
Creating directory m4
Copying file m4/codeset.m4
Copying file m4/fcntl-o.m4
Copying file m4/gettext.m4
Copying file m4/glibc2.m4
Copying file m4/glibc21.m4
Copying file m4/iconv.m4
Copying file m4/intdiv0.m4
Copying file m4/intl.m4
Copying file m4/intldir.m4
Copying file m4/intlmacosx.m4
Copying file m4/intmax.m4
Copying file m4/inttypes-pri.m4
Copying file m4/inttypes_h.m4
Copying file m4/lcmessage.m4
Copying file m4/lib-ld.m4
Copying file m4/lib-link.m4
Copying file m4/lib-prefix.m4
Copying file m4/lock.m4
Copying file m4/longlong.m4
Copying file m4/nls.m4
Copying file m4/po.m4
Copying file m4/printf-posix.m4
Copying file m4/progtest.m4
Copying file m4/size_max.m4
Copying file m4/stdint_h.m4
Copying file m4/threadlib.m4
Copying file m4/uintmax_t.m4
Copying file m4/visibility.m4
Copying file m4/wchar_t.m4
Copying file m4/wint_t.m4
Copying file m4/xsize.m4
Copying file po/Makefile.in.in
Copying file po/Makevars.template
Copying file po/Rules-quot
Copying file po/boldquot.sed
Copying file po/[email protected]
Copying file po/[email protected]
Copying file po/insert-header.sin
Copying file po/quot.sed
Copying file po/remove-potcdate.sin
glibtoolize: putting auxiliary files in .'. glibtoolize: linking file./ltmain.sh'
glibtoolize: putting macros in AC_CONFIG_MACRO_DIR, m4'. glibtoolize: linking filem4/libtool.m4'
glibtoolize: linking file m4/ltoptions.m4' glibtoolize: linking filem4/ltsugar.m4'
glibtoolize: linking file m4/ltversion.m4' glibtoolize: linking filem4/lt~obsolete.m4'
configure.ac:75: warning: The 'AM_PROG_MKDIR_P' macro is deprecated, and will soon be removed.
configure.ac:75: You should use the Autoconf-provided 'AC_PROG_MKDIR_P' macro instead,
configure.ac:75: and use '$(MKDIR_P)' instead of '$(mkdir_p)'in your Makefile.am files.
configure.ac:62: installing './compile'
configure.ac:14: installing './config.guess'
configure.ac:14: installing './config.sub'
configure.ac:8: installing './install-sh'
configure.ac:8: installing './missing'
Makefile.am: installing './INSTALL'
automake: error: cannot open < gtk-doc.make: No such file or directory
autoreconf: automake failed with exit status: 1

Initialized build system. For a common configuration please run:

./configure --enable-gtk-doc

$ ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking dependency style of gcc... none
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking minix/config.h usability... no
checking minix/config.h presence... no
checking for minix/config.h... no
checking whether it is safe to define EXTENSIONS... yes
checking build system type... x86_64-apple-darwin12.3.0
checking host system type... x86_64-apple-darwin12.3.0
checking how to print strings... printf
checking for a sed that does not truncate output... /usr/bin/sed
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld
checking if the linker (/usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld) is GNU ld... no
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm
checking the name lister (/usr/bin/nm) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 196608
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking how to convert x86_64-apple-darwin12.3.0 file names to x86_64-apple-darwin12.3.0 format... func_convert_file_noop
checking how to convert x86_64-apple-darwin12.3.0 file names to toolchain format... func_convert_file_noop
checking for /usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld option to reload object files... -r
checking for objdump... no
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @file support... no
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm output from gcc object... ok
checking for sysroot... no
checking for mt... no
checking if : is a manifest tool... no
checking for dsymutil... dsymutil
checking for nmedit... nmedit
checking for lipo... lipo
checking for otool... otool
checking for otool64... no
checking for -single_module linker flag... yes
checking for -exported_symbols_list linker flag... yes
checking for -force_load linker flag... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fno-common -DPIC
checking if gcc PIC flag -fno-common -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld) supports shared libraries... yes
checking dynamic linker characteristics... darwin12.3.0 dyld
checking how to hardcode library paths into programs... immediate
checking for dlopen in -ldl... yes
checking whether a program can dlopen itself... yes
checking whether a statically linked program can dlopen itself... yes
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking whether make supports nested variables... (cached) yes
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking dependency style of gcc... (cached) none
checking for flex... flex
checking lex output file root... lex.yy
checking lex library... none needed
checking whether yytext is a pointer... no
checking whether ln -s works... yes
checking whether gcc and cc understand -c and -o together... yes
checking for gcc option to accept ISO C99... none needed
checking whether NLS is requested... yes
checking for msgfmt... no
checking for gmsgfmt... :
checking for xgettext... no
checking for msgmerge... no
checking for ld used by GCC... /usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld
checking if the linker (/usr/llvm-gcc-4.2/libexec/gcc/i686-apple-darwin11/4.2.1/ld) is GNU ld... no
checking for shared library run path origin... done
checking for CFPreferencesCopyAppValue... no
checking for CFLocaleCopyCurrent... no
checking for GNU gettext in libc... no
checking for iconv... no, consider installing GNU libiconv
checking for GNU gettext in libintl... no
checking whether to use NLS... no
checking whether to generate man pages... no
checking for gnutls_global_init in -lgnutls... no
configure: WARNING: *** GNUTLS was not found. You will not be able to use HTTPS or Metalink checksumming
checking for deflate in -lz... no
configure: WARNING: *** ZLIB was not found. You will not be able to use gzip compression
checking for idna_to_ascii_8z in -lidn... no
configure: WARNING: *** LIBIDN was not found. You will not be able to use IDN support
checking for pthread_attr_init in -lpthread... no
checking for clock_gettime in -lrt... no
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking for inttypes.h... (cached) yes
checking libintl.h usability... no
checking libintl.h presence... no
checking for libintl.h... no
checking locale.h usability... yes
checking locale.h presence... yes
checking for locale.h... yes
checking netdb.h usability... yes
checking netdb.h presence... yes
checking for netdb.h... yes
checking netinet/in.h usability... yes
checking netinet/in.h presence... yes
checking for netinet/in.h... yes
checking stddef.h usability... yes
checking stddef.h presence... yes
checking for stddef.h... yes
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking for strings.h... (cached) yes
checking sys/socket.h usability... yes
checking sys/socket.h presence... yes
checking for sys/socket.h... yes
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking for unistd.h... (cached) yes
checking for stdbool.h that conforms to C99... yes
checking for _Bool... yes
checking for inline... inline
checking for int16_t... yes
checking for int32_t... yes
checking for int8_t... yes
checking for off_t... yes
checking for pid_t... yes
checking for C/C++ restrict keyword... __restrict
checking for size_t... yes
checking for ssize_t... yes
checking for uint16_t... yes
checking for uint32_t... yes
checking for uint8_t... yes
checking for ptrdiff_t... yes
checking for working alloca.h... no
checking for alloca... no
checking whether alloca.c' needs Cray hooks... no checking stack direction for C alloca... -1 checking vfork.h usability... no checking vfork.h presence... no checking for vfork.h... no checking for fork... no checking for vfork... no checking for stdlib.h... (cached) yes checking for GNU libc compatible malloc... no checking for stdlib.h... (cached) yes checking for unistd.h... (cached) yes checking for sys/param.h... yes checking for getpagesize... no checking for working mmap... no checking for stdlib.h... (cached) yes checking for GNU libc compatible realloc... no checking for clock_gettime... no checking for dprintf... no checking for dup2... no checking for futimens... no checking for gettimeofday... no checking for localtime_r... no checking for memchr... no checking for memmove... no checking for memset... no checking for mkdir... no checking for munmap... no checking for select... no checking for setlocale... no checking for socket... no checking for strcasecmp... no checking for strchr... no checking for strdup... no checking for strerror... no checking for strncasecmp... no checking for strndup... no checking for strrchr... no checking for strstr... no checking for strlcpy... no checking for vasprintf... no checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating Makefile config.status: creating po/Makefile.in config.status: error: cannot find input file:include/Makefile.in'
Ants-MacBook-Pro:mget-master ant$ make
cd . && /bin/sh ./config.status config.h
config.status: creating config.h
make all-recursive
Making all in po
make[2]: *** No rule to make target `all'. Stop.
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

I'm trying to replace wget with mget, but currently there are some really glaring things which are quite useful to implement:

• --accept / --reject (accept / reject flags for directories / files / extensions (mime types))
• --mirror option
• --adjust-extension (according to MIME type)
• --ignore-case
• --restrict_file_names=windows,unix
• --wait seconds (waits some seconds, for a new connection)

- --random-wait randomizes wait time 1..--wait

These suggestions also should be able to put in the ~/.mgetrc, especially header flags.

Why does -p conflict with recursive downloading (it stops downloading after getting the page requirements and does not go any further if you want to mirror something with getting the page requirements .....)?

Hi Tim.

I've given up trying building the whole thing with MingW/MSVC. Cygwin works fine except for this in tests/libtest.c:

gcc -c -Wall -DHAVE_CONFIG_H=1 -DHAVE_LIBPTHREAD=1 -I. -I./include -I./libmget -I./src -std=gnu99  -g3 -O0 tests/libtest.c
tests/libtest.c: In function '_server_thread':
tests/libtest.c:89:30: error: unknown field 'sa_handler' specified in initializer
tests/libtest.c:89:30: warning: missing braces around initializer
tests/libtest.c:89:30: warning: (near initialization for '(anonymous).<anonymous>')

I use this diff to fix it:

--- Git-latest/tests/libtest.c  2014-08-19 22:42:21 +0000
+++ tests/libtest.c     2014-08-23 14:48:18 +0000
@@ -85,8 +85,10 @@
        unsigned it;
        int byterange, authorized;
        time_t modified;
+       struct sigaction sa;

-       sigaction(SIGTERM, &(struct sigaction) { .sa_handler = nop }, NULL);
+       sa.sa_handler = nop;
+       sigaction(SIGTERM, &sa, NULL);

        while (!terminate) {
                mget_tcp_deinit(&tcp);

My Cygwin uses gcc 4.5.3. Is that too old for this?