gaenserich / hostsblock Goto Github PK

Review all files to make sure they comply with bash best practices, e.g. using "<" instead of cat, etc.

I installed hostsblock from AUR, the mentioned /etc/hostsblock/rc.conf file wasn't created (it appears to be that the hostsblock.conf file is the file that holds the configuration now, so the documentation is probably out of date)

$ ls /etc/hostsblock 
black.list  hosts.head  hostsblock.conf  white.list

so I just ran

$ sudo /usr/bin/hostsblock
gzip: invalid option -- 'z'
Try `gzip --help' for more information.

on a subsequent run there was no message, and I got my shell back after a few seconds

edit: when I clear the cache directory in /var/cache/hostsblock/ I get the message back

It can find a url including http(s):// just fine:

sudo hostsblock-urlcheck -f test/hostsblock.conf http://www.foo.com
[DETAIL] Using configuration file test/hostsblock.conf.
Checking to see if url is blocked or unblocked...

'www.foo.com' NOT BLOCKED/REDIRECTED
    1) Block www.foo.com
    2) Block www.foo.com and delete all whitelist url entries containing www.foo.com
    3) Keep unblocked (default)
1-3 (default: 3)

But with foo.com or www.foo.com:

$ sudo hostsblock-urlcheck -f test/hostsblock.conf foo.com
[DETAIL] Using configuration file test/hostsblock.conf.
Checking to see if url is blocked or unblocked...

'-f' NOT BLOCKED/REDIRECTED
    1) Block -f
    2) Block -f and delete all whitelist url entries containing -f
    3) Keep unblocked (default)
1-3 (default: 3):

It's been some time since hostsblock scripts have been using hostsblock.conf and not rc.conf (which probably should have been deleted already).
So the recent adaway source list should be added to this file as well as moving someonewhocares.org to the highly recommended group together with it.
#54

After the latest commits I can see better processing in producing the hosts.block file with only 2 exceptions.
I have the following two lines as the 1st and 3rd entries:

0.0.0.0 0.0.0.0
0.0.0.0 0.0.0.0 www.zergnet.com

I don't think this is caused by the fact that I use 0.0.0.0 (which works fine for me) instead of 127.0.0.1 (which creates a little larger file).

Hello,
Please how to temporary disable the whole block list?
I suspect my hostsblock config may causes my inability to download any data from Google Webmaster Tools. On my main computer with hostblocks it occurs with four different browsers (plus firefox no plugin). Clicking the "Download this table" on GWT gives no result but the following error in Firefox console:

AssertionError: Assertion failed: GA not loaded yet.

There's no visible URL on GWT's button so I cannot make use of hostsblock-url-check in this case.
It works on my laptop therefore the need to check my main computer's network conf.

Reduce the number of instances of grep piping to another grep instance by using the "-f" flag. Will probably have to create a whole "hostsblock" directory under "/usr/lib" to contain those "grep -f" files and hostsblock-common.sh. This should hopefully improve performance.

I have HostsMan running in a Windows machine configured to use exactly the same sources as Linux hostsblock, but it produces considerably larger list.

The sources used are:
http://winhelp2002.mvps.org/hosts.zip
http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&mimetype=plaintext
http://www.malwaredomainlist.com/hostslist/hosts.txt
http://hosts-file.net/ad_servers.asp
http://hostsfile.mine.nu/Hosts.zip
http://someonewhocares.org/hosts/hosts
http://sysctl.org/cameleon/hosts

When I use Meld to compare two lists (after stripping everything other than domain names) I see that there are also a number of domain names missing in the Windows hosts list, but it has 147957 entries whereas hosts.block has 127335.

For instance, I saw large chunks of entries ending in hosts.block such as:

a.collective-media.net...302br.net
ad*.adk2.co
ad.amtk-media.com...302br.net
ad.doubleclick.net...302br.net
ad-emea.doubleclick.net...302br.net
adfarm.mediaplex.com...302br.net
admin*.testandtarget.omniture.com
ads.pointroll.com...302br.net
ams*.ib.adnxs.com

In trying out hostsblock check on a Slackware machine that uses Networkmanager combined with dnsmasq the script fails for me due to a number of what appear to be wrong assumptions in the check_dnsmasq_config function in /usr/lib/hostsblock-common.sh

The background is that when someone uses Networkmanager to manage the network it becomes responsible for starting dnsmasq (by setting dns=dnsmasq). When doing this Networkmanager starts dnsmasq and it no longer uses a dnsmasq config file; instead it reads files that are put in a user defined conf-dir directory(user can pick the name of that directory). Note that this is something that can be the case without Networkmanager as well (see config file for dnsmasq)

For reference I get the following output for the options dnsmasq runs with:

bash-4.3$ ps -eo comm,args| grep dnsm dnsmasq /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/var/run/NetworkManager/dnsmasq.pid --listen-address=127.0.0.1 --conf-file=/var/run/NetworkManager/dnsmasq.conf --cache-size=400 --proxy-dnssec --conf-dir=/etc/NetworkManager/dnsmasq.d

1. The above line has both a conf-file and a conf-dir, however the settings that are added in the conf-directory DO NOT get added to the conf-file written by Networkmanager. So if you want to check the configuration you need to read both the conf-file and the content of all the files in the conf-dir. If you don't read the content of the conf-dir the check will always fail as the user has no control over the content of the conf-file.

2. Note that I explicitly write "conf-file" instead of the "config-file" that you grep for in the code in this function. It should be like this if I look at the above output of ps. Moreover this is the way how these options are specified in the man-page of dnsmasq, see http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html (search for conf-file).

3. Some of the dnsmasq options you check for are hardcoded in the networkmanager code that starts dnsmasq and don't appear in either the config file or the config directory you are trying to read. See line 273 - 290 in http://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/src/dns-manager/nm-dns-dnsmasq.c for this. You would miss for example that the listen-address is in fact configured correctly.

For now I gave up on trying to get the script working after this; however a quick glance through shows that I would have issues with the dhcpcd_config function as well as that also appears to look for files that don't exist. This is set in /etc/resolv.conf as Networkmanager manages this.

Personally I would opt for giving clear instructions of what settings should be used instead of these checks. Simply because getting these checks functioning correctly is hard because of distro and setup specific differences in network-config.

Sites that use a 302 redirect on a URL cause cURL to break, its fixed by adding "-L" or "--location" to the cURL arguments. However hostsblock will still fail to parse the resulting file if its compressed, because it'll save it as the initial URL and not as the redirected URL, which doesn't end with a file extension resulting in it being treated as a plaintext file. I'd submit a pull request to fix it, but I'm not skilled enough in BASH to do so.

In hostsblock.sh and hostsblock-common.sh there is a variable misspelled 4 times.
"whilelist" Should be "whitelist" (L -> T). This causes white-listing not applying when updating hosts.

First run works OK, subsequent one returns following:

$ sudo /usr/bin/hostsblock
[/var/cache/hostsblock/hosts-file.net.download.hosts.zip]
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of /var/cache/hostsblock/hosts-file.net.download.hosts.zip or
        /var/cache/hostsblock/hosts-file.net.download.hosts.zip.zip, and cannot find /var/cache/hostsblock/hosts-file.net.download.hosts.zip.ZIP, period.
[WARN] FAILED to unzip hosts-file.net.download.hosts.zip.
[/var/cache/hostsblock/hostsfile.org.Downloads.BadHosts.unx.zip]
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of /var/cache/hostsblock/hostsfile.org.Downloads.BadHosts.unx.zip or
        /var/cache/hostsblock/hostsfile.org.Downloads.BadHosts.unx.zip.zip, and cannot find /var/cache/hostsblock/hostsfile.org.Downloads.BadHosts.unx.zip.ZIP, period.
[WARN] FAILED to unzip hostsfile.org.Downloads.BadHosts.unx.zip.
[/var/cache/hostsblock/winhelp2002.mvps.org.hosts.zip]
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of /var/cache/hostsblock/winhelp2002.mvps.org.hosts.zip or
        /var/cache/hostsblock/winhelp2002.mvps.org.hosts.zip.zip, and cannot find /var/cache/hostsblock/winhelp2002.mvps.org.hosts.zip.ZIP, period.
[WARN] FAILED to unzip winhelp2002.mvps.org.hosts.zip.

PS: I've found this problem reported in Arch Forums hostsblock thread too but thought it's better to track it here.

Provide hostsblock.service and hostsblock.timer files to allow hostsblock to be used with systemd.

There is an alternative hosts blocking software which is written in python, well documentated README.md and only changes /etc/hosts. It uses the KISS principle which you can just python updateHostsFile.py to use it. I hope you can add it to the README.md. I also hope that you can add a license to hostsblock (I recommend wftpl.

The hostsblock annotation file (by default /var/lib/hostsblock.db), which tracks which blocklists contain which entries (so that users can be informed which blocklist may contain overly-aggressive entries when using hostsblock-urlcheck) gets very large as just a plain text file (currently 157M on my box), and is also not in a sensible order (each pass of the target file compilation loop just appends to the same file). To make this file smaller and more human-readable, hostsblock should compress (with gzip/pigz, so as to just re-use the same mechanism used to compress previous target hosts files) and sort this file.
Ideas on implementation:
*hostsblock.sh: if gzip/pigz is detected, just "sort -u hostsblock.db | gzip/pigz -9dc > hostsblock.db.gz && rm hostsblock.db" etc.
*hostsblock-urlcheck: have a conditional to select either hostsblock.db.gz or fall back to hostsblock.db, add a gzip/pigz to the end of the write procedures for detected file.

Under 0.999.3-1 the whitelist is not applied to the newly processed hosts file.

To reproduce, use hostsblock-urlcheck to explicity add a URL to the white list (eg, http://www.lrb.co.uk/) and then run hostsblock. The newly generated /etc/hosts file will include the whitelisted URL.

To debug, I appended a message to the relevant line in /usr/bin/hostsblock, like so
sed "s/ \!.*$//g" | sort -u | grep -vf "$whitelist" >> "$hostsfile" && _notify 4 "Whitelist applied +++++"; then

This never gets printed.

The output of hostsblock (with level 4 debugging) is:

[INFO] Old /etc/hosts will not be recycled into new version.
[DETAIL] Backing up old version of /etc/hosts...
/usr/lib/hostsblock-common.sh: line 55: 0: command not found
[WARN] FAILED to compress /etc/hosts with 0.
[DETAIL] Using a hostshead file, so overwriting /etc/hosts with /etc/hostsblock/hosts.head...
[INFO] Replaced existing /etc/hosts with .
[INFO] Compiling block entries into /etc/hosts...
[INFO] Compiled block entries into /etc/hosts.
[DETAIL] Skipping redirect entries...
[INFO] Appending blacklisted entries to /etc/hosts...
[INFO] Appended blacklisted entries to /etc/hosts.
[INFO] /etc/hosts: 354005 urls redirected to 127.0.0.1.
[INFO] Executing postprocessing...
[INFO] Postprocessing completed.
[DETAIL] Cleaning up temporary directory /dev/shm/hostsblock...
[NOTE] Cleaned up /dev/shm/hostsblock.
[INFO] DONE.

The error message about 0 not found points to /etc/hostsblock/hostblock.conf where I have tried both 0 and no here:

backup_old="0"

Sorry I haven't been able to debug more...

After installing hostblock on my system and running it with default config I found that yandex.ru is blocked.

For ex-USSR users (especially in Russia) It is like block google.com by default. It is main search engine and etc.

I don't know yet which blacklist contains that entry, but it definitely should be turned off by default.

I don't know about other distros or MacOS but adding the following line at the very end displays a nice notification each time when this cron job is completed under Ubuntu:

DISPLAY=:0.0 XAUTHORITY=~/.Xauthority notify-send "Hostsblock updated just now!"

While testing the new pre-alpha version I get the error "USING no DNSCACHER, BUT THE 'hostsfile' VARIABLE IS NOT SET TO /etc/hosts"
But I am using DNSCACHER.

Today I received "xxxxx FAILED" message for the first time, and I only saw it when I opened the log file after hostsblock ended quietly as normal.
Apparently no action was taken in such a case, although perhaps it would be better if hostsblock could (a) simply move on to the next server and complete the whole process using that server's hostsblock list in the cache, or (b) re-try connecting to such a server a (configurable?) specified number of times at (configurable?) specified intervals, and perhaps switch to mode (a) if it still FAILED.

I can't find a usage section in README.md. How do I use it?

Hi.
It seems that everywhere the gzip command is used, there is a -z parameter passed along, which makes it fail.
The consequences are that the hostsblock.db.gz isn't created properly and then, other commands fail too.

The fix is pretty straightforward: remove the -z parameter.
BTW, the -z parameter is also used with the command pigz (parallel gzip), but as I don't use that program, I'm not sure if this issue affects it too. I'd guess it does.

I think it is time to organize the code into src, man and something else as it seems a bit too long to ls and it may be easier to get all the src into /usr/bin, man into /usr/share/man.

Just FYI, dnsmasq will read hosts files and generate zones from them.

So you could use that for ALL of the heavy-listing. It even has a config option / flag (-b) to instantly report failed resolution for host-local addresses (127.anything, 0.0.0.0) and can cache somewhere around a thousand names in RAM (uses ~8MB RAM after running 5 days heavy browsing). That would make it a hard dependency though, so ...

Source: http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html

Can even specify a directory containing multiple hosts-formatted files ;)

Right now, /etc/hosts.block does will not always be updated when the while/black.list files are changed. This is because $changed is only set to true when the downloaded files change, but not when these configuration files change. The $changed part should therefore also monitor these files.

A small group of entries repeat themselves in the target hosts file despite filtering. Textfile encoding is suspected. The repeated entries:
ads.cnn.com
ads.doubleclick.com
ads.theweek.com
adsyndication.msn.com
advertising.demandmedia.com
beacon.scorecardresearch.com
cdn.betrad.com
click.linksynergy.com
content.dl-rms.com
csi.gstatic.com
dsa.csdata1.com
example.com
getglue.com
mediakit.theonion.com
nbcpeacock.rresults.com
ogp.me
pixel.mathtag.com
ru.youporn.com
s7.addthis.com
widgets.getglue.com
www.nbcudigitaladops.com
www.rooshv.com

I hope you can add it to the README.md to show how to use hostsblock in polipo by converting /etc/hosts

sudo sed 's/127.0.0.1 //' /etc/hosts >> /etc/polipo/forbidden

Hi, in hostsblock.conf all DEFAULT settings are set for a NO DNS CACHER setup. But that one:

hostsfile="/etc/hosts.block"  # For use with a dnscacher, e.g dnsmasq

Of course unless I'm wrong. Please enlight me.

I suggest that somewhere in the script you perform chmod on the files hostsblock creates. If the root user's umask settings are restrictive, then the default permissions may not allow the files to be read by non-root processes.

Version 0.999.2-1 from aur

I am just trying to understand the format for the file /var/lib/hostsblock.db.
It seems to me it is something like "%IP %address ! %source"
However

• line 262 in common: _strip_entries " $@$" "$annotate" . The extra $ in the first argument makes it match the address and the end of line. I would expect "$@ !"
• Is there a special format for local files ? the "!" character is escape in many calls to sed and grep. So the database ends up with "%IP %address \! %source"

Provide detailed man pages (hostsblock-urlcheck.8 hostsblock.8 hostsblock.conf.8).

I am now researching for xargs and parallel.

While testing the new pre-alpha version, got this error: "Configuration file /etc/hostsblock/hostsblock.conf NOT FOUND, using defaults."
It seems the file "rc.conf" should be renamed as "hostsblock.conf"

hostsblock-0.999.2-1 is installed on two of my machines running Arch; are up-to-date, AUR included (as of yesterday).
hostsblock (and dnsmask) conf files are identical on both machines. It works on box One does but not on the other.

box One
With DNS caching on, hosts.block is updated and so on, single issue:

oct. 29 00:03:35 One hostsblock[3169]: gzip: invalid option -- 'z'
oct. 29 00:03:35 One hostsblock[3169]: Try `gzip --help' for more information.
oct. 29 00:03:38 One hostsblock[3169]: sort: write failed: standard output: Broken pipe
oct. 29 00:03:38 One hostsblock[3169]: sort: write error

from line 210 in /usr/bin/hostsblock --which is absent from the same version script on box Zero ~8-|

Box Zero
With DNS caching ON, hostsblock exit and hosts.block file is not updated

oct. 29 00:00:01 Zero hostsblock[13141]: [FATAL] NO VIABLE DNSCACHER CONFIGURATION FOUND. EXITING...
oct. 29 00:00:01 Zero systemd[1]: hostsblock.service: Main process exited, code=exited, status=1/FAILURE
oct. 29 00:00:01 Zero systemd[1]: Failed to start Block bad domains system-wide.
oct. 29 00:00:01 Zero systemd[1]: hostsblock.service: Unit entered failed state.
oct. 29 00:00:01 Zero systemd[1]: hostsblock.service: Failed with result 'exit-code'.

I tripled checked if dnsmasq.conf are identical between the two boxes and they are.

But I noted /usr/bin/hostsblock differ consequently. pacman -Qi says both are v0.999.2-1, with a different size ?#-|

For the moment I changed hostsblock on box One to run without DNS caching, and it runs. Though it still checks dnsmask:
/usr/bin/hostsblock

[INFO] Auto-detecting dnscaching settings...
[INFO] DNSMASQ does NOT have the correct 'listen-address' entry.
[INFO] DNSMASQ incorrectly configured.
[INFO] Checking blocklists for updates...

I suspect a script change forgetting to update version. What's your suggestion?

I wanted to install this package, but I didn't find it on AUR. So I decided to adopt the "general installation" from this git: I downloaded the tarball and extracted it, but I found out that the rc.conf link pointed to nowhere... now I see that it has been deleted from the git 5 days ago.
What happened? :(

In addition to "0.0.0.0" and "0.0.0.0 0.0.0.0" I've mentioned before, today I've noticed this line in my hosts.block file - from http://hostsfile.mine.nu: "0.0.0.0 PNAP-WDC002"

Although these are harmless, I wonder if we could get rid of such entries by adding this (hopefully harmless) command as well: sed "/^[0-9].*[0-9][$\r]/d"

I've noticed that the blocklist at http://someonewhocares.org/hosts/hosts gets updated too frequently, and when I checked I didn't see such frequent updates there. For instance, hostsblock somehow has recently found changes every day although the file itself has this line: "Last updated: Nov 19th, 2015 at 21:40"
I wonder if this is caused by a server flaw or what.

Tested with pre-alpha, it looks like the directories are created, the file is extracted from 7z, and later there's no information about redirections. Below are parts of log created with verbosity=4:

rlwpx.free.fr.WPFF.hrsk.7z is a 7z archive. Will use 7za to extract it...
Extracting entries from /var/cache/hostsblock/rlwpx.free.fr.WPFF.hrsk.7z...
Created directory /dev/shm/hostsblock/rlwpx.free.fr.WPFF.hrsk.7z.d.
Moved rlwpx.free.fr.WPFF.hrsk.7z to /dev/shm/hostsblock/rlwpx.free.fr.WPFF.hrsk.7z.d.
[...]
Un7zipped rlwpx.free.fr.WPFF.hrsk.7z.
Extracting obvious entries from rlwpx.free.fr.WPFF.hrsk.7z...
Extracted obvious entries from rlwpx.free.fr.WPFF.hrsk.7z.
Extracting less-obvious entries from rlwpx.free.fr.WPFF.hrsk.7z
Extracted less-obvious entries from rlwpx.free.fr.WPFF.hrsk.7z.
Deleting /dev/shm/hostsblock/rlwpx.free.fr.WPFF.hrsk.7z.d...

The annotation file (by default /var/lib/hostsblock.db) contains entries that do not include their source.

I don't know if this is very easy (e.g. a little modification in the "DOWNLOAD BLOCKLISTS" section of hostsblock.sh) or difficult to implement without much change in the present code, but I thought it would be very handy if we could enter a command like hostsblock.sh -v 3 -checkonly &>/tmp/hostsblock-lists-status.log to see if there are any pending updates.

Note: What I have in mind actually is to try and create, in due course, an app indicator similar to https://github.com/Sadi58/grive-indicator and https://github.com/Sadi58/indicator-chars
And my "work-in-progress" to this end (adding several files under subfolder ./src/indicator/ only) is now here: https://github.com/Sadi58/hostsblock/tree/master/src/indicator

to run hostsblock with a pdnsd needs:

1. /etc/hostsblock/rc.conf:
   postprocess(){
   /etc/rc.d/pdnsd restart
   }
2. /etc/pdnsd.conf (http://members.home.nl/p.a.rombouts/pdnsd/doc.html#sourcesection):
   source {
   owner = localhost;
   file = /etc/hosts.block;
   }

Configuration that hostsblock script checks for is not the only possible, f.i. I usually avoid modifying dnsmasq.conf directly and place config overrides in /etc/dnsmasq.d/hostsblock instead. Also, "prepend domain-name-servers 127.0.0.1" is not always required.

It would be helpful to have a way to disable hostsblock config checks and assume that everything is already configured and working properly.

I believe that hostsblock.conf should mention dnscacher variable and its recognized values. In my case dnsmasq auto-configuration fails (due to non-standard dnsmasq config) so I set dnscacher=manual which works well up to the point of hostsblock trying to restart it resulting in:

[WARN] FAILED to restart manual.

That could be fixed by enhancing below test condition:

# IF WE HAVE A DNS CACHER, LET'S RESTART IT
if [ "$dnscacher" != "none" ]; then
    _notify 3 "Restarting $dnscacher..."

Instead of having all of hostsblock run as root, have only the needed processes do so, e.g.

• sending signals to any dnscachers to restart
• write to files under /etc

Will sudo work for these other situations, or is there another method?

I saw this adaway source file at https://github.com/StevenBlack/hosts/blob/master/data/adaway.org/update.info, I hope that it can be a default as I found the adaway sources list nice.

After the latest commits I've begun getting the text below followed by a space before each line in the hostsblock.log file:
[�[0;33mINFO�[0m]
The odd character above (white question mark against black background) is displayed in my text editor as a square including 00 in the first line and 18 or 1B in the second line (my eyes are a bit old ;-)
I use this launcher script:

#!/bin/bash
/etc/hostsblock/hostsblock -v 3 &>/var/log/hostsblock.log
cat <<EOF >> /var/log/hostsblock.log
Hostsblock completed at `date +'%d/%m/%Y %H:%M'`
EOF

Line# 261 is:
echo " $@ " >> "$whitelist"

It should be:
echo " $@" >> "$whitelist"

I should add that this makes manully added whitelist urls via hostsblock-urlcheck.sh not work after the hosts.block file is updated.