AdBlocking with TomatoUSB Router – Ad free Internet for all your devices
The Internet is flooded populated with advertisements today. There are AdBlock plugins for browsers. But what about mobile devices like non-rooted or non-jailbroken Androids and iOS devices?
Recently, I chanced upon a Kickstarter project called AdTrap. It introduces a small zero configuration device that removes advertisements from your Internet connection before they reach any of your home devices. This device is connected between your modem and home router. Sounds cool? Yes if you are willing to fork out US$132 (inclusive of US$12 shipping if outside US / Canada).
Do I need the AdTrap project to achieve that? No!
So I gave it a thought – with dnsmasq on the TomatoUSB-enabled router (e.g. ASUS RT-N66U, Linksys E4200), I could probably achieve ad blocking with DNS Cache Poisoning on the TomatoUSB-enabled router. Too geek? In simpler terms, I can make the TomatoUSB-enabled router resolve known advertisement domain names / hostnames to invalid addresses like 0.0.0.0.
As a result, I will be able to filter advertisements from the Internet before they reach any of my devices at home.
I started to google around to see if anybody has used the DNS Cache Poisoning technique (using dnsmasq) and found that somebody actually did the scripts (http://goo.gl/mhykQ)! Did a little modification to it and here it is.
Updates
29 Apr 2014
Added Pixelserv for ASUS RT-AC68U. Download Pixelserv (ARM) v43-2.
28 Apr 2014
Pixelserv does not work on ASUS RT-AC68U as it was originally compiled for Broadcom architecture. The ASUS RT-AC68U is based on ARM architecture. Instead of using Pixelserv, I decided to use NGINX web server to perform the role of pixelserv. Check out AdBlocking with NGINX: Serving 1 pixel GIF and 204 No Content.
7 Dec 2013
Updated the script to use pixelserv V31. The changes are in the base64 encoded block. Previous version of pixelserv requires a parameter “-n br0” which is no longer required in V31. Updated the base64 block to reflect that.
Pre-requisites
- TomatoUSB-enabled router like ASUS RT-N66U, RT-N16, Linksys E4200. ASUS RT-AC66U
- Flashed to TomatoUSB firmware (I am using TomatoUSB on my RT-N66U and RT-AC66U)
- Geeky mind to do some troubleshooting
Inserting the AdBlocking script
- Using a web browser, login to the TomatoUSB web administration page
- Navigate to Administration -> Scripts -> WAN Up tab
- Copy and paste the contents below. Then save and reboot the router
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 | ## ALL-U-NEED Ad Blocking v3.9e ## http://goo.gl/mhykQ ## Original script by YAQUI ## Updated by ~nephelim~, Syl, jochen, groosh, ng12345, ray123, mstombs ## base64 decoder by Danny Chouinard's sleep 10 ADB="/tmp/ADBLOCK.sh" { cat <<'ENDF' >$ADB #!/bin/sh OPTIMISE="Y" GETS="1 2 3 4 5 6" TRIM_BEGIN=2 S1="http://mirror1.malwaredomains.com/files/justdomains" S2="http://www.malwaredomainlist.com/hostslist/hosts.txt" S3="http://someonewhocares.org/hosts/hosts" S4="http://winhelp2002.mvps.org/hosts.txt" S5="http://sysctl.org/cameleon/hosts" S6="http://hosts-file.net/ad_servers.asp" USEWHITELIST="Y" # N/Y/R for remote WURL="http://example.com/whitelist.txt" WHITE="intel.com www.shadowandy.net" BLACK="" USEPIXELSERV="N" PXL_IP=192.168.1.10 PXL_EXE="/tmp/pixelserv" PXL_URL="http://www.example.com/pixelserv" UPLOAD="N" FTP_SERVER="example.com" FTP_USER="" FTP_PASS="" FTP_PORT=21 FTP_PATH="/gen" ADD_CONF="N" USEHOSTS="N" ROUTER="Y" NIP="0.0.0.0" ENDF } UPDATE="Y" AUP() { if [[ "$UPDATE" == "Y" ]] ; then if [[ "$(cru l | grep AdUpd | cut -d '#' -f2)" != "AdUpd" ]] ; then cru a AdUpd "0 4 * * * $ADB" fi fi } #### DO NOT EDIT BELOW #### b64="openssl enc -base64 -d" [[ "$(echo WQ==|$b64)" != "Y" ]] && b64="b64" b64(){ awk 'BEGIN{b64="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"} {for(i=1;i<=length($0);i++){c=index(b64,substr($0,i,1));if(c--) for(b=0;b<6;b++){o=o*2+int(c/32);c=(c*2)%64;if(++obc==8){if(o) {printf"%c",o}else{system("echo -en \"\\0\"")}obc=o=0}}}}';} { cat <<'ENDF'| $b64 |gunzip >>$ADB H4sICJEmo1IAA2RlY29kZWQudHh0AK1Xe2/aSBD/359iuvHFkKsx5Jq7KtSRCI+A SgCBc+kpJBG1F1gVbGo7j17od7/ZWRsMJG2kXpMq693Zmd+8Zyv9s4Gt72ln9Y7N rHi+sCbcZ5pz3ks+Yz5fMK3a7TSSDTfwx0xrdgfOINmZBlEcMe2y0WrXk62HqYg5 0/CfGMPVFegVlANvbCjC9TUsl3KP6blFBEuYhHwBJgfDNCVvI90y/OBz4H0z8gxs GxjDm2WIp9zXwjmYY9ARc2HuHWU/Z6Mo1sZCCq62+7k8PGVO0yUqly6lXuma8Gvf 5d3epzbdVeCZfjGo91qf6u1Bvf+3QvNPBo6iegPmI+h487b+qb4+fJjwGMzu+oQW F/225k7ngQe/ry9J4GIsbSAm8DksHhfVUau3ApJbCC8Yw0I88lnEw/sd22zJafXK EM04WrMEZZB3AL6I2QzMQevsYtAvwS7LMhCOVwrUZsFkwkOo1E7b3epHqPf73f4x uCPfD2KI4lEYr29r8o/G70czYJ1Wz04wMqk5/pLtO1Uy/UWnVm/YRW0chCBA+NKD zqAMXgCKwV04s9lQH+iCgdYr2XqOu9MAdNxfQsQ9MKLlzTSOF1fR9cGxZV3dWNcH w5xVOBjm9eWwtDTyWvPn14Y5dTGPF9NrEtOjxFSCQ/hDYtJiMecpL4xl1qxXauiC EjQdp2eVCqVhOPSbmCjHoDfpoxr4PndjEfhorVkQcbnJlprvgvkAR5IM3heXKjsE tDGyzfPAE2PBveNlHILpARuGLE++QldJCEzmmPROEg2fQz76Qg71Ap9vUtoblMre JaJVFpFUJyeUVj5/UAwUB52IUc+vaIGUwVYWEiNFnt1OybXzmrQ9pm90N18fL927 mBTDH3NcygPSHW7RIZhdskQz5KoUw9XhSrmtGD0GPwB3OvInPIJI+C4HgiYVfgv8 UcTCnxSyZUXqL/eTOJ3fZ08ylYcC2Om3zimEZTSZAgubFlnDPQwfy8LVTenwr2Gh SL+lKxjG1we0r5YJzfFx9mgWuKOZrLLy1LrRLc9QVYzk1QYkbTdRdvJEWonqkYx2 WZRMrLXrSIIT4gobZU0XYE44HqBWt6f1s1ZnXWvkHpCj3VGs7p6cqFK7bXEpUWX/ D+tFCk5LI1ZZdODk8k8YCkebobAdBgnkTG+Q3WZ/n/TA2zIy9BxhTc7zpMwvBAl2 mTJ9JaWNMgdFnay6U1rX2pdNZ7upXDZbTr3dGji7TUW6M8ImSh4lOjRIElLArLRs SZKlKlvWsGAN8f/EyFseU24gG/LZyyL7mdaxDo9L7E/Pxwc1SXghrajrz0SUcEjR yt6eJACsMsFQ20kwq967obPyEx3kf0H110TcNm6V5SSF6aftSvUjg1RSkpZDH4Wp vUSHNPLJ2d3epq/xu3XeGtR3/bxtwygIZWQBDVWFQiEbLjR07bKQV8C8AzNQOUgw SPPRwxcwG2AUDDAod5+6fZza2PcnNHVO2J1GWZwUy8I084tQ+Ki/YHusrNYM29F3 QyWaFLFcsdvbYlf2ACH9hmzFOCd8jz/m9OJbL//GLu3vdxpv7GL+ybP1YsJYN00U rCkItN4EwQpMF1kQiEKVJumWbYORochs3CtkqvPKEuSQakc5xOk6WEMesMfO4AOd 53drFRIB9+NQ8Ejdbji9i95m8vba3UrtpVmQ6f3uhVPvP5PW8WJxR97SkektOrUP 5kJ99CqDAZi95KPbd9RKDp1IlZA4zYx/ffd/45eZwBq1zuB8u1S9EHvuIi23OLBI kmSs/fPdu3QjUwVwtFoaOk5+OOQbz5yXBDbHQhF/SrBqeitCUjmhlqzkQDbyvJBH kW3tW4rx0ljpo8putXH2Ol2AupjFY9fy/Gg+ir4W5CxOiU3vBHmcfsCHD/VuQ0Px vkkRaK+fQpo8IbAvM8xCrNRq2yBx61aKeQbkK/RQAzbytAkry47dtIvSaeZeadQ4 SxTypu7CHN3F0yAU8SgW91IJd8rNSPzL7cPiu/cyXcxR9M137SPSdGXp0/bHZO4J 7wX2y0RpfAQEi9fATunVO9AcixlXCij8zx5n7digQYgstH5YGsm1nWclpGK30z+V IyII73wfi/GP2ghGn3zjyJKdOhmLdrJcQcMxQT4q5esG5zU50NA8IMuS7BX0l4qM 9tMKQtlJYZ0EDhkelad3rxSk/Qf82ewR1Q8AAA== ENDF } chmod 775 $ADB $ADB AUP |
Verifying that the script is running
- After the TomatoUSB router has rebooted, ssh into the router
- Verify that script_wanup.sh is created by typing “ls /tmp” without the quotes
- Verify that the adblock script is running by typing “cat /var/log/messages | grep ADBLOCK” without the quotes
It should show that it has N amount of entries for hosts and dnsmasq is running - Congratulations! Ad block is running!
How do I remove the script?
To remove the script. Simply delete the content of the WAN Up tab, save and reboot. The /tmp/script_wanup.sh should go away.
What is in the base64 text block
In TomatoUSB, all scripts are limited to 4096 characters hence some scripts are gziped and converted to base64. The contents of the base64 block is based on the following script. You can also verify by checking the contents of the file /tmp/script_wanup.sh. You do not have to copy and paste the below script anywhere. I am showing here for illustration purposes only. If you are geeky enough, you can also use the command
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 | ARGS=$# GEN="/tmp/gen" TMP="/tmp/temp" CONF="/tmp/conf" HOSTS="/tmp/hosts" WFILE="/tmp/white" if [[ $ARGS != 0 ]] || [[ "$(ps | grep -e '--conf' | grep 'nobody')" == "" ]]; then rm -f $GEN.md5 rm -f $GEN.last fi CLR() { rm -f $GEN rm -f $TMP rm -f $CONF rm -f $WFILE } PXL() { if [[ "$USEPIXELSERV" == "Y" ]]; then if [[ ! -x $PXL_EXE ]]; then wget -O $PXL_EXE $PXL_URL chmod +x $PXL_EXE fi ifconfig br0:0 $PXL_IP if [[ "$(pidof pixelserv)" == "" ]]; then $PXL_EXE $PXL_IP; sleep 1 ; else kill -SIGUSR1 $(pidof pixelserv); fi if [[ "$(pidof pixelserv)" == "" ]]; then logger ADBLOCK ERROR: cannot start pixelserv else eval "NIP=$PXL_IP" fi fi } NC() { UNDEF=0 for i in $GETS; do eval url="\$S$i" P1=$(echo $url| sed 's|^http[s]*://[^/]*\(/.*\)$|\1|') H1=$(echo $url| sed 's|^http[s]*://\([^/]*\)/.*$|\1|') for x in 1 2 3; do time=$(echo -e "HEAD $P1 HTTP/1.1\r\nHost: $H1\r\nConnection: close\r\n"| nc -w 5 $H1 80|grep -i Last-Modified:|tr -d "\r") if [ "$time" != "" ]; then break; fi done if [ "$time" == "" ]; then UNDEF=1; fi echo $time>>$GEN.new done if [ $UNDEF -eq 1 ]; then rm -f $GEN.last; fi if [ -f $GEN.last ]; then MD1=$(md5sum $GEN.last|cut -d " " -f1) MD2=$(md5sum $GEN.new|cut -d " " -f1) if [ "$MD1" == "$MD2" ]; then logger ADBLOCK: no changes since last time, exiting. rm -f $GEN.new exit fi fi mv -f $GEN.new $GEN.last } TRIM() { sed -ie ' s/\#.*$// s/^127\.0\.0\.1[ \t]*// s/[ \t]*$// s/^::1[ \t]*// s/localhost$// /^$/d' $TMP } DS() { for i in $GETS; do eval url="\$S$i" if wget $url -O - | tr -d "\r" > $TMP ; then if [[ $i -ge $TRIM_BEGIN ]]; then TRIM ; fi cat $TMP >> $GEN logger ADBLOCK: $url else logger ADBLOCK ERROR: cannot get $url fi done } TST(){ MD5=$(md5sum $GEN|cut -d " " -f1) if [[ -f $GEN.md5 ]] && [[ $MD5 == $(cat $GEN.md5) ]]; then logger ADBLOCK: no changes since last time, exiting. CLR; exit else echo $MD5>$GEN.md5 fi } LWHT() { if [[ "$USEWHITELIST" == "Y" ]]; then for site in $WHITE do sed -i "/$(echo $site|sed 's/\./\\./g')/d" $GEN done elif [[ "$USEWHITELIST" == "R" ]]; then if wget $WURL -O - | tr -d "\r" > $WFILE ; then logger ADBLOCK: whitelist $WURL sed -i -e '/\#.*$/ s/\#.*$//' -e '/^$/d' $WFILE for site in $(cat $WFILE) do sed -i "/$(echo $site|sed 's/\./\\./g')/d" $GEN done else logger ADBLOCK ERROR: cannot get whitelist $WURL fi fi echo "$BLACK" |sed 's/[ \t]*/\n/g'|sed '/^$/d' >> $GEN } OPT() { if [[ "$OPTIMISE" == "Y" ]]; then logger ADBLOCK: sorting hosts... if [[ "$USEHOSTS" == "Y" ]]; then sort -u -o $TMP $GEN else awk -F '.' 'BEGIN{ORS=""}{for(i=NF;i>0;i--)print $i"#";print "\n"}' $GEN|sort| awk -F '#' 'BEGIN{ORS="";d = "%"}{if(index($0,d)!=1&&NF!=0){d=$0;print $--NF; for(i=--NF;i>0;i--)print "."$i;print "\n"}}' > $TMP fi logger ADBLOCK: hosts sorted. fi mv -f $TMP $GEN } CNT() { TOT=$(wc -l < $GEN) logger ADBLOCK: $TOT entries } FTPUP() { if [[ "$UPLOAD" == "Y" ]]; then if [[ "$ROUTER" == "Y" ]]; then ftpput -u $FTP_USER -p $FTP_PASS -P $FTP_PORT $FTP_SERVER $FTP_PATH $GEN else ncftpput -u $FTP_USER -p $FTP_PASS -P $FTP_PORT $FTP_SERVER $FTP_PATH $GEN fi fi } FDNSM() { if [[ "$USEHOSTS" == "Y" ]]; then cp -f $GEN $HOSTS chmod 644 $HOSTS sed -i -e 's|^|'$NIP' |' $HOSTS sed -i -e '1i127.0.0.1 localhost' $HOSTS else sed -i 's|^.*$|address=/&/'$NIP'|' $GEN fi } LCFG() { if [[ "$USEHOSTS" == "Y" ]]; then cat /etc/dnsmasq.conf >> $CONF cat >> $CONF <<EOF addn-hosts=/tmp/hosts EOF else cat /etc/dnsmasq.conf >> $GEN fi } ADDCFG() { if [[ "$ADD_CONF" == "Y" ]]; then if [[ "$USEHOSTS" == "Y" ]]; then eval "CFG=$CONF" else eval "CFG=$GEN" fi cat >> $CFG <<EOF dhcp-authoritative cache-size=2048 log-async=5 EOF fi } LBLK() { service dnsmasq stop if [[ "$USEHOSTS" == "Y" ]]; then dnsmasq --conf-file=$CONF else dnsmasq --conf-file=$GEN fi } FS() { if ps | grep 'dnsmasq' | grep 'nobody' ; then logger ADBLOCK: dnsmasq is running else logger ADBLOCK ERROR: restarting dnsmasq... dnsmasq fi } CLR PXL NC DS TST LWHT CNT OPT CNT FTPUP if [[ "$ROUTER" == "Y" ]]; then FDNSM LCFG ADDCFG LBLK FS fi CLR |
The AdBlock Sources
I have actually updated the AdBlock sources to one that is more suitable for me. The codes that are responsible for the AdBlock sources are:
1 2 3 4 5 6 7 8 9 | OPTIMISE="Y" GETS="1 2 3 4 5 6" TRIM_BEGIN=2 S1="http://mirror1.malwaredomains.com/files/justdomains" S2="http://www.malwaredomainlist.com/hostslist/hosts.txt" S3="http://someonewhocares.org/hosts/hosts" S4="http://winhelp2002.mvps.org/hosts.txt" S5="http://sysctl.org/cameleon/hosts" S6="http://hosts-file.net/ad_servers.asp" |
You can choose which hosts files will be downloaded to block ads, update the variable GETS. I am downloading all of them.
A little explanation about TRIM_BEGIN=2:
S1 contains hostname only, it will be copied without formatting. While S2 to S6 are file formatted to replace hosts file, these files will be formatted to works with AdBlock. TRIM_BEGIN tells AdBlock which sources need formatting (from S2 to the end). Well usually you don’t need to touch this. Yes, you can add new blacklists to it.
If OPTIMISE=”Y”, AdBlock will remove duplicated entries.
Custom Whitelist and Blacklist
You can add inline whitelist and blacklist to the script by updating this chunk of the codes:
1 2 3 4 5 | USEWHITELIST="Y" # N/Y/R for remote WURL="http://example.com/whitelist.txt" WHITE="intel.com www.shadowandy.net" BLACK="" |
Auto Updating of AdBlock sources
By setting UPDATE=”Y”, AdBlock will update the AdBlock entries at 4am daily according to the time specified in the cron rule.
1 2 3 4 5 6 7 8 | UPDATE="Y" AUP() { if [[ "$UPDATE" == "Y" ]] ; then if [[ "$(cru l | grep AdUpd | cut -d '#' -f2)" != "AdUpd" ]] ; then cru a AdUpd "0 4 * * * $ADB" fi fi } |
Getting Pixelserv to work
Pixelserv is a super minimal web server whose sole purpose is to serve a 1 x 1 pixel transparent gif file regardless of request. When USEPIXELSERV is enabled (set to “Y” instead of “N”). The ad blocked hostnames are resolved to PXL_IP (which actually points to the router itself). This result in your web browser requesting resource from Pixelserv.
1 2 3 4 | USEPIXELSERV="N" PXL_IP=192.168.1.10 PXL_EXE="/tmp/pixelserv" PXL_URL="http://www.example.com/pixelserv" |
To get Pixelserv to work, simply do the following:
- Download Pixelserv V31 or Pixelserv (ARM) V34-2 (for RT-AC68U).
For non-ARM routers (e.g. RT-AC66U, RT-N66U) use the Pixelserv V31. For ARM-based routers (e.g. RT-AC68U, RT-N56U), use the Pixelserv (ARM) V34-2.
- Unpack the package and upload it to your webhost or the public folder of your Dropbox
- Update PXL_URL to point to the location of pixelserv on your webhost or Dropbox without the https (e.g. http://dl.dropbox.com/u/pixelserv)
- Update PXL_IP to a unused IP address on your network (e.g. 192.168.1.10)
- Update USEPIXELSERV to “Y”
- Change TomatoUSB Web Admin page to run on port 8080 (TomatoUSB –> Administration –> Admin Access –> HTTP Port)
- Save the changes and restart the router
Do note that you will need to append :8080 to your router IP if you intend to access the Web Admin in the future.
Some useful hosts to whitelist
I will update the below list whenever I find some desktop / web applications breaks or do not work as they should. Do share your findings too.
1 2 3 4 5 6 7 8 9 10 11 12 | swupmf.adobe.com # required for Adobe Application Manager to work www.namecheap.com # 1 of 3 required for Namecheap.com to work properly manage.www.namecheap.com # 2 of 3 required for Namecheap.com to work properly support.namecheap.com # 3 of 3 required for Namecheap.com to work properly clients1.google.com # Required for Android / iOS WiFi to detect that it is not using a public hotspot clients2.google.com # Required for Android / iOS WiFi to detect that it is not using a public hotspot clients3.google.com # Required for Android / iOS WiFi to detect that it is not using a public hotspot clients4.google.com # Required for Android / iOS WiFi to detect that it is not using a public hotspot clients5.google.com # Required for Android / iOS WiFi to detect that it is not using a public hotspot clients6.google.com # Required for Android / iOS WiFi to detect that it is not using a public hotspot clients7.google.com # Required for Android / iOS WiFi to detect that it is not using a public hotspot s.amazon-adsystem.com # Required for Amazon Android App to work |
