Web Filtering HTTPS Traffic on Raspberry PI

Update (December 2015): The new and improved tutorial how to build HTTPS filtering Squid with Diladele Web Safety ICAP web filter is available from http://docs.diladele.com/administrator_guide_4_4/install/rpi/index.html

This article will tell you how to compile and configure Squid proxy server running on Raspberry PI, capable of filtering encrypted HTTPS connections using Diladele Web Safety ICAP content filtering server. Being able to look into HTTPS contents greatly increases your ability to control what is allowed and accepted within your network while keeping inappropriate contents away.

Why Should We Filter HTTPS?

HTTPS protocol was designed to provide secure means of communications between internet browser and remote web servers. In order to achieve this goal HTTPS protocol encrypts data passing through established connections so that it cannot be decrypted in reasonable amount of time thus preventing anyone from sniffing the contents interchanged over this connection. This protocol was primarily invented to enable safe and secure communication between the user and financial sites or government institutions over the insecure medium such as the Internet.

Recently more and more web sites started to use HTTPS encrypted communications to increase online privacy of users. Google who as first enabled HTTPS for all its searches by default probably initiated this trend. Although there are no doubts that HTTPS encryption is a good thing for safety on the wire we must take into account that it also creates several problems for controlled networks typically found at home or offices. The main problem here is the essence of the HTTPS protocol itself – no one except the browser and the web server is able to see and thus filter transferred data. This may not always be desired. Contents that are usually blocked suddenly become immediately accessible by anyone. As an example imagine a school network where minors can see questionable content by just mistyping a search term in Google. Moreover the law often forces administrators in educational institutions to block access to such content (e.g. CIPA for educational environments) and encrypted access to web sites makes it nearly impossible to fulfill such an obligation.

In order to overcome these limitations it is advised to setup HTTPS filtering of web contents with help of SSL bump feature of Squid proxy server and Diladele Web Safety web filter.

How It Works

In order to filter web requests user’s browser needs to be explicitly directed to use the proxy that is deployed in the same network. It is also possible to set the transparent proxy but we are not going to explain how this is done in this tutorial because steps involved are quite different from explicit proxy setup.

When a user tries to navigate to a web site, browser sends the request to proxy server, asking it to get the requested page on his behalf. The proxy establishes a new connection to the remote site and returns the response to browser. If normal HTTP is used then proxy is able to see the original contents of the response and filter it. In case of HTTPS the flow of data is a little different. Browser asks the proxy to establish a virtual tunnel between itself and remote server and then sends encrypted data through the proxy. Domain name to which a virtual tunnel is being established is usually known, so proxy is able to block this virtual tunnel when it finds out that domain name belongs to a prohibited category. Unfortunately this is not a complete solution as there are a lot of sites on the Internet which are general in nature (like Google or YouTube) but allow you to easily navigate to something undesired.

To improve the quality of web filtering and get access to contents in encrypted connections, browsers in the network may be setup to trust proxy to act on their behalf for establishing HTTPS connections, filtering them and passing the allowed data to clients while blocking everything that is not allowed. Although this assumption is too strict to be implemented in public networks, it is easily doable in controlled home, educational or corporate environments where administrators act as sole owners of network devices and may force any trusting rules. After established trust browser is able to ask proxy to connect to a remote site in a safe manner with HTTPS, proxy is able to decrypt the traffic, filter it, encrypt it again and pass it to browser. As browser trusts the proxy it continues working with filtered HTTPS without any errors or warnings.

Unfortunately the default Squid version included into Raspbian OS for Raspberry PI does not contain compile switches necessary for successful HTTPS filtering. We need to recompile Squid proxy, reinstall and reconfigure it with additional list of options.

Build Squid with SSL Bump and ICAP Client

Before compiling it is considered a good practice to bring the operation system to a most recent state. This can be done by running the following commands in the terminal.

$sudo apt-get update && sudo apt-get upgrade && sudo reboot

In order to build the Squid from source we need to install some build tools.

$ sudo apt-get install devscripts build-essential fakeroot libssl-dev pkg-config

Default version of Squid in Raspbian Wheezy is too old so we will need to do the following trick to get Squid version from Raspbian Jessy repositories. The idea is to temporary switch our repositories to Jessy, get Squid’s source code and switch back to Wheezy’s repositories.

# fetch the source for the package to re-build from the jessy repositories
$ echo "Copying sources lists..."
$ sudo cp /etc/apt/sources.list /etc/apt/sources.list.default
$ sudo cp /etc/apt/sources.list /etc/apt/sources.list.jessy
$ sudo cp /etc/apt/sources.list /etc/apt/sources.list.wheezy

$ echo "Adding source repositories..."
$ sudo echo "deb-src http://archive.raspbian.org/raspbian jessie main contrib non-free" >> /etc/apt/sources.list.jessy
$ sudo echo "deb-src http://archive.raspbian.org/raspbian wheezy main contrib non-free" >> /etc/apt/sources.list.wheezy

$ echo "Getting Squid from Jessy..."
$ sudo cp /etc/apt/sources.list.jessy /etc/apt/sources.list
$ sudo apt-get update
$ sudo apt-get source squid3=3.3.8-1.2 --download-only
$ sudo apt-get source libecap2=0.2.0-1 --download-only

$ echo "Getting Squid dependencies from Wheezy..."
$ sudo cp /etc/apt/sources.list.wheezy /etc/apt/sources.list
$ sudo apt-get update
$ sudo apt-get build-dep squid3

$ echo "Reverting default repository..."
$ sudo mv /etc/apt/sources.list.default /etc/apt/sources.list
$ sudo apt-get update

Now we have sources of Squid 3.3.8 in current folder. Running the following command gets all necessary dependencies for Squid from Wheezy repos and unpacks Squid source package together with all system integration scripts and patches provided by Debian/Raspbian developers.

$ echo "Running dpkg-source..."
$ sudo dpkg-source -x squid3_3.3.8-1.2.dsc
$ sudo dpkg-source -x libecap_0.2.0-1.dsc

Before we build Squid we need to build and install dependency library which is not included into Wheezy.

$ echo "Building libecap2..."
$ pushd libecap-0.2.0
$ dpkg-buildpackage -rfakeroot -b
$ popd

$ echo "Installing libecap2..."
$ sudo dpkg --install *.deb

Sources are unpacked into squid3-3.3.8 folder. We need to modify configure options in debian/rules and debian/control to include compiler switches (–enable-ssl and –enable-ssl-crtd) necessary for HTTPS filtering.

$ pushd squid3-3.3.8
$ echo "Patching Squid source..."
$ sudo patch debian/rules < "../rules.patch"
$ sudo patch debian/control < "../control.patch"
$ sudo patch src/ssl/gadgets.cc < "../gadgets.cc.patch"
$ echo "Building Squid..."
$ sudo dpkg-buildpackage -rfakeroot -b
$ popd

As the rules.patch and control.patch are rather big they are not included into the article directly. All patches are part of the corresponding download archive. Please note, that one file in source code of Squid Proxy needs to be adjusted too (src/ssl/gadgets.cc). This change is needed to prevent Firefox error sec_error_inadequate_key_usage that usually occurs when doing HTTPS filtering with latest versions of Firefox browser. If you use only Google Chrome, Microsoft Internet Explorer or Apple Safari this step is not required.

Please take into account that the build process is rather slow (it took approximately 10 hours on stock Raspberry PI) while all required *.DEB packages need to be built on Raspberry PI and cannot be cross compiled.

Install Diladele Web Safety

SSL Bumping feature alone is not enough to block questionable web content. We also need the filtering server that could be paired with Squid. We will use Diladele Web Safety (DDWS) formerly known as QuintoLabs Content Security (qlproxy) for the filtering and blocking part. It is an ICAP daemon capable of integrating existing Squid proxy and providing rich content filtering functionality out of the box. It may be used to block illegal or potentially malicious file downloads, remove annoying advertisements, prevent access to various categories of the web sites and block resources with explicit content.

We will use version 3.4.0 of qlproxy. It was designed specifically with HTTPS filtering in mind and contains rich web administrator console to perform routine tasks right from the browser.

By default, DDWS comes with four polices preinstalled. Strict policy contains web filter settings put on maximum level and is supposed to protect minors and K12 students from inappropriate contents on the Internet. Relaxed policy blocks only excessive advertisements and was supposed to be used by network administrators, teachers and all those who do not need filtered access to web but would like to evade most ads. Third policy is tailored to white list only browsing and the last group contains less restrictive web filtering settings suitable for normal web browsing without explicitly adult contents shown.
In order to install Diladele Web Safety for Squid Proxy, download package for Raspberry PI from Diladele B.V. web site at http://www.quintolabs.com using browser or just run the following command in terminal.

$ wget http://updates.diladele.com/qlproxy/binaries/3.4.0.9307/armhf/release/rpi/qlproxy-3.4.0.9307_armhf.deb

Administration console of Diladele Web Safety is built using Python Django framework and is usually managed by Apache web server. To install packages required for correct functioning of web UI run the following commands in the terminal.

$sudo apt-get -y install python-setuptools
$sudo easy_install django==1.5
$sudo apt-get –y install apache2 libapache2-mod-wsgi

Install the DEB package and perform integration with Apache by running the following commands.

$ sudo dpkg --install qlproxy-3.4.0.9307_armhf.deb
$ sudo a2dissite 000-default
$ sudo a2ensite qlproxy.conf
$ sudo service apache2 restart

Configure Squid for ICAP Filtering and HTTP Bumping

The Squid packages we have compiled previously need to be installed on the system. To perform installation run the following commands.

$ sudo apt-get install ssl-cert
$ sudo apt-get install squid-langpack
$ sudo dpkg --install squid3-common_3.3.8-1.2_all.deb
$ sudo dpkg --install squid3_3.3.8-1.2_armhf.deb
$ sudo dpkg --install squidclient_3.3.8-1.2_armhf.deb

In order to recreate original SSL certificates of the remote web sites during HTTPS filtering Squid uses a separate process named ssl_crtd that needs to be configured like this.

$ sudo ln -s /usr/lib/squid3/ssl_crtd /bin/ssl_crtd
$ sudo /bin/ssl_crtd -c -s /var/spool/squid3_ssldb
$ sudo chown -R proxy:proxy /var/spool/squid3_ssldb

Finally, modify Squid configuration file in /etc/squid3/squid.conf to integrate it with Diladele Web Safety as ICAP server. Due to the size of the patch file its text not included into the article directly but is part of the download archive.

$ sudo cp /etc/squid3/squid.conf /etc/squid3/squid.conf.default
$ sudo patch /etc/squid3/squid.conf < "../squid.conf.patch"
$ sudo /usr/sbin/squid3 -k parse

From now on Squid is capable of HTTPS filtering and we may continue filtering adjustments from Web UI of Diladele Web Safety.
Navigate to http://YOUR_PROXY_IP_ADDRESS/ and login with default name root and password P@ssw0rd. Select Settings / HTTPS Filtering / Filtering Mode. Diladele Web Safety may either filter specific HTTPS sites or all of them with exclusions. Total filtering is more tailored to providing very safe network environments.

Select the desired mode, click Save Settings, add target domains or exclusions as you like and then restart ICAP server by clicking on the green button in the top right corner as indicated on the following screenshots.

webui1

webui2

webui3

Run the following command in terminal on the proxy.

$ sudo service squid3 restart

Navigate to google.com and see that HTTPS filtering is indeed active. The following warning shows that Squid was able to bump the SSL connection, filtered it and encrypted in again using Diladele Web Safety’s generated certificate.

webui5

In order to get rid of these warnings, we must install the myca.der certificate file from into the browser and mark it as trusted. Again navigate to http://YOUR_PROXY_IP_ADDRESS. Select Settings / HTTPS Filtering / Certificates and select the one that matches your operating system or device. Instructions on how to install the certificate in each operating system or device is slightly different, the following screens show how to install the DER file in Microsoft Internet Explorer. For other devices please take a look at Online Documentation of Diladele Web Safety.

ie_cert1

ie_cert2

ie_cert3

ie_cert4

ie_cert5

Reopen your browser, navigate to Google and make sure the certificate warning is away. If you click on the lock icon in the internet address box then it clearly indicates the google.com was signed by proxy’s certificate and not by original certificate by google.

ie_cert6

ie_cert7

If you try to search Google with some adult only terms (e.g. NSFW) Diladele Web Safety blocks the access to explicit contents showing its denied page.

ie_cert8

Please be sure to change the default certificates that come with installation package of Diladele Web Safety to something more unique for your network. For instructions on how to regenerate your own certificates for this purpose consult Online Documentation of Diladele Web Safety.

Resume

Now we have HTTPS web filtering up and running and our network environment become a little safer for those who need protection at most. Next steps would be direct all clients browsers to use Squid proxy, regenerate the default proxy certificates, setup authentication and authorization to get user specific reports in Diladele Web Safety, integrate it with e.g. Active Directory using Squid’s support for Kerberos authentication and optionally setup transparent HTTPS filtering. It is also advisable to setup the caching DNS server on Squid proxy to further increase speed of connections.

Links

  1. Diladele B.V. web site at http://www.quintolabs.com
  2. Online Documentation of Diladele Web Safety
  3. Squid Proxy Wiki on SSL Bumping
  4. Raspbian Web Site.
  5. Download archive with installation scripts and patches

About sichent

sichent
This entry was posted in ARM, Diladele, HTTPS, ICAP, Raspberry PI, squid, web filter and tagged , , . Bookmark the permalink.

43 Responses to Web Filtering HTTPS Traffic on Raspberry PI

  1. Great walkthrough, thank you so much for sharing 🙂

    btw, there is now a new version of qlproxy to qlproxy-3.1.0.2992_armhf.deb as the file in your patch script is returning a 404 🙂

  2. Matthew Potgieter says:

    Great walkthrough, thanks.

    I have done on an old desktop this last week, and it worked very nicely, but I am now itching to try it on my Pi for home use… What is the performance like? Smooth or is there a noticeable lag?

    What about reports for both HTTP and HTTPS. Will Diladele be able to offer this to me, or can we install Free-SA or SARG side by side with Diladele? I have tried, but haven’t been able to get both websites to run at the same time… Any pointers in the right direction would be appreciated.

    As to going forward, how would we make it transparent for both HTTP and HTTPS? I played around building a normal transparent HTTP squid proxy with one nic last week, but I have checked the squid.conf file and the SSL Bump lines wouldn’t allow for the same configuration of http_port. Are their any documents on how to proceed?

    • sichent says:

      Hi Matthew,

      Well we did extensive testing on RPI and you can say that it technically works but performance wise it is far from being perfect. Major limiting factors are CPU horsepower and design of ethernet on RPI that makes it completely impossible to download faster than a couple of megabytes per second. So for a typical household now we suggest something like Atom… this may change however when more-core-arm announced this year start to become affordable.

      If you have https bump in place then all URLs from HTTPS connections will be included into reports automatically, so hopefully if collection reports present in Diladele satisfies all requirements it will be enough. If there are any reports not present – just send a feature request to support@diladele.com describing what is needed and it will probably be included into next release.

      For transparent settings for HTTPS on CentOS for example see – http://xmodulo.com/2014/04/transparent-https-filtering-proxy-centos.html, we will base our documentation of this article after a while. And yes it is possible to have transparent squid for http/https and for explicit proxying at the same time (not on RPI though as it has only one NIC).

      Raf

      • Matthew says:

        Hi Sichent,

        Thanks for the swift reply.

        Good to know about the performance trouble (means that my openElec RPi is safe for the meantime). You have however have got me rather excited with your link about the example on CentOS… Will be trying that this weekend I think.

        I would still like to integrate either Free-SA or SARG to run alongside Diladele though, as the pretty graphs are always nice to take to show off to the boss. But, I’ll have to do a bit more research on how to do that.

        Thanks for taking the time to reply… Much appreciated.

  3. sichent says:

    Thanks Matthew, I have included your request – I cannot say it will be in 3.3 but we will definitely include it as a lot of users asking for nice graphics not just dummy tables 🙂 in reports. See issue at https://github.com/ra-at-diladele-com/qlproxy_external/issues/356

  4. Robert Hanuschke says:

    Very good tutorial and the scripts work like a charm, made everything so easy!

    One question though.
    In the Administrators Guide chapter for filtering HTTPS traffic, there is the following command at the end:
    sudo apt-mark hold squid3 squid3-common
    Shouldn’t that also be included in the tutorial here and in the scripts to not lose the adjusted squid installation?

  5. Matthew Robinson says:

    Thanks for writing up this info page and scripts! I’ve installed Diladele Web Safety on a Raspberry Pi, and was wondering what the next step is if I want to set up as a transparent proxy. I see you have a writeup on “Transparent SSL / HTTPS filtering on CentOS” – is it much different to do this on the Pi?

  6. Kleanthis says:

    Will this project work for a small office serving 20 users?

    • sichent says:

      Hello Kleanthis,
      Unfortunately to properly handle 20 users load RPI is just not enough. I was not able to be more than 3MB per second from its NIC (due to internal limitations). You would use a proper desktop/server cpu from intel/amd. Not sure about all those new 8 core ARM chips coming out…

  7. Pedro Silva says:

    Hi sichent,

    I’m having some problems when run this command “sudo dpkg-buildpackage -rfakeroot -b” for squid3-3.3.8. But it gives me always errors. some times gives me an error right at the beginning, sometimes past 20min, 5min other, is never in the same place. I’ve tried more than 30 times. Even as root, with rasbian as default, but the problem persists. I’ve tried with scripts, but it’s the same thing. Errors, segmentation fault, etc.

    This is a project for school, and I’m getting desperate. Can you help me?

    Best Regards,
    Pedro Silva

    • sichent says:

      Hello Pedro, have you tested it on another RPI may be this is a hardware issue? There are no issues during compilation at all here. May be you have overclocked your RPI and it is not stable enough?

      Raf

      • Pedro Silva says:

        Hi sichent,

        Thank you for your reply.
        I had my RPI overclocked with 1Ghz, now is completely default without any overclocking, without any modification. and I’m coming at the point I mentioned that had problems, until now no problem. I hope that’s the problem. 🙂

        I will keep in touch.

        Thank you!

      • sichent says:

        🙂 hope it will work

      • Pedro Silva says:

        It works!
        It seems that the problema was the overclocking.

        Thank you!

      • Pedro Silva says:

        Hi again,

        I’m getting this errors when reload squid servisse

        2014/08/28 15:13:25| Warning: empty ACL: acl qlproxy_https_targets dstdomain “/opt/qlproxy/etc/squid/https_targets.conf”
        2014/08/28 15:13:25| Warning: empty ACL: acl qlproxy_icap_edomains dstdomain “/opt/qlproxy/etc/squid/icap_exclusions_domains.conf”
        2014/08/28 15:13:25| Warning: empty ACL: acl qlproxy_icap_etypes rep_mime_type “/opt/qlproxy/etc/squid/icap_exclusions_contenttypes.conf”

        Why this happening?
        How can i know the proxy address to put in my browser?

      • sichent says:

        Hello Pedro, these are warnings not errors, they can be ignored. The ip address of your proxy *is* the IP browsers need to talk to. Port 3128.

  8. Pedro Silva says:

    Hi,

    So there is no problem with having that warnings?
    Proxy working, thanks.

    • sichent says:

      No problems at all. If you are interested, these are generated by squid as you do not have any exclusions in ICAP domains, content type, SSL exclusions etc. So the files being included by squid are empty – hence the warnings. Just ignore them.

  9. Benoit says:

    Thank you for this tutorial.
    Thus, 05_squid.sh gives me the following error :
    dpkg: error processing squid3-common_3.3.8-1.1_all.deb (–install):
    cannot access archive: No such file or directory
    Errors were encountered while processing:
    squid3-common_3.3.8-1.1_all.deb

    Is this file still available?

  10. What are the content types we can block globally? Can we please give us the codes?

    How can we disable file monitoring completely? I only need web filtering, no file or ad filtering needed. Need to save resources and speed things up!

  11. Daniel G says:

    Hello sichent!
    I try to install the system mit the script above on a raspberry pi. I have a fresh install of raspbian. The 3rd script stops with an error while downloading squid3 from the jessie repository. Version 3.3.8-1.2 is not found. What can I do?

  12. Imam says:

    hi sichent
    nice to meet you… im try you’re tutorial but i have error with version packge like this.. can u help me?
    http://prntscr.com/5lv8i3

    i do add repository jessy to my pi but error.. u can see on top

    plz help me.. i want using my pi to catch https 🙂

  13. Richard says:

    Hi, nice work. I’m a newbie to all this RPi stuff but still managing. Great post. Is there a fix for the missing “jessie” squid sources yet? I found this: http://snapshot.debian.org/package/squid3/3.3.8-1.2/
    Will it work? Are these the sources needed?

    • sichent says:

      Hello Richard, Frankly I do not know. I am waiting for the release of Debian / Raspbian 8 which will make the things much easier…

      • Richard says:

        Hi, thanks for the quick reply and pointers. I see, I think. I’ve modified your scripts and my sources.list for “jessie” throughout and then run the whole build through with squid3 (3.4.8-5). I missed out the download and build of libecap2 as well as that all seemed to be taken care of with the “apt-get build-dep squid3” having set for “jessie”. Applying the control.patch failed and looking at it I thought it might, and also not needed being a different version? Everything else seems to be going OK. I found an article on “apt pinning” http://www.raspberrypi.org/forums/viewtopic.php?f=66&t=47944 which helped too.

    • sichent says:

      The Pidora distrib may have Squid with HTTPS Bump support already built in as it is in CentOS / RedHat 7 for example…

  14. Richard says:

    I have my Squid 3.4.8 running nicely. My main issue was realising that my browser certificates issue when running the https proxy with “intercept” (transparently) was related to my ssl_bump setting in squid.conf. Setting it to “ssl_bump server-first” if you are filtering with an icap client or to “ssl_bump none” if not fixes that issue.

    • sichent says:

      Glad that it worked! May be you could share your installation scripts with all of us? I will add these to the instructions for upcoming release of qlproxy 4.0 (end of January)

  15. Benoit says:

    Hello Sichent,
    Now that the Raspberry PI 2 is released, will you provide the same features (https filtering) but on Rpi 2 ? I think user experience should be very nice with this new hardware :o)

    Regards,
    Benoît

  16. Pingback: Router and squid and https | Dictator Dad's Words of Wisdom

  17. Cameron says:

    Anyone looked into running on the banana pi ? Higher speced than the Raspberry’s and can get a 4x gbe ports version . or 1xgbe version. Would be amazing to have running inline/transparent as small footprint for home or small business.

    • sichent says:

      Hello Cameron, the deep inspection of HTML contents is soo CPU intensive that the RPI (or banana) deployments for now are more of an experiment 😦 Lightweight Intel Atom or Core seem to be more suited for the job).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s