Update (December 2015): The new and improved tutorial how to build HTTPS filtering Squid with Diladele Web Safety ICAP web filter is available from http://docs.diladele.com/administrator_guide_4_4/install/rpi/index.html
This article will tell you how to compile and configure Squid proxy server running on Raspberry PI, capable of filtering encrypted HTTPS connections using Diladele Web Safety ICAP content filtering server. Being able to look into HTTPS contents greatly increases your ability to control what is allowed and accepted within your network while keeping inappropriate contents away.
Why Should We Filter HTTPS?
HTTPS protocol was designed to provide secure means of communications between internet browser and remote web servers. In order to achieve this goal HTTPS protocol encrypts data passing through established connections so that it cannot be decrypted in reasonable amount of time thus preventing anyone from sniffing the contents interchanged over this connection. This protocol was primarily invented to enable safe and secure communication between the user and financial sites or government institutions over the insecure medium such as the Internet.
Recently more and more web sites started to use HTTPS encrypted communications to increase online privacy of users. Google who as first enabled HTTPS for all its searches by default probably initiated this trend. Although there are no doubts that HTTPS encryption is a good thing for safety on the wire we must take into account that it also creates several problems for controlled networks typically found at home or offices. The main problem here is the essence of the HTTPS protocol itself – no one except the browser and the web server is able to see and thus filter transferred data. This may not always be desired. Contents that are usually blocked suddenly become immediately accessible by anyone. As an example imagine a school network where minors can see questionable content by just mistyping a search term in Google. Moreover the law often forces administrators in educational institutions to block access to such content (e.g. CIPA for educational environments) and encrypted access to web sites makes it nearly impossible to fulfill such an obligation.
In order to overcome these limitations it is advised to setup HTTPS filtering of web contents with help of SSL bump feature of Squid proxy server and Diladele Web Safety web filter.
How It Works
In order to filter web requests user’s browser needs to be explicitly directed to use the proxy that is deployed in the same network. It is also possible to set the transparent proxy but we are not going to explain how this is done in this tutorial because steps involved are quite different from explicit proxy setup.
When a user tries to navigate to a web site, browser sends the request to proxy server, asking it to get the requested page on his behalf. The proxy establishes a new connection to the remote site and returns the response to browser. If normal HTTP is used then proxy is able to see the original contents of the response and filter it. In case of HTTPS the flow of data is a little different. Browser asks the proxy to establish a virtual tunnel between itself and remote server and then sends encrypted data through the proxy. Domain name to which a virtual tunnel is being established is usually known, so proxy is able to block this virtual tunnel when it finds out that domain name belongs to a prohibited category. Unfortunately this is not a complete solution as there are a lot of sites on the Internet which are general in nature (like Google or YouTube) but allow you to easily navigate to something undesired.
To improve the quality of web filtering and get access to contents in encrypted connections, browsers in the network may be setup to trust proxy to act on their behalf for establishing HTTPS connections, filtering them and passing the allowed data to clients while blocking everything that is not allowed. Although this assumption is too strict to be implemented in public networks, it is easily doable in controlled home, educational or corporate environments where administrators act as sole owners of network devices and may force any trusting rules. After established trust browser is able to ask proxy to connect to a remote site in a safe manner with HTTPS, proxy is able to decrypt the traffic, filter it, encrypt it again and pass it to browser. As browser trusts the proxy it continues working with filtered HTTPS without any errors or warnings.
Unfortunately the default Squid version included into Raspbian OS for Raspberry PI does not contain compile switches necessary for successful HTTPS filtering. We need to recompile Squid proxy, reinstall and reconfigure it with additional list of options.
Build Squid with SSL Bump and ICAP Client
Before compiling it is considered a good practice to bring the operation system to a most recent state. This can be done by running the following commands in the terminal.
$sudo apt-get update && sudo apt-get upgrade && sudo reboot
In order to build the Squid from source we need to install some build tools.
$ sudo apt-get install devscripts build-essential fakeroot libssl-dev pkg-config
Default version of Squid in Raspbian Wheezy is too old so we will need to do the following trick to get Squid version from Raspbian Jessy repositories. The idea is to temporary switch our repositories to Jessy, get Squid’s source code and switch back to Wheezy’s repositories.
# fetch the source for the package to re-build from the jessy repositories
$ echo "Copying sources lists..."
$ sudo cp /etc/apt/sources.list /etc/apt/sources.list.default
$ sudo cp /etc/apt/sources.list /etc/apt/sources.list.jessy
$ sudo cp /etc/apt/sources.list /etc/apt/sources.list.wheezy
$ echo "Adding source repositories..."
$ sudo echo "deb-src http://archive.raspbian.org/raspbian jessie main contrib non-free" >> /etc/apt/sources.list.jessy
$ sudo echo "deb-src http://archive.raspbian.org/raspbian wheezy main contrib non-free" >> /etc/apt/sources.list.wheezy
$ echo "Getting Squid from Jessy..."
$ sudo cp /etc/apt/sources.list.jessy /etc/apt/sources.list
$ sudo apt-get update
$ sudo apt-get source squid3=3.3.8-1.2 --download-only
$ sudo apt-get source libecap2=0.2.0-1 --download-only
$ echo "Getting Squid dependencies from Wheezy..."
$ sudo cp /etc/apt/sources.list.wheezy /etc/apt/sources.list
$ sudo apt-get update
$ sudo apt-get build-dep squid3
$ echo "Reverting default repository..."
$ sudo mv /etc/apt/sources.list.default /etc/apt/sources.list
$ sudo apt-get update
Now we have sources of Squid 3.3.8 in current folder. Running the following command gets all necessary dependencies for Squid from Wheezy repos and unpacks Squid source package together with all system integration scripts and patches provided by Debian/Raspbian developers.
$ echo "Running dpkg-source..."
$ sudo dpkg-source -x squid3_3.3.8-1.2.dsc
$ sudo dpkg-source -x libecap_0.2.0-1.dsc
Before we build Squid we need to build and install dependency library which is not included into Wheezy.
$ echo "Building libecap2..."
$ pushd libecap-0.2.0
$ dpkg-buildpackage -rfakeroot -b
$ echo "Installing libecap2..."
$ sudo dpkg --install *.deb
Sources are unpacked into squid3-3.3.8 folder. We need to modify configure options in debian/rules and debian/control to include compiler switches (–enable-ssl and –enable-ssl-crtd) necessary for HTTPS filtering.
$ pushd squid3-3.3.8
$ echo "Patching Squid source..."
$ sudo patch debian/rules < "../rules.patch"
$ sudo patch debian/control < "../control.patch"
$ sudo patch src/ssl/gadgets.cc < "../gadgets.cc.patch"
$ echo "Building Squid..."
$ sudo dpkg-buildpackage -rfakeroot -b
As the rules.patch and control.patch are rather big they are not included into the article directly. All patches are part of the corresponding download archive. Please note, that one file in source code of Squid Proxy needs to be adjusted too (src/ssl/gadgets.cc). This change is needed to prevent Firefox error sec_error_inadequate_key_usage that usually occurs when doing HTTPS filtering with latest versions of Firefox browser. If you use only Google Chrome, Microsoft Internet Explorer or Apple Safari this step is not required.
Please take into account that the build process is rather slow (it took approximately 10 hours on stock Raspberry PI) while all required *.DEB packages need to be built on Raspberry PI and cannot be cross compiled.
Install Diladele Web Safety
SSL Bumping feature alone is not enough to block questionable web content. We also need the filtering server that could be paired with Squid. We will use Diladele Web Safety (DDWS) formerly known as QuintoLabs Content Security (qlproxy) for the filtering and blocking part. It is an ICAP daemon capable of integrating existing Squid proxy and providing rich content filtering functionality out of the box. It may be used to block illegal or potentially malicious file downloads, remove annoying advertisements, prevent access to various categories of the web sites and block resources with explicit content.
We will use version 3.4.0 of qlproxy. It was designed specifically with HTTPS filtering in mind and contains rich web administrator console to perform routine tasks right from the browser.
By default, DDWS comes with four polices preinstalled. Strict policy contains web filter settings put on maximum level and is supposed to protect minors and K12 students from inappropriate contents on the Internet. Relaxed policy blocks only excessive advertisements and was supposed to be used by network administrators, teachers and all those who do not need filtered access to web but would like to evade most ads. Third policy is tailored to white list only browsing and the last group contains less restrictive web filtering settings suitable for normal web browsing without explicitly adult contents shown.
In order to install Diladele Web Safety for Squid Proxy, download package for Raspberry PI from Diladele B.V. web site at http://www.quintolabs.com using browser or just run the following command in terminal.
Administration console of Diladele Web Safety is built using Python Django framework and is usually managed by Apache web server. To install packages required for correct functioning of web UI run the following commands in the terminal.
$sudo apt-get -y install python-setuptools
$sudo easy_install django==1.5
$sudo apt-get –y install apache2 libapache2-mod-wsgi
Install the DEB package and perform integration with Apache by running the following commands.
$ sudo dpkg --install qlproxy-18.104.22.16807_armhf.deb
$ sudo a2dissite 000-default
$ sudo a2ensite qlproxy.conf
$ sudo service apache2 restart
Configure Squid for ICAP Filtering and HTTP Bumping
The Squid packages we have compiled previously need to be installed on the system. To perform installation run the following commands.
$ sudo apt-get install ssl-cert
$ sudo apt-get install squid-langpack
$ sudo dpkg --install squid3-common_3.3.8-1.2_all.deb
$ sudo dpkg --install squid3_3.3.8-1.2_armhf.deb
$ sudo dpkg --install squidclient_3.3.8-1.2_armhf.deb
In order to recreate original SSL certificates of the remote web sites during HTTPS filtering Squid uses a separate process named ssl_crtd that needs to be configured like this.
$ sudo ln -s /usr/lib/squid3/ssl_crtd /bin/ssl_crtd
$ sudo /bin/ssl_crtd -c -s /var/spool/squid3_ssldb
$ sudo chown -R proxy:proxy /var/spool/squid3_ssldb
Finally, modify Squid configuration file in /etc/squid3/squid.conf to integrate it with Diladele Web Safety as ICAP server. Due to the size of the patch file its text not included into the article directly but is part of the download archive.
$ sudo cp /etc/squid3/squid.conf /etc/squid3/squid.conf.default
$ sudo patch /etc/squid3/squid.conf < "../squid.conf.patch"
$ sudo /usr/sbin/squid3 -k parse
From now on Squid is capable of HTTPS filtering and we may continue filtering adjustments from Web UI of Diladele Web Safety.
Navigate to http://YOUR_PROXY_IP_ADDRESS/ and login with default name
root and password
P@ssw0rd. Select Settings / HTTPS Filtering / Filtering Mode. Diladele Web Safety may either filter specific HTTPS sites or all of them with exclusions. Total filtering is more tailored to providing very safe network environments.
Select the desired mode, click Save Settings, add target domains or exclusions as you like and then restart ICAP server by clicking on the green button in the top right corner as indicated on the following screenshots.
Run the following command in terminal on the proxy.
$ sudo service squid3 restart
Navigate to google.com and see that HTTPS filtering is indeed active. The following warning shows that Squid was able to bump the SSL connection, filtered it and encrypted in again using Diladele Web Safety’s generated certificate.
In order to get rid of these warnings, we must install the myca.der certificate file from into the browser and mark it as trusted. Again navigate to http://YOUR_PROXY_IP_ADDRESS. Select Settings / HTTPS Filtering / Certificates and select the one that matches your operating system or device. Instructions on how to install the certificate in each operating system or device is slightly different, the following screens show how to install the DER file in Microsoft Internet Explorer. For other devices please take a look at Online Documentation of Diladele Web Safety.
Reopen your browser, navigate to Google and make sure the certificate warning is away. If you click on the lock icon in the internet address box then it clearly indicates the google.com was signed by proxy’s certificate and not by original certificate by google.
If you try to search Google with some adult only terms (e.g. NSFW) Diladele Web Safety blocks the access to explicit contents showing its denied page.
Please be sure to change the default certificates that come with installation package of Diladele Web Safety to something more unique for your network. For instructions on how to regenerate your own certificates for this purpose consult Online Documentation of Diladele Web Safety.
Now we have HTTPS web filtering up and running and our network environment become a little safer for those who need protection at most. Next steps would be direct all clients browsers to use Squid proxy, regenerate the default proxy certificates, setup authentication and authorization to get user specific reports in Diladele Web Safety, integrate it with e.g. Active Directory using Squid’s support for Kerberos authentication and optionally setup transparent HTTPS filtering. It is also advisable to setup the caching DNS server on Squid proxy to further increase speed of connections.