December 23, 2015 – the http://docs.diladele.com/tutorials/filtering_https_traffic_squid_pfsense/index.html contains a new tutorial for HTTPS filtering Squid + Web Safety ICAP Server 4.3 and pfSense 2.2.
If you are updating from Diladele Web Safety 3.X, be sure to follow the upgrade instructions. Please *NOTE*: after uninstall of Diladele Web Safety 3.2/3.3, remove qlproxy user and group, reboot your pfSense box and add qlproxy user and group again!!!. Please also note, all scripts mentioned in this article can be downloaded here.
This article will tell you how to install and configure Squid proxy capable of filtering encrypted HTTPS connections using Diladele Web Safety ICAP content filtering server running on pfSense Firewall 2.1.5 (amd64). Being able to look into HTTPS contents greatly increases your ability to control what is allowed and accepted within your network while keeping inappropriate contents away.
Why We Need to Filter HTTPS
HTTPS protocol was designed to provide secure means of communications between internet browser and remote web servers. In order to achieve this goal HTTPS protocol encrypts data passing through established connections so that it cannot be decrypted in reasonable amount of time thus preventing anyone from sniffing the contents interchanged over this connection. This protocol was primarily invented to enable safe and secure communication between the user and financial sites or government institutions over the insecure medium such as the Internet.
Recently more and more web sites started to use HTTPS encrypted communications to increase online privacy of users. Google who as first enabled HTTPS for all its searches by default probably initiated this trend. Although there are no doubts that HTTPS encryption is a good thing for safety on the wire we must take into account that it also creates several problems for controlled networks typically found at home or offices. The main problem here is the essence of the HTTPS protocol itself – no one except the browser and the web server is able to see and thus filter transferred data. This may not always be desired. Contents that are usually blocked suddenly become immediately accessible by anyone. As an example imagine a school network where minors can see questionable content by just mistyping a search term in Google. Moreover the law often forces administrators in educational institutions to block access to such content (e.g. CIPA for educational environments) and encrypted access to web sites makes it nearly impossible to fulfill such an obligation.
In order to overcome these limitations it is advised to setup HTTPS filtering of web contents with help of SSL bump feature of Squid proxy server and Diladele Web Safety web filter.
How It Works
In order to filter web requests user’s browser needs to be explicitly directed to use the proxy that is deployed in the same network. It is also possible to set the transparent proxy but we are not going to explain how this is done in this tutorial because steps involved may be quite different from explicit proxy setup.
When a user tries to navigate to a web site, browser sends the request to proxy server, asking it to get the requested page on his behalf. The proxy establishes a new connection to the remote site and returns the response to browser. If normal HTTP is used then proxy is able to see the original contents of the response and filter it. In case of HTTPS the flow of data is a little different. Browser asks the proxy to establish a virtual tunnel between itself and remote server and then sends encrypted data through the proxy. Domain name to which a virtual tunnel is being established is usually known, so proxy is able to block this virtual tunnel when it finds out that domain name belongs to a prohibited category. Unfortunately this is not a complete solution as there are a lot of sites on the Internet which are general in nature (like Google or YouTube) but allow you to easily navigate to something undesired.
To improve the quality of web filtering and get access to contents in encrypted connections, browsers in the network may be setup to trust proxy to act on their behalf for establishing HTTPS connections, filtering them and passing the allowed data to clients while blocking everything that is not allowed. Although this assumption is too strict to be implemented in public networks, it is easily doable in controlled home, educational or corporate environments where administrators act as sole owners of network devices and may force any trusting rules. After established trust browser is able to ask proxy to connect to a remote site in a safe manner with HTTPS, proxy is able to decrypt the traffic, filter it, encrypt it again and pass it to browser. As browser trusts the proxy it continues working with filtered HTTS without any errors or warnings.
I assume you have already installed pfSense 2.1.4 amd64 with two NIC (LAN and WAN). Lan IP address is 192.168.1.1. I also assume you have already done the initial login to the Web UI of pfSense and completed the initial setup wizard and successfully rebooted the pfSense box at least once.
Step 1 – Install Squid built with SSL decryption support.
Login to Web UI and select System -> Packages -> Available Packages. Find and install package squid3-dev.
Wait until squid3-dev package is installed and after installation, reboot your pfSense box.
Login to Web UI, select Status -> Services. The green dot on the right side will indicate Squid service is up and running.
Now adjust your browser configuration to point to pfSense box and try browsing the web.
The response from Squid indicates that we do not have access to it.
To get the access we will add our subnet to the allowed ACL. Select Services -> Proxy server and click the ACL tab. Add 192.168.1.0/24 to the “Allowed subnets” field. Scroll all the way down and click Save. Reload Squid service in Status -> Services. Try browsing web again and see that Squid is now working.
Step 2 – Install Diladele Web Safety for Squid Proxy
SSL Bumping feature alone is not enough to block questionable web content. We also need the filtering server that could be paired with Squid. We will use Diladele Web Safety (called qlproxy) for the filtering and blocking part. It is an ICAP daemon capable of integrating existing Squid proxy and providing rich content filtering functionality out of the box. It may be used to block illegal or potentially malicious file downloads, remove annoying advertisements, prevent access to various categories of the web sites and block resources with explicit content.
We will use the stable release version 3.3 of qlproxy. It was designed specifically with HTTPS filtering in mind and contains rich web administrator console to perform routine tasks right from the browser.
By default, qlproxy comes with four polices preinstalled. Strict policy contains web filter settings put on maximum level and is supposed to protect minors and K12 students from inappropriate contents on the Internet. Relaxed policy blocks only excessive advertisements and was supposed to be used by network administrators, teachers and all those who do not need filtered access to web but would like to evade most ads. Third policy is tailored to white list only browsing and the last group contains less restrictive web filtering settings suitable for normal web browsing without explicitly adult contents shown.
Diladele Web Safety uses qlproxy user and group to run. Normally it creates those upon installation but for some reason they are not saved during reboots so we must create required users manually. Go to System -> User Manager select Groups Tab and add a new group qlproxy.
Click Save and then select the Users tab to add a new user qlproxy. Do not forget to make in a member of qlproxy group. Enter some arbitrary password.
Again click Save.
In order to install all required libraries and programs we will use 3 scripts. First script installs Python and all needed libraries for the Web UI of Diladele Web Safety, second installs Apache web server that runs the Web UI and third installs Diladele Web Safety itself.
The first script is named
01_django.sh and looks like the following. It should be run from the pfSense command like (e.g. using Putty) as
sh 01_django.sh. Please watch out for possible errors!
# setup some configuration variables ARCH=`uname -m` # bail out on any error set -e # prepare the environment to get some packages from FreeBSD 8 PACKAGESITE=http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/$ARCH/packages-8.3-release/Latest/ export PACKAGESITE # add python (not sure about py27-ldap) pkg_add -r python27 py27-sqlite3 py27-pip # the following command does not work as cc compiler is not present in pfSense, so no LDAP browsing from Web UI will work! # someone knows how to overcome this? may be install openldap libraries from FreeBSD somehow??? #pkg_add -r py27-ldap2 # add django 1.5 /usr/local/bin/pip install Django==1.5 # and report echo "01_django.sh script has finished successfully, please run script 02_apache.sh!"
Now we need to install Apache server, copy the following to
02_apache.sh script and run it in the console of pfSense box as
sh 02_apache.sh too. This script install all prerequisites for Apache web server and configures it to serve Diladele Web Safety’s Web UI on port 8080 (as standard port 80 is already taken by pfSense Web UI).
# setup some configuration variables ARCH=`uname -m` # bail out on any error set -e # prepare the environment to get some packages from FreeBSD 8 PACKAGESITE=http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/$ARCH/packages-8.3-release/Latest/ export PACKAGESITE # add apache pkg_add -r apache22 ap22-mod_wsgi # in order to correctly start up apache at boot time init script needs to be renamed cp /usr/local/etc/rc.d/apache22 /usr/local/etc/rc.d/apache22.sh # make apache autostart sed -i '' 's/apache22_enable=\"NO\"/apache22_enable=\"YES\"/' /usr/local/etc/rc.d/apache22.sh # make apache listen on 8080 port sed -i '' 's/Listen 80/Listen 8080/' /usr/local/etc/apache22/httpd.conf # and include the virtual hosts sed -i '' 's/\#Include etc\/apache22\/extra\/httpd-vhosts.conf/Include etc\/apache22\/extra\/httpd-vhosts.conf/' /usr/local/etc/apache22/httpd.conf # and report echo "02_apache.sh script has finished successfully, please run script 03_diladele.sh!"
Finally run the
03_diladele.sh script by typing
sh 03_diladele.sh in pfSense console. It will download latest stable build of Diladele Web Safety and adjusts Apache configuration for the Web UI.
# setup some configuration variables ARCH=`uname -m` DDWS_VERSION=3.4.0 DDWS_BUILD=9307 # see if qlproxy group exists echo "Searching for group qlproxy..." getent group qlproxy >/dev/null if [ $? -ne 0 ] ; then echo "Group qlproxy is not found, please add it through pfSense Web UI." exit 1 else echo "Group qlproxy already exists." fi # see if qlproxy user exists echo "Searching for user qlproxy..." getent passwd qlproxy >/dev/null if [ $? -ne 0 ] ; then echo "User qlproxy is not found, please add it through pfSense Web UI." exit 2 else echo "User qlproxy already exists." fi # how to check user qlproxy is in qlproxy group??? # get latest version of diladele icap server fetch http://updates.diladele.com/qlproxy/binaries/$DDWS_VERSION.$DDWS_BUILD/$ARCH/release/freebsd8/qlproxy-$DDWS_VERSION-$ARCH.tbz # and install it pkg_add qlproxy-$DDWS_VERSION-$ARCH.tbz # now copy default apache virtual hosts file if [ -f /usr/local/etc/apache22/extra/httpd-vhosts.conf.default ]; then echo "Not saving default vhosts file" else cp /usr/local/etc/apache22/extra/httpd-vhosts.conf /usr/local/etc/apache22/extra/httpd-vhosts.conf.default echo "default vhosts file is backed up" fi # virtual hosts file needs to contaion only diladele virtual host echo "NameVirtualHost *:8080" > /usr/local/etc/apache22/extra/httpd-vhosts.conf echo "Include /usr/local/etc/apache22/extra/qlproxy_virtual_host" >> /usr/local/etc/apache22/extra/httpd-vhosts.conf # restart apache /usr/local/etc/rc.d/apache22.sh restart # and report echo "03_diladele.sh script has finished successfully!" echo "Please, reboot your pfSense box and login to Diladele Web Safety's Web UI at http://18.104.22.168:8080!"
Now reboot your pfSense box and login to http://22.214.171.124:8080 using root and P@ssw0rd credentials to finally see the Web UI of Diladele Web Safety.
Step 3 – Integrate Squid Proxy and Diladele Web Safety
To integrate qlproxy and squid, go to Services / Proxy Server, scroll all the way down and in Custom ACLs (Before AUTH) field type:
icap_enable on icap_preview_enable on icap_preview_size 4096 icap_persistent_connections on icap_send_client_ip on icap_send_client_username on icap_client_username_header X-Client-Username icap_service qlproxy1 reqmod_precache bypass=0 icap://127.0.0.1:1344/reqmod icap_service qlproxy2 respmod_precache bypass=0 icap://127.0.0.1:1344/respmod acl qlproxy_icap_edomains dstdomain "/opt/qlproxy/etc/squid/icap_exclusions_domains.conf" acl qlproxy_icap_etypes rep_mime_type "/opt/qlproxy/etc/squid/icap_exclusions_contenttypes.conf" adaptation_access qlproxy1 deny qlproxy_icap_edomains adaptation_access qlproxy2 deny qlproxy_icap_edomains adaptation_access qlproxy2 deny qlproxy_icap_etypes adaptation_access qlproxy1 allow all adaptation_access qlproxy2 allow all
Click Save and restart Squid proxy. Try to browse to some adult site and see that HTTP filtering works correctly.
Now we need to enable SSL Filtering to make Diladele Web Safety filter the HTTPS requests too. Create a certification authority in System / Cert Manager.
This certificate will be used to bump the HTTPS connections. Go to Services / Proxy Server, scroll to SSL Man-in-the-Middle filtering and fill the fields as indicated on the following screenshot. Note we are not filling the port settings as we are not doing transparent HTTPS filtering for now. Save and restart Squid service.
If you navigate to google.com you may clearly see the HTTPS connection was NOT bumped. The reason for this is a missing directive “ssl_bump server-first all“ in Squid. Add it before the “icap_enable on” directive added earlier. Save and restart Squid. Now your SSL connection to Google will be bumped.
To get rid of this warning, we need to install the root CA certificate from pfSense box as trusted in your browser(s). Download the certificates from pfSense using WinSCP for example and import it into trusted certificates as indicated on the following screenshot (instructions for Google Chrome and Internet Explorer may be different).
After importing of certificate, reopen your browser, navigate to Google and make sure the certificate warning is away. If you click on the lock icon in the internet address box then it clearly indicates the google.com was signed by proxy’s certificate and not by original certificate by google.
If you try to search Google with some adult only terms (e.g. NSFW) Diladele Web Safety blocks the access to explicit contents showing its denied page.
Setup Automatic Updates and Reporting
By default periodic package that runs automatic updates of definition files and report generation scripts is not installed on pfSense. We will use cron functionality to run the scripts manually. Open System / Packages in pfSense Web UI and install the cron package. After that open Services / Cron and add two cron entries:
- Run the command to update definition files every 59 minutes as root user –
- Run the command to generate reports of browsing activities every day at 01:00 as root user –
Now we have HTTPS web filtering up and running and our network environment become a little safer for those who need protection at most. Next steps would be direct all client browsers to use Squid proxy, correctly setup authentication and authorization to get user specific reports in Diladele Web Safety and optionally setup transparent HTTPS filtering. It is also advisable to enable caching DNS server on pfSense proxy to further increase speed of connections.
If you happen to be gladly using this product be sure to purchase the home, educational or business license that will make the work required to have it up and running in native pfSense format possible.