Filtering HTTPS Traffic with Squid on pfSense 2.1.5

December 23, 2015 – the http://docs.diladele.com/tutorials/filtering_https_traffic_squid_pfsense/index.html contains a new tutorial for HTTPS filtering Squid + Web Safety ICAP Server 4.3 and pfSense 2.2.

If you are updating from Diladele Web Safety 3.X, be sure to follow the upgrade instructions. Please *NOTE*: after uninstall of Diladele Web Safety 3.2/3.3, remove qlproxy user and group, reboot your pfSense box and add qlproxy user and group again!!!. Please also note, all scripts mentioned in this article can be downloaded here.

This article will tell you how to install and configure Squid proxy capable of filtering encrypted HTTPS connections using Diladele Web Safety ICAP content filtering server running on pfSense Firewall 2.1.5 (amd64). Being able to look into HTTPS contents greatly increases your ability to control what is allowed and accepted within your network while keeping inappropriate contents away.

Why We Need to Filter HTTPS

HTTPS protocol was designed to provide secure means of communications between internet browser and remote web servers. In order to achieve this goal HTTPS protocol encrypts data passing through established connections so that it cannot be decrypted in reasonable amount of time thus preventing anyone from sniffing the contents interchanged over this connection. This protocol was primarily invented to enable safe and secure communication between the user and financial sites or government institutions over the insecure medium such as the Internet.

Recently more and more web sites started to use HTTPS encrypted communications to increase online privacy of users. Google who as first enabled HTTPS for all its searches by default probably initiated this trend. Although there are no doubts that HTTPS encryption is a good thing for safety on the wire we must take into account that it also creates several problems for controlled networks typically found at home or offices. The main problem here is the essence of the HTTPS protocol itself – no one except the browser and the web server is able to see and thus filter transferred data. This may not always be desired. Contents that are usually blocked suddenly become immediately accessible by anyone. As an example imagine a school network where minors can see questionable content by just mistyping a search term in Google. Moreover the law often forces administrators in educational institutions to block access to such content (e.g. CIPA for educational environments) and encrypted access to web sites makes it nearly impossible to fulfill such an obligation.

In order to overcome these limitations it is advised to setup HTTPS filtering of web contents with help of SSL bump feature of Squid proxy server and Diladele Web Safety web filter.

How It Works

In order to filter web requests user’s browser needs to be explicitly directed to use the proxy that is deployed in the same network. It is also possible to set the transparent proxy but we are not going to explain how this is done in this tutorial because steps involved may be quite different from explicit proxy setup.

When a user tries to navigate to a web site, browser sends the request to proxy server, asking it to get the requested page on his behalf. The proxy establishes a new connection to the remote site and returns the response to browser. If normal HTTP is used then proxy is able to see the original contents of the response and filter it. In case of HTTPS the flow of data is a little different. Browser asks the proxy to establish a virtual tunnel between itself and remote server and then sends encrypted data through the proxy. Domain name to which a virtual tunnel is being established is usually known, so proxy is able to block this virtual tunnel when it finds out that domain name belongs to a prohibited category. Unfortunately this is not a complete solution as there are a lot of sites on the Internet which are general in nature (like Google or YouTube) but allow you to easily navigate to something undesired.

To improve the quality of web filtering and get access to contents in encrypted connections, browsers in the network may be setup to trust proxy to act on their behalf for establishing HTTPS connections, filtering them and passing the allowed data to clients while blocking everything that is not allowed. Although this assumption is too strict to be implemented in public networks, it is easily doable in controlled home, educational or corporate environments where administrators act as sole owners of network devices and may force any trusting rules. After established trust browser is able to ask proxy to connect to a remote site in a safe manner with HTTPS, proxy is able to decrypt the traffic, filter it, encrypt it again and pass it to browser. As browser trusts the proxy it continues working with filtered HTTS without any errors or warnings.

Assumptions

I assume you have already installed pfSense 2.1.4 amd64 with two NIC (LAN and WAN). Lan IP address is 192.168.1.1. I also assume you have already done the initial login to the Web UI of pfSense and completed the initial setup wizard and successfully rebooted the pfSense box at least once.

image001

Step 1 – Install Squid built with SSL decryption support.

Login to Web UI and select System -> Packages -> Available Packages. Find and install package squid3-dev.

image003

Wait until squid3-dev package is installed and after installation, reboot your pfSense box.

image005

Login to Web UI, select Status -> Services. The green dot on the right side will indicate Squid service is up and running.

image007

Now adjust your browser configuration to point to pfSense box and try browsing the web.

image009

The response from Squid indicates that we do not have access to it.

image011

To get the access we will add our subnet to the allowed ACL. Select Services -> Proxy server and click the ACL tab. Add 192.168.1.0/24 to the “Allowed subnets” field. Scroll all the way down and click Save. Reload Squid service in Status -> Services. Try browsing web again and see that Squid is now working.

image013

image015

Step 2 – Install Diladele Web Safety for Squid Proxy

SSL Bumping feature alone is not enough to block questionable web content. We also need the filtering server that could be paired with Squid. We will use Diladele Web Safety (called qlproxy) for the filtering and blocking part. It is an ICAP daemon capable of integrating existing Squid proxy and providing rich content filtering functionality out of the box. It may be used to block illegal or potentially malicious file downloads, remove annoying advertisements, prevent access to various categories of the web sites and block resources with explicit content.

We will use the stable release version 3.3 of qlproxy. It was designed specifically with HTTPS filtering in mind and contains rich web administrator console to perform routine tasks right from the browser.

By default, qlproxy comes with four polices preinstalled. Strict policy contains web filter settings put on maximum level and is supposed to protect minors and K12 students from inappropriate contents on the Internet. Relaxed policy blocks only excessive advertisements and was supposed to be used by network administrators, teachers and all those who do not need filtered access to web but would like to evade most ads. Third policy is tailored to white list only browsing and the last group contains less restrictive web filtering settings suitable for normal web browsing without explicitly adult contents shown.

Diladele Web Safety uses qlproxy user and group to run. Normally it creates those upon installation but for some reason they are not saved during reboots so we must create required users manually. Go to System -> User Manager select Groups Tab and add a new group qlproxy.

image017

image019

Click Save and then select the Users tab to add a new user qlproxy. Do not forget to make in a member of qlproxy group. Enter some arbitrary password.

image021

Again click Save.

image023

In order to install all required libraries and programs we will use 3 scripts. First script installs Python and all needed libraries for the Web UI of Diladele Web Safety, second installs Apache web server that runs the Web UI and third installs Diladele Web Safety itself.

The first script is named 01_django.sh and looks like the following. It should be run from the pfSense command like (e.g. using Putty) as sh 01_django.sh. Please watch out for possible errors!

# setup some configuration variables
ARCH=`uname -m`

# bail out on any error
set -e

# prepare the environment to get some packages from FreeBSD 8
PACKAGESITE=http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/$ARCH/packages-8.3-release/Latest/
export PACKAGESITE

# add python (not sure about py27-ldap)
pkg_add -r python27 py27-sqlite3 py27-pip

# the following command does not work as cc compiler is not present in pfSense, so no LDAP browsing from Web UI will work!
# someone knows how to overcome this? may be install openldap libraries from FreeBSD somehow???
#pkg_add -r py27-ldap2

# add django 1.5
/usr/local/bin/pip install Django==1.5

# and report
echo "01_django.sh script has finished successfully, please run script 02_apache.sh!"

Now we need to install Apache server, copy the following to 02_apache.sh script and run it in the console of pfSense box as sh 02_apache.sh too. This script install all prerequisites for Apache web server and configures it to serve Diladele Web Safety’s Web UI on port 8080 (as standard port 80 is already taken by pfSense Web UI).

# setup some configuration variables
ARCH=`uname -m`

# bail out on any error
set -e

# prepare the environment to get some packages from FreeBSD 8
PACKAGESITE=http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/$ARCH/packages-8.3-release/Latest/
export PACKAGESITE

# add apache
pkg_add -r apache22 ap22-mod_wsgi

# in order to correctly start up apache at boot time init script needs to be renamed
cp /usr/local/etc/rc.d/apache22 /usr/local/etc/rc.d/apache22.sh

# make apache autostart
sed -i '' 's/apache22_enable=\"NO\"/apache22_enable=\"YES\"/' /usr/local/etc/rc.d/apache22.sh

# make apache listen on 8080 port
sed -i '' 's/Listen 80/Listen 8080/' /usr/local/etc/apache22/httpd.conf

# and include the virtual hosts
sed -i '' 's/\#Include etc\/apache22\/extra\/httpd-vhosts.conf/Include etc\/apache22\/extra\/httpd-vhosts.conf/' /usr/local/etc/apache22/httpd.conf

# and report
echo "02_apache.sh script has finished successfully, please run script 03_diladele.sh!"

Finally run the 03_diladele.sh script by typing sh 03_diladele.sh in pfSense console. It will download latest stable build of Diladele Web Safety and adjusts Apache configuration for the Web UI.

# setup some configuration variables
ARCH=`uname -m`
DDWS_VERSION=3.4.0
DDWS_BUILD=9307

# see if qlproxy group exists
echo "Searching for group qlproxy..."
getent group qlproxy >/dev/null
if [ $? -ne 0 ] ; then
    echo "Group qlproxy is not found, please add it through pfSense Web UI."
    exit 1
else
    echo "Group qlproxy already exists."
fi

# see if qlproxy user exists
echo "Searching for user qlproxy..."
getent passwd qlproxy >/dev/null
if [ $? -ne 0 ] ; then
    echo "User qlproxy is not found, please add it through pfSense Web UI."
    exit 2
else
    echo "User qlproxy already exists."
fi

# how to check user qlproxy is in qlproxy group???

# get latest version of diladele icap server
fetch http://updates.diladele.com/qlproxy/binaries/$DDWS_VERSION.$DDWS_BUILD/$ARCH/release/freebsd8/qlproxy-$DDWS_VERSION-$ARCH.tbz

# and install it
pkg_add qlproxy-$DDWS_VERSION-$ARCH.tbz

# now copy default apache virtual hosts file
if [ -f /usr/local/etc/apache22/extra/httpd-vhosts.conf.default ]; then
    echo "Not saving default vhosts file"
else
    cp /usr/local/etc/apache22/extra/httpd-vhosts.conf /usr/local/etc/apache22/extra/httpd-vhosts.conf.default
    echo "default vhosts file is backed up"
fi

# virtual hosts file needs to contaion only diladele virtual host
echo "NameVirtualHost *:8080" > /usr/local/etc/apache22/extra/httpd-vhosts.conf
echo "Include /usr/local/etc/apache22/extra/qlproxy_virtual_host" >> /usr/local/etc/apache22/extra/httpd-vhosts.conf

# restart apache
/usr/local/etc/rc.d/apache22.sh restart 

# and report
echo "03_diladele.sh script has finished successfully!"
echo "Please, reboot your pfSense box and login to Diladele Web Safety's Web UI at http://192.169.1.1:8080!"

Now reboot your pfSense box and login to http://192.169.1.1:8080 using root and P@ssw0rd credentials to finally see the Web UI of Diladele Web Safety.

image029

Step 3 – Integrate Squid Proxy and Diladele Web Safety

To integrate qlproxy and squid, go to Services / Proxy Server, scroll all the way down and in Custom ACLs (Before AUTH) field type:

icap_enable on
icap_preview_enable on
icap_preview_size 4096
icap_persistent_connections on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_header X-Client-Username
icap_service qlproxy1 reqmod_precache bypass=0 icap://127.0.0.1:1344/reqmod
icap_service qlproxy2 respmod_precache bypass=0 icap://127.0.0.1:1344/respmod

acl qlproxy_icap_edomains dstdomain "/opt/qlproxy/etc/squid/icap_exclusions_domains.conf"
acl qlproxy_icap_etypes rep_mime_type "/opt/qlproxy/etc/squid/icap_exclusions_contenttypes.conf"

adaptation_access qlproxy1 deny qlproxy_icap_edomains
adaptation_access qlproxy2 deny qlproxy_icap_edomains
adaptation_access qlproxy2 deny qlproxy_icap_etypes
adaptation_access qlproxy1 allow all
adaptation_access qlproxy2 allow all

Click Save and restart Squid proxy. Try to browse to some adult site and see that HTTP filtering works correctly.

image031

image033

image035

Now we need to enable SSL Filtering to make Diladele Web Safety filter the HTTPS requests too. Create a certification authority in System / Cert Manager.

image037

Click Save.

image039

This certificate will be used to bump the HTTPS connections. Go to Services / Proxy Server, scroll to SSL Man-in-the-Middle filtering and fill the fields as indicated on the following screenshot. Note we are not filling the port settings as we are not doing transparent HTTPS filtering for now. Save and restart Squid service.

image041

If you navigate to google.com you may clearly see the HTTPS connection was NOT bumped. The reason for this is a missing directive “ssl_bump server-first all“ in Squid. Add it before the “icap_enable on” directive added earlier. Save and restart Squid. Now your SSL connection to Google will be bumped.

image043

image045

To get rid of this warning, we need to install the root CA certificate from pfSense box as trusted in your browser(s). Download the certificates from pfSense using WinSCP for example and import it into trusted certificates as indicated on the following screenshot (instructions for Google Chrome and Internet Explorer may be different).

image047

image049

image051

image053

After importing of certificate, reopen your browser, navigate to Google and make sure the certificate warning is away. If you click on the lock icon in the internet address box then it clearly indicates the google.com was signed by proxy’s certificate and not by original certificate by google.

image055

If you try to search Google with some adult only terms (e.g. NSFW) Diladele Web Safety blocks the access to explicit contents showing its denied page.

image057

Setup Automatic Updates and Reporting

By default periodic package that runs automatic updates of definition files and report generation scripts is not installed on pfSense. We will use cron functionality to run the scripts manually. Open System / Packages in pfSense Web UI and install the cron package. After that open Services / Cron and add two cron entries:

  1. Run the command to update definition files every 59 minutes as root user – /bin/sh /usr/local/etc/periodic/daily/511.qlproxy_update
  2. Run the command to generate reports of browsing activities every day at 01:00 as root user – /bin/sh /usr/local/etc/periodic/daily/510.qlproxy_report

image061

Resume

Now we have HTTPS web filtering up and running and our network environment become a little safer for those who need protection at most. Next steps would be direct all client browsers to use Squid proxy, correctly setup authentication and authorization to get user specific reports in Diladele Web Safety and optionally setup transparent HTTPS filtering. It is also advisable to enable caching DNS server on pfSense proxy to further increase speed of connections.

If you happen to be gladly using this product be sure to purchase the home, educational or business license that will make the work required to have it up and running in native pfSense format possible.

Links

  1. Diladele B.V. web site at http://www.quintolabs.com
  2. Online Documentation of Diladele Web Safety
  3. Squid Proxy Wiki on SSL Bumping
  4. pfSense Web Site
  5. All scripts mentioned in this article

About sichent

sichent
This entry was posted in Diladele, FreeBSD, ICAP, pfSense, squid, web filter and tagged , , . Bookmark the permalink.

210 Responses to Filtering HTTPS Traffic with Squid on pfSense 2.1.5

  1. Christian says:

    Thanks a lot for the tutorial.

    Just configured it using local auth and under transparent mode. Can we use it under ldap auth mode ? I tried to enable ldap and squid keeps loading saved configuration.

    • sichent says:

      Hi Cristian, I have not tried the LDAP configuration. The qlproxy version 3.2 will support assigning members of Active Directory (LDAP) groups as members of policies. When this is ready for public I will try to implement it using pfSense + Squid.

  2. Thomas says:

    Hey,
    i download the file like you wrote here:
    fetch -o /usr/local/lib/libheimntlm.so.10 http://e-sac.siteseguro.ws/pfsense/8/amd64/All/ldd/libheimntlm.so.10
    and still get an error after “/usr/local/sbin/squid -v”
    Feedback is:
    /libexec/ld-elf.so.1: /usr/local/lib/libheimntlm.so.10: unsupported file layout

    Do you have an idea hoe to fix this? I really love to work with squid3 and http/https

    • sichent says:

      Hi Thomas, please make sure your pfSense architecture matches the one you are downloading binaries for – e.g. amd64 == amd64.

      • Thomas says:

        Hi sichent,
        my architecture is i386. Do you know a place i can download the files i need? I´m not shure i can use http://e-sac.siteseguro.ws/pfsense/8/All/ldd/..
        Thanks for your fast answer and the nice tutorial 🙂

      • sichent says:

        Seems the i386 files are not available on this site. May be from FreeBSD 8 i386 system?

      • Thomas says:

        Hey sichent,

        it works fine with: http://e-sac.siteseguro.ws/pfsense/8/All/ldd/
        Now i need a NAT forward to force my https on my squid3 or dansguardian. But i still get an error. My forward looks like this:

        Portforward 2:
        Interface: LAN
        Protocol: TCP
        Source: LAN subnet
        Destination: any
        Destination Port: HTTPs to HTTPs
        Redirect IP: 127.0.0.1
        Redirect Target Port: 8080

        If i put my proxy manuale in it works fine.

      • sichent says:

        Hi Thomas, I cannot help with port forwarding, the HOWTO implies all browsers are set to use proxy explicitly. I might publish something on this topic later when I better understand the transparent scenario…

  3. Greg Zartman says:

    Good howto. However, the download links now point to qlproxy3.2 and if I try to install following these instructions I get sql errors on the web interface. Note: I also tried django1.5

    Thanks

  4. Greg Zartman says:

    Here is the specific error I get. I even went to the dev website and followed their instructions:

    Request Method: GET
    Request URL: http://192.168.0.254:8081/
    Django Version: 1.5
    Exception Type: DatabaseError
    Exception Value:
    attempt to write a readonly database
    Exception Location: /usr/local/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py in execute, line 362
    Python Executable: /usr/local/bin/python
    Python Version: 2.7.2
    Python Path:
    [‘/usr/local/var/qlproxy/console’,
    ‘/usr/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg’,
    ‘/usr/local/lib/python2.7/site-packages/pip-1.0.2-py2.7.egg’,
    ‘/usr/local/lib/python27.zip’,
    ‘/usr/local/lib/python2.7’,
    ‘/usr/local/lib/python2.7/plat-freebsd8’,
    ‘/usr/local/lib/python2.7/lib-tk’,
    ‘/usr/local/lib/python2.7/lib-old’,
    ‘/usr/local/lib/python2.7/lib-dynload’,
    ‘/usr/local/lib/python2.7/site-packages’]
    Server time: Sat, 29 Mar 2014 23:38:50 -0700

    • sichent says:

      Hello Greg,

      I went through all instructions once again for qlproxy 3.2 and did not see this error. Anyway the error message indicates the *.sqlite databases are read only for unknown reason. Please run the following command to see if error message goes away:

      chmod +w /usr/local/var/qlproxy/console/qlproxy.sqlite

      The reason for this *may* be your httpd WSGI server does not run as qlproxy user… and if this is the case there might be some other problems coming (like inability to restart qlproxy from web ui). Could you run the # ps aux | grep qlproxy when Web UI of qlproxy is up and running? The output shows the user under which Web UI runs.

  5. Greg Zartman says:

    sichent,

    The databases not being writable by apache was the problem. I made user www a member of the qlproxy group and all is fine.

    Suggest you add a couple steps to your howto:

    pw usermod www -G qlproxy

    The /usr/local/var/glproxy/console dir also needs to be group writable. After install it is only user writable to the user qlproxy. Unless apache runs as the user qlproxy, it the dbase one be accessible.

    I ended up creating the www user and group in the pfsense user utility so that the relation between the qlproxy group and the www user will survive backup/restore.

    Thanks for the tip! Qlproxy is now up and running for me.

    Greg

    • sichent says:

      Hi Greg, the thing is web ui of qlproxy runs by the WSGI daemon run by the Apache. So WSGI script of qlproxy explicitly states it needs to be run as qlproxy user. So there is no need to add qlproxy/apache user to qlproxy group. The problem may be in a different place and by changing apache/qlproxy group settings we are just curing the problem and not its roots 😦

  6. Greg Zartman says:

    I was able to do this without adding www to the qlproxy group, but instead set the perms of the qlproxy dir to 755.

  7. Jerome says:

    Integrate Squid Proxy and Diladele Web Safety

    This section doesn’t work – it seems to break Squid and stop it from starting.

    I assume Services > Proxy Server > Integrations box is where we should paste the code under that section, however you walkthrough says Services > Proxy Server > Custom Settings / Custom Options field type which doesn’t seem to exist in PFSense 2.1…

    I’ve wasted a ton of time on this now and can’t get it working – could you please advise?

    Thanks,

    • sichent says:

      Hello Jerome, what entries are in squid cache.log? It may explain the reason for failed start.

      • Jerome says:

        Hi,

        Your wording is wrong. You say “Then to integrate qlproxy and squid, go to Services / Proxy Server, scroll all the way down and in Custom Settings / Custom Options field type:”

        As this “Custom Settings / Custom Options field” doesn’t exist in 2.1 (or atleast 2.1.1 which i’m using).

        It should instead read:

        “Then to integrate qlproxy and squid, go to Services / Proxy Server, scroll all the way down to Custom ACLS (Before_Auth) or Custom ACLS (After_Auth) and add the following to either (depending on when you want Auth done).

        Other than this it was a great walk though. I’d perhaps add a bit on HTTPS filtering / captive portal in PFSense Squid.

        e.g.

        “Patch captive portal
        Enable this option to force captive portal to non transparent proxy users.
        NOTE: You may need to reapply captive portal config after changing this option. ”

        As this is a great option for forcing users to use your proxy. As the only other alternative is no internet access…

      • sichent says:

        Hi Jerome, I have not tried with 2.1.1 yet, I will try to carefully update the post when I have time for that. Please send the parts you’d like to be updated to support@diladele.com and I will try to include it into the post. Thanks!

    • Greg Zartman says:

      I too spend ALOT of time trying to get this working on PFSense and eventually just gave up. It kind of sort of works, but I ran into multiple problem from my clients seemingly loosing the Root CA to Diladele filtering mainstream web pages.

      This doesn’t seem ready for prime time on PFsense. It’s more of a Alpha level add-on.

  8. rf says:

    I too am having issue with the lines you have to paste, these ones seem to be the most troublesome:

    acl qlproxy_icap_edomains dstdomain
    “/usr/local/etc/qlproxy/squid/icap_exclusions_domains.conf”
    acl qlproxy_icap_etypes rep_mime_type
    “/usr/local/etc/qlproxy/squid/icap_exclusions_contenttypes.conf”
    adaptation_access qlproxy1 deny qlproxy_icap_edomains
    adaptation_access qlproxy2 deny qlproxy_icap_edomains
    adaptation_access qlproxy2 deny qlproxy_icap_etypes
    adaptation_access qlproxy1 allow all
    adaptation_access qlproxy2 allow all

  9. rf says:

    i managed to fix the config issue – the two files for the acl’s must be on the same line as the acl entry, not below.

    However, ssl bump isn’t working for me yet, and i’m not sure why.. I can still see the google certificate, for instance.

    Not sure what I’m looking for in the logs to see why this is, though:

    2014/04/15 15:31:55.679| Initializing https proxy context
    2014/04/15 15:31:55.680| support.cc(1000) sslCreateClientContext: Using SSLv2/SSLv3.
    2014/04/15 15:31:55.681| support.cc(1052) sslCreateClientContext: Setting RSA key generation callback.
    2014/04/15 15:31:55.681| support.cc(1059) sslCreateClientContext: Setting certificate verification callback.
    2014/04/15 15:31:55.681| support.cc(1063) sslCreateClientContext: Setting CA certificate locations.
    2014/04/15 15:31:55.681| Initializing http_port 192.168.4.217:3128 SSL context
    2014/04/15 15:31:55.681| Using certificate in /usr/pbi/squid-amd64/etc/squid/serverkey.pem
    2014/04/15 15:31:55.681| support.cc(1489) readSslX509CertificatesChain: Certificate is self-signed, will not be chained
    2014/04/15 15:31:55.694| support.cc(1351) contextMethod: Using SSLv2/SSLv3.
    2014/04/15 15:31:55.694| support.cc(798) configureSslContext: Setting RSA key generation callback.
    2014/04/15 15:31:55.694| support.cc(801) configureSslContext: Setting CA certificate locations.
    2014/04/15 15:31:55.694| support.cc(844) configureSslContext: Not requiring any client certificates

    all seems ok?

    • rf says:

      I should add – even before configuring diladele I wasn’t able to get bump to work, even after simply enabling it in the proxy server part of pfsense… so i’m not sure where the issue leis.

      • sichent says:

        Several users already reported this issue – seems pfSense UI does not generate the ‘ssl_bump server-first all’ directive (not checked yet)

      • rf says:

        oh yes. turns out when I google “ssl_bump server-first all” i’d already visited the link on the forum where it’s discussed.. my eyes are tired!

      • rf says:

        One other thing – not that I think this is related to your tutorial – but even installing the certificate from the PFSense machine and marking it trusted, etc, does not stop the “this connection is untrusted” appearing, even after restarting the browser with the certificate installed.. any thoughts on that ?

      • sichent says:

        What browser? Chrome and IE use system store on Windows for example, but Firefox has its own.

  10. rf says:

    Both chrome and firefox, in linux. Both have options under preferences/settings to store the certificate. I’ll keep playing around..

    • sichent says:

      I did not try in Linux, could you please share your finding when/if you are able to fix it?

      • rf says:

        The only way I could get it “fixed” was to tell squid/pfsense to ignore the certificate errors, as that’s an option in squid. Not sure if that was implied in this guide and the option was broken in the latter version of PFSense.

  11. cirkit says:

    Hi
    I am facing this update definations error, pl help
    Running update script at Wed Apr 16 11:33:46 IST 2014
    ********************************************************************************
    * *
    * Welcome to Diladele Web Safety Updater *
    * *
    ********************************************************************************
    No newer version than 3.2.0.4CAF is found, your version is up-to-date.
    Traceback (most recent call last):
    File “/usr/local/bin/qlproxy_update.py”, line 296, in
    result = main()
    File “/usr/local/bin/qlproxy_update.py”, line 281, in main
    if update_adblock(force) == True:
    File “/usr/local/bin/qlproxy_update.py”, line 213, in update_adblock
    verify_definitions(adblock_data + “.download”)
    File “/usr/local/bin/qlproxy_update.py”, line 115, in verify_definitions
    verify_easylist(os.path.join(root, file))
    File “/usr/local/bin/qlproxy_update.py”, line 104, in verify_easylist
    ret = subprocess.call([exe, name])
    File “/usr/local/lib/python2.7/subprocess.py”, line 493, in call
    return Popen(*popenargs, **kwargs).wait()
    File “/usr/local/lib/python2.7/subprocess.py”, line 679, in __init__
    errread, errwrite)
    File “/usr/local/lib/python2.7/subprocess.py”, line 1228, in _execute_child
    raise child_exception
    OSError: [Errno 2] No such file or directory

    • sichent says:

      Unfortunately the path in /usr/local/bin/qlproxy_update.py around line 96 is incorrect, it should read dir = os.path.join(qlproxy_opt, "bin") instead of dir = os.path.join(qlproxy_opt, "Sbin"). Could you please change it and try again?

      • cirkit says:

        Yes! it worked properly and the definations updated well.
        running update script at Wed Apr 16 13:22:21 IST 2014
        ********************************************************************************
        * *
        * Welcome to Diladele Web Safety Updater *
        * *
        ********************************************************************************
        No newer version than 3.2.0.4CAF is found, your version is up-to-date.
        Updated adblock definition files.
        Updated privacy definition files.
        Updated domains definition files.
        Update finished SUCCESSFULLY!

        In my case too the ssl bump does not work, can you guide? My version of pfsense is 2.1 amd64 and diladele is 3.2 amd64 squid3-dev version is 3.3.10

  12. sichent says:

    Hi cirkit, what is the exact version of pfSense – 2.1.0 or 2.1.2 – seems it makes a lot of difference.

  13. cirkit says:

    Ok! I managed to get SSL bump work and along with that Diladele I had to enable authentication under proxy server – local databse & users added under pfsense user manger and group-qlproxy.

    • rf says:

      Are you able to see the Diladele “Access denied” page when viewing a domain/site blocked which is https ? That’s my stumbling block right now.. http:// is fine. https:// does not work, I just get “Error 111 (net::ERR_TUNNEL_CONNECTION_FAILED): Unknown error”. I see on an http site, squid reports a “GET” request where for https, “CONNECT” is reported. I’m guessing that’s the difference? I’ll continue to read…

      • rf says:

        ssl-bump is working( i think ?), If I goto a non-filtered site I see my own CA under the ssl certificate, but I am just unable to get an error page from diladele to appear when visiting a blocked site (ie. facebook)

      • cirkit says:

        No, for SSL bumps I only see the message ” proxy server refused connection” in firefox, IE does not show any message or error simply refuses to connect.

  14. sichent says:

    Hi rf, if you see the site identified with you CA instead of the original then SSL bump really works. The thing is – it is important *how* you are accessing the squid. If it is explicit proxy – you will never see the blocked page as all modern browsers do not show any blocked requests from CONNECT tunnels… if you do transparent intercept (which I have not tried yet on pfSense) you *will see* the blocked page. It is weird I know – but I do not see any other explanation… BTW it is interesting what happens if qlproxy does not *block* but *redirect* request – will browser show it???

  15. sichent says:

    The post is updated for 2.1.3 pfSense (amd64).

  16. Nitesh Vasant says:

    Thanks

  17. ryan says:

    when i try to log in to Diladele Web UI all i see is “It works!” on web page.
    can u help ?
    tanks

    • sichent says:

      Hello Ryan! Most probably your virtual host configuration is incorrect, please see if qlproxy_virtual_host is included into httpd.conf and port numbers are correct.

      • Aucesar-BR says:

        Hi, in which section qlproxy_virtual_host must be inserted? Thanks.

      • sichent says:

        Hello Aucesar, please see the attached scripts – these do it all automatically. Just run them (understanding what each does!)

      • Aucesar says:

        Hi Sichent, the problem was I changed default port on the script to 8090, so we need to change this in the qlproxy_virtual_host file too…. Thanks for reply. Did you had the steps to integrate Active Directory using Pfsense? Thanks again….

  18. Tim Haynes says:

    Wonderful tutorial – thank you for putting it together!

    I ran into some errors when installing python, django, and apache. Do you have any suggestions that might help? I am hesitant to continue the install, given that each of these three installations generated errors that differ from your output

    -*-*-*–*-*-*–*-*-*–*-*-*–*-*-*–*-*-*–*-*-*–*-*-*–*-*-*–*-*-*–*-*-*–*-*-*–*-*-*-

    [2.1.3-RELEASE][admin@.]/root(8): pkg_add -r python27 py27-sqlite3 py27-pip
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/Latest/python27.tbz… Done.
    pkg_install EOL is scheduled for 2014-09-01. Please consider migrating to pkgng
    http://blogs.freebsdish.org/portmgr/2014/02/03/time-to-bid-farewell-to-the-old-pkg_-tools/

    ====
    Note that some of the standard modules are provided as separate
    ports since they require extra dependencies:

    bsddb databases/py-bsddb
    gdbm databases/py-gdbm
    sqlite3 databases/py-sqlite3
    tkinter x11-toolkits/py-tkinter

    Install them as needed.
    ====

    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/Latest/py27-sqlite3.tbz… Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/All/sqlite3-3.8.2_1.tbz… Done.
    pkg_install EOL is scheduled for 2014-09-01. Please consider migrating to pkgng
    http://blogs.freebsdish.org/portmgr/2014/02/03/time-to-bid-farewell-to-the-old-pkg_-tools/
    pkg_install EOL is scheduled for 2014-09-01. Please consider migrating to pkgng
    http://blogs.freebsdish.org/portmgr/2014/02/03/time-to-bid-farewell-to-the-old-pkg_-tools/
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/Latest/py27-pip.tbz… Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/All/py27-setuptools-2.0.1.tbz… Done.
    pkg_install EOL is scheduled for 2014-09-01. Please consider migrating to pkgng
    http://blogs.freebsdish.org/portmgr/2014/02/03/time-to-bid-farewell-to-the-old-pkg_-tools/
    /bin/ed: not found
    pkg_add: command ‘/usr/bin/printf ‘1a\n./pip-1.4.1-py2.7.egg\n.\nw\nq\n’ | /bin/ed /usr/local/lib/python2.7/site-packages/easy-install.pth’ failed
    pkg_install EOL is scheduled for 2014-09-01. Please consider migrating to pkgng
    http://blogs.freebsdish.org/portmgr/2014/02/03/time-to-bid-farewell-to-the-old-pkg_-tools/

    [2.1.3-RELEASE][admin@.]/root(9): /usr/local/bin/pip install Django==1.5
    Downloading/unpacking Django==1.5
    Downloading Django-1.5.tar.gz (8.0MB): 8.0MB downloaded
    Running setup.py egg_info for package Django

    warning: no previously-included files matching ‘__pycache__’ found under directory ‘*’
    warning: no previously-included files matching ‘*.py[co]’ found under directory ‘*’
    Installing collected packages: Django
    Running setup.py install for Django
    changing mode of build/scripts-2.7/django-admin.py from 644 to 755

    warning: no previously-included files matching ‘__pycache__’ found under directory ‘*’
    warning: no previously-included files matching ‘*.py[co]’ found under directory ‘*’
    changing mode of /usr/local/bin/django-admin.py to 755
    Could not find .egg-info directory in install record for Django==1.5
    Successfully installed Django
    Cleaning up…

    [2.1.3-RELEASE][admin@.]/root(10): pkg_add -r apache22 ap22-mod_wsgi
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/Latest/apache22.tbz… Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/All/expat-2.1.0.tbz… Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/All/perl5-5.16.3_6.tbz… Done.
    /usr/bin/makewhatis: not found
    pkg_add: command ‘/usr/bin/makewhatis /usr/local/lib/perl5/5.16/perl/man’ failed
    /usr/bin/makewhatis: not found
    pkg_add: command ‘/usr/bin/makewhatis /usr/local/lib/perl5/5.16/man’ failed
    Removing /usr/local/etc/perl5_version… Done.
    Creating /usr/local/etc/perl5_version… Done.
    cd: can’t cd to /usr/include
    Removing stale symlinks from /usr/bin…
    Skipping /usr/bin/perl
    Skipping /usr/bin/perl5
    Done.
    Creating various symlinks in /usr/bin…
    Symlinking /usr/local/bin/perl5.16.3 to /usr/bin/perl
    Symlinking /usr/local/bin/perl5.16.3 to /usr/bin/perl5
    Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/All/pcre-8.34.tbz… Done.
    pkg_install EOL is scheduled for 2014-09-01. Please consider migrating to pkgng
    http://blogs.freebsdish.org/portmgr/2014/02/03/time-to-bid-farewell-to-the-old-pkg_-tools/
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/All/db42-4.2.52_5.tbz… Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/All/gdbm-1.11.tbz… Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/All/apr-1.4.8.1.5.3.tbz… Done.
    ===> Creating users and/or groups.
    Using existing group ‘www’.
    Using existing user ‘www’.
    /usr/local/share/examples/apache22/httpd.conf -> /usr/local/etc/apache22/httpd.conf
    pkg_install EOL is scheduled for 2014-09-01. Please consider migrating to pkgng
    http://blogs.freebsdish.org/portmgr/2014/02/03/time-to-bid-farewell-to-the-old-pkg_-tools/

    To run apache www server from startup, add apache22_enable=”YES”
    in your /etc/rc.conf. Extra options can be found in startup script.

    Your hostname must be resolvable using at least 1 mechanism in
    /etc/nsswitch.conf typically DNS or /etc/hosts or apache might
    have issues starting depending on the modules you are using.

    Error: Unable to get ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/Latest/ap22-mod_wsgi.tbz: File unavailable (e.g., file not found, no access)
    pkg_add: unable to fetch ‘ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/Latest/ap22-mod_wsgi.tbz’ by URL

  19. ryan says:

    it was httpd.conf file
    Thanks sichent

  20. Stephan says:

    Hi

    I did everything step by step but if i log on to the website with local-Ip:8080 it shows “its working” and not showing the information, any suggestions ?

  21. Stephan says:

    Hi
    Thanks, i got it working now.
    I have one problem, my pfsense box is outside of my network i can’t get the proxy server to work properly is your pfsense box inside the same network subnet ?

  22. Tim Haynes says:

    Okay, no replies to my massive “help me” post, so maybe if I ask more directed questions, I will get a hand. 🙂

    Is anybody else getting this error during install of python, django, and apache?

    “pkg_install EOL is scheduled for 2014-09-01. Please consider migrating to pkgng”

    Also, anybody else getting this error during python install?

    “/bin/ed: not found
    pkg_add: command ‘/usr/bin/printf ’1a\n./pip-1.4.1-py2.7.egg\n.\nw\nq\n’ | /bin/ed /usr/local/lib/python2.7/site-packages/easy-install.pth’ failed”

    • sichent says:

      Hello Tim, I have seen the error about pkg_install but ignored it. I never saw error /bin/ed. What version of pfSense you are using (2.1.3)? Nano or normal? What architecture?

      • Tim Haynes says:

        Thanks for replying! I will ignore the pkg_install errors. I am using 2.1.3 normal on i386. When I install pfSense using the “automated install” which is supposed to do everything by itself, it errors out. When I do an “advanced install” and answer all the questions, it seems to install fine. However, I have low confidence due to the errors during automated install. Perhaps I should redownload, reburn to CD, and reinstall using the “automated” install to see if that fixes any potentially missed files.

      • sichent says:

        Hi Tim, unfortunately I did not try on x86 platform. Going to x64 is not an option? I also do not have *any* errors during automatic install so may be this is the ground issue you are facing..

  23. Tim Haynes says:

    Okay, I reinstalled with a fresh CD of pfSense, and I was able to fully install everything, just as your tutorial shows. Great! It’s not actually filtering content for me, which I hope to work on tonight. (i.e. viewing adult content is not blocked) Thanks for your help thus far.

    • sichent says:

      Hi Tim, We have a new qlproxy 3.3 for pfSense available for install from http://quintolabs.com/soon.php. It contains advanced exclusions and custom categories. May be you could try it? The installation folder is changed though and some manual steps needed for qlproxy 3.2 is now automated. Moving from 3.2 to 3.3 will be not so easy as moving from 3.3 alpha to 3.3 release.

      • Tim Haynes says:

        Alpha software… hmmm… Is it possible to crash and then nobody can get out to the WAN? If it crashes and only takes itself down, that’s fine. I just wouldn’t want to install it and find out that an hourly crash keeps taking down Internet access for everybody.

      • sichent says:

        The core code is quite stable… but it is ok to wait for RC of course…

  24. Tim Haynes says:

    So I have everything installed and configured, but am having a problem getting the filter to do anything in transparent mode, even under HTTP (haven’t tried HTTPS yet). If I specify the proxy in IE or Chrome (pfsense port 3128) then it will hit squid and qlproxy. If I turn on transparent HTTP proxying and remove the proxy settings from the browser, then it doesn’t go through squid or qlproxy.

    Any suggestions of what to check? I double-checked all of the config file edits suggested in this tutorial.

    • sichent says:

      Hello Tim, thw howto explicitly states I have not tried transparent filtering with pfSense… but please take a look at generated squid conf file – the squid ui in pfsense sometime generates incorrect conf entries…

  25. Tim Haynes says:

    Looks like I was able to get transparent HTTP and HTTPS proxy working simply by enabling “Allow users on interface” in the Proxy server’s General settings. With that on, my LAN devices don’t need to specify a proxy, but their connections pass through squid and qlproxy. When I install the cert, HTTPS works with no cert errors. Fantastic! I can live with having to install a cert on any devices that need it, and they all get HTTP/HTTPS transparent content filtering.

    • sichent says:

      The question now is how to exclude specific domains from SSL Bump? In transparent filtering Squid cannot do this it needs to be done from the firewall that should not intercept traffic to such sites…

      • Tim Haynes says:

        I see what you mean. I can live without being able to exclude domains from SSL Bump, because the alternative right now is no HTTPS filtering at all, which would mean that Google Image Search, or any HTTPS site (even the obviously bad ones), are wide-open for the children and teenagers on my network.

  26. cirkit says:

    Hi Sichent
    I built a pfsense server with Diladele.
    Initially the lan ip of my server was 192.168.100.40, later I changed to 192.168.1.1,
    My definition updates as of today are dated 4th June 2014.
    since the change, my report generation is not working, I keep seeing the old reports only, new reports cannot be viewed, I have corrected the report script as stated above, but it still keeps showing older reports only, when I run the report update script, the output is as below

    Pl help…

    Running report collection script script at Wed Jun 11 16:24:15 IST 2014
    2014-06-11 05:54:16,259 Diladele Web Safety Monitor is starting…
    2014-06-11 05:54:16,260 Using database engine django.db.backends.sqlite3
    2014-06-11 05:54:16,260 Using database name /usr/local/var/qlproxy/console/monitor.sqlite
    2014-06-11 05:54:16,261 Enumerating folder /usr/local/var/qlproxy/monitor…
    2014-06-11 05:54:16,419 Found 8428 monitor files to upload, total size 21.0 MB…
    2014-06-11 05:54:16,419 Uploading file 1 of 8428, /usr/local/var/qlproxy/monitor/a9974b5a-9ed0-49b0-9197-f254c51bdaba.monitor…
    2014-06-11 05:54:16,461 File /usr/local/var/qlproxy/monitor/a9974b5a-9ed0-49b0-9197-f254c51bdaba.monitor removed.
    2014-06-11 05:54:16,461 Uploading file 2 of 8428, /usr/local/var/qlproxy/monitor/97bbdafa-c56a-4973-829d-843957cb26da.monitor…
    2014-06-11 05:54:16,474 File /usr/local/var/qlproxy/monitor/97bbdafa-c56a-4973-829d-843957cb26da.monitor removed.
    2014-06-11 05:54:16,474 Uploading file 3 of 8428, /usr/local/var/qlproxy/monitor/eda26041-128e-4513-849c-6404221c8dcb.monitor…
    2014-06-11 05:54:16,498 File /usr/local/var/qlproxy/monitor/eda26041-128e-4513-849c-6404221c8dcb.monitor removed.
    2014-06-11 05:54:16,498 Uploading file 4 of 8428, /usr/local/var/qlproxy/monitor/bfdfce46-d9fa-4725-8258-477d8d1713f8.monitor…
    2014-06-11 05:54:16,516 File /usr/local/var/qlproxy/monitor/bfdfce46-d9fa-4725-8258-477d8d1713f8.monitor removed.
    2014-06-11 05:54:16,516 Uploading file 5 of 8428, /usr/local/var/qlproxy/monitor/4e52406b-6186-46df-a30c-0aff1a21f62a.monitor…
    2014-06-11 05:54:16,524 File /usr/local/var/qlproxy/monitor/4e52406b-6186-46df-a30c-0aff1a21f62a.monitor removed.
    2014-06-11 05:54:16,524 Uploading file 6 of 8428, /usr/local/var/qlproxy/monitor/cfeb34cb-918b-4d8d-ab6f-bd33e91e1bbd.monitor…
    Traceback (most recent call last):
    File “/usr/local/var/qlproxy/console/import.py”, line 339, in
    main()
    File “/usr/local/var/qlproxy/console/import.py”, line 334, in main
    upload(leave)
    File “/usr/local/var/qlproxy/console/import.py”, line 270, in upload
    upload_file(path, count, len(files))
    File “/usr/local/var/qlproxy/console/import.py”, line 205, in upload_file
    entries = parser.get_entries()
    File “/usr/local/var/qlproxy/console/import.py”, line 109, in get_entries
    if self.parse_event(event, line):
    File “/usr/local/var/qlproxy/console/import.py”, line 128, in parse_event
    event.user_name = self.cache.get(“UserName”, fields.popleft())
    IndexError: pop from an empty deque
    Running report generation script script at Wed Jun 11 16:24:16 IST 2014
    2014-06-11 05:54:16,885 Diladele Web Safety Reporter is starting…
    2014-06-11 05:54:16,885 Using database engine django.db.backends.sqlite3
    2014-06-11 05:54:16,886 Using database name /usr/local/var/qlproxy/console/monitor.sqlite
    2014-06-11 05:54:17,124 Updated overall statistics
    2014-06-11 05:54:18,839 Updated daily statistics
    2014-06-11 05:54:19,110 Updated statistics for IP 192.168.100.74
    2014-06-11 05:54:19,115 Updated statistics for IP 192.168.100.75
    2014-06-11 05:54:19,120 Updated statistics for IP 192.168.100.38
    2014-06-11 05:54:19,125 Updated statistics for IP 192.168.100.76
    2014-06-11 05:54:19,130 Updated statistics for IP 192.168.100.244
    2014-06-11 05:54:19,135 Updated statistics for IP 192.168.100.243
    2014-06-11 05:54:19,140 Updated statistics for IP 192.168.100.242
    2014-06-11 05:54:19,145 Updated statistics for IP 192.168.100.231
    2014-06-11 05:54:19,418 Updated statistics for User jigna
    2014-06-11 05:54:19,422 Updated statistics for User test2
    2014-06-11 05:54:19,427 Updated statistics for User nitesh
    2014-06-11 05:54:19,432 Updated statistics for User test1
    2014-06-11 05:54:19,436 Updated statistics for User –

    • sichent says:

      Hello Cirkit,

      Seems the upload script cannot parse the log entries.. could you send me the contents of /usr/local/var/qlproxy/monitor/ to me to support@diladele.com for analysis?

      Thanks!

      • cirkit says:

        Hi Sichent
        The errors pointed to some corruption in monitor files so I deleted about 5000 old monitor files and kept the ones of June 2014 and re-ran the update report script and now it shows the current reports correctly…one query it still keeps showing old users which I have deleted and no more exist…how do I delete those ,rotate the report log…or completely start fresh from today erasing all previous instances ,incidents, ip logs, userlogs.

      • sichent says:

        Hello Cirkit, the easiest way to reset the reports database is to run the following commands (but please make copies of *.sqlite files first!!!):

        mv /usr/local/var/qlproxy/console/qlproxy.sqlite qlproxy.sqlite.bak
        mv /usr/local/var/qlproxy/console/report.sqlite report.sqlite.bak
        mv /usr/local/var/qlproxy/console/monitor.sqlite monitor.sqlite.bak
        python /usr/local/var/qlproxy/console/manage.py syncdb –database=monitor
        python /usr/local/var/qlproxy/console/manage.py syncdb –database=report

        I have not tried these commands, so please adjust if I missed the paths. Hopefully the idea is clear. Note we are NOT resetting the qlproxy database as it stores configuration not reports!

      • sichent says:

        I have updated the docs site (please note it is tailored to qlproxy 3.3 beta for now – http://docs.diladele.com/administrator_guide_3_3/system_administration/reset_monitoring_data.html)

  27. Tim Haynes says:

    Hi again, Sichent

    So it looks like everything was working fine for about 2 days, and now I am constantly getting icap errors (TCP_MISS/500 all over the squid logs). Any ideas?

    To the rest of the bunch, I just now discovered that qlproxy has a support forum here, which could help some of us out: http://groups.google.com/group/quintolabs-content-security-for-squid-proxy

    • sichent says:

      Hello Tim, please take a look if qlproxyd is running (ps -aux | grep qlp) (or in the web ui) and see the log of qlproxy in /usr/local/var/qlproxy/logs – this might give the clue what happened to the daemon. Also cache.log of squid gives more information *why* icap 500 errors occur.

  28. A Mohan Rao says:

    Hello,
    as per your instructions i have do same.
    but on chrome google or gmail both are not working. In firefox only google search working also the same gmail is still not working. i have install a fresh pfsense firewall. but facing same problem all time and i using transparent http or https proxy .

    Thanks
    waiting for your positive and good response.

  29. cirkit says:

    Hi Sichent,
    Thanks for new ver 3.3 guide for pfsense,
    The link to the update procedure from 3.2 to 3.3 contains a batch file and an exe file, is this to be used with pfsense?
    Can you please send links to the earlier procedures for ver 3.2 for pfsense 2.1 & 2.1.3 as I would like to store them for reference before I migrate to the new version, a text file would be highly appreciated…..weblinks would also do.
    Thanks

    • sichent says:

      Hello cirkit, the file needs to be run on Windows PC. Actually these are SQL scripts that can be run on any sqlite compatible platform. For now we do not have automatic updates.

  30. cirkit says:

    Hi Sichent..cirkit again,
    I had installed a pfsense 2.1 firewall with Intel E2180 2.0Ghz CPU, 2GB ddr2 Ram,80GB HDD, G31 Intel Motherboard With two Realtek Nic’s. There are 80 computers behind this firewall, My input bandwidth is ADSL 8mbps / 2 mbps, Squid 3.3.10 & Diladele 3.2.04. After 4.5 days of operation I started receiving the fatal ” Icap protocol error”, I restarted squid and qlproxy but I it kept failing every few minutes, The GUI of Pfsense was showing 84% memory used, Finally I had to restart the Firewall / proxy and then it stabilized, what do you think might be the problem and what do you suggest.. I looked into qlproxy logs and squid access and cache logs but could not find any hint there.

    • sichent says:

      May be Squid cache? Or number of file descriptors? If you disable icap then the bug still exists?

      • cirkit says:

        With icap disabled the system does not crash.

        Cirkit

      • sichent says:

        Do you have any crash dumps of qlproxyd? Any errors in the squid logs? What ICAP server is unavailable – does it mean qlproxyd is not running (use ps aux| grep qlp to see if it is running). No errors without ICAP may mean invalid number of file descriptors is not reached as ICAAP uses twice as much as usually)

      • cirkit says:

        will post the outputs on further crashes..for now system is UP.

      • sichent says:

        First let’s see if the error about file descriptors is really in the cache.log. I remember something like this was discussed in the squid mailing list about squid-dev in pfsense…

  31. cirkit says:

    what have I to do for ” file descriptors : can you throw some light on this issue and suggest any changes in config that I can do and check-out…

  32. cirkit says:

    Hi Sichent,
    Please help, Once again my pfsense server started giving ” Icap protocol error” after 4 days, last time it crashed after 4.5 days, I am attaching the qlproxy logs, the moment I disabled icap from squid conf, the system resumed, one hint which I would like to give you is the status of my memory utilization from the gui of pfsense, on a 2gb system, after reboot the utilization was 27% in the first 24hrs, then 53% in the next 24hrs, then 79 in the next 24hrs then 93% and it crashed….

  33. cirkit says:

    Hi sichent Sorry I forgot to attach the qlproxy logs in the earlier comment,
    here they are

    [2014-Jun-19 17:31:01] [info] Diladele Web Safety is starting…
    [2014-Jun-19 17:31:02] [info] Diladele Web Safety is started
    [2014-Jun-19 17:31:02] [info] need to create filtering services, creating…
    [2014-Jun-19 17:59:04] [info] need to reload filtering services, reloading…
    [2014-Jun-19 18:59:03] [info] need to reload filtering services, reloading…
    [2014-Jun-19 19:59:04] [info] need to reload filtering services, reloading…
    [2014-Jun-19 20:59:07] [info] need to reload filtering services, reloading…
    [2014-Jun-19 21:59:05] [info] need to reload filtering services, reloading…
    [2014-Jun-19 22:59:04] [info] need to reload filtering services, reloading…
    [2014-Jun-19 23:59:05] [info] need to reload filtering services, reloading…
    [2014-Jun-20 00:59:05] [info] need to reload filtering services, reloading…
    [2014-Jun-20 01:59:05] [info] need to reload filtering services, reloading…
    [2014-Jun-20 02:59:04] [info] need to reload filtering services, reloading…
    [2014-Jun-20 03:59:03] [info] need to reload filtering services, reloading…
    [2014-Jun-20 04:59:03] [info] need to reload filtering services, reloading…
    [2014-Jun-20 05:59:03] [info] need to reload filtering services, reloading…
    [2014-Jun-20 06:59:04] [info] need to reload filtering services, reloading…
    [2014-Jun-20 07:59:03] [info] need to reload filtering services, reloading…
    [2014-Jun-20 08:59:04] [info] need to reload filtering services, reloading…
    [2014-Jun-20 09:59:04] [info] need to reload filtering services, reloading…
    [2014-Jun-20 10:07:28] [info] need to reload filtering services, reloading…
    [2014-Jun-20 10:39:52] [info] need to reload filtering services, reloading…
    [2014-Jun-20 10:59:03] [info] need to reload filtering services, reloading…
    [2014-Jun-20 11:59:04] [info] need to reload filtering services, reloading…

    No errors and after this I started receiving icap protocol errors

    here is my squid conf
    4 http_port 192.168.1.1:3128 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=10MB cert=/usr/pbi/squid-amd64/etc/squid/serverkey.pem capath=/usr/pbi/squid-amd64/share/certs/
    6 icp_port 0
    7 dns_v4_first off
    8 pid_filename /var/run/squid.pid
    9 cache_effective_user proxy
    10 cache_effective_group proxy
    11 error_default_language en
    12 icon_directory /usr/pbi/squid-amd64/etc/squid/icons
    13 visible_hostname localhost
    14 cache_mgr admin@localhost
    15 access_log /var/squid/log/access.log
    16 cache_log /var/squid/log/cache.log
    17 cache_store_log none
    18 netdb_filename /var/squid/log/netdb.state
    19 pinger_enable on
    20 pinger_program /usr/pbi/squid-amd64/libexec/squid/pinger
    21 sslcrtd_program /usr/pbi/squid-amd64/libexec/squid/ssl_crtd -s /var/squid/lib/ssl_db -M 4MB -b 2048
    22 sslcrtd_children 5
    23 sslproxy_capath /usr/pbi/squid-amd64/share/certs/
    25 logfile_rotate 1
    26 debug_options rotate=1
    27 shutdown_lifetime 3 seconds
    29 acl localnet src 192.168.1.0/24
    30 via off
    31 uri_whitespace strip
    33 acl dynamic urlpath_regex cgi-bin \?
    34 cache deny dynamic
    36 cache_mem 128 MB
    37 maximum_object_size_in_memory 32 KB
    38 memory_replacement_policy heap GDSF
    39 cache_replacement_policy heap LFUDA
    40 cache_dir ufs /var/squid/cache 16384 16 256
    41 minimum_object_size 0 KB
    42 maximum_object_size 4 KB
    43 offline_mode off
    44 cache_swap_low 90
    45 cache_swap_high 95
    46 cache allow all
    57 acl allsrc src all
    58 acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 3128 3127 1025-65535 88
    59 acl sslports port 443 563 9097
    64 acl purge method PURGE
    65 acl connect method CONNECT
    68 acl HTTP proto HTTP
    69 acl HTTPS proto HTTPS
    70 acl unrestricted_hosts src “/var/squid/acl/unrestricted_hosts.acl”
    71 acl blacklist dstdom_regex -i “/var/squid/acl/blacklist.acl”
    72 http_access allow manager localhost
    74 http_access deny manager
    75 http_access allow purge localhost
    76 http_access deny purge
    77 http_access deny !safeports
    78 http_access deny CONNECT !sslports
    85 request_body_max_size 0 KB
    86 delay_pools 1
    87 delay_class 1 2
    88 delay_parameters 1 -1/-1 -1/-1
    89 delay_initial_bucket_level 100
    90 delay_access 1 allow allsrc
    96 icap_enable off
    97 icap_preview_enable on
    98 icap_preview_size 4096
    99 icap_persistent_connections on
    100 icap_send_client_ip on
    101 icap_send_client_username on
    102 icap_client_username_header X-Client-Username
    103 icap_service qlproxy1 reqmod_precache bypass=0 icap://127.0.0.1:1344/reqmod
    104 icap_service qlproxy2 respmod_precache bypass=0 icap://127.0.0.1:1344/respmod
    106 acl qlproxy_icap_edomains dstdomain “/usr/local/etc/qlproxy/squid/icap_exclusions_domains.conf”
    107 acl qlproxy_icap_etypes rep_mime_type “/usr/local/etc/qlproxy/squid/icap_exclusions_contenttypes.conf”
    109 adaptation_access qlproxy1 deny qlproxy_icap_edomains
    110 adaptation_access qlproxy2 deny qlproxy_icap_edomains
    111 adaptation_access qlproxy2 deny qlproxy_icap_etypes
    112 adaptation_access qlproxy1 allow all
    113 adaptation_access qlproxy2 allow all
    115 icap_service_failure_limit -1
    117 acl NoXForwardedFor dst “/usr/pbi/squid-amd64/etc/squid/NoXForwardedFor.txt”
    118 request_header_access X-Forwarded-For deny NoXForwardedFor
    120 acl nossl_sites dstdomain “/usr/pbi/squid-amd64/etc/squid/nossl_sites.txt”
    121 ssl_bump none nossl_sites
    124 http_access allow unrestricted_hosts
    126 http_access deny blacklist
    127 auth_param basic program /usr/pbi/squid-amd64/libexec/squid/basic_ncsa_auth /var/etc/squid.passwd
    128 auth_param basic children 5
    129 auth_param basic realm Please enter your credentials to access the proxy
    130 auth_param basic credentialsttl 5 minutes
    131 acl password proxy_auth REQUIRED
    135 always_direct allow all
    136 ssl_bump server-first all
    137 http_access allow unrestricted_hosts
    138 http_access allow password localnet
    140 http_access deny allsrc

    here is my squid cache log
    CPU Usage: 0.343 seconds = 0.183 user + 0.160 sys
    Maximum Resident Size: 87152 KB
    Page faults with physical i/o: 0
    2014/06/20 07:30:51 kid1| Starting Squid Cache version 3.3.10 for amd64-portbld-freebsd8.3…
    2014/06/20 07:30:51 kid1| parse error while reading template file: /usr/pbi/squid-amd64/etc/squid/errors/en/error-details.txt
    2014/06/20 07:30:51 kid1| Unable to load default error language files. Reset to backups.
    2014/06/20 07:30:51 kid1| parse error while reading template file: /usr/pbi/squid-amd64/etc/squid/errors/templates/error-details.txt
    2014/06/20 07:30:51 kid1| WARNING: failed to find or read error text file error-details.txt
    2014/06/20 07:30:51| pinger: Initialising ICMP pinger …
    2014/06/20 07:41:04 kid1| Starting new basicauthenticator helpers…
    2014/06/20 13:35:49 kid1| Warning: empty ACL: acl qlproxy_icap_etypes rep_mime_type “/usr/local/etc/qlproxy/squid/icap_exclusions_contenttypes.conf”
    2014/06/20 13:35:50 kid1| parse error while reading template file: /usr/pbi/squid-amd64/etc/squid/errors/en/error-details.txt
    2014/06/20 13:35:50 kid1| Unable to load default error language files. Reset to backups.
    2014/06/20 13:35:50 kid1| parse error while reading template file: /usr/pbi/squid-amd64/etc/squid/errors/templates/error-details.txt
    2014/06/20 13:35:50 kid1| WARNING: failed to find or read error text file error-details.txt
    2014/06/20 13:35:50| pinger: Initialising ICMP pinger …
    2014/06/20 13:35:51 kid1| Warning: empty ACL: acl qlproxy_icap_etypes rep_mime_type “/usr/local/etc/qlproxy/squid/icap_exclusions_contenttypes.conf”
    2014/06/20 13:35:51 kid1| parse error while reading template file: /usr/pbi/squid-amd64/etc/squid/errors/en/error-details.txt
    2014/06/20 13:35:51 kid1| Unable to load default error language files. Reset to backups.
    2014/06/20 13:35:51 kid1| parse error while reading template file: /usr/pbi/squid-amd64/etc/squid/errors/templates/error-details.txt
    2014/06/20 13:35:51 kid1| WARNING: failed to find or read error text file error-details.txt
    2014/06/20 13:35:51| pinger: Initialising ICMP pinger …
    2014/06/20 13:35:51 kid1| Starting new basicauthenticator helpers…
    2014/06/20 13:36:24 kid1| FATAL: dying from an unhandled exception: !theOptionsFetcher
    2014/06/20 13:36:27 kid1| Starting Squid Cache version 3.3.10 for amd64-portbld-freebsd8.3…
    2014/06/20 13:36:27 kid1| parse error while reading template file: /usr/pbi/squid-amd64/etc/squid/errors/en/error-details.txt
    2014/06/20 13:36:27 kid1| Unable to load default error language files. Reset to backups.
    2014/06/20 13:36:27 kid1| parse error while reading template file: /usr/pbi/squid-amd64/etc/squid/errors/templates/error-details.txt
    2014/06/20 13:36:27 kid1| WARNING: failed to find or read error text file error-details.txt
    2014/06/20 13:36:27| pinger: Initialising ICMP pinger …
    2014/06/20 13:36:27 kid1| Starting new basicauthenticator helpers…

  34. sichent says:

    Try to disable caching on Squid and enable back icap – let it run for a while – seems you are running out of memory with caching enabled.

  35. cirkit says:

    I reduced memory allocation on the proxy server GUI of pfsense from 128Mb to 8Mb, How do I completely disable caching?….I am watching the performance…will soon update…..Bit if I am running out of memory why do I receive ” icap protocol error” only after 4 days every time…My daily load behind the firewall is same and bandwidth utilization is also fairly constant?

  36. cirkit says:

    By the time I finished writing the above comment, I started getting the “icap protocol error”
    output os ps aux | grep qlp is
    [2.1-RELEASE][root@pfsense.myftp.biz]/root(1): ps aux| grep qlp
    qlproxy 5613 0.0 1.3 161000 27776 ?? I Mon08PM 0:59.50 httpd: (wsgi:di
    qlproxy 83984 0.0 0.1 86864 2396 ?? IN Mon08PM 0:06.08 httpd: (wsgi:di
    root 25126 0.0 0.1 9068 1292 0 S+ 4:47PM 0:00.00 grep qlp

    • sichent says:

      Good then I do not see the qlproxyd running – it might be a serious error. Not sure how to debug it – does pfsense has the gdb debugger built in?

  37. cirkit says:

    Hi sichent
    I have an exactly similar setup and same configuration running squid3-dev and dansguardian on pfsense 2.1 and that setup has never experienced such issues of running out of memory.

  38. sichent says:

    Then seems it is not a memory issue, may I ask to stop qlproxy, run gdb something like this:

    % gdb /usr/local/sbin/qlproxyd –nodaemon
    handle SIGPIPE pass nostop noprint
    handle SIGTERM pass nostop noprint
    handle SIGUSR1 pass nostop noprint
    handle SIGSEGV stop
    handle SIGABRT stop
    run
    [wait for crash]
    backtrace
    generate-core-file
    quit

    then start squid and wait until crash occurs after that please send me the core file (packed) to support@diladele.com
    Thanks!

  39. cirkit says:

    I am confused about the memory utilization increasing every 24 hrs ( exactly 24hrs…..any squid3 cron task may be doing this?)
    should I double my RAM from 2Gb to 4Gb and check…( I dought it may last another 4 days..totaling to 8 days)
    my cron tasks as below

    0 * * * * root /usr/bin/nice -n20 newsyslog
    1,31 0-5 * * * root /usr/bin/nice -n20 adjkerntz -a
    1 3 1 * * root /usr/bin/nice -n20 /etc/rc.update_bogons.sh
    */60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 sshlockout
    1 1 * * * root /usr/bin/nice -n20 /etc/rc.dyndns.update
    */60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot
    30 12 * * * root /usr/bin/nice -n20 /etc/rc.update_urltables
    59 * * * * root sh /usr/local/etc/periodic/daily/511.qlproxy_update
    50 * * * * root sh /usr/local/etc/periodic/daily/510.qlproxy_report
    30 07 * * * root /var/etc/pppoe_restart_pppoe0
    0 0 * * * root /usr/pbi/squid-amd64/sbin/squid -k rotate -f /usr/pbi/squid-amd64/etc/squid/squid.conf
    */15 * * * * root /usr/local/pkg/swapstate_check.php
    */30 * * * * root /usr/local/bin/php /usr/local/www/sarg.php 0
    0 */24 * * * root /usr/local/bin/php /usr/local/www/sarg.php 1

  40. cirkit says:

    Do you recommend to disable ” all request monitoring” and observe?
    But the GUI of pfsense has never reported CPU utilization above 5%.
    NItesh

  41. cirkit says:

    Presently This system runs at a school with 80 computers. Its is OFF time now, children & staff both have left, there are no computers on-line ..then too the memory utilization is 91% and cpu utilization 2%…Is there any way to bring down this memory utilization
    There is a utility called “free” which gives you detail of memory utilization..here is the output

    [2.1-RELEASE][root@pfsense.myftp.biz]/root(13): free
    SYSTEM MEMORY INFORMATION:
    mem_wire: 1560576000 ( 1488MB) [ 76%] Wired: disabled for paging out
    mem_active: + 245309440 ( 233MB) [ 11%] Active: recently referenced
    mem_inactive:+ 101527552 ( 96MB) [ 4%] Inactive: recently not referenced
    mem_cache: + 54484992 ( 51MB) [ 2%] Cached: almost avail. for allocation
    mem_free: + 86528000 ( 82MB) [ 4%] Free: fully available for allocation
    mem_gap_vm: + 647168 ( 0MB) [ 0%] Memory gap: UNKNOWN
    ————– ———— ———– ——
    mem_all: = 2049073152 ( 1954MB) [100%] Total real memory managed
    mem_gap_sys: + 63000576 ( 60MB) Memory gap: Kernel?!
    ————– ———— ———–
    mem_phys: = 2112073728 ( 2014MB) Total real memory available
    mem_gap_hw: + 35409920 ( 33MB) Memory gap: Segment Mappings?!
    ————– ———— ———–
    mem_hw: = 2147483648 ( 2048MB) Total real memory installed

    SYSTEM MEMORY SUMMARY:
    mem_used: 1904943104 ( 1816MB) [ 88%] Logically used memory
    mem_avail: + 242540544 ( 231MB) [ 11%] Logically available memory
    ————– ———— ———– ——
    mem_total: = 2147483648 ( 2048MB) [100%] Logically total memory
    [2.1-RELEASE][root@pfsense.myftp.biz]/root(14):

    Nitesh

  42. cirkit says:

    yes, no change

    • sichent says:

      Then the icap filtering server is not the reason for memory utilization. What if you run ps aux and analyze which process takes most of the memory? (and please try to get the core dump crash file for qlproxy)

  43. cirkit says:

    this is the output of ps aux…I am trying for the core dump.
    [2.1-RELEASE][root@pfsense.myftp.biz]/root(1): ps aux
    USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
    root 11 200.0 0.0 0 32 ?? RL Mon08PM 10707:49.99 [idle]
    root 0 0.0 0.0 0 160 ?? DLs Mon08PM 17:12.13 [kernel]
    root 1 0.0 0.0 3200 120 ?? ILs Mon08PM 0:00.05 /sbin/init —
    root 2 0.0 0.0 0 16 ?? DL Mon08PM 0:00.00 [g_event]
    root 3 0.0 0.0 0 16 ?? DL Mon08PM 10:06.84 [g_up]
    root 4 0.0 0.0 0 16 ?? DL Mon08PM 12:03.07 [g_down]
    root 5 0.0 0.0 0 16 ?? DL Mon08PM 0:00.00 [crypto]
    root 6 0.0 0.0 0 16 ?? DL Mon08PM 0:00.00 [crypto returns]
    root 7 0.0 0.0 0 16 ?? DL Mon08PM 0:01.24 [fdc0]
    root 8 0.0 0.0 0 16 ?? DL Mon08PM 0:00.00 [sctp_iterator]
    root 9 0.0 0.0 0 16 ?? DL Mon08PM 0:12.21 [pfpurge]
    root 10 0.0 0.0 0 16 ?? DL Mon08PM 0:00.00 [audit]
    root 12 0.0 0.0 0 336 ?? WL Mon08PM 50:47.40 [intr]
    root 13 0.0 0.0 0 32 ?? DL Mon08PM 0:00.36 [ng_queue]
    root 14 0.0 0.0 0 16 ?? DL Mon08PM 1:25.36 [yarrow]
    root 15 0.0 0.0 0 320 ?? DL Mon08PM 0:06.24 [usb]
    root 16 0.0 0.0 0 16 ?? DL Mon08PM 0:00.00 [xpt_thrd]
    root 17 0.0 0.0 0 16 ?? DL Mon08PM 0:28.37 [pagedaemon]
    root 18 0.0 0.0 0 16 ?? DL Mon08PM 0:00.14 [vmdaemon]
    root 19 0.0 0.0 0 16 ?? DL Mon08PM 0:00.00 [pagezero]
    root 20 0.0 0.0 0 16 ?? DL Mon08PM 0:00.33 [idlepoll]
    root 21 0.0 0.0 0 16 ?? DL Mon08PM 0:38.94 [bufdaemon]
    root 22 0.0 0.0 0 16 ?? DL Mon08PM 2:57.73 [syncer]
    root 23 0.0 0.0 0 16 ?? DL Mon08PM 0:09.03 [vnlru]
    root 24 0.0 0.0 0 16 ?? DL Mon08PM 0:01.66 [softdepflush]
    root 37 0.0 0.0 0 32 ?? DL Mon08PM 0:03.56 [zfskern]
    root 58 0.0 0.0 0 16 ?? DL Mon08PM 0:39.92 [md0]
    root 63 0.0 0.0 0 16 ?? DL Mon08PM 0:22.79 [md1]
    root 304 0.0 0.0 6908 824 ?? INs Mon08PM 15:14.71 /usr/local/sbin/check_reload_status
    root 306 0.0 0.0 6908 0 ?? IWN – 0:00.00 check_reload_status: Monitoring daemon of check_reload_status
    root 315 0.0 0.0 5248 548 ?? Is Mon08PM 0:00.01 /sbin/devd
    root 5170 0.0 0.1 8296 1096 ?? SN 5:30PM 0:00.14 /bin/sh /var/db/rrd/updaterrd.sh
    root 5264 0.0 0.1 82708 2056 ?? Ss Mon08PM 0:07.94 /usr/local/sbin/httpd -DNOHTTPACCEPT
    qlproxy 5613 0.0 0.1 161000 2776 ?? I Mon08PM 1:02.35 httpd: (wsgi:diladele.lan) (httpd)
    www 8596 0.0 0.0 86804 468 ?? I Mon08PM 0:00.20 /usr/local/sbin/httpd -DNOHTTPACCEPT
    www 9011 0.0 0.0 86804 560 ?? I Mon08PM 0:00.20 /usr/local/sbin/httpd -DNOHTTPACCEPT
    www 9176 0.0 0.0 84756 0 ?? IW – 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
    www 9335 0.0 0.0 84756 0 ?? IW – 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
    root 9576 0.0 0.0 7036 952 ?? Is Mon08PM 0:00.01 /usr/local/sbin/sshlockout_pf 15
    root 11186 0.0 0.1 15268 1808 ?? Is Mon08PM 0:00.02 /usr/sbin/sshd
    root 11439 0.0 0.0 7036 764 ?? Is Mon08PM 0:00.01 /usr/local/sbin/sshlockout_pf 15
    root 15182 0.0 0.0 6872 728 ?? Is Mon08PM 0:00.01 dhclient: rl0 [priv] (dhclient)
    root 17466 0.0 0.0 2716 880 ?? S 5:49PM 0:00.00 sleep 55
    root 19606 0.0 0.0 6956 960 ?? Ss Mon08PM 1:33.87 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -f /var/etc/syslog.conf
    root 20031 0.0 0.0 2716 880 ?? SN 5:49PM 0:00.00 sleep 60
    _dhcp 22820 0.0 0.0 6872 744 ?? Is Mon08PM 0:00.02 dhclient: rl0 (dhclient)
    root 29499 0.0 0.0 8984 1004 ?? Is Mon08PM 0:00.02 /usr/sbin/inetd -wW -R 0 -a 127.0.0.1 /var/etc/inetd.conf
    root 37420 0.0 0.0 5780 896 ?? Ss Mon08PM 1:07.67 /usr/local/sbin/apinger -c /var/etc/apinger.conf
    root 37727 0.0 0.0 14384 1028 ?? I Mon08PM 0:01.79 /usr/local/bin/rrdtool –
    root 40230 0.0 0.1 25032 1992 ?? SNs 7:30AM 0:00.49 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_wan.conf -p /var/run/pppoe_wan.pid -s ppp pppoeclient
    root 50900 0.0 0.3 15264 7164 ?? SNs 5:30PM 0:00.07 /usr/local/bin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
    root 52210 0.0 0.1 7928 1476 ?? INs 5:30PM 0:00.00 /usr/sbin/cron -s
    root 53069 0.0 0.1 26268 1308 ?? S Mon08PM 0:12.62 /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf
    root 53259 0.0 0.0 139044 0 ?? IWs – 0:00.00 /usr/local/bin/php
    root 56679 0.0 0.1 147492 2668 ?? I Mon08PM 0:04.46 /usr/local/bin/php
    root 57841 0.0 0.0 139044 224 ?? Is Mon08PM 0:00.08 /usr/local/bin/php
    nobody 60017 0.0 0.1 10100 1360 ?? I Mon08PM 0:15.97 /usr/local/sbin/dnsmasq –all-servers –rebind-localhost-ok –stop-dns-rebind –dns-forward-max=5000 —
    qlproxy 61806 0.0 4.1 114704 84496 ?? INs 5:30PM 0:07.55 /usr/local/sbin/qlproxyd –log info
    dhcpd 63206 0.0 0.0 17104 908 ?? Ss Mon08PM 0:20.13 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcp
    www 64298 0.0 0.0 84756 0 ?? IW – 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
    www 65779 0.0 0.0 84756 0 ?? IW – 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
    www 65950 0.0 0.0 84756 0 ?? IW – 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
    root 68741 0.0 0.4 34288 8984 ?? INs 5:30PM 0:00.00 /usr/pbi/squid-amd64/sbin/squid -f /usr/pbi/squid-amd64/etc/squid/squid.conf
    proxy 69110 0.0 1.7 69104 34252 ?? SN 5:30PM 0:02.60 (squid-1) -f /usr/pbi/squid-amd64/etc/squid/squid.conf (squid)
    root 69223 0.0 0.2 151972 4636 ?? I 5:55PM 0:33.42 /usr/local/bin/php
    proxy 69297 0.0 0.2 15312 3744 ?? IN 5:30PM 0:00.12 (ssl_crtd) -s /var/squid/lib/ssl_db -M 4MB -b 2048 (ssl_crtd)
    proxy 69421 0.0 0.2 15312 3120 ?? IN 5:30PM 0:00.01 (ssl_crtd) -s /var/squid/lib/ssl_db -M 4MB -b 2048 (ssl_crtd)
    proxy 69561 0.0 0.2 15312 3120 ?? IN 5:30PM 0:00.01 (ssl_crtd) -s /var/squid/lib/ssl_db -M 4MB -b 2048 (ssl_crtd)
    proxy 69799 0.0 0.2 15312 3120 ?? IN 5:30PM 0:00.01 (ssl_crtd) -s /var/squid/lib/ssl_db -M 4MB -b 2048 (ssl_crtd)
    proxy 69934 0.0 0.2 15312 3120 ?? IN 5:30PM 0:00.01 (ssl_crtd) -s /var/squid/lib/ssl_db -M 4MB -b 2048 (ssl_crtd)
    proxy 69968 0.0 0.1 11252 1788 ?? IN 5:30PM 0:00.01 (unlinkd) (unlinkd)
    proxy 70038 0.0 0.1 11252 2136 ?? SN 5:30PM 0:00.09 (pinger) (pinger)
    proxy 70286 0.0 0.1 18600 2964 ?? IN 5:31PM 0:00.01 (basic_ncsa_auth) /var/etc/squid.passwd (basic_ncsa_auth)
    root 81871 0.0 0.2 13488 3160 ?? SNs 5:30PM 0:01.63 /usr/local/sbin/openvpn –config /var/etc/openvpn/server1.conf
    root 83786 0.0 0.1 82708 2056 ?? SNs Mon08PM 0:06.81 /usr/local/sbin/httpd -DNOHTTPACCEPT
    qlproxy 83984 0.0 0.1 86864 2332 ?? IN Mon08PM 0:06.14 httpd: (wsgi:diladele.lan) (httpd)
    root 87331 0.0 0.0 5784 0 ?? IWs – 0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
    root 87871 0.0 0.0 5784 252 ?? I Mon08PM 0:00.22 minicron: helper /usr/local/bin/ping_hosts.sh (minicron)
    root 88147 0.0 0.0 5784 0 ?? IWs – 0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /etc/rc.expireaccounts
    root 88565 0.0 0.0 5784 244 ?? I Mon08PM 0:00.02 minicron: helper /etc/rc.expireaccounts (minicron)
    root 88891 0.0 0.0 5784 0 ?? IWs – 0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /etc/rc.update_alias_url_data
    root 89108 0.0 0.0 5784 192 ?? I Mon08PM 0:00.00 minicron: helper /etc/rc.update_alias_url_data (minicron)
    www 91428 0.0 0.0 86804 740 ?? I Mon09PM 0:00.16 /usr/local/sbin/httpd -DNOHTTPACCEPT
    www 91461 0.0 0.0 86804 472 ?? I Mon09PM 0:00.16 /usr/local/sbin/httpd -DNOHTTPACCEPT
    www 91767 0.0 0.0 84756 0 ?? IW – 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
    root 92545 0.0 0.2 26168 3768 ?? Ss 5:49PM 0:00.03 sshd: root@pts/0 (sshd)
    www 96210 0.0 0.0 84756 0 ?? IWN – 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
    www 96277 0.0 0.0 84756 0 ?? IWN – 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
    www 96502 0.0 0.0 84756 0 ?? IWN – 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
    www 96520 0.0 0.0 84756 0 ?? IWN – 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
    www 96597 0.0 0.0 84756 0 ?? IWN – 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
    root 8294 0.0 0.0 19480 0 v0 IWs – 0:00.00 login [pam] (login)
    root 9787 0.0 0.0 8296 0 v0 IW – 0:00.00 -sh (sh)
    root 11155 0.0 0.0 8296 876 v0 I+ Mon08PM 0:00.00 /bin/sh /etc/rc.initial
    root 26032 0.0 0.6 26084 12016 v0- S Mon08PM 0:54.04 /usr/sbin/tcpdump -s 256 -v -S -l -n -e -ttt -i pflog0
    root 26141 0.0 0.0 5780 720 v0- I Mon08PM 0:41.75 logger -t pf -p local0.info
    root 89877 0.0 0.0 8296 544 v0- S Mon08PM 0:03.70 /bin/sh /usr/local/pkg/sqpmon.sh
    root 20068 0.0 0.1 8268 2492 0 S 5:49PM 0:00.01 /bin/tcsh
    root 20671 0.0 0.1 7992 1460 0 R+ 5:49PM 0:00.00 ps aux
    root 93325 0.0 0.1 8296 1508 0 Is 5:49PM 0:00.00 -sh (sh)
    root 93774 0.0 0.1 8296 1532 0 S 5:49PM 0:00.00 /bin/sh /etc/rc.initial

    • sichent says:

      Everything seems to be ok in this output. What makes you think you have high RAM usage? Have you moved to a new version of pfSense compared with what you had before (when no high memory usage conditions were not in place?)

      • cirkit says:

        yes i can see..but the gui says 91%

      • cirkit says:

        no i have not upgraded this system..the guiu immediately showd 20-30% utilisation if i reboot it and then the usage shown increases day by day and finally crashes on the 4th day…this is the third incident repeating every 4 days……it runs pfsense 2.1 amd 64 right from day one

      • sichent says:

        Well in this case the only thing I can recommend it to write to pfsense mailing list/forum. The users there are very helpful and hopefully then will be able to understand when GUI takes this number from…

  44. cirkit says:

    yes I’ll write to them, in the mean time I will build another setup to test the same….I am a hardware engineer and I have atleast 30-35 types of motherboard-cpu combo’s available at time for testing…I will test this with a different motherboard cpu combo…off late THE GIGABYTE GA-C1037UN Celeron dual-core 1.8ghz mini-itx 17w motherboard is available with onboard 2 giga-lan nic’s..tomorrow I will giva Diladele a try on this setup.

  45. cirkit says:

    Hi Sichent
    My new pfsense 2.1 amd64 with squid3-dev 3.3.10, v2.2.6 & qlproxy 3.3.0 is fully functional on the new setup of ga-c1037un with 2gb ddr3 lan and two onboard realtek nic’s, i’ll put this system at the school where the earlier system kept crashing of icap protocol error every 4 days. i’ll keep you updated on the results.
    cirkit

  46. cirkit says:

    Hi sichent,
    I was trying to use the file download blocking feature of diladele, I could block .exe files, .pdf files but i could not block .mp3 files, the expression used was .*/.mp3, the mp3 sites do not directly provide a link to download instead mention ” use right click and save as” to download, under these conditions diladele does not block the download, may be there is another way to achieve this..pl suggest.
    cirkit

  47. cirkit says:

    Also How Do I stop streaming of mp3, mp4, mkv files online. These files are played online rather than downloading.

  48. cirkit says:

    test

  49. cirkit says:

    I am trying to post a link but the site isn’t allowing me to do so, how do I send you the link?

  50. silvio says:

    hi sichent how can i operate dansguardian properly ??? any suggestion ?? thanks sir

  51. Pingback: Tweaking the Home Network – pfSense Firewall [Part 3: Monitoring Your Network] | Rob the Geek

  52. cirkit says:

    All the scripts for integrating qlproxy with pfsense 2.1.4 work correctly.

  53. mattmon says:

    I’m having a problem since this post was updated for pfSense 2.1.5…

    Following the instructions creates a working transparent proxy, however after rebooting, the proxy no longer funtions for unknown reasons.

    qlproxy logs show no problems, squid log shows this error:

    The ssl_crtd helpers are crashing too rapidly, need help!

    I can get rid of the error with the following commands, but proxy still does not intercept traffic.

    rm -rf /var/squid/lib/ssl_db
    cd /usr/pbi/squid-amd64/libexec/squid
    ./ssl_crtd -c -s /var/squid/lib/ssl_db
    chown -R proxy:proxy /var/squid/lib/

    Can anyone help?

    • mattmon says:

      I resolved the problem by:

      uninstall squid3-dev
      install then uninstall squid3
      reinstall squid3-dev

      pfSense + Diladele is far too fragile for production use.

      Unfortunate because it’s caused by pfSense’s squid3-dev package, it’s not the fault of Diladele.

      • sichent says:

        Hello Matt, yes I also got this feeling making changes for each tutorial for pfSense 2.1.3, 2.1.4 and now for 2.1.5. Seems with each release something that was working before broke and later fixed again.

        Well if you have another machine and do not object installing a real proxy – then ubuntu 14 or centos 7 is the way to follow…

  54. cirkit says:

    What exactly has changed in the install scripts / method between pfsense 2.1.4 & 2.1.5, I upgraded two systems from 2.1.3 to 2.1.4 with qlproxy previously installed and everything went smooth….no errors.

  55. Mike says:

    Hi, worked perfectly up until installing the qlproxy, had to download the package manually and install it.

    Also LDAP support within qlproxy settings do not work, is this because of the py27-ldap2 part of the script is missing from the freebsd site? can this be installed seperate? I need this to work with my AD if possible.

    TIA

    • sichent says:

      Hello Mike, what exactly went wrong with qlproxy download? We have updated the script two days ago to match the latest version (3.4.0.9307). Do you have the same version in your downloaded archive?

      Ad for LDAP – were there any errors or just it did not work at all? I mean – read warning when doing LDAP server settings and Active Directory group configuration?

      • Mike says:

        Hi,

        No I downloaded the package manually as the script claimed that no package could be found. I downloaded version 3.3 of olproxy and installed it after the scripts where run, this appears to be working a charm.

        Squid connects to my AD fine, I put the AD settings into qlproxy and the error:

        Failure! Cannot enumerate users in the LDAP server, error No module named ldap.

        when searching for AD usernames or groups, so I cannot add any groups/users from my AD within the policies.

        Looking down you script, you have a section that says:

        # the following command does not work as cc compiler is not present in pfSense, so no LDAP browsing from Web UI will work!
        # someone knows how to overcome this? may be install openldap libraries from FreeBSD somehow???
        #pkg_add -r py27-ldap2

        Would this be the cause?

        I have tried to download the py27-ldap2 package and install it, as per your script, I get a cc error as it is not present. Has a workaround been found for this?

        Thanks in advance

      • sichent says:

        Thanks Mike, now it is clear. Yes you are correct we could not find the workaround for the ldap on pfsense. Almost the same instructions work without a flaw in FreeBSD 8 (which is pfSense is based upon).

        Please share your findings if you manage to find them.

  56. Mike says:

    Hi Sichent,

    Just to update you, I spoke to Diladele support who advised the only way may be to copy the binaries needed from full freebsd to pfsense, because PFSense is a stripped down version, but wasn’t sure what binaries to copy or where to copy them to.

    The second option was a second box with full freebsd and install the proxy/diladele on that to run the filtering.

    Other than the slight niggle with LDAP in pfsense (which is no one’s fault really), this is a really good tutorial. Thanks again.

    Mike

  57. mct says:

    sichent, excellent tutorial, and many thanks.
    Errors:
    fetch: http://updates.diladele.com/qlproxy/binaries/3.4.0.9307/i386/release/freebsd8/qlproxy-3.4.0-i386.tbz: Not Found
    pkg_add: can’t stat package file ‘qlproxy-3.4.0-i386.tbz’
    Not saving default vhosts file
    Performing sanity check on apache22 configuration:
    httpd: Syntax error on line 446 of /usr/local/etc/apache22/httpd.conf: Syntax er ror on line 2 of /usr/local/etc/apache22/extra/httpd-vhosts.conf: Could not open configuration file /usr/local/etc/apache22/extra/qlproxy_virtual_host: No such file or directory

    Ideas?

  58. mct says:

    sichent,
    Thanks. Seems to have installed properly. I am getting hung up on https filtering, but to me that is an issue for a different day.
    However, logs directly from Diladele shows this error, and unknown if related to the git error above.
    —-WARNING!
    Currently installed product version 3.4.0.9307 is out of date (latest is 3.3.0.E807).
    Please visit the Diladele B.V. web site at http://www.quintolabs.com.
    Updated adblock definition files.
    Updated privacy definition files.
    Updated categories definition files.
    Update finished SUCCESSFULLY!

    Again, thank you.
    mct

    • sichent says:

      Hello mct, just ignore this warning for now; the version you have is 3.4 RC and our public servers think the latest is 3.3. After three weeks all public servers will be updated with 3.4 and warning will go away.

  59. Grunge says:

    hi,

    thanks for this guide. im just a newbie on pfsense, i want to ask how can i upload the tbz binary on the pfsense using the winscp? Thanks

    • sichent says:

      Hello Grunge, the scripts attached to article should do the job for you, unfortunately you also need to upload them 🙂 just enable the ssh access to your pfsense box from console, then download WinSCP and connect to your pfSense box’s IP address and upload scripts from archive. Now go to console and run then one by one. If you have never worked with *nixes then please get a book on the subject first 🙂

  60. mct says:

    sichent,
    I took the box down, reinstalled I386 pfsense, logged in, rebooted, installed squid3-dev, rebooted, set proxy to point at the pfsense box, and I am unable to connect to the proxy server.
    Squid settings are loopback, 3128. Ideas?

  61. roux says:

    Hello.
    I just did this and is working 100% for me thanks for the great HOW TO. I just have one problem. I have a office 365 account and it does not accept my Cert, it keeps saying the name on cert is invalid or does not match the name on the site…
    Is there a way can exclude a site like outlook.office365.com from the proxy?

    thanks

  62. I’m about to go through this – but couldn’t this be made into a package?

  63. I think since you have to go into the shell anyway to run the scripts – using fetch http://updates.diladele.com/qlproxy/binaries/3.4.0.9307/amd64/release/freebsd9/qlproxy-3.4.0-amd64.tbz (get exact link from the webpage), might be a better tip than using winscp.

  64. devs says:

    Hi sichent, i ijust installed diladele 3.4 on pfsense 2.1.5.it works fine,

    except “Report upload log of Diladele Web Safety cannot be found, error :[Errno 2] No such file or directory: ‘/opt/qlproxy/var/log/cron_report.log” for this error log when i click “Report Upload Log” on the dashboard/reports.

    Prior clicking, just notice that the “daily statistics is “no data available” ,on overall statistics values on the corresponding “countof” list is 0

    Thanks

    • sichent says:

      Hello devs,
      The report cron job (if you set it up as indicated in tutorial) will run the report script that will generate the log entries and reports.

      • devs says:

        What if i miss it from your tutorial,,what just i did during installing the third script 03_diladele.sh ,i pick some errors fetching the diladele 3.4 installer. And I manually fetch the intaller in the pfsense shell.and continue manually the run command in the 03_diladele.sh in the shell.Thanks

  65. devs says:

    02_apache.sh script has finished successfully, please run script 03_diladele.sh!
    [2.1.3-RELEASE][root@pfsense.cghc]/root(3): sh 03_diladele.sh
    Searching for group qlproxy…
    Group qlproxy already exists.
    Searching for user qlproxy…
    User qlproxy already exists.
    fetch: http://updates.diladele.com/qlproxy/binaries/3.4.0.9307/i386/release/free bsd8/qlproxy-3.4.0-i386.tbz: Not Found
    pkg_add: can’t stat package file ‘qlproxy-3.4.0-i386.tbz’
    default vhosts file is backed up
    Performing sanity check on apache22 configuration:
    httpd: Syntax error on line 446 of /usr/local/etc/apache22/httpd.conf: Syntax er ror on line 2 of /usr/local/etc/apache22/extra/httpd-vhosts.conf: Could not open configuration file /usr/local/etc/apache22/extra/qlproxy_virtual_host: No such file or directory
    03_diladele.sh script has finished successfully!
    Please, reboot your pfSense box and login to Diladele Web Safety’s Web UI at htt p://192.169.1.1:8080!
    [2.1.3-RELEASE][root@pfsense.cghc]/root(4):

    Hi Siichent, installing again the diladele in another pfsense machine as test.This the sample what ii encounter running 03_diladele.sh, Following step by step your guide lead mo to this situation.
    How can i solve this

    Thanks

  66. devs says:

    Many thanks sichent,Hope i could have report in the dashboard.thats the issue im facing while deploying Squid3+pfsense 2.1.5+diladele 3.4. And i wonder if you use TOR browser , in this setup.Can diladele 3.4 intercept https and http in that browser??

  67. cirkit says:

    Hi Sichent, I installed qlproxy-3.4.0.tbz on my pfsense 2.1.4 amd64 after pkg_delete qlproxy-3.3.0, removed and added qlproxy group & user, qlproxy 3.4.0 works fine but nothing is getting updated under reports…no ip’s no domains…Monitoring works fine
    the below is the output of report generation

    Running report generation script script at Mon Oct 20 01:51:01 IST 2014
    2014-10-19 15:21:02,125 Diladele Web Safety Reporter is starting…
    2014-10-19 15:21:02,125 Using database engine django.db.backends.sqlite3
    2014-10-19 15:21:02,126 Using database name /opt/qlproxy/var/console/../db/monitor.sqlite
    2014-10-19 15:21:02,400 Updated total statistics (0 seconds)
    2014-10-19 15:21:02,404 Updated daily statistics (0 seconds)
    2014-10-19 15:21:02,404 Updating total IP statistics…
    2014-10-19 15:21:02,405 Updated total IP statistics (0 seconds)
    2014-10-19 15:21:02,405 Updating total user statistics…
    2014-10-19 15:21:02,406 Updated total user statistics (0 seconds)
    2014-10-19 15:21:02,407 Updating total hosts statistics…
    2014-10-19 15:21:02,408 Updated total host statistics for 0 hosts (0 seconds)
    2014-10-19 15:21:02,408 Updating statistics for each day…
    2014-10-19 15:21:02,409 Updated statistics for 0 days (0 seconds)
    2014-10-19 15:21:02,409 Updating statistics for each IP…
    2014-10-19 15:21:02,410 Updated statistics for 0 user IPs (0 seconds)
    2014-10-19 15:21:02,410 Updating statistics for each user…
    2014-10-19 15:21:02,410 Updated statistics for 0 user names (0 seconds)
    2014-10-19 15:21:02,411 Reports are generated SUCCESSFULLY in 0 seconds!

    • sichent says:

      The monitoring module has some settings to accumulate data before dumping – may be you did not have a lot of blocks – I hope after a while the there will be enough data to generate reports. See Web UI / Monitoring / Settings.

  68. Tim says:

    Hi sichent

    I see that qlproxy 4 is available for download (https://groups.google.com/forum/#!topic/quintolabs-content-security-for-squid-proxy/imiDlnGcKRU)

    I have 3.2.0.4 running on pfsense 2.1.3 but would like to upgrade to the current versions of both. Do you see a problem with me ignoring the upgrade paths from 3.2 to 3.3 to 3.4 to 4.0 and just uninstalling 3.2 and installing 4.0? I’d love to save some time here.

    Thanks

    • sichent says:

      Hello Tim, unfortunately we can support only latest 3.3 to 4.0 upgrade. If you are familiar with Python please take a look at /opt/qlproxy/var/console/upgrade*.py files – this might give a clue how to adjust upgrade instructions for 3.3 to your case.

  69. Tim Haynes says:

    Well, I successfully upgraded pfsense to 2.1.5, removed qlproxy 3.2 as per instructions. I installed qlproxy 4 with the guidance on this page but stopped short of installing qlproxy 3.4 instead issuing the command from here: http://docs.diladele.com/administrator_guide_4_0/installation_and_removal/install_on_freebsd.html#install-diladele-web-safety

    fetch http://packages.diladele.com/qlproxy/4.0.0.303E/amd64/release/freebsd8/qlproxy-4.0.0-amd64.tbz
    pkg_add qlproxy-4.0.0-amd64.tbz

    It seems to have worked, sort of. I can access the new WebUI for Diladele (looks nice!) but when I try to actually get qlproxy and up and running, I get this:

    [2.1.5-RELEASE][root@pfSense.hit.net]/root(8): /opt/qlproxy/bin/restart.sh
    restarting Diladele Web Safety…
    qlproxyd not running? (check /opt/qlproxy/var/run/qlproxyd.pid).
    Starting qlproxyd.
    /opt/qlproxy/bin/qlproxyd: 1: Syntax error: “(” unexpected
    /usr/local/etc/rc.d/qlproxyd.sh: WARNING: failed to start qlproxyd
    cannot restart, error 1
    [2.1.5-RELEASE][root@pfSense.hit.net]/root(9):

    I suspect the binary is not right for my architecture. I’m running on i386, but the fetch/pkg_add was for amd64. I couldn’t successfully guess at what the i386 binary fetch location would be. Is there an i386 version available?

    Tanks

  70. Tim says:

    Perhaps this problem is a permission problem? http://stackoverflow.com/questions/23163098/django-admin-cant-access-admin-backend-attempt-to-write-a-readonly-database seems to indicate so. I am not sure where this DB is, and what user should be owning it, though. Any ideas?

    • sichent says:

      /opt/qlproxy/var/db/qlproxy.sqlite user is qlproxy, group is qlproxy too

      • Tim says:

        Chown’d them all, with no luck. 😦
        [2.1.5-RELEASE][root@pfSense.hit.net]/opt/qlproxy/var/db(7): ls -al
        total 378
        drwxr-xr-x 2 1003 2000 512 Dec 4 06:52 .
        drwxr-xr-x 9 1003 2000 512 Dec 4 06:52 ..
        -rw-r–r– 1 qlproxy qlproxy 81920 Dec 1 13:57 monitor.sqlite
        -rw-r–r– 1 qlproxy qlproxy 230400 Dec 1 13:57 qlproxy.sqlite
        -rw-r–r– 1 qlproxy qlproxy 37888 Dec 1 13:57 report.sqlite

      • sichent says:

        The owner of parent folder is numeric! It means the UID does not correspond to qlproxy. Try chowning from /opt

      • Tim Haynes says:

        Thanks, that worked! Now I can access the Diladele web ui, navigate around it, everything looks good.

        Now, hopefully the last brick in this wall: squid doesn’t seem to be doing anything. Squid is running according to pfSense and Diladele, but when I turn on Transparent proxy (or even use it as an explicit proxy on 3128) it just hangs. I eventually get an error. The access.log is over here if you want to take a peek: http://pastebin.com/9brcHPRe

        Once again, a million thanks! I think we are almost there… 😀

  71. Tim says:

    Another clue. In the Diladele webui, the runtime information for squid is:
    Exit Code: 1
    STDOUT:
    STDERR: client: ERROR: Cannot connect to 127.0.0.1:3128: Operation timed out

    This is true… if I try to telnet to port 3128 on localhost, it doesn’t connect. Odd – all of the GUIs tell me squid processes are running. I cannot even find the squid.conf file to see what it’s set to. There is no squid.conf in the usual locations. The only one I can find in the filesystem is ]/opt/qlproxy/var/console/squid/templates/squid/squid.conf but I suspect that’s not a live config file.

  72. Pavan says:

    Hi Sichent,Can you upload a entire configuration video that will be of great help.

    Thanks
    Pavan

  73. Jay says:

    Sichent.. can you update the diladele script? don’t know what to do next after fetching the diladele..
    🙂

  74. Tomaz says:

    Hi, whn pfsense 2.2?

  75. xersys says:

    Hi there.

    I’m installing qlproxy to pfsense 2.2 but everytime i run the package install it gives me a message /tmp/qlproxy-4.0.0-x86.tbz is not a valid package. Is their a signed qlproxy package that is compatible with pfsense 2.2.

    • sichent says:

      Hello xersys, the 2.2 of pfSense is not supported (it is based on FreeBSD 10 and not on FreeBSD 8 as 2.1). A lot of has changed and our product is not ready for this yet.

  76. Lexus34 says:

    Hi all.
    I’m installing qlproxy-4.1.0 on pfsense 2.1.5-RELEASE (amd64) FreeBSD 8.3-RELEASE-p16

    I receive an error

    #######################################################################

    ImportError at /accounts/login/

    No module named views

    Request Method: GET
    Request URL: http://192.168.0.250:8080/accounts/login/
    Django Version: 1.5
    Exception Type: ImportError
    Exception Value:

    No module named views

    Exception Location: /opt/qlproxy/var/console/www/views.py in , line 18
    Python Executable: /usr/local/bin/python
    Python Version: 2.7.2
    Python Path:

    [‘/opt/qlproxy/var/console’,
    ‘/usr/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg’,
    ‘/usr/local/lib/python2.7/site-packages/pip-1.0.2-py2.7.egg’,
    ‘/usr/local/lib/python27.zip’,
    ‘/usr/local/lib/python2.7’,
    ‘/usr/local/lib/python2.7/plat-freebsd8’,
    ‘/usr/local/lib/python2.7/lib-tk’,
    ‘/usr/local/lib/python2.7/lib-old’,
    ‘/usr/local/lib/python2.7/lib-dynload’,
    ‘/usr/local/lib/python2.7/site-packages’]

    ############################################################

    Grateful for any help.

  77. Carlos de Lima says:

    After running the script, the browser does not access.

  78. Hey! I tried to run the scripts but its not even working, here is the error.
    01_django.sh: pkg_add: not found

    I’ve also fixed the links to the packages in the scripts but did not helped. what should I do??

  79. I am getting an an error when I want to browse Web UI of Diladele.

    ImproperlyConfigured at /accounts/login/
    Creating a ModelForm without either the ‘fields’ attribute or the ‘exclude’ attribute is prohibited; form NetworkForm needs updating.
    Request Method: GET
    Request URL: http://10.1.1.1:8080/accounts/login/
    Django Version: 1.8.4
    Exception Type: ImproperlyConfigured
    Exception Value:
    Creating a ModelForm without either the ‘fields’ attribute or the ‘exclude’ attribute is prohibited; form NetworkForm needs updating.
    Exception Location: /usr/local/lib/python2.7/site-packages/django/forms/models.py in __new__, line 274
    Python Executable: /usr/local/bin/python
    Python Version: 2.7.2
    Python Path:
    [‘/opt/qlproxy/var/console’,
    ‘/usr/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg’,
    ‘/usr/local/lib/python2.7/site-packages/pip-1.0.2-py2.7.egg’,
    ‘/usr/local/lib/python27.zip’,
    ‘/usr/local/lib/python2.7’,
    ‘/usr/local/lib/python2.7/plat-freebsd8’,
    ‘/usr/local/lib/python2.7/lib-tk’,
    ‘/usr/local/lib/python2.7/lib-old’,
    ‘/usr/local/lib/python2.7/lib-dynload’,
    ‘/usr/local/lib/python2.7/site-packages’]
    Server time: Sun, 27 Sep 2015 10:21:33 -0500

  80. shadowwalkers says:

    now I can get to Diladele Web Safety’s Web UI but the login username/password is not working.
    I’ve used the default: root and P@ssw0rd

  81. Armando says:

    hi there.. can this diladele webcont filter works on pfsense machine that running on bridgemode? i need a guide on how to setup this. thks

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s