Daniel M. Wilcox

QubesOS Web Browser Proxies

2022-06-15

Reading Time: ~14 minute(s)

My sincerest thanks to Dave of d10.dev for a thorough review of this piece and his excellent posts and tools for QubesOS over the years.

Browsing the web is the most dangerous thing you do with a computer.

We use electronic banking with the same browser we use to read the news, doom scroll, and splunk on Wikipedia. What if something happens to our web browser?

A simple solution is to install two web browsers – say Chrome and Firefox – or use two VMs. Chrome is only for banking and Firefox is for everything else. “Fixed” you say.

Two problems:

We don’t trust the web sites we’re visiting entirely, even our bank’s or web mail providers’. They could be hacked and distribute malware and we could end up with a web browser that installs ransomware or mines Bitcoin.
We’re human and we make mistakes. We contaminate our web browser meant only for banking. We click on bad links, ignore security warnings, and put login credentials into questionable web sites.

Qubes can help us with #1 by limiting the blast radius of a web browser being exploited. But help for “being only human”? Yes we can do it! Click wildly, browse freely, and let the computer sort it out.

Demo

What does this look like?

We map our workflows to specific AppVMs and limit them to only the level of netork access they require to do their jobs.

tldr; AppVM networking is enabled but with everything denied and the default gateway provides a filtering proxy (tinyproxy) as the only egress.

Proxy Unaware Traffic Dropped

Without using the proxy traffic simply doesn’t flow.

user@web:~$ wget http://google.com
--2021-06-12 22:42:40--  http://google.com/
Resolving google.com (google.com)... 142.250.191.46, 2607:f8b0:4005:80f::200e
Connecting to google.com (google.com)|142.250.191.46|:80... failed: No route to host.
Connecting to google.com (google.com)|2607:f8b0:4005:80f::200e|:80... failed: Network is unreachable.

user@web:~$ traceroute google.com
traceroute to google.com (142.250.191.46), 30 hops max, 60 byte packets
 1  10.137.0.36 (10.137.0.36)  0.538 ms  0.505 ms  0.492 ms
 2  10.137.0.36 (10.137.0.36)  0.481 ms !X  0.464 ms !X  0.452 ms !X

Note: DNS works – because it is passed by default if networking is enabled at all for an AppVM. But there is no path for other traffic from the AppVM to the Internet without going via the proxy.

Proxy Aware Traffic Filtered

This example filtering proxy is configured to allow access to Github and nothing else.

user@web:~$ http_proxy=10.137.0.36:8888 wget http://github.com -O/dev/null
--2022-03-20 00:20:53--  http://github.com/
Connecting to 10.137.0.36:8888... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/ [following]
--2022-03-20 00:20:54--  https://github.com/
Connecting to 10.137.0.36:8888... connected.
Proxy request sent, awaiting response... 200 OK
<snip saving file>

Github access is allowed. It redirects to the HTTPS version of the site and downloads index.html.

user@web:~$ http_proxy=10.137.0.36:8888 wget http://google.com
--2022-03-20 00:20:32--  http://google.com/
Connecting to 10.137.0.36:8888... connected.
Proxy request sent, awaiting response... 403 Filtered
2022-03-20 00:20:32 ERROR 403: Filtered.

But Google access is not allowed and is returned a ‘Permission Denied’ status code.

ProxyVM Setup

Given the wide use of proxys for web filtering it should be of little surprise they are our best option for creating web browsing compartments. See QubesOS Options for Controlling Egress for paths not taken (or taken and severely regretted).

Note: Qubes atttempts to provide domain filtering functionality via its’ qubes-firewall however in it all domain names (e.g. gmail.com) are immediately resolved and stored as IP addresses. Unfortunately the IP addresses used by many web services change over time and by location. This results in a frustrating web experience as pages get blocked because the firewall rules don’t map to the same IPs returned for the same name at a different time and location. This has been a driving motivation for the solution laid out below.

Create Template for ProxyVMs

Below I lay out how to create a minimal VM in Qubes for use as a ‘ProxyVM’.

sudo qubes-dom0-update qubes-template-debian-11-minimal

This takes a few minutes to download (~200MB) and install and will start and stop the VM to initially configure it (assign static IPs etc).

Start the new VM and specify the root user since minimal VM templates do not come with sudo configured.

qvm-run -u root debian-11-minimal xterm

Note: To paste into an xterm use ‘Shift + Insert’

Update the new template VM:

apt update
apt list --upgradable
apt upgrade
apt autoremove

I like to keep a clean copy of templates so I can always start over.

Always shutdown a template VM to ensure changes are saved to its image before doing operations like cloning or making a new AppVM based on it.

from Dom0:

qvm-shutdown debian-11-minimal
qvm-clone debian-11-minimal debian-11-proxy

This consumes extra disk space and is another template to keep up to date (but only if you use both). Skip it if desired and simply read debian-11-proxy below as debian-11-minimal.

Setup Tinyproxy in TemplateVM

Get a shell in the new ProxyVM template:

qvm-run -u root debian-11-proxy xterm

Install packages required to act as a ProxyVM:

apt install qubes-core-agent-networking nftables tinyproxy

Qubes will disable tinyproxy by default as it normally uses tinyproxy for it’s update proxy (to update template VMs).

# note it is disabled by the file
# /usr/lib/systemd/system/tinyproxy.service.d/30_not_needed_in_qubes_by_default.conf
systemctl status tinyproxy

You can leave that file alone – we can enable it using Qubes Settings on a per VM basis in the Qube Manager.

More important is how we can vary proxy configuration between different proxy VMs. Rather than changing the configuration inside the VM template we only need to change how tinyproxy finds its config when started. And then we can keep per-VM configuration separate from the base proxy image.

We can do this by editing the default environment used by systemd to start tinyproxy. And if we point tinyproxy at a config in /rw every ProxyVM can have different configuration and filtering rules.

In debian-11-proxy edit /etc/default/tinyproxy and add the following line:

FLAGS="-c /rw/config/tinyproxy/tinyproxy.conf"

You can review the tinyproxy systemd config at /lib/systemd/system/tinyproxy.service to see how the defaults file is used.

Configure the TemplateVM as a Qubes Firewall VM

Setting this is required to control firewall rules of attached VMs (the feature is inherited by AppVMs using the template so it should not need to be set on individual ProxyVMs).

qvm-features debian-11-proxy qubes-firewall 1
# confirm
qvm-features debian-11-proxy qubes-firewall

Create ProxyVM in Dom0

Create the first ProxyVM:

qvm-create --template debian-11-proxy --label green proxy-bank
qvm-run -u root proxy-bank xterm

Finish Tinyproxy Setup in ProxyVM

Unfortunately I haven’t found a good way to seed /rw directories on an initial basis from TemplateVMs. So to make tinyproxy work on the new proxy-bank AppVM we must put a config at /rw/config/tinyproxy/tinyproxy.conf.

mkdir /rw/config/tinyproxy
# Run the above to compare the default config with the modified one below
cat /etc/tinyproxy/tinyproxy.conf | egrep -v '^#|^$'
# You can copy your own tinyproxy.conf and modify it but pay special attention
# to copy the Listen, Allow and Filter* lines.

Put this in /rw/config/tinyproxy/tinyproxy.conf:

User tinyproxy
Group tinyproxy
Port 8888
Listen 0.0.0.0
Timeout 600
ErrorFile 403 "/usr/share/tinyproxy/403.html"
DefaultErrorFile "/usr/share/tinyproxy/default.html"
LogFile "/var/log/tinyproxy/tinyproxy.log"
LogLevel Info
PidFile "/run/tinyproxy/tinyproxy.pid"
MaxClients 100
MinSpareServers 5
MaxSpareServers 20
StartServers 10
Allow 10.137.0.0/16
DisableViaHeader Yes
Filter "/rw/config/tinyproxy/filter"
FilterExtended On
FilterDefaultDeny Yes
ConnectPort 443

Note: Be sure and review your Qubes AppVMs IP address range – I have gotten reports of 10.138.0.0/24 in use instead of the above in some configurations.

This is the default config modified to add filtering, use the Qubes IP address range, and to only allow port 443 (HTTPS) for CONNECT (the HTTP verb used to set up TCP socket forwarding, usually for encrypted traffic).

Since we reference the filter file in the tinyproxy config it must also exist. Copy the below lines into /rw/config/tinyproxy/filter:

# Specify domains to allow below using extended regex.
# Mozilla Add-ons
addons.mozilla.org
(.*)\.cdn\.mozilla\.net
# Debian updates
security\.debian\.org
security-cdn\.debian\.org
deb\.qubes-os\.org
deb\.debian\.org
http\.debian\.net
cdn-fastly\.deb\.debian\.org

This allows access to the Mozilla add-ons site to install plugins for Firefox.

Whenever the above filter changes tinyproxy needs a reload – a soft reload may be done by running killall -HUP tinyproxy as root.

Version 1.11 of tinyproxy (in bullseye backports) support extremely soft reloads via SIGUSR1 where existing connections are not interrupted (though not usually a huge deal for a single AppVM).

See Appendix for additional filter file clauses for commonly used sites.

Finish the ‘On-Host’ ProxyVM Setup

With generic tinyproxy configuration is in place. Next we must make it possible for client AppVMs to reach tinyproxy.

Normally VMs in Qubes, even those acting as the network provider for other VMs have a strict firewall in place. This firewall enforces VM isolation by dropping traffic between client VMs and only allows traffic out toward the Internet.

In order to use our ProxyVMs we have to poke a hole in this firewall to allow client VMs to access the proxy. This means modifying the INPUT chain to allow traffic to tinyproxy. Normally the only traffic from an AppVM to a NetVM would be handled on the FORWARD chain.

View the existing INPUT chain on the default (‘filter’) table:

iptables -L INPUT -v -n --line-numbers

Find line, probably 2nd from the bottom, on INPUT chain directly above REJECT target reject-with icmp-host-prohibited message – and use it’s line number (the REJECT will be bumped down) – it is 6 for me.

iptables -I INPUT 6 -p tcp -s 10.137.0.0/16 --dport 8888 -j ACCEPT

Review the INPUT chain to ensure it looks good and make it permanent by adding the same line as above (if different than mine) to /rw/config/qubes-firewall-user-script:

iptables -I INPUT 6 -p tcp -s 10.137.0.0/16 --dport 8888 -j ACCEPT

Ensure the firewall script is enabled with:

chmod a+x /rw/config/qubes-firewall-user-script

Shutdown the new ProxyVM so we can restart it and ensure everything comes up fine without intervention:

shutdown -h now

Finish ProxyVM Setup in Dom0

# Enables proxy to forward network traffic
qvm-prefs --set proxy-bank provides_network True
# confirm it was set
qvm-prefs --get proxy-bank provides_network
# enables tinyproxy to start
qvm-service --enable proxy-bank tinyproxy
# confirm it was set
qvm-service --list proxy-bank

Use Qubes Manager to setup the following firewall rules for the new proxy VM:

Restrict internet to following connections
Allow TCP port 80 to *
Allow TCP port 443 to * Click Apply or OK

Verify the firewall of the ProxyVM in Dom0 by running:

qvm-firewall proxy-bank list

Start-up the ProxyVM for Testing

Start the new ProxyVM to ensure it starts with the new Qubes preferences and services in place.

From Dom0 start a root shell on the ProxyVM:

qvm-run -u root proxy-bank xterm

On the ProxyVM, check tinyproxy is running + that port 8888 is allowed in the INPUT chain:

systemctl status tinyproxy
journalctl -u tinyproxy
iptables -L -n -v

Configuring a Client AppVM

One option is to create a new client VM using the ProxyVM. This creates a new VM called banking from the debian-11 template and uses proxy-bank to provide the network:

qvm-create --template debian-11 --label yellow --property netvm=proxy-bank banking

Another option is to reconfigure an existing VM either in the ‘Qubes Settings’ for that VM or on the command line as follows:

qvm-prefs --set $my_appvm netvm proxy-bank

A VM reconfigured this way will need to be restarted to pick up the changes.

Important

You must configure the new client AppVMs to have networking enabled but all connections denied.

Only traffic sent to tinyproxy can be filtered. Using the FORWARD firewall chain, aka what is configured in Qube Settings, will bypass tinyproxy entirely as the ProxyVM will be acting like a packet router and passing traffic.

Remember, the ProxyVM is not really in control of its’ network forwarding table – the qubes-firewall is in control and the ProxyVM will forward whatever the qubes-firewall has configured.

To ensure your client AppVMs cannot simply go around the ProxyVM by not configuring a proxy, configure the qubes-firewall to deny all traffic for each client AppVM.

To do this open the Qube Settings for the client AppVM and select the Firewall rules tab.
Use the radio button at the top of the tab to select Limit outgoing Internet connections to …
Delete any rules of allowed traffic flows and click Apply in the lower corner.

This keeps networking enabled but no specific connections are allowed. All traffic sent from the client AppVM to the ProxyVM with the intention of being forwarded will be dropped. But traffic addressed directly to our ProxyVM from the client VM is allowed, and is how tinyproxy filtering works.

Using the Proxy

To configure everything to use the ProxyVM we need to get its IP address. We can use that to point our web browser at the proxy, setup our terminal environment to use the proxy by default, and configure our package manager to use the proxy (needed for a StandaloneVM).

Open the Qube Settings for the ProxyVM.
On the right-hand side under the Networking heading is the IP. Write it down somewhere for later.

Configure Firefox

In the client VM setup with the ProxyVM open Firefox ESR
Press the Alt key on your keyboard which will make a tool bar of menus appear across the top of the window
Click the Edit menu and select Preferences
Scroll all the way to the bottom and under the Network Settings section click the button labelled Settings…
Under Configure Proxy Access to the Internet select the radio button corresponding to Manual proxy configuration
Enter your ProxyVM IP address into the input box for HTTP Proxy and check the box to Use this proxy server for all protocols
Click OK to save and close the preferences tab

Configure Your Terminal

Add the following to the bottom of your $HOME/.bashrc or other shell configuration to more easily use the proxy at the command line:

PROXY="http://$(ip route get 8.8.8.8| awk '/via/ {print $3}'):8888"
HTTP_PROXY=$PROXY
HTTPS_PROXY=$PROXY
http_proxy=$PROXY
https_proxy=$PROXY
export HTTP_PROXY HTTPS_PROXY http_proxy https_proxy

Apply the change to your current shell by ‘sourcing’ the file:

source ~/.bashrc

Configure Debian Package Manager (APT)

Note: This primarily useful for a StandaloneVM but not your TemplateVMs. TemplateVMs use a different tinyproxy configuration in Qubes to access update servers.

To use the proxy, create a file in the ‘drop folder’ for apt package manager.

Create /etc/apt/apt.conf.d/80proxy using the below as a template. And replace the IP address 10.137.0.36 below with your ProxyVMs IP:

Acquire::https::proxy "http://10.137.0.36:8888/";
Acquire::http::proxy "http://10.137.0.36:8888/";
Acquire::ftp::proxy "http://10.137.0.36:8888/";

Configure Outbound Ports Not via the Proxy

This can be useful for setting up local mail clients which may need ports for sending and receiving email. Whatever your use case the pattern is the same.

Using the example port of 993 (IMAPS) do the following to enable traffic to flow from your client AppVM through your ProxyVM (and not via tinyproxy).

Open the Qube Settings for your client AppVM and select the Firewall rules tab
Click the green plus (+) button and a New Address form will pop-up
Fill in the IP address or domain name you need to reach, for example mail.google.com. Or if it really has to be any address put a *.
Select the Protocol radio for your protocol. IMAPS is TCP so we will click that
And we can either type 993 into the Port/Service field or select imaps from the drop-down

The above will create the firewall rule in your ProxyVM for your client VM to access the specified email server. However your ProxyVM itself is not allowed to access the email server on port 993 because its NetVM doesn’t allow it.

To permit the ProxyVM to pass the traffic on-ward follow the same steps again but for the ProxyVM this time. By creating the rules in both the ProxyVM and the ProxyVMs’ network provider you can bypass the proxy for unsupported protocols.

Always remember creating firewall rules for a VM creates the rules in the VMs’ network provider and not in the VM itself.

Common Operations - Adding New Sites

workflow of determining site URL
add base URL to filter file in ProxyVM + HUP tinyrproxy
load URL in app VM w/ inspector + network tab enabled
review page for breakage and if broken in unacceptable way determine next asset URL to unblock in ProxyVM
repeat until ProxyVM sufficiently opened to allow site operation

Troubleshooting Tips

tcpdump in the netvm of your ProxyVM determine interface by using ProxyVM IP to find the interface being used by your network VM to talk to it – ip route show (since each VM will get it’s own interface in it’s netvm)
look in the tinyproxy log file: /var/log/tinyproxy/tinyproxy.log you can also adjust the log level in /rw/config/tinyproxy/tinyproxy.conf

References

Appendix - Filter file snippets

Docker

download\.docker\.com
registry-1\.docker\.io
auth\.docker\.io
index\.docker\.io
production\.cloudflare\.docker\.com

Alpine Linux

Often needed for updating Alpine Docker images

dl-cdn\.alpinelinux\.org

Github

github\.com
(.*)\.github\.com
github\.githubassets\.com
objects\.githubusercontent\.com

Python

pypi\.python\.org
pypi\.org
files\.pythonhosted\.org

GoLang

Packages may be hosted anywhere expect to have some issues downloading and be prepared to extend this list.

golang\.org
gopkg\.in
go\.googlesource\.com
code\.googlesource\.com
google\.golang\.org

NodeJS

Packages may be hosted anywhere expect to have some issues downloading and be prepared to extend this list.

deb\.nodesource\.com
dl\.yarnpkg\.com
(.*)\.yarnpkg\.com
registry\.yarnpkg\.com
registry\.npmjs\.org

Anti list

Things I didn’t mean to do but needed to anyway.

browse templates under -itl repos here

https://yum.qubes-os.org/

upgrade vm kernel

sudo qubes-dom0-update --best --allowerasing kernel-qubes-vm

upgrade dom0 kernel

sudo qubes-dom0-update --best --allowerasing kernel

This will normally keep 3 versions for both of these and will remove the oldest version.

clean out old templates

dnf remove qubes-template-fedora-30-minimal

./danielwilcox.uk

Opinions expressed on this site are my own.