Firewall rules using VPC as target should allow/deny traffic based on private IP, not external IP

[askfongjojo]

While investigating oxidecomputer/omicron#3373, I've dug in a bit more on just the default firewall rule behavior (w/o making any changes to the VPC fw rules). This is the opte firewall setup for a brand new project and default VPC (and no change to the fw rules):

BRM44220011 # /opt/oxide/opte/bin/opteadm list-layers -p opte6
NAME         RULES IN   RULES OUT  DEF IN   DEF OUT  FLOWS     
gateway      1          5          deny     deny     0         
firewall     3          0          deny     stateful allow 0         
router       0          2          allow    deny     0         
nat          1          2          allow    allow    0         
overlay      1          1          deny     deny     0  

BRM44220011 # /opt/oxide/opte/bin/opteadm dump-layer firewall -p opte6
Layer firewall
======================================================================
Inbound Flows
----------------------------------------------------------------------
PROTO  SRC IP           SPORT  DST IP           DPORT  HITS     ACTION                
TCP    172.20.17.42     64834  172.30.0.5       5201   0        no-op                 

Outbound Flows
----------------------------------------------------------------------
PROTO  SRC IP           SPORT  DST IP           DPORT  HITS     ACTION                
TCP    172.30.0.5       5201   172.20.17.42     64834  1        no-op                 

Inbound Rules
----------------------------------------------------------------------
ID     PRI    HITS   PREDICATES                             ACTION            
2      65534  1      inner.ip.proto=TCP                     "Stateful Allow"
                     inner.ulp.dst=22                      

1      65534  1      meta: vni=7524168                      "Stateful Allow"
0      65534  0      inner.ip.proto=ICMP                    "Stateful Allow"
DEF    --     0      --                                     "deny"

Outbound Rules
----------------------------------------------------------------------
ID     PRI    HITS   PREDICATES                             ACTION            
DEF    --     47     --                                     "stateful allow

I created an instance in it and ran an iperf3 server on it.

ubuntu@default-fwrules:~$ iperf3 -s -D
ubuntu@default-fwrules:~$ exit
logout
Connection to 172.20.26.45 closed.
pisces-2:docs angela$ nc -vz 172.20.26.45 5201
Connection to 172.20.26.45 port 5201 [tcp/targus-getdata1] succeeded!

Based on the default rules, the only port accessible on tcp should be 22. So it'd look like the deny-all default is not taking effect.

Next, I tried disabling the allow-ssh rule and saw that the change was reflected in the opte port:

BRM44220011 # /opt/oxide/opte/bin/opteadm dump-layer firewall -p opte6
Layer firewall
======================================================================
Inbound Flows
----------------------------------------------------------------------
PROTO  SRC IP           SPORT  DST IP           DPORT  HITS     ACTION                
TCP    172.20.17.42     64897  172.30.0.5       22     0        no-op                 

Outbound Flows
----------------------------------------------------------------------
PROTO  SRC IP           SPORT  DST IP           DPORT  HITS     ACTION                
TCP    172.30.0.5       22     172.20.17.42     64897  1        no-op                 

Inbound Rules
----------------------------------------------------------------------
ID     PRI    HITS   PREDICATES                             ACTION            
4      65534  1      meta: vni=7524168                      "Stateful Allow"
3      65534  0      inner.ip.proto=ICMP                    "Stateful Allow"
DEF    --     0      --                                     "deny"

Outbound Rules
----------------------------------------------------------------------
ID     PRI    HITS   PREDICATES                             ACTION            
DEF    --     50     --                                     "stateful allow"

Afterwards, I was still able to SSH to the vm. This further confirms that there is no firewall at all.

[askfongjojo]

In case caching plays a part in this problem, it'll help there are commands I can run to invalidate the firewall rule cache. This way we can better isolate the problem. @bnaecker / @luqmana - do you have any suggestions for the next step here?

[bnaecker]

If you mean the Unified Flow Table, then that can be done with opteadm clear-uft.

[askfongjojo]

After digging in a bit more, I think the issue here isn't that there was no firewall in effect. If I disable the "allow-internal-inbound" rule that is meant to allow all traffic within the same VPC, I am no longer allowed to tcp-connect to ports from my workstation, except for port 22.

It seems that the external IP range is inadvertently filtered in the same way as internal IP range. In other words, the allow-internal-inbound which has VPC as target and source (host filter) allows all traffic between IPs on the same external subnet as well. Since my workstation is on a 172.20.x.x IP address, it's considered being on the same subnet as the VM's 172.20.26.x external IP.

The only way I can actually achieve "allow-internal-inbound" is to create a rule that uses a specific subnet as the source and target, resulting in rules that look like these in opte:

Inbound Rules
----------------------------------------------------------------------
ID     PRI    HITS   PREDICATES                             ACTION            
173    0      0      inner.ip.proto=UDP                     "Stateful Allow"
                     inner.ip6.src=fdaf:4243:35d6::/64     

172    0      0      inner.ip.proto=UDP                     "Stateful Allow"
                     inner.ip.src=172.30.0.0/22            

171    0      0      inner.ip.proto=TCP                     "Stateful Allow"
                     inner.ip6.src=fdaf:4243:35d6::/64     

170    0      0      inner.ip.proto=TCP                     "Stateful Allow"
                     inner.ip.src=172.30.0.0/22            

169    65534  0      inner.ip.proto=ICMP                    "Stateful Allow"
DEF    --     140    --                                     "deny"

I am not sure if the vpc source/target should filter based on the external IP address. If user needs to restrict access to VMs' external IP, they probably should do so by specifying specific subnet CIDR or IP addresses as the target. (The fact that we have a default rule named "allow-internal-inbound" seems to suggest that it's for allowing traffic on private IP only.)

[rcgoodfellow]

It seems like we need to be able to discern between boundary services ingress traffic and internal VPC traffic. Because these are both on the same Geneve VNI we need additional information. At first glance, there are two options.

1. Use Geneve TLVs. Every ingress packet that gets NAT'd by boundary services would also have a TLV added to the packet indicating this is from an external source.
2. Use the source address of the underlay packet. This address is not a normal underlay address in the RFD 63 sense. It has its own (for better or worse) addressing scheme that is currently just an ad-hoc fd00:99::1/64 address. When we get to multi-switch this will shift to an anycast address or a pair of addresses on different /64s. But at any rate, it will be distinguishable from sled underlay addresses.

Option 1 is where we ultimately want to be. If complications arise and other critical path work takes precedence option 2 may be an ok path to take.