Introduction

Managing network traffic in an Kubernetes based platform can quickly become a challenge in terms on manageability. This Blog post documents an approach based on a discussion with Thomas Hikade how to implement a customer requirement keeping the environment manageable.

The idea discussed can be summarized as following:

To manage access to services within the OCP cluster we use NetworkPolicy rules
To manage access to the intranet services (which are outside OCP) we use the EgressFirewall and matching NetworkPolicy configurations. In this document we use the network 135.xxx.xxx.xxx/8 as the intranet network.
To manage access to the internet services we setup EgressIP so that we can use the corporate network infrastructure to manage access to internet services or use the EgressFirewall object as well.

Managing outgoing network traffic on RedHat Openshift v4 using OVNKubernetes CNI

When running workload on Redhat Openshift (OCP) by default all network traffic is allowed. From the security perspective this is a challenge as any application (or intruder) could access other pods and applications with sensitive data or even use the cluster to gain access to the customers intranet.
To manage and limit the allowed network traffic, OCP (as a Kubernetes implementation) provides the NetworkPolicy API. Using the NetworkPolicy API it is possible to secure and limit the network traffic for the workload fine grained and therefore increase the security of the deployment. However there is one drawback using the NetworkPolicy API namely that it can use IP addresses in CIDR format only to configure access to networks / hosts outside of the platform. Unfortunately the NetworkPolicy API does not allow using DNS names to manage access.
While this is at least manageable for intranet access (although it is error prune) it becomes a real issue to control access to internet services which might change IPs without notifications.
Luckily Redhat Openshift provides a way to better manage the access to external services, which allows to use IP addresses in CIDR format as well as DNS names by using the EgressFirewall API.

Note: Redhat Openshift provides currently two network plugins namely the OVN-Kubernetes network plugin and the Openshift SDN network plugin. This article is written based on the OVN-Kubernetes network plugin.

Furthermore in many companies access to external services is anyhow managed via outgoing firewall and/or proxy rules so that it makes sense to make use of the existing rule setup in the existing infrastructure. To ease that approach Redhat Openshift provides the ability to configure an EgressIP for a certain workload based on namespaceSelector or a podSelector. Using the EgressIP all outgoing traffic of a certain workload appears to originate from one configured IP which is helpful if you want to manage access to external networks for example on a corporate firewall.

The test case

In the following steps we’ll setup a test scenario where we use NetworkPolicy API and the EgressFirewall API to protect our environment while we are making use of the existing enterprise infrastructure to manage access to internet services. I.e. we use the NetworkPolicy API to control access within the OCP cluster, we utilize EgressFirewall API to manage the access to the intranet and leave it to the corporate network to manage the access to internet services.

To test the setup we are running our workload in three namespaces on the OCP, namely client, noaccess and server. In the server and noaccess namespace we are running an HTTP service, which is accessed by the client . Furthermore the client is accessing an intranet HTTP server in the 135.0.0.0/8 network as well as an internet service like for example redhat.com.

Setting up the test case

The test case requires to setup three namespaces on OCP with some basic applications deployed and to setup an HTTP server on the intranet.

Setup OCP namespaces

Lets first create the OCP namespaces for the client and server workload and add a label with the required annotation to enable NetworkPolicy audit logging:

1

for name in client server noaccess; do oc delete ns ${name} || true ; oc new-project ${name} ; oc label ns ${name} namespace=${name} ; oc annotate namespaces ${name} k8s.ovn.org/acl-logging='{ "deny": "alert", "allow": "alert" }'; done

Annotate the worker nodes

To enable the EgressIP we need to annotate at least one node with the label k8s.ovn.org/egress-assignable="". We’ve labeled all the worker nodes by running:

1

oc get nodes | grep worker | awk '{print $1}' | while read line ; do  oc label node $line k8s.ovn.org/egress-assignable="" ; done

Deploy NGINX for testing to the server and noaccess namespace

To simulate server workload on the OCP we are simply deploying an nginx server in the server and noaccess namespace which will be accessed via curl from the client namespace:

1
2


oc run web --namespace=server --image=nginx --labels="app=web" --expose --port=80
oc run web --namespace=noaccess --image=nginx --labels="app=web" --expose --port=80

and verify that the nginx application started up correctly and is in ready state by running:

1
2


oc -n server get all
oc -n noaccess get all

Deploy rhel-tools pod to test access to services

To test access to the resources on the OCP platform and to internet as well as intranet services we deploy the rhel-tools pod to the client namespace.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


cat <<EOF| oc apply -n client -f -
apiVersion: v1
kind: Pod
metadata:
  name: client
spec:
  containers:
    - name: client
      image: registry.access.redhat.com/rhel7/rhel-tools
      command: ["/bin/sh", "-c"]
      args:
        ["sleep inf"]
EOF

and verify that the pod is started properly and in ready state by running:

1

oc -n client get all

Once the pod is ready we are good to test the access to the nginx server pod. So first we start a bash to the client pod:

1

oc -n client exec -it pod/client -- bash

and then use curl to access the nginx server in the server namespace:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


[root@client /]# curl http://web.server
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

so all good and we are able to access the nginx server running in the server namespace.

Note: Repeat the above test with the nginx server running in the noaccess namespace by using the URL http://web.noaccess

Setup the HTTP server on the intranet

To verify that the correct EgressIP is used we are setting up an Apache HTTP Server on an intranet server and log the client IP in the Apache’s access log file by using the combined log-format (which was the default in our case).

Install Apache HTTP server on intranet server

To start an HTTP server we just install the respective HTTPD package using yum:

1

yum -y install httpd.x86_64

Verify the HTTP server configuration

In our case we’ve had to reconfigure the HTTP server to listen on a non-default port as the default HTTP server port 80 has been in use.

1
2
3
4
5


[root@host conf]# diff httpd.conf httpd.conf.ori
45c45
< Listen 5822
---
> Listen 80

This also required to open the firewalld to allow server access via port 5822 for the network being used to access the HTTP server from the OCP workload:

1
2


firewall-cmd --zone=libvirt --add-port=5822/tcp --permanent
firewall-cmd --zone=libvirt --add-port=5822/tcp

Setup a simple HTTP server index.html

To verify that we hit the correct HTTP server we setup the following index.html file:

1
2
3
4
5
6


cat > /var/www/html/index.html << EOF
<html>
<head/>
<body>2innovate.at test HTTP Server</^Cdy>
</html>
EOF

and then start the the HTTP server simply by running:

1

apachectl start

Verify that the HTTP server is running by checking that the port 5822 is in LISTEN state:

1
2


[root@host conf]# netstat -tan | grep 5822
tcp6       0      0 :::5822                 :::*                    LISTEN     

Now we should be good to test the access to the HTTP server from our client pod:

1
2
3
4
5


[root@client /]# curl http://135.xxx.xxx.xxx:5822/
<html>
<head/>
<body>2innovate.at test HTTP Server</^Cdy>
</html>

Checking the HTTP access log of the server we can see that the source IP is logged as the pod’s node IP running the pod:

1
2


[root@host logs]# tail -f /etc/httpd/logs/access_log
192.168.50.14 - - [11/Sep/2023:13:48:12 +0200] "GET / HTTP/1.1" 200 66 "-" "curl/7.29.0"

Checking the node hosting the pod and the node’s IP confirms what we expect:

1
2
3
4
5
6


[hhuebler@host tmp]$ oc -n client get pods -o wide
NAME     READY   STATUS    RESTARTS   AGE    IP            NODE        NOMINATED NODE   READINESS GATES
client   1/1     Running   0          104m   10.131.0.36   compute-1   <none>           <none>
[hhuebler@host tmp]$ oc get nodes -o wide | grep worker
NAME        STATUS   ROLES    AGE    VERSION            INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                                                        KERNEL-VERSION                 CONTAINER-RUNTIME
compute-1   Ready    worker   548d   v1.25.11+1485cc9   192.168.50.14   <none>        Red Hat Enterprise Linux CoreOS 412.86.202307251341-0 (Ootpa)   4.18.0-372.64.1.el8_6.x86_64   cri-o://1.25.3-5.rhaos4.12.git44a2cb2.el8

Setting up the EgressIP for the client namespace to be able to manage outgoing traffic

Before setting up an EgressIP we need to make sure that at least one node is labeled with the label k8s.ovn.org/egress-assignable: "".

Before assigning an EgressIP to a namespace and/or pods we need to make sure that the IP is not in use and that it is from the node’s IP network. The author recommends to reserve a separate range of IP addresses for the EgressIP addresses and omit that range from the DHCP pool.

YAML to create EgressIP address

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


cat <<EOF | oc -n client apply -f - 
apiVersion: k8s.ovn.org/v1
kind: EgressIP
metadata:
  name: client-egress
spec:
  egressIPs:
  - 192.168.50.100
  namespaceSelector:
    matchLabels:
      namespace: client
EOF

Verify that the EgressIP has been created and assigned correctly (don’t worry that the EgressIP is on a different node as the pod - OCP will manage that for you):

1
2
3


[hhuebler@host ~]$ oc -n client get egressip
NAME            EGRESSIPS        ASSIGNED NODE   ASSIGNED EGRESSIPS
client-egress   192.168.50.100   compute-0       192.168.50.100

As the EgressIP has been assigned correctly we should now see that the 192.168.50.100 is logged in the HTTP servers access log of we access the server from our client again:

1
2
3
4
5
6


[root@client /]# date && curl http://135.xxx.xxx.xxx:5822/
Mon Sep 11 12:02:27 UTC 2023
<html>
<head/>
<body>2innovate.at test HTTP Server</^Cdy>
</html>

and checking the HTTP server access log we now confirms that the EgressIP is used for the outgoing traffic:

1

192.168.50.100 - - [11/Sep/2023:14:02:50 +0200] "GET / HTTP/1.1" 200 66 "-" "curl/7.29.0"

So this proves that the EgressIP is indeed used to access the server. This setup can therefore be used to use the corporate network infrastructure to manage access to the internet services from our workload running on OCP as we know the originator IP address for the requests from a specific workload.

Using NetworkPolicy rules to limit access to other services

Using the NetworkPolicy API we can manage the access to services within the current namespace, to other namespaces and to certain IP addresses. The downside however is as mentioned above that we can only use IP addresses in the CIDR format to configure allowed target IPs.

Therefore as mentioned above we will split the config up in the following areas:

To manage access to services withing the OCP cluster we use NetworkPolicy rules
To manage access to the intranet services (which are outside OCP) we use the EgressFirewall configurations in combination with NetworkPolicy rules
To manage access to the internet services we setup EgressIP so that we can use the corporate network infrastructure to manage access to internet services. If we don’t have an existing network infrastructure which allows to manage outgoing traffic we can use EgressFirewall configurations as well.

To setup this strategy we start with the following NetworkPolicy object:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


cat <<EOF | oc -n client apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-except-internal-and-intranet
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress: []
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              # clusterNetwork (oc get networks.config.openshift.io cluster -o yaml)
              - 10.128.0.0/14
              # serviceNetwork (oc get networks.config.openshift.io cluster -o yaml)
              - 172.30.0.0/16
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: "allow-to-openshift-dns"
spec:
  egress:
    - to:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: openshift-dns

  podSelector: {}
  policyTypes:
    - Egress
EOF

Note: We need to add the rule allow-to-openshift-dns to be able to access sites via names and not IPs only.

From this NetworkPolicy object we allow all traffic at the NetworkPolicy level except:

OCP internal networks (oc get networks.config.openshift.io cluster -o yaml):
- clusterNetwork (10.128.0.0/14)
- serviceNetwork (172.30.0.0/16) Note: We need to list the clusterNetwork as well as the serviceNetwork in the except-list, as otherwise all internal OCP traffic would be allowed and NetworkPolicy limitations would not apply.

This allows us to manage all traffic outside the OCP cluster either:

at the corporate network level with the help of the EgressIP
or manage this traffic using an EgressFirewall object. We need to consider that different OCP workload might need access to different services on the intranet / internet.

As soon as we apply this NetworkPolicy we’ll see that none of the curls we previously ran successfully works anymore which is what we expect.

Run some tests now from the running pod in the client namespace like for example:

curl http://web.noaccess
curl http://web.server

and all these curl commands should hang at this stage of the configuration.

However the following URLs should work:

curl -I -L https://redhat.com
curl http://135.xxx.xxx.xxx:5822/

as this are intranet and internet URL which is allowed as the IP address is not in the except rule and hence allowed. As we don’t have any corporate setup to block this access it succeeds.

Allow access to the workload in the server namespace

To allow access to the OCP internal services we need to setup the usual NetworkPolicy rule to allow that access. Now let’s create this rule:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


cat <<EOF | oc -n client apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-to-server-namespace
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
      - podSelector: {}
  egress:
    - to:
      - namespaceSelector:
          matchLabels:
            namespace: server
EOF

Once we created this rule the following URL should work again: curl http://web.server however the following curl curl http://web.noaccess should still hang.

Control the access to intranet and / or internet services

To manage the access to the intranet and / or internet (in case we don’t have a corporate firewall managing the access to the internet) URLs - we need to setup an EgressFirewall rule (which must be named default). Setting up an EgressFirewall rule allows us to allow or deny access to specific IPs or DNSNames. Right, using an EgressFirewall allows us to specify a cidrSelector or a dnsName with either an Allow or Deny specification.

So to allow access to http://135.xxx.xxx.xxx:5822/ we need to create an EgressFirewall rule allowing that access like the following (we’ll discuss the the other rules in more detail below):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


cat <<EOF | oc -n client apply -f -
apiVersion: k8s.ovn.org/v1
kind: EgressFirewall
metadata:
  name: default
spec:
  egress:
  - type: Allow 
    to:
      dnsName: www.orf.at
  - type: Allow 
    to:
      dnsName: 2innovate.at
    ports:
    - port: 80
      protocol: TCP
    - port: 443
      protocol: TCP
  - type: Allow
    to:
      cidrSelector: 135.xxx.xxx.xxx/32
  - type: Deny 
    to:
      dnsName: web.server
  - type: Deny
    to:
      cidrSelector: 0.0.0.0/0
EOF

Once you’ve created this EgressFirewall named default like above you should be able to access curl http://135.xxx.xxx.xxx:5822 again.

What if we have the network `135.xxx.xxx.xxx/8` in the except list of the `NetworkPolicy` manifest?

In case the NetworkPolicy named allow-all-except-internal-and-intranet has the network 135.xxx.xxx.xxx/8 in its except list - for whatever reasons - we need to allow the access to this network via a NetworkPolicy definition as well.

Only in case you have the network 135.xxx.xxx.xxx/8 in the except list of the NetworkPolicy manifest named allow-all-except-internal-and-intranet we need to add a NetworkPolicy to allow access to this IP or network. Let’s create the following rule.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


cat <<EOF | oc -n client apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: "allow-to-ip-135.xxx.xxx.xxx"
spec:
  egress:
    - to:
      - ipBlock:
          cidr: 135.xxx.xxx.xxx/32
  podSelector: {}
  policyTypes:
    - Egress
EOF

Note: This discussion shows us that the NetworkPolicy definitions have priority over the EgressFirewall. I.e. what is not allowed at the NetworkPolicy can’t be allowed at the EgressFirewall level.

As soon as we create the NetworkPolicy allow-to-ip-135.xxx.xxx.xxx we can again access the site at curl http://135.xxx.xxx.xxx:5822 as access is allowed at the NetworkPolicy and the EgressFirewall level.

Some additional notes on the EgressFirewall/default

In this EgressFirewall rule named default we have some more additional rules we should comment on:

EgressFirewall rule - Deny all

The rule list entry:

1
2
3


  - type: Deny
    to:
      cidrSelector: 0.0.0.0/0

denies access to all servers which are not explicitly allowed by another list entry. As we don’t have the site redhat.com in the allow list access to https://redhat.com does not work anymore.
Remember - we could access https://redhat.com before we created the EgressFirewall/default rule.

This relevant if we don’t have an external network infrastructure to manage access to external services and we need to do that from the OCP environment.

EgressFirewall rule for OCP workload

The following list entry in the egress list:

1
2
3


  - type: Deny 
    to:
      dnsName: web.server

is useless and does not make sense as this addresses an OCP internal service which is not accessed via the Egress. Hence it should be removed. This rule was added for demonstration purposes only.

EgressFirewall rule limiting allowed ports

The following rule grants access to the host 2innovate.at but only via the ports 80 and 443:

1
2
3
4
5
6
7
8


  - type: Allow 
    to:
      dnsName: 2innovate.at
    ports:
    - port: 80
      protocol: TCP
    - port: 443
      protocol: TCP

EgressFirewall rule to an internet host without limiting allowed ports

The following rule allows access to the host www.orf.at without any limitations regarding the ports being accessed:

1
2
3


  - type: Allow 
    to:
      dnsName: www.orf.at

Summary

In this Blog post we’ve discussed a possible setup to secure network access on the Redhat Openshift platform using NetworkPolicy, EgressFirewall and EgressIP objects. We have shown that we need NetworkPolicy objects to manage access to services within the OCP platform.

To manage access to the internet or intranet services we can use EgressFirewall objects or rely on existing corporate network infrastructure. So our network protection looks like as follows:

Approach to manage outgoing traffic on Openshift v4 OVNKubernetes network plugin