by Milan Lenčo & Pavel Kotúček | Leave us your feedback on this post!

As part of a webinar, in cooperation with the Linux Foundation Networking, we have created two repositories with examples from our demonstration “Building CNFs with FD.io VPP and Network Service Mesh + VPP Traceability in Cloud-Native Deployments“:

Check out our full-webinar, in cooperation with the Linux Foundation Networking on YouTube:

What is Network Service Mesh (NSM)?

Recently, Network Service Mesh (NSM) has been drawing lots of attention in the area of network function virtualization (NFV). Inspired by Istio, Network Service Mesh maps the concept of a Service Mesh to L2/L3 payloads. It runs on top of (any) CNI and builds additional connections between Kubernetes Pods in the run-time, based on the Network Service definition deployed via CRD.

Unlike Contiv-VPP, for example, NSM is mostly controlled from within applications through the provided SDK. This approach has its pros and cons.

Pros: Gives programmers more control over the interactions between their applications and NSM

Cons: Requires a deeper understanding of the framework to get things right

Another difference is, that NSM intentionally offers only the minimalistic point-to-point connections between pods (or clients and endpoints in their terminology). Everything that can be implemented via CNFs, is left out of the framework. Even things as basic as connecting a service chain with external physical interfaces, or attaching multiple services to a common L2/L3 network, is not supported and instead left to the users (programmers) of NSM to implement.

Integration of NSM with Ligato

At PANTHEON.tech, we see the potential of NSM and decided to tackle the main drawbacks of the framework. For example, we have developed a new plugin for Ligato-based control-plane agents, that allows seamless integration of CNFs with NSM.

Instead of having to use the low-level and imperative NSM SDK, the users (not necessarily software developers) can use the standard northbound (NB) protobuf API, in order to define the connections between their applications and other network services in a declarative form. The plugin then uses NSM SDK behind the scenes to open the connections and creates corresponding interfaces that the CNF is then ready to use.

The CNF components, therefore, do not have to care about how the interfaces were created, whether it was by Contiv, via NSM SDK, or in some other way, and can simply use logical interface names for reference. This approach allows us to decouple the implementation of the network function provided by a CNF from the service networking/chaining that surrounds it.

The plugin for Ligato-NSM integration is shipped both separately, ready for import into existing Ligato-based agents, and also as a part of our NSM-Agent-VPP and NSM-Agent-Linux. The former extends the vanilla Ligato VPP-Agent with the NSM support while the latter also adds NSM support but omits all the VPP-related plugins when only Linux networking needs to be managed.

Furthermore, since most of the common network features are already provided by Ligato VPP-Agent, it is often unnecessary to do any additional programming work whatsoever to develop a new CNF. With the help of the Ligato framework and tools developed at Pantheon, achieving the desired network function is often a matter of defining network configuration in a declarative way inside one or more YAML files deployed as Kubernetes CRD instances. For examples of Ligato-based CNF deployments with NSM networking, please refer to our repository with CNF examples.

Finally, included in the repository is also a controller for K8s CRD defined to allow deploying network configuration for Ligato-based CNFs like any other Kubernetes resource defined inside YAML-formatted files. Usage examples can also be found in the repository with CNF examples.

CNF Chaining using Ligato & NSM (example from LFN Webinar)

In this example, we demonstrate the capabilities of the NSM agent – a control-plane for Cloud-native Network Functions deployed in a Kubernetes cluster. The NSM agent seamlessly integrates the Ligato framework for Linux and VPP network configuration management, together with Network Service Mesh (NSM) for separating the data plane from the control plane connectivity, between containers and external endpoints.

In the presented use-case, we simulate a scenario in which a client from a local network needs to access a web server with a public IP address. The necessary Network Address Translation (NAT) is performed in-between the client and the webserver by the high-performance VPP NAT plugin, deployed as a true CNF (Cloud-Native Network Functions) inside a container. For simplicity, the client is represented by a K8s Pod running image with cURL installed (as opposed to being an external endpoint as it would be in a real-world scenario). For the server-side, the minimalistic TestHTTPServer implemented in VPP is utilized.

In all the three Pods an instance of NSM Agent is run to communicate with the NSM manager via NSM SDK and negotiate additional network connections to connect the pods into a chain client:

Client <-> NAT-CNF <-> web-server (see diagrams below)ne

The agents then use the features of the Ligato framework to further configure Linux and VPP networking around the additional interfaces provided by NSM (e.g. routes, NAT).

The configuration to apply is described declaratively and submitted to NSM agents in a Kubernetes native way through our own Custom Resource called CNFConfiguration. The controller for this CRD (installed by cnf-crd.yaml) simply reflects the content of applied CRD instances into an ETCD datastore from which it is read by NSM agents. For example, the configuration for the NSM agent managing the central NAT CNF can be found in cnf-nat44.yaml.

More information about cloud-native tools and network functions provided by PANTHEON.tech can be found on our website here.

Networking Diagram

Routing Diagram

Steps to recreate the Demo

Clone the following repository.
Create Kubernetes cluster; deploy CNI (network plugin) of your preference
Install Helm version 2 (latest NSM release v0.2.0 does not support Helm v3)
Run helm init to install Tiller and to set up a local configuration for the Helm

Create a service account for Tiller

$ kubectl create serviceaccount --namespace kube-system tiller
$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
$ kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'

Deploy NSM using Helm:

$ helm repo add nsm https://helm.nsm.dev/
$ helm install --set insecure=true nsm/nsm

Deploy ETCD + controller for CRD, both of which will be used together to pass configuration to NSM agents:
```
$ kubectl apply -f cnf-crd.yaml
```
Submit the definition of the network topology for this example to NSM:
```
$ kubectl apply -f network-service.yaml
```
Deploy and start simple VPP-based webserver with NSM-Agent-VPP as control-plane:
```
$ kubectl apply -f webserver.yaml
```
Deploy VPP-based NAT44 CNF with NSM-Agent-VPP as control-plane:
```
$ kubectl apply -f cnf-nat44.yaml
```
Deploy Pod with NSM-Agent-Linux control-plane and curl for testing connection to the webserver through NAT44 CNF:
```
$ kubectl apply -f client.yaml
```

Test connectivity between client and webserver:

$ kubectl exec -it  client curl 80.80.80.80/show/version

To confirm that client’s IP is indeed source NATed (from 192.168.100.10 to 80.80.80.102) before reaching the web server, one can use the VPP packet tracing:

$ kubectl exec -it webserver vppctl trace add memif-input 10
$ kubectl exec -it client curl 80.80.80.80/show/version
$ kubectl exec -it webserver vppctl show trace

00:01:04:655507: memif-input
  memif: hw_if_index 1 next-index 4
    slot: ring 0
00:01:04:655515: ethernet-input
  IP4: 02:fe:68:a6:6b:8c -> 02:fe:b8:e1:c8:ad
00:01:04:655519: ip4-input
  TCP: 80.80.80.100 -> 80.80.80.80
...