Posts

[PyConSK19] Automated visualization and presentation of test results

PANTHEON.tech’s developer Tibor Frank recently held a presentation at PyConSK19 in Bratislava. Here is his full-length presentation and some notes on top of it.

Motivation

Patterns, trends, and correlations that might remain undetected in text-based data can be exposed and recognized easier with data visualization.

Why do we need to visualize data? Because people like pictures and they hate piles of numbers. When we want or need to provide data to our audience, the best way how to communicate it is to display it. But we must display it in a form and structure our intended audience can consume. They must process the information in a fraction of second and in the correct way.

A real-world example

When I was building my house, a sales representative approached me and proposed, that he has the best thermal isolation for houses on the market.

He told me that it is 10% better than the second one and showed me this picture. Not a graph, a picture, as this is only a picture with one blue rectangle and two lines. From this point, I wasn’t interested in his product anymore. But I was curious about the price of this magic material. He told me that it is only 30% more expensive than the second one. Then I asked him to leave.

Of course, I was curious so I did some research later in the evening. I found the information about both materials and I visualized it with a few clicks in Excel.

What’s the difference between these two visualizations? The first one is good only as an illustration in marketing materials.

From a technician point of view, I am missing information about what the graph is trying to show me:

  • Title
  • Axes with titles
  • Numbers
  • Physical quantities
  • Legend

The same things which we were told to use at elementary school. It’s simple, isn’t it?

To be honest, this graph confirmed that his product is 10% better, but still, 30% more expensive.

Attributes of a good design

When we are talking about visualization we must also talk about design. In general, there are four attributes of a good design. It must be:

  • Beautiful –  because people must find pleasure in it
  • Gratifying – to enjoy it
  • Logical –  everything should be in the right place; it must be self-descriptive, no need for further explanation
  • Functional – the interactions between components must work and it must fit the bigger picture

Besides these attributes, we should choose the right point of view. Look at these two pictures. Which one do you like the best?

Do we want to show details? Or a bigger picture? Or both?

We need to decide before we start according to the nature of data we visualize, the information we want to communicate and the intended audience.

We must ask ourselves who will be the customer for our visualized data.

  • Management
  • Marketing
  • Customers
  • Skilled professionals
  • Anyone

We should keep it in mind while preparing the visuals.

What do we visualize?

Data. Lots of data. 30GB of data, twice a day. Our data is the result of performance tests. I am working on an FD.io project which performs Continuous System and Integration Testing (CSIT) of the Vector Packet Processor (VPP). As I said, we focus on performance. The tests measure the packet throughput and packet latency in plenty of configurations of the VPP. For more information, visit FD.io’s website. 

The Fast Data Project So, what do we visualize?

1. Performance test results for the product release:

  • Packet throughput
  • Packet Latency
  • Speedup multi-core throughput – increase in performance if we increase the number of processor cores used for data processing

2.Performance over a defined time period:

  • Packet throughput trend

Where does the data come from?

FD.io CSIT hierarchy (https://docs.fd.io/)

Jenkins runs thousands of tests on our test-beds and provides the results as robot frameworks’ output.xml files. The files are processed and the data from them visualized. As a final step we use Sphinx to generate html pages which are then published on the web.

What we use

Nothing surprising, we use standard python tools:

  • Python 2.7 3.6
  • Numpy
  • Pandas
  • Plot.ly
  • Sphinx

I will not bother you with these tools, you might know them better than me.

Plots

We use Plot.ly in offline mode to generate various kinds of plots. Hundreds of dynamically generated plots are then put together by Sphinx to create the release Report or the Continuous Performance Trending.

I’ll start with the plots published in the Report. In general, we run all performance tests ten times to prevent any anomalies in the results. We calculate the mean value and standard deviation for all tests and then work with them.

Packet throughput – Statistical box plot

The elementary information about the performance is visualized by the statistical box plot. This kind of plot provides us all information about statistical data – minimum, first quartile, median, third quartile, maximum and outliers. This information is displayed in the hover box.

As you can see, the X-axis lists indices of individual test suites as listed in Graph Legend; and the Y-axis presents measured Packet throughput values [Mpps]. Both axes start with zero value so we know the scale. The tests in each plot are grouped and ordered by the chosen criteria. This grouping is written in the plot title (area, topology, processor architecture, NIC, frame size, number of cores, test type, measured property).

From this graph we can also see the variability of results – the results in the first graph are in all runs almost the same (there are lines instead of boxes), the results in the second one vary in a quite big range. It says that the reliability of these results are lower than in the first case and there might be an issue in the tested functionality, tests or infrastructure & more.

Packet Latency – Scatter plot with error bars

When we measure the packet latency, we get minimal, average and maximal values in both directions of data flows. The best way we found to visualize it, is the scatter plot with error bars.

The dot represents the average value and the error bars the minimum and maximum values.

The rest of the graph is similar to the previous one.

Speedup – Scatter plot with annotations

Speedup is the increase of the packet throughput if we increase the number of processor cores used for data processing. We measure the throughput using 1, 2 and 4 cores.

Again, we use the Scatter plot. The measured values are represented by dots connected by solid lines. In the ideal case, it would be a straight line. But it is not. So we added dashed lines to show how it would be in the ideal world. In real life, there are limitations not only in software but also in hardware. They are shown as dotted lines – Link, NIC, and PCIe bus limits. These limits cannot be overcome by the software.

3D Data

Some of our visualizations present three-dimensional data, in this example, the packet throughput is measured in a configuration with two changing parameters.

The easiest way to visualize it is to use excel and by a few clicks, we can create a graph like this one (above). It looks quite good but:

  • The orientation in the graph is not so easy. What if there were 20 samples in a row?
  • It is difficult to read the results
  • In some cases, which are very probable, some bars can be hidden behind another one
  • And Plot.ly does NOT support this kind of graph because, as they say, it is not needed. And they are right

So we had to look for a better solution. And we found the heat-map. It presents three-dimensional data in two-dimensional space. It is easy to process all information at one quick sight. We can quickly find any anomalies in this pattern as we expect the highest value to be in the top left corner and decreasing to the right and down.

Packet throughput trending – Scatter plot

 

The trending graphs show us the trend of packet throughput over the chosen time period which is 90 days in this case. The data points (again average value of 10 samples) are displayed as dots. The trend lines are calculated by JumpAvg algorithm developed by our colleague. It is based on the minimum description length (MDL) principle.

What is important in the visualization of a trend, are changes in the trend. We mark them by color circles: red for regression and green for progression. These changes in trend can be easily spotted by testers and/or developers so we immediately know the effect of merged changes on the product performance.

Tibor Frank

Vector Packet Processing 101: VPP Plugins & Binary API

In the first part of our new series, we will be building our first VPP platform plug-in, using basic examples. We will start with a first-dive into plugin creation and finish with introducing VAPI into this configuration.

If you do not know what VPP is, please visit our introductory post regarding VPP and why you should consider using it.

Table of contents:

  • How to write a new VPP Plugin
    • 1. Preparing your new VPP plugin
    • 2. Building & running your new plugin
  • How to create new API messages
  • How to call the binary API
    • Additional C/C++ Examples

How to write a new VPP Plugin

The principle of VPP is, that you can plugin a new graph node, adapt it to your networks purposes and run it right off the bat. Including a new plugin does not mean, you need to change your core-code with each new addition. Plugins can be either included in the processing graph, or they can be built outside the source tree and become an individual component in your build.

Furthermore, this separation of plugins makes crashes a matter of a simple process restart, which does not require your whole build to be restarted because of one plugin failure.

1. Preparing your new VPP plugin

The easiest way how to create a new plugin that integrates with VPP, is to reuse the sample code at “src/examples/sample-plugin”. The sample code implements a trivial “macswap” algorithm that demonstrates the plugins run-time integration with the VPP graph hierarchy, API and CLI.

  • To create a new plugin based on the sample plugin, copy and rename the sample plugin directory

cp -r src/examples/sample-plugin/sample src/plugins/newplugin

#replace 'sample' with 'newplugin'. as always, take extra care with sed!
cd src/plugins/newplugin
fgrep -il "SAMPLE" * | xargs sed -i.bak 's/SAMPLE/NEWPLUGIN/g'
fgrep -il "sample" * | xargs sed -i.bak 's/sample/newplugin/g'
rm *.bak*
rename 's/sample/newplugin/g' *

There are the are following files:

    • node.c – implements functionality of this graph node (swap source and destination address) -update according to your requirements.
    • newplugin.api – defines plugin’s API, see below
    • newplugin.c, newplugin_test.c – implements plugin functionality, API handlers, etc.
  • Update CMakeLists.txt in newplugin directory to reflect your requirements:
add_vpp_plugin(newplugin
  SOURCES
  node.c
  newplugin.c

  MULTIARCH_SOURCES
  node.c

  API_FILES
  newplugin.api

  API_TEST_SOURCES
  newplugin_test.c

  COMPONENT vpp-plugin-newplugin
)
  • Update sample.c to hook your plugin into the VPP graph properly:
VNET_FEATURE_INIT (newplugin, static) = 
{
 .arc_name = "device-input",
 .node_name = "newplugin",
 .runs_before = VNET_FEATURES ("ethernet-input"),
};
  • Update newplugin.api to define your API requests/replies. For more details see “API message creation” below.
  • Update node.c to do required actions on input frames, such as handling incoming packets and more

2. Building & running your new plugin

  • Build vpp and your plugin. New plugins will be built and integrated automatically, based on the CMakeLists.txt
make rebuild
  • (Optional) Build & install vpp packages for your platform
make pkg-deb
cd build-root
sudo dpkg -i *.deb
  • The binary-api header files you can include later are located in build-root/build-vpp_debug-native/vpp/vpp-api/vapi
    •  If vpp is installed, they are located in /usr/include/vapi
  • Run vpp and check whether your plugin is loaded (newplugin has to be loaded and listed using the show plugin CLI command)
make run
...
load_one_plugin:189: Loaded plugin: nat_plugin.so (Network Address Translation)
load_one_plugin:189: Loaded plugin: newplugin_plugin.so (Sample VPP Plugin)
load_one_plugin:189: Loaded plugin: nsim_plugin.so (network delay simulator plugin)
...
DBGvpp# show plugins
...
 Plugin Version Description
 1. ioam_plugin.so 19.01-rc0~144-g0c2319f Inbound OAM
 ...
 x. newplugin_plugin.so 1.0 Sample VPP Plugin
 ...

How to create new API messages

API messages are defined in *.api files – see src/vnet/devices/af_packet.api, src/vnet/ip/ip.api, etc. These API files are used to generate corresponding message handlers. There are two types of API messages – non-blocking and blocking. These messages are used to communicate with the VPP Engine to configure and modify data path processing.

Non-blocking messages use one request and one reply message. Message replies can be auto-generated, or defined manually. Each request contains two mandatory fields – “client-index” and “context“, and each reply message contains mandatory fields – “context” and “retval“.

  • API message with auto-generated reply

autoreply define ip_table_add_del
{
 u32 client_index;
 u32 context;
 u32 table_id;
...
};
  • API message with manually defined reply
define ip_neighbor_add_del
{
 u32 client_index;
 u32 context;
 u32 sw_if_index;
...
};
define ip_neighbor_add_del_reply
{
 u32 context;
 i32 retval;
 u32 stats_index;
...
};

Blocking messages use one request and series of replies defined in *.api file. Each request contains two mandatory fields – “client-index” and “context“, and each reply message contains mandatory field – “context“.

  • Blocking message is defined using two structs – *-dump and *_details

define ip_fib_dump
{
 u32 client_index;
 u32 context;
...
};
define ip_fib_details
{
 u32 context;
...
};

Once you define a message in an API file, you have to define and implement the corresponding handlers for given request/reply message. These handlers are defined in one of component/plugin file and they use predefined naming – vl_api_…_t_handler – for each API message.

Here is an example for existing API messages (you can check it in src/vnet/ip component):

#define foreach_ip_api_msg \
_(IP_FIB_DUMP, ip_fib_dump) \
_(IP_NEIGHBOR_ADD_DEL, ip_neighbor_add_del) \
...
static void vl_api_ip_neighbor_add_del_t_handler (vl_api_ip_neighbor_add_del_t * mp, vlib_main_t * vm)
{
...
 REPLY_MACRO2 (VL_API_IP_NEIGHBOR_ADD_DEL_REPLY,
...
static void vl_api_ip_fib_dump_t_handler (vl_api_ip_fib_dump_t * mp)
{
...
 send_ip_fib_details (am, reg, fib_table, pfx, api_rpaths, mp->context);
...

Request and reply handlers are usually defined in api_format.c (or in plugin). Request uses a predefined naming – api_… for each API message and you have to also define help for each API message :

static int api_ip_neighbor_add_del (vat_main_t * vam)
{
...
  /* Construct the API message */
  M (IP_NEIGHBOR_ADD_DEL, mp);
  /* send it... */
  S (mp);
  /* Wait for a reply, return good/bad news */
  W (ret);
  return ret;
}
static int api_ip_fib_dump (vat_main_t * vam)
{
...
  M (IP_FIB_DUMP, mp);
  S (mp);
  /* Use a control ping for synchronization */
  MPING (CONTROL_PING, mp_ping);
  S (mp_ping);
  W (ret);
  return ret;
}
#define foreach_vpe_api_msg \
...
_(ip_neighbor_add_del, \
 "(<intfc> | sw_if_index <id>) dst <ip46-address> " \
 "[mac <mac-addr>] [vrf <vrf-id>] [is_static] [del]") \
...
_(ip_fib_dump, "") \
...

Replies can be auto-generated or manually defined.

  • auto-generated reply using define foreach_standard_reply_retval_handler, with predefined naming
  • manually defined reply with details

How to call the binary API

In order to call the binary API, we will introduce VAPI to our configuration.

VAPI is the high-level C/C++ binary API. Please refer to src/vpp-api/vapi/vapi_doc.md for details.

VAPI’s multitude of advantages include:

  • All headers in a single place – /usr/include/vapi => simplifies code generation
  • Hidden internals – one no longer has to care about message IDs, byte-order conversion
  • Easier binapi calls – passing user provided context between callbacks

We can use the following C++ code to call our new plugins’s binary API.

#include <cstdlib>
#include <iostream>
#include <cassert>

//necessary includes & macros
#include <vapi/vapi.hpp>
#include <vapi/vpe.api.vapi.hpp>
DEFINE_VAPI_MSG_IDS_VPE_API_JSON

//include the desired modules / plugins
#include <vapi/newplugin.api.vapi.hpp>
DEFINE_VAPI_MSG_IDS_NEWPLUGIN_API_JSON

using namespace vapi;
using namespace std;

//parameters for connecting
static const char *app_name = "test_client";
static const char *api_prefix = nullptr;
static const int max_outstanding_requests = 32;
static const int response_queue_size = 32;

#define WAIT_FOR_RESPONSE(param, ret)      \
  do                                       \
    {                                      \
      ret = con.wait_for_response (param); \
    }                                      \
  while (ret == VAPI_EAGAIN)

//global connection object
Connection con;

void die(int exit_code)
{
    //disconnect & cleanup
    vapi_error_e rv = con.disconnect();
    if (VAPI_OK != rv) {
        fprintf(stderr, "error: (rc:%d)", rv);
    }

    exit(exit_code);
}

int main()
{
    //connect to VPP
    vapi_error_e rv = con.connect(app_name, api_prefix, max_outstanding_requests, response_queue_size);

    if (VAPI_OK != rv) {
        cerr << "error: connecting to vlib";
        return rv;
    }

    try
    {
        Newplugin_macswap_enable_disable cl(con);

        auto &mp = cl.get_request().get_payload();

        mp.enable_disable = true;
        mp.sw_if_index = 5;

        auto rv = cl.execute ();
        if (VAPI_OK != rv) {
            throw exception{};
        }

        WAIT_FOR_RESPONSE (cl, rv);
        if (VAPI_OK != rv) {
            throw exception{};
        }

        //verify the reply
        auto &rp = cl.get_response ().get_payload ();
        if (rp.retval != 0) {
            throw exception{};
        }
    }
    catch (...)
    {
        cerr << "Newplugin_macswap_enable_disable ERROR" << endl;
        die(1);
    }

    die(0);
}

Additional C/C++ Examples

Furthermore, you are encouraged to try the minimal VAPI example provided in vapi_minimal.zip. This example creates a loopback interface, assigns it an IPv4 address and then prints the address.
Follow these steps:

  • Install VPP
  • Extract the archive, build & run examples
unzip vapi_minimal.zip
mkdir build; cd build
cmake ..
make

#c example
sudo ./vapi_minimal
#c++ example
sudo ./vapi_minimal_cpp

In conclusion, we have:

  • successfully built and ran our first VPP plugin
  • created and called an API message in VPP

Our next post will introduce and highlight the key reasons, why you should consider Honeycomb/hc2vpp in your VPP build.


You can contact us at https://pantheon.tech/

Explore our Pantheon GitHub. Follow us on Twitter.

Watch our YouTube Channel.

PANTHEON.tech presents: Vector Packet Processing (VPP) Guide

Welcome to our new series on how to build and program FD.io‘s Vector Packet Processing framework, also known as VPP.

The name stems from VPP’s usage of vector processing, which can process multiple packets at a time with low latency. Single packet processing and high latency were a common occurrence in the older, scalar processing approach, which VPP aims to make obsolete.

What will this series include?

This four-part series will include the following features, with the ultimate goal on getting to know your VPP framework and adapting it to your network:

  1. Binary API
  2. Honeycomb/hc2vpp
  3. Ligato VPP Agent
  4. gRPC/REST

Why should I start using Vector Package Processing?

The main advantages are:

  • high performance with a proven technology
  • production level quality
  • flexible and extensible

The principle of VPP is, that you can plugin a new graph node, adapt it to your networks purposes and run it right off the bat. Including a new plugin does not mean, you need to change your core-code with each new addition. Plugins can be either included in the processing graph, or they can be built outside the source tree and become an individual component in your build.

Furthermore, this separation of plugins makes crashes a matter of a simple process restart, which does not require your whole build to be restarted because of one plugin failure.

For a full list of features, please visit the official Vector Package Processing Wiki.You can also check our previous installments on VPP integration.

Preparation of VPP packages

In order to build and start with VPP yourself, you will have to:

  1. Download VPP’s repository from this page or follow the installation instructions
  2. Clone the repository inside your system, or from VPP’s GitHub

Enjoy and explore the repository as you wish. We will continue exploring the Binary API in the next part of our series.


You can contact us at https://pantheon.tech/

Explore our Pantheon GitHub.

Watch our YouTube Channel.

Integrate VPP & Honeycomb & Extension of VPP Services

In this short article, I would like to share our experience in the field of integrating VPP and Honeycomb, and about the extension of VPP services. Among our colleagues are many developers who contribute to both projects, as well as people who work on integrating these two projects. These developers also work on integrating them with the rest of the networking world.

Let’s define the basic terms.

What is VPP?

According to its wiki page, it is “an extensible framework that provides out-of-the-box production quality switch/router functionality”. There is definitely more to say about VPP, but what’s most important is that it:

  • provides switch and router functionality
  • is in production quality level
  • is platform independent

“Platform independent” means, that it is up to your decision where you will run it (virtualized environment, bare-metal or others). VPP is a piece of software, which is by default spread in the form of packages. Final VPP packages are available from Nexus repositories. Let’s say we decide to use stable VPP in version 17.04 on a stable Ubuntu version 16.04. You can download all available packages from the corresponding Nexus site. If there is no such platform available at Nexus, you can still download VPP and build it on the platform, which you need.

VPP will process packets, which flow in your network similarly to a physical router, but with one big advantage: you do not need to buy a router. You can use whatever physical device you have and just install the VPP modules.

What is Honeycomb?

It is a specific VPP interface. Honeycomb provides NETCONF and RESTCONF interface on northbound and stores required configuration (in form of XML or JSON) in local data store. There is also the hc2vpp project, which calls the corresponding VPP API as reaction to a new configuration stored in data store.

In VPP, there is a special CLI that is used to communicate with VPP. It is in text form (similarly as in OS). To make it easier to use VPP, we also have Honeycomb. It provides an interface, which is somewhere between a GUI and a CLI. You can also request VPP state or statistics of via XML, and you will get the response in an XML form. Honeycomb can be installed in the same way as VPP, through packages, which can be accessed from the Nexus site.

Where can the combination of VPP and Honeycomb be used?

We’ve already showcased several use cases on our PANTHEON.tech YouTube channel:

Another alternative is to use the two as vCPE (Virtual Customer Premises Equipment) as specified in this draft. One of projects which wants to implement it is ONAP. VPP used as vCPE-endpoint for the internet connection from a provider. According to this use case, vCPE should provide several services. In standalone VPP, such services aren’t supported, but they still can be added to a machine where VPP is running. For demonstration, we have chosen DHCP and DNS.

DHCP

In this case, we have two VMs. VM0 simulates the client side (DHCP client) which wants IP address to be assigned to interface enp0s9. VM1 contains VPP and a DHCP server. The DHCP request is broadcasted via enp0s9 at VM0 to VPP1 via port 192.168.40.2. VPP1 is set as proxy DHCP server and DHCP request message is forwarded to 192.168.60.2, where the DHCP server will response with a DHCP offer. Finally, after all DHCP configuration steps are done, interface enp0s9 at VM0 is configured with IP address 192.168.40.10.

DHCP

DNS

In this case, we also have two VMs. VM0 simulates the client side (DNS client) which needs to resolve domain name to IP address. This request is routed via local port to VPP1, where it is routed to DNS server in VM1. If this resolution is required for the first time, then the request will be sent to the external DNS server. Otherwise, local DNS server will serve this request.

DNS

Jozef Glončák

Software Engineer