From 0bb692dde188190c17496cdc854ee8ba3d7d9ff7 Mon Sep 17 00:00:00 2001
From: Kevin Deems <kevin8deems@gmail.com>
Date: Tue, 10 Nov 2020 13:54:03 -0500
Subject: [PATCH 01/17] [Docker Updates] Dockerize 20.10

Updates our Docker tag and pulls a new Ubuntu image

Commit log:
* Starting to get a docker container for onvm 20.10

* Finished compiling and updated markdown file
---
 docs/Docker.md     |  4 ++--
 scripts/Dockerfile | 30 ++++++++++++++++++------------
 2 files changed, 20 insertions(+), 14 deletions(-)

diff --git a/docs/Docker.md b/docs/Docker.md
index 190a3e6fd..d74fd4e45 100644
--- a/docs/Docker.md
+++ b/docs/Docker.md
@@ -34,7 +34,7 @@ sudo ./docker.sh -h HUGEPAGES -o ONVM -n NAME [-D DEVICES] [-d DIRECTORY] [-c CO
   - This will start a container with two NIC devices mapped in, /dev/uio0 and /dev/uio1, the hugepage directory at `/mnt/huge` mapped in, and the openNetVM source directory at `/root/openNetVM` mapped into the container with the name of Basic_Monitor_NF.
 
     ```bash
-    sudo ./docker.sh -h /mnt/huge -o /root/openNetVM -n Speed_Tester_NF -D /dev/uio0 -c "./examples/speed_tester/go.sh 1 -d 1"
+    sudo ./docker.sh -h /mnt/huge -o /root/openNetVM -n Speed_Tester_NF -D /dev/uio0 -c "./examples/start_nf.sh speed_tester 1 -d 1"
     ```
 
   - This will start a container with one NIC device mapped in, /dev/uio0 , the hugepage directory at `/mnt/huge` mapped in, and the openNetVM source directory at `/root/openNetVM` mapped into the container with the name of Speed_Tester_NF. Also, the container will be started in detached mode (no connection to it) and it will run the go script of the simple forward NF.
@@ -162,7 +162,7 @@ Older Dockerfiles
 
 If you want to use an older ONVM version on Ubuntu 14, take a look at the [Available Tags][onvm-docker-tags].
 The 18.03 tag runs ONVM when it had been set up for an older version of Ubuntu.
-The `latest` dockerfile runs on Ubuntu 18.04 and is called `latest`.
+The `latest` dockerfile runs on Ubuntu 20.04 and is called `latest`.
 
 [docker]: ../scripts/docker.sh
 [onvm-docker]: https://hub.docker.com/r/sdnfv/opennetvm/
diff --git a/scripts/Dockerfile b/scripts/Dockerfile
index bb25e3a99..8bdf4483d 100644
--- a/scripts/Dockerfile
+++ b/scripts/Dockerfile
@@ -34,9 +34,9 @@
 # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
-FROM ubuntu:18.04
-MAINTAINER "Kevin Deems" <kevin8deems@gmail.com>
-LABEL version="OpenNetVM v19.07"
+FROM ubuntu:20.04
+LABEL maintainer="Kevin Deems kevin8deems@gmail.com"
+LABEL version="OpenNetVM v20.10"
 LABEL vendor="SDNFV @ UCR and GW"
 LABEL github="github.com/sdnfv/openNetVM"
 
@@ -47,13 +47,19 @@ ENV ONVM_NUM_HUGEPAGES=1024
 
 WORKDIR $ONVM_HOME
 
+# Ubuntu 20.04 image hangs unless timezone is set
+RUN DEBIAN_FRONTEND=noninteractive \
+    ln -fs /usr/share/zoneinfo/America/New_York /etc/localtime
+
 RUN apt-get update -y && apt-get upgrade -y && \
-    apt-get install -y build-essential \
-                       sudo \
-                       gdb \
-                       python \
-                       libnuma-dev \
-                       vim \
-                       less \
-                       git \
-                       net-tools \
+    apt-get install -y \
+    build-essential \
+    sudo \
+    gdb \
+    python \
+    libnuma-dev \
+    vim \
+    less \
+    git \
+    net-tools \
+    bc
\ No newline at end of file

From dda73e45e1396be0e120a80cbca0b09f69279969 Mon Sep 17 00:00:00 2001
From: JackKuo <kuohfjack@gmail.com>
Date: Wed, 11 Nov 2020 02:57:20 +0800
Subject: [PATCH 02/17] [Documentation] Fix comment typo

Fixes typos in onvm_pkt_helper.h

Commit log:
* Fix comment typo
---
 onvm/onvm_nflib/onvm_pkt_helper.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/onvm/onvm_nflib/onvm_pkt_helper.h b/onvm/onvm_nflib/onvm_pkt_helper.h
index 93490363e..de62defa6 100644
--- a/onvm/onvm_nflib/onvm_pkt_helper.h
+++ b/onvm/onvm_nflib/onvm_pkt_helper.h
@@ -200,19 +200,19 @@ int
 onvm_pkt_swap_ether_hdr(struct rte_ether_hdr* ether_hdr);
 
 /**
- * Generates a UDP packet with the provided values
+ * Swap the IP header values
  */
 int
 onvm_pkt_swap_ip_hdr(struct rte_ipv4_hdr* ip_hdr);
 
 /**
- * Generates a TCP packet with the provided values
+ * Swap the TCP header values
  */
 int
 onvm_pkt_swap_tcp_hdr(struct rte_tcp_hdr* tcp_hdr);
 
 /**
- * Generates a UDP packet with the provided values
+ * Generates a TCP packet with the provided values
  */
 struct rte_mbuf*
 onvm_pkt_generate_tcp(struct rte_mempool* pktmbuf_pool, struct rte_tcp_hdr* tcp_hdr, struct rte_ipv4_hdr* iph,

From d5ca4581b7b8806b456c4bff2fc14774189ac87b Mon Sep 17 00:00:00 2001
From: JackKuo <kuohfjack@gmail.com>
Date: Wed, 11 Nov 2020 03:02:03 +0800
Subject: [PATCH 03/17] [Documentation] Update Installation README

Fixes to the install README

Commit log:
* Update installation README
---
 docs/Install.md | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/docs/Install.md b/docs/Install.md
index 3d599a711..efe6f6332 100644
--- a/docs/Install.md
+++ b/docs/Install.md
@@ -21,7 +21,7 @@ Check System
 
     Install the Linux Kernel headers package for your kernel version.
     ```sh
-    sudo apt-get install build-essential linux-headers-$(uname -r) git bc
+    sudo apt-get install build-essential linux-headers-$(uname -r) git bc gcc make pkg-config
     ```
     If your distribution didn't come with Python or came with an earlier version, you will need to install Python 3 v3.4+. 
 
@@ -50,7 +50,7 @@ Check System
    ```sh
    sudo apt-get install libnuma-dev
    ```
-   If installing libnuma-dev fails, your system may not be up to date. To fix this, run:
+   If installing libnuma-dev fails, your package information may not be up to date. To fix this, run:
    ```sh
    sudo apt-get update
    ```
@@ -105,7 +105,7 @@ Set up Environment
     echo export ONVM_NUM_HUGEPAGES=1024 >> ~/.bashrc
     ```
 
-    ONVM_NIC_PCI is a variable that specifies NIC ports to be bound to DPDK.  If ONVM_NIC_PCI is not specified, the default action is to bind all non-active 10G NIC ports to DPDK. Note, NIC PCI device IDs may not be the same across all hosts. In that case, please retrieve this information for your host before setting the variable.
+    ONVM_NIC_PCI is a variable that specifies NIC ports to be bound to DPDK.  If ONVM_NIC_PCI is not specified, the default action is to bind all non-active 10G NIC ports to DPDK. Note, NIC PCI device IDs may not be the same across all hosts. In that case, please retrieve this information for your host before setting the variable. (e.g., via `lspci`)
     ```sh
     export ONVM_NIC_PCI=" 07:00.0 07:00.1 "
     ```
@@ -117,7 +117,7 @@ Set up Environment
 7. Disable ASLR since it makes sharing memory with NFs harder:
    ```sh
    sudo sh -c "echo 0 > /proc/sys/kernel/randomize_va_space"
-    ```
+   ```
 
 Configure and compile DPDK
 --
@@ -130,6 +130,8 @@ Configure and compile DPDK
 
     The [install script](../scripts/install.sh) will automatically run the [environment setup script](../scripts/setup_environment.sh), which configures your local environment.  This should be run once for every reboot, as it loads the appropraite kernel modules and can bind your NIC ports to the DPDK driver.
 
+    If `which python` shows **not found** in your environment, please link `python` to Python3, e.g., via `link -s` or `alias python=$(which python3)`.
+
 Run DPDK HelloWorld Application
 --
 

From d056dc8aba2d422525d06695a63a314920f9d607 Mon Sep 17 00:00:00 2001
From: JackKuo <kuohfjack@gmail.com>
Date: Tue, 8 Dec 2020 23:07:09 +0800
Subject: [PATCH 04/17] [Bug Fix] Fix relative path problem about web

Instead of using .. we use the $ONVM_HOME variable in the onvm go script to find the web directory.

Commit log:
* Fix relative path problem about web
---
 onvm/go.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/onvm/go.sh b/onvm/go.sh
index 6b4cb3e21..421357a86 100755
--- a/onvm/go.sh
+++ b/onvm/go.sh
@@ -255,7 +255,7 @@ fi
 
 if [ "${stats}" = "-s web" ]
 then
-    cd ../onvm_web/ || usage
+    cd "$ONVM_HOME"/onvm_web/ || usage
     if [ -n "${web_port}" ]
     then
         . start_web_console.sh -p "${web_port}"

From 0cb551d93d4421f8f3b85e396ca13ce04a38c1e8 Mon Sep 17 00:00:00 2001
From: Kevin Deems <kevin8deems@gmail.com>
Date: Sun, 7 Mar 2021 13:57:59 -0500
Subject: [PATCH 05/17] [Bug Fix] Fix json configuration and duplicate instance
 id bugs

Fixes bug where NF rings would not be cleared after deallocation, as well as an underflow bug in stats.

Commit log:
* Testing onvm config bug

* Fixed instance id stats issue

* Added strlenn to account for null terminator

* Reverting some linting changes
---
 examples/example_config.json         |  3 +--
 examples/start_nf.sh                 |  1 +
 onvm/onvm_mgr/onvm_nf.c              | 40 +++++++++++++++++++---------
 onvm/onvm_mgr/onvm_stats.c           | 14 ++++++++--
 onvm/onvm_nflib/onvm_config_common.c | 29 +++++++++++---------
 onvm/onvm_nflib/onvm_config_common.h | 12 +++++++++
 6 files changed, 71 insertions(+), 28 deletions(-)

diff --git a/examples/example_config.json b/examples/example_config.json
index 2191bb1c1..5c1302a41 100644
--- a/examples/example_config.json
+++ b/examples/example_config.json
@@ -4,10 +4,9 @@
                 "memory_channels": 3,
                 "portmask": 2
         },
-
         "onvm": {
                 "output": "stdout",
                 "serviceid": 1,
                 "instanceid": 3
         }
-}
+}
\ No newline at end of file
diff --git a/examples/start_nf.sh b/examples/start_nf.sh
index 85078392a..f1a9cf840 100755
--- a/examples/start_nf.sh
+++ b/examples/start_nf.sh
@@ -45,6 +45,7 @@ then
   config=$2
   shift 2
   exec sudo "$BINARY" -F "$config" "$@"
+  exit 0
 fi
 
 # Check if -- is present, if so parse dpdk/onvm specific args
diff --git a/onvm/onvm_mgr/onvm_nf.c b/onvm/onvm_mgr/onvm_nf.c
index 674424ed0..aad5f291f 100644
--- a/onvm/onvm_mgr/onvm_nf.c
+++ b/onvm/onvm_mgr/onvm_nf.c
@@ -117,6 +117,16 @@ onvm_nf_init_lpm_region(struct lpm_request *req_lpm);
 static void
 onvm_nf_init_ft(struct ft_request *ft);
 
+/*
+ * Function that clears the tx, rx, & msg_q rings
+ * in case an NF uses this iid in the future
+ *
+ * Input  : An nf struct
+ * Output : none
+ */
+static void
+onvm_nf_clear_rings(struct onvm_nf *nf);
+
 /*
  *  Set up the DPDK rings which will be used to pass packets, via
  *  pointers, between the multi-process server and NF processes.
@@ -192,7 +202,7 @@ onvm_nf_check_status(void) {
                                 onvm_nf_init_lpm_region(req_lpm);
                                 break;
                         case MSG_REQUEST_FT:
-                                ft = (struct ft_request *) msg->msg_data;
+                                ft = (struct ft_request *)msg->msg_data;
                                 onvm_nf_init_ft(ft);
                                 break;
                         case MSG_NF_STARTING:
@@ -370,8 +380,8 @@ onvm_nf_stop(struct onvm_nf *nf) {
                         rte_pktmbuf_free(pkts[i]);
         }
         nf_msg_pool = rte_mempool_lookup(_NF_MSG_POOL_NAME);
-        while (rte_ring_dequeue(nfs[nf_id].msg_q, (void**)(&msg)) == 0) {
-                rte_mempool_put(nf_msg_pool, (void*)msg);
+        while (rte_ring_dequeue(nfs[nf_id].msg_q, (void **)(&msg)) == 0) {
+                rte_mempool_put(nf_msg_pool, (void *)msg);
         }
 
         /* Free info struct */
@@ -380,7 +390,7 @@ onvm_nf_stop(struct onvm_nf *nf) {
         if (nf_info_mp == NULL)
                 return 1;
 
-        rte_mempool_put(nf_info_mp, (void*)nf);
+        rte_mempool_put(nf_info_mp, (void *)nf);
 
         /* Further cleanup is only required if NF was succesfully started */
         if (nf_status != NF_RUNNING && nf_status != NF_PAUSED)
@@ -392,6 +402,9 @@ onvm_nf_stop(struct onvm_nf *nf) {
         /* Reset stats */
         onvm_stats_clear_nf(nf_id);
 
+        /* Free rings if another NF uses this instance id */
+        onvm_nf_clear_rings(&nfs[nf_id]);
+
         /* Remove this NF from the service map.
          * Need to shift all elements past it in the array left to avoid gaps */
         nf_per_service_count[service_id]--;
@@ -429,7 +442,7 @@ onvm_nf_stop(struct onvm_nf *nf) {
 static void
 onvm_nf_init_lpm_region(struct lpm_request *req_lpm) {
         struct rte_lpm_config conf;
-        struct rte_lpm* lpm_region;
+        struct rte_lpm *lpm_region;
 
         conf.max_rules = req_lpm->max_num_rules;
         conf.number_tbl8s = req_lpm->num_tbl8s;
@@ -474,6 +487,13 @@ onvm_nf_relocate_nf(uint16_t dest, uint16_t new_core) {
         return 0;
 }
 
+static void
+onvm_nf_clear_rings(struct onvm_nf *nf) {
+        rte_ring_free(nf->rx_q);
+        rte_ring_free(nf->tx_q);
+        rte_ring_free(nf->msg_q);
+}
+
 static void
 onvm_nf_init_rings(struct onvm_nf *nf) {
         unsigned instance_id;
@@ -489,20 +509,16 @@ onvm_nf_init_rings(struct onvm_nf *nf) {
         rq_name = get_rx_queue_name(instance_id);
         tq_name = get_tx_queue_name(instance_id);
         msg_q_name = get_msg_queue_name(instance_id);
-        nf->rx_q =
-                rte_ring_create(rq_name, ringsize, socket_id, RING_F_SC_DEQ); /* multi prod, single cons */
-        nf->tx_q =
-                rte_ring_create(tq_name, ringsize, socket_id, RING_F_SC_DEQ); /* multi prod, single cons */
-        nf->msg_q =
-                rte_ring_create(msg_q_name, msgringsize, socket_id,
-                                RING_F_SC_DEQ); /* multi prod, single cons */
 
+        nf->rx_q = rte_ring_create(rq_name, ringsize, socket_id, RING_F_SC_DEQ); /* multi prod, single cons */
         if (nf->rx_q == NULL)
                 rte_exit(EXIT_FAILURE, "Cannot create rx ring queue for NF %u\n", instance_id);
 
+        nf->tx_q = rte_ring_create(tq_name, ringsize, socket_id, RING_F_SC_DEQ); /* multi prod, single cons */
         if (nf->tx_q == NULL)
                 rte_exit(EXIT_FAILURE, "Cannot create tx ring queue for NF %u\n", instance_id);
 
+        nf->msg_q = rte_ring_create(msg_q_name, msgringsize, socket_id, RING_F_SC_DEQ); /* multi prod, single cons */
         if (nf->msg_q == NULL)
                 rte_exit(EXIT_FAILURE, "Cannot create msg queue for NF %u\n", instance_id);
 }
diff --git a/onvm/onvm_mgr/onvm_stats.c b/onvm/onvm_mgr/onvm_stats.c
index 6f0e370f8..8d0863d55 100644
--- a/onvm/onvm_mgr/onvm_stats.c
+++ b/onvm/onvm_mgr/onvm_stats.c
@@ -455,10 +455,21 @@ onvm_stats_display_nfs(unsigned difftime, uint8_t verbosity_level) {
                 const uint64_t act_next = nfs[i].stats.act_next;
                 const uint64_t act_buffer = nfs[i].stats.tx_buffer;
                 const uint64_t act_returned = nfs[i].stats.tx_returned;
+
+                /* On onvm_stats_clear_nf, subtraction causes underflow */
+                if (unlikely(rx == 0))
+                        nf_rx_last[i] = 0;
                 const uint64_t rx_pps = (rx - nf_rx_last[i]) / difftime;
+                if (unlikely(tx == 0))
+                        nf_tx_last[i] = 0;
                 const uint64_t tx_pps = (tx - nf_tx_last[i]) / difftime;
-                const uint64_t tx_drop_rate = (tx_drop - nf_tx_drop_last[i]) / difftime;
+                if (unlikely(rx_drop == 0))
+                        nf_rx_drop_last[i] = 0;
                 const uint64_t rx_drop_rate = (rx_drop - nf_rx_drop_last[i]) / difftime;
+                if (unlikely(tx_drop == 0))
+                        nf_tx_drop_last[i] = 0;
+                const uint64_t tx_drop_rate = (tx_drop - nf_tx_drop_last[i]) / difftime;
+
                 const uint64_t num_wakeups = nf_wakeup_infos[i].num_wakeups;
                 const uint64_t prev_num_wakeups = nf_wakeup_infos[i].prev_num_wakeups;
                 const uint64_t wakeup_rate = (num_wakeups - prev_num_wakeups) / difftime;
@@ -571,7 +582,6 @@ onvm_stats_display_nfs(unsigned difftime, uint8_t verbosity_level) {
                 fprintf(stats_out, "-----------------\n");
                 onvm_stats_display_client_wakeup_thread_context(difftime);
         }
-
 }
 
 /***************************Helper functions**********************************/
diff --git a/onvm/onvm_nflib/onvm_config_common.c b/onvm/onvm_nflib/onvm_config_common.c
index ed72fd75b..6fbccfd94 100644
--- a/onvm/onvm_nflib/onvm_config_common.c
+++ b/onvm/onvm_nflib/onvm_config_common.c
@@ -49,6 +49,11 @@
 
 #define IS_NULL_OR_EMPTY_STRING(s) ((s) == NULL || strncmp(s, "", 1) == 0 ? 1 : 0)
 
+size_t
+strlenn(const char* __s) {
+        return strlen(__s) + 1;
+}
+
 cJSON*
 onvm_config_parse_file(const char* filename) {
         if (IS_NULL_OR_EMPTY_STRING(filename)) {
@@ -420,7 +425,7 @@ onvm_config_create_onvm_args(cJSON* onvm_config, int* onvm_argc, char** onvm_arg
                 return -1;
         }
 
-        (*onvm_argv)[0] = malloc(sizeof(char) * strlen(FLAG_R));
+        (*onvm_argv)[0] = malloc(sizeof(char) * strlenn(FLAG_R));
         if ((*onvm_argv)[0] == NULL) {
                 printf("Unable to allocate space for onvm_argv[0]\n");
                 free(service_id_string);
@@ -428,7 +433,7 @@ onvm_config_create_onvm_args(cJSON* onvm_config, int* onvm_argc, char** onvm_arg
                 return -1;
         }
 
-        memcpy((*onvm_argv)[0], FLAG_R, strlen(FLAG_R));
+        memcpy((*onvm_argv)[0], FLAG_R, strlenn(FLAG_R));
 
         snprintf(service_id_string, sizeof(char) * MAX_SERVICE_ID_SIZE, "%d", service_id);
         (*onvm_argv)[1] = service_id_string;
@@ -443,7 +448,7 @@ onvm_config_create_onvm_args(cJSON* onvm_config, int* onvm_argc, char** onvm_arg
                         return -1;
                 }
 
-                (*onvm_argv)[2] = malloc(sizeof(char) * strlen(FLAG_N));
+                (*onvm_argv)[2] = malloc(sizeof(char) * strlenn(FLAG_N));
                 if ((*onvm_argv)[2] == NULL) {
                         printf("Could not allocate space for instance id in argv\n");
                         free(instance_id_string);
@@ -452,7 +457,7 @@ onvm_config_create_onvm_args(cJSON* onvm_config, int* onvm_argc, char** onvm_arg
                         free((*onvm_argv)[0]);
                         return -1;
                 }
-                memcpy((*onvm_argv)[2], FLAG_N, strlen(FLAG_N));
+                memcpy((*onvm_argv)[2], FLAG_N, strlenn(FLAG_N));
                 snprintf(instance_id_string, sizeof(char) * MAX_SERVICE_ID_SIZE, "%d", instance_id);
                 (*onvm_argv)[3] = instance_id_string;
         }
@@ -491,7 +496,7 @@ onvm_config_create_dpdk_args(cJSON* dpdk_config, int* dpdk_argc, char** dpdk_arg
                 return -1;
         }
 
-        core_string = (char*)malloc(sizeof(char) * strlen(cJSON_GetObjectItem(dpdk_config, "corelist")->valuestring));
+        core_string = (char*)malloc(sizeof(char) * strlenn(cJSON_GetObjectItem(dpdk_config, "corelist")->valuestring));
         if (core_string == NULL) {
                 printf("Unable to allocate space for core string\n");
                 free(*dpdk_argv);
@@ -500,7 +505,7 @@ onvm_config_create_dpdk_args(cJSON* dpdk_config, int* dpdk_argc, char** dpdk_arg
         }
 
         core_string = strncpy(core_string, cJSON_GetObjectItem(dpdk_config, "corelist")->valuestring,
-                              strlen(cJSON_GetObjectItem(dpdk_config, "corelist")->valuestring));
+                              strlenn(cJSON_GetObjectItem(dpdk_config, "corelist")->valuestring));
 
         if (onvm_config_extract_memory_channels(dpdk_config, &mem_channels) < 0) {
                 printf("Unable to extract memory channels\n");
@@ -522,12 +527,12 @@ onvm_config_create_dpdk_args(cJSON* dpdk_config, int* dpdk_argc, char** dpdk_arg
 
         snprintf(mem_channels_string, mem_channels_string_size, "%d", mem_channels);
 
-        arg_size[0] = strlen(FLAG_L);
-        arg_size[1] = strlen(core_string);
-        arg_size[2] = strlen(FLAG_N);
-        arg_size[3] = strlen(mem_channels_string);
-        arg_size[4] = strlen(PROC_TYPE_SECONDARY);
-        arg_size[5] = strlen(FLAG_DASH);
+        arg_size[0] = strlenn(FLAG_L);
+        arg_size[1] = strlenn(core_string);
+        arg_size[2] = strlenn(FLAG_N);
+        arg_size[3] = strlenn(mem_channels_string);
+        arg_size[4] = strlenn(PROC_TYPE_SECONDARY);
+        arg_size[5] = strlenn(FLAG_DASH);
 
         for (i = 0; i < *dpdk_argc; ++i) {
                 (*dpdk_argv)[i] = (char*)malloc(arg_size[i]);
diff --git a/onvm/onvm_nflib/onvm_config_common.h b/onvm/onvm_nflib/onvm_config_common.h
index f4de86aec..45d9e5a9f 100644
--- a/onvm/onvm_nflib/onvm_config_common.h
+++ b/onvm/onvm_nflib/onvm_config_common.h
@@ -55,6 +55,18 @@
 
 /*****************************API************************************/
 
+/*
+ * Uses strlen to get string length *including*
+ * null terminator '\0'
+ *
+ * @param __s
+ *   String to measure
+ * @return
+ *   Size of the string including null terminator
+ */
+size_t
+strlenn(const char* __s);
+
 /**
  * Parses a config file into a cJSON struct.
  * This struct contains all information stored within the config file

From ea1180628af279ff925d397277f74717ff2618a2 Mon Sep 17 00:00:00 2001
From: Leslie Monis <lesliemonis@gmail.com>
Date: Mon, 8 Mar 2021 01:02:20 +0530
Subject: [PATCH 06/17] [Feature] Add support for jumbo frames

Add support for ports to send and receive packets with
length up to 9600 bytes. Use the -j option when running the
manager to enable this support.

Commit log:
* Add support for jumbo frames
---
 onvm/go.sh                |  7 +++++--
 onvm/onvm_mgr/onvm_args.c | 14 +++++++++++---
 onvm/onvm_mgr/onvm_init.c | 14 +++++++++++++-
 onvm/onvm_mgr/onvm_init.h |  3 +--
 4 files changed, 30 insertions(+), 8 deletions(-)

diff --git a/onvm/go.sh b/onvm/go.sh
index 421357a86..4a84c7cb2 100755
--- a/onvm/go.sh
+++ b/onvm/go.sh
@@ -16,6 +16,8 @@ function usage {
         echo -e "\tRuns ONVM the same way as above, but prints statistics to stdout"
         echo -e "$0 -k 3 -n 0xF0 -m 2,3,4 -s stdout -c"
         echo -e "\tRuns ONVM the same way as above, but enables shared cpu support"
+        echo -e "$0 -k 3 -n 0xF0 -m 2,3,4 -s stdout -c -j"
+        echo -e "\tRuns ONVM the same way as above, but allows ports to send and receive jumbo frames"
         echo -e "$0 -k 3 -n 0xF0 -m 2,3,4 -s stdout -t 42"
         echo -e "\tRuns ONVM the same way as above, but shuts down after 42 seconds"
         echo -e "$0 -k 3 -n 0xF0 -m 2,3,4 -s stdout -l 64"
@@ -110,7 +112,7 @@ then
     exit 1
 fi
 
-while getopts "a:r:d:s:t:l:p:z:cvm:k:n:" opt; do
+while getopts "a:r:d:s:t:l:p:z:cvm:k:n:j" opt; do
     case $opt in
         a) virt_addr="--base-virtaddr=$OPTARG";;
         r) num_srvc="-r $OPTARG";;
@@ -152,6 +154,7 @@ while getopts "a:r:d:s:t:l:p:z:cvm:k:n:" opt; do
             else
                 nf_cores=$OPTARG
             fi;;
+        j) jumbo_frames_flag="-j";;
         \?) echo "Unknown option -$OPTARG" && usage
             ;;
     esac
@@ -269,7 +272,7 @@ fi
 sudo rm -rf /mnt/huge/rtemap_*
 # watch out for variable expansion
 # shellcheck disable=SC2086
-sudo "$SCRIPTPATH"/onvm_mgr/"$RTE_TARGET"/onvm_mgr -l "$cpu" -n 4 --proc-type=primary ${virt_addr} -- -p ${ports} -n ${nf_cores} ${num_srvc} ${def_srvc} ${stats} ${stats_sleep_time} ${verbosity_level} ${ttl} ${packet_limit} ${shared_cpu_flag}
+sudo "$SCRIPTPATH"/onvm_mgr/"$RTE_TARGET"/onvm_mgr -l "$cpu" -n 4 --proc-type=primary ${virt_addr} -- -p ${ports} -n ${nf_cores} ${num_srvc} ${def_srvc} ${stats} ${stats_sleep_time} ${verbosity_level} ${ttl} ${packet_limit} ${shared_cpu_flag} ${jumbo_frames_flag}
 
 if [ "${stats}" = "-s web" ]
 then
diff --git a/onvm/onvm_mgr/onvm_args.c b/onvm/onvm_mgr/onvm_args.c
index 9989e9f77..211abc8dc 100644
--- a/onvm/onvm_mgr/onvm_args.c
+++ b/onvm/onvm_mgr/onvm_args.c
@@ -79,6 +79,9 @@ uint8_t global_verbosity_level = 1;
 /* global flag for enabling shared core logic - extern in init.h */
 uint8_t ONVM_NF_SHARE_CORES = 0;
 
+/* global flag for jumbo frames - extern in init.h */
+uint8_t ONVM_USE_JUMBO_FRAMES = 0;
+
 /* global var for program name */
 static const char *progname;
 
@@ -126,11 +129,12 @@ parse_app_args(uint8_t max_ports, int argc, char *argv[]) {
             {"nf-cores", required_argument, NULL, 'n'},  {"default-service", required_argument, NULL, 'd'},
             {"stats-out", no_argument, NULL, 's'},       {"stats-sleep-time", no_argument, NULL, 'z'},
             {"time_to_live", no_argument, NULL, 't'},    {"packet_limit", no_argument, NULL, 'l'},
-            {"verbocity-level", no_argument, NULL, 'v'}, {"enable_shared_cpu", no_argument, NULL, 'c'}};
+            {"verbocity-level", no_argument, NULL, 'v'}, {"enable_shared_cpu", no_argument, NULL, 'c'},
+            {"jumbo_frames", no_argument, NULL, 'j'}};
 
         progname = argv[0];
 
-        while ((opt = getopt_long(argc, argvopt, "p:r:n:d:s:t:l:z:v:c", lgopts, &option_index)) != EOF) {
+        while ((opt = getopt_long(argc, argvopt, "p:r:n:d:s:t:l:z:v:cj", lgopts, &option_index)) != EOF) {
                 switch (opt) {
                         case 'p':
                                 if (parse_portmask(max_ports, optarg) != 0) {
@@ -192,6 +196,9 @@ parse_app_args(uint8_t max_ports, int argc, char *argv[]) {
                                 onvm_config->flags.ONVM_NF_SHARE_CORES = 1;
                                 ONVM_NF_SHARE_CORES = 1;
                                 break;
+                        case 'j':
+                                ONVM_USE_JUMBO_FRAMES = 1;
+                                break;
                         default:
                                 printf("ERROR: Unknown option '%c'\n", opt);
                                 usage();
@@ -216,7 +223,8 @@ usage(void) {
             "\t-t TTL: time to live, how many seconds to wait until exiting (optional)\n"
             "\t-l PACKET_LIMIT: how many millions of packets to recieve before exiting (optional)\n"
             "\t-v VERBOCITY_LEVEL: verbocity level of the stats output (optional)\n"
-            "\t-c ENABLE_SHARED_CORE: allow the NFs to share a core based on mutex sleep/wakeups (optional)\n",
+            "\t-c ENABLE_SHARED_CORE: allow the NFs to share a core based on mutex sleep/wakeups (optional)\n"
+            "\t-j JUMBO_FRAMES: allow the ports to send and receive jumbo frames (optional)\n",
             progname);
 }
 
diff --git a/onvm/onvm_mgr/onvm_init.c b/onvm/onvm_mgr/onvm_init.c
index a0d9e7527..25884dc96 100644
--- a/onvm/onvm_mgr/onvm_init.c
+++ b/onvm/onvm_mgr/onvm_init.c
@@ -283,10 +283,17 @@ set_default_config(struct onvm_configuration *config) {
  */
 static int
 init_mbuf_pools(void) {
+        uint16_t mbuf_size;
+
+        if (ONVM_USE_JUMBO_FRAMES)
+                mbuf_size = 9600 + RTE_ETHER_CRC_LEN + RTE_ETHER_HDR_LEN + MBUF_OVERHEAD;
+        else
+                mbuf_size = RTE_MBUF_DEFAULT_DATAROOM + MBUF_OVERHEAD;
+
         /* don't pass single-producer/single-consumer flags to mbuf create as it
          * seems faster to use a cache instead */
         printf("Creating mbuf pool '%s' [%u mbufs] ...\n", PKTMBUF_POOL_NAME, NUM_MBUFS);
-        pktmbuf_pool = rte_mempool_create(PKTMBUF_POOL_NAME, NUM_MBUFS, MBUF_SIZE, MBUF_CACHE_SIZE,
+        pktmbuf_pool = rte_mempool_create(PKTMBUF_POOL_NAME, NUM_MBUFS, mbuf_size, MBUF_CACHE_SIZE,
                                           sizeof(struct rte_pktmbuf_pool_private), rte_pktmbuf_pool_init, NULL,
                                           rte_pktmbuf_init, NULL, rte_socket_id(), NO_FLAGS);
 
@@ -363,6 +370,11 @@ init_port(uint8_t port_num) {
                     port_num, port_conf.rx_adv_conf.rss_conf.rss_hf, local_port_conf.rx_adv_conf.rss_conf.rss_hf);
         }
 
+        if (ONVM_USE_JUMBO_FRAMES) {
+                local_port_conf.rxmode.max_rx_pkt_len = 9600 + RTE_ETHER_CRC_LEN + RTE_ETHER_HDR_LEN;
+                local_port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_JUMBO_FRAME;
+        }
+
         if ((retval = rte_eth_dev_configure(port_num, rx_rings, tx_rings, &local_port_conf)) != 0)
                 return retval;
 
diff --git a/onvm/onvm_mgr/onvm_init.h b/onvm/onvm_mgr/onvm_init.h
index f133038e4..85d11e9c8 100644
--- a/onvm/onvm_mgr/onvm_init.h
+++ b/onvm/onvm_mgr/onvm_init.h
@@ -79,8 +79,6 @@
 
 #define MBUF_CACHE_SIZE 512
 #define MBUF_OVERHEAD (sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM)
-#define RX_MBUF_DATA_SIZE 2048
-#define MBUF_SIZE (RX_MBUF_DATA_SIZE + MBUF_OVERHEAD)
 
 #define NF_INFO_SIZE sizeof(struct onvm_nf_init_cfg)
 
@@ -126,6 +124,7 @@ extern uint8_t global_verbosity_level;
 /* Custom flags for onvm */
 extern struct onvm_configuration *onvm_config;
 extern uint8_t ONVM_NF_SHARE_CORES;
+extern uint8_t ONVM_USE_JUMBO_FRAMES;
 
 /* For handling shared core logic */
 extern struct nf_wakeup_info *nf_wakeup_infos;

From a35035d66ac44b5a5a6e2566af057a625f433c64 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=E5=90=B4=E8=8A=83?=
 <45660411+PengWu-wp@users.noreply.github.com>
Date: Wed, 25 Aug 2021 00:41:01 +0800
Subject: [PATCH 07/17] [Bug Fix] Update NF_TAG in aesdecrypt.c

Fixes the NF_TAG of aesdecrypt in openNetVM/examples/aes_decrypt/aesdecrypt.c.

Commit log:
* Update aesdecrypt.c
---
 examples/aes_decrypt/aesdecrypt.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/examples/aes_decrypt/aesdecrypt.c b/examples/aes_decrypt/aesdecrypt.c
index 1810f9375..dd2e0c570 100644
--- a/examples/aes_decrypt/aesdecrypt.c
+++ b/examples/aes_decrypt/aesdecrypt.c
@@ -60,7 +60,7 @@
 #include "onvm_nflib.h"
 #include "onvm_pkt_helper.h"
 
-#define NF_TAG "aes_encrypt"
+#define NF_TAG "aes_decrypt"
 
 /* number of package between each print */
 static uint32_t print_delay = 1000000;

From 211ac80edadec26f444a8202cdffe1c925a45e47 Mon Sep 17 00:00:00 2001
From: Catherine Meadows <meadowsc@gwu.edu>
Date: Tue, 24 Aug 2021 12:42:45 -0400
Subject: [PATCH 08/17] [Documentation] Moongen documentation updates

Updates Moongen installation guide to work with new DPDK version.

Commit log:

* updates to moongen install guide

* updates for documentation

* update dpdk version

* add missing dependencies
---
 docs/moongen.md | 125 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 82 insertions(+), 43 deletions(-)

diff --git a/docs/moongen.md b/docs/moongen.md
index cbb412566..fd241658b 100644
--- a/docs/moongen.md
+++ b/docs/moongen.md
@@ -1,26 +1,33 @@
-MoonGen Installation (DPDK-2.0 Version)
+MoonGen Installation (DPDK-19.5 Version)
 ===================
 
-#### Welcome to installation memo for [MoonGen](http://scholzd.github.io/MoonGen/install.html), MoonGen is a "Scriptable High-Speed Packet Generator". 
+#### Welcome to installation memo for [MoonGen](http://scholzd.github.io/MoonGen/install.html). MoonGen is a "Scriptable High-Speed Packet Generator". 
 ----------
 
-1. Preparation steps 
+## 1. Preparation steps
 ===================
-Installation steps are assuming that you have already got OpenNetVM installed. If you have already got OpenNetVM installed, please following the steps below for a double check of your system.
 
-1.1 check if you have available hugepages
+Installation steps are assuming that you already have [openNetVM installed](./Install.md). Follow the steps below to double check your system configuration.
 
-`$grep -i huge /proc/meminfo`
+### 1.1 Check for available hugepages
 
-If ***HugePages_Free*** 's value equals to 0, which means there is no free hugepages available, you probably have to reboot your machine, by `$sudo reboot` to get some released hugepages. 
+`$ grep -i huge /proc/meminfo`
 
-1.2 check if you have available ports binded to DPDK
+If the ***HugePages_Free*** 's value equals 0, which means there are no free hugepages available, there may be a few reasons why: 
+- The manager crashed, but an NF(s) is still running.
+    - In this case, either kill them manually by hitting Ctrl+C or run `$ sudo pkill NF_NAME` for every NF that you have ran.
+- The manager and NFs are not running, but something crashed without freeing hugepages.
+    - To fix this, please run `$ sudo rm -rf /mnt/huge/*` to remove all files that contain hugepage data.
+- The above two cases are not met, something weird is happening:
+    - A reboot might fix this problem and free memory: `$ sudo reboot`
 
-`$cd dirctory_of_your_installed_dpdk`
+### 1.2 Check NIC ports are bound to DPDK
 
-`$./tools/dpdk_nic_bind.py  --status`
+`$ cd dirctory_of_your_installed_dpdk`
 
-if you got the follwing binding information indicating that you have the two 10-Gigabit NIC ports binded with DPDK driver, then you are fine, please jump to step 1.4, otherwise, please jump to step 1.3.
+`$ ./usertools/dpdk-devbind.py  --status`
+
+If you got the follwing similar binding information indicating that you have the two 10-Gigabit NIC ports bound with the DPDK driver, jump to step 1.4. Otherwise, please jump to step 1.3.
 
 ```
 Network devices using DPDK-compatible driver
@@ -34,73 +41,105 @@ Network devices using kernel driver
 0000:05:00.1 '82576 Gigabit Network Connection' if=eth0 drv=igb unused=igb_uio *Active*
 ```
 
-1.3 bind the 10G ports to DPDK
+### 1.3 Bind the 10G ports to DPDK
 
-1.3.1 load in your uio linux kernel module
+An example of incorrect bindings is as follows: 
 
-`$sudo modprobe uio`
+```
+Network devices using DPDK-compatible driver
+============================================
+<none>
 
-1.3.2 load in your igb_uio, which is in DPDK kernel module, e.g x86_64-native-linuxapp-gcc 
+Network devices using kernel driver
+===================================
+0000:05:00.0 '82576 Gigabit Network Connection' if=eth0 drv=igb unused=igb_uio *Active*
+0000:05:00.1 '82576 Gigabit Network Connection' if=eth1 drv=igb unused=igb_uio
+0000:07:00.0 '82599EB 10-Gigabit SFI/SFP+ Network Connection' if=eth2 drv=ixgbe unused=igb_uio *Active*
+0000:07:00.1 '82599EB 10-Gigabit SFI/SFP+ Network Connection' if=eth3 drv=ixgbe unused=igb_uio
+```
 
-`$sudo insmod x86_64-native-linuxapp-gcc/kmod/igb_uio.ko`
+In our example above, we see two 10G capable NIC ports that we could use with description '82599EB 10-Gigabit SFI/SFP+ Network Connection'.
 
-if it showed up as alredy bind, use `$sudo rmmod igb_uio`, and then perform `$sudo insmod x86_64-native-linuxapp-gcc/kmod/igb_uio.ko`. 
+One of the two NIC ports, 07:00.0, is active shown by the *Active* at the end of the line. Since the Linux Kernel is currently using that port, network interface eth2, we will not be able to use it with openNetVM. We must first disable the network interface in the Kernel, and then proceed to bind the NIC port to the DPDK Kernel module, igb_uio:
 
-1.3.3 bind the 10G ports to DPDK
+`$ sudo ifconfig eth2 down`
 
-`$sudo ./tools/dpdk_nic_bind.py -b igb_uio 07:00.0`
+Rerun the status command, ./usertools/dpdk-devbind.py --status, to see that it is not active anymore. Once that is done, proceed to bind the NIC port to the DPDK Kenrel module:
 
-`$sudo ./tools/dpdk_nic_bind.py -b igb_uio 07:00.1`
+`$ sudo ./usertools/dpdk-devbind.py -b igb_uio 07:00.0`
 
-1.4 check if g++ and gcc are updated with version higher than 4.7
+Check the status again, `$ ./usertools/dpdk-devbind.py --status`, and assure the output is similar to our example below:
 
-`$g++ --version`
+```
+Network devices using DPDK-compatible driver
+============================================
+0000:07:00.0 '82599EB 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe
 
-`$gcc --version`
+Network devices using kernel driver
+===================================
+0000:05:00.0 '82576 Gigabit Network Connection' if=eth0 drv=igb unused=igb_uio *Active*
+0000:05:00.1 '82576 Gigabit Network Connection' if=eth1 drv=igb unused=igb_uio
+0000:07:00.1 '82599EB 10-Gigabit SFI/SFP+ Network Connection' if=eth3 drv=ixgbe unused=igb_uio
+```
+
+1.4 check if g++ and gcc are updated with version higher than 4.8
+
+`$ g++ --version`
+
+`$ gcc --version`
 
 if not, please add the repository using:
 
-`$sudo add-apt-repository ppa:ubuntu-toolchain-r/test`
+`$ sudo add-apt-repository ppa:ubuntu-toolchain-r/test`
+
+Then, to install it use:
+
+`$ sudo apt-get update`
 
-then, to install it use:
+`$ sudo apt-get install g++-4.8`
 
-`$sudo apt-get update`
+and then change the default compiler to use update-alternatives:
 
-`$sudo apt-get install g++-4.7`
+`$ sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 40 --slave /usr/bin/g++ g++ /usr/bin/g++-4.8`
 
-and then change the default compiler use update-alternatives:
+`$ sudo update-alternatives --config gcc`
 
-`$sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.7 40 --slave /usr/bin/g++ g++ /usr/bin/g++-4.7`
+Install other dependencies with:
 
-`$sudo update-alternatives --config gcc`
+```
+$ sudo apt-get install -y build-essential cmake linux-headers-`uname -r` pciutils libnuma-dev
+$ sudo apt install cmake
+$ sudo apt install libtbb2
+```
 
 
-2. MoonGen Installation 
+## 2. MoonGen Installation
 ===================
 
-2.1 get the resource from github, and checkout the dpdk2.0 branch
+### 2.1 Get the resource from github, and checkout the dpdk19.5 branch
 
-`$git clone https://github.com/emmericp/MoonGen`
+`$ git clone https://github.com/emmericp/MoonGen` 
 
-`$cd MoonGen`
+`$ cd MoonGen`
 
-`$git checkout dpdk2.0`
+`$ git checkout dpdk-19.05`
 
-`$sudo git submodule update --init`
+`$ sudo git submodule update --init`
 
-2.2 build the resource
+### 2.2 Build the resource
 
-`$sudo ./build.sh`
+`$ sudo ./build.sh`
 
-2.3 set up hugetable
+### 2.3 Set up hugetable
 
-`$sudo ./setup-hugetlbfs.sh`
+`$ sudo ./setup-hugetlbfs.sh`
 
-2.4 execute the test, configure the ***quality-of-service-test.lua*** with your destination ip address (ip address for the server you want to sent packets to) in line 60 and line 177, and your source ip address (ip address for the machine you are executing MoonGen on) in line 68 and line 165, and run with command: 
+### 2.4 Execute the test
+Configure the ***quality-of-service-test.lua*** with your destination ip address (ip address for the server you want to sent packets to) and your source ip address (ip address for the machine you are executing MoonGen on), and run with command: 
 
-`$sudo ./build/MoonGen  ./examples/quality-of-service-test.lua 0 1`
+`$ sudo ./build/MoonGen  ./examples/quality-of-service-test.lua 0 1`
 
-and if sample log showed up as following, you are fine, please use ***Ctrl+C*** to stop generating packets:
+and if the sample log outputs the following, your configutation is correct. Use ***Ctrl+C*** to stop generating packets:
 
 ```
 wenhui@nimbnode16:~/MoonGen$ sudo ./build/MoonGen ./examples/quality-of-service-test.lua 0 0

From 73ee8fd4b1a06383c4b4a473f815daea91a4d69b Mon Sep 17 00:00:00 2001
From: Lauren Hahn <62489998+Lhahn01@users.noreply.github.com>
Date: Thu, 26 Aug 2021 12:24:51 -0400
Subject: [PATCH 09/17] [Bug Fix] Fixed Issue #284 Linter Cppcheck

Fixes a version error in the Github actions pipeline.

Commit log:

* Updated cppcheck

* Updated cppcheck install version

* cppcheck updated

* cppcheck install version updated

* Revert "cppcheck updated"

* Updated cppcheck install

* Updated GitHub Action Style documentation
---
 .github/workflows/Linter.yml | 4 ++--
 style/github-action-style.md | 2 +-
 style/install-linters.sh     | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/.github/workflows/Linter.yml b/.github/workflows/Linter.yml
index d265ee554..b5fff4a15 100644
--- a/.github/workflows/Linter.yml
+++ b/.github/workflows/Linter.yml
@@ -24,13 +24,13 @@ jobs:
          chmod +x style/run-lint.sh
          style/run-lint.sh c
   Cppcheck:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-20.04
     steps:
       - name: Checkout
         uses: actions/checkout@v2
       - name: Install Requirements
         run: |
-         sudo apt-get install cppcheck=1.82-1
+         sudo apt-get install cppcheck=1.90-4build1
       - name: User's Lint
         run: |
          chmod +x style/run-lint.sh
diff --git a/style/github-action-style.md b/style/github-action-style.md
index b02c73ba3..93d29ae6a 100644
--- a/style/github-action-style.md
+++ b/style/github-action-style.md
@@ -35,7 +35,7 @@ The Linter can be broken down into a few simple steps.
 
 * The following three linters need to be installed before you are able to run the linter in your terminal. \
 **pylint 2.4.4** \
-**Cppcheck 1.82** \
+**Cppcheck 1.90-4build1** \
 **ShellCheck 0.7.0**
 
 Run **./style/install-linters.sh** to install the different static tools.
diff --git a/style/install-linters.sh b/style/install-linters.sh
index 0a33e66d5..e94a15407 100755
--- a/style/install-linters.sh
+++ b/style/install-linters.sh
@@ -1,8 +1,8 @@
 #!/bin/bash
 # A script to install static analysis tools pylint, shellcheck, and cppcheck.
 
-# Install cppcheck-1.82-1
-sudo apt-get install cppcheck=1.82-1 -y
+# Install cppcheck=1.90-4build1
+sudo apt-get install cppcheck=1.90-4build1 -y
 cppcheck --version
 
 # Install shellcheck-v0.7.0

From 07fc80066b96ab10796747443f326a48d32f6f54 Mon Sep 17 00:00:00 2001
From: Noah Chinitz <62520433+NoahChinitzGWU@users.noreply.github.com>
Date: Tue, 31 Aug 2021 11:36:02 -0400
Subject: [PATCH 10/17] [Unit Test] [Bug Fixes] Add Messaging Unit Tests and
 Fix Issue #293 Message Memory Leak (#296)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This PR includes a new NF called Basic Message Test that tests the functionality of an NF sending messages to itself. This NF was created in response to Issue #293 where a memory leak was found. The memory leak occurred when a message was trying to be sent and the message queue had been filled; the message was never freed and put back into the shared memory pool. Three tests are ran: the first making sure that one message (with the correct data) can be sent to an NF, next it tests if 10 messages can be sent to an NF, and finally it tests to make sure that even if the ring buffer is overflowed that the appropriate messages are received by the NF. Moreover, there were changes made to onvm_sc_common.c in which if there was no packet sent to the function then we would return the first instance ID associated with the given service ID.

Commit log:

* [Update] Fixed Issue #293

* [Update] Deleted dead code

* [Update] Abstracted SID so that it works with whatever SID given

* [Update] Changed name

* [Update] Changed dir name

* [Update] Changed code in regards to PR #296 comments

* [Update] Drop packets

* [Update] Sends more than one msg

* [Update] Working on making output cleaner as well as creating a struct to hold 'global data'

* [Update] Added checking of memory

* [Update] Added working struct. Working on cleaner output and automation of test

* [Update] Added first Test

* [Update] Working on Unit Tests

* [Update] Verson 1 of Tests. Looking at messaging code now

* [Update] Testing fixed. Changing Verbosity

* Delete Makefile

* Delete README.md

* Delete go.sh

* Delete test_messaging.c

* [Update] Ready for Review.

* debug files

* [Update] Test Version 3.0

* restored old go.sh file

* [Update] Version 3.1 of Tests. Added destroy funciton

* [Update] Fixed destroy function. Added NF to examples/Makefile. Still working on hanging execution.

* Hanging occurs when IID != SID

* Update launch.json

* - Added GDB configuration for VSCode.
 - Convert from service ID to instance ID when sending message in /examples/test_messaging
 - Add support to look up instance ID of a message by passing NULL for the packet

* Added function comments

* [Update] fixed file according to PR comments

* Delete launch.json

* Delete tasks.json

* Delete gdb.sh

* [Update] Deleted unnecessary files

* Changed test making sure all messages have been dequeued

* [Update] cleaned up the tests

* [Update] Ran Linter and finalized tests

* [Update] Print out socket ID information for manager and network functions

* Revert "[Update] Print out socket ID information for manager and network functions"

This reverts commit 8ac16b850477194d1394a6d154b620bca4bbcf65.

* [Update] Print out socket ID information for manager and network functions

* [Updated] Fixed tests

* [Update] added 4th test phase

* Revert "[Update] Print out socket ID information for manager and network func…"

* [Update] Changing Workflow

* [Update] Fixed Workflow to be cleaner and more organized

* Update onvm/onvm_nflib/onvm_nflib.c
---
 examples/Makefile                        |   2 +-
 examples/test_messaging/Makefile         |  67 ++++
 examples/test_messaging/README.md        |  26 ++
 examples/test_messaging/go.sh            |  20 ++
 examples/test_messaging/test_messaging.c | 406 +++++++++++++++++++++++
 onvm/onvm_nflib/onvm_nflib.c             |   9 +-
 onvm/onvm_nflib/onvm_nflib.h             |  10 +
 onvm/onvm_nflib/onvm_sc_common.c         |   6 +-
 onvm/onvm_nflib/onvm_sc_common.h         |   6 +-
 9 files changed, 546 insertions(+), 6 deletions(-)
 create mode 100644 examples/test_messaging/Makefile
 create mode 100644 examples/test_messaging/README.md
 create mode 100755 examples/test_messaging/go.sh
 create mode 100644 examples/test_messaging/test_messaging.c
 mode change 100755 => 100644 onvm/onvm_nflib/onvm_nflib.c

diff --git a/examples/Makefile b/examples/Makefile
index 3ce592312..e997c7690 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -40,7 +40,7 @@ $(error "Please define RTE_SDK environment variable")
 endif
 
 # To add new examples, append the directory name to this variable
-examples = bridge basic_monitor simple_forward speed_tester flow_table test_flow_dir aes_encrypt aes_decrypt flow_tracker load_balancer arp_response nf_router scaling_example load_generator payload_scan firewall simple_fwd_tb l2fwd
+examples = bridge basic_monitor simple_forward speed_tester flow_table test_flow_dir aes_encrypt aes_decrypt flow_tracker load_balancer arp_response nf_router scaling_example load_generator payload_scan firewall simple_fwd_tb l2fwd test_messaging
 
 ifeq ($(NDPI_HOME),)
 $(warning "Skipping ndpi_stats NF as NDPI_HOME is not set")
diff --git a/examples/test_messaging/Makefile b/examples/test_messaging/Makefile
new file mode 100644
index 000000000..ffb501a59
--- /dev/null
+++ b/examples/test_messaging/Makefile
@@ -0,0 +1,67 @@
+#                    openNetVM
+#      https://github.com/sdnfv/openNetVM
+#
+# BSD LICENSE
+#
+# Copyright(c)
+#          2015-2021 George Washington University
+#          2015-2021 University of California Riverside
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in
+# the documentation and/or other materials provided with the
+# distribution.
+# The name of the author may not be used to endorse or promote
+# products derived from this software without specific prior
+# written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+
+# Default target, can be overriden by command line or environment
+include $(RTE_SDK)/mk/rte.vars.mk
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+# binary name
+APP = test_messaging
+
+# all source are stored in SRCS-y
+SRCS-y := test_messaging.c
+
+ONVM= $(SRCDIR)/../../onvm
+
+CFLAGS += $(WERROR_FLAGS) -O3 $(USER_FLAGS)
+
+CFLAGS += -I$(ONVM)/onvm_nflib
+CFLAGS += -I$(ONVM)/lib
+LDFLAGS += $(ONVM)/onvm_nflib/$(RTE_TARGET)/libonvm.a
+LDFLAGS += $(ONVM)/lib/$(RTE_TARGET)/lib/libonvmhelper.a -lm
+
+# workaround for a gcc bug with noreturn attribute
+# http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
+ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
+CFLAGS_main.o += -Wno-return-type
+endif
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/test_messaging/README.md b/examples/test_messaging/README.md
new file mode 100644
index 000000000..4064dd115
--- /dev/null
+++ b/examples/test_messaging/README.md
@@ -0,0 +1,26 @@
+Test Messaging
+==
+This is an example NF that acts as a unit test for sending messsages from an NF to itself.
+The NF runs the following unit tests:
+
+ 1. Send/Receive a single message and check for data correctness
+ 2. Send/Receive a batch of messages
+ 3. Send a batch of messages larger than the ring and verify there are no memory pool leaks
+ 
+ Each test is allowed to run for a maximum of 5 seconds.
+ 
+Compilation and Execution
+--
+```
+cd examples
+make
+cd test_messaging
+./go.sh SERVICE_ID
+```
+
+Config File Support
+--
+This NF supports the NF generating arguments from a config file. For
+additional reading, see [Examples.md](../../docs/Examples.md)
+
+See `../example_config.json` for all possible options that can be set.
\ No newline at end of file
diff --git a/examples/test_messaging/go.sh b/examples/test_messaging/go.sh
new file mode 100755
index 000000000..03fc3bb36
--- /dev/null
+++ b/examples/test_messaging/go.sh
@@ -0,0 +1,20 @@
+#!/bin/bash
+
+#The go.sh script is a convinient way to run start_nf.sh without specifying NF_NAME
+
+NF_DIR=${PWD##*/}
+
+if [ ! -f ../start_nf.sh ]; then
+  echo "ERROR: The ./go.sh script can only be used from the NF folder"
+  echo "If running from other directory use examples/start_nf.sh"
+  exit 1
+fi
+
+# only check for running manager if not in Docker
+if [[ -z $(pgrep -u root -f "/onvm/onvm_mgr/.*/onvm_mgr") ]] && ! grep -q "docker" /proc/1/cgroup
+then
+    echo "NF cannot start without a running manager"
+    exit 1
+fi
+
+../start_nf.sh "$NF_DIR" "$@"
diff --git a/examples/test_messaging/test_messaging.c b/examples/test_messaging/test_messaging.c
new file mode 100644
index 000000000..6fbf4edba
--- /dev/null
+++ b/examples/test_messaging/test_messaging.c
@@ -0,0 +1,406 @@
+/*********************************************************************
+ *                     openNetVM
+ *              https://sdnfv.github.io
+ *
+ *   BSD LICENSE
+ *
+ *   Copyright(c)
+ *            2015-2021 George Washington University
+ *            2015-2021 University of California Riverside
+ *   All rights reserved.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * The name of the author may not be used to endorse or promote
+ *       products derived from this software without specific prior
+ *       written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * test_messaging.c - unit test to ensure NFs can send messages to themselves
+ ********************************************************************/
+
+#include <errno.h>
+#include <getopt.h>
+#include <inttypes.h>
+#include <stdarg.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/queue.h>
+#include <unistd.h>
+
+#include <rte_common.h>
+#include <rte_ip.h>
+#include <rte_mbuf.h>
+#include <rte_mempool.h>
+#include <rte_ring.h>
+#include <rte_malloc.h>
+
+#include "onvm_nflib.h"
+#include "onvm_pkt_helper.h"
+#include "onvm_common.h"
+
+#define NF_TAG "test_messaging"
+#define MAGIC_NUMBER 11
+
+#define TEST_FAILED -1
+#define TEST_INCOMPLETE 0
+#define TEST_PASSED 1
+
+#define TEST_TIME_LIMIT 5
+
+struct test_msg_data{
+        int tests_passed;
+        int test_phase;
+        int test_msg_count;
+        int ring_count;
+        int mempool_count;
+        uint16_t address;
+        struct rte_mempool* msg_pool;
+        struct rte_ring *msg_q;
+        int test_status[3];
+        uint64_t start_time;
+};
+
+void
+nf_setup(struct onvm_nf_local_ctx *nf_local_ctx);
+
+void
+nf_msg_handler(void *msg_data, struct onvm_nf_local_ctx *nf_local_ctx);
+
+/*
+ * Frees memory allocated to the test_msg_data struct
+ */ 
+static void
+destroy_test_msg_data(struct test_msg_data **test_msg_data) {
+        if (test_msg_data == NULL) {
+                return;
+        }
+        rte_free(*test_msg_data);
+        (*test_msg_data) = NULL;
+        return;
+}
+
+/*
+ * Clears the message queues after each test
+ */
+static void
+clear_msg_queue(struct test_msg_data *test_state) {
+        void* msg = NULL;
+        while (rte_ring_count(test_state->msg_q) != 0) {
+                rte_ring_dequeue(test_state->msg_q, &msg);
+                rte_mempool_put(test_state->msg_pool, (void *)msg);
+        }
+        rte_free(msg);
+}
+
+/*
+ * Print a usage message
+ */
+static void
+usage(const char *progname) {
+        printf("Usage:\n");
+        printf("%s [EAL args] -- [NF_LIB args]\n", progname);
+}
+
+/*
+ * Parse the application arguments.
+ */
+static int
+parse_app_args(int argc, char *argv[], const char *progname, __attribute__((unused)) struct onvm_nf *nf) {
+        int c;
+        while ((c = getopt(argc, argv, "p:")) != -1) {
+                switch (c) {
+                        case '?':
+                                usage(progname);
+                                if (optopt == 'p')
+                                        RTE_LOG(INFO, APP, "Option -%c requires an argument.\n", optopt);
+                                else if (isprint(optopt))
+                                        RTE_LOG(INFO, APP, "Unknown option `-%c'.\n", optopt);
+                                else
+                                        RTE_LOG(INFO, APP, "Unknown option character `\\x%x'.\n", optopt);
+                                return -1;
+                        default:
+                                usage(progname);
+                                return -1;
+                }
+        }
+        return optind;
+}
+
+/*
+ * Sets up the NF and state data
+ */
+void
+nf_setup(struct onvm_nf_local_ctx *nf_local_ctx) {
+        struct rte_mempool *pktmbuf_pool;
+        struct test_msg_data *test_state;
+        static struct rte_mempool *nf_msg_pool;
+        pktmbuf_pool = rte_mempool_lookup(PKTMBUF_POOL_NAME);
+        if (pktmbuf_pool == NULL) {
+                onvm_nflib_stop(nf_local_ctx);
+                rte_exit(EXIT_FAILURE, "Cannot find mbuf pool!\n");
+        }
+        nf_msg_pool = rte_mempool_lookup(_NF_MSG_POOL_NAME);
+        if (nf_msg_pool == NULL)
+                rte_exit(EXIT_FAILURE, "No NF Message mempool - bye\n");
+
+        test_state = (struct test_msg_data *)rte_malloc(NULL, sizeof(struct test_msg_data), 0);
+        test_state->tests_passed = 0;
+        test_state->test_phase = 0;
+        test_state->address = nf_local_ctx->nf->service_id;
+        test_state->msg_pool = nf_msg_pool;
+        test_state->msg_q = nf_local_ctx->nf->msg_q;
+        test_state->ring_count = 0;
+        test_state->mempool_count = 0;
+        test_state->test_msg_count = 0;
+        test_state->start_time = rte_get_tsc_cycles();
+        memset(test_state->test_status, TEST_INCOMPLETE, sizeof(int)*3);
+        nf_local_ctx->nf->data = (void *)test_state;
+}
+
+/*
+ * Initiates each test phase
+ */
+static int
+test_handler(struct onvm_nf_local_ctx *nf_local_ctx) {
+        struct test_msg_data *test_state;
+        test_state = (struct test_msg_data *)nf_local_ctx->nf->data;
+
+        if (( (rte_get_tsc_cycles() - test_state->start_time) / rte_get_timer_hz()) > TEST_TIME_LIMIT) {
+                // it took too long
+                // we failed this test
+                test_state->test_status[test_state->test_phase - 1] = TEST_FAILED;
+                // reset start time clock
+                test_state->start_time = rte_get_tsc_cycles();
+                // clear ring and move on to next test
+                clear_msg_queue(test_state);
+                test_state->test_phase++;
+        }
+
+        // If we haven't started any tests yet, start phase 1
+        if (0 == test_state->test_phase) {
+                printf("\nTEST MESSAGING STARTED\n");
+                printf("---------------------------\n");
+                clear_msg_queue(test_state);
+                test_state->mempool_count = rte_mempool_avail_count(test_state->msg_pool);
+                int *msg_int;
+                msg_int = (int*)rte_malloc(NULL, sizeof(int), 0);
+                if (msg_int == NULL) {
+                        printf("Message was not able to be malloc'd\n");
+                }
+                *msg_int = MAGIC_NUMBER;
+                int ret = onvm_nflib_send_msg_to_nf(test_state->address, msg_int);
+                if (ret != 0) {
+                        printf("Message was unable to be sent\n");
+                }
+                test_state->test_phase++;
+        } else if (1 == test_state->test_phase) {
+                printf("TEST 1: Send/Receive One Message with correct data...\n");
+                printf("---------------------------\n");
+                if (TEST_PASSED == test_state->test_status[0]) {
+                        printf("PASSED TEST 1\n");
+                        printf("---------------------------\n");
+                } else if (TEST_FAILED == test_state->test_status[0]) {
+                        printf("FAILED TEST 1\n");
+                        printf("---------------------------\n");
+                }
+                if (TEST_INCOMPLETE != test_state->test_status[0]) {
+                        printf("TEST 2: Send/Receive Multiple Messages...\n");
+                        printf("---------------------------\n");
+                        clear_msg_queue(test_state);
+                        for (int i = 0; i < 10; i++) {
+                                int* msg_int = (int*)rte_malloc(NULL, sizeof(int), 0);
+                                *msg_int = i;
+                                onvm_nflib_send_msg_to_nf(test_state->address, msg_int);
+                        }
+                        test_state->test_phase++;
+                }
+        } else if (2 == test_state->test_phase) {
+                if (TEST_PASSED == test_state->test_status[1]) {
+                        printf("PASSED TEST 2\n");
+                        printf("---------------------------\n");
+
+                } else if (TEST_FAILED == test_state->test_status[1]) {
+                        printf("FAILED TEST 2\n");
+                        printf("---------------------------\n");
+                }
+                if (TEST_INCOMPLETE != test_state->test_status[1]) {
+                        printf("TEST 3: Message Ring Overflow...\n");
+                        printf("---------------------------\n");
+                        clear_msg_queue(test_state);
+                        for (int i = 0; i < (int)(rte_ring_get_size(test_state->msg_q) + 2); i++) {
+                                int* msg_int = (int*)rte_malloc(NULL, sizeof(int), 0);
+                                *msg_int = i;
+                                onvm_nflib_send_msg_to_nf(test_state->address, msg_int);
+                        }
+                        test_state->test_phase++;
+                }
+        } else if (3 == test_state->test_phase) {
+                if (TEST_PASSED == test_state->test_status[2]) {
+                        printf("PASSED TEST 3\n");
+                        printf("---------------------------\n");
+                } else if (TEST_FAILED == test_state->test_status[2]) {
+                        printf("FAILED TEST 3\n");
+                        printf("---------------------------\n");
+                }
+                if (TEST_INCOMPLETE != test_state->test_status[2]) {
+                        clear_msg_queue(test_state);
+                        test_state->test_phase++;
+                }
+        } else if (4 == test_state->test_phase) {
+                printf("Passed %d/3 Tests\n", test_state->tests_passed);
+                return 1;
+        }
+        return 0;
+}
+
+/*
+ * Checks for test correctness
+ */
+void
+nf_msg_handler(void *msg_data, struct onvm_nf_local_ctx *nf_local_ctx) {
+        struct test_msg_data *test_state;
+        test_state = (struct test_msg_data *)nf_local_ctx->nf->data;
+        // Tests if one message can be sent to itself
+        if (1 == test_state->test_phase) {
+                test_state->test_msg_count++;
+                if (1 == test_state->test_msg_count) {
+                        if (rte_ring_count(test_state->msg_q) != 0) {
+                                printf("FAILURE: Shouldn't be any messages left, but there are %d in the ring. This may cause future tests to fail.\n", rte_ring_count(test_state->msg_q));
+                                printf("---------------------------\n");
+                                test_state->test_status[0] = TEST_FAILED;
+                        } else {
+                                if (*((int *)msg_data) == MAGIC_NUMBER) {
+                                        test_state->tests_passed++;
+                                        test_state->test_status[0] = TEST_PASSED;
+                                        test_state->test_msg_count = 0;
+                                } else {
+                                        printf("FAILURE: Received %d instead of %d\n", *((int *)msg_data), MAGIC_NUMBER);
+                                        printf("---------------------------\n");
+                                        test_state->test_status[0] = TEST_FAILED;
+                                }
+                        }
+                }
+        } else if (2 == test_state->test_phase) { // Tests if multiple messages can be sent to itself
+                if (*((int *)msg_data) != test_state->test_msg_count) {
+                        printf("FAILURE: Received %d instead of %d\n", *((int *)msg_data), test_state->test_msg_count);
+                        printf("---------------------------\n");
+                        test_state->test_status[1] = TEST_FAILED;
+                } else {
+                        test_state->test_msg_count++;
+                }
+                if (10 == test_state->test_msg_count) {
+                        if (rte_ring_count(test_state->msg_q) != 0) {
+                                printf("FAILURE: Shouldn't be any messages left, but there are %d in the ring. This may cause future tests to fail.\n", rte_ring_count(test_state->msg_q));
+                                printf("---------------------------\n");
+                                test_state->test_status[1] = TEST_FAILED;
+                        } else {
+                                test_state->tests_passed++;
+                                test_state->test_status[1] = TEST_PASSED;
+                                test_state->test_msg_count = 0;
+                        }
+                }
+        } else if (3 == test_state->test_phase) { // Tests to see if even with a ring overflow it can still handle the messages
+                if (*((int *)msg_data) != test_state->test_msg_count) {
+                        printf("FAILURE: Received %d instead of %d\n", *((int *)msg_data), test_state->test_msg_count);
+                        printf("---------------------------\n");
+                        test_state->test_status[2] = TEST_FAILED;
+                } else {
+                        test_state->test_msg_count++;
+                }
+                if ((int)(rte_ring_get_size(test_state->msg_q) - 1) == test_state->test_msg_count) {
+                        if (rte_ring_count(test_state->msg_q) != 0) {
+                                printf("FAILURE: Shouldn't be any messages left, but there are %d in the ring. This may cause future tests to fail.\n", rte_ring_count(test_state->msg_q));
+                                printf("---------------------------\n");
+                                test_state->test_status[2] = TEST_FAILED;
+                        } else if ((int)rte_mempool_avail_count(test_state->msg_pool) == test_state->mempool_count - 1) {
+                                // only pass if there wasn't a memory leak
+                                test_state->tests_passed++;
+                                test_state->test_status[2] = TEST_PASSED;
+                                test_state->test_msg_count = 0;
+                        } else {
+                                printf("FAILURE: %d messages have not been deallocated back to the memory pool.\n", test_state->mempool_count - rte_mempool_avail_count(test_state->msg_pool));
+                                printf("---------------------------\n");
+                                test_state->test_status[2] = TEST_FAILED;
+                        }
+                }
+        }
+        printf("---------------------------\n");
+        rte_free(msg_data);
+}
+
+/*
+ * Not concerned with packets, so they are dropped.
+ */
+static int
+packet_handler(__attribute__((unused)) struct rte_mbuf *pkt, struct onvm_pkt_meta *meta,
+               __attribute__((unused)) struct onvm_nf_local_ctx *nf_local_ctx) {
+        meta->action = ONVM_NF_ACTION_DROP;
+        return 0;
+}
+
+/*
+ * Creates function table and local context. Runs NF.
+ */
+int
+main(int argc, char *argv[]) {
+        int arg_offset;
+        struct onvm_nf_local_ctx *nf_local_ctx;
+        struct onvm_nf_function_table *nf_function_table;
+        const char *progname = argv[0];
+
+        nf_local_ctx = onvm_nflib_init_nf_local_ctx();
+        onvm_nflib_start_signal_handler(nf_local_ctx, NULL);
+
+        nf_function_table = onvm_nflib_init_nf_function_table();
+        nf_function_table->pkt_handler = &packet_handler;
+        nf_function_table->setup = &nf_setup;
+        nf_function_table->msg_handler = &nf_msg_handler;
+        nf_function_table->user_actions = &test_handler;
+
+        if ((arg_offset = onvm_nflib_init(argc, argv, NF_TAG, nf_local_ctx, nf_function_table)) < 0) {
+                onvm_nflib_stop(nf_local_ctx);
+                if (arg_offset == ONVM_SIGNAL_TERMINATION) {
+                        printf("Exiting due to user termination\n");
+                        return 0;
+                } else {
+                        rte_exit(EXIT_FAILURE, "Failed ONVM init\n");
+                }
+        }
+
+        argc -= arg_offset;
+        argv += arg_offset;
+
+        if (parse_app_args(argc, argv, progname, nf_local_ctx->nf) < 0) {
+                destroy_test_msg_data(nf_local_ctx->nf->data);
+                onvm_nflib_stop(nf_local_ctx);
+                rte_exit(EXIT_FAILURE, "Invalid command-line arguments\n");
+        }
+
+        onvm_nflib_run(nf_local_ctx);
+        destroy_test_msg_data((struct test_msg_data**)&nf_local_ctx->nf->data);
+        onvm_nflib_stop(nf_local_ctx);
+
+        return 0;
+}
diff --git a/onvm/onvm_nflib/onvm_nflib.c b/onvm/onvm_nflib/onvm_nflib.c
old mode 100755
new mode 100644
index 2e5ecb33b..50f44320e
--- a/onvm/onvm_nflib/onvm_nflib.c
+++ b/onvm/onvm_nflib/onvm_nflib.c
@@ -727,7 +727,14 @@ onvm_nflib_send_msg_to_nf(uint16_t dest, void *msg_data) {
         msg->msg_type = MSG_FROM_NF;
         msg->msg_data = msg_data;
 
-        return rte_ring_enqueue(nfs[dest].msg_q, (void*)msg);
+        uint16_t instance_id = onvm_sc_service_to_nf_map(dest, NULL);
+        ret = rte_ring_enqueue(nfs[instance_id].msg_q, (void*)msg);
+        if (ret != 0) {
+                RTE_LOG(WARNING, APP, "Destination NF ring is full! Unable to enqueue msg to ring\n");
+                rte_mempool_put(nf_msg_pool, (void*)msg);
+                return ret;
+        }
+        return 0;
 }
 
 void
diff --git a/onvm/onvm_nflib/onvm_nflib.h b/onvm/onvm_nflib/onvm_nflib.h
index 858f95e45..e83b0073c 100644
--- a/onvm/onvm_nflib/onvm_nflib.h
+++ b/onvm/onvm_nflib/onvm_nflib.h
@@ -188,6 +188,16 @@ onvm_nflib_start_nf(struct onvm_nf_local_ctx *nf_local_ctx, struct onvm_nf_init_
 int
 onvm_nflib_handle_msg(struct onvm_nf_msg *msg, struct onvm_nf_local_ctx *nf_local_ctx);
 
+/**
+ * Sends a message to another NF
+ *
+ * @param dest_nf
+ *    The destination NF's service ID
+ * @param msg_data
+ *    Pointer to the message that is sent
+ * @return
+ *    0 on success, or a negative value on error
+ */
 int
 onvm_nflib_send_msg_to_nf(uint16_t dest_nf, void *msg_data);
 
diff --git a/onvm/onvm_nflib/onvm_sc_common.c b/onvm/onvm_nflib/onvm_sc_common.c
index 6dcd506c4..45370cd1e 100644
--- a/onvm/onvm_nflib/onvm_sc_common.c
+++ b/onvm/onvm_nflib/onvm_sc_common.c
@@ -55,8 +55,10 @@ onvm_sc_service_to_nf_map(uint16_t service_id, struct rte_mbuf *pkt) {
         if (num_nfs_available == 0)
                 return 0;
 
-        if (pkt == NULL)
-                return 0;
+        if (pkt == NULL) {
+                uint16_t instance_id = services[service_id][0];
+                return instance_id;
+        }
 
         uint16_t instance_index = pkt->hash.rss % num_nfs_available;
         uint16_t instance_id = services[service_id][instance_index];
diff --git a/onvm/onvm_nflib/onvm_sc_common.h b/onvm/onvm_nflib/onvm_sc_common.h
index d0ed4a032..84efbde75 100644
--- a/onvm/onvm_nflib/onvm_sc_common.h
+++ b/onvm/onvm_nflib/onvm_sc_common.h
@@ -51,10 +51,12 @@ extern uint16_t **services;
 extern uint16_t *nf_per_service_count;
 
 /********************************Interfaces***********************************/
-
+/* Returns the instance ID associated with the given service ID and packet.
+   If using for a message, given packet is NULL and the function returns 
+   the first instance ID associated with the given service ID for messaging */
 uint16_t
 onvm_sc_service_to_nf_map(uint16_t service_id,
-                          struct rte_mbuf *pkt); /*, uint16_t *nf_per_service_count, uint16_t **services);*/
+                          struct rte_mbuf *pkt);
 
 /* append a entry to serivce chain, 0 means appending successful, 1 means failed*/
 int

From d6483ede290ed5b7f9af651d3d03327f01b56de4 Mon Sep 17 00:00:00 2001
From: Noah Chinitz <62520433+NoahChinitzGWU@users.noreply.github.com>
Date: Thu, 2 Sep 2021 11:24:34 -0400
Subject: [PATCH 11/17] [Logging] Log physical socket in use

Logs the physical socket ID whenever logical core ID is also printed out, and outputs socket ID to CSV in the NF directory.

Commit log:
* Print out physical socket ID
---
 onvm/onvm_mgr/main.c         | 38 ++++++++++++++++++------------------
 onvm/onvm_nflib/onvm_nflib.c |  6 ++++--
 2 files changed, 23 insertions(+), 21 deletions(-)

diff --git a/onvm/onvm_mgr/main.c b/onvm/onvm_mgr/main.c
index 4c1bbd53d..99dee45af 100644
--- a/onvm/onvm_mgr/main.c
+++ b/onvm/onvm_mgr/main.c
@@ -84,7 +84,7 @@ master_thread_main(void) {
         const uint64_t start_time = rte_get_tsc_cycles();
         uint64_t total_rx_pkts;
 
-        RTE_LOG(INFO, APP, "Core %d: Running master thread\n", rte_lcore_id());
+        RTE_LOG(INFO, APP, "Socket %d, Core %d: Running master thread\n", rte_socket_id(), rte_lcore_id());
 
         if (stats_destination == ONVM_STATS_WEB) {
                 RTE_LOG(INFO, APP, "ONVM stats can be viewed through the web console\n");
@@ -130,7 +130,7 @@ master_thread_main(void) {
         rte_pdump_uninit();
 #endif
 
-        RTE_LOG(INFO, APP, "Core %d: Initiating shutdown sequence\n", rte_lcore_id());
+        RTE_LOG(INFO, APP, "Socket %d, Core %d: Initiating shutdown sequence\n", rte_socket_id(), rte_lcore_id());
 
         /* Stop all RX and TX threads */
         worker_keep_running = 0;
@@ -140,7 +140,7 @@ master_thread_main(void) {
                 if (nfs[i].status != NF_RUNNING)
                         continue;
 
-                RTE_LOG(INFO, APP, "Core %d: Notifying NF %" PRIu16 " to shut down\n", rte_lcore_id(), i);
+                RTE_LOG(INFO, APP, "Socket %d, Core %d: Notifying NF %" PRIu16 " to shut down\n", rte_socket_id(), rte_lcore_id(), i);
                 onvm_nf_send_msg(i, MSG_STOP, NULL);
 
                 /* If in shared core mode NFs might be sleeping */
@@ -154,13 +154,13 @@ master_thread_main(void) {
         /* Wait to process all exits */
         for (shutdown_iter_count = 0; shutdown_iter_count < MAX_SHUTDOWN_ITERS && num_nfs > 0; shutdown_iter_count++) {
                 onvm_nf_check_status();
-                RTE_LOG(INFO, APP, "Core %d: Waiting for %" PRIu16 " NFs to exit\n", rte_lcore_id(), num_nfs);
+                RTE_LOG(INFO, APP, "Socket %d, Core %d: Waiting for %" PRIu16 " NFs to exit\n", rte_socket_id(), rte_lcore_id(), num_nfs);
                 sleep(sleeptime);
         }
 
         if (num_nfs > 0) {
-                RTE_LOG(INFO, APP, "Core %d: Up to %" PRIu16 " NFs may still be running and must be killed manually\n",
-                        rte_lcore_id(), num_nfs);
+                RTE_LOG(INFO, APP, "Socket %d, Core %d: Up to %" PRIu16 " NFs may still be running and must be killed manually\n",
+                        rte_socket_id(), rte_lcore_id(), num_nfs);
         }
 
         /* Clean up the shared memory */
@@ -171,7 +171,7 @@ master_thread_main(void) {
                 }
         }
 
-        RTE_LOG(INFO, APP, "Core %d: Master thread done\n", rte_lcore_id());
+        RTE_LOG(INFO, APP, "Socket %d, Core %d: Master thread done\n", rte_socket_id(), rte_lcore_id());
 }
 
 /*
@@ -186,7 +186,7 @@ rx_thread_main(void *arg) {
         cur_lcore = rte_lcore_id();
 
         onvm_stats_gen_event_info("Rx Start", ONVM_EVENT_WITH_CORE, &cur_lcore);
-        RTE_LOG(INFO, APP, "Core %d: Running RX thread for RX queue %d\n", cur_lcore, rx_mgr->id);
+        RTE_LOG(INFO, APP, "Socket %d, Core %d: Running RX thread for RX queue %d\n", rte_socket_id(), cur_lcore, rx_mgr->id);
 
         for (; worker_keep_running;) {
                 /* Read ports */
@@ -206,7 +206,7 @@ rx_thread_main(void *arg) {
                 }
         }
 
-        RTE_LOG(INFO, APP, "Core %d: RX thread done\n", rte_lcore_id());
+        RTE_LOG(INFO, APP, "Socket %d, Core %d: RX thread done\n", rte_socket_id(), rte_lcore_id());
         return 0;
 }
 
@@ -220,10 +220,10 @@ tx_thread_main(void *arg) {
 
         onvm_stats_gen_event_info("Tx Start", ONVM_EVENT_WITH_CORE, &cur_lcore);
         if (tx_mgr->tx_thread_info->first_nf == tx_mgr->tx_thread_info->last_nf - 1) {
-                RTE_LOG(INFO, APP, "Core %d: Running TX thread for NF %d\n", cur_lcore,
+                RTE_LOG(INFO, APP, "Socket %d, Core %d: Running TX thread for NF %d\n", rte_socket_id(), cur_lcore,
                         tx_mgr->tx_thread_info->first_nf);
         } else if (tx_mgr->tx_thread_info->first_nf < tx_mgr->tx_thread_info->last_nf) {
-                RTE_LOG(INFO, APP, "Core %d: Running TX thread for NFs %d to %d\n", cur_lcore,
+                RTE_LOG(INFO, APP, "Socket %d, Core %d: Running TX thread for NFs %d to %d\n", rte_socket_id(), cur_lcore,
                         tx_mgr->tx_thread_info->first_nf, tx_mgr->tx_thread_info->last_nf - 1);
         }
 
@@ -250,7 +250,7 @@ tx_thread_main(void *arg) {
                 onvm_pkt_flush_all_nfs(tx_mgr, NULL);
         }
 
-        RTE_LOG(INFO, APP, "Core %d: TX thread done\n", rte_lcore_id());
+        RTE_LOG(INFO, APP, "Socket %d, Core %d: TX thread done\n", rte_socket_id(), rte_lcore_id());
         return 0;
 }
 
@@ -276,10 +276,10 @@ wakeup_thread_main(void *arg) {
         struct wakeup_thread_context *wakeup_ctx = (struct wakeup_thread_context *)arg;
 
         if (wakeup_ctx->first_nf == wakeup_ctx->last_nf - 1) {
-                RTE_LOG(INFO, APP, "Core %d: Running Wakeup thread for NF %d\n", rte_lcore_id(),
+                RTE_LOG(INFO, APP, "Socket %d, Core %d: Running Wakeup thread for NF %d\n", rte_socket_id(), rte_lcore_id(),
                         wakeup_ctx->first_nf);
         } else if (wakeup_ctx->first_nf < wakeup_ctx->last_nf) {
-                RTE_LOG(INFO, APP, "Core %d: Running Wakeup thread for NFs %d to %d\n", rte_lcore_id(),
+                RTE_LOG(INFO, APP, "Socket %d, Core %d: Running Wakeup thread for NFs %d to %d\n", rte_socket_id(), rte_lcore_id(),
                         wakeup_ctx->first_nf, wakeup_ctx->last_nf - 1);
         }
 
@@ -379,7 +379,7 @@ main(int argc, char *argv[]) {
         /* Offset cur_lcore to start assigning TX cores */
         cur_lcore += (rx_lcores - 1);
 
-        RTE_LOG(INFO, APP, "%d cores available in total\n", rte_lcore_count());
+        RTE_LOG(INFO, APP, "%d Sockets, %d Cores available in total\n", rte_socket_count(), rte_lcore_count());
         RTE_LOG(INFO, APP, "%d cores available for handling manager RX queues\n", rx_lcores);
         RTE_LOG(INFO, APP, "%d cores available for handling TX queues\n", tx_lcores);
         if (ONVM_NF_SHARE_CORES)
@@ -432,7 +432,7 @@ main(int argc, char *argv[]) {
                 tx_mgr[i]->tx_thread_info->last_nf = RTE_MIN((i + 1) * nfs_per_tx + 1, (unsigned)MAX_NFS);
                 cur_lcore = rte_get_next_lcore(cur_lcore, 1, 1);
                 if (rte_eal_remote_launch(tx_thread_main, (void *)tx_mgr[i], cur_lcore) == -EBUSY) {
-                        RTE_LOG(ERR, APP, "Core %d is already busy, can't use for nf %d TX\n", cur_lcore,
+                        RTE_LOG(ERR, APP, "Socket %d, Core %d is already busy, can't use for nf %d TX\n", rte_socket_id(), cur_lcore,
                                 tx_mgr[i]->tx_thread_info->first_nf);
                         onvm_main_free(tx_lcores,rx_lcores, tx_mgr, rx_mgr, wakeup_ctx);
                         return -1;
@@ -454,7 +454,7 @@ main(int argc, char *argv[]) {
                 }
                 cur_lcore = rte_get_next_lcore(cur_lcore, 1, 1);
                 if (rte_eal_remote_launch(rx_thread_main, (void *)rx_mgr[i], cur_lcore) == -EBUSY) {
-                        RTE_LOG(ERR, APP, "Core %d is already busy, can't use for RX queue id %d\n", cur_lcore,
+                        RTE_LOG(ERR, APP, "Socket %d, Core %d is already busy, can't use for RX queue id %d\n", rte_socket_id(), cur_lcore,
                                 rx_mgr[i]->id);
                         onvm_main_free(tx_lcores,rx_lcores, tx_mgr, rx_mgr, wakeup_ctx);
                         return -1;
@@ -472,8 +472,8 @@ main(int argc, char *argv[]) {
                         wakeup_ctx[i]->last_nf = RTE_MIN((i + 1) * nfs_per_wakeup_thread + 1, (unsigned)MAX_NFS);
                         cur_lcore = rte_get_next_lcore(cur_lcore, 1, 1);
                         if (rte_eal_remote_launch(wakeup_thread_main, (void*)wakeup_ctx[i], cur_lcore) == -EBUSY) {
-                                RTE_LOG(ERR, APP, "Core %d is already busy, can't use for nf %d wakeup thread\n",
-                                        cur_lcore, wakeup_ctx[i]->first_nf);
+                                RTE_LOG(ERR, APP, "Socket %d, Core %d is already busy, can't use for nf %d wakeup thread\n",
+                                        rte_socket_id(), cur_lcore, wakeup_ctx[i]->first_nf);
                                 onvm_main_free(tx_lcores, rx_lcores, tx_mgr, rx_mgr, wakeup_ctx);
                                 return -1;
                         }
diff --git a/onvm/onvm_nflib/onvm_nflib.c b/onvm/onvm_nflib/onvm_nflib.c
index 50f44320e..4e95ed2a2 100644
--- a/onvm/onvm_nflib/onvm_nflib.c
+++ b/onvm/onvm_nflib/onvm_nflib.c
@@ -533,7 +533,7 @@ onvm_nflib_start_nf(struct onvm_nf_local_ctx *nf_local_ctx, struct onvm_nf_init_
 
         RTE_LOG(INFO, APP, "Using Instance ID %d\n", nf->instance_id);
         RTE_LOG(INFO, APP, "Using Service ID %d\n", nf->service_id);
-        RTE_LOG(INFO, APP, "Running on core %d\n", nf->thread_info.core);
+        RTE_LOG(INFO, APP, "Running on Socket %d, Core %d\n", rte_socket_id(), nf->thread_info.core);
 
         if (nf->flags.time_to_live)
                 RTE_LOG(INFO, APP, "Time to live set to %u\n", nf->flags.time_to_live);
@@ -1328,7 +1328,7 @@ onvm_nflib_stats_summary_output(uint16_t id) {
         const char clr[] = {27, '[', '2', 'J', '\0'};
         const char topLeft[] = {27, '[', '1', ';', '1', 'H', '\0'};
         const char *csv_suffix = "_stats.csv";
-        const char *csv_stats_headers = "NF tag, NF instance ID, NF service ID, NF assigned core, RX total,"
+        const char *csv_stats_headers = "NF tag, NF instance ID, NF service ID, NF assigned socket, NF assigned core, RX total,"
                                         "RX total dropped, TX total, TX total dropped, NF sent out, NF sent to NF,"
                                         "NF dropped, NF next, NF tx buffered, NF tx buffered, NF tx returned";
         const uint64_t rx = nfs[id].stats.rx;
@@ -1355,6 +1355,7 @@ onvm_nflib_stats_summary_output(uint16_t id) {
         printf("NF tag: %s\n", nf_tag);
         printf("NF instance ID: %d\n", instance_id);
         printf("NF service ID: %d\n", service_id);
+        printf("NF assigned socket: %d\n", rte_socket_id());
         printf("NF assigned core: %d\n", core);
         printf("----------------------------------------------------\n");
         printf("RX total: %ld\n", rx);
@@ -1397,6 +1398,7 @@ onvm_nflib_stats_summary_output(uint16_t id) {
         fprintf(csv_fp, "\n%s", nf_tag);
         fprintf(csv_fp, ", %d", instance_id);
         fprintf(csv_fp, ", %d", service_id);
+        fprintf(csv_fp, ", %d", rte_socket_id());
         fprintf(csv_fp, ", %d", core);
         fprintf(csv_fp, ", %ld", rx);
         fprintf(csv_fp, ", %ld", rx_drop);

From e101afa945cc1a8704c1ea530f3430b86eec724a Mon Sep 17 00:00:00 2001
From: Benjamin De Vierno <54540257+bdevierno1@users.noreply.github.com>
Date: Mon, 25 Oct 2021 14:12:34 -0400
Subject: [PATCH 12/17] [New NF] L3switch(l3fwd)

A l3 forwarding NF based on DPDK's example. There are two modes - longest prefix match table and hash table lookup.
There are a few differences between this version and DPDK's. Use of openNetVM's flow API was broadly used. This allows data to be stored with each rule or flow when added to the table. This is in contrast to DPDK's version where it uses the key value to get an index. The index value is then used to perform a lookup in an array.

Commit log:

* dpdk: Update ONVM to use DPDK v20.05

* onvm: Update onvm to use updated DPDK APIs

* Update the onvm manager, nflib and examples to use the
updated DPDK APIs.

* Update install.sh to enable the igb_uio module build
  (CONFIG_RTE_EAL_IGB_UIO is disabled by default since v20.02)

* switch strncpy to memcpy

strncpy raises (precautionary) werrors during onvm compilation with ubuntu 20 compiler (gcc 9); gnu/gcc recommends use of memcpy to avoid warning

* Layer 3 switch

* Layer 3 switch

* Added Readme.

* Edit makefile

* Edit makefile

* Add symbolic link

* Free tables upon shutdown

* Update free function

* Edit readme.

* rename handler

* update pktgen-dpdk submodule to v20.03

* pktgen submodule updates

* pktgen submodule to 807b4d

* Table pointers to use global variable.

* modify pktgen config args

lua pktgen.set_mac() now takes three args: port, src|dst, mac addr

* remove pktgen blacklist

blacklists outdated as of pktgen v2.7.0

* UPdate dpdk functions to use latest.

* Rename files

* Remove global variables

Co-authored-by: Vivek Jain <jain.vivek.anand@gmail.com>
Co-authored-by: Sreya Nalla <sreyanalla@gwu.edu>
Co-authored-by: Benjamin De Vierno <bdevierno97@gwu.edu>
Co-authored-by: sreyan <sreyan@localhost>
Co-authored-by: Dennis Afanasev <dennis.afa@gmail.com>
---
 examples/Makefile          |   2 +-
 examples/l3fwd/Makefile    |  62 +++++++
 examples/l3fwd/README.md   |  39 +++++
 examples/l3fwd/go.sh       |   1 +
 examples/l3fwd/l3fwd.c     | 348 +++++++++++++++++++++++++++++++++++++
 examples/l3fwd/l3fwd.h     |  91 ++++++++++
 examples/l3fwd/l3fwd_em.c  | 154 ++++++++++++++++
 examples/l3fwd/l3fwd_lpm.c | 132 ++++++++++++++
 8 files changed, 828 insertions(+), 1 deletion(-)
 create mode 100755 examples/l3fwd/Makefile
 create mode 100644 examples/l3fwd/README.md
 create mode 120000 examples/l3fwd/go.sh
 create mode 100755 examples/l3fwd/l3fwd.c
 create mode 100644 examples/l3fwd/l3fwd.h
 create mode 100644 examples/l3fwd/l3fwd_em.c
 create mode 100644 examples/l3fwd/l3fwd_lpm.c

diff --git a/examples/Makefile b/examples/Makefile
index e997c7690..9e431dc01 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -40,7 +40,7 @@ $(error "Please define RTE_SDK environment variable")
 endif
 
 # To add new examples, append the directory name to this variable
-examples = bridge basic_monitor simple_forward speed_tester flow_table test_flow_dir aes_encrypt aes_decrypt flow_tracker load_balancer arp_response nf_router scaling_example load_generator payload_scan firewall simple_fwd_tb l2fwd test_messaging
+examples = bridge basic_monitor simple_forward speed_tester flow_table test_flow_dir aes_encrypt aes_decrypt flow_tracker load_balancer arp_response nf_router scaling_example load_generator payload_scan firewall simple_fwd_tb l2fwd test_messaging l3fwd
 
 ifeq ($(NDPI_HOME),)
 $(warning "Skipping ndpi_stats NF as NDPI_HOME is not set")
diff --git a/examples/l3fwd/Makefile b/examples/l3fwd/Makefile
new file mode 100755
index 000000000..d7a09a633
--- /dev/null
+++ b/examples/l3fwd/Makefile
@@ -0,0 +1,62 @@
+#                    openNetVM
+#      https://github.com/sdnfv/openNetVM
+#
+# BSD LICENSE
+#
+# Copyright(c)
+#          2015-2020 George Washington University
+#          2015-2020 University of California Riverside
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in
+# the documentation and/or other materials provided with the
+# distribution.
+# The name of the author may not be used to endorse or promote
+# products derived from this software without specific prior
+# written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+RTE_TARGET ?= $(RTE_TARGET)
+
+# Default target, can be overriden by command line or environment
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = l3fwd
+
+# all source are stored in SRCS-y
+SRCS-y := l3fwd.c l3fwd_lpm.c l3fwd_em.c
+
+# OpenNetVM path
+ONVM= $(SRCDIR)/../../onvm
+
+CFLAGS += -O3 $(USER_FLAGS)
+
+CFLAGS += -I$(ONVM)/onvm_nflib
+CFLAGS += -I$(ONVM)/lib
+LDFLAGS += $(ONVM)/onvm_nflib/$(RTE_TARGET)/libonvm.a
+LDFLAGS += $(ONVM)/lib/$(RTE_TARGET)/lib/libonvmhelper.a -lm
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/l3fwd/README.md b/examples/l3fwd/README.md
new file mode 100644
index 000000000..254334cca
--- /dev/null
+++ b/examples/l3fwd/README.md
@@ -0,0 +1,39 @@
+Layer 3 Switch
+==
+l3switch is based on DPDK's [l3fwd example](https://doc.dpdk.org/guides/sample_app_ug/l3_forward.html) that leverages the openNetVM's flow director API. This NF has two modes, exact match and longest prefix match. In longest prefix match mode, a lookup matches a packet to a destination port. The lookup is based on the destination IP address. In exact match mode, a lookup is is based on the packet 5-tuple i.e. destination and src of the packets port and IP.
+
+Hash entry number refers to the number of flow rules when running in exact match mode.
+
+Compilation and Execution
+--
+```
+cd examples
+make
+cd l3switch
+./go.sh SERVICE_ID [PRINT_DELAY]
+
+OR
+
+./go.sh -F CONFIG_FILE -- -- [-p PRINT_DELAY]
+
+OR
+
+sudo ./build/l3switch -l CORELIST -n 3 --proc-type=secondary -- -r SERVICE_ID -- [-p PRINT_DELAY]
+```
+
+App Specific Arguments
+--
+  -p <print_delay>: number of packets between each print, e.g. `-p 1` prints every packets. Default is every 1000000 packets.
+  -e : Enables exact match mode.
+  -h <hash entry number> : Sets the hash entry number.
+
+For example: ./go.sh 1 -e -h 7
+
+Will enable exact match mode with hash entry number set to 7. Hash entry number will default to 4 if not manually set or not running in exact match mode.
+
+Config File Support
+--
+This NF supports the NF generating arguments from a config file. For
+additional reading, see [Examples.md](../../docs/Examples.md)
+
+See `../example_config.json` for all possible options that can be set.
diff --git a/examples/l3fwd/go.sh b/examples/l3fwd/go.sh
new file mode 120000
index 000000000..40babd806
--- /dev/null
+++ b/examples/l3fwd/go.sh
@@ -0,0 +1 @@
+../go.sh
\ No newline at end of file
diff --git a/examples/l3fwd/l3fwd.c b/examples/l3fwd/l3fwd.c
new file mode 100755
index 000000000..3cf6a92b3
--- /dev/null
+++ b/examples/l3fwd/l3fwd.c
@@ -0,0 +1,348 @@
+/*********************************************************************
+ *                     openNetVM
+ *              https://sdnfv.github.io
+ *
+ *   BSD LICENSE
+ *
+ *   Copyright(c)
+ *            2015-2020 George Washington University
+ *            2015-2020 University of California Riverside
+ *   All rights reserved.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * The name of the author may not be used to endorse or promote
+ *       products derived from this software without specific prior
+ *       written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * l3switch.c - Layer 3 forwarding application with either exact match or LPM table.
+ ********************************************************************/
+
+
+#include <errno.h>
+#include <getopt.h>
+#include <inttypes.h>
+#include <stdarg.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/queue.h>
+#include <unistd.h>
+
+#include <rte_common.h>
+#include <rte_ip.h>
+#include <rte_lpm.h>
+#include <rte_mbuf.h>
+#include <rte_malloc.h>
+
+#include "onvm_nflib.h"
+#include "onvm_flow_table.h"
+#include "onvm_pkt_helper.h"
+#include "l3fwd.h"
+
+#define NF_TAG "l3switch"
+
+/* Print a usage message. */
+static void
+usage(const char *progname) {
+        printf("Usage:\n");
+        printf("%s [EAL args] -- [NF_LIB args] -- -p <print_delay>\n", progname);
+        printf("%s -F <CONFIG_FILE.json> [EAL args] -- [NF_LIB args] -- [NF args]\n\n", progname);
+        printf("Flags:\n");
+        printf(" -e : Enable exact match. \n");
+        printf(" -l : Enable longest prefix match. \n");
+        printf(" -h : Specifies the hash entry number in decimal to be setup. Default is 4. \n");
+}
+
+/* Parse the application arguments. */
+static int
+parse_app_args(int argc, char *argv[], const char *progname, struct state_info *stats) {
+        int c;
+
+        while ((c = getopt(argc, argv, "h:p:e")) != -1) {
+                switch (c) {
+                        case 'h':
+                                stats->hash_entry_number = strtoul(optarg, NULL, 10);
+                                break;
+                        case 'p':
+                                stats->print_delay = strtoul(optarg, NULL, 10);
+                                break;
+                        case 'e':
+                                stats->l3fwd_lpm_on = 0;
+                                stats->l3fwd_em_on = 1;
+                                break;
+                        case '?':
+                                usage(progname);
+                                if (optopt == 'p')
+                                        RTE_LOG(INFO, APP, "Option -%c requires an argument.\n", optopt);
+                                else if (isprint(optopt))
+                                        RTE_LOG(INFO, APP, "Unknown option `-%c'.\n", optopt);
+                                else
+                                        RTE_LOG(INFO, APP, "Unknown option character `\\x%x'.\n", optopt);
+                                return -1;
+                        default:
+                                usage(progname);
+                                return -1;
+                }
+        }
+        return optind;
+}
+
+/*
+ * This function displays stats. It uses ANSI terminal codes to clear
+ * screen when called. It is called from a single non-master
+ * thread in the server process, when the process is run with more
+ * than one lcore enabled.
+ */
+static void
+print_stats(__attribute__((unused)) struct onvm_nf_local_ctx *nf_local_ctx) {
+        const char clr[] = {27, '[', '2', 'J', '\0'};
+        const char topLeft[] = {27, '[', '1', ';', '1', 'H', '\0'};
+        uint64_t total_packets = 0;
+
+        struct onvm_nf *nf = nf_local_ctx->nf;
+        struct state_info *stats = (struct state_info *)nf->data;
+
+        /* Clear screen and move to top left */
+        printf("\nPort statistics ====================================");
+        int i;
+        for (i = 0; i < ports->num_ports; i++) {
+                printf("\nStatistics for port %u ------------------------------"
+                          "\nPackets forwarded to: %20"PRIu64,
+                           ports->id[i],
+                           stats->port_statistics[ports->id[i]]);
+
+               total_packets += stats->port_statistics[ports->id[i]];
+        }
+        printf("\nAggregate statistics ==============================="
+                   "\nTotal packets forwarded: %17"PRIu64
+                   "\nPackets dropped: %18"PRIu64,
+                   total_packets,
+                   stats->packets_dropped);
+        printf("\n====================================================\n");
+
+        printf("\n\n");
+}
+
+/*
+ * This function checks for valid ipv4 packets. Updates the
+ * src and destination ethernet addresses of packets. It then performs a lookup
+ * for the destination port. If the destination port value returned is not valid/not binded to dpdk,
+ * the packet is forwarded back to the port of incoming traffic.
+ */
+static int
+packet_handler(struct rte_mbuf *pkt, struct onvm_pkt_meta *meta,
+               __attribute__((unused)) struct onvm_nf_local_ctx *nf_local_ctx) {
+        static uint32_t counter = 0;
+
+        struct onvm_nf *nf = nf_local_ctx->nf;
+        struct state_info *stats = (struct state_info *)nf->data;
+
+        if (counter++ == stats->print_delay) {
+               print_stats(nf_local_ctx);
+               counter = 0;
+        }
+        struct rte_ether_hdr *eth_hdr;
+        struct ipv4_hdr *ipv4_hdr;
+        uint16_t dst_port;
+
+        eth_hdr = rte_pktmbuf_mtod(pkt, struct rte_ether_hdr *);
+        if (onvm_pkt_is_ipv4(pkt)) {
+                /* Handle IPv4 headers.*/
+                ipv4_hdr = rte_pktmbuf_mtod_offset(pkt, struct ipv4_hdr *,
+                                                   sizeof(struct rte_ether_hdr));
+
+#ifdef DO_RFC_1812_CHECKS
+                /* Check to make sure the packet is valid (RFC1812) */
+                if (is_valid_ipv4_pkt(ipv4_hdr, pkt->pkt_len) < 0) {
+                        meta->action = ONVM_NF_ACTION_DROP;
+                        packets_dropped++;
+                        return 0;
+                }
+#endif
+                if (stats->l3fwd_lpm_on) {
+                        dst_port = lpm_get_ipv4_dst_port(ipv4_hdr, pkt->port, stats);
+                } else {
+                        dst_port = em_get_ipv4_dst_port(pkt, stats);
+                }
+                if (dst_port >= RTE_MAX_ETHPORTS ||
+                        get_initialized_ports(dst_port) == 0)
+                        dst_port = pkt->port;
+
+#ifdef DO_RFC_1812_CHECKS
+                /* Update time to live and header checksum */
+                --(ipv4_hdr->time_to_live);
+                ++(ipv4_hdr->hdr_checksum);
+#endif
+                /* dst addr */
+                *(uint64_t *)&eth_hdr->d_addr = stats->dest_eth_addr[dst_port];
+
+                /* src addr */
+                rte_ether_addr_copy(&stats->ports_eth_addr[dst_port], &eth_hdr->s_addr);
+
+                meta->destination = dst_port;
+                stats->port_statistics[dst_port]++;
+                meta->action = ONVM_NF_ACTION_OUT;
+        } else {
+                meta->action = ONVM_NF_ACTION_DROP;
+                stats->packets_dropped++;
+        }
+        return 0;
+}
+
+/*
+ * This function displays the ethernet addressof each initialized port.
+ * It saves the ethernet addresses in the struct ether_addr array.
+ */
+static void
+l3fwd_initialize_ports(struct state_info *stats) {
+        uint16_t i;
+        for (i = 0; i < ports->num_ports; i++) {
+                rte_eth_macaddr_get(ports->id[i], &stats->ports_eth_addr[ports->id[i]]);
+                printf("Port %u, MAC address: %02X:%02X:%02X:%02X:%02X:%02X\n\n",
+                        ports->id[i],
+                        stats->ports_eth_addr[ports->id[i]].addr_bytes[0],
+                        stats->ports_eth_addr[ports->id[i]].addr_bytes[1],
+                        stats->ports_eth_addr[ports->id[i]].addr_bytes[2],
+                        stats->ports_eth_addr[ports->id[i]].addr_bytes[3],
+                        stats->ports_eth_addr[ports->id[i]].addr_bytes[4],
+                        stats->ports_eth_addr[ports->id[i]].addr_bytes[5]);
+        }
+}
+
+/* 
+ * This function pre-init dst MACs for all ports to 02:00:00:00:00:xx.
+ * Destination mac addresses are saved in th dest_eth_addr array.
+ */
+static void
+l3fwd_initialize_dst(struct state_info *stats) {
+        uint16_t i;
+        for (i = 0; i < ports->num_ports; i++) {
+                stats->dest_eth_addr[ports->id[i]] =
+                        RTE_ETHER_LOCAL_ADMIN_ADDR + ((uint64_t)ports->id[i] << 40);
+                *(uint64_t *)(stats->val_eth + ports->id[i]) = stats->dest_eth_addr[ports->id[i]];
+        }
+}
+
+/* This function frees all allocated data structures and hash tables. */
+static void
+free_tables(struct state_info *stats) {
+        if (stats->lpm_tbl != NULL) {
+                rte_lpm_free(stats->lpm_tbl);
+        }
+        if (stats->em_tbl != NULL) {
+                onvm_ft_free(stats->em_tbl);
+        }
+}
+
+void
+nf_setup(struct onvm_nf_local_ctx *nf_local_ctx) {
+        struct onvm_nf *nf = nf_local_ctx->nf;
+        struct state_info *stats = (struct state_info *)nf->data;
+        l3fwd_initialize_ports(stats);
+        l3fwd_initialize_dst(stats);
+        /*
+         * Hash flags are valid only for exact macth,
+         * reset them to default for longest-prefix match.
+         */
+        if (stats->l3fwd_lpm_on) {
+                if (setup_lpm(stats) < 0) {
+                     onvm_nflib_stop(nf_local_ctx);
+                     rte_free(stats);
+                     rte_exit(EXIT_FAILURE, "Unable to setup LPM\n");
+                }
+                stats->hash_entry_number = HASH_ENTRY_NUMBER_DEFAULT;
+                printf("\nLongest prefix match enabled. \n");
+        } else {
+                if (setup_hash(stats) < 0) {
+                     onvm_nflib_stop(nf_local_ctx);
+                     rte_free(stats);
+                     rte_exit(EXIT_FAILURE, "Unable to setup Hash\n");
+                }
+                printf("Hash table exact match enabled. \n");
+                printf("Hash entry number set to: %d\n", stats->hash_entry_number);
+        }
+}
+
+int
+main(int argc, char *argv[]) {
+        int arg_offset;
+        struct onvm_nf_local_ctx *nf_local_ctx;
+        struct onvm_nf_function_table *nf_function_table;
+        const char *progname = argv[0];
+
+        nf_local_ctx = onvm_nflib_init_nf_local_ctx();
+        onvm_nflib_start_signal_handler(nf_local_ctx, NULL);
+
+        nf_function_table = onvm_nflib_init_nf_function_table();
+        nf_function_table->pkt_handler = &packet_handler;
+        nf_function_table->setup = &nf_setup;
+
+        if ((arg_offset = onvm_nflib_init(argc, argv, NF_TAG, nf_local_ctx, nf_function_table)) < 0) {
+                onvm_nflib_stop(nf_local_ctx);
+                if (arg_offset == ONVM_SIGNAL_TERMINATION) {
+                        printf("Exiting due to user termination\n");
+                        return 0;
+                } else {
+                        rte_exit(EXIT_FAILURE, "Failed ONVM init\n");
+                }
+        }
+
+        argc -= arg_offset;
+        argv += arg_offset;
+
+        /*
+         * The following allocates a struct which keeps track of all NF state information.
+         * Longest prefix match is enabled by default as well as default values for print delay
+         * and hash entry number.
+         */
+        struct onvm_nf *nf = nf_local_ctx->nf;
+        struct state_info *stats = rte_calloc("state", 1, sizeof(struct state_info), 0); // Will be freed my manager.
+        if (stats == NULL) {
+                onvm_nflib_stop(nf_local_ctx);
+                rte_exit(EXIT_FAILURE, "Unable to initialize NF stats.");
+        }
+        stats->print_delay = 1000000;
+        stats->l3fwd_lpm_on = 1;
+        stats->l3fwd_em_on = 0;
+        stats->hash_entry_number = HASH_ENTRY_NUMBER_DEFAULT;
+        nf->data = (void *)stats;
+
+        /* Parse application arguments. */
+        if (parse_app_args(argc, argv, progname, stats) < 0) {
+                onvm_nflib_stop(nf_local_ctx);
+                rte_exit(EXIT_FAILURE, "Invalid command-line arguments\n");
+        }
+
+        if (ports->num_ports == 0) {
+                onvm_nflib_stop(nf_local_ctx);
+                rte_exit(EXIT_FAILURE, "No Ethernet ports. Ensure ports binded to dpdk. - bye\n");
+        }
+        onvm_nflib_run(nf_local_ctx);
+
+        free_tables(stats);
+        onvm_nflib_stop(nf_local_ctx);
+        printf("If we reach here, program is ending\n");
+        return 0;
+}
diff --git a/examples/l3fwd/l3fwd.h b/examples/l3fwd/l3fwd.h
new file mode 100644
index 000000000..685e88387
--- /dev/null
+++ b/examples/l3fwd/l3fwd.h
@@ -0,0 +1,91 @@
+/*********************************************************************
+ *                     openNetVM
+ *              https://sdnfv.github.io
+ *
+ *   BSD LICENSE
+ *
+ *   Copyright(c)
+ *            2015-2020 George Washington University
+ *            2015-2020 University of California Riverside
+ *   All rights reserved.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * The name of the author may not be used to endorse or promote
+ *       products derived from this software without specific prior
+ *       written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * l3switch.h - This application performs L3 forwarding.
+ ********************************************************************/
+
+#include "onvm_flow_table.h"
+
+#ifndef __L3_SWITCH_H_
+#define __L3_SWICTH_H_
+
+/* Hash parameters. */
+#ifdef RTE_ARCH_64
+/* default to 4 million hash entries (approx) */
+#define L3FWD_HASH_ENTRIES              (1024*1024*4)
+#else
+/* 32-bit has less address-space for hugepage memory, limit to 1M entries */
+#define L3FWD_HASH_ENTRIES              (1024*1024*1)
+#endif
+
+#define HASH_ENTRY_NUMBER_DEFAULT       4
+#define NB_SOCKETS        8
+
+/*Struct that holds all NF state information */
+struct state_info {
+        struct lpm_request *l3switch_req;
+        struct rte_lpm *lpm_tbl;
+        struct onvm_ft *em_tbl;
+        struct rte_ether_addr ports_eth_addr[RTE_MAX_ETHPORTS];
+        uint64_t port_statistics[RTE_MAX_ETHPORTS];
+        xmm_t val_eth[RTE_MAX_ETHPORTS];
+        uint64_t dest_eth_addr[RTE_MAX_ETHPORTS];
+        uint64_t packets_dropped;
+        uint32_t print_delay;
+        uint32_t hash_entry_number;
+        int8_t l3fwd_lpm_on;
+        int8_t l3fwd_em_on;
+};
+
+/* Function pointers for LPM or EM functionality. */
+
+int
+setup_lpm(struct state_info *stats);
+
+int
+setup_hash(struct state_info *stats);
+
+uint16_t
+lpm_get_ipv4_dst_port(void *ipv4_hdr, uint16_t portid, struct state_info *stats);
+
+uint16_t
+em_get_ipv4_dst_port(struct rte_mbuf *pkt, struct state_info *stats);
+
+int
+get_initialized_ports(uint8_t if_out);
+
+#endif // __L3_SWICTH_H_
diff --git a/examples/l3fwd/l3fwd_em.c b/examples/l3fwd/l3fwd_em.c
new file mode 100644
index 000000000..e9fc75236
--- /dev/null
+++ b/examples/l3fwd/l3fwd_em.c
@@ -0,0 +1,154 @@
+#include <errno.h>
+#include <getopt.h>
+#include <inttypes.h>
+#include <stdarg.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <rte_common.h>
+#include <rte_cycles.h>
+#include <rte_hash.h>
+#include <rte_ip.h>
+#include <rte_mbuf.h>
+
+#include "l3fwd.h"
+#include "onvm_flow_table.h"
+#include "onvm_nflib.h"
+#include "onvm_pkt_helper.h"
+
+#define BYTE_VALUE_MAX 256
+
+struct ipv4_l3fwd_em_route {
+    struct onvm_ft_ipv4_5tuple key;
+    uint8_t if_out;
+};
+
+// src_addr, dst_addr, src_port, dst_port, proto
+struct ipv4_l3fwd_em_route ipv4_l3fwd_em_route_array[] = {
+    {{RTE_IPV4(101, 0, 0, 0), RTE_IPV4(100, 10, 0, 1),  0, 1, IPPROTO_TCP}, 0},
+    {{RTE_IPV4(201, 0, 0, 0), RTE_IPV4(200, 20, 0, 1),  1, 0, IPPROTO_TCP}, 1},
+    {{RTE_IPV4(111, 0, 0, 0), RTE_IPV4(100, 30, 0, 1),  0, 2, IPPROTO_TCP}, 2},
+    {{RTE_IPV4(211, 0, 0, 0), RTE_IPV4(200, 40, 0, 1),  2, 0, IPPROTO_TCP}, 3},
+};
+
+/* Struct that holds info about each flow, and is stored at each flow table entry. */
+struct data {
+    uint8_t  if_out;
+};
+
+#define IPV4_L3FWD_EM_NUM_ROUTES \
+    (sizeof(ipv4_l3fwd_em_route_array) / sizeof(ipv4_l3fwd_em_route_array[0]))
+
+static inline void
+populate_ipv4_few_flow_into_table(struct onvm_ft *h) {
+    uint32_t i;
+    int32_t ret;
+
+    for (i = 0; i < IPV4_L3FWD_EM_NUM_ROUTES; i++) {
+        struct ipv4_l3fwd_em_route entry;
+        union ipv4_5tuple_host newkey;
+        struct data *data = NULL;
+
+        entry = ipv4_l3fwd_em_route_array[i];
+        int tbl_index = onvm_ft_add_key(h, &entry.key, (char **)&data);
+        data->if_out = entry.if_out;
+        if (tbl_index < 0)
+            rte_exit(EXIT_FAILURE, "Unable to add entry %u\n", i);
+        printf("\nAdding key:");
+        _onvm_ft_print_key(&entry.key);
+    }
+    printf("Hash: Adding 0x%" PRIx64 " keys\n",
+        (uint64_t)IPV4_L3FWD_EM_NUM_ROUTES);
+}
+
+#define NUMBER_PORT_USED 4
+static inline void
+populate_ipv4_many_flow_into_table(struct onvm_ft *h, unsigned int nr_flow) {
+    unsigned i;
+
+    for (i = 0; i < nr_flow; i++) {
+        struct ipv4_l3fwd_em_route entry;
+        union ipv4_5tuple_host newkey;
+
+        uint8_t a = (uint8_t)
+            ((i/NUMBER_PORT_USED)%BYTE_VALUE_MAX);
+        uint8_t b = (uint8_t)
+            (((i/NUMBER_PORT_USED)/BYTE_VALUE_MAX)%BYTE_VALUE_MAX);
+        uint8_t c = (uint8_t)
+            ((i/NUMBER_PORT_USED)/(BYTE_VALUE_MAX*BYTE_VALUE_MAX));
+
+        /* Create the ipv4 exact match flow */
+        memset(&entry, 0, sizeof(entry));
+        switch (i & (NUMBER_PORT_USED - 1)) {
+        case 0:
+            entry = ipv4_l3fwd_em_route_array[0];
+            entry.key.dst_addr = RTE_IPV4(101, c, b, a);
+            break;
+        case 1:
+            entry = ipv4_l3fwd_em_route_array[1];
+            entry.key.dst_addr = RTE_IPV4(201, c, b, a);
+            break;
+        case 2:
+            entry = ipv4_l3fwd_em_route_array[2];
+            entry.key.dst_addr = RTE_IPV4(111, c, b, a);
+            break;
+        case 3:
+            entry = ipv4_l3fwd_em_route_array[3];
+            entry.key.dst_addr = RTE_IPV4(211, c, b, a);
+            break;
+        }
+        struct data *data = NULL;
+        int tbl_index = onvm_ft_add_key(h, &entry.key, (char **)&data);
+        data->if_out = entry.if_out;
+        if (tbl_index < 0)
+            rte_exit(EXIT_FAILURE, "Unable to add entry %u\n", i);
+        printf("\nAdding key:");
+        _onvm_ft_print_key(&entry.key);
+    }
+    printf("Hash: Adding 0x%x keys\n", nr_flow);
+}
+
+uint16_t
+em_get_ipv4_dst_port(struct rte_mbuf *pkt, struct state_info *stats) {
+    struct data *data = NULL;
+    struct onvm_ft_ipv4_5tuple key;
+    uint8_t port;
+
+    int ret = onvm_ft_fill_key(&key, pkt);
+
+    if (ret < 0)
+            return pkt->port;
+
+    int tbl_index = onvm_ft_lookup_key(stats->em_tbl, &key, (char **)&data);
+    if (tbl_index < 0)
+            return 1;
+    port = data->if_out;
+    return port;
+}
+
+int
+setup_hash(struct state_info *stats) {
+    stats->em_tbl = onvm_ft_create(L3FWD_HASH_ENTRIES, sizeof(struct ipv4_l3fwd_em_route));
+    if (stats->em_tbl == NULL) {
+            printf("Unable to create flow table");
+            return -1;
+    }
+    if (stats->hash_entry_number != HASH_ENTRY_NUMBER_DEFAULT) {
+        /* For testing hash matching with a large number of flows we
+         * generate millions of IP 5-tuples with an incremented dst
+         * address to initialize the hash table. */
+        /* populate the ipv4 hash */
+        populate_ipv4_many_flow_into_table(stats->em_tbl, stats->hash_entry_number);
+    } else {
+        /*
+         * Use data in ipv4/ipv6 l3fwd lookup table
+         * directly to initialize the hash table.
+         */
+        /* populate the ipv4 hash */
+        populate_ipv4_few_flow_into_table(stats->em_tbl);
+    }
+    return 0;
+}
diff --git a/examples/l3fwd/l3fwd_lpm.c b/examples/l3fwd/l3fwd_lpm.c
new file mode 100644
index 000000000..23629c665
--- /dev/null
+++ b/examples/l3fwd/l3fwd_lpm.c
@@ -0,0 +1,132 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <inttypes.h>
+#include <sys/types.h>
+#include <string.h>
+#include <sys/queue.h>
+#include <stdarg.h>
+#include <errno.h>
+#include <getopt.h>
+#include <stdbool.h>
+
+#include <rte_debug.h>
+#include <rte_ether.h>
+#include <rte_ethdev.h>
+#include <rte_cycles.h>
+#include <rte_mbuf.h>
+#include <rte_ip.h>
+#include <rte_tcp.h>
+#include <rte_udp.h>
+#include <rte_lpm.h>
+#include <rte_lpm6.h>
+#include <rte_malloc.h>
+
+#include "onvm_nflib.h"
+#include "onvm_pkt_helper.h"
+#include "l3fwd.h"
+
+/* Shared data structure containing host port info. */
+extern struct port_info *ports;
+
+struct ipv4_l3fwd_lpm_route {
+        uint32_t ip; // destination address
+        uint8_t  depth;
+        uint8_t  if_out;
+};
+
+static struct ipv4_l3fwd_lpm_route ipv4_l3fwd_lpm_route_array[] = {
+        {RTE_IPV4(1, 1, 1, 0), 24, 0},
+        {RTE_IPV4(2, 1, 1, 0), 24, 1},
+        {RTE_IPV4(3, 1, 1, 0), 24, 2},
+        {RTE_IPV4(4, 1, 1, 0), 24, 3},
+        {RTE_IPV4(5, 1, 1, 0), 24, 4},
+        {RTE_IPV4(6, 1, 1, 0), 24, 5},
+        {RTE_IPV4(7, 1, 1, 0), 24, 6},
+        {RTE_IPV4(8, 1, 1, 0), 24, 7},
+};
+
+#define IPV4_L3FWD_LPM_NUM_ROUTES \
+        (sizeof(ipv4_l3fwd_lpm_route_array) / sizeof(ipv4_l3fwd_lpm_route_array[0]))
+
+#define IPV4_L3FWD_LPM_MAX_RULES         1024
+#define IPV4_L3FWD_LPM_NUMBER_TBL8S (1 << 8)
+
+int
+setup_lpm(struct state_info *stats) {
+        struct rte_lpm6_config config;
+        int i, status, ret;
+        char name[64];
+
+        /* create the LPM table */
+        stats->l3switch_req = (struct lpm_request *)rte_malloc(NULL, sizeof(struct lpm_request), 0);
+
+        if (!stats->l3switch_req) return -1;
+
+        snprintf(name, sizeof(name), "fw%d-%"PRIu64, rte_lcore_id(), rte_get_tsc_cycles());
+        stats->l3switch_req->max_num_rules = IPV4_L3FWD_LPM_MAX_RULES;
+        stats->l3switch_req->num_tbl8s = IPV4_L3FWD_LPM_NUMBER_TBL8S;
+        stats->l3switch_req->socket_id = rte_socket_id();
+        snprintf(stats->l3switch_req->name, sizeof(name), "%s", name);
+        status = onvm_nflib_request_lpm(stats->l3switch_req);
+
+        if (status < 0) {
+                printf("Cannot get lpm region for l3switch\n");
+                return -1;
+        }
+        rte_free(stats->l3switch_req);
+
+        stats->lpm_tbl = rte_lpm_find_existing(name);
+
+        if (stats->lpm_tbl == NULL) {
+                printf("No existing LPM_TBL\n");
+                return -1;
+        }
+        /* populate the LPM table */
+        for (i = 0; i < IPV4_L3FWD_LPM_NUM_ROUTES; i++) {
+
+                /* skip unused ports */
+                if (get_initialized_ports(ipv4_l3fwd_lpm_route_array[i].if_out) == 0)
+                        continue;
+
+                ret = rte_lpm_add(stats->lpm_tbl,
+                        ipv4_l3fwd_lpm_route_array[i].ip,
+                        ipv4_l3fwd_lpm_route_array[i].depth,
+                        ipv4_l3fwd_lpm_route_array[i].if_out);
+
+                if (ret < 0) {
+                        printf("Unable to add entry %u to the l3fwd LPM table. \n", i);
+                        return -1;
+                }
+
+                printf("\nLPM: Adding route 0x%08x / %d (%d)\n",
+                        (unsigned)ipv4_l3fwd_lpm_route_array[i].ip,
+                        ipv4_l3fwd_lpm_route_array[i].depth,
+                        ipv4_l3fwd_lpm_route_array[i].if_out);
+                printf("IP: %" PRIu8 ".%" PRIu8 ".%" PRIu8 ".%" PRIu8,
+                        ipv4_l3fwd_lpm_route_array[i].ip >>24 & 0xFF, (ipv4_l3fwd_lpm_route_array[i].ip >> 16) & 0xFF,
+                        (ipv4_l3fwd_lpm_route_array[i].ip >> 8) & 0xFF, (ipv4_l3fwd_lpm_route_array[i].ip) & 0xFF);
+        }
+        return 0;
+}
+
+uint16_t
+lpm_get_ipv4_dst_port(void *ipv4_hdr, uint16_t portid, struct state_info *stats) {
+        uint32_t next_hop;
+        return (uint16_t) ((rte_lpm_lookup(stats->lpm_tbl,
+                rte_be_to_cpu_32(((struct rte_ipv4_hdr *)ipv4_hdr)->dst_addr),
+                &next_hop) == 0) ? next_hop : portid);
+}
+
+/*
+ * This helper function checks if the destination port
+ * is a valid port number that is currently binded to dpdk.
+ */
+int
+get_initialized_ports(uint8_t if_out) {
+        for (int i = 0; i < ports->num_ports; i++) {
+                if (ports->id[i] == if_out)
+                        return 1;
+        }
+        return 0;
+}

From 3149348328460175db02b001a310b96877f780d0 Mon Sep 17 00:00:00 2001
From: Rohit M P <rohitmp99@gmail.com>
Date: Mon, 25 Oct 2021 23:45:27 +0530
Subject: [PATCH 13/17] [New NF] Fair Queue NF

This PR adds a new example - Fair Queue with Round Robin Scheduler

The NF uses the advanced rings mode, classifies IPv4 packets based on the header information, and simulates a round-robin dequeue.

The CRC32 hash from the DPDK library has been used to classify the packets.

The NF uses a custom implementation of the standard FIFO queuing system. An implementation using the rte_ring structure from the DPDK library failed to scale for large number of queues.

Commit log:

* Initial import

* Update NF list

* Update examples/fair_queue/fair_queue.c

prevent failure in destructor

Co-authored-by: Catherine Meadows <meadowsc@gwu.edu>

* Update examples/fair_queue/README.md

Co-authored-by: Tim Wood <timwood@gwu.edu>
Co-authored-by: Catherine Meadows <meadowsc@gwu.edu>
Co-authored-by: Dennis Afanasev <dennis.afa@gmail.com>
---
 examples/Makefile                       |   2 +-
 examples/fair_queue/.gitignore          |   2 +
 examples/fair_queue/Makefile            |  68 ++++
 examples/fair_queue/README.md           |  19 +
 examples/fair_queue/fair_queue.c        | 447 ++++++++++++++++++++++++
 examples/fair_queue/fair_queue_helper.h | 261 ++++++++++++++
 examples/fair_queue/go.sh               |   1 +
 7 files changed, 799 insertions(+), 1 deletion(-)
 create mode 100644 examples/fair_queue/.gitignore
 create mode 100644 examples/fair_queue/Makefile
 create mode 100644 examples/fair_queue/README.md
 create mode 100644 examples/fair_queue/fair_queue.c
 create mode 100644 examples/fair_queue/fair_queue_helper.h
 create mode 120000 examples/fair_queue/go.sh

diff --git a/examples/Makefile b/examples/Makefile
index 9e431dc01..a7b41bc26 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -40,7 +40,7 @@ $(error "Please define RTE_SDK environment variable")
 endif
 
 # To add new examples, append the directory name to this variable
-examples = bridge basic_monitor simple_forward speed_tester flow_table test_flow_dir aes_encrypt aes_decrypt flow_tracker load_balancer arp_response nf_router scaling_example load_generator payload_scan firewall simple_fwd_tb l2fwd test_messaging l3fwd
+examples = bridge basic_monitor simple_forward speed_tester flow_table test_flow_dir aes_encrypt aes_decrypt flow_tracker load_balancer arp_response nf_router scaling_example load_generator payload_scan firewall simple_fwd_tb l2fwd test_messaging l3fwd fair_queue
 
 ifeq ($(NDPI_HOME),)
 $(warning "Skipping ndpi_stats NF as NDPI_HOME is not set")
diff --git a/examples/fair_queue/.gitignore b/examples/fair_queue/.gitignore
new file mode 100644
index 000000000..112e2d9b7
--- /dev/null
+++ b/examples/fair_queue/.gitignore
@@ -0,0 +1,2 @@
+build/
+fair_queue/
diff --git a/examples/fair_queue/Makefile b/examples/fair_queue/Makefile
new file mode 100644
index 000000000..da97b989e
--- /dev/null
+++ b/examples/fair_queue/Makefile
@@ -0,0 +1,68 @@
+#                    openNetVM
+#      https://github.com/sdnfv/openNetVM
+#
+# BSD LICENSE
+#
+# Copyright(c)
+#          2015-2017 George Washington University
+#          2015-2017 University of California Riverside
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in
+# the documentation and/or other materials provided with the
+# distribution.
+# The name of the author may not be used to endorse or promote
+# products derived from this software without specific prior
+# written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+# Default target, can be overriden by command line or environment
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = fair_queue
+
+# all source are stored in SRCS-y
+SRCS-y := fair_queue.c
+
+# OpenNetVM path
+ONVM= $(SRCDIR)/../../onvm
+
+CFLAGS += $(WERROR_FLAGS) -O3 $(USER_FLAGS)
+
+CFLAGS += -I$(ONVM)/onvm_nflib
+CFLAGS += -I$(ONVM)/lib
+LDFLAGS += $(ONVM)/onvm_nflib/$(RTE_TARGET)/libonvm.a
+LDFLAGS += $(ONVM)/lib/$(RTE_TARGET)/lib/libonvmhelper.a -lm
+
+# workaround for a gcc bug with noreturn attribute
+# http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12603
+ifeq ($(CONFIG_RTE_TOOLCHAIN_GCC),y)
+CFLAGS_main.o += -Wno-return-type
+endif
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/fair_queue/README.md b/examples/fair_queue/README.md
new file mode 100644
index 000000000..28120f46a
--- /dev/null
+++ b/examples/fair_queue/README.md
@@ -0,0 +1,19 @@
+Fair Queue
+==
+The Fair Queue NF simulates fair queueing by classifying packets based on the 5-tuple of Source IP, Source Port, Destination IP, Destination Port, and Protocol into separate queues and dequeuing the packets in a round robin fashion.
+
+Contributed by [Rohit MP](https://gist.github.com/rohit-mp) from NITK
+
+Compilation and Execution
+--
+```
+cd examples
+make
+cd fair_queue
+./go.sh SERVICE_ID -d DESTINATION_ID [-n NUM_QUEUES] [-p]
+```
+
+App Specific Arguments
+--
+  - `-n <num_queues>`: Number of queues for the fair queuing system.
+  - `-p`: Print per queue statistics
diff --git a/examples/fair_queue/fair_queue.c b/examples/fair_queue/fair_queue.c
new file mode 100644
index 000000000..4803caab9
--- /dev/null
+++ b/examples/fair_queue/fair_queue.c
@@ -0,0 +1,447 @@
+/*********************************************************************
+ *                     openNetVM
+ *              https://sdnfv.github.io
+ *
+ *   BSD LICENSE
+ *
+ *   Copyright(c)
+ *            2020 National Institute of Technology Karnataka, Surathkal
+ *   All rights reserved.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * The name of the author may not be used to endorse or promote
+ *       products derived from this software without specific prior
+ *       written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * fair_queue.c - Simulates fair queueing by categorizing packets based
+ *      on IPv4 header values into separate queues and dequeue
+ *      packets in a round robin fashion.
+ ********************************************************************/
+
+#include <errno.h>
+#include <getopt.h>
+#include <inttypes.h>
+#include <stdarg.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/queue.h>
+#include <unistd.h>
+
+#include <rte_common.h>
+#include <rte_cycles.h>
+#include <rte_ip.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+
+#include "cJSON.h"
+#include "onvm_nflib.h"
+#include "onvm_pkt_helper.h"
+
+#include "fair_queue_helper.h"
+
+#define NF_TAG "fair_queue"
+
+#define PKT_READ_SIZE ((uint16_t)32)
+
+#define FQ_STATS_MSG                                                       \
+        "\n"                                                               \
+        "QUEUE ID         rx_pps  /  tx_pps             rx  /  tx      \n" \
+        "               drop_pps  /  drop_pps      rx_drop  /  tx_drop \n" \
+        "--------------------------------------------------------------\n"
+
+#define FQ_STATS_CONTENT                                                             \
+        "%2u             %9" PRIu64 " / %-9" PRIu64 "   %11" PRIu64 " / %-11" PRIu64 \
+        "\n"                                                                         \
+        "               %9" PRIu64 " / %-9" PRIu64 "   %11" PRIu64 " / %-11" PRIu64 "\n\n"
+
+static uint32_t destination;
+static uint8_t print_stats_flag;
+
+/* For advanced rings scaling */
+rte_atomic16_t signal_exit_flag;
+struct child_spawn_info {
+        struct onvm_nf_init_cfg *child_cfg;
+        struct onvm_nf *parent;
+};
+
+void
+sig_handler(int sig);
+
+void *
+start_child(void *arg);
+
+void *
+do_fq_stats_display(void *arg);
+
+/*
+ * Print a usage message
+ */
+static void
+usage(const char *progname) {
+        printf("Usage:\n");
+        printf("%s [EAL args] -- [NF_LIB args] -- -d <destination> -p <print_delay> -D <tb_depth> -R <tb_rate>\n",
+               progname);
+        printf("%s -F <CONFIG_FILE.json> [EAL args] -- [NF_LIB args] -- [NF args]\n\n", progname);
+        printf("Flags:\n");
+        printf(" - `-d <dst>`: Destination service ID to forward to\n");
+        printf(" - `-n <num_queues>`: Number of queues to simulate round robin fair queueing\n");
+        printf(
+            " - `-p`: Print per queue stats on the terminal (Not recommended for use with large value of "
+            "num_queues)\n");
+}
+
+/*
+ * Parse the application arguments.
+ */
+static int
+parse_app_args(int argc, char *argv[], const char *progname, struct onvm_nf *nf) {
+        int c, dst_flag = 0, num_queues_flag = 0;
+        struct fairqueue_t *fairqueue;
+        print_stats_flag = 0;    /* No per queue output by default */
+        uint16_t num_queues = 2; /* Default number of queueus */
+
+        while ((c = getopt(argc, argv, "d:n:p")) != -1) {
+                switch (c) {
+                        case 'd':
+                                destination = strtoul(optarg, NULL, 10);
+                                dst_flag = 1;
+                                break;
+                        case 'n':
+                                num_queues = strtoul(optarg, NULL, 10);
+                                num_queues_flag = 1;
+                                break;
+                        case 'p':
+                                print_stats_flag = 1;
+                                break;
+                        case '?':
+                                usage(progname);
+                                if (optopt == 'd')
+                                        RTE_LOG(INFO, APP, "Option -%c requires an argument.\n", optopt);
+                                else if (optopt == 'n')
+                                        RTE_LOG(INFO, APP, "Option -%c requires an argument.\n", optopt);
+                                else if (isprint(optopt))
+                                        RTE_LOG(INFO, APP, "Unknown option `-%c'.\n", optopt);
+                                else
+                                        RTE_LOG(INFO, APP, "Unknown option character `\\x%x'.\n", optopt);
+                                return -1;
+                        default:
+                                usage(progname);
+                                return -1;
+                }
+        }
+
+        if (!dst_flag) {
+                RTE_LOG(INFO, APP, "Fair Queue NF requires destination flag -d.\n");
+                return -1;
+        }
+
+        if (!num_queues_flag) {
+                RTE_LOG(INFO, APP, "Default number of queues (2) used. Specify a number using flag -n.\n");
+        }
+
+        /* Setup fair queue */
+        setup_fairqueue(&fairqueue, num_queues);
+        nf->data = (void *)fairqueue;
+
+        return optind;
+}
+
+void *
+do_fq_stats_display(__attribute__((unused)) void *arg) {
+        const char clr[] = {27, '[', '2', 'J', '\0'};
+        const char topLeft[] = {27, '[', '1', ';', '1', 'H', '\0'};
+        struct fairqueue_t *fair_queue;
+
+        fair_queue = (struct fairqueue_t *)arg;
+
+        while (!rte_atomic16_read(&signal_exit_flag)) {
+                printf("%s%s", clr, topLeft);
+                printf("%s", FQ_STATS_MSG);
+
+                for (uint16_t i = 0; i < fair_queue->num_queues; i++) {
+                        struct fairqueue_queue *fq;
+                        uint64_t rx, rx_last, tx, tx_last;
+                        uint64_t rx_drop, tx_drop, rx_drop_last, tx_drop_last;
+
+                        fq = fair_queue->fq[i];
+
+                        rx = fq->rx;
+                        rx_last = fq->rx_last;
+                        tx = fq->tx;
+                        tx_last = fq->tx_last;
+                        rx_drop = fq->rx_drop;
+                        rx_drop_last = fq->rx_drop_last;
+                        tx_drop = fq->tx_drop;
+                        tx_drop_last = fq->tx_drop_last;
+
+                        printf(FQ_STATS_CONTENT, i + 1, rx - rx_last, tx - tx_last, rx, tx, rx_drop - rx_drop_last,
+                               tx_drop - tx_drop_last, rx_drop, tx_drop);
+
+                        fq->rx_last = rx;
+                        fq->tx_last = tx;
+                        fq->rx_drop_last = rx_drop;
+                        fq->tx_drop_last = tx_drop;
+                }
+                sleep(1);
+        }
+
+        return NULL;
+}
+
+void
+sig_handler(int sig) {
+        if (sig != SIGINT && sig != SIGTERM)
+                return;
+
+        /* Will stop the processing for all spawned threads in advanced rings mode */
+        rte_atomic16_set(&signal_exit_flag, 1);
+}
+
+static int
+tx_loop(struct onvm_nf_local_ctx *nf_local_ctx) {
+        void *pktsTX[PKT_READ_SIZE];
+        int tx_batch_size;
+        struct onvm_pkt_meta *meta;
+        struct rte_ring *tx_ring;
+        struct rte_ring *msg_q;
+        struct onvm_nf *nf;
+        struct onvm_nf_msg *msg;
+        struct rte_mempool *nf_msg_pool;
+        struct rte_mbuf *pkt;
+        struct fairqueue_t *fair_queue;
+
+        nf = nf_local_ctx->nf;
+        fair_queue = (struct fairqueue_t *)nf->data;
+        onvm_nflib_nf_ready(nf);
+
+        /* Get rings from nflib */
+        tx_ring = nf->tx_q;
+        msg_q = nf->msg_q;
+        nf_msg_pool = rte_mempool_lookup(_NF_MSG_POOL_NAME);
+
+        printf("Process %d handling packets using advanced rings\n", nf->instance_id);
+        if (onvm_threading_core_affinitize(nf->thread_info.core) < 0)
+                rte_exit(EXIT_FAILURE, "Failed to affinitize to core %d\n", nf->thread_info.core);
+
+        tx_batch_size = 0;
+        while (!rte_atomic16_read(&signal_exit_flag)) {
+                /* Check for a stop message from the manager */
+                if (unlikely(rte_ring_count(msg_q) > 0)) {
+                        msg = NULL;
+                        rte_ring_dequeue(msg_q, (void **)(&msg));
+                        if (msg->msg_type == MSG_STOP) {
+                                rte_atomic16_set(&signal_exit_flag, 1);
+                        } else {
+                                printf("Received message %d, ignoring", msg->msg_type);
+                        }
+                        rte_mempool_put(nf_msg_pool, (void *)msg);
+                }
+
+                /* Dequeue packet from the fair queue system */
+                pkt = NULL;
+                fairqueue_dequeue(fair_queue, &pkt);
+
+                if (pkt == NULL) {
+                        continue;
+                }
+
+                meta = onvm_get_pkt_meta((struct rte_mbuf *)pkt);
+                meta->action = ONVM_NF_ACTION_TONF;
+                meta->destination = destination;
+                pktsTX[tx_batch_size++] = pkt;
+                if (tx_batch_size == PKT_READ_SIZE) {
+                        rte_ring_enqueue_bulk(tx_ring, pktsTX, tx_batch_size, NULL);
+                        tx_batch_size = 0;
+                }
+        }
+        return 0;
+}
+
+static int
+rx_loop(struct onvm_nf_local_ctx *nf_local_ctx) {
+        void *pkts[PKT_READ_SIZE];
+        void *pktsDrop[PKT_READ_SIZE];
+        struct onvm_pkt_meta *meta;
+        uint16_t i, nb_pkts;
+        int tx_batch_size;
+        struct rte_ring *rx_ring;
+        struct rte_ring *tx_ring;
+        struct rte_ring *msg_q;
+        struct onvm_nf *nf;
+        struct onvm_nf_msg *msg;
+        struct rte_mempool *nf_msg_pool;
+        struct fairqueue_t *fair_queue;
+
+        nf = nf_local_ctx->nf;
+        fair_queue = (struct fairqueue_t *)nf->data;
+        onvm_nflib_nf_ready(nf);
+
+        /* Get rings from nflib */
+        rx_ring = nf->rx_q;
+        tx_ring = nf->tx_q;
+        msg_q = nf->msg_q;
+        nf_msg_pool = rte_mempool_lookup(_NF_MSG_POOL_NAME);
+
+        printf("Process %d handling packets using advanced rings\n", nf->instance_id);
+        if (onvm_threading_core_affinitize(nf->thread_info.core) < 0)
+                rte_exit(EXIT_FAILURE, "Failed to affinitize to core %d\n", nf->thread_info.core);
+
+        tx_batch_size = 0;
+        while (!rte_atomic16_read(&signal_exit_flag)) {
+                /* Check for a stop message from the manager */
+                if (unlikely(rte_ring_count(msg_q) > 0)) {
+                        msg = NULL;
+                        rte_ring_dequeue(msg_q, (void **)(&msg));
+                        if (msg->msg_type == MSG_STOP) {
+                                rte_atomic16_set(&signal_exit_flag, 1);
+                        } else {
+                                printf("Received message %d, ignoring", msg->msg_type);
+                        }
+                        rte_mempool_put(nf_msg_pool, (void *)msg);
+                }
+
+                nb_pkts = rte_ring_dequeue_burst(rx_ring, pkts, PKT_READ_SIZE, NULL);
+
+                for (i = 0; i < nb_pkts; i++) {
+                        if (fairqueue_enqueue(fair_queue, pkts[i]) == -1) {
+                                meta = onvm_get_pkt_meta((struct rte_mbuf *)pkts[i]);
+                                meta->action = ONVM_NF_ACTION_DROP;
+                                meta->destination = destination;
+                                pktsDrop[tx_batch_size++] = pkts[i];
+                                if (tx_batch_size == 32) {
+                                        rte_ring_enqueue_bulk(tx_ring, pktsDrop, tx_batch_size, NULL);
+                                        tx_batch_size = 0;
+                                }
+                        }
+                }
+        }
+        return 0;
+}
+
+void *
+start_child(void *arg) {
+        struct onvm_nf_local_ctx *child_local_ctx;
+        struct onvm_nf_init_cfg *child_init_cfg;
+        struct onvm_nf *parent;
+        struct child_spawn_info *spawn_info;
+
+        spawn_info = (struct child_spawn_info *)arg;
+        child_init_cfg = spawn_info->child_cfg;
+        parent = spawn_info->parent;
+        child_local_ctx = onvm_nflib_init_nf_local_ctx();
+
+        if (onvm_nflib_start_nf(child_local_ctx, child_init_cfg) < 0) {
+                printf("Failed to spawn child NF\n");
+                return NULL;
+        }
+
+        /* Keep track of parent for proper termination */
+        child_local_ctx->nf->thread_info.parent = parent->instance_id;
+        child_local_ctx->nf->data = parent->data;
+
+        tx_loop(child_local_ctx);
+
+        /* Invalidate the child data to avoid double freeing of memory */
+        child_local_ctx->nf->data = NULL;
+        onvm_nflib_stop(child_local_ctx);
+        return NULL;
+}
+
+int
+main(int argc, char *argv[]) {
+        pthread_t tx_thread;
+        pthread_t stats_thread;
+        struct onvm_nf_local_ctx *nf_local_ctx;
+        struct onvm_nf_function_table *nf_function_table;
+        struct onvm_nf *nf;
+        int arg_offset;
+
+        const char *progname = argv[0];
+
+        nf_local_ctx = onvm_nflib_init_nf_local_ctx();
+
+        /* If we're using advanced rings also pass a custom cleanup function,
+         * this can be used to handle NF specific (non onvm) cleanup logic */
+        rte_atomic16_init(&signal_exit_flag);
+        rte_atomic16_set(&signal_exit_flag, 0);
+        onvm_nflib_start_signal_handler(nf_local_ctx, sig_handler);
+
+        /* No need to define a function table as adv rings won't run onvm_nflib_run */
+        nf_function_table = NULL;
+
+        if ((arg_offset = onvm_nflib_init(argc, argv, NF_TAG, nf_local_ctx, nf_function_table)) < 0) {
+                onvm_nflib_stop(nf_local_ctx);
+                if (arg_offset == ONVM_SIGNAL_TERMINATION) {
+                        printf("Exiting due to user termination\n");
+                        return 0;
+                } else {
+                        rte_exit(EXIT_FAILURE, "Failed ONVM init\n");
+                }
+        }
+
+        argc -= arg_offset;
+        argv += arg_offset;
+
+        nf = nf_local_ctx->nf;
+
+        if (parse_app_args(argc, argv, progname, nf) < 0) {
+                onvm_nflib_stop(nf_local_ctx);
+                rte_exit(EXIT_FAILURE, "Invalid command-line arguments\n");
+        }
+
+        struct onvm_nf_init_cfg *child_cfg;
+        child_cfg = onvm_nflib_init_nf_init_cfg(nf->tag);
+
+        /* Prepare init data for the child */
+        child_cfg->service_id = nf->service_id;
+        struct child_spawn_info *child_data = malloc(sizeof(struct child_spawn_info));
+        child_data->child_cfg = child_cfg;
+        child_data->parent = nf;
+
+        /* Increment the children count so that stats are displayed and NF does proper cleanup */
+        rte_atomic16_inc(&nf->thread_info.children_cnt);
+
+        pthread_create(&tx_thread, NULL, start_child, (void *)child_data);
+        if (print_stats_flag) {
+                pthread_create(&stats_thread, NULL, do_fq_stats_display, nf->data);
+        }
+
+        rx_loop(nf_local_ctx);
+        pthread_join(tx_thread, NULL);
+        if (print_stats_flag) {
+                pthread_join(stats_thread, NULL);
+        }
+
+        /* Free data passed to the child thread */
+        free(child_data);
+
+        /* Free allocated memory for fairqueue */
+        destroy_fairqueue((struct fairqueue_t **)&nf->data);
+        onvm_nflib_stop(nf_local_ctx);
+
+        printf("If we reach here, program is ending\n");
+        return 0;
+}
diff --git a/examples/fair_queue/fair_queue_helper.h b/examples/fair_queue/fair_queue_helper.h
new file mode 100644
index 000000000..47ee09432
--- /dev/null
+++ b/examples/fair_queue/fair_queue_helper.h
@@ -0,0 +1,261 @@
+/*********************************************************************
+ *                     openNetVM
+ *              https://sdnfv.github.io
+ *
+ *   BSD LICENSE
+ *
+ *   Copyright(c)
+ *            2020 National Institute of Technology Karnataka, Surathkal
+ *   All rights reserved.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * The name of the author may not be used to endorse or promote
+ *       products derived from this software without specific prior
+ *       written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * fair_queue_helper.h - functions and structures required to maintain
+ *      multiple queue and simulate fair queueing
+ ********************************************************************/
+
+#include <onvm_flow_table.h>
+#include <rte_cycles.h>
+#include <rte_hash_crc.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_ring.h>
+#include <rte_spinlock.h>
+
+#define QUEUE_SIZE (NF_QUEUE_RINGSIZE)
+
+/* Structure of each queue */
+struct fairqueue_queue {
+        uint32_t head;                                          // head of the queue
+        uint32_t tail;                                          // tail of the queue
+        struct rte_mbuf **pkts;                                 // array of packet pointers
+        rte_spinlock_t lock;                                    // lock to access the queue
+        uint64_t rx, rx_last, tx, tx_last;                      // maintaining stats
+        uint64_t rx_drop, tx_drop, rx_drop_last, tx_drop_last;  // maintaining stats
+};
+
+/* Fair queue structure */
+struct fairqueue_t {
+        struct fairqueue_queue **fq;  // pointer to each queue
+        uint16_t num_queues;          // number of queues
+};
+
+/*
+ * Allocate memory to the fairqueue_t structure and initialize the variables.
+ */
+static int
+setup_fairqueue(struct fairqueue_t **fairqueue, uint16_t num_queues) {
+        uint16_t i;
+
+        *fairqueue = (struct fairqueue_t *)rte_malloc(NULL, sizeof(struct fairqueue_t), 0);
+        if ((*fairqueue) == NULL) {
+                rte_exit(EXIT_FAILURE, "Unable to allocate memory for fair queue structure.\n");
+        }
+
+        (*fairqueue)->fq =
+            (struct fairqueue_queue **)rte_malloc(NULL, sizeof(struct fairqueue_queue *) * num_queues, 0);
+        if ((*fairqueue)->fq == NULL) {
+                rte_free(*fairqueue);
+                rte_exit(EXIT_FAILURE, "Unable to allocate memory for fair queue pointers.\n");
+        }
+
+        (*fairqueue)->num_queues = num_queues;
+
+        for (i = 0; i < num_queues; i++) {
+                (*fairqueue)->fq[i] = (struct fairqueue_queue *)rte_malloc(NULL, sizeof(struct fairqueue_queue), 0);
+                if ((*fairqueue)->fq[i] == NULL) {
+                        for (uint16_t j = 0; j < i; j++) {
+                                rte_free((*fairqueue)->fq[j]);
+                        }
+                        rte_free((*fairqueue)->fq);
+                        rte_free(*fairqueue);
+                        rte_exit(EXIT_FAILURE, "Unable to allocate memory for queue %d.\n", i + 1);
+                }
+                struct fairqueue_queue *fq = (*fairqueue)->fq[i];
+
+                fq->pkts = (struct rte_mbuf **)rte_malloc(NULL, sizeof(struct rte_mbuf *) * QUEUE_SIZE, 0);
+                if (fq->pkts == NULL) {
+                        for (uint16_t j = 0; j < i; j++) {
+                                rte_free((*fairqueue)->fq[j]);
+                        }
+                        rte_free((*fairqueue)->fq[i]);
+                        rte_free((*fairqueue)->fq);
+                        rte_free(*fairqueue);
+                        rte_exit(EXIT_FAILURE, "Unable to create queue %d.\n", i + 1);
+                }
+                fq->head = 0;
+                fq->tail = 0;
+                rte_spinlock_init(&fq->lock);
+
+                fq->rx = 0;
+                fq->rx_last = 0;
+                fq->tx = 0;
+                fq->tx_last = 0;
+                fq->rx_drop = 0;
+                fq->rx_drop_last = 0;
+                fq->tx_drop = 0;
+                fq->tx_drop_last = 0;
+        }
+        return 0;
+}
+
+/*
+ * Free memory allocated to the fairqueue_t struct.
+ */
+static int
+destroy_fairqueue(struct fairqueue_t **fairqueue) {
+        uint16_t i;
+
+        if ((*fairqueue) == NULL) {
+                return 0;
+        }
+
+        for (i = 0; i < (*fairqueue)->num_queues; i++) {
+                rte_free((*fairqueue)->fq[i]->pkts);
+                rte_free((*fairqueue)->fq[i]);
+        }
+        rte_free((*fairqueue)->fq);
+        rte_free(*fairqueue);
+        (*fairqueue) = NULL;
+        return 0;
+}
+
+/*
+ * Classify the incoming pkts based on the 5-tuple from the pkt header
+ * (src IP, dst IP, src port, dst port, protocol)
+ * into the queues of the fairqueue_t struct.
+ * Internally called by `fairqueue_enqueue`.
+ */
+static int
+get_enqueue_qid(struct fairqueue_t *fairqueue, struct rte_mbuf *pkt) {
+        struct onvm_ft_ipv4_5tuple key;
+        uint32_t hash_value;
+        int ret;
+
+        /* Obtain the 5-tuple values */
+        ret = onvm_ft_fill_key(&key, pkt);
+        if (ret < 0) {
+                printf("Received a non-IP4 packet!\n");
+                return -1;
+        }
+
+        /* Classify pkts based on the 5-tuple values */
+        hash_value = fairqueue->num_queues;
+        hash_value = rte_hash_crc_4byte(key.proto, hash_value);
+        hash_value = rte_hash_crc_4byte(key.src_addr, hash_value);
+        hash_value = rte_hash_crc_4byte(key.dst_addr, hash_value);
+        hash_value = rte_hash_crc_4byte(key.src_port, hash_value);
+        hash_value = rte_hash_crc_4byte(key.dst_port, hash_value);
+
+        return hash_value % fairqueue->num_queues;
+}
+
+/*
+ * Enqueue pkt to one of the queues of the fairqueue_t struct.
+ */
+static int
+fairqueue_enqueue(struct fairqueue_t *fairqueue, struct rte_mbuf *pkt) {
+        int qid;
+        struct fairqueue_queue *fq;
+
+        qid = get_enqueue_qid(fairqueue, pkt);
+        if (qid == -1) {
+                return -1;
+        }
+        fq = fairqueue->fq[qid];
+
+        rte_spinlock_lock(&fq->lock);
+        if (fq->head == fq->tail && fq->pkts[fq->head] != NULL) {
+                fq->rx_drop += 1;
+                return -1;
+        }
+        fq->pkts[fq->tail] = pkt;
+        fq->tail = (fq->tail + 1) % QUEUE_SIZE;
+        rte_spinlock_unlock(&fq->lock);
+
+        fq->rx += 1;
+
+        return 0;
+}
+
+/*
+ * Iterates through each queue atmost once and stops on finding a non-empty queue.
+ * Return the queue_id if queue isn't empty.
+ * Return -1 otherwise.
+ * Internally called by `fairqueue_dequeue`.
+ */
+static int
+get_dequeue_qid(struct fairqueue_t *fairqueue) {
+        static uint16_t qid = 0;
+        uint16_t start_qid;
+        struct fairqueue_queue *fq;
+
+        start_qid = qid;
+        do {
+                fq = fairqueue->fq[qid];
+
+                if (fq->pkts[fq->head] == NULL) {
+                        qid = (qid + 1) % fairqueue->num_queues;
+                        continue;
+                }
+
+                uint16_t temp_qid = qid;
+                qid = (qid + 1) % fairqueue->num_queues;
+                return temp_qid;
+        } while (qid != start_qid);
+
+        return -1;
+}
+
+/*
+ * Dequeue pkt from the fairqueue_t queues in a round robin fashion.
+ */
+static int
+fairqueue_dequeue(struct fairqueue_t *fairqueue, struct rte_mbuf **pkt) {
+        int qid;
+        struct fairqueue_queue *fq;
+
+        qid = get_dequeue_qid(fairqueue);
+
+        if (qid == -1) {
+                *pkt = NULL;
+                return -1;
+        }
+
+        fq = fairqueue->fq[qid];
+
+        /* Dequeue pkt */
+        rte_spinlock_lock(&fq->lock);
+        *pkt = fq->pkts[fq->head];
+        fq->pkts[fq->head] = NULL;
+        fq->head = (fq->head + 1) % QUEUE_SIZE;
+        rte_spinlock_unlock(&fq->lock);
+
+        fq->tx += 1;
+
+        return 0;
+}
diff --git a/examples/fair_queue/go.sh b/examples/fair_queue/go.sh
new file mode 120000
index 000000000..40babd806
--- /dev/null
+++ b/examples/fair_queue/go.sh
@@ -0,0 +1 @@
+../go.sh
\ No newline at end of file

From 15fc6f72815d10940ecd3ff1d61b3ef4a2c3ccc3 Mon Sep 17 00:00:00 2001
From: Tim Wood <timwood@gwu.edu>
Date: Mon, 25 Oct 2021 14:16:29 -0400
Subject: [PATCH 14/17] [Release Notes] ONVM Release Notes 21.09

Release notes for ONVM 21.09.

Commit log:

* initial releases doc

* Add descriptions

Co-authored-by: Lauren Hahn <62489998+Lhahn01@users.noreply.github.com>

* Update docs/Releases.md

Co-authored-by: Noah Chinitz <62520433+NoahChinitzGWU@users.noreply.github.com>

* Update docs/Releases.md

Co-authored-by: Elie Henne <63778310+elliotthenne@users.noreply.github.com>

* Update Releases.md

Co-authored-by: Lauren Hahn <62489998+Lhahn01@users.noreply.github.com>
Co-authored-by: Noah Chinitz <62520433+NoahChinitzGWU@users.noreply.github.com>
Co-authored-by: Elie Henne <63778310+elliotthenne@users.noreply.github.com>
---
 docs/Releases.md | 43 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)

diff --git a/docs/Releases.md b/docs/Releases.md
index 55939a7e8..31ca741c6 100644
--- a/docs/Releases.md
+++ b/docs/Releases.md
@@ -13,6 +13,49 @@ use a date based versioning system.  Now, a release version can look
 like `17.11` where the "major" number is the year and the "minor" number
 is the month.
 
+## v21.10 (10/2020): Bug Fixes, Test cases, Dev Environment Improvements
+This release focused on general bug fixing and improving our test/development environments.
+A CloudLab template will be available with the latest release here: [https://www.cloudlab.us/p/GWCloudLab/onvm](https://www.cloudlab.us/p/GWCloudLab/onvm)
+
+**New Features and NFs**
+
+- [[243](https://github.com/sdnfv/openNetVM/pull/243)] Adds L3 Switch example based on DPDK `l3fwd` sample code. This NF can forward packets either using longest prefix match or a hash table lookup. 
+- [[254](https://github.com/sdnfv/openNetVM/pull/254)] Adds Fair Queue NF that demonstrates how to use advanced rings mode to directly access packets and distribute them to a set of child NFs. Packets are "classified" using a CRC32 hash and assigned to a queue. Queues are then read from in Round Robin order to process packets of different types in a fair way. Contributed by  ([Rohit M P](https://github.com/rohit-mp)) from NITK. 
+- [[277](https://github.com/sdnfv/openNetVM/pull/277)] Adds support for Jumbo frame packets. Enable by adding a `-j` flag to the manager's go.sh script.
+
+
+**Testing and Development Improvements**
+
+- [[296](https://github.com/sdnfv/openNetVM/pull/296)] Adds unit test for NF messaging infrastructure and fixes memory leak related to overflow of message pools [[Issue 293](https://github.com/sdnfv/openNetVM/pull/293)]. 
+- [[297](https://github.com/sdnfv/openNetVM/pull/297)] Adds VS Code profile to simplify debugging of NFs. 
+- [[302](https://github.com/sdnfv/openNetVM/pull/302)] Adds NF chain performance test to measure and plot inter-NF throughput and latency.
+
+
+**Miscellaneous Bug and Documentation Fixes**
+
+- [[304](https://github.com/sdnfv/openNetVM/pull/304)] Fixes the NF_TAG of `aes_decrypt` in `openNetVM/examples/aes_decrypt/aesdecrypt.c`.
+- [[300](https://github.com/sdnfv/openNetVM/pull/300)] Updates MoonGen installation document to work with the new DPDK version.
+- [[270](https://github.com/sdnfv/openNetVM/pull/270)] Fixes issues with relative path in the onvm go script to find the web directory. Now using `$ONVM_HOME` instead of `..`.
+- [[272](https://github.com/sdnfv/openNetVM/pull/272)] Fixes two bugs (including Issue #233) where the NF rings would not be cleared after deallocation and an underflow bug in stats.
+- [[265](https://github.com/sdnfv/openNetVM/pull/265)] Updates Install README to provide further clarification as well as to include a missing package. 
+- [[267](https://github.com/sdnfv/openNetVM/pull/267)] Fixes typos in `onvm_pkt_helper.h`.
+
+Contributors:
+
+- Dennis Afanasev ([dennisafa](https://github.com/dennisafa))
+- Noah Chinitz ([NoahChinitzGWU](https://github.com/NoahChinitzGWU))
+- Benjamin De Vierno ([bdevierno1](https://github.com/bdevierno1))
+- Kevin Deems ([kevindweb](https://github.com/kevindweb))
+- Lauren Hahn ([lhahn01](https://github.com/Lhahn01))
+- Elliott (Elie) Henne ([elliotthenne](https://github.com/elliotthenne))
+- Vivek Jain ([vivek-anand-jain](https://github.com/Vivek-anand-jain))
+- Jack Kuo [JackKuo-tw](https://github.com/JackKuo-tw)
+- Catherine Meadows ([catherinemeadows](https://github.com/catherinemeadows))
+- Rohit M P ([rohit-mp](https://github.com/rohit-mp))
+- Leslie Monis [lesliemonis](https://github.com/lesliemonis)
+- Peng Wu ([PengWu-wp](https://github.com/PengWu-wp))
+
+
 ## v20.10 (10/2020): OS/Dependency Updates, Bug Fixes, New NFs
 A CloudLab template will be available with the latest release here: [https://www.cloudlab.us/p/GWCloudLab/onvm](https://www.cloudlab.us/p/GWCloudLab/onvm)
 

From 67a4d1a3e4b6350a9d60e3306c226eb6e987b9f5 Mon Sep 17 00:00:00 2001
From: Noah Chinitz <62520433+NoahChinitzGWU@users.noreply.github.com>
Date: Wed, 1 Dec 2021 20:22:51 -0500
Subject: [PATCH 15/17] [Bug Fix] ARP endianess fixed (#317)

Changes the source IP address in ARP reply from CPU byte ordering to BE.

Commit log:
* ARP endianess fixed
---
 examples/arp_response/arp_response.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/examples/arp_response/arp_response.c b/examples/arp_response/arp_response.c
index f7f4b7b71..6b23a0009 100644
--- a/examples/arp_response/arp_response.c
+++ b/examples/arp_response/arp_response.c
@@ -249,7 +249,7 @@ send_arp_reply(int port, struct rte_ether_addr *tha, uint32_t tip, struct onvm_n
         out_arp_hdr->arp_opcode = rte_cpu_to_be_16(RTE_ARP_OP_REPLY);
 
         rte_ether_addr_copy(&ports->mac[port], &out_arp_hdr->arp_data.arp_sha);
-        out_arp_hdr->arp_data.arp_sip = state_info->source_ips[ports->id[port]];
+        out_arp_hdr->arp_data.arp_sip = rte_cpu_to_be_32(state_info->source_ips[ports->id[port]]);
 
         out_arp_hdr->arp_data.arp_tip = tip;
         rte_ether_addr_copy(tha, &out_arp_hdr->arp_data.arp_tha);

From df351541ce025cecab79e6cdcf484d9ed28dfc08 Mon Sep 17 00:00:00 2001
From: Aaron Hill <62488058+aaronmhill01@users.noreply.github.com>
Date: Wed, 26 Jan 2022 08:42:51 -0500
Subject: [PATCH 16/17] [Bug Fix] speed_tester jumbo frames crash fix (#316)

Speed tester will no longer crash if you tell it to load a PCAP trace that contains jumbo frames without using the -j flag.

Commit log:

* [UPDATE] speed_tester jumbo frames crash fix

* [UPDTE] speed_tester jumbo frame usability improvement

Co-authored-by: aaron <aaron@localhost>
---
 examples/speed_tester/speed_tester.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/examples/speed_tester/speed_tester.c b/examples/speed_tester/speed_tester.c
index 4b487d30f..361d3d23c 100644
--- a/examples/speed_tester/speed_tester.c
+++ b/examples/speed_tester/speed_tester.c
@@ -310,6 +310,7 @@ nf_setup(struct onvm_nf_local_ctx *nf_local_ctx) {
         const u_char *packet;
         struct pcap_pkthdr header;
         char errbuf[PCAP_ERRBUF_SIZE];
+        uint32_t max_elt_size;
 
         if (pcap_filename != NULL) {
                 printf("Replaying %s pcap file\n", pcap_filename);
@@ -326,10 +327,24 @@ nf_setup(struct onvm_nf_local_ctx *nf_local_ctx) {
 
                 i = 0;
 
+                /* 
+                 * max_elt_size is the maximum preallocated memory size permitted for each packet, 
+                 * adjusted for the memory offset of the rte_mbuf struct and header/tail lengths
+                 */
+                
+                max_elt_size = pktmbuf_pool->elt_size - sizeof(struct rte_mbuf) - pktmbuf_pool->header_size - pktmbuf_pool->trailer_size;
+
                 while (((packet = pcap_next(pcap, &header)) != NULL) && (i < packet_number)) {
                         struct onvm_pkt_meta *pmeta;
                         struct onvm_ft_ipv4_5tuple key;
 
+                        /* Length of the packet cannot exceed preallocated storage size */
+                        if (header.caplen > max_elt_size) {
+                                nf_local_ctx->nf->stats.tx_drop++;
+                                nf_local_ctx->nf->stats.act_drop++;
+                                continue;
+                        }
+
                         pkt = rte_pktmbuf_alloc(pktmbuf_pool);
                         if (pkt == NULL)
                                 break;
@@ -355,7 +370,7 @@ nf_setup(struct onvm_nf_local_ctx *nf_local_ctx) {
                 onvm_nflib_return_pkt_bulk(nf_local_ctx->nf, pkts, pkts_generated);
         } else {
 #endif
-                /*  use default number of initial packets if -c has not been used */
+                /* Use default number of initial packets if -c has not been used */
                 packet_number = (use_custom_pkt_count ? packet_number : DEFAULT_PKT_NUM);
                 struct rte_mbuf *pkts[packet_number];
 

From 152271d6c3a31de5948d4bd8019205a3d4e3c846 Mon Sep 17 00:00:00 2001
From: Tim Wood <timwood@gwu.edu>
Date: Mon, 31 Jan 2022 12:22:56 -0500
Subject: [PATCH 17/17] ONVM 21.10

ONVM 21.10 Release
---
 docs/Releases.md | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/docs/Releases.md b/docs/Releases.md
index 31ca741c6..c862aa9ac 100644
--- a/docs/Releases.md
+++ b/docs/Releases.md
@@ -29,7 +29,7 @@ A CloudLab template will be available with the latest release here: [https://www
 - [[296](https://github.com/sdnfv/openNetVM/pull/296)] Adds unit test for NF messaging infrastructure and fixes memory leak related to overflow of message pools [[Issue 293](https://github.com/sdnfv/openNetVM/pull/293)]. 
 - [[297](https://github.com/sdnfv/openNetVM/pull/297)] Adds VS Code profile to simplify debugging of NFs. 
 - [[302](https://github.com/sdnfv/openNetVM/pull/302)] Adds NF chain performance test to measure and plot inter-NF throughput and latency.
-
+- [[308](https://github.com/sdnfv/openNetVM/pull/308)] Adds socket ID information to NF and manager logging print statements. 
 
 **Miscellaneous Bug and Documentation Fixes**
 
@@ -39,6 +39,9 @@ A CloudLab template will be available with the latest release here: [https://www
 - [[272](https://github.com/sdnfv/openNetVM/pull/272)] Fixes two bugs (including Issue #233) where the NF rings would not be cleared after deallocation and an underflow bug in stats.
 - [[265](https://github.com/sdnfv/openNetVM/pull/265)] Updates Install README to provide further clarification as well as to include a missing package. 
 - [[267](https://github.com/sdnfv/openNetVM/pull/267)] Fixes typos in `onvm_pkt_helper.h`.
+- [[317](https://github.com/sdnfv/openNetVM/pull/317)] Fixes the ARP NF endianness for source IP addresses. 
+- [[306](https://github.com/sdnfv/openNetVM/pull/306)] Updates linter installation script to use newer versions of cppcheck and Ubuntu.
+- [[316](https://github.com/sdnfv/openNetVM/pull/316)] Fixes Speed Tester NF so that it does not crash while loading a PCAP trace with jumbo frames without the correct flags.
 
 Contributors: