Quiesce logging in pod for log generator workload

This along with cloud-bulldozer/benchmark-wrapper#273
helps suppress any unneeded logging in the pod, so that we can accurately and easily count
the number of log emssages recevied in a backend like kafka merely by using the offsets.
The plan is to deploy the log generator pods in a separate namesapce and forward those logs
to a topic in kafka. That way we would be able to reliably count the messages received just
by looking at kafka topic offset. Otherwise there would be other logs from the log generator pods
as well as benchmark-operator pod that would make it hard to reliably count logs received just by
kafka offset.

Signed-off-by: Sai Sindhur Malleni <smalleni@redhat.com>

docs/log_generator.md

@@ -26,6 +26,8 @@ This data will also be indexed if Elasticsearch information is provided.
 
 `timeout` how long, in seconds, after have been sent to allow the backend service to receive all the messages (default: 600)
 
+`snafu_disable_logs` Disable all logging in the pod from the snafu logger, thereby only leaving the generated log messages on stdout
+
 ### Verification variables:
 
 To verify your messages have been received by the backend aggregator you must provide information for ONLY ONE of the supported

roles/log_generator/templates/log_generator.yml

@@ -49,6 +49,8 @@ spec:
             value: "{{ test_user | default("ripsaw") }}"
           - name: clustername
             value: "{{ clustername }}"
+         - name: snafu_logs
+           value: "{{ workload_args.snafu_disable_logs | default(false) }}"
 {% if elasticsearch is defined %}
           - name: es
             value: "{{ elasticsearch.url }}"
@@ -73,7 +75,7 @@ spec:
         args:
           - > 
 {% if workload_args.pod_count | default(1) | int > 1 %}
-            echo "Waiting for all pods to be Ready";
+            if [[ "${snafu_disable_logs}" == "False" ]]; then echo "Waiting for all pods to be Ready"; fi;
             redis-cli -h {{ bo.resources[0].status.podIP }} INCR "log-generator-{{ trunc_uuid }}" > /dev/null 2>&1;
             pods=`redis-cli -h {{ bo.resources[0].status.podIP }} GET "log-generator-{{ trunc_uuid }}"`;
             while [[ $pods != {{ workload_args.pod_count | int }} ]]; do