Iqss/#8380 counter-processor version update

doc/sphinx-guides/source/admin/make-data-count.rst

@@ -83,9 +83,9 @@ Configure Counter Processor
 
 * Change to the directory where you installed Counter Processor.
 
-  * ``cd /usr/local/counter-processor-0.0.1``
+  * ``cd /usr/local/counter-processor-0.1.04``
 
-* Download :download:`counter-processor-config.yaml <../_static/admin/counter-processor-config.yaml>` to ``/usr/local/counter-processor-0.0.1``.
+* Download :download:`counter-processor-config.yaml <../_static/admin/counter-processor-config.yaml>` to ``/usr/local/counter-processor-0.1.04``.
 
 * Edit the config file and pay particular attention to the FIXME lines.
 
@@ -98,7 +98,7 @@ Soon we will be setting up a cron job to run nightly but we start with a single
 
 * Change to the directory where you installed Counter Processor.
 
-  * ``cd /usr/local/counter-processor-0.0.1``
+  * ``cd /usr/local/counter-processor-0.1.04``
 
 * If you are running Counter Processor for the first time in the middle of a month, you will need create blank log files for the previous days. e.g.:
 

doc/sphinx-guides/source/developers/make-data-count.rst

@@ -51,7 +51,7 @@ Once you are done with your configuration, you can run Counter Processor like th
 
 ``su - counter``
 
-``cd /usr/local/counter-processor-0.0.1``
+``cd /usr/local/counter-processor-0.1.04``
 
 ``CONFIG_FILE=counter-processor-config.yaml python36 main.py``
 
@@ -84,7 +84,7 @@ Second, if you are also sending your SUSHI report to Make Data Count, you will n
 
 ``curl -H "Authorization: Bearer $JSON_WEB_TOKEN" -X DELETE https://$MDC_SERVER/reports/$REPORT_ID``
 
-To get the ``REPORT_ID``, look at the logs generated in ``/usr/local/counter-processor-0.0.1/tmp/datacite_response_body.txt``
+To get the ``REPORT_ID``, look at the logs generated in ``/usr/local/counter-processor-0.1.04/tmp/datacite_response_body.txt``
 
 To read more about the Make Data Count api, see https://github.com/datacite/sashimi
 

doc/sphinx-guides/source/installation/prerequisites.rst

@@ -96,7 +96,7 @@ PostgreSQL
 ----------
 
 Installing PostgreSQL
-=======================
+=====================
 
 The application has been tested with PostgreSQL versions up to 13 and version 10+ is required. We recommend installing the latest version that is available for your OS distribution. *For example*, to install PostgreSQL 13 under RHEL7/derivative::
 
@@ -425,20 +425,30 @@ Counter Processor has only been tested on el7 (see "Linux" above). Please note t
 As root, download and install Counter Processor::
 
         cd /usr/local
-        wget https://github.com/CDLUC3/counter-processor/archive/v0.0.1.tar.gz
-        tar xvfz v0.0.1.tar.gz
+        wget https://github.com/CDLUC3/counter-processor/archive/v0.1.04.tar.gz
+        tar xvfz v0.1.04.tar.gz
+        cd /usr/local/counter-processor-0.1.04
+
+Installing GeoLite Country Database
+===================================
 
-As root, change to the Counter Processor directory you just created, download the GeoLite2-Country tarball, untar it, and copy the geoip database into place::
+Counter Processor can report per country results if the optional GeoLite Country Database is installed. At present, this database is free but to use it one must signing an agreement (EULA) with MaxMind. 
+(The primary concern appears to be that individuals can opt-out of having their location tracked via IP address and, due to various privacy laws, MaxMind needs a way to comply with that for products it has "sold" (for no cost in this case). Their agreement requires you to either configure automatic updates to the GeoLite Country database or be responsible on your own for managing take down notices.)
+The process required to sign up, download the database, and to configure automated updating is described at https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/ and the links from that page.
 
-        cd /usr/local/counter-processor-0.0.1
-        wget https://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz
+As root, change to the Counter Processor directory you just created, download the GeoLite2-Country tarball from MaxMind, untar it, and copy the geoip database into place::
+
+        <download or move the GeoLite2-Country.tar.gz to the /usr/local/counter-processor-0.1.04 directory>
         tar xvfz GeoLite2-Country.tar.gz
         cp GeoLite2-Country_*/GeoLite2-Country.mmdb maxmind_geoip
 
+Creating a counter User
+=======================
+
 As root, create a "counter" user and change ownership of Counter Processor directory to this new user::
 
         useradd counter
-        chown -R counter:counter /usr/local/counter-processor-0.0.1
+        chown -R counter:counter /usr/local/counter-processor-0.1.04
 
 Installing Counter Processor Python Requirements
 ================================================
@@ -456,7 +466,7 @@ Install Python 3.6::
 Install Counter Processor Python requirements::
 
         python3.6 -m ensurepip
-        cd /usr/local/counter-processor-0.0.1
+        cd /usr/local/counter-processor-0.1.04
         pip3 install -r requirements.txt
 
 See the :doc:`/admin/make-data-count` section of the Admin Guide for how to configure and run Counter Processor.

scripts/vagrant/setup-counter-processor.sh

@@ -11,7 +11,7 @@ python3.6 -m ensurepip
 COUNTER_USER=counter
 echo "Ensuring Unix user '$COUNTER_USER' exists"
 useradd $COUNTER_USER || :
-COMMIT='a73dbced06f0ac2f0d85231e4d9dd4f21bee8487'
+COMMIT='7974dad259465ba196ef639f48dea007cae8f9ac'
 UNZIPPED_DIR="counter-processor-$COMMIT"
 if [ ! -e $UNZIPPED_DIR ]; then
   ZIP_FILE="${COMMIT}.zip"
@@ -20,19 +20,7 @@ if [ ! -e $UNZIPPED_DIR ]; then
   unzip $ZIP_FILE
 fi
 cd $UNZIPPED_DIR
-GEOIP_DIR='maxmind_geoip'
-GEOIP_FILE='GeoLite2-Country.mmdb'
-GEOIP_PATH_TO_FILE="$GEOIP_DIR/$GEOIP_FILE"
-if [ ! -e $GEOIP_PATH_TO_FILE ]; then
-  echo "let's do this thing"
-  TARBALL='GeoLite2-Country.tar.gz'
-  wget https://geolite.maxmind.com/download/geoip/database/$TARBALL
-  tar xfz GeoLite2-Country.tar.gz
-  # Glob (*) below because of directories like "GeoLite2-Country_20181204".
-  GEOIP_UNTARRED_DIR='GeoLite2-Country_*'
-  mv $GEOIP_UNTARRED_DIR/$GEOIP_FILE $GEOIP_PATH_TO_FILE
-  rm -rf $TARBALL $GEOIP_UNTARRED_DIR
-fi
+echo "Installation of the GeoLite2 country database for counter-processor can no longer be automated. See the Installation Guide for the manual installation process."
 pip3 install -r requirements.txt
 # For now, parsing sample_logs/counter_2018-05-08.log
 for i in `echo {00..31}`; do

src/main/java/edu/harvard/iq/dataverse/makedatacount/DatasetMetricsServiceBean.java

@@ -118,7 +118,12 @@ public List<DatasetMetrics> parseSushiReport(JsonObject report){
 
     public List<DatasetMetrics> parseSushiReport(JsonObject report, Dataset dataset) {
         List<DatasetMetrics> datasetMetricsAll = new ArrayList<>();
-        JsonArray reportDatasets = report.getJsonArray("report_datasets");
+        //Current counter-processor v 0.1.04+ format
+        JsonArray reportDatasets = report.getJsonArray("report-datasets");
+        if(reportDatasets==null) {
+            //Try counter-processor v 0.0.1 name
+            reportDatasets = report.getJsonArray("report_datasets");
+        }
         for (JsonValue reportDataset : reportDatasets) {
             List<DatasetMetrics> datasetMetricsDataset = new ArrayList<>();
             String globalId = null;