Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Iqss/#8380 counter-processor version update #8391

Conversation

qqmyers
Copy link
Member

@qqmyers qqmyers commented Feb 2, 2022

What this PR does / why we need it: This PR updates the parsing of the counter-processor reports in Dataverse to be compatible with the v0.1.04 (latest) version of counter processor and updates the documents to reference that version. It also updates the documentation related to MaxMind's change from making the GeoLite Country database available from a fixed URL to something you have to sign up for and sign an agreement to download. The changes point to MaxMind's documentation about why this was done and how to obtain the database, and notes that this is an optional addition to counter-processor (used to report per country results).

Which issue(s) this PR closes:

Special notes for your reviewer: The vagrant script references the non-working GeoLite db download URL. I think it is currently failing gracefully but this section could perhaps be removed. Since the new process requires having an account/signing a EULA with MaxMind, probably configuring a cron job to run their update script, etc. I'm not sure it is something we could/should automate. So it may make sense to do something like write a message to say 'follow the guides to install the optional GeoLite Country db for counter-processor to capture per country MakeDataCount statistics.' I'm hoping someone might suggest a change for the script.

Suggestions on how to test this: I'll try to test at QDR and report on/fix any issues I find there. For QA testing, configure for Make Data Counts. Update the counter-processor version to v0.1.04 and test that daily results are still being captured. The update should also be backward compatible, i.e. running this PR with counter processor v0.0.1 should also work.

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

Is there a release notes update needed for this change?:

Additional documentation:

Copy link
Member

@pdurbin pdurbin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't running the code but overall, looks great. I'm excited that we won't be held back on an old releases and I'm glad to see the GeoLite issue get resolved. I'm left a few comments.

@@ -425,20 +425,30 @@ Counter Processor has only been tested on el7 (see "Linux" above). Please note t
As root, download and install Counter Processor::

cd /usr/local
wget https://github.com/CDLUC3/counter-processor/archive/v0.0.1.tar.gz
wget https://github.com/CDLUC3/counter-processor/archive/v0.1.04.tar.gz
tar xvfz v0.0.1.tar.gz
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tar still shows the old version.


As root, change to the Counter Processor directory you just created, download the GeoLite2-Country tarball, untar it, and copy the geoip database into place::
Counter Processor can report per country results if the optional GeoLite Country Database is installed. At present, this database is free but to use it one nust signing an agreement (EULA) with MaxMind.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"nust"

@@ -83,9 +83,9 @@ Configure Counter Processor

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a general comment I'm adding to the top... the branch has a # in it. I forget if this causes problems deploying to QA servers or not.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW: it is easy to rename the branch but I think that closes the PR (any way around that?) - let me know if this is needed for the QA build.


As root, change to the Counter Processor directory you just created, download the GeoLite2-Country tarball, untar it, and copy the geoip database into place::
Counter Processor can report per country results if the optional GeoLite Country Database is installed. At present, this database is free but to use it one nust signing an agreement (EULA) with MaxMind.
(The primary concern appears to be that individuals can opt-out of having their location tracked via IP address and, due to various privacy laws, MaxMind needs a way to comply with that for products it has 'sold' (for no cost in this case). Their agreement requires you to either configure automatic updates to the GeoLite Country database or be responsible on your own for managing take down notices.)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd put "sold" in double quotes instead of single quotes.

@@ -11,7 +11,7 @@ python3.6 -m ensurepip
COUNTER_USER=counter
echo "Ensuring Unix user '$COUNTER_USER' exists"
useradd $COUNTER_USER || :
COMMIT='a73dbced06f0ac2f0d85231e4d9dd4f21bee8487'
COMMIT='7974dad259465ba196ef639f48dea007cae8f9ac'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to change this right now but we could switch to a release instead of commit when downloading. Originally, we used a commit because there were no releases ( CDLUC3/counter-processor#2 ).

@pdurbin pdurbin assigned qqmyers and unassigned pdurbin Mar 14, 2022
@qqmyers qqmyers assigned pdurbin and unassigned qqmyers Mar 14, 2022
Copy link
Member

@pdurbin pdurbin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't run the code but the changes make sense.

@pdurbin pdurbin removed their assignment Mar 14, 2022
@kcondon kcondon self-assigned this Mar 21, 2022
qqmyers added a commit to QualitativeDataRepository/dataverse that referenced this pull request Apr 13, 2022
@kcondon kcondon removed their assignment Apr 14, 2022
@qqmyers
Copy link
Member Author

qqmyers commented Apr 14, 2022

To make counter-processor 0.1.04 the new default version (e.g. with install instructions for it in the guides, etc.), some additional work is needed on the PR to update the python version (to 3.7+) and update it's install instructions/scripts, etc.

@coveralls
Copy link

coveralls commented Apr 14, 2022

Coverage Status

Coverage decreased (-0.0005%) to 19.001% when pulling 6d4d16b on GlobalDataverseCommunityConsortium:IQSS/#8380-counterprocesor_version_update into 4e991dd on IQSS:develop.

@kcondon kcondon self-assigned this Apr 14, 2022
Installing GeoLite Country Database
===================================

Counter Processor can report per country results if the optional GeoLite Country Database is installed. At present, this database is free but to use it one must signing an agreement (EULA) with MaxMind.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Counter Processor can report per country results if the optional GeoLite Country Database is installed. At present, this database is free but to use it one must signing an agreement (EULA) with MaxMind.
Counter Processor can report per country results if the optional GeoLite Country Database is installed. At present, this database is free but to use it one must sign an agreement (EULA) with MaxMind.

@kcondon kcondon merged commit 0f4ff80 into IQSS:develop Apr 19, 2022
@pdurbin pdurbin added this to the 5.11 milestone May 4, 2022
qqmyers added a commit to QualitativeDataRepository/dataverse that referenced this pull request May 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants