spinning up branches: phase 1

[pdurbin]

At tech hours we just talked about the idea of spinning up arbitrary branches on AWS. I was asked to make an issue we can estimate and I'm going to try to keep this small but the whiteboard notes below reflect some future thinking as well.

Definition of done:

• The https://github.com/IQSS/dataverse-ansible repo is modified to run on arbitrary pre-release (usually unmerged) branches rather than being hard coded to a specific release. See enable spinning up of arbitrary branches dataverse-ansible#22
• A script can be passed an arbitrary branch name and it spins up an EC2 instance, clones the main Dataverse git repo, clones the https://github.com/IQSS/dataverse-ansible repo, builds the artifacts, and deploys Dataverse.

[djbrooke]

Thanks @pdurbin, we'll estimate this today.

[pdurbin]

@donsizemore just fixed up dataverse-ansible so that it will deploy arbitrary branches (thanks!!). You just add branch name to tests/group_vars/vagrant.yml like this:

murphy:dataverse-ansible pdurbin$ git diff
diff --git a/tests/group_vars/vagrant.yml b/tests/group_vars/vagrant.yml
index 35fa041..f276df7 100644
--- a/tests/group_vars/vagrant.yml
+++ b/tests/group_vars/vagrant.yml
@@ -8,7 +8,7 @@ dataverse:
     blocked_endpoints: "admin,test"
     blocked_policy: "localhost-only"
     location: "http://localhost:8080/api"
-  branch:
+  branch: "4926-unlock-dataset-api"
   copyright: "© Your Institution"
   db:
     name: dvndb
murphy:dataverse-ansible pdurbin$ 

And then you run vagrant up. Heads up to @dlmurphy that this might be immediately useful to you when writing messaging. It works today.

[matthew-a-dunlap]

My first goal in picking this up is to explore scripting the creation of an EC2 instance and dropping in relevant information (e.g. branch name) for ansible to pick up and use for bringing up Dataverse

[matthew-a-dunlap]

Current state: the shell script is now creating an EC2 instance and passing the branch as expected to ansible. There are one or two outstanding issues related to having ansible work in our current usecase. One is that the current ansible process has an upgrade step which causes it to upgrade itself when running it locally, causing an error. The other is that for some reason ansible seems to use the latest release (its default functionality) even after building whatever arbitrary branch that it was pointed to. These are things to look into on Monday.

[donsizemore]

want me to comment out the upgrade step permanently? it wasn’t causing problems in Vagrant

the release version is also an extra variable you can pass to ansible; you’ll want to coordinate that with the specified branch

[matthew-a-dunlap]

@donsizemore that sounds great (removing the step). For the release version, not all branches have been released, what does ansible expect in that case? It seemed when I had the branch as develop and the release version as 4.9.2 (hardcoded default) none of the new develop changes were deployed

[donsizemore]

there was/is no standard for naming custom files per version, so i punted and had it explode the dvinstall.zip for a given version (and i bump the default with each release)

if there’s a branch specified ansible does a git clone and a mvn package, then overwrites the warfile before deploying.

if you want we can pull necessary modified files from a given branch (usually schema.xml or some bit of json) but i’d be interested in seeing the output of your ansible run; phil and i each verified that it will deploy a given branch properly.

[matthew-a-dunlap]

@donsizemore i may have something wrong in my setup, I'll pick up on it Monday!

[matthew-a-dunlap]

Looks like my issue with the branch was due to the order of operations of where ansible looks for attributes. I've worked around it and things flow correctly!

Right now the script & ansible only support branches on the IQSS dataverse repo due to similar variable issues. I decided to save that for phase 2 of this story.

[dlmurphy]

I just walked through the instructions and had a bear of a time trying to get things working. With some help from @matthew-a-dunlap I managed to eventually successfully spin up a branch and then terminate it. Here are the points where I had difficulty, our guide could perhaps address these:

• Adding the AWS CLI executable to my command line path was a very confusing process. Matthew linked me to Amazon's guide, but it doesn't explain the process very well, and we ended up having to find a stack overflow answer that finally got it to work. We should consider explaining this part of the AWS setup process in our guide.

• I followed all the steps under "Configure AWS CLI", but when I ran the script it totally ignored the config files. I ended up having to use the AWS configure command and input the config parameters that way. Maybe we should just instruct users to run that command instead of creating those text files.

[kcondon]

@pdurbin
I had aws cli already installed so I was able to spin up two separate branches using the command supplied, verified it worked, then destroyed the instance using the provided command.

Would like to discuss any doc changes that might help with Derek's reported issues. Otherwise seems ready to merge.

I did find one issue: lots of errors in logs about not being able to send email to smtp.yourinstitution.edu when doing basic tasks in app.

Also, it was pointed out that build number does not indicate which branch is running, wondering whether you can poke build.properties to hack that.

[pdurbin]

@kcondon thanks, that's good feedback. For the SMTP thing I opened IQSS/dataverse-ansible#28 but the branch should already be there. It was added in IQSS/dataverse-ansible@d4dc7cb . What's missing from the build number is the commit. Let's chat about it to make sure we're seeing the same thing. We can discuss doc changes too, like you said.

[donsizemore]

@pdurbin dataverse-ansible should be munging build.properties when a branch is passed. i'll check on that.

[pdurbin]

I sat with @dlmurphy to discuss his two bullets above and made improvements to the docs in 562c9c7 and 746eec5

I also spoke with @kcondon and @djbrooke and it sounds like we're happy with how phase 1 has turned out so I'm moving this to QA.

@donsizemore yes, the branch is being added to the bottom right corner but not the commit. I'll ping you in IRC or in the dataverse-ansible repo about this. Thanks for adding the branch in IQSS/dataverse-ansible@d4dc7cb !