Create public demo of UI

[patcon]

Re-ticketed from #219 (comment)

Thanks for your consideration, @lightandluck :)

I hope it's not presumptuous to transpose your and Rob's comments -- into 🔍facts, ❤️feelings, and 💡ideas -- so as to better engage with it.

(Generally, facts are things we all agree to be true. Feelings are subjective, not disputable, and optionally supported by facts. Ideas are ways forward, ideally based on facts & feelings.)

1. 🔍 @weatherpattern and @lightandluck spoke recently about the premise of a public demo.
2. ❤️ I feel that a demo that people can play with would be great.
3. 🔍 Current staging url is not accessible to public.
4. 🔍 Long discussion must happen before staging data is made public.
5. 💡 We could have a stripped down demo website.
6. ❤️ I feel that we don't have resources to maintain a separate demo.
7. 💡 We could create lots of screenshots and videos.

And now my own 🎉

1. ❤️ I feel that screenshots and videos might have their own upkeep and maintenance burden.
2. 🔍 Feature flags are a code/deploy pattern.
3. 🔍 Feature flags help avoid maintaining multiple codebases.
4. 🔍 Feature flags help the same codebase to run in multiple variations/deployments.
5. ❤️ I feel that feature flags are a great pattern even without this demo opportunity.
6. 💡 We could move toward using feature flags, in anticipation of offering a public demo.
7. ❤️ I feel that a demo or some explorable piece is important to unlocking more community resources.
8. ❤️ I feel that the sentiment of "There aren't enough resources [to provide a public demo]" does not fully account for the fact that a public demo is all about augmenting low human resources.
9. 🔍 EDGI and the Web Monitoring pipeline regularly make news.
10. 🔍 People in public can easily see the impact of this product through news reports.
11. ❤️ I feel that Web Monitoring is an exciting project!
12. ❤️ I feel that Web Monitoring has high capacity for community contribution.

[lightandluck]

This is great, Pat. I like how this help frames the discussion. Just a quick correction: it was @weatherpattern and I that had the chat, not Rob.

[lightandluck]

Thanks for your enthusiasm and love the feelings you have about the project! I think I share all of them. I recognize having something like this would help augment our limited resources and feature flags would be helpful ongoing.

A sticking point is that access to the api, and therefore everything useful (content & diffing), is restricted by login credentials. And how to open that up to the public is the long conversation I was referencing. (What types of users, what types of permissions, do we need login for certain things, etc.) This conversation probably goes beyond our working group as well. So, I'm not sure it would be as easy to change as flipping a bit.

[lightandluck]

And just to reiterate, I want this as badly as you do! (I mean I just suggested it a couple days ago) But figuring out how to do this well, at least, I don't have the bandwidth for. We're really trying hard to get our deployment down and also get off Versionista. I would love to revisit this once those are settled.

[patcon]

Totally. Thanks @lightandluck! I definitely feel where y'all are coming from. I'm equally excited to see action on the horizon as opposed to now :)

Without expectation of near-term action, I'll add a few more comments, hopefully building on yours:

1. 🔍 The dynamic real API is restricted by login credentials.
2. 🔍 The swagger JS client can be pointed to any API endpoint
3. 🔍 The API endpoint can be within web-monitoring-ui itself.
4. 🔍 A minimal set of static JSON fixtures can provide a mock API.
5. 🔍 A script could be written to quickly re-generate these fixtures from the real API as needed.
6. ❤️ I feel that even a single page being returned from mock /pages endpoint (with bare minimum of associated data from other endpoints) could be illustrative.
7. 💡 Store some minimal raw fixture data in web-monitoring-ui. When feature flag is enabled, the app could serve the data for itself, and override WEB_MONITORING_DB_URL. The served data would be for a single page and a bit of version history: just enough to drive a minimal UI experience.

No pressure to respond, but pls do feel free to correct any errors above.

[lightandluck]

Thanks, Pat, for sparking this convo! I've been thinking about it a lot over the last few days. I agree, all the facts are true. And honestly if it was up to me only I would

1. 💡Use feature-flags to hide annotations and assigned pages, instead of the hack we have now. (edit: We probably will need auth in some form for annotations, but for the demo, we could turn it off completely.)
2. 💡Create an api endpoint where we could just get the content and diffs without login.
3. ❤️ I feel the main point of contention was showing annotations and the public misinterpreting that.
4. ❤️ I don't think there's anything wrong with just showing content and diffing now.
5. 🔍 Others in Archiving should weigh in on this (Dawn, Matt P, SC members? I don't want to ping anybody yet).
6. ❤️ There may be some unresolved feelings that would make the "opening up to public" convo more difficult than just technical aspects.
7. 🔍 We do need to do it.
8. ❤️ This goes beyond WM and goes into the shared dream of integrating Scanner into IA (which I won't stop talking about until it happens). 😁

[Mr0grog]

Can we solve this for now by simply:

1. Disconnecting staging from the scraper (so we aren’t live importing into it), and
2. Creating a [non-admin] account on staging with publicly known credentials?

There was some back-and-forth a while ago on edgi-govdata-archiving/web-monitoring-db#152 about whether we even want staging to match production (which it currently sort-of does because the scraper sends the same data to both staging and production), and giving up on that and doing this also makes it easy to onboard new contributors — they can just connect to staging with the publicly known credentials.

That’s less simple than a version totally lacking in auth requirements, but it’s really low-effort and almost as good (maybe better since development and testing against it will be similar to production — not requiring any auth in a test/dev environment is an easy way to make mistakes about what can be expected once auth is back on in production).

/cc @danielballan since he was in the other discussion about staging matching/not matching production.

[Mr0grog]

Note we’ve had rough agreement that we don’t need to be careful about access to changes more than a month old, so as long as we strip any newer data from its DB and disconnect it from the scraper, we should be a-ok in the “can we make this public” department.

[Mr0grog]

/cc @gretchengehrke from the analyst team on privacy concerns.

[lightandluck]

@Mr0grog I like this idea! And looks like Dan and Patcon gave the thumbs up too.

This would also make it a lot easier to show people what we're doing and help recruit volunteers.

One thing to flag is that because we've been re-working the annotations, we'll probably want to import data with the settled changes.

[weatherpattern]

A public demo would be great. If we open things to the public, even for just a demo, we should determine if the terms of use docs need to be implemented, and if so, put them in place.

…

On Wed, Aug 29, 2018 at 12:13 PM Kevin Nguyen ***@***.***> wrote: @Mr0grog <https://github.com/Mr0grog> I like this idea! And looks like Dan and Patcon gave the thumbs up too. This would also make it a lot easier to show people what we're doing and help recruit volunteers. One thing to flag is that because we've been re-working the annotations, we'll probably want to import data with the settled changes. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#220 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABwEVsvAToL8VbczggSP4szf3oFKShSGks5uVr3DgaJpZM4TcHpM> .

-- http://www.weatherpattern.com http://twitter.com/weatherpattern

[Mr0grog]

because we've been re-working the annotations, we'll probably want to import data with the settled changes.

Does that apply here, though? I don’t think we should be surfacing analysts’ actual annotations in a public demo site, or at least not any time soon (this was the biggest and [in my view] most reasonable of all the original privacy issues). They certainly shouldn’t be on staging, regardless of private/public concerns.

[weatherpattern]

I am not sure, it depends on the eventual functionality of the demo. I raised the question to help ensure it is thought about. By the way, the clinical produced two versions of the summaries of use, one including annotations and one without.

…

On Thu, Aug 30, 2018 at 3:56 PM Rob Brackett ***@***.***> wrote: because we've been re-working the annotations, we'll probably want to import data with the settled changes. Does that apply here, though? I don’t think we should be surfacing analysts’ actual annotations in a public demo site, or at least not any time soon (this was the *biggest* and [in my view] most reasonable of all the original privacy issues). They certainly shouldn’t be on staging, regardless of private/public concerns. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#220 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABwEVpP9eryc3UXBD49rtCSHZHZ6lxekks5uWENegaJpZM4TcHpM> .

-- http://www.weatherpattern.com http://twitter.com/weatherpattern

[lightandluck]

I don’t think we should be surfacing analysts’ actual annotations in a public demo site

Oh, yea, you're right! For some reason, I was thinking that the data input from the scraping was bringing along annotations, but of course that's not happening.

Whenever the form gets turned on, it will be usable in the demo? We just won't fill it with real data?

[Mr0grog]

We just won't fill it with real data?

Well, we won’t fill it with our analyst team’s data. People would still be free to use it however they want, either by putting in meaningless stuff or real analysis. And we’d continue to make the same non-guarantee we do today about staging: everything could be flushed and totally reset at any time and we don’t consider any of its data to be worth preserving. At least that’s my proposal here.

That said, I did just think of one thing we probably want to be able to lock down here — imports.

[lightandluck]

+1 Rob's proposal

[Mr0grog]

OK, I’ve turned off live importing of data to staging, so from this point forward, it will cease to receive all the same data that we pull out of Versionista and send to production. That’s probably something we should have done anyway, but it’s also the first step here :)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in seven days if no further activity occurs. If it should not be closed, please comment! Thank you for your contributions.

[Mr0grog]

OK! We now have authorization support in DB, and I’ve created a user on the staging system (public.access@envirodatagov.org) that can only view.

What else needs to happen here?

• Probably add credentials for this user to our docs
• Is it critical that we get updated pages in versions from production moving regularly to staging? (I’d like to say no, though that’s a thing we should eventually get to.)
• Anything else?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in seven days if no further activity occurs. If it should not be closed, please comment! Thank you for your contributions.