Stub out LMS during MFE development

[jmbowman]

MFEs allow for simple, light-weight development environments with one glaring exception - the need to launch devstack or Tutor to interact with back end services, especially the LMS for authentication. This drags multiple Python services, databases, a search engine, etc. into the required development environment. We're making progress on supporting cloud-hosted dev environments that avoid the need to install all this on a laptop, but even better would be not to need them at all.

We'd like to start by replacing the LMS with a stub server that returns canned responses to the most common requests made to it by MFEs. Rather than building out a service from scratch which would leave us holding the bulk of the development and maintenance burden, we should use an existing stub server. The leading candidate is https://github.com/pact-foundation/pact-stub-server from the Pact contract testing framework, for the following reasons:

• We have already started using Pact, and believe broader usage could help reduce the number of production incidents involving broken communications between services.
• It is implemented in Rust for performance, and is distributed as a single binary.
• The associated Pact specification already has robust support for many kinds of stubbing needs, including the ability to change the state of the mocked service and alter the responses accordingly.
• It is an actively maintained component of a larger project with commercial backing, which is unlikely to be suddenly abandoned without a solid migration plan.

Here is more information about Pact in the context of Open edX.

The anticipated parts of this project are roughly:

Tasks

Beta Give feedback

1. Identify the communications with the LMS that need to be stubbed out
2. Figure out the best way to define the mock responses for these requests. This may be writing the Pact file by hand, or it may be creating some Pact tests between the services which will generate this file accordingly.
3. Generate the necessary Pact file.
4. Document and/or automate a simple process for using the stub server with an MFE instead of needing an actual LMS instance.
5. Write a plan for covering the stubbed communications with Pact tests (if they were already written in order to create the Pact file, this may only cover the steps to get them running in CI).
6. Document the process of similarly stubbing out new endpoints or additional back end services.
7. Identify and prioritize similar work that could be done to eliminate the need for other back end services during development of particular MFEs.

An unmerged hackathon project from a few months ago attempted a different approach to creating a mock authentication service, that code may be useful at least for identifying some of the request/response sequences that need to be stubbed out. It also has the beginnings of Pact testing for it.

[robrap]

@jmbowman: I removed @swayamrana as the assignee and marked this as "No Status" so we can re-prioritize. I'm not sure if you said this might go to fed-bom?

[Syed-Ali-Abbas-Zaidi]

Identify the communications with the LMS that need to be stubbed out

Each MFE relies on the cookie edx-jwt-cookie-header-payload; this cookie contains the JWT token needed for authentication purposes. The frontend code responsible for authentication is housed in frontend-platform so that it can be utilized by each MFE. This serves as the initial step for each MFE. For example, if this cookie is not set, the MFE will redirect the user to the backend, i.e., edx-platform. Therefore, the first step is to stub out this authentication-related communication between the MFE and LMS.

How we achieved the above mentioned scenario ☝️

What we did first was generate a JWT token with no expiry (we tweaked the functionality related to expiry on the backend). This JWT token was created for the admin user edx@example.com. We stored this JWT token in the constants of frontend-platform. After that, we created an environment variable named PACT_STUB_ENABLED in our MFE. If this environment variable is set to true, frontend-platform will not search for the cookie in the browser; instead, it will retrieve the cookie from constants and communicate to our MFE that the user is authenticated and the MFE is ready to load.

Figure out the best way to define the mock responses for these requests. This may be writing the Pact file by hand, or it may be creating some Pact tests between the services which will generate this file accordingly.

The best and much easier way is to simply create the mocked responses file by hand.

Generate the necessary Pact file

We created this Pact file for the Profile MFE as it is one of the simpler MFEs of edx. It has only a single route, /u/<user-name>, which calls a couple of GET APIs to fetch some information related to the user, such as Preferred Languages, Earned Certificates, etc. We mocked all these APIs in our Pact file, and finally, the frontend of the MFE started rendering (without devstack - yayyyyy! 🎉 ).

Our Questions/Concerns:

1. Does this work improve local development? Yes, we know that it removes devstack from the dependency, but it adds one more dependency. Now, if a developer wants to introduce a new feature, they have to mock that API manually.
2. What about POST APIs? Where do we store data? If we mock POST api in pact file then it will not reflect the data entered by the user. (We did RnD on using SQL or some other database to store/receive data. But Pact Stub Server only allows .json file for its interactions)

cc: @jmbowman