Async resources feedback

Background

Based on CMCS staff feedback, the eRegulations team updated the site to give users the ability to view policy-related materials (rules, subregulatory guidance, implementation resources, etc.) for a specific regulation section. The updated site also includes a resources search feature that searches all links to policy-related materials. In late June 2022, we sent out an email survey asking for feedback on these new features.

This report was exported from Dovetail (CMCS Dovetail account required 🔒)

Findings

Participants were able to navigate to a familiar regulation section and found the associated subregualtory guidance to be useful

• P1 said they were able to quickly navigate to the citation they needed even though they only knew the part number and a specific keyword.

• P2 said they had to review the site’s video demonstration to remind themselves how to navigate the site. They found it helpful to have the demo on the site and said it would be useful for both new users and anyone who may need a refresher. They also said they liked that the resources sidebar included a comprehensive list of resources related to the subpart, and that the resources were organized by resource type (e.g., Technical Assistance for States, Toolkits).

• P3 found it “tremendously useful” to have subregulatory guidance organized by subpart.

Participants thought the results listing on the Resources page was easy to scan and they found it helpful to be able to conduct both broad and narrow searches

• The majority of participants (P1 and P2) said the resources listing was easy to scroll through. P1 said they liked having the resources listed in chronological order, and P2 said the information in the the listing helped them “get a flavor for what each resource covers relative to the content.” P1 also mentioned finding the related regulations links to be helpful.

• P1 said they weren’t sure when a particular SMD letter was released and were able to easily find it using the search resources feature.

• P2 said they liked being able to search with keywords or citations, and they would use the Resources page search if they didn't know a specific citation. If they knew the citation, they’d prefer to start with the regulation, then view associated resources for that specific citation, because they’d get more focused resource results. They also said they liked the different filter options on the Resources page.

• P3 said, “Being able to filter by so many different categories, creates a much more efficient and precise search.”

All three participants said they would use these new eRegulations features for their work

• All three participants were enthusiastic about using eRegulations to research regulations and associated resources. P2 said, “It’s so much easier to have all the resources, both regulations and sub-regulatory guidance, available from one website. I often have multiple browsers open when conducting research as I’m toggling between Medicaid.gov and whatever site I’m using at the time to look up the regulation."

• Although all three participants said they would also use the Resources page for their work, as noted above, P2 said they would only use the Resources page if they didn’t know the citation — if they did know the citation, they would start by viewing the regulation. The other two participants (P1 and P3) said they would use the Resources page, with P3 saying, “Yes, I will use this frequently. It is incredibly valuable. Thank you for putting together such a tremendous resource!”

Although the majority of participants were happy with the new features, two participants encountered bugs

• P1 encountered a bug when trying to filter by SMDLs without first selecting a regulation. We quickly resolved the issue and informed the participant.

• P2 encountered a bug when trying to view resources for a specific section (§ 438.3) from a regulation text page. When they clicked on the “View § 438.3 resources (77)” link, the resources sidebar didn’t update with the section’s resources as intended. We weren’t able to replicate this bug.

Things to consider

• Since these features are new, we need to make sure people are aware they’re available, and know how to navigate to them. P3 said they were worried people might not know to navigate to the Resources page to “search for terms within specific types of guidance.”

• A fourth participant (P4) didn’t complete the full survey, but shared some feedback via email. When executing a keyword search, they expected to see results that included both new and historical program names, because there are many programs with new names and the association with the historical name is critical. They provided a few examples: “SCHIP” and “CHIP,” “MSIS” and “T-MSIS,” and “non-MAGI eligibility” and ”disability eligibility regs.”