User matching problem with Azure OIDC

[ZadkielSantospirito]

Hi here,

We have an issue where Moodle users are not being matched with their M365 accounts causing them to receive the following error:

"Invalid login: User not found in Moodle. If this site has the "authpreventaccountcreation" setting enabled, this may mean you need an administrator to create an account for you first"

We are able to manually match users by uploading a csv file in the user matching tool in the plugin's sync settings but can't work out why the user matching is failing in the first place.

Any help greatly appreciated.

[ZadkielSantospirito]

For additional context we are using Moodle 3.9 and are on version 3.9.8 of the plugin.

In the logs of the \local_o365\task\usersync tasks here is an example of an affected user:

76791d08c633:20213 ......... Syncing user (M365 username)
76791d08c633:20213 ......... User doesn't exist in Moodle
76791d08c633:20213 ......... Not creating a Moodle user because that sync option is disabled.

[weilai-irl]

Hi @ZadkielSantospirito,

When logging in or running the usersync task, Moodle will try to find a Moodle user whose username matches the UPN of the Microsoft 365 user. Moodle gets the UPN from either Graph API call (when running the usersync task) or from ID token on successful user login (when logging in using auth_oidc plugin).

Could you check the UPN of the user who tries to login, and the username of the Moodle account that is expected to be for the same user, and confirm if they match please.

The error messages in both cases indicate that a Moodle user with username matching the UPN is not found, and while Moodle tries to create a new account with the username, the Moodle option "authpreventaccountcreation" is enabled, so new account cannot be created according to this setting.

Regards,
Lai

[ZadkielSantospirito]

Hi @weilai-irl,

Thanks for getting back to me.

The issue is that the UPN does exist in Moodle in these cases, the UPN does actually exist in Moodle, it's simply not being picked up during the usersync task. We use the Moodle email address as the field that contains the UPN rather than the Moodle username and those do always match in these cases.

We use a in-house middleware system to create the accounts in Moodle that passes the username and UPN to Moodle using a web service so we do have authpreventaccountcreation enabled (since as far as we understand, that would end up creating duplicate accounts - just to check, is that correct?).

Do you have any suggestions about what might be preventing the UPNs that are in Moodle from being found?

All the best,

Zad

[ZadkielSantospirito]

Hi @weilai-irl,

Any thoughts on the above?

All the best,

Zad

[weilai-irl]

Hi @ZadkielSantospirito,

Since you are matching Microsoft 365 account UPN with Moodle account email instead of username, do you have the "Match Azure usernames to moodle emails instead of moodle usernames during the sync" option checked in the user sync settings? By default matching is only checked between Microsoft UPN and Moodle account username.

Regards,
Lai

[ZadkielSantospirito]

Hi @weilai-irl ,

Thanks for getting back to me.

Yes, we do have that setting enabled in the sync settings.

Regards,
Zad

[weilai-irl]

Hi @ZadkielSantospirito,

I was reading your original reporting of the issue, and it looks that you were able to match Microsoft 365 accounts with Moodle accounts using the "User Matching" advanced tool. Then this is expected.

The "Match Azure usernames to moodle emails instead of moodle usernames during the sync" option in the "User sync" section of the "Sync Settings" tab is only applied when running the user sync scheduled task, and is not considered on user login.

When a user tries to login using their Microsoft 365 account, auth_oidc will check (1A) if the user's UPN is already matched to a Moodle user (in auth_oidc_tokens table), and if not, (1B) if the user's UPN is the same as the username of any existing Moodle user; then if either (1A) and (1B) are matched, it will check (2) if the user uses auth_oidc; if (2) is true, then it will try to login the user. It will not try to match Microsoft 365 account UPN with Moodle account email in this process.

So what you experienced is how the system is designed to work.

Regards,
Lai

[ZadkielSantospirito]

Hi @weilai-irl,

Thanks for the information. Does this mean that there is no way to run scheduled tasks to match Azure usernames to Moodle emails? If I'm understanding you correctly this means there's only a manual sync possible for the option to match Azure usernames with Moodle emails which seems strange.

We aren't trying to get the users matched on login - we are using the \local_o365\task\usersync scheduled task to match the Moodle usernames and Moodle emails - is that not something that's possible to do?

All the best,
Zad

[weilai-irl]

Hi @ZadkielSantospirito,

If what you want to achieve is just to map a Moodle account with a Microsoft 365 account, then either the "User matching" tool or the usersync scheduled task can do it.

When using the usersync scheduled task, assuming you have "Match Azure usernames to moodle emails instead of moodle usernames during the sync" option enabled, consider the following situation:

• You have an existing Moodle user with username "testuser1" and email "testuser1@example.com", using manual authentication.
• You have a Microsoft user with UPN "testuser1@example.com".
• When running the usersync task, the users will be matched, you will see something like the following in the log.

......... Syncing user testuser1@example.com
......... Found a user in Azure AD that seems to match a user in Moodle
......... moodle username: testuser1, aad upn: testuser1@example.com
......... Matched user, but did not switch them to OIDC.
......... Adding o365 object record for user.
......... User is now synced.

This will not change the authentication method of the user, so the user still needs to login using their Moodle username and password. Upon login, in the Microsoft block, the user will see a text "You are almost connected to Microsoft 365". The user will need to click the link in the block and login in the popup window to finish the connection, but even without it, the users are mapped.

Regards,
Lai

[ZadkielSantospirito]

Hi @weilai-irl,

The issue is that we are using OIDC for authentication so we need the users to be matched before before they log in.

Not sure if it helps but here's an example of the log output showing both a user that is matched and a user where matching has failed (I've replaced the actual upns as they are actual email addresses for data protection purposes).

ff71b7d5f0d5:9634 ......... Syncing user @bbk.ac.uk
ff71b7d5f0d5:9634 ......... The user is already using OpenID for authentication.
ff71b7d5f0d5:9634 ......... User is now synced.
ff71b7d5f0d5:9634 ......... Updating Moodle user data from Azure AD user data.
ff71b7d5f0d5:9634 ......... User is now updated.
ff71b7d5f0d5:9634 .........
ff71b7d5f0d5:9634 ......... Syncing user @bbk.ac.uk
ff71b7d5f0d5:9634 ......... User doesn't exist in Moodle
ff71b7d5f0d5:9634 ......... Not creating a Moodle user because that sync option is disabled.

While the logs say that the second user doesn't exist in Moodle, they can be found when searching under the user search if the email field is used for filtering and their UPN is put in for the email address.

Hoping this can help find what's not set up right.

All the best,

Zad

[ZadkielSantospirito]

Hi @weilai-irl,

To try to help you understand the problem, I'm attaching a screengrab of our settings:

Hope this helps!

All the best,

Zad

[ZadkielSantospirito]

Hi @weilai-irl,

Hope you're well.

Do you have any advice that can help us with our user matching issue? Any help will be very welcome indeed.

All the best,

Zad

[weilai-irl]

Hi @ZadkielSantospirito,

Sorry for not getting back to you. I was a bit busy last week.

I ran a test in your scenario and it generates different results. Here are my details:

• In Moodle, I updated the user sync settings as set in your screenshot, and temporarily disabled the user sync task.
• In Moodle, I created an account using auth_oidc, with username "user1" and email address"user1@example.com".
• In Microsoft, I created an account with UPN "user1@example.com".
• I ran the user sync task in Moodle, and it shows the user is matched.

......... Syncing user user1@example.com
......... The user is already using OIDC for authentication.
......... Adding o365 object record for user.
......... User is now synced.
......... Updating Moodle user data from Azure AD user data.
......... User is now updated.

Note compared to other already matched users, it contains a line "Adding o365 object record for user".

• I then confirmed the user can log in.

If this still doesn't work for you, could you confirm the Microsoft 365 user UPN and Moodle user name are example match, even for letter cases please.

Regards,
Lai

[phager-at]

Hi,

The mentioned test case was a lucky path due to the username still being equal to the first part of the UPN. Further down the method existing users are only identified by comparing all username variants (UPN, lower case UPN, UPN without @ and domain part, etc.) against the array's keys which are only usernames, no matter the setting to compare UPNs against Moodle mail addresses instead. The fix seems simple: just make the mail address the first selected column, so the returned array is indexed by Moodle user's mail addresses.

The only TODO remaining after my PRs concerns installations with (Moodle) setting allowaccountssameemail set and multiple users sharing the same mail address which can lead to problems for unique matching of users.

Regards,
Philipp

[weilai-irl]

@phager-at

Hi Philipp,

Thank you for making the pull request. What you described makes perfect sense. I'll include the PR in the next release after review and test.

Regards,
Lai

[ZadkielSantospirito]

Hi @weilai-irl and @phager-at,

Thanks for the attention on this issue. Is this believed to be able to fi our issue? Our UPNs are all uniform, are all lowercase, all contain @ and there are no instances that have been found in the unlinked accounts where duplicate emails have been the problem.

All the best,

Zad

[weilai-irl]

Hi @ZadkielSantospirito,

Please test out the latest release on your site (ideally in a test environment first, of course) to see if the changes helps. Please report back if it works.

@phager-at Thank you again for your contribution in this one.

Regards,
Lai

[phager-at]

Hi @ZadkielSantospirito,

Please test out the latest release on your site (ideally in a test environment first, of course) to see if the changes helps. Please report back if it works.

@phager-at Thank you again for your contribution in this one.

Regards, Lai

Glad I could be helpful!

[ZadkielSantospirito]

Hi @weilai-irl,

We've asked our Moodle hosts to upgrade our staging site to the latest version of the plugin however version 4.0.1 requires Moodle 4.0 and we are on 3.9.

There was also a 3.9.10 version released at the same time - can you confirm that this version also contains the relevant fix?

Many thanks,

Zad

[weilai-irl]

Hi @ZadkielSantospirito,

I confirm the latest release 3.9.10 for Moodle 3.9 contains the same fix.

Regards,
Lai

[ZadkielSantospirito]

Hi @weilai-irl,

We've had our Moodle hosts apply the 3.9.10 fix on our staging site however it has not fixed our issue.

What might be worth mentioning is that we discovered during our testing that students were being properly synched and that it was only staff that are not synching correctly.

The reason this might be significant is that staff accounts don't have their Moodle username as the first part of their UPN as opposed to students who do. Because of this we were very hopeful that the update would address the problem but for some still unknown reason it has not.

It seems like our issue does have some relation to the problem identified by @phager-at however there is still some issue remaining preventing the \local_o365\task\usersync task from properly matching staff accounts.

Our staff UPNs are in the format [initial].[surname]@bbk.ac.uk (so for instance my UPN is z.santospirito@bbk.ac.uk) and our student UPNs are in the format [username]@student.bbk.ac.uk.

Hoping this information will help with solving the problem.

Thanks in advance,

Zad

[ZadkielSantospirito]

Hi @weilai-irl,

Just following up on my previous message. We've now released the updated plugin to our production environment and it still hasn't resolved the issues we're having. Is there any advice you can offer? More than happy to provide any additional information you need.

All the best,

Zad

[weilai-irl]

Hi @ZadkielSantospirito,

Could you provide some more information to help me performing more investigation on this please. The information I need are:

• The Microsoft 365 account UPN and email for a user that has this issue.
• For the Moodle account that's supposed to be matched to the Microsoft account in question, I would need its username and email address. Take a note at the user ID too - I won't need it, but you will for the other information.
• Find if there is a matching record in the mdl_local_o365_objects table, by running SELECT * FROM mdl_local_o365_object WHERE type='user' AND o365name LIKE "[MICROSOFT ACCOUNT UPN]"; in your Moodle database, where [MICROSOFT ACCOUNT UPN] is the UPN of your Microsoft account. This should return exactly one record, with the moodleid field matching the Moodle user ID. Please confirm.
• Find if there is a matching record in the mdl_auth_oidc_token tab, by running SELECT * FROM mdl_auth_oidc_token WHERE oidcusername LIKE "[MICROSOFT ACCOUNT UPN]"; in your Moodle database, where [MICROSOFT ACCOUNT UPN] is the UPN of your Microsoft account. This query may return one or zero record. Please post back the result.

Regards,
Lai

[ZadkielSantospirito]

Hi @weilai-irl

Thanks for getting back to me.

The user's UPN (and username) is laura.green@bbk.ac.uk. I've run the queries on the database and neither table returns a result for the user. I've run control tests with a successfully matched user and the queries return the result expected.

Hopefully this helps identify what's going on.

All the best,

Zad

[weilai-irl]

Hi @ZadkielSantospirito,

Could you provide the details of the Moodle user that's expected to be matched with the named Microsoft 365 user (laura.green@bbk.ac.uk) please. I'll need her:

• username, and
• email address

If this is still doesn't work, I'd like to arrange a quick call with you, as it might be easier to figure the issue that way.

Regards,
Lai

[ZadkielSantospirito]

Hi @weilai-irl,

No problem, the user's username is ublgre006 and their email address is the same as the UPN (if you remember we're using the option "Match Azure usernames to Moodle emails instead of Moodle usernames during the sync").

If that doesn't work a quick call would be great. Let me know what you'd need from me to set that up.

All the best,

Zad

[weilai-irl]

Hi @ZadkielSantospirito,

I ran a test in my dev environment based on your answer, but it still gave different results as yours. I think a call might be easier.

Please send me an email to arrange a call so that we can walk through this together. My email can be found in my user profile.

Regards,
Lai

[weilai-irl]

After a call, we found out that this is caused by the user sync task falls back to use username, rather than email address as configured, as the key of the existing user cache, because it finds duplicate active Moodle accounts with duplicate email addresses. The logic is at

o365-moodle/local/o365/classes/feature/usersync/main.php

Lines 982 to 987 in 5786739

	if (local_o365_users_with_same_email_exist()) {
	// Match by email, but duplicate email exists, revert to match by username.
	$select = "SELECT LOWER(u.username) AS username, LOWER(u.email) AS email, ";
	} else {
	$select = "SELECT LOWER(u.email) AS email, LOWER(u.username) AS username, ";
	}

.

The current logic is if such duplicate accounts exist, then the complete existing user cache falls back to use username. This is too broad and need to be updated to reduce the impact of the fallback.

Lai

[weilai-irl]

Hi @ZadkielSantospirito,

I have provided a fix to the issue that we found. Please apply the fix to your test environment and give it a try.

Regards,
Lai

[ZadkielSantospirito]

Hi @weilai-irl,

Thanks for this - we really appreciate your help.

Could I just clarify whether the intention is for this fix to go into the main plugin at some point or would be be relying on staying on a forked version moving forwards?

All the best,

Zad

[weilai-irl]

@ZadkielSantospirito,

The fix is going to be included in the next release of the plugins.

Regards,
Lai

[weilai-irl]

Hi all,

The fix to the issue has been included in the release from today. Please upgrade your plugins to the latest version to see the changes.

I'm going to close the issue now.

Regards,
Lai