&tl;dr

If you’re using CrmServiceClient and getting what seems to be a valid connection but cannot really use it because of the “The user authentication failed!” error, check the servers’ clocks in your infrastructure.

Bo-o-o-oring

I have some perfect reasonable code deployed in Azure and that code has been humming along, talking to Dynamics 365 On-Premises for as long as I can remember. The code uses Xrm Tooling (yes, from time to time we listen to our own advice). Then one day – kabrakeeboom! – we get the log full of the following messages

Message: The user authentication failed!
StackTrace: at Microsoft.Xrm.Tooling.Connector
.CrmServiceClient.Execute(OrganizationRequest request)
blah-blah, our code

Quick check with the customer’s technical support – nothing has changed, no updates were installed on any of the servers. Sample code to troubleshoot:

string cs = "bleugh";
using (var client = new CrmServiceClient(cs))  
{
    if (client.IsReady)
    {
        Trace.WriteLine($"{client.GetMyCrmUserId()}");
        var foo = client.Execute(new WhoAmIRequest()) 
                        as WhoAmIResponse;
        Trace.WriteLine($"{foo.UserId}");
    }
}

Here’s the kick: IsReady is true and GetMyCrmUserId returns a valid value (probably cached) but Execute spits out “The user authentication failed”. Adding logging showed that coveted token gets invalidated immediately after the connection is made. It gets better: XrmToolBox would successfully connect to the instance as well but any attempt to use anything pops up “The user authentication failed!”

I’ll spare you the pain of ADFS debugging.

You know how they always synchronize watches in the action movies (there is even a song about it)? Now I know the reason.

ADFS server had its clock synchronized with the time service that was drifting over the time. Into the future. Once the drift was over 2 minutes, ADFS continued to issue tokens but they now were invalid the moment they left the server. Why nobody noticed and UI didn’t break? Because Dynamics 365 server was drifting as well, perfectly in sync with ADFS.