OAuth2 access token response parsing fails with nested JSON object

[buckett]

Summary

When parsing OAuth2 access token response a nested JSON object causes the response parsing to fail.

Actual Behavior

When attempting to use Spring Security OAuth to allow logins against a provider that responds with objects in their access token reponse an error message is shown:

An error occurred reading the OAuth 2.0 Access Token Response: JSON parse error: Cannot deserialize instance of `java.lang.String` out of START_OBJECT token; nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.lang.String` out of START_OBJECT token
 at [Source: (ByteArrayInputStream); line: 7, column: 14] (through reference chain: java.util.LinkedHashMap["object"]); nested exception is org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Cannot deserialize instance of `java.lang.String` out of START_OBJECT token; nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.lang.String` out of START_OBJECT token
 at [Source: (ByteArrayInputStream); line: 7, column: 14] (through reference chain: java.util.LinkedHashMap["object"])

Expected Behavior

According to the OAuth spec https://tools.ietf.org/html/rfc6749#section-5.1 clients must ignore values they don't understand. The value should either end up in the additionalParameters of the OAuth2AccessTokenResponse or it should be ignored.

Configuration

Jackson is being used to parse the JSON response (seems to be default in my spring-boot application).

Version

Spring Security 5.1.3, issue also looks to be present on master.

Sample

You can see a test case that currently fails in: https://github.com/spring-projects/spring-security/compare/master...buckett:oauth-response?expand=1

[buckett]

An example response that triggers this bug is:

{
   "access_token": "access-token-1234",
   "token_type": "bearer",
   "expires_in": "3600",
   "scope": "read write",
   "refresh_token": "refresh-token-1234",
   "object": {"param": "value" },
   "custom_parameter": "custom-value"
}

[buckett]

To workaround this I copied OAuth2AccessTokenResponseHttpMessageConverter into my own project and updated the tokenResponseParameters to be a Map<String, Object>. Here's a gist of the file: https://gist.github.com/buckett/7fe338901b4fcfefb6cfce637629af3f

Then I just connect this converter up to the RestTemplate that's used by the OAuth 2 code:

        .oauth2Login().tokenEndpoint().accessTokenResponseClient(accessTokenResposeClient());

[..snipped..]

    private OAuth2AccessTokenResponseClient<OAuth2AuthorizationCodeGrantRequest> accessTokenResposeClient() {
        DefaultAuthorizationCodeTokenResponseClient client = new DefaultAuthorizationCodeTokenResponseClient();
        RestTemplate restTemplate = new RestTemplate(Arrays.asList(
                new FormHttpMessageConverter(), new OAuth2AccessTokenResponseHttpMessageConverter()));
        HttpClient requestFactory = HttpClientBuilder.create().build();
        restTemplate.setRequestFactory(new HttpComponentsClientHttpRequestFactory(requestFactory));
        restTemplate.setErrorHandler(new OAuth2ErrorResponseErrorHandler());
        client.setRestOperations(restTemplate);
        return client;
    }

This isn't the actual code I'm using as I've removed other changes that aren't needed so it may not work as expected.

[jgrandja]

@buckett The solution you implemented with providing your own AbstractHttpMessageConverter<OAuth2AccessTokenResponse> is exactly what you would do for these kind of custom token responses. This is the main extension point for RestTemplate, the HttpMessageConverter for request/response serializing/deserializing. OAuth2AccessTokenResponseHttpMessageConverter is simply a default implementation.

It's not clear to me if you're still having a problem or are you expecting a different way of implementing your setup?

As per spec:

The parameters are serialized into a JavaScript Object Notation (JSON)
structure by adding each parameter at the highest structure level.
Parameter names and string values are included as JSON strings.
Numerical values are included as JSON numbers. The order of
parameters does not matter and can vary.

Based on my understanding how the spec reads, each parameter must be at the highest (root) structure level and are either strings or numbers. However, your example token response has a JSON object value at the root level with the parameter names/values at the next level below the object. So therefore the value of the root level parameter is an object and not a string or number as the spec dictates.

Does this make sense?

[buckett]

When I read the spec I was reading that paragraph as applying to the previously specified parameters (access_token, token_type, expires_in, refresh_token, scope).

It also goes on to say that the "the client MUST ignore unrecognized values names in the response", which again suggests we should be a little more forgiving in how we deal with unexpected value names.

If nothing else it would have been helpful if this was easier to code around as this took me a little while to debug (I came across this upgrading from Spring 5 to Spring 5.1, Spring 5 was forgiving of extra values I think).

[jgrandja]

As per spec:

The client MUST ignore unrecognized value names in the response. The
sizes of tokens and other values received from the authorization
server are left undefined. The client should avoid making
assumptions about value sizes. The authorization server SHOULD
document the size of any value it issues.

I see your point based on the reference in bold.

If nothing else it would have been helpful if this was easier to code around

Our goal is to make it easy for the user so this is valuable feedback for us. Do you have a suggestion on improving? Are you possibly interested in submitting a PR for this improvement?

[Gyurmatag]

When will this be fixed?

[jgrandja]

@Gyurmatag We'll do our best to fix this soon.

[jscoder1009]

when will this change be available? I checked my project maven dependencies and I am on spring-security-oauth2-core-5.2.1.RELEASE.jar. Once the change is live which version will have this change? thanks!

[jscoder1009]

To workaround this I copied OAuth2AccessTokenResponseHttpMessageConverter into my own project and updated the tokenResponseParameters to be a Map<String, Object>. Here's a gist of the file: https://gist.github.com/buckett/7fe338901b4fcfefb6cfce637629af3f

Then I just connect this converter up to the RestTemplate that's used by the OAuth 2 code:

        .oauth2Login().tokenEndpoint().accessTokenResponseClient(accessTokenResposeClient());

[..snipped..]

    private OAuth2AccessTokenResponseClient<OAuth2AuthorizationCodeGrantRequest> accessTokenResposeClient() {
        DefaultAuthorizationCodeTokenResponseClient client = new DefaultAuthorizationCodeTokenResponseClient();
        RestTemplate restTemplate = new RestTemplate(Arrays.asList(
                new FormHttpMessageConverter(), new OAuth2AccessTokenResponseHttpMessageConverter()));
        HttpClient requestFactory = HttpClientBuilder.create().build();
        restTemplate.setRequestFactory(new HttpComponentsClientHttpRequestFactory(requestFactory));
        restTemplate.setErrorHandler(new OAuth2ErrorResponseErrorHandler());
        client.setRestOperations(restTemplate);
        return client;
    }

This isn't the actual code I'm using as I've removed other changes that aren't needed so it may not work as expected.

how are you able to cast "client" to OAuth2AccessTokenResponseClient? any help is greatly appreciated.

[rwinch]

The change is in 5.3.0.BUILD-SNAPSHOT and will be in 5.3.0.RELEASE as indicated by the milestone selected with this issue.

[jscoder1009]

The change is in 5.3.0.BUILD-SNAPSHOT and will be in 5.3.0.RELEASE as indicated by the milestone selected with this issue.

thank you for the update.

[jscoder1009]

The change is in 5.3.0.BUILD-SNAPSHOT and will be in 5.3.0.RELEASE as indicated by the milestone selected with this issue.

I as was just wondering if there any options to reference to the snapshot version ? our app is live but broken for facebook login. I greatly appreciate your response.

[rwinch]

Please see the relevant section of the reference documentation https://docs.spring.io/spring-security/site/docs/5.3.0.BUILD-SNAPSHOT/reference/html5/#getting

[jscoder1009]

I upgraded to 5.3.0 for org.springframework.security.oauth2.core and still facing the issue. I am using facebook OAuth2 for user authentication.

org.springframework.http.converter.HttpMessageNotReadableException: An error occurred reading the OAuth 2.0 Error: Invalid JSON input: Cannot deserialize instance of java.lang.String out of START_OBJECT token; nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of java.lang.String out of START_OBJECT token
at [Source: (sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 1, column: 10] (through reference chain: java.util.LinkedHashMap["error"]); nested exception is org.springframework.http.converter.HttpMessageNotReadableException: Invalid JSON input: Cannot deserialize instance of java.lang.String out of START_OBJECT token; nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of java.lang.String out of START_OBJECT token
at [Source: (sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 1, column: 10] (through reference chain: java.util.LinkedHashMap["error"])
at org.springframework.security.oauth2.core.http.converter.OAuth2ErrorHttpMessageConverter.readInternal(OAuth2ErrorHttpMessageConverter.java:78) ~[spring-security-oauth2-core-5.3.0.RELEASE.jar!/:5.3.0.RELEASE]

[jgrandja]

@jscoder1009 This ticket is related to a parsing error for a Successful Token Response. However, based on the stacktrace, the issue you are having happens during an attempt to parse an Error Token Response.

org.springframework.security.oauth2.core.http.converter.OAuth2ErrorHttpMessageConverter.readInternal(OAuth2ErrorHttpMessageConverter.java:78)

It looks like Facebook does not return a correctly formatted OAuth 2.0 Error as defined in 5.2. Error Response.

You can override the default OAuth2ErrorResponseErrorHandler assigned to RestTemplate.setErrorHandler() in the associated OAuth2AccessTokenResponseClient (eg. DefaultAuthorizationCodeTokenResponseClient) to work around this issue with Facebook. Please see the ref doc for further details on custom configuration.

[josephbleau]

I landed here after a very long excursion! I wanted to report that Strava is also returning an object in one of their response attributes. Upgrading to 5.3.0 resolved the issue for me (was on 5.2.x).

https://developers.strava.com/docs/authentication/

Example Response

{
  "token_type": "Bearer",
  "expires_at": 1568775134,
  "expires_in": 21600,
  "refresh_token": "e5n567567...",
  "access_token": "a4b945687g...",
  "athlete": {
    #{summary athlete representation}
  }
}

Thanks everyone for their hard work!

[jscoder1009]

spring:
security:
oauth2:
client:
registration:
facebook:
client-id: xxddd...
clientSecret: secrettt....
redirect-uri: https://example.com/login/oauth2/code/facebook

Issue was redirect-uri was going as http. I explicitly added callback URL with https and this fixed my issue.

[keyuls]

@jscoder1009 where did you add callback URL with https?

[jmgomez77]

Dear all,

It seems the bug appears again in 5.4.6. Is this possible? As a result, the upgrade of Keycloak from 10.0 to 15.0 fails, since new values were added to the openid well-known configuration.

Below is the stack.

Kind regards.

Caused by: java.lang.RuntimeException: com.nimbusds.oauth2.sdk.ParseException: Unexpected type of JSON object member with key mtls_endpoint_aliases
	at org.springframework.security.oauth2.client.registration.ClientRegistrations.parse(ClientRegistrations.java:232) ~[spring-security-oauth2-client-5.4.6.jar:5.4.6]
	at org.springframework.security.oauth2.client.registration.ClientRegistrations.lambda$oidc$0(ClientRegistrations.java:157) ~[spring-security-oauth2-client-5.4.6.jar:5.4.6]
	at org.springframework.security.oauth2.client.registration.ClientRegistrations.getBuilder(ClientRegistrations.java:209) ~[spring-security-oauth2-client-5.4.6.jar:5.4.6]
	... 90 common frames omitted
Caused by: com.nimbusds.oauth2.sdk.ParseException: Unexpected type of JSON object member with key mtls_endpoint_aliases
	at com.nimbusds.oauth2.sdk.util.JSONObjectUtils.getGeneric(JSONObjectUtils.java:161) ~[oauth2-oidc-sdk-8.36.jar:8.36]
	at com.nimbusds.oauth2.sdk.util.JSONObjectUtils.getJSONObject(JSONObjectUtils.java:827) ~[oauth2-oidc-sdk-8.36.jar:8.36]
	at com.nimbusds.oauth2.sdk.as.AuthorizationServerMetadata.parse(AuthorizationServerMetadata.java:2042) ~[oauth2-oidc-sdk-8.36.jar:8.36]
	at com.nimbusds.openid.connect.sdk.op.OIDCProviderMetadata.parse(OIDCProviderMetadata.java:1362) ~[oauth2-oidc-sdk-8.36.jar:8.36]
	at org.springframework.security.oauth2.client.registration.ClientRegistrations.parse(ClientRegistrations.java:229) ~[spring-security-oauth2-client-5.4.6.jar:5.4.6]
	... 92 common frames omitted
Caused by: com.nimbusds.oauth2.sdk.ParseException: Unexpected type: class java.util.LinkedHashMap
	at com.nimbusds.oauth2.sdk.util.JSONUtils.to(JSONUtils.java:98) ~[oauth2-oidc-sdk-8.36.jar:8.36]
	at com.nimbusds.oauth2.sdk.util.JSONObjectUtils.getGeneric(JSONObjectUtils.java:159) ~[oauth2-oidc-sdk-8.36.jar:8.36]
	... 96 common frames omitted

[simonpai]

@jmgomez77
It seems to be a different issue caused by a bug in an underlying library com.nimbusds:oauth2-oidc-sdk:8.x:
https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions/issues/377/nimbus-8-prevent-auto-discover-mechanism

It has been fixed in 9.x and I think you can work around it with maven/gradle dependency management.

[yomal]

Hi all, I am facing the same issue using 5.1.7, and i am using this as for the client credentials flow.. Is there any workaround for this?

[laurocesar]

I have the same issue in version 5.4.5 and Keycloak 16.1.0:

Caused by: java.lang.RuntimeException: com.nimbusds.oauth2.sdk.ParseException: Unexpected type of JSON object member with key mtls_endpoint_aliases
        at org.springframework.security.oauth2.client.registration.ClientRegistrations.parse(ClientRegistrations.java:232)
        at org.springframework.security.oauth2.client.registration.ClientRegistrations.lambda$oidc$0(ClientRegistrations.java:157)
        at org.springframework.security.oauth2.client.registration.ClientRegistrations.getBuilder(ClientRegistrations.java:209)
        ... 113 common frames omitted
Caused by: com.nimbusds.oauth2.sdk.ParseException: Unexpected type of JSON object member with key mtls_endpoint_aliases
        at com.nimbusds.oauth2.sdk.util.JSONObjectUtils.getGeneric(JSONObjectUtils.java:161)
        at com.nimbusds.oauth2.sdk.util.JSONObjectUtils.getJSONObject(JSONObjectUtils.java:827)
        at com.nimbusds.oauth2.sdk.as.AuthorizationServerMetadata.parse(AuthorizationServerMetadata.java:2042)
        at com.nimbusds.openid.connect.sdk.op.OIDCProviderMetadata.parse(OIDCProviderMetadata.java:1362)
        at org.springframework.security.oauth2.client.registration.ClientRegistrations.parse(ClientRegistrations.java:229)
        ... 115 common frames omitted
Caused by: com.nimbusds.oauth2.sdk.ParseException: Unexpected type: class java.util.LinkedHashMap
        at com.nimbusds.oauth2.sdk.util.JSONUtils.to(JSONUtils.java:100)
        at com.nimbusds.oauth2.sdk.util.JSONObjectUtils.getGeneric(JSONObjectUtils.java:159)
        ... 119 common frames omitted

[imaxkhan]

hi issue with getting token from azure b2c
this method from internal library raise the error
return this.restOperations.exchange(request, OAuth2AccessTokenResponse.class);

[invalid_token_response] An error occurred while attempting to retrieve the OAuth 2.0 Access Token Response: Error while extracting response for type [class org.springframework.security.oauth2.core.endpoint.OAuth2AccessTokenResponse] and content type [application/json;charset=utf-8][invalid_token_response]