Cache write is failing when converting ASCII-8BIT characters to UTF-8

[salbertson]

I'm getting a Encoding::UndefinedConversionError exception when trying to cache GitHub responses. Some responses will be cached, but some will fail to write because of an encoding conversion issue.

I'm using faraday 0.8.8 and faraday-http-cache 0.4.0 and here's the backtrace.

Any help would be greatly appreciated.

[lucasmazza]

@salbertson there is any public URL that is producing this error that we can use to debug it?

[salbertson]

@lucasmazza here's one.

https://api.github.com/repos/thoughtbot/factory_girl_rails

[salbertson]

I created a branch with a test to reproduce the issue. My first thought was to skip the write if the encoding fails.

salbertson/faraday-http-cache@master...sa-skip-write-when-encoding-errors

[lucasmazza]

@salbertson thanks for the report. I'm not sure of how to fix this without messing up with the response, as the issue lies with net/http not handling encoding properly and JSON failing hard when converting the body to UTF-8, but it sucks to ship with broken defaults like that.

On your app you can swap the serializer to Yajl or Marshal and they should handle the serialization better.

In the mean time, a PR that emits a warning with the storage logger (instead of puts) will be very appreciated 😄.

[salbertson]

@lucasmazza using Marshal worked perfectly. Thanks, I'll get a pull request opened for the warning.

[monfresh]

I just ran into this as well, and I was able to get rid of the error by using Marshal as the serializer, but now I'm getting this error for some URLs:

incompatible marshal file format (can't be read)
    format version 4.8 required; 123.34 given

Any idea what might be causing this and how to fix it?

Thanks!

[skorth]

Using Rails 4.2.0.rc3, warden-github, octokit and faraday-http-cache runs into a ASCII-8BIT to UTF-8 error. Its about the response body that holds a german "umlaut" like ü,ä or ö. As long as i use the faraday-http-cache middleware i get this error. Only way is to set Marshal as serializer instead of JSON. Didn't found any other solution for now.