Blazor compile-time HTML encoding breaks £

[rynowak]

Do the blazor workshop up to https://github.com/dotnet-presentations/blazor-workshop/blob/master/docs/02-customize-a-pizza.md#add-additional-toppings - should take like 10 min

using 16.2p2 (public preview) and .NET Core 3.0 preview 6

You'll notice that the pound-sign (£) has been replaced in the generated code with a 65533 character - the replacement character. This means that instead of £ we tried to write some invalid unicode sequence and the encoder turned it into the replacement character.

Sorry, I don't have a more isolated repro of this and I tried a few things.

[rynowak]

@ajaybhargavb

[rynowak]

It might be the case here that the item-template is missing a BOM. If I create a new component using the item template and save, it doesn't have one.

[ajaybhargavb]

I looked at this and we have a lot of inconsistencies in our template. I created a new project with dotnet new blazorserverside and inspected each of the .razor file for BOM. Here is what I found,

Contains BOM:

• MyApp/Pages/_Imports.razor
• MyApp/Pages/Index.razor
• MyApp/Shared/MainLayout.razor
• MyApp/App.razor

Does not contain BOM:

• MyApp/_Imports.razor
• MyApp/Pages/_Host.cshtml
• MyApp/Pages/Counter.razor
• MyApp/Pages/FetchData.razor
• MyApp/Shared/NavMenu.razor

On top of this, VS New Item creation has inconsistencies as well,

• Creating a .razor file from Razor Component item template creates a file with no BOM
• Creating a .razor file from Razor View or Class item template creates a file with BOM

I also put the pound-sign (£) in many of these files and found that it does not render correctly on files with no BOM.

@rynowak @SteveSandersonMS @danroth27, do we want to require .razor files to contains UTF-8 BOM to work correctly?

Also, I filed an issue to handle the inconsistencies in our templates.

[SteveSandersonMS]

From your investigation, it sounds like we do need them to contain a BOM. But that's a bit awkward, since it's increasingly common these days not to use BOMs and for no-BOM to signal "UTF-8".

Is it possible to handle this in the Razor compiler by making it interpret no-BOM files as UTF-8?

[ajaybhargavb]

@mkArtakMSFT, I'm assigning this to myself.

[Andrzej-W]

I think the real problem is that Visual Studio saves files without BOM with ANSI encoding. I have Polish Windows (ANSI 1250) and English Visual Studio. Now lets assume I have a simple text file with only ASCII (English) text. I can open it in VS, add some Polish letters ąćęłńóśźż and save the file. If original file has BOM it will be saved as UTF-8 with BOM. If original file doesn't have BOM it will be saved as ANSI 1250.

In my projects I always have .editorconfig file. It is now widely supported. Maybe you should add a very basic .editorconfig file to every template similar to this:

# http://EditorConfig.org
root = true
[*]
charset = utf-8
# charset = utf-8-bom
indent_style = space
indent_size = 4
tab_width = 4
trim_trailing_whitespace = true
end_of_line = crlf
insert_final_newline = true

Probably there are a lot of people who are not aware that VS supports .editorconfig file. If you add it to the default template you will increase its popularity.

[ajaybhargavb]

After more investigation, turns out Blazor isn't the cause of this issue. Reading UTF-8 files without BOM works just fine. The cause of this issue was that for some reason Visual Studio was saving the .razor files as windows 1252 code page and when blazor tried to read it as UTF-8, that caused problems. We have fixed already fixed our templates so this shouldn't happen on newer projects.
For anyone seeing this issue, the workaround is to save the file in UTF8 encoding with/without BOM.