Skip to content

Embed picnic data in source to make it trim-friendly #534

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

Rob-Hague
Copy link
Contributor

Move the constant data for picnic from the bz2 files embedded as resources in the assembly to be embedded as static properties in the source. Those properties can then be analysed and trimmed away by the trimmer when unused.

In a test app which generates an ECDH key pair (i.e. does not use picnic), this results in the following changes to the size of BouncyCastle.Cryptography.dll:

before after
untrimmed 6989 KB 6975 KB
trimmed 3993 KB 2791 KB

That is, the untrimmed assembly size is unaffected with this change, but the trimmed assembly is reduced by 1202 KB which is exactly the size of the bz2 files.

Of the remaining 2791 KB, 2229 KB is the sike bz2 data, which is not treated here.

The source code was generated by reflecting over instances of the existing classes:

using Org.BouncyCastle.Pqc.Crypto.Picnic;
using System;
using System.CodeDom.Compiler;
using System.IO;
using System.Linq;
using System.Reflection;

LowmcConstantsL5 lc = new();

FieldInfo[] fields = lc.GetType().GetFields(BindingFlags.NonPublic | BindingFlags.Instance);

using StreamWriter sw = new(@"C:\tmp\LowmcConstants.txt");
using IndentedTextWriter tw = new(sw);

tw.Indent++; //namespace
tw.Indent++; //class
tw.WriteLine(); // to get tabs on next line

foreach (FieldInfo uintField in fields.Where(f => f.FieldType == typeof(uint[])))
{
    uint[] data = (uint[])uintField.GetValue(lc);

    int nonZeroDataLength = data.AsSpan().LastIndexOfAnyExcept(0u) + 1;

    tw.WriteLine($"/// <summary>Length: {nonZeroDataLength}</summary>");
    tw.WriteLine($"private static readonly uint[] s_{uintField.Name} = new uint[]");
    tw.WriteLine("{");
    tw.Indent++;

    foreach (uint[] chunk in data.Take(nonZeroDataLength).Chunk(8))
    {
        tw.Write("0x");
        tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X8"))));
        tw.WriteLine(",");
    }

    tw.Indent--;
    tw.WriteLine("};");
    tw.WriteLine();
}

Contributes to #422

Move the constant data for picnic from the bz2 files embedded as resources in the
assembly to be embedded as static properties in the source. Those properties can
then be analysed and trimmed away by the trimmer when unused.

In a test app which generates an ECDH key pair (i.e. does not use picnic), this
results in the following changes to the size of BouncyCastle.Cryptography.dll:

|           |  before |   after |
| --------- | ------- | ------- |
| untrimmed | 6989 KB | 6975 KB |
|   trimmed | 3993 KB | 2791 KB |

That is, the untrimmed assembly size is unaffected with this change, but the
trimmed assembly is reduced by 1202 KB which is exactly the size of the bz2 files.

Of the remaining 2791 KB, 2229 KB is the sike bz2 data, which is not treated here.

The source code was generated by reflecting over instances of the existing classes:

  using Org.BouncyCastle.Pqc.Crypto.Picnic;
  using System;
  using System.CodeDom.Compiler;
  using System.IO;
  using System.Linq;
  using System.Reflection;

  LowmcConstantsL5 lc = new();

  FieldInfo[] fields = lc.GetType().GetFields(BindingFlags.NonPublic | BindingFlags.Instance);

  using StreamWriter sw = new(@"C:\tmp\LowmcConstants.txt");
  using IndentedTextWriter tw = new(sw);

  tw.Indent++; //namespace
  tw.Indent++; //class
  tw.WriteLine(); // to get tabs on next line

  foreach (FieldInfo uintField in fields.Where(f => f.FieldType == typeof(uint[])))
  {
      uint[] data = (uint[])uintField.GetValue(lc);

      int nonZeroDataLength = data.AsSpan().LastIndexOfAnyExcept(0u) + 1;

      tw.WriteLine($"/// <summary>Length: {nonZeroDataLength}</summary>");
      tw.WriteLine($"private static readonly uint[] s_{uintField.Name} = new uint[]");
      tw.WriteLine("{");
      tw.Indent++;

      foreach (uint[] chunk in data.Take(nonZeroDataLength).Chunk(8))
      {
          tw.Write("0x");
          tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X8"))));
          tw.WriteLine(",");
      }

      tw.Indent--;
      tw.WriteLine("};");
      tw.WriteLine();
  }
@peterdettman
Copy link
Collaborator

Merged, thanks for the PR. If anyone wants to do something similar for Sike it would be welcome (it will be removed completely in 3.0, but that's probably still a ways off).

@Rob-Hague
Copy link
Contributor Author

Great, thanks. I'll give it a go - it seems slightly more complicated but still viable.

@Rob-Hague Rob-Hague deleted the bz2 branch May 20, 2024 12:21
Rob-Hague added a commit to Rob-Hague/bc-csharp that referenced this pull request May 21, 2024
Follow-up to bcgit#534, this time for the constant Sike data.

In the same test app on top of the same branch:

|           |  before bcgit#534 | after bcgit#534 |     now |
| --------- | ------------ | ---------- | ------- |
| untrimmed |      6989 KB |    6975 KB | 6892 KB |
|   trimmed |      3993 KB |    2791 KB |  564 KB |

i.e. the assembly now trims nicely with no cost to the untrimmed size.

As before, the source code was generated by reflecting over instances of the
existing classes:

  using Org.BouncyCastle.Pqc.Crypto.Sike;
  using System;
  using System.CodeDom.Compiler;
  using System.Diagnostics;
  using System.IO;
  using System.Linq;
  using System.Reflection;

  P434 lc = new(isCompressed: true);

  string[] bz2FieldNames =
  [
      "ph2_path",
      "ph3_path",
      "A_gen",
      "B_gen",
      "XQB3",
      "A_basis_zero",
      "B_basis_zero",
      "B_gen_3_tors",
      "g_R_S_im",
      "g_phiR_phiS_re",
      "g_phiR_phiS_im",
      "Montgomery_RB1",
      "Montgomery_RB2",
      "threeinv",
      "u_entang",
      "u0_entang",
      "table_r_qr",
      "table_r_qnr",
      "table_v_qr",
      "table_v_qnr",
      "v_3_torsion",
      "T_tate3",
      "T_tate2_firststep_P",
      "T_tate2_P",
      "T_tate2_firststep_Q",
      "T_tate2_Q",
      "ph2_T",
      "ph3_T1",
      "ph3_T2",
  ];

  FieldInfo[] fields = lc.GetType().GetFields(BindingFlags.NonPublic | BindingFlags.Instance);

  using StreamWriter sw = new(@"C:\tmp\out.txt");
  using IndentedTextWriter tw = new(sw);

  tw.Indent++; //class
  tw.WriteLine(); // to get tabs on next line

  foreach (string fieldName in bz2FieldNames)
  {
      FieldInfo field = fields.Single(f => f.Name == fieldName);

      if (field.FieldType == typeof(uint[]))
      {
          uint[] data = (uint[])field.GetValue(lc);

          int nonZeroDataLength = data.AsSpan().LastIndexOfAnyExcept(0u) + 1;

          tw.WriteLine($"private static readonly uint[] s_{field.Name} = new uint[{nonZeroDataLength}]");
          tw.WriteLine("{");
          tw.Indent++;

          foreach (uint[] chunk in data.Take(nonZeroDataLength).Chunk(8))
          {
              tw.Write("0x");
              tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X8"))));
              tw.WriteLine(",");
          }

          tw.Indent--;
          tw.WriteLine("};");
          tw.WriteLine();
      }
      else if (field.FieldType == typeof(ulong[]))
      {
          ulong[] data = (ulong[])field.GetValue(lc);

          int nonZeroDataLength = data.AsSpan().LastIndexOfAnyExcept(0u) + 1;

          tw.WriteLine($"private static readonly ulong[] s_{field.Name} = new ulong[{nonZeroDataLength}]");
          tw.WriteLine("{");
          tw.Indent++;

          foreach (ulong[] chunk in data.Take(nonZeroDataLength).Chunk(4))
          {
              tw.Write("0x");
              tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X16"))));
              tw.WriteLine(",");
          }

          tw.Indent--;
          tw.WriteLine("};");
          tw.WriteLine();
      }
      else if (field.FieldType == typeof(ulong[][]))
      {
          ulong[][] data = (ulong[][])field.GetValue(lc);

          tw.WriteLine($"private static readonly ulong[][] s_{field.Name} = new ulong[{data.Length}][]");
          tw.WriteLine("{");
          tw.Indent++;
          for (int i = 0; i < data.Length; i++)
          {
              int nonZeroDataLength = data[i].AsSpan().LastIndexOfAnyExcept(0u) + 1;

              tw.WriteLine($"new ulong[{nonZeroDataLength}]");
              tw.WriteLine("{");
              tw.Indent++;

              foreach (ulong[] chunk in data[i].Take(nonZeroDataLength).Chunk(4))
              {
                  tw.Write("0x");
                  tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X16"))));
                  tw.WriteLine(",");
              }

              tw.Indent--;
              tw.WriteLine("},");

          }
          tw.Indent--;
          tw.WriteLine("};");
          tw.WriteLine();
      }
      else if (field.FieldType == typeof(ulong[][][]))
      {
          ulong[][][] data = (ulong[][][])field.GetValue(lc);

          tw.WriteLine($"private static readonly ulong[][][] s_{field.Name} = new ulong[{data.Length}][][]");
          tw.WriteLine("{");
          tw.Indent++;
          for (int i = 0; i < data.Length; i++)
          {
              tw.WriteLine($"new ulong[{data[i].Length}][]");
              tw.WriteLine("{");
              tw.Indent++;

              for (int j = 0; j < data[i].Length; j++)
              {
                  int nonZeroDataLength = data[i][j].AsSpan().LastIndexOfAnyExcept(0u) + 1;

                  tw.WriteLine($"new ulong[{nonZeroDataLength}]");
                  tw.WriteLine("{");
                  tw.Indent++;

                  foreach (ulong[] chunk in data[i][j].Take(nonZeroDataLength).Chunk(4))
                  {
                      tw.Write("0x");
                      tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X16"))));
                      tw.WriteLine(",");
                  }

                  tw.Indent--;
                  tw.WriteLine("},");
              }

              tw.Indent--;
              tw.WriteLine("},");
          }
          tw.Indent--;
          tw.WriteLine("};");
          tw.WriteLine();
      }
      else
      {
          Debug.Fail(field.FieldType.ToString());
      }
  }
@Rob-Hague
Copy link
Contributor Author

I've got the change at master...Rob-Hague:bc-csharp:sike

It's currently based on GitHub/master so doesn't include this change and fails the picnic tests because it removes <EmbeddedResource Include="pqc\crypto\**\*.bz2" /> from the .csproj. Should I submit it against GitHub/master, this branch or wait for GitHub/master to update (or you can directly cherry-pick my branch)?

Presumably there are no conflicts whichever way.

Thanks

hubot pushed a commit that referenced this pull request May 22, 2024
Rob-Hague added a commit to Rob-Hague/bc-csharp that referenced this pull request May 22, 2024
Follow-up to 76e247 (bcgit#534),
this time for the constant Sike data.

In the same test app on top of the same branch:

|           |  before bcgit#534 | after bcgit#534 |     now |
| --------- | ------------ | ---------- | ------- |
| untrimmed |      6989 KB |    6975 KB | 6892 KB |
|   trimmed |      3993 KB |    2791 KB |  564 KB |

i.e. the assembly now trims nicely with no cost to the untrimmed size.

As before, the source code was generated by reflecting over instances of the
existing classes:

  using Org.BouncyCastle.Pqc.Crypto.Sike;
  using System;
  using System.CodeDom.Compiler;
  using System.Diagnostics;
  using System.IO;
  using System.Linq;
  using System.Reflection;

  P434 lc = new(isCompressed: true);

  string[] bz2FieldNames =
  [
      "ph2_path",
      "ph3_path",
      "A_gen",
      "B_gen",
      "XQB3",
      "A_basis_zero",
      "B_basis_zero",
      "B_gen_3_tors",
      "g_R_S_im",
      "g_phiR_phiS_re",
      "g_phiR_phiS_im",
      "Montgomery_RB1",
      "Montgomery_RB2",
      "threeinv",
      "u_entang",
      "u0_entang",
      "table_r_qr",
      "table_r_qnr",
      "table_v_qr",
      "table_v_qnr",
      "v_3_torsion",
      "T_tate3",
      "T_tate2_firststep_P",
      "T_tate2_P",
      "T_tate2_firststep_Q",
      "T_tate2_Q",
      "ph2_T",
      "ph3_T1",
      "ph3_T2",
  ];

  FieldInfo[] fields = lc.GetType().GetFields(BindingFlags.NonPublic | BindingFlags.Instance);

  using StreamWriter sw = new(@"C:\tmp\out.txt");
  using IndentedTextWriter tw = new(sw);

  tw.Indent++; //class
  tw.WriteLine(); // to get tabs on next line

  foreach (string fieldName in bz2FieldNames)
  {
      FieldInfo field = fields.Single(f => f.Name == fieldName);

      if (field.FieldType == typeof(uint[]))
      {
          uint[] data = (uint[])field.GetValue(lc);

          int nonZeroDataLength = data.AsSpan().LastIndexOfAnyExcept(0u) + 1;

          tw.WriteLine($"private static readonly uint[] s_{field.Name} = new uint[{nonZeroDataLength}]");
          tw.WriteLine("{");
          tw.Indent++;

          foreach (uint[] chunk in data.Take(nonZeroDataLength).Chunk(8))
          {
              tw.Write("0x");
              tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X8"))));
              tw.WriteLine(",");
          }

          tw.Indent--;
          tw.WriteLine("};");
          tw.WriteLine();
      }
      else if (field.FieldType == typeof(ulong[]))
      {
          ulong[] data = (ulong[])field.GetValue(lc);

          int nonZeroDataLength = data.AsSpan().LastIndexOfAnyExcept(0u) + 1;

          tw.WriteLine($"private static readonly ulong[] s_{field.Name} = new ulong[{nonZeroDataLength}]");
          tw.WriteLine("{");
          tw.Indent++;

          foreach (ulong[] chunk in data.Take(nonZeroDataLength).Chunk(4))
          {
              tw.Write("0x");
              tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X16"))));
              tw.WriteLine(",");
          }

          tw.Indent--;
          tw.WriteLine("};");
          tw.WriteLine();
      }
      else if (field.FieldType == typeof(ulong[][]))
      {
          ulong[][] data = (ulong[][])field.GetValue(lc);

          tw.WriteLine($"private static readonly ulong[][] s_{field.Name} = new ulong[{data.Length}][]");
          tw.WriteLine("{");
          tw.Indent++;
          for (int i = 0; i < data.Length; i++)
          {
              int nonZeroDataLength = data[i].AsSpan().LastIndexOfAnyExcept(0u) + 1;

              tw.WriteLine($"new ulong[{nonZeroDataLength}]");
              tw.WriteLine("{");
              tw.Indent++;

              foreach (ulong[] chunk in data[i].Take(nonZeroDataLength).Chunk(4))
              {
                  tw.Write("0x");
                  tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X16"))));
                  tw.WriteLine(",");
              }

              tw.Indent--;
              tw.WriteLine("},");

          }
          tw.Indent--;
          tw.WriteLine("};");
          tw.WriteLine();
      }
      else if (field.FieldType == typeof(ulong[][][]))
      {
          ulong[][][] data = (ulong[][][])field.GetValue(lc);

          tw.WriteLine($"private static readonly ulong[][][] s_{field.Name} = new ulong[{data.Length}][][]");
          tw.WriteLine("{");
          tw.Indent++;
          for (int i = 0; i < data.Length; i++)
          {
              tw.WriteLine($"new ulong[{data[i].Length}][]");
              tw.WriteLine("{");
              tw.Indent++;

              for (int j = 0; j < data[i].Length; j++)
              {
                  int nonZeroDataLength = data[i][j].AsSpan().LastIndexOfAnyExcept(0u) + 1;

                  tw.WriteLine($"new ulong[{nonZeroDataLength}]");
                  tw.WriteLine("{");
                  tw.Indent++;

                  foreach (ulong[] chunk in data[i][j].Take(nonZeroDataLength).Chunk(4))
                  {
                      tw.Write("0x");
                      tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X16"))));
                      tw.WriteLine(",");
                  }

                  tw.Indent--;
                  tw.WriteLine("},");
              }

              tw.Indent--;
              tw.WriteLine("},");
          }
          tw.Indent--;
          tw.WriteLine("};");
          tw.WriteLine();
      }
      else
      {
          Debug.Fail(field.FieldType.ToString());
      }
  }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants