-
Notifications
You must be signed in to change notification settings - Fork 569
Embed picnic data in source to make it trim-friendly #534
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Move the constant data for picnic from the bz2 files embedded as resources in the assembly to be embedded as static properties in the source. Those properties can then be analysed and trimmed away by the trimmer when unused. In a test app which generates an ECDH key pair (i.e. does not use picnic), this results in the following changes to the size of BouncyCastle.Cryptography.dll: | | before | after | | --------- | ------- | ------- | | untrimmed | 6989 KB | 6975 KB | | trimmed | 3993 KB | 2791 KB | That is, the untrimmed assembly size is unaffected with this change, but the trimmed assembly is reduced by 1202 KB which is exactly the size of the bz2 files. Of the remaining 2791 KB, 2229 KB is the sike bz2 data, which is not treated here. The source code was generated by reflecting over instances of the existing classes: using Org.BouncyCastle.Pqc.Crypto.Picnic; using System; using System.CodeDom.Compiler; using System.IO; using System.Linq; using System.Reflection; LowmcConstantsL5 lc = new(); FieldInfo[] fields = lc.GetType().GetFields(BindingFlags.NonPublic | BindingFlags.Instance); using StreamWriter sw = new(@"C:\tmp\LowmcConstants.txt"); using IndentedTextWriter tw = new(sw); tw.Indent++; //namespace tw.Indent++; //class tw.WriteLine(); // to get tabs on next line foreach (FieldInfo uintField in fields.Where(f => f.FieldType == typeof(uint[]))) { uint[] data = (uint[])uintField.GetValue(lc); int nonZeroDataLength = data.AsSpan().LastIndexOfAnyExcept(0u) + 1; tw.WriteLine($"/// <summary>Length: {nonZeroDataLength}</summary>"); tw.WriteLine($"private static readonly uint[] s_{uintField.Name} = new uint[]"); tw.WriteLine("{"); tw.Indent++; foreach (uint[] chunk in data.Take(nonZeroDataLength).Chunk(8)) { tw.Write("0x"); tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X8")))); tw.WriteLine(","); } tw.Indent--; tw.WriteLine("};"); tw.WriteLine(); }
Merged, thanks for the PR. If anyone wants to do something similar for Sike it would be welcome (it will be removed completely in 3.0, but that's probably still a ways off). |
Great, thanks. I'll give it a go - it seems slightly more complicated but still viable. |
Follow-up to bcgit#534, this time for the constant Sike data. In the same test app on top of the same branch: | | before bcgit#534 | after bcgit#534 | now | | --------- | ------------ | ---------- | ------- | | untrimmed | 6989 KB | 6975 KB | 6892 KB | | trimmed | 3993 KB | 2791 KB | 564 KB | i.e. the assembly now trims nicely with no cost to the untrimmed size. As before, the source code was generated by reflecting over instances of the existing classes: using Org.BouncyCastle.Pqc.Crypto.Sike; using System; using System.CodeDom.Compiler; using System.Diagnostics; using System.IO; using System.Linq; using System.Reflection; P434 lc = new(isCompressed: true); string[] bz2FieldNames = [ "ph2_path", "ph3_path", "A_gen", "B_gen", "XQB3", "A_basis_zero", "B_basis_zero", "B_gen_3_tors", "g_R_S_im", "g_phiR_phiS_re", "g_phiR_phiS_im", "Montgomery_RB1", "Montgomery_RB2", "threeinv", "u_entang", "u0_entang", "table_r_qr", "table_r_qnr", "table_v_qr", "table_v_qnr", "v_3_torsion", "T_tate3", "T_tate2_firststep_P", "T_tate2_P", "T_tate2_firststep_Q", "T_tate2_Q", "ph2_T", "ph3_T1", "ph3_T2", ]; FieldInfo[] fields = lc.GetType().GetFields(BindingFlags.NonPublic | BindingFlags.Instance); using StreamWriter sw = new(@"C:\tmp\out.txt"); using IndentedTextWriter tw = new(sw); tw.Indent++; //class tw.WriteLine(); // to get tabs on next line foreach (string fieldName in bz2FieldNames) { FieldInfo field = fields.Single(f => f.Name == fieldName); if (field.FieldType == typeof(uint[])) { uint[] data = (uint[])field.GetValue(lc); int nonZeroDataLength = data.AsSpan().LastIndexOfAnyExcept(0u) + 1; tw.WriteLine($"private static readonly uint[] s_{field.Name} = new uint[{nonZeroDataLength}]"); tw.WriteLine("{"); tw.Indent++; foreach (uint[] chunk in data.Take(nonZeroDataLength).Chunk(8)) { tw.Write("0x"); tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X8")))); tw.WriteLine(","); } tw.Indent--; tw.WriteLine("};"); tw.WriteLine(); } else if (field.FieldType == typeof(ulong[])) { ulong[] data = (ulong[])field.GetValue(lc); int nonZeroDataLength = data.AsSpan().LastIndexOfAnyExcept(0u) + 1; tw.WriteLine($"private static readonly ulong[] s_{field.Name} = new ulong[{nonZeroDataLength}]"); tw.WriteLine("{"); tw.Indent++; foreach (ulong[] chunk in data.Take(nonZeroDataLength).Chunk(4)) { tw.Write("0x"); tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X16")))); tw.WriteLine(","); } tw.Indent--; tw.WriteLine("};"); tw.WriteLine(); } else if (field.FieldType == typeof(ulong[][])) { ulong[][] data = (ulong[][])field.GetValue(lc); tw.WriteLine($"private static readonly ulong[][] s_{field.Name} = new ulong[{data.Length}][]"); tw.WriteLine("{"); tw.Indent++; for (int i = 0; i < data.Length; i++) { int nonZeroDataLength = data[i].AsSpan().LastIndexOfAnyExcept(0u) + 1; tw.WriteLine($"new ulong[{nonZeroDataLength}]"); tw.WriteLine("{"); tw.Indent++; foreach (ulong[] chunk in data[i].Take(nonZeroDataLength).Chunk(4)) { tw.Write("0x"); tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X16")))); tw.WriteLine(","); } tw.Indent--; tw.WriteLine("},"); } tw.Indent--; tw.WriteLine("};"); tw.WriteLine(); } else if (field.FieldType == typeof(ulong[][][])) { ulong[][][] data = (ulong[][][])field.GetValue(lc); tw.WriteLine($"private static readonly ulong[][][] s_{field.Name} = new ulong[{data.Length}][][]"); tw.WriteLine("{"); tw.Indent++; for (int i = 0; i < data.Length; i++) { tw.WriteLine($"new ulong[{data[i].Length}][]"); tw.WriteLine("{"); tw.Indent++; for (int j = 0; j < data[i].Length; j++) { int nonZeroDataLength = data[i][j].AsSpan().LastIndexOfAnyExcept(0u) + 1; tw.WriteLine($"new ulong[{nonZeroDataLength}]"); tw.WriteLine("{"); tw.Indent++; foreach (ulong[] chunk in data[i][j].Take(nonZeroDataLength).Chunk(4)) { tw.Write("0x"); tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X16")))); tw.WriteLine(","); } tw.Indent--; tw.WriteLine("},"); } tw.Indent--; tw.WriteLine("},"); } tw.Indent--; tw.WriteLine("};"); tw.WriteLine(); } else { Debug.Fail(field.FieldType.ToString()); } }
I've got the change at master...Rob-Hague:bc-csharp:sike It's currently based on GitHub/master so doesn't include this change and fails the picnic tests because it removes Presumably there are no conflicts whichever way. Thanks |
Follow-up to 76e247 (bcgit#534), this time for the constant Sike data. In the same test app on top of the same branch: | | before bcgit#534 | after bcgit#534 | now | | --------- | ------------ | ---------- | ------- | | untrimmed | 6989 KB | 6975 KB | 6892 KB | | trimmed | 3993 KB | 2791 KB | 564 KB | i.e. the assembly now trims nicely with no cost to the untrimmed size. As before, the source code was generated by reflecting over instances of the existing classes: using Org.BouncyCastle.Pqc.Crypto.Sike; using System; using System.CodeDom.Compiler; using System.Diagnostics; using System.IO; using System.Linq; using System.Reflection; P434 lc = new(isCompressed: true); string[] bz2FieldNames = [ "ph2_path", "ph3_path", "A_gen", "B_gen", "XQB3", "A_basis_zero", "B_basis_zero", "B_gen_3_tors", "g_R_S_im", "g_phiR_phiS_re", "g_phiR_phiS_im", "Montgomery_RB1", "Montgomery_RB2", "threeinv", "u_entang", "u0_entang", "table_r_qr", "table_r_qnr", "table_v_qr", "table_v_qnr", "v_3_torsion", "T_tate3", "T_tate2_firststep_P", "T_tate2_P", "T_tate2_firststep_Q", "T_tate2_Q", "ph2_T", "ph3_T1", "ph3_T2", ]; FieldInfo[] fields = lc.GetType().GetFields(BindingFlags.NonPublic | BindingFlags.Instance); using StreamWriter sw = new(@"C:\tmp\out.txt"); using IndentedTextWriter tw = new(sw); tw.Indent++; //class tw.WriteLine(); // to get tabs on next line foreach (string fieldName in bz2FieldNames) { FieldInfo field = fields.Single(f => f.Name == fieldName); if (field.FieldType == typeof(uint[])) { uint[] data = (uint[])field.GetValue(lc); int nonZeroDataLength = data.AsSpan().LastIndexOfAnyExcept(0u) + 1; tw.WriteLine($"private static readonly uint[] s_{field.Name} = new uint[{nonZeroDataLength}]"); tw.WriteLine("{"); tw.Indent++; foreach (uint[] chunk in data.Take(nonZeroDataLength).Chunk(8)) { tw.Write("0x"); tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X8")))); tw.WriteLine(","); } tw.Indent--; tw.WriteLine("};"); tw.WriteLine(); } else if (field.FieldType == typeof(ulong[])) { ulong[] data = (ulong[])field.GetValue(lc); int nonZeroDataLength = data.AsSpan().LastIndexOfAnyExcept(0u) + 1; tw.WriteLine($"private static readonly ulong[] s_{field.Name} = new ulong[{nonZeroDataLength}]"); tw.WriteLine("{"); tw.Indent++; foreach (ulong[] chunk in data.Take(nonZeroDataLength).Chunk(4)) { tw.Write("0x"); tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X16")))); tw.WriteLine(","); } tw.Indent--; tw.WriteLine("};"); tw.WriteLine(); } else if (field.FieldType == typeof(ulong[][])) { ulong[][] data = (ulong[][])field.GetValue(lc); tw.WriteLine($"private static readonly ulong[][] s_{field.Name} = new ulong[{data.Length}][]"); tw.WriteLine("{"); tw.Indent++; for (int i = 0; i < data.Length; i++) { int nonZeroDataLength = data[i].AsSpan().LastIndexOfAnyExcept(0u) + 1; tw.WriteLine($"new ulong[{nonZeroDataLength}]"); tw.WriteLine("{"); tw.Indent++; foreach (ulong[] chunk in data[i].Take(nonZeroDataLength).Chunk(4)) { tw.Write("0x"); tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X16")))); tw.WriteLine(","); } tw.Indent--; tw.WriteLine("},"); } tw.Indent--; tw.WriteLine("};"); tw.WriteLine(); } else if (field.FieldType == typeof(ulong[][][])) { ulong[][][] data = (ulong[][][])field.GetValue(lc); tw.WriteLine($"private static readonly ulong[][][] s_{field.Name} = new ulong[{data.Length}][][]"); tw.WriteLine("{"); tw.Indent++; for (int i = 0; i < data.Length; i++) { tw.WriteLine($"new ulong[{data[i].Length}][]"); tw.WriteLine("{"); tw.Indent++; for (int j = 0; j < data[i].Length; j++) { int nonZeroDataLength = data[i][j].AsSpan().LastIndexOfAnyExcept(0u) + 1; tw.WriteLine($"new ulong[{nonZeroDataLength}]"); tw.WriteLine("{"); tw.Indent++; foreach (ulong[] chunk in data[i][j].Take(nonZeroDataLength).Chunk(4)) { tw.Write("0x"); tw.Write(string.Join(", 0x", chunk.Select(u => u.ToString("X16")))); tw.WriteLine(","); } tw.Indent--; tw.WriteLine("},"); } tw.Indent--; tw.WriteLine("},"); } tw.Indent--; tw.WriteLine("};"); tw.WriteLine(); } else { Debug.Fail(field.FieldType.ToString()); } }
Move the constant data for picnic from the bz2 files embedded as resources in the assembly to be embedded as static properties in the source. Those properties can then be analysed and trimmed away by the trimmer when unused.
In a test app which generates an ECDH key pair (i.e. does not use picnic), this results in the following changes to the size of BouncyCastle.Cryptography.dll:
That is, the untrimmed assembly size is unaffected with this change, but the trimmed assembly is reduced by 1202 KB which is exactly the size of the bz2 files.
Of the remaining 2791 KB, 2229 KB is the sike bz2 data, which is not treated here.
The source code was generated by reflecting over instances of the existing classes:
Contributes to #422