Sftp Uploads and BufferSize File Corruption

[RickStrahl]

I'm investigating a file corruption issue with large FTP uploads when using custom buffer sizes.

It looks like failures occur only with files larger and 11 megs or so, and only if the default buffer size is changed. I'm attaching a large generated file and when uploading this file I end up with an output file of the same size, but corrupted data. It doesn't happen on all files, but

The following code fails with the fail attached:

[TestMethod]
public void RawUploadTest()
{
	int BufferSize = 24000;
	string localFile = @"C:\webconnection\Fox\SENDFILETEST.ZZZ";
	string username = "tester";
	string password = "password";
	string host = "127.0.0.1";
	int port = 23;

	var methods = new List<AuthenticationMethod>();
	methods.Add(new PasswordAuthenticationMethod(username, password));

	using (var sftpClient = new SftpClient(host, port, username, password))
	{
		sftpClient.OperationTimeout = TimeSpan.FromSeconds(20);
		sftpClient.BufferSize = (uint) BufferSize;

		sftpClient.Connect();
		int index = 0;
		byte[] buffer = new byte[BufferSize];
		int offset = 0;

		using (var file = File.OpenRead(localFile))
		{
			using (SftpFileStream s = sftpClient.OpenWrite("SendFileTest.zzz"))
			{
				long length = file.Length;
				int bytesRead = 1;
				while (bytesRead > 0)
				{
					bytesRead = file.Read(buffer, 0, BufferSize );
					if (bytesRead > 0)
					{
						s.Write(buffer, 0, bytesRead);
						offset += bytesRead;
						index++;
					}
				}
			}
		}			
	}

	var crc1 = EncryptionUtils.GetChecksumFromFile(localFile, "SHA256");
	var crc2 = EncryptionUtils.GetChecksumFromFile(@"c:\utl\data\SendFileTest.zzz", "SHA256");
	Console.WriteLine(crc1);
	Console.WriteLine(crc2);
	Assert.AreEqual(crc1, crc2);
}

Changing the buffersize to its default size of 50,000 makes this work. Playing around with various sizes it seems to work between 40-60000 - anything smaller and larger seems to fail, but not sure if that's coincidental.

Not really a big issue since I can certainly ignore the buffersize (we're using SftpClient in an FTP abstraction wrapper), but something seems very wonky about this. Small files under 10 megs seem to work without issue - it's only with larger files that this shows up. Depending on the buffersize settings (our default was around 24k) the file differences start showing around the 11 meg mark with transposed byte sequences for a small block.

As an aside: Using the higher level DownloadFile() function seems to work fine.

SENDFILETEST.zip

[arqbackup]

I see this when writing a file via SFTP. A buffer size of 32768 works; a buffer size of 65536 causes transposed byte sequences. Related to SFTP packet size?

[drieseng]

This is a duplicate of issue #70.
Please try using version 2016.1.0-beta1 or the upcoming 2016.1.0-beta2 release.

[RickStrahl]

Works with the 2016.1.0-beta1 package. Tried a host of different combination that were previously failing and it looks like it all works now.

Thank you