Specified key is not a valid size for this algorithm

SHA512 uses hexadecimal digits. Hexadecimal digits are 4 bits per. I think you need 32.

Does my key have to be in HEX?  I tried the following key and got the same error message:

tmpStr = Encrypt.Encrypt("12345", true, "abcdefghijklmnopqrstuvwxyz123456")

No, it should not need to be in hex.

I still get the error using this as the key: abcdefghijklmnopqrstuvwxyz123456

Any thoughts?

Each of those characters is a byte, not a bit.
I think you need to shorten it to match the keylength you're after.

Wy are you using UTF8 for encoding?can you as part of your test output what the key length is and the string that is being passed to the encrypt function.

I modified my the Encryption Program to return the KeyHashSize and I modified my calling program to tell me how many bytes are in my key.  Here are the results:

String to encrypt: 12345
Key: abcdefghijklmnopqrstuvwxyz123456
Results: Encryption Key: 256     Key in Bytes: 32

I'm still getting the invalid key size error.  I should say that I modified my original post to use SHA256CryptoServiceProvider hashmd5 = new SHA256CryptoServiceProvider();

As to why I'm using UTF8 for encoding:  I copied this algorithm from a post that I found on the web.

Any other thoughts?

Which .net framework are you working with?
http://msdn.microsoft.com/en-us/library/system.security.cryptography.sha512cryptoserviceprovider.aspx

You have several from triple des, des, and sha512 within the DLL.

.Net Framework 4

The computeHASH method requires three parameters.
Or it takes a stream of data

Your key is effectively an integer representing the number of bytes in the string you passed.
http://msdn.microsoft.com/en-us/library/xsy6z64h%28v=vs.100%29.aspx

http://msdn.microsoft.com/en-us/library/system.security.cryptography.sha512%28v=vs.100%29.aspx

I don't see where it requires 3 parameters:

HashAlgorithm.ComputeHash Method (Byte[])

I am passing it a byte array by calling UTF8Encoding.UTF8.GetBytes(key).

GetBytes reports the bytes as int.

The link http://msdn.microsoft.com/en-us/library/system.security.cryptography.sha512%28v=vs.100%29.aspx 
ref ComputeHash there are three options.

The link dealing with the GetBytes method of the UTF-8 returns an INT and not a string array.

I see your point about GetBytes but here is a test that I've done and the attached file shows the results.

namespace CVL
{
    public partial class test : System.Web.UI.Page
    {
        protected void Page_Load(object sender, EventArgs e)
        {

        }
        protected void EncryptBtn_Click(object sender, EventArgs e)
        {
            string key = KeyValue.Text;
            byte[] keyArray;
            keyArray = UTF8Encoding.UTF8.GetBytes(key);
            string tmpStr = keyArray.Count().ToString() + "<br />";
            
            foreach (byte k in keyArray)
            {
                tmpStr = tmpStr + k + "<br />";
            }
            StatusMsg.Text = tmpStr;
            //CryptorEngine encrypt = new CryptorEngine();
            //StatusMsg.Text = encrypt.Encrypt(EncryptValue.Text, true, key);
            //StatusMsg.Text = StatusMsg.Text + "   Key in Bytes: " + tmpStr;
        }
    }
}

Select allOpen in new window

So from what I can tell I am returning a byte array that has 32 elements which is the key size.
UTF8.GetBytes.bmp

I am not sure what the output is.

32 is the number of bytes in the array.  The values after represent the bytes of the key.  I'm showing that my key is 32 bytes and this line

keyArray = UTF8Encoding.UTF8.GetBytes(key);

is returning a byte array.

I have narrowed down the error to coming from the 3DES code but still don't know why.  This screen shot shows the key I'm using (16 characters), number of bytes in byte array and the values in the byte array.
3DESerror.bmp

Found the problem.  The ComputeHash is returning a 32 byte array regardless of how few characters are in the key.  The 3DES needs a key that is 16 or 24 bytes.

Since I can't use 3DES, do you have any suggestions for finishing the encryption?  I need it to be FIPs compliant.

I think 3des has to use a hex key 0-9 and a-f
http://www.cryptosys.net/3des.html

I installed the app on 3 other servers and don't have the problem on them so I've narrowed the problem down to something specific on the production server.  One of the things I discovered is the production server doesn't seem to be referencing the corrrect algorithm (see screen shot).  The error indicates that it is using the RijindaelManaged which is not FIPs compliant.  In my code I am using Rijindael as shown below.

        public string Encrypt(string PlainText, string key, string Salt="21stIn", 
            string HashAlgorithm = "SHA1", int PasswordIterations = 2,
            string InitialVector = "OFRna73m*aze01xY", int KeySize = 256)
        {
            // If string is null or whitespace, return toEncrypt
            if (String.IsNullOrEmpty(PlainText) || String.IsNullOrWhiteSpace(PlainText) || PlainText.Equals(DBNull.Value))
                return "";

            // Strip blanks from end of string
            PlainText = PlainText.Trim();

            System.Configuration.AppSettingsReader settingsReader = new AppSettingsReader();
            if (String.IsNullOrEmpty(key))
                // Get the key from config file
                key = (string)settingsReader.GetValue("SecurityKey", typeof(String));
            
            byte[] InitialVectorBytes = Encoding.ASCII.GetBytes(InitialVector);
            byte[] SaltValueBytes = Encoding.ASCII.GetBytes(Salt);
            byte[] PlainTextBytes = Encoding.UTF8.GetBytes(PlainText);
            PasswordDeriveBytes DerivedPassword = new PasswordDeriveBytes(key, SaltValueBytes, HashAlgorithm, PasswordIterations);  
            byte[] KeyBytes = DerivedPassword.GetBytes(KeySize / 8);
            [b]Rijndael SymmetricKey = Rijndael.Create();[/b]            SymmetricKey.Mode = CipherMode.CBC;
            byte[] CipherTextBytes = null;
            using (ICryptoTransform Encryptor = SymmetricKey.CreateEncryptor(KeyBytes, InitialVectorBytes))
            {
                using (MemoryStream MemStream = new MemoryStream())
                {
                    using (CryptoStream CryptoStream = new CryptoStream(MemStream, Encryptor, CryptoStreamMode.Write))
                    {
                        CryptoStream.Write(PlainTextBytes, 0, PlainTextBytes.Length);
                        CryptoStream.FlushFinalBlock();
                        CipherTextBytes = MemStream.ToArray();
                        MemStream.Close();
                        CryptoStream.Close();
                    }
                }
            }
            SymmetricKey.Clear();
            return Convert.ToBase64String(CipherTextBytes);
        }

Select allOpen in new window

So my question is, how do I find out why the production server is using the Managed version while the other servers are not?  I've compared the IIS settings between the servers and I don't see a difference.
FIPs-Error.bmp

.net version for the site? Double check the steps you follow when setting this up from scratch, versus the setup on the production server. Is the DLL have to be part f authorized components on the production system?

I haven't setup any of the servers.  I have access to the servers.  How do I check if DLL has to be part of authorized components?  How do I authorize it?  Production server is running Windows Server 2003.

In IIS 6 you need to check web service extensions to see if the one you need is added and approved.

I checked Web Service Extensions and both machines have ASP.NET v4.0.30319.  I checked the aspnet_isapii.dll on both machines and they are the same size.  The only difference I see is that the machine that the DLL works on has Windows SharePoint Services v3 allowed under Web Extensions where the machine where the DLL is not working does not.  I am not using SharePoint Services that I am aware of.

Is the DLL that includes these functions registered on both systems?
i.e. when you setting up your software on a new system, what process do you go through?
Was the same process followed for the production system?

That I can't answer because my group does not set up the systems.

You have a custom DLL that includes your encryption functions. Does that DLL get added to the production environment or is your encrypt.encrypt method in the code uses a system DLL function rather than your own.

I use the following to register my custom DLL:

%SystemRoot%\Microsoft.NET\Framework\v4.0.30319\regasm EncryptDecrypt.dll /codebase /tlb:EnCryptDecrypt.tlb

I run this from the folder where my DLL is.

I did as you suggested and still have the same problem.  At this point I think the problem has to be on the Production server and not with my code.  I am able to run on a different production server ok so at this point I am going to have to install my application on that server.  I appreciate all your help.

You mean you added the comment identifier and it reflects the correct response but still getting the same error? Double check the calling page to make sure it is not the one with the wrong crypto selector.

I would like to close this question without a result.  The problem still exists on the Production Server and we used a different server that didn't have the problem.

Did this exchange help you at all?  I.e. comments helped to lead you to move this application to a new server, pick and assign whatever point value you want and close the question, alternatively you can ask to delete this question.

My problem still exists but your help enabled me to narrow down the problem to a specific server.  Since we are putting a plan in place to migrate our applications from that server to another one where the DLL works I am not going to pursue this any longer.  Thank you for your time and help on this.