RC2 Key Incompatibility between NT/2000 and XP/2003

Problem Description

Data encrypted with a 128-bit RC2 key on Windows 2000 or Windows NT refuses to be decrypted on a Windows 2003 or XP machine. The symptoms are as though the encryption and decryption keys do not match (see Article PS020711101), although in fact the keys are identical. The error returned is:

Persits.CryptoManager.1 error '800a0010'
Bad Data.

or

Persits.CryptoManager.1 (0x800A0010)
Bad Data

Solution

Microsoft Strong and Enhanced cryptographic service providers (CSP) on Windows NT and Windows 2000 generate RC2 keys that have the effective key length of 40 bits by default regardless of the requested key length. Thus, the AspEncrypt methods GenerateKey and GenerateKeyFromPassword with the last argument (Key Size) set to 128 still generate 40-bit RC2 keys on Windows NT and 2000, although the property Key.Length returns 128.

NOTE: The above statement is only true for RC2 keys. With the algorithms RC4, DES, 3DES, and 3DES2, the effective key length matches the requested key length on all platforms.

The cryptographic providers on Windows XP and 2003, on the other hand, generate RC2 keys with the effective key length matching that of Key.Length. As a result, 128-bit keys generated on Windows NT/2000 and those generated on Windows 2003/XP do not match (as their effective lengths are 40 and 128, respectively).

AspEncrypt 2.2 (released on September 16, 2003) provides a workaround for this problem by offering a new property, Key.EffectiveLength which can be used to retrieve and specify the effective key length for RC2 keys.

On Windows XP/2003, to generate a key that matches a 128-bit RC2 created on a Windows NT/2000 box, use the following code:

' Use SHA and RC2 by default
Set Key = Context.GenerateKeyFromPassword("pwd", , , 128)
Key.EffectiveLength = 40
Text = Key.DecryptText(Blob)

It is recommended that all your RC2-encrypted data be decrypted and then re-encrypted using a key with the effective key length of 128, as follows:

Set Key = Context.GenerateKeyFromPassword("pwd", , , 128)
Key.EffectiveLength = 128
Set Blob = Key.EncryptText(...)

Upgrades to AspEncrypt 2.2 are absolutely free for registered users. The new version can be downloaded at www.aspencrypt.com/download.html.