Subject: Key exchange trashing memory

Key exchange trashing memory

From: James Sumners <>
Date: Sun, 21 Oct 2012 19:27:52 -0400

I'm writing an Objective-C interface to libssh2. During my initial
testing I was communicating with a server that reports the version
string "OpenSSH_5.5p1 Debian-6+squeeze2" and didn't have any problems.
Now, I'm testing against a server that reports simply "OpenSSH_6.0"
(it's a slightly outdated Arch Linux server; OpenSSH 6.1 is available
as an update) and the key exchange is failing. I'm using libssh2
version 1.4.3_DEV.

I recorded a video of what is happening -- . This video shows the
problem being run through LLDB.

In the video, you can see that in the `void libssh2_sha1(const unsigned
char *message, unsigned long len, unsigned char *out)` function in
openssl.c (line 512) the `len` and `out` variables are getting
overwritten after the call to `EVP_DigestInit`. Also shown in the video
is a patch that allows the function to finish, but causes a crash later
on in the process.

Can anyone shed some light on what is happening here?

The trace for the key exchange is:

[libssh2] 0.229073 Key Ex: Sent KEX:

[libssh2] 0.229620 Key Ex: Sent HOSTKEY: ssh-rsa,ssh-dss
[libssh2] 0.229920 Key Ex: Sent CRYPT_CS:

[libssh2] 0.230242 Key Ex: Sent CRYPT_SC:

[libssh2] 0.230549 Key Ex: Sent MAC_CS:

[libssh2] 0.230828 Key Ex: Sent MAC_SC:

[libssh2] 0.248835 Key Ex: Sent COMP_CS: none
[libssh2] 0.249782 Key Ex: Sent COMP_SC: none
[libssh2] 0.250296 Key Ex: Sent LANG_CS:
[libssh2] 0.250665 Key Ex: Sent LANG_SC:
[libssh2] 0.251181 Key Ex: Agreed on KEX method: diffie-hellman-group14-sha1
[libssh2] 0.251501 Key Ex: Agreed on HOSTKEY method: ssh-rsa
[libssh2] 0.251799 Key Ex: Agreed on CRYPT_CS method: aes128-ctr
[libssh2] 0.252076 Key Ex: Agreed on CRYPT_SC method: aes128-ctr
[libssh2] 0.252362 Key Ex: Agreed on MAC_CS method: hmac-sha1
[libssh2] 0.252644 Key Ex: Agreed on MAC_SC method: hmac-sha1
[libssh2] 0.252915 Key Ex: Agreed on COMP_CS method: none
[libssh2] 0.253183 Key Ex: Agreed on COMP_SC method: none
[libssh2] 0.262499 Key Ex: Initiating Diffie-Hellman Group14 Key Exchange
[libssh2] 0.273457 Key Ex: Sending KEX packet 30
[libssh2] 0.380532 Key Ex: Server's MD5 Fingerprint:
[libssh2] 0.381142 Key Ex: Server's SHA1 Fingerprint:

Received on 2012-10-22