Hi,
I have gone through many suggestion most of them ask us to check the
locking mechanism. Following the suggestion, I was able to find the
issue.. Many people who have faced this issue, after resolving the issue
have not updated the reason. I am just explaining my scenario here, and how
it was resolved.
My application uses openssl with many other opensource third party
libraries, like a open source web server mongoose, libssh2, gsoap, libcurl
and also open pegasus. In all the mentioned libraries most of the
documentation recommends the application to maintain the openssl thread
callbacks, except for two libraries mongoose and Open pegasus. Both of
these libraries register and de-register openssl thread call backs when
needed. This used to cause thread locking set by the application to be
reset. This resulted in openssl data structure's to get corrupted for other
openssl calls from libssh2, gsoap and libcurl to get corrupted.
I have also reported the same as a bug on Open Pegasus project.
My suggestion when you face this issue, assuming everything is fine from
your application, Please check the third party library for the openssl
callback registration function CRYPTO_set_locking_callback()
Srikanth Bemineni
On Tue, Feb 17, 2015 at 12:21 PM, Srikanth Bemineni <
bemineni.srikanth_at_gmail.com> wrote:
> Hi,
>
> Some more details on the issue that I found out in the openssl bug list
> archive
>
> #1764: openssl-0.9.8i random generator - bug
> http://rt.openssl.org/Ticket/Display.html?id=1764#2630:
> Segmentation-fault in ssleay_rand_bytes() after generating a large
> (INT_MAX) random buffer
>
> http://rt.openssl.org/Ticket/Display.html?id=2630&user=guest&pass=guest
>
>
> Srikanth Bemineni
>
>
> On Tue, Feb 17, 2015 at 12:29 AM, Srikanth Bemineni <
> bemineni.srikanth_at_gmail.com> wrote:
>
>> Hi,
>>
>> We are seeing this random crash in libssh2 while copying files in a multi
>> threaded application. Our application is threaded application, and each
>> file transfer happens in its own thread. Each thread creates its own
>> libssh2 session and a channel to transfer the file. Occasionally we see
>> this random crash in openssl library. Below is the stack trace of the crash.
>>
>>
>>
>> some_application.exe!sha1_block_data_order() + 0x2b5c
>> bytes
>> some_application.exe!SHA1_Update(SHAstate_st * c, const
>> void * data_, unsigned __int64 len) Line 326 C
>> some_application.exe!update(env_md_ctx_st * ctx, const
>> void * data, unsigned __int64 count) Line 78 + 0x34 bytes C
>> some_application.exe!EVP_DigestUpdate(env_md_ctx_st * ctx,
>> const void * data, unsigned __int64 count) Line 252 C
>> some_application.exe!ssleay_rand_bytes(unsigned char *
>> buf, int num, int pseudo, int lock) Line 499 C
>> some_application.exe!ssleay_rand_nopseudo_bytes(unsigned
>> char * buf, int num) Line 542 C
>> some_application.exe!RAND_bytes(unsigned char * buf, int
>> num) Line 165 + 0x11 bytes C
>>
>> some_application.exe!_libssh2_transport_send(_LIBSSH2_SESSION * session,
>> const unsigned char * data, unsigned __int64 data_len, const unsigned char
>> * data2, unsigned __int64 data2_len) Line 820 C
>>
>> some_application.exe!_libssh2_channel_write(_LIBSSH2_CHANNEL * channel, int
>> stream_id, const unsigned char * buf, unsigned __int64 buflen) Line 2060 +
>> 0x46 bytes C
>>
>> some_application.exe!libssh2_channel_write_ex(_LIBSSH2_CHANNEL * channel,
>> int stream_id, const char * buf, unsigned __int64 buflen) Line 2109 + 0x24
>> bytes C
>> some_application.exe!Ssha::scp_write_file(QString *
>> local_path, _LIBSSH2_CHANNEL * channel_new) Line 761 + 0x1b bytes C++
>> > some_application.exe!Ssha::PutFile(QString * local_path,
>> QString * remote_path) Line 854 + 0x4a bytes C++
>>
>>
>> When I look at the session errmsg Its says "Unable to send channel data"
>> with errcode LIBSSH2_ERROR_EAGAIN
>>
>> The locking mechanism is also in place for open ssl. We did check that
>> openssl global data is locked and released by the mutex. Is there anything
>> that we are missing from the libssh2 perspective ?
>>
>> init()
>> {
>> libssh2_init(0);
>> CRYPTO_malloc_init();
>> CRYPTO_thread_setup()
>> }
>>
>> shutDownSequence()
>> {
>> libssh2_exit();
>> CRYPTO_thread_cleanup();
>> }
>>
>> void locking_function(int mode, int n, const char *file, int line)
>> {
>> if (mode & CRYPTO_LOCK)
>> {
>> if(mutex_buf[n] != NULL)
>> {
>> mutex_buf[n]->lock();
>> }
>> }
>> else
>> {
>> if(mutex_buf[n] != NULL)
>> {
>> mutex_buf[n]->unlock();
>> }
>> }
>> }
>>
>> unsigned long id_function()
>> {
>> return (unsigned long)QThread::currentThreadId();
>> }
>>
>> void Cb_function(CRYPTO_THREADID *id)
>> {
>> CRYPTO_THREADID_set_numeric(id, (unsigned
>> long)QThread::currentThreadId());
>> }
>>
>> bool CRYPTO_thread_setup()
>> {
>> int i;
>> int num = CRYPTO_num_locks();
>> for (i = 0; i < num; i++)
>> {
>> lock_count[i] = 0;
>> mutex_buf[i] = new QMutex(QMutex::NonRecursive);
>> }
>> CRYPTO_set_id_callback(id_function);
>> CRYPTO_THREADID_set_callback(Cb_function);
>> CRYPTO_set_locking_callback(locking_function);
>> CRYPTO_set_dynlock_create_callback(dyn_create_function);
>> CRYPTO_set_dynlock_lock_callback(dyn_lock_function);
>> CRYPTO_set_dynlock_destroy_callback(dyn_destroy_function);
>> return true;
>> }
>>
>> void CRYPTO_thread_cleanup()
>> {
>> int i;
>> int num = CRYPTO_num_locks();
>> if(crypto_initialized != 0)
>> {
>> CRYPTO_set_id_callback(NULL);
>> CRYPTO_THREADID_set_callback(NULL);
>> CRYPTO_set_locking_callback(NULL);
>> CRYPTO_set_dynlock_create_callback(NULL);
>> CRYPTO_set_dynlock_lock_callback(NULL);
>> CRYPTO_set_dynlock_destroy_callback(NULL);
>> for (i = 0; i < num; i++)
>> {
>> if(mutex_buf[i])
>> delete(mutex_buf[i]);
>> }
>> crypto_initialized = 0;
>> }
>> }
>>
>>
>> I see a similar issue reported as a bug in
>> http://trac.libssh2.org/ticket/212 . The resolution says adding
>> libssh2_init(0); fixed the issue. This has already been taken care in our
>> code, but we still see the crash.
>>
>> Srikanth Bemineni
>>
>
>
_______________________________________________
libssh2-devel http://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel
Received on 2015-02-20