Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CloudWatchLogs request errors #238

Closed
jimmyt857 opened this issue Jul 15, 2016 · 12 comments
Closed

CloudWatchLogs request errors #238

jimmyt857 opened this issue Jul 15, 2016 · 12 comments
Labels
bug This issue is a bug.

Comments

@jimmyt857
Copy link

When calling CreateLogGroup, CreateLogStream, DeleteLogStream, or PutLogEvents, I am getting curl error 35 (ssl connect error). Any of the first 3 will just fail after ~10 attempts, but for some reason PutLogEvents always succeeds on the 7th attempt. Here is an example log trace:

[INFO] 2016-07-15 03:29:18 CurlHandleContainer [139736288929792] Initializing CurlHandleContainer with size 25
[TRACE] 2016-07-15 03:29:18 AWSClient [139736288929792] Found body, but content-length has not been set, attempting to compute content-length
[DEBUG] 2016-07-15 03:29:18 AWSAuthV4Signer [139736288929792] Calculated sha256 ed5df988d44e451e18fedf10d89327e33aa4ec818bf5eac665b2679d77c22dee for payload.
[DEBUG] 2016-07-15 03:29:18 AWSAuthV4Signer [139736288929792] Canonical Header String: content-length:31
content-type:application/x-amz-json-1.1
host:logs.us-west-2.amazonaws.com
user-agent:aws-sdk-cpp/0.13.4 Linux/3.13.0-74-generic x86_64
x-amz-content-sha256:ed5df988d44e451e18fedf10d89327e33aa4ec818bf5eac665b2679d77c22dee
x-amz-date:20160715T032918Z
x-amz-target:Logs_20140328.CreateLogGroup

[DEBUG] 2016-07-15 03:29:18 AWSAuthV4Signer [139736288929792] Signed Headers value:content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-target
[DEBUG] 2016-07-15 03:29:18 AWSAuthV4Signer [139736288929792] Canonical Request String: POST
/

content-length:31
content-type:application/x-amz-json-1.1
host:logs.us-west-2.amazonaws.com
user-agent:aws-sdk-cpp/0.13.4 Linux/3.13.0-74-generic x86_64
x-amz-content-sha256:ed5df988d44e451e18fedf10d89327e33aa4ec818bf5eac665b2679d77c22dee
x-amz-date:20160715T032918Z
x-amz-target:Logs_20140328.CreateLogGroup

content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-target
ed5df988d44e451e18fedf10d89327e33aa4ec818bf5eac665b2679d77c22dee
[DEBUG] 2016-07-15 03:29:18 AWSAuthV4Signer [139736288929792] Final String to sign: AWS4-HMAC-SHA256
20160715T032918Z
20160715/us-west-2/logs/aws4_request
6bde1138796478370ca249eb333f7dc6fb618b90cb07f5874081196b130dd249
[DEBUG] 2016-07-15 03:29:18 AWSAuthV4Signer [139736288929792] Final computed signing hash: ca9443049e51356c14eba7fb8025aa0ebc589d4c7b4324df833df4f8c4300651
[DEBUG] 2016-07-15 03:29:18 AWSAuthV4Signer [139736288929792] Signing request with: AWS4-HMAC-SHA256 Credential=AKIAIMTDWTMSLPNECNJA/20160715/us-west-2/logs/aws4_request, SignedHeaders=content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-target, Signature=ca9443049e51356c14eba7fb8025aa0ebc589d4c7b4324df833df4f8c4300651
[DEBUG] 2016-07-15 03:29:18 AWSClient [139736288929792] Request Successfully signed
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] Making request to https://logs.us-west-2.amazonaws.com
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] Including headers:
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] authorization: AWS4-HMAC-SHA256 Credential=AKIAIMTDWTMSLPNECNJA/20160715/us-west-2/logs/aws4_request, SignedHeaders=content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-target, Signature=ca9443049e51356c14eba7fb8025aa0ebc589d4c7b4324df833df4f8c4300651
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] content-length: 31
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] content-type: application/x-amz-json-1.1
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] host: logs.us-west-2.amazonaws.com
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] user-agent: aws-sdk-cpp/0.13.4 Linux/3.13.0-74-generic x86_64
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] x-amz-content-sha256: ed5df988d44e451e18fedf10d89327e33aa4ec818bf5eac665b2679d77c22dee
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] x-amz-date: 20160715T032918Z
[TRACE] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] x-amz-target: Logs_20140328.CreateLogGroup
[DEBUG] 2016-07-15 03:29:18 CurlHandleContainer [139736288929792] Attempting to acquire curl connection.
[DEBUG] 2016-07-15 03:29:18 CurlHandleContainer [139736288929792] No current connections available in pool. Attempting to create new connections.
[DEBUG] 2016-07-15 03:29:18 CurlHandleContainer [139736288929792] attempting to grow pool size by 2
[INFO] 2016-07-15 03:29:18 CurlHandleContainer [139736288929792] Pool successfully grown by 2
[DEBUG] 2016-07-15 03:29:18 CurlHandleContainer [139736288929792] Returning connection handle 0x42746a0
[DEBUG] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] Obtained connection handle 0x42746a0
[ERROR] 2016-07-15 03:29:18 CurlHttpClient [139736288929792] Curl returned error code 35
[DEBUG] 2016-07-15 03:29:18 CurlHandleContainer [139736288929792] Releasing curl handle 0x42746a0
[DEBUG] 2016-07-15 03:29:18 CurlHandleContainer [139736288929792] Notifying waiting threads.
[DEBUG] 2016-07-15 03:29:18 AWSClient [139736288929792] Request returned error. Attempting to generate appropriate error codes from response
[WARN] 2016-07-15 03:29:18 AWSClient [139736288929792] Request failed, now waiting 0 ms before attempting again.
@JonathanHenson
Copy link
Contributor

This actually looks like an TLS issue (error code 35). Can you turn on the verbose line in the curl client so I can see the TLS handshake output?

@jimmyt857
Copy link
Author

Sure; do you mean these two lines?

@JonathanHenson
Copy link
Contributor

Just the verbose line.

Sent from my iPhone

On Jul 17, 2016, at 1:48 PM, jimmyt857 notifications@github.com wrote:

Sure; do you mean these two lines?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

@jimmyt857
Copy link
Author

Here is the output from several attempts to connect. It's mostly this "Illegar parameter" error, but there is 1 Decrypt error instead.

@JonathanHenson
Copy link
Contributor

Ah... I think you are using the wrong libcurl. You need the one linked and compiled against OpenSSL. GnuTLS isn't compatible with our load balancers.

Sent from my iPhone

On Jul 18, 2016, at 11:22 AM, jimmyt857 notifications@github.com wrote:

Here is the output from several attempts to connect. It's mostly this "Illegar parameter" error, but there is 1 Decrypt error instead.

Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
Hostname was NOT found in DNS cache
Trying 54.240.253.17...
Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (#6)
found 173 certificates in /etc/ssl/certs/ca-certificates.crt
gnutls_handshake() failed: Illegal parameter
Closing connection 6
Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
Hostname was found in DNS cache
Trying 54.240.253.17...
Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (#7)
found 173 certificates in /etc/ssl/certs/ca-certificates.crt
gnutls_handshake() failed: Illegal parameter
Closing connection 7
Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
Hostname was found in DNS cache
Trying 54.240.253.17...
Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (#8)
found 173 certificates in /etc/ssl/certs/ca-certificates.crt
gnutls_handshake() failed: Illegal parameter
Closing connection 8
Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
Hostname was found in DNS cache
Trying 54.240.253.17...
Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (#9)
found 173 certificates in /etc/ssl/certs/ca-certificates.crt
gnutls_handshake() failed: Decrypt error
Closing connection 9
Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
Hostname was found in DNS cache
Trying 54.240.253.17...
Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (#10)
found 173 certificates in /etc/ssl/certs/ca-certificates.crt
gnutls_handshake() failed: Illegal parameter
Closing connection 10
Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
Hostname was found in DNS cache
Trying 54.240.253.17...
Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (#11)
found 173 certificates in /etc/ssl/certs/ca-certificates.crt
gnutls_handshake() failed: Decrypt error
Closing connection 11
Rebuilt URL to: https://logs.us-west-2.amazonaws.com/
Hostname was found in DNS cache
Trying 54.240.253.17...
Connected to logs.us-west-2.amazonaws.com (54.240.253.17) port 443 (#12)
found 173 certificates in /etc/ssl/certs/ca-certificates.crt
gnutls_handshake() failed: Illegal parameter
Closing connection 12
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

@jimmyt857
Copy link
Author

My curl --version is outputting
curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
so I think it is using openssl. Also, it is only these cloudwatch commands that have the problem.

Do you have a suggestion for how to install curl? I tried this repository https://github.com/curl/curl but I get compile issues when I try to build the aws library afterwards.

@jimmyt857
Copy link
Author

Ok, I was able to get this working. I was originally getting the following errors when building:

aws-sdk-cpp/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:123:5: error: invalid conversion from ‘void*’ to ‘CURL* {aka Curl_easy*}’ [-fpermissive]
curl_easy_setopt(handle, CURLOPT_NOSIGNAL, 1L);
^
aws-sdk-cpp/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:124:5: error: invalid conversion from ‘void*’ to ‘CURL* {aka Curl_easy*}’ [-fpermissive]
curl_easy_setopt(handle, CURLOPT_TIMEOUT_MS, m_requestTimeout);
^
aws-sdk-cpp/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:125:5: error: invalid conversion from ‘void*’ to ‘CURL* {aka Curl_easy*}’ [-fpermissive]
curl_easy_setopt(handle, CURLOPT_CONNECTTIMEOUT_MS, m_connectTimeout);

After adding the casts, I was able to build and it worked. Could we add these casts to the library?

@JonathanHenson
Copy link
Contributor

We don't get those errors when we build, so to evaluate this I need to know versions of the library and header files you are using.

curl --version is a fully built and linked binary. This is very much different from what you've actually built and linked against. There are multiple variants of the libcurl header files, some which use openssl, some gnutls, and some that use other dependencies.

I am certain, assuming the verbose output you showed is correct, that you were building against the gnutls variant. It may work against some of our load balancers, but it is also known widely to not work in most cases, so you shouldn't use it.

@jimmyt857
Copy link
Author

I am using the current version of https://github.com/curl/curl, building as follows:

autoreconf -i
./configure --with-ssl=/usr/lib/ssl --libdir=/usr/lib/x86_64-linux-gnu
make
sudo make install

@ambasta
Copy link

ambasta commented Jul 25, 2016

Hi,

Piggybacking on this issue, I too am getting:

In file included from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/include/aws/core/http/curl/CurlHandleContainer.h:23:0,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:16:
/home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp: In member function ‘void Aws::Http::CurlHandleContainer::SetDefaultOptionsOnHandle(void*)’:
/home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:123:5: error: invalid conversion from ‘void*’ to ‘CURL* {aka Curl_easy*}’ [-fpermissive]
     curl_easy_setopt(handle, CURLOPT_NOSIGNAL, 1L);
     ^
In file included from /usr/include/curl/curl.h:2420:0,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/include/aws/core/http/curl/CurlHandleContainer.h:23,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:16:
/usr/include/curl/easy.h:29:22: note:   initializing argument 1 of ‘CURLcode curl_easy_setopt(CURL*, CURLoption, ...)’
 CURL_EXTERN CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...);
                      ^
In file included from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/include/aws/core/http/curl/CurlHandleContainer.h:23:0,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:16:
/home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:124:5: error: invalid conversion from ‘void*’ to ‘CURL* {aka Curl_easy*}’ [-fpermissive]
     curl_easy_setopt(handle, CURLOPT_TIMEOUT_MS, m_requestTimeout);
     ^
In file included from /usr/include/curl/curl.h:2420:0,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/include/aws/core/http/curl/CurlHandleContainer.h:23,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:16:
/usr/include/curl/easy.h:29:22: note:   initializing argument 1 of ‘CURLcode curl_easy_setopt(CURL*, CURLoption, ...)’
 CURL_EXTERN CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...);
                      ^
In file included from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/include/aws/core/http/curl/CurlHandleContainer.h:23:0,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:16:
/home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:125:5: error: invalid conversion from ‘void*’ to ‘CURL* {aka Curl_easy*}’ [-fpermissive]
     curl_easy_setopt(handle, CURLOPT_CONNECTTIMEOUT_MS, m_connectTimeout);
     ^
In file included from /usr/include/curl/curl.h:2420:0,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/include/aws/core/http/curl/CurlHandleContainer.h:23,
                 from /home/amitprakash/aws-sdk/aws-sdk-cpp-0.13.8/aws-cpp-sdk-core/source/http/curl/CurlHandleContainer.cpp:16:
/usr/include/curl/easy.h:29:22: note:   initializing argument 1 of ‘CURLcode curl_easy_setopt(CURL*, CURLoption, ...)’
 CURL_EXTERN CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...);
                      ^
make[2]: *** [aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/build.make:1164: aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/source/http/curl/CurlHandleContainer.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:86: aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/all] Error 2
make: *** [Makefile:128: all] Error 2

I've curl compiled w/ openssl

net-misc/curl-7.50.0::gentoo USE="ldap ssl -adns -http2 -idn -ipv6 -kerberos -metalink -rtmp -samba -ssh -static-libs {-test} -threads" CURL_SSL="openssl -axtls -gnutls -libressl -mbedtls -nss -polarssl (-winssl)"

@jimmyt857
Copy link
Author

Since I have to make this edit every time I update my version of the library, I've been looking at this file some more; CURL* seems to be used already in several other places in the file. In fact, both instances of this function being called in this file pass a CURL*, and it isn't called from anywhere else.

@JonathanHenson JonathanHenson added the bug This issue is a bug. label Oct 5, 2016
@singku
Copy link
Contributor

singku commented May 26, 2017 •
edited
Loading

@jimmyt857 @ambasta Hello, it seems you have fixed the original problem after compiling curl with openssl. Is the casting problem still exists?

@singku singku closed this as completed Jun 26, 2017
cobookman pushed a commit to cobookman/aws-sdk-cpp that referenced this issue Jan 17, 2022
@bretambrose
Windows high performance counter precision fix; crt init/shutdown fix (…
8fcc21d 
…aws#238)

* Windows high performance counter precision fix; crt init/shutdown fix
* Submodule update
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug This issue is a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ambasta @jimmyt857 @singku @JonathanHenson