ETCD with TLS showing error "transport: remote error: tls: bad certificate" for certificates generated using OpenSSL #8603
Comments
|
What is the output of |
|
And do not advertise default route address |
|
Thank you for the link Please find the output of server.pem |
|
@keyankay SSL config looks good, but you are advertising the default route Try fixing: To ? |
|
I still get the same error (used 2 options with and without 0.0.0.0). Same issue both the times 2017-09-26 09:29:51.305220 I | etcdserver: published {Name:default ClientURLs:[https://10.53.70.188:2379]} to cluster cdf818194e3a8c32 ./etcd --client-cert-auth --trusted-ca-file=/root/cfssl/ca.pem --cert-file=/root/cfssl/server.pem --key-file=/root/cfssl/server-key.pem --advertise-client-urls https://10.53.70.188:2379 --listen-client-urls https://10.53.70.188:2379 |
|
I also checked in 3.2.7, the problem exists. [root@vm-188 etcd-v3.1.10-linux-amd64]# ./etcd --client-cert-auth --trusted-ca-file=/root/cfssl/ca.pem --cert-file=/root/cfssl/server.pem --key-file=/root/cfssl/server-key.pem --advertise-client-urls https://10.53.70.188:2379 --listen-client-urls https://10.53.70.188:2379 2017-09-26 10:07:16.421663 I | embed: ClientTLS: cert = /root/cfssl/server.pem, key = /root/cfssl/server-key.pem, ca = , trusted-ca = /root/cfssl/ca.pem, client-cert-auth = true However, when i try to connect from client, i get an error sudo curl -v --noproxy '*' --cacert /root/cfssl/ca.pem --cert /root/cfssl/client.pem --key /root/cfssl/client-key.pem -L https://10.53.70.188:2379/v2/keys/foo -XPUT -d value=bar -v
ca.pemserver.pemCertificate: client.pemcertificate: |
|
Can you regenerate certs with |
|
I did with another set with the extended key usage, but not self signed (With a intermediate CA). I get a different error when i do with intermediate and CA Chain Here i am posting with all the relevant logs and information ./etcd --client-cert-auth --trusted-ca-file=/root/etcd/server_certs/ca-chain.cert.pem --cert-file=/root/etcd/server_certs/server.cert.pem --key-file=/root/etcd/server_certs/server.key.pem --advertise-client-urls https://10.53.70.188:2379 --listen-client-urls https://0.0.0.0:2379 2017-09-26 12:14:29.212009 I | embed: ClientTLS: cert = /root/etcd/server_certs/server.cert.pem, key = /root/etcd/server_certs/server.key.pem, ca = , trusted-ca = /root/etcd/server_certs/ca-chain.cert.pem, client-cert-auth = true ETCD client log: [root@vm-188 cfssl]# curl -v --noproxy '*' --cacert /root/etcd/server_certs/ca-chain.cert.pem --cert /root/etcd/client_certs/client.cert.pem --key /root/etcd/client_certs/client.key.pem -L https://localhost:2379/v2/keys/foo -XPUT -d value=bar -v
Please find the certificate details : Server Certificate: ClientCertificate: |
How did you generate certs? Please provide us with concise reproducible steps. |
|
I generated the certificates (in two methods). Second method is at the end of this comment First Method: https://jamielinux.com/docs/openssl-certificate-authority/appendix/root-configuration-file.html However please find the concise steps here: First you need to download these two files and follow these steps
Eventhough, i did not see any startup issues in 3.1.10, my request still did not go through in this version also. With 3.1.10 I get this error from etcd
Second Method:
If you provide me an email id, i can mail the certificates |
On your second logs, I don't see any |
|
I tried again with CA certificate with X509V3 Subject Alternative Name using version etcd-v3.1.10 (where there is no error observed in etcd side). Certificates generated as mentioned in in etcdv-3.2.7 i see the following error"etcdserver/api/v3rpc: Failed to dial 10.53.70.188:2379: connection error: desc = "transport: remote error: tls: bad certificate"; please retry." Execution command `Certificate: Server.pem Client certificate I get the error
|
|
Sorry, couldn't have time to reproduce. In the meantime, could you also try http://play.etcd.io/install with latest etcd release? It explains |
|
I've been able to reproduce this on 3.2.5 and 3.2.7. Doesn't seem to be occurring on a 3.1.9 cluster. The "bad certificate" error only comes up when etcd starts up, but the cluster is reporting itself as healthy (and works without any other noticeable issues) otherwise. In our case, we're using etcd for Openshift Origin. We used the steps to generate the certs directly from their documentation (see step 7): https://docs.openshift.org/3.6/admin_guide/backup_restore.html#backup-restore-adding-etcd-hosts |
|
@rezie |
|
@keyankay Do you mean whether I generated both the client and server sets? If so, then yes; the steps I followed are all in the documentation that I linked. |
|
In case somebody else comes across this issue, I was able to reproduce in 3.2 (3.2.11) build by using client certs with an empty SAN section. By adding the server IP address into the SAN, I was able to get connections to re-establish. |
|
I'm experiencing the same issue with official docker image 3.2.16 despite the cerificates are completely fine (used terraform tls provider to generate the complete chain). Here is the output of docker logs: $ docker logs etcd
2018-02-22 21:33:55.922579 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=https://etcd-1.internal:2379
2018-02-22 21:33:55.922665 I | pkg/flags: recognized and used environment variable ETCD_CA_FILE=/etc/etcd/certs/etcd_ca.pem
2018-02-22 21:33:55.922673 I | pkg/flags: recognized and used environment variable ETCD_CERT_FILE=/etc/etcd/certs/etcd_server.pem
2018-02-22 21:33:55.922683 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=true
2018-02-22 21:33:55.922700 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/etcd-data
2018-02-22 21:33:55.922726 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=https://etcd-1.internal:2380
2018-02-22 21:33:55.922733 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd-1.internal=https://etcd-1.internal:2380
2018-02-22 21:33:55.922789 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
2018-02-22 21:33:55.922802 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-k8-cluster
2018-02-22 21:33:55.922809 I | pkg/flags: recognized and used environment variable ETCD_KEY_FILE=/etc/etcd/certs/etcd_server_key.pem
2018-02-22 21:33:55.922818 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
2018-02-22 21:33:55.922829 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
2018-02-22 21:33:55.922848 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd-1.internal
2018-02-22 21:33:55.922859 I | pkg/flags: recognized and used environment variable ETCD_PEER_CA_FILE=/etc/etcd/certs/etcd_ca.pem
2018-02-22 21:33:55.922866 I | pkg/flags: recognized and used environment variable ETCD_PEER_CERT_FILE=/etc/etcd/certs/etcd_peer.pem
2018-02-22 21:33:55.922873 I | pkg/flags: recognized and used environment variable ETCD_PEER_CLIENT_CERT_AUTH=true
2018-02-22 21:33:55.922882 I | pkg/flags: recognized and used environment variable ETCD_PEER_KEY_FILE=/etc/etcd/certs/etcd_peer_key.pem
2018-02-22 21:33:55.922891 I | pkg/flags: recognized and used environment variable ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/certs/etcd_ca.pem
2018-02-22 21:33:55.922898 I | pkg/flags: recognized and used environment variable ETCD_PROXY=off
2018-02-22 21:33:55.922935 I | pkg/flags: recognized and used environment variable ETCD_TRUSTED_CA_FILE=/etc/etcd/certs/etcd_ca.pem
2018-02-22 21:33:55.923002 I | etcdmain: etcd Version: 3.2.16
2018-02-22 21:33:55.923023 I | etcdmain: Git SHA: 121edf046
2018-02-22 21:33:55.923041 I | etcdmain: Go Version: go1.8.5
2018-02-22 21:33:55.923046 I | etcdmain: Go OS/Arch: linux/amd64
2018-02-22 21:33:55.923051 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
2018-02-22 21:33:55.923130 I | embed: peerTLS: cert = /etc/etcd/certs/etcd_peer.pem, key = /etc/etcd/certs/etcd_peer_key.pem, ca = /etc/etcd/certs/etcd_ca.pem, trusted-ca = /etc/etcd/certs/etcd_ca.pem, client-cert-auth = true
2018-02-22 21:33:55.924260 I | embed: listening for peers on https://0.0.0.0:2380
2018-02-22 21:33:55.924316 I | embed: listening for client requests on 0.0.0.0:2379
2018-02-22 21:33:55.935526 I | pkg/netutil: resolving etcd-1.internal:2380 to 10.0.2.24:2380
2018-02-22 21:33:55.936757 I | pkg/netutil: resolving etcd-1.internal:2380 to 10.0.2.24:2380
2018-02-22 21:33:55.936811 I | etcdserver: name = etcd-1.internal
2018-02-22 21:33:55.936818 I | etcdserver: data dir = /etcd-data
2018-02-22 21:33:55.936825 I | etcdserver: member dir = /etcd-data/member
2018-02-22 21:33:55.936832 I | etcdserver: heartbeat = 100ms
2018-02-22 21:33:55.936838 I | etcdserver: election = 1000ms
2018-02-22 21:33:55.936843 I | etcdserver: snapshot count = 100000
2018-02-22 21:33:55.936877 I | etcdserver: advertise client URLs = https://etcd-1.internal:2379
2018-02-22 21:33:55.936886 I | etcdserver: initial advertise peer URLs = https://etcd-1.internal:2380
2018-02-22 21:33:55.936901 I | etcdserver: initial cluster = etcd-1.internal=https://etcd-1.internal:2380
2018-02-22 21:33:55.943476 I | etcdserver: starting member dc051765c3f0f8ee in cluster fdb09331e905ba38
2018-02-22 21:33:55.943506 I | raft: dc051765c3f0f8ee became follower at term 0
2018-02-22 21:33:55.943516 I | raft: newRaft dc051765c3f0f8ee [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2018-02-22 21:33:55.943522 I | raft: dc051765c3f0f8ee became follower at term 1
2018-02-22 21:33:55.956019 W | auth: simple token is not cryptographically signed
2018-02-22 21:33:55.965842 I | etcdserver: starting server... [version: 3.2.16, cluster version: to_be_decided]
2018-02-22 21:33:55.966966 I | embed: ClientTLS: cert = /etc/etcd/certs/etcd_server.pem, key = /etc/etcd/certs/etcd_server_key.pem, ca = /etc/etcd/certs/etcd_ca.pem, trusted-ca = /etc/etcd/certs/etcd_ca.pem, client-cert-auth = true
2018-02-22 21:33:55.967382 I | etcdserver/membership: added member dc051765c3f0f8ee [https://etcd-1.internal:2380] to cluster fdb09331e905ba38
2018-02-22 21:33:56.451204 I | raft: dc051765c3f0f8ee is starting a new election at term 1
2018-02-22 21:33:56.451538 I | raft: dc051765c3f0f8ee became candidate at term 2
2018-02-22 21:33:56.451894 I | raft: dc051765c3f0f8ee received MsgVoteResp from dc051765c3f0f8ee at term 2
2018-02-22 21:33:56.452056 I | raft: dc051765c3f0f8ee became leader at term 2
2018-02-22 21:33:56.452173 I | raft: raft.node: dc051765c3f0f8ee elected leader dc051765c3f0f8ee at term 2
2018-02-22 21:33:56.453644 I | etcdserver: setting up the initial cluster version to 3.2
2018-02-22 21:33:56.453803 I | etcdserver: published {Name:etcd-1.internal ClientURLs:[https://etcd-1.internal:2379]} to cluster fdb09331e905ba38
2018-02-22 21:33:56.454507 I | embed: ready to serve client requests
2018-02-22 21:33:56.455495 I | embed: serving client requests on [::]:2379
2018-02-22 21:33:56.469146 N | etcdserver/membership: set the initial cluster version to 3.2
2018-02-22 21:33:56.477496 I | etcdserver/api: enabled capabilities for version 3.2
WARNING: 2018/02/22 21:33:56 Failed to dial 0.0.0.0:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.The single node cluster appears to be healthy though: I'm sure that there is nothing wrong with certificates as I use similar chains in other applications and they work just fine. |
|
Seems like you don't specify SAN field in your certs? |
|
@gyuho in fact I do and specify both DNS names and IP (for simplicity I'm testing now one node): |
|
If I switch to the latest version 3.3.1 I get this: My peer certificate has no any |
|
I am hitting the same issue with etcd 3.2.14 |
|
I think the cause is missing or unclear documentation on what the server/peer & client certificates need to have in CN & SAN in order for it to work. |
|
Met the same issue on 3.3.3 cluster, etcdctl works fine, but not curl. I could do health check through curl, but put key request will return error "transport: authentication handshake failed: remote error: tls: bad certificate" |
Etcd requires clientAuth and serverAuth usage in the server certificate due to the embedded grpc server. [1] [1](etcd-io/etcd#8603) Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1593635
Etcd requires clientAuth and serverAuth usage in the server certificate due to the embedded grpc server. [1] [1](etcd-io/etcd#8603) Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1593635
Etcd requires clientAuth and serverAuth usage in the server certificate due to the embedded grpc server. [1] [1](etcd-io/etcd#8603) Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1593635
|
My solution was to hit https://:2379 instead of https://127.0.0.1:2379 as I had previously. This made me think that the problem was the server certificate not including 12.0.0.1 in the SAN, but a repeat test proved 127.0.0.1 in the SAN did not solve the problem |
A cert with this X509v3 usage can not be used for client auth is must also have include |
|
I think I am also running into this on the current version of etcd, with some changes. CA pemserver pemOf interest a curl connection times out with no server hello received the etcd journalctl log is saying that the CA is unknown even when the ca.pem is passed through via system unit. |
|
had the same problem when was trying to use a shared certificate with a CN (common name) something like |
|
Suppose you are using systemd to manage your ETCD cluster. Make sure your servers have correct NTP configuration and correct time (no matter what timezone you choose). sudo timedatectl set-ntp true
sudo timedatectl status
sudo systemctl restart etcdETCD uses cert to encrypt and authenticate traffic, as you see: If your server NTP and time isn't correct, it will not work correctly, for example, ETCD will complain
|
|
Im experiencing similar issue and have tried regenerating and using client and server pems from cfssl Testing to each node via curl seems to connect no issue. `sudo curl -v --noproxy '*' --cacert /etc/etcd/certs/ca.pem --cert /etc/etcd/certs/client.pem --key /etc/etcd/certs/client-key.pem -L https://10.86.170.54:2379/v2/keys/foo -XPUT -d value=bar -v
When I check systemd status I see continuous logs for remote error "tls: bad certificate" , verified adding name aswell as IP when generating certs and also tried adding the hostname parameter to no avail
|
@sfuerte could you elaborate a little more on the above, I have trouble understanding the SAN piece and where can verify my settings |
|
https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=1950471 This appears to be waits happening for me |
|
I read in the docs of version 3.5: so I put this in SAN, and that worked for me: SAN: "IP: 127.0.0.1, IP: (eth1 local address ), DNS: localhost, DNS: (root domain) " localhost is required because etcd internally works with this. |
|
Closing as part of migration of questions to GitHub discussions. |
ETCD 3.2.5 started with openssl certificates as follows
etcdserver/api/v3rpc: Failed to dial 0.0.0.0:2379: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
The ca-chain and server certificates were generated as mentioned here
https://jamielinux.com/docs/openssl-certificate-authority/appendix/root-configuration-file.html
as well as here
https://coreos.com/os/docs/latest/generate-self-signed-certificates.html
Please find the complete log here
[root@vm-188 etcd-v3.2.5-linux-amd64]# ./etcd --client-cert-auth --trusted-ca-file=/root/cfssl/ca.pem --cert-file=/root/cfssl/server.pem --key-file=/root/cfssl/server-key.pem --advertise-client-urls https://0.0.0.0:2379 --listen-client-urls https://10.53.70.188:2379 2017-09-25 19:31:46.145497 I | etcdmain: etcd Version: 3.2.5
2017-09-25 19:31:46.145663 I | etcdmain: Git SHA: d0d1a87
2017-09-25 19:31:46.145678 I | etcdmain: Go Version: go1.8.3
2017-09-25 19:31:46.145690 I | etcdmain: Go OS/Arch: linux/amd64
2017-09-25 19:31:46.145707 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
2017-09-25 19:31:46.145734 W | etcdmain: no data-dir provided, using default data-dir ./default.etcd
2017-09-25 19:31:46.145827 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2017-09-25 19:31:46.146912 I | embed: listening for peers on http://localhost:2380
2017-09-25 19:31:46.147068 I | embed: listening for client requests on 10.53.70.188:2379
2017-09-25 19:31:46.151360 I | etcdserver: name = default
2017-09-25 19:31:46.151384 I | etcdserver: data dir = default.etcd
2017-09-25 19:31:46.151397 I | etcdserver: member dir = default.etcd/member
2017-09-25 19:31:46.151418 I | etcdserver: heartbeat = 100ms
2017-09-25 19:31:46.151430 I | etcdserver: election = 1000ms
2017-09-25 19:31:46.151441 I | etcdserver: snapshot count = 100000
2017-09-25 19:31:46.151464 I | etcdserver: advertise client URLs = https://0.0.0.0:2379
2017-09-25 19:31:46.197402 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 387
2017-09-25 19:31:46.197544 I | raft: 8e9e05c52164694d became follower at term 43
2017-09-25 19:31:46.197646 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 43, commit: 387, applied: 0, lastindex: 387, lastterm: 43]
2017-09-25 19:31:46.244364 W | auth: simple token is not cryptographically signed
2017-09-25 19:31:46.269470 I | etcdserver: starting server... [version: 3.2.5, cluster version: to_be_decided]
2017-09-25 19:31:46.269551 I | embed: ClientTLS: cert = /root/cfssl/server.pem, key = /root/cfssl/server-key.pem, ca = , trusted-ca = /root/cfssl/ca.pem, client-cert-auth = true
2017-09-25 19:31:46.270799 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2017-09-25 19:31:46.271154 N | etcdserver/membership: set the initial cluster version to 3.2
2017-09-25 19:31:46.271269 I | etcdserver/api: enabled capabilities for version 3.2
2017-09-25 19:31:46.298335 I | raft: 8e9e05c52164694d is starting a new election at term 43
2017-09-25 19:31:46.298399 I | raft: 8e9e05c52164694d became candidate at term 44
2017-09-25 19:31:46.298440 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 44
2017-09-25 19:31:46.298474 I | raft: 8e9e05c52164694d became leader at term 44
2017-09-25 19:31:46.298494 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 44
2017-09-25 19:31:46.299064 I | etcdserver: published {Name:default ClientURLs:[https://0.0.0.0:2379]} to cluster cdf818194e3a8c32
2017-09-25 19:31:46.321000 I | embed: ready to serve client requests
2017-09-25 19:31:46.321265 E | etcdmain: forgot to set Type=notify in systemd service file?
2017-09-25 19:31:46.322924 I | embed: serving client requests on 10.53.70.188:2379
2017-09-25 19:31:46.328396 I | etcdserver/api/v3rpc: Failed to dial 10.53.70.188:2379: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
The text was updated successfully, but these errors were encountered: