can not connect the cluster with the cert and muli-addr

[longquanwjx]

etcd server version 3.4.13,
i write a demo to connect the cluster with the cert, but it get can not cannot connect
i have three nodes

node1

./etcd \
  --data-dir=./data/  \
  --name=etcd0 \
  --cert-file=./etcd.pem \
  --key-file=./etcd-key.pem \
  --trusted-ca-file=./ca.pem \
  --peer-cert-file=./etcd.pem \
  --peer-key-file=./etcd-key.pem \
  --peer-trusted-ca-file=./ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --listen-peer-urls=https://192.168.8.133:2380 \
  --initial-advertise-peer-urls=https://192.168.8.133:2380 \
  --listen-client-urls=https://192.168.8.133:2379,http://127.0.0.1:2379 \
  --advertise-client-urls=https://192.168.8.133:2379 \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster="etcd0=https://192.168.8.133:2380,etcd1=https://192.168.8.134:2380,etcd2=https://192.168.8.135:2380" \
  --initial-cluster-state=new \
  --auto-compaction-mode=periodic \
  --auto-compaction-retention=1 \
  --max-request-bytes=33554432 \
  --quota-backend-bytes=6442450944 \
  --heartbeat-interval=250 \
  --election-timeout=2000

node2

./etcd \
  --data-dir=./data/  \
  --name=etcd1 \
  --cert-file=./etcd.pem \
  --key-file=./etcd-key.pem \
  --trusted-ca-file=./ca.pem \
  --peer-cert-file=./etcd.pem \
  --peer-key-file=./etcd-key.pem \
  --peer-trusted-ca-file=./ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --listen-peer-urls=https://192.168.8.134:2380 \
  --initial-advertise-peer-urls=https://192.168.8.134:2380 \
  --listen-client-urls=https://192.168.8.134:2379,http://127.0.0.1:2379 \
  --advertise-client-urls=https://192.168.8.134:2379 \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster="etcd0=https://192.168.8.133:2380,etcd1=https://192.168.8.134:2380,etcd2=https://192.168.8.135:2380" \
  --initial-cluster-state=new \
  --auto-compaction-mode=periodic \
  --auto-compaction-retention=1 \
  --max-request-bytes=33554432 \
  --quota-backend-bytes=6442450944 \
  --heartbeat-interval=250 \
  --election-timeout=2000  

node3

./etcd \
  --data-dir=./data/  \
  --name=etcd2 \
  --cert-file=./etcd.pem \
  --key-file=./etcd-key.pem \
  --trusted-ca-file=./ca.pem \
  --peer-cert-file=./etcd.pem \
  --peer-key-file=./etcd-key.pem \
  --peer-trusted-ca-file=./ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth \
  --listen-peer-urls=https://192.168.8.135:2380 \
  --initial-advertise-peer-urls=https://192.168.8.135:2380 \
  --listen-client-urls=https://192.168.8.135:2379,http://127.0.0.1:2379 \
  --advertise-client-urls=https://192.168.8.135:2379 \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster="etcd0=https://192.168.8.133:2380,etcd1=https://192.168.8.134:2380,etcd2=https://192.168.8.135:2380" \
  --initial-cluster-state=new \
  --auto-compaction-mode=periodic \
  --auto-compaction-retention=1 \
  --max-request-bytes=33554432 \
  --quota-backend-bytes=6442450944 \
  --heartbeat-interval=250 \
  --election-timeout=2000    

i test with etcdctl, it test ok

knt@knt-20:~/work/3.4.13$ etcdctl --command-timeout=3s  --cacert=./ca.pem --cert=./etcd.pem --key=./etcd-key.pem --endpoints="https://192.168.8.135:2379,https://192.168.8.134:2379,https://192.168.8.133:2379" endpoint status -w=table
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.8.135:2379 |  781976c9acf7bd7 |  3.4.13 |   25 kB |      true |      false |         8 |          9 |                  9 |        |
| https://192.168.8.134:2379 | 82f03684ff969f47 |  3.4.13 |   25 kB |     false |      false |         8 |          9 |                  9 |        |
| https://192.168.8.133:2379 | c7a71eb7058e525c |  3.4.13 |   25 kB |     false |      false |         8 |          9 |                  9 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

i write a demo
but cannot connect with muli-addr

#include <etcd/Client.hpp>
#include <etcd/Response.hpp>
#include <string>
#include <iostream>

int main()
{

        etcd::Client* m = etcd::Client::WithSSL("https://192.168.8.133:2379,https://192.168.8.134:2379", "./ca.pem", "./etcd.pem", "./etcd-key.pem");
        etcd::Response resp = m->set("foo", "test", 15).get();
        printf("error_code %d, error_message %s, action %s\n", resp.error_code(),
           resp.error_message().c_str(), resp.action().c_str());
}

print

this->channel : 0x5629620d38a0error_code 14, error_message Connect Failed, action

when i change to single addr, or, remove cert in cluster ,it is ok

#include <etcd/Client.hpp>
#include <etcd/Response.hpp>
#include <string>
#include <iostream>

int main()
{

        etcd::Client* m = etcd::Client::WithSSL("https://192.168.8.133:2379", "./ca.pem", "./etcd.pem", "./etcd-key.pem");
        etcd::Response resp = m->set("foo", "test", 15).get();
        printf("error_code %d, error_message %s, action %s\n", resp.error_code(),
           resp.error_message().c_str(), resp.action().c_str());
}

print

this->channel : 0x558c44b878a0error_code 0, error_message , action set

[sighingnow]

(I have formatted the description a bit to make it more readable).

I cannot reproduce the error, I guess you are deploying etcd on different nodes (different ip and hostname) using the same SSL keys?

[sighingnow]

You could get a pesudo-distributed deployment via

1. setup the SSL keys

./security-config/setup-ca.sh

2. start the first etcd instance

etcd \
    --data-dir=./default.etcd0/ \
    --name=etcd0 \
    --cert-file security-config/certs/etcd0.example.com.crt \
    --key-file security-config/private/etcd0.example.com.key \
    --client-cert-auth \
    --trusted-ca-file security-config/certs/ca.crt \
    --peer-cert-file security-config/certs/etcd0.example.com.crt \
    --peer-key-file security-config/private/etcd0.example.com.key \
    --peer-client-cert-auth \
    --peer-trusted-ca-file security-config/certs/ca.crt \
    --listen-peer-urls=https://127.0.0.1:2380 \
    --initial-advertise-peer-urls=https://127.0.0.1:2380 \
    --listen-client-urls=https://127.0.0.1:2379 \
    --advertise-client-urls=https://127.0.0.1:2379 \
    --initial-cluster-token=etcd-cluster \
    --initial-cluster="etcd0=https://127.0.0.1:2380,etcd1=https://127.0.0.1:2480" \
    --initial-cluster-state=new \
    --debug

3. start the second etcd instance

etcd \
    --data-dir=./default.etcd1/ \
    --name=etcd1 \
    --cert-file security-config/certs/etcd0.example.com.crt \
    --key-file security-config/private/etcd0.example.com.key \
    --client-cert-auth \
    --trusted-ca-file security-config/certs/ca.crt \
    --peer-cert-file security-config/certs/etcd0.example.com.crt \
    --peer-key-file security-config/private/etcd0.example.com.key \
    --peer-client-cert-auth \
    --peer-trusted-ca-file security-config/certs/ca.crt \
    --listen-peer-urls=https://127.0.0.1:2480 \
    --initial-advertise-peer-urls=https://127.0.0.1:2480 \
    --listen-client-urls=https://127.0.0.1:2479 \
    --advertise-client-urls=https://127.0.0.1:2479 \
    --initial-cluster-token=etcd-cluster \
    --initial-cluster="etcd0=https://127.0.0.1:2380,etcd1=https://127.0.0.1:2480" \
    --initial-cluster-state=new \
    --debug

[longquanwjx]

yes, i deploy etcd on different nodes (different ip and host), because our production environment is different nodes
i generate the cert using this link
https://github.com/opsnull/follow-me-install-kubernetes-cluster
the ca-config.json file

cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF

the ca-csr.json file

cat > ca-csr.json <<EOF
{
  "CN": "kubernetes-ca",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "opsnull"
    }
  ],
  "ca": {
    "expiry": "876000h"
 }
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

the etcd-csr.json file

cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "172.27.138.251",
    "172.27.137.229",
    "172.27.138.239"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "opsnull"
    }
  ]
}
EOF

cfssl gencert -ca=/opt/k8s/work/ca.pem \
    -ca-key=/opt/k8s/work/ca-key.pem \
    -config=/opt/k8s/work/ca-config.json \
    -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

ls etcd*pem

[sighingnow]

When generating the keys have you replaces the "hosts" field in etcd-csr.json with your own hosts?

[longquanwjx]

yes, i replaced,
host in my case is
"hosts": [
"127.0.0.1",
"192.168.8.133",
"192.168.8.134",
"192.168.8.135"
]

i test cert with the etcdctl , it can get result success,
and it test cert with single addr ,it can get result success
i think the cert is correct

knt@knt-20:~/work/3.4.13$ etcdctl --command-timeout=3s  --cacert=./ca.pem --cert=./etcd.pem --key=./etcd-key.pem --endpoints="https://192.168.8.135:2379,https://192.168.8.134:2379,https://192.168.8.133:2379" endpoint status -w=table
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.8.135:2379 |  781976c9acf7bd7 |  3.4.13 |   25 kB |      true |      false |         8 |          9 |                  9 |        |
| https://192.168.8.134:2379 | 82f03684ff969f47 |  3.4.13 |   25 kB |     false |      false |         8 |          9 |                  9 |        |
| https://192.168.8.133:2379 | c7a71eb7058e525c |  3.4.13 |   25 kB |     false |      false |         8 |          9 |                  9 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

[sighingnow]

I will try the multi-host setting latter.

[longquanwjx]

ok, thank you

[sighingnow]

Will be fixed by #89, please add a common hostname, e.g., "etcd" here to the "hosts" when self-signing the certificate, and pass it to WithSSL:

etcd::Client* m = etcd::Client::WithSSL("https://192.168.8.133:2379",
                                        "./ca.pem", "./etcd.pem", "./etcd-key.pem", "etcd");