Support IPv6 addresses across all YugabyteDB servers & CLIs

[schoudhury]

Today only IPv4 addresses are supported for all the various YugabyteDB servers and CLIs. This should be enhanced to support IPv6 addresses as well.

Some context from https://en.wikipedia.org/wiki/IPv6_address

IPv6 is the successor to the first addressing infrastructure of the Internet, Internet Protocol version 4 (IPv4). In contrast to IPv4, which defined an IP address as a 32-bit value, IPv6 addresses have a size of 128 bits. Therefore, IPv6 has a vastly enlarged address space compared to IPv4.

An IPv6 address is represented as eight groups of four hexadecimal digits, each group representing 16 bits (two octets, a group sometimes also called a hextet[6][7]). The groups are separated by colons (:). An example of an IPv6 address is: 2001:0db8:85a3:0000:0000:8a2e:0370:7334

[synnack]

So I took a look at the source..

Looks like most of the problems can be fixed in yb/util/net/net_util.cc:

• the AF_INET hint before getaddrinfo should go from HostToInetAddrInfo
• HostPort::ParseString port splitting should be more intelligent for [::]:7100
• Bind for hostnames should probably bind more sockets than one, localhost may return both 127.0.0.1 and ::1, bind to both.

Wherever there is 0.0.0.0, it should be replaced by ::, IPv6 sockets also listen on IPv4 by default (except on OpenBSD or when this is explicitly disabled, in which case you'll want to bind to multiple sockets as well, one on 0.0.0.0 and one on ::).

[rkarthik007]

Hi @synnack,

Awesome thanks for going through the code and pointing out the fix, we will definitely take a look at this! Also, if you're interested in contributing a fix, we'd love to work with you on enabling you. cc @iSignal @bmatican

Do you also mind adding your self to the contributors list: https://github.com/yugabyte/yugabyte-db/blob/master/CONTRIBUTORS.md

[synnack]

I have something working:

diff --git a/src/yb/util/net/net_util.cc b/src/yb/util/net/net_util.cc
index 020497d6d..a48d07f68 100644
--- a/src/yb/util/net/net_util.cc
+++ b/src/yb/util/net/net_util.cc
@@ -140,24 +140,41 @@ Status HostPort::RemoveAndGetHostPortList(
   return Status::OK();
 }
 
+
+// Accepts entries like: ::1, 127.0.0.1, [::1]:7100, 0.0.0.0:7100, f.q.d.n:7100
 Status HostPort::ParseString(const string& str, uint16_t default_port) {
-  std::pair<string, string> p = strings::Split(str, strings::delimiter::Limit(":", 1));
+  uint32_t port;
 
-  // Strip any whitespace from the host.
-  StripWhiteSpace(&p.first);
+  // We look for the last colon and see if there is a valid port behind it
+  // If it looks like a valid port and there is only one colon, it's host:port
+  // If it looks like a valid port and the string starts with [ and there's a ]
+  // before the last colon, it's an IPv6 address [host]:port
+  // otherwise, just a host.
+  size_t pos = str.rfind(':');
+  if (pos != string::npos && pos > 1 && pos + 1 < str.length()
+      && SimpleAtoi(str.substr(pos + 1), &port)
+      && ((str[0] == '[' && str[pos - 1] == ']')
+          || count(str.begin(), str.end(), ':') == 1)) {
+
+    if (port == 0 || port > numeric_limits<uint16_t>::max()) {
+      return STATUS(InvalidArgument, "Invalid port", str);
+    }
 
-  // Parse the port.
-  uint32_t port;
-  if (p.second.empty() && strcount(str, ':') == 0) {
-    // No port specified.
-    port = default_port;
-  } else if (!SimpleAtoi(p.second, &port) ||
-             port > 65535) {
-    return STATUS(InvalidArgument, "Invalid port", str);
+    // Brackets indicate the address for IPv6 urls
+    if (str[0] == '[' && str[pos - 1] == ']') {
+      host_ = str.substr(1, pos - 2);
+      port_ = port;
+      return Status::OK();
+    } else {
+      host_ = str.substr(0, pos);
+      port_ = port;
+      return Status::OK();
+    }
   }
 
-  host_.swap(p.first);
-  port_ = port;
+  // Not in host:port or [host]:port format, assume it's a host
+  host_ = str;
+  port_ = default_port;
   return Status::OK();
 }
 
@@ -184,7 +201,6 @@ const string getaddrinfo_rc_to_string(int rc) {
 Result<std::unique_ptr<addrinfo, AddrinfoDeleter>> HostToInetAddrInfo(const std::string& host) {
   struct addrinfo hints;
   memset(&hints, 0, sizeof(hints));
-  hints.ai_family = AF_INET;
   hints.ai_socktype = SOCK_STREAM;
   struct addrinfo* res = nullptr;
   int rc = 0;
diff --git a/thirdparty/build_definitions/squeasel.py b/thirdparty/build_definitions/squeasel.py
index 150fdda59..ac305dd69 100644
--- a/thirdparty/build_definitions/squeasel.py
+++ b/thirdparty/build_definitions/squeasel.py
@@ -27,7 +27,7 @@ class SqueaselDependency(Dependency):
 
     def build(self, builder):
         log_prefix = builder.log_prefix(self)
-        compile_command = [builder.get_c_compiler(), '-std=c99', '-O3', '-DNDEBUG', '-fPIC', '-c',
+        compile_command = [builder.get_c_compiler(), '-std=c99', '-O3', '-DNDEBUG', '-DUSE_IPV6', '-fPIC', '-c',
                            'squeasel.c']
         compile_command += builder.compiler_flags + builder.c_flags
         log_output(log_prefix, compile_command)
diff --git a/src/yb/util/net/dns_resolver.cc b/src/yb/util/net/dns_resolver.cc
index 33f4c834e..d0b183387 100644
--- a/src/yb/util/net/dns_resolver.cc
+++ b/src/yb/util/net/dns_resolver.cc
@@ -136,12 +136,10 @@ Result<InetAddress> PickResolvedAddress(
   if (error) {
     return STATUS_FORMAT(NetworkError, "Resolve failed $0: $1", host, error.message());
   }
-  std::vector<InetAddress> addresses, addresses_v6;
+  std::vector<InetAddress> addresses;
   for (const auto& entry : entries) {
-    auto& dest = entry.endpoint().address().is_v4() ? addresses : addresses_v6;
-    dest.emplace_back(entry.endpoint().address());
+    addresses.emplace_back(entry.endpoint().address());
   }
-  addresses.insert(addresses.end(), addresses_v6.begin(), addresses_v6.end());
   if (addresses.empty()) {
     return STATUS_FORMAT(NetworkError, "No endpoints resolved for: $0", host);
   }
diff --git a/src/yb/master/master_main.cc b/src/yb/master/master_main.cc
index fa9539792..c842182c2 100644
--- a/src/yb/master/master_main.cc
+++ b/src/yb/master/master_main.cc
@@ -66,7 +66,7 @@ namespace master {
 
 static int MasterMain(int argc, char** argv) {
   // Reset some default values before parsing gflags.
-  FLAGS_rpc_bind_addresses = strings::Substitute("0.0.0.0:$0", kMasterDefaultPort);
+  FLAGS_rpc_bind_addresses = strings::Substitute("[::]:$0", kMasterDefaultPort);
   FLAGS_webserver_port = kMasterDefaultWebPort;
   FLAGS_default_memory_limit_to_ram_ratio = 0.10;
   // For masters we always want to fsync the WAL files.
diff --git a/src/yb/server/rpc_server.cc b/src/yb/server/rpc_server.cc
index be4e53d1d..5be8d9db8 100644
--- a/src/yb/server/rpc_server.cc
+++ b/src/yb/server/rpc_server.cc
@@ -57,7 +57,7 @@ using strings::Substitute;
 using std::unique_ptr;
 using std::make_unique;
 
-DEFINE_string(rpc_bind_addresses, "0.0.0.0",
+DEFINE_string(rpc_bind_addresses, "::",
               "Comma-separated list of addresses to bind to for RPC connections. "
               "Currently, ephemeral ports (i.e. port 0) are not allowed.");
 TAG_FLAG(rpc_bind_addresses, stable);
diff --git a/src/yb/server/server_base.cc b/src/yb/server/server_base.cc
index 410accf75..d58f7b3f5 100644
--- a/src/yb/server/server_base.cc
+++ b/src/yb/server/server_base.cc
@@ -107,7 +107,7 @@ using strings::Substitute;
 namespace yb {
 namespace server {
 
-static const string kWildCardHostAddress = "0.0.0.0";
+static const string kWildCardHostAddress = "::";
 
 namespace {
 
diff --git a/src/yb/server/webserver.cc b/src/yb/server/webserver.cc
index 0462637d8..ed8788ae4 100644
--- a/src/yb/server/webserver.cc
+++ b/src/yb/server/webserver.cc
@@ -98,7 +98,7 @@ Webserver::Webserver(const WebserverOptions& opts, const std::string& server_nam
   : opts_(opts),
     context_(nullptr),
     server_name_(server_name) {
-  string host = opts.bind_interface.empty() ? "0.0.0.0" : opts.bind_interface;
+  string host = opts.bind_interface.empty() ? "[::]" : opts.bind_interface;
   http_address_ = host + ":" + boost::lexical_cast<string>(opts.port);
 }
 

Parsing succeeds, listens and connects over IPv6 (this socket still also accepts IPv4 traffic):
yb-master 9266 root 32u IPv6 169841 0t0 TCP *:7100 (LISTEN)
yb-master 9266 root 36u IPv6 169843 0t0 TCP *:7000 (LISTEN)
yb-master 9266 root 40u IPv6 171198 0t0 TCP [2001:470:....]:41394->[2001:982:...]:7100 (SYN_SENT)
yb-master 9266 root 42u IPv6 171200 0t0 TCP [2001:470:...]:41398->[2001:982:...]:7100 (SYN_SENT)

Still requires a bit more testing/validation before I make a pull request, but all in all I think it is working.

[synnack]

Possibly concerned about the lack of StripWhitespace in ParseString now, doesn't really seem to be the right place for it though, and did not want to modify the string from the parent.

[iSignal]

The functionality is checked in. I will document the net_address_filter flag needed at #5813.