[BACKPORT pg15-cherrypicks] all: Bulk port from master - 98

Summary:
 f5ad1fb [#23855] Fixing the calculation of automatic refactoring count.
 eed826d [#22519] YSQL: Move ExplicitRowLockBuffer class into separate file
 99a27e6 [PLAT-14522] Taking yba-ctl backups with prometheus HTTPS
 e2a84b0 [PLAT-15044] Add preflight check for node addition in provider
 7e40d89 [#20769] XCluster: Dynamically apply cdc_wal_retention_time_secs for XCluster
 aa41478 [#23858] build: fix ./yb_build.sh release --gcc11
 bcf7f47 [PLAT-13910] Improve IAM credentials fetch logging and add retries
 303a202 [#23778] xCluster: Remove the capability to rename xCluster replication groups
 Excluded: 58fd26e [#23652] YSQL: Fix TestPgRegressAnalyze.java timeout / database drop failure on TSAN build
 31da65b [doc] yb_enable_bitmapscan flag (#23854)
 798db14 [#20335] DocDB: Use MonoClock for write query metric
 80779d8 [#23860] xCluster: Add automatic ddl mode proto fields
 afd763d Revert "Revert "[PLAT-14786] Add support to node_agent install to use bind ip and node_external_fqdn""
 e86951a [#23841] docdb: Disable stack trace tracking in TSAN builds
 d600608 [PLAT-15244] Fix schedule not getting updated on edit schedule API call
 3c0df09 [PLAT-15214][PLAT-15232]YBC version upgrade to 2.2.0.0-b6 and enable YBC verbose by default
 aa7372e [#23478] YSQL: fix connection manager session variable case sensitivity issue
 0d53558 [PLAT-14810][PLAT-14811][YBA CLI] Support adding and editing EIT configurations
 ffa537e [PLAT-10706][dr] Support retry-ability of failover and switchover
 Excluded: 5ae4558 [#23578] YSQL: Add HELP and TYPE to :13000/prometheus-metrics
 3aa7459 [PLAT-15016] Handle gflag_group changes for ENHANCED_PG_COMPATIBILITY group in 2024.1.3
 c89356c [PLAT-10592][YBA] Changes to support global tserver/master service in K8s
 7fc3b76 [#3893] YCQL: Align 2 system_schema.* tables with Cassandra
 da6274e [23646] Test Stability: Fix PgMiniTest.FollowerReads
 5fa6dc9 [PLAT-15180][Platform][UI][PITR]Create Restore Backup modal
 be0d1d1 [PLAT-15247][Platform][Backup]Create Backup scheduled policy List

Test Plan: Jenkins: rebase: pg15-cherrypicks

Reviewers: jason, tfoucher

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D37981

docs/content/preview/explore/ysql-language-features/postgresql-compatibility.md

@@ -52,7 +52,7 @@ Conversely, if you are using EPCM on a universe, you cannot set any of the featu
 | Batch nested loop join | [yb_enable_batchednl](../../../reference/configuration/yb-tserver/#yb-enable-batchednl) | 2.20 | 2024.1 |
 | Efficient communication<br>between PostgreSQL and DocDB | [pg_client_use_shared_memory](../../../reference/configuration/yb-tserver/#pg-client-use-shared-memory) | 2024.1 | | Yes |
 | Ascending indexing by default | [yb_use_hash_splitting_by_default](../../../reference/configuration/yb-tserver/#yb-use-hash-splitting-by-default) | 2024.1 | |
-| Bitmap scan | [enable_bitmapscan](../../../reference/configuration/yb-tserver/#enable-bitmapscan) | 2024.1.3<br>(Planned) | |
+| YugabyteDB bitmap scan | [yb_enable_bitmapscan](../../../reference/configuration/yb-tserver/#yb-enable-bitmapscan) | 2024.1.3<br>(Planned) | |
 | Parallel query | | Planned | |
 
 ### Released
@@ -119,11 +119,11 @@ Default ascending indexing provides feature compatibility and is the default in
 
 The following features are planned for EPCM in future releases.
 
-#### Bitmap scan
+#### YugabyteDB bitmap scan
 
-Configuration parameter: `enable_bitmapscan=true`
+Configuration parameter: `yb_enable_bitmapscan=true`
 
-Bitmap scans use multiple indexes to answer a query, with only one scan of the main table. Each index produces a "bitmap" indicating which rows of the main table are interesting. Bitmap scans can improve the performance of queries containing AND and OR conditions across several index scans. Bitmap scan provides improved performance parity.
+Bitmap scans use multiple indexes to answer a query, with only one scan of the main table. Each index produces a "bitmap" indicating which rows of the main table are interesting. Bitmap scans can improve the performance of queries containing AND and OR conditions across several index scans. YugabyteDB bitmap scan provides feature compatibility and improved performance parity. For YugabyteDB relations to use a bitmap scan, the PostgreSQL parameter `enable_bitmapscan` must also be true (the default).
 
 #### Parallel query
 

docs/content/preview/reference/configuration/yb-tserver.md

@@ -1733,14 +1733,27 @@ Default: `1GB`
 
 ##### enable_bitmapscan
 
-{{<badge/tp>}} Enables or disables the query planner's use of bitmap-scan plan types.
+PostgreSQL parameter to enable or disable the query planner's use of bitmap-scan plan types.
 
 Bitmap Scans use multiple indexes to answer a query, with only one scan of the main table. Each index produces a "bitmap" indicating which rows of the main table are interesting. Multiple bitmaps can be combined with AND or OR operators to create a final bitmap that is used to collect rows from the main table.
 
-Bitmap scans follow the same work_mem behavior as PostgreSQL: each individual bitmap is bounded by work_mem. If there are n bitmaps, it means we may use n * work_mem memory.
+Bitmap scans follow the same `work_mem` behavior as PostgreSQL: each individual bitmap is bounded by `work_mem`. If there are n bitmaps, it means we may use `n * work_mem` memory.
 
 Bitmap scans are only supported for LSM indexes.
 
+Default: true
+
+##### yb_enable_bitmapscan
+
+{{<badge/tp>}} Enables or disables the query planner's use of bitmap scans for YugabyteDB relations. Both [enable_bitmapscan](#enable-bitmapscan) and `yb_enable_bitmapscan` must be set to true for a YugabyteDB relation to use a bitmap scan. If `yb_enable_bitmapscan` is false, the planner never uses a YugabyteDB bitmap scan.
+
+| enable_bitmapscan | yb_enable_bitmapscan | Result |
+| :--- | :---  | :--- |
+| true | false | Default. Bitmap scans allowed only on temporary tables, if the planner believes the bitmap scan is most optimal. |
+| true | true  | Default for [Enhanced PostgreSQL Compatibility](../../../explore/ysql-language-features/postgresql-compatibility/#enhanced-postgresql-compatibility-mode). Bitmap scans are allowed on temporary tables and YugabyteDB relations, if the planner believes the bitmap scan is most optimal. |
+| false | false | Bitmap scans allowed only on temporary tables, but only if every other scan type is also disabled / not possible. |
+| false | true  | Bitmap scans allowed on temporary tables and YugabyteDB relations, but only if every other scan type is also disabled / not possible. |
+
 Default: false
 
 ##### yb_bnl_batch_size

docs/content/stable/explore/ysql-language-features/postgresql-compatibility.md

@@ -50,7 +50,7 @@ Conversely, if you are using EPCM on a universe, you cannot set any of the featu
 | Batch nested loop join | [yb_enable_batchednl](../../../reference/configuration/yb-tserver/#yb-enable-batchednl) | 2.20 | 2024.1 |
 | Efficient communication<br>between PostgreSQL and DocDB | [pg_client_use_shared_memory](../../../reference/configuration/yb-tserver/#pg-client-use-shared-memory) | 2024.1 | | Yes |
 | Ascending indexing by default | [yb_use_hash_splitting_by_default](../../../reference/configuration/yb-tserver/#yb-use-hash-splitting-by-default) | 2024.1 | |
-| Bitmap scan | [enable_bitmapscan](../../../reference/configuration/yb-tserver/#enable-bitmapscan) | 2024.1.3<br>(Planned) | |
+| YugabyteDB bitmap scan | [yb_enable_bitmapscan](../../../reference/configuration/yb-tserver/#yb-enable-bitmapscan) | 2024.1.3<br>(Planned) | |
 | Parallel query | | Planned | |
 
 ### Released
@@ -117,11 +117,11 @@ Default ascending indexing provides feature compatibility and is the default in
 
 The following features are planned for EPCM in future releases.
 
-#### Bitmap scan
+#### YugabyteDB bitmap scan
 
-Configuration parameter: `enable_bitmapscan=true`
+Configuration parameter: `yb_enable_bitmapscan=true`
 
-Bitmap scans use multiple indexes to answer a query, with only one scan of the main table. Each index produces a "bitmap" indicating which rows of the main table are interesting. Bitmap scans can improve the performance of queries containing AND and OR conditions across several index scans. Bitmap scan provides improved performance parity.
+Bitmap scans use multiple indexes to answer a query, with only one scan of the main table. Each index produces a "bitmap" indicating which rows of the main table are interesting. Bitmap scans can improve the performance of queries containing AND and OR conditions across several index scans. YugabyteDB bitmap scan provides feature compatibility and improved performance parity. For YugabyteDB relations to use a bitmap scan, the PostgreSQL parameter `enable_bitmapscan` must also be true (the default).
 
 #### Parallel query
 

docs/content/stable/reference/configuration/yb-tserver.md

@@ -1715,14 +1715,27 @@ Default: `1GB`
 
 ##### enable_bitmapscan
 
-{{<badge/tp>}} Enables or disables the query planner's use of bitmap-scan plan types.
+PostgreSQL parameter to enable or disable the query planner's use of bitmap-scan plan types.
 
 Bitmap Scans use multiple indexes to answer a query, with only one scan of the main table. Each index produces a "bitmap" indicating which rows of the main table are interesting. Multiple bitmaps can be combined with AND or OR operators to create a final bitmap that is used to collect rows from the main table.
 
-Bitmap scans follow the same work_mem behavior as PostgreSQL: each individual bitmap is bounded by work_mem. If there are n bitmaps, it means we may use n * work_mem memory.
+Bitmap scans follow the same `work_mem` behavior as PostgreSQL: each individual bitmap is bounded by `work_mem`. If there are n bitmaps, it means we may use `n * work_mem` memory.
 
 Bitmap scans are only supported for LSM indexes.
 
+Default: true
+
+##### yb_enable_bitmapscan
+
+{{<badge/tp>}} Enables or disables the query planner's use of bitmap scans for YugabyteDB relations. Both [enable_bitmapscan](#enable-bitmapscan) and `yb_enable_bitmapscan` must be set to true for a YugabyteDB relation to use a bitmap scan. If `yb_enable_bitmapscan` is false, the planner never uses a YugabyteDB bitmap scan.
+
+| enable_bitmapscan | yb_enable_bitmapscan | Result |
+| :--- | :---  | :--- |
+| true | false | Default. Bitmap scans allowed only on temporary tables, if the planner believes the bitmap scan is most optimal. |
+| true | true  | Default for [Enhanced PostgreSQL Compatibility](../../../explore/ysql-language-features/postgresql-compatibility/#enhanced-postgresql-compatibility-mode). Bitmap scans are allowed on temporary tables and YugabyteDB relations, if the planner believes the bitmap scan is most optimal. |
+| false | false | Bitmap scans allowed only on temporary tables, but only if every other scan type is also disabled / not possible. |
+| false | true  | Bitmap scans allowed on temporary tables and YugabyteDB relations, but only if every other scan type is also disabled / not possible. |
+
 Default: false
 
 ##### yb_bnl_batch_size

java/yb-client/src/main/java/org/yb/client/AlterUniverseReplicationRequest.java

@@ -87,7 +87,7 @@ ByteBuf serialize(Message header) {
             .setRemoveTableIgnoreErrors(removeTableIgnoreErrors);
 
     if (newReplicationGroupName != null) {
-      builder.setNewReplicationGroupId(newReplicationGroupName);
+      builder.setDEPRECATEDNewReplicationGroupId(newReplicationGroupName);
     }
 
     // If all bootstrap IDs are null, it is not required.

java/yb-cql/src/test/java/org/yb/cql/TestSystemTables.java

@@ -225,6 +225,53 @@ public void testEmptySystemTables() throws Exception {
     assertEquals(0, session.execute("SELECT * FROM system_schema.views;").all().size());
   }
 
+  @Test
+  public void testSystemTableColumns() throws Exception {
+    // Table system_schema.aggregates
+    List<Row> results = session.execute("SELECT * FROM system_schema.columns WHERE " +
+        "keyspace_name = 'system_schema' AND table_name = 'aggregates'").all();
+    assertEquals(8, results.size());
+    verifyColumnSchema(results.get(0), "system_schema", "aggregates",
+        "keyspace_name", "partition_key", 0, "text", "none");
+    verifyColumnSchema(results.get(1), "system_schema", "aggregates",
+        "aggregate_name", "clustering", 0, "text", "asc");
+    verifyColumnSchema(results.get(2), "system_schema", "aggregates",
+        "argument_types", "clustering", 1, "frozen<list<text>>", "asc");
+    verifyColumnSchema(results.get(3), "system_schema", "aggregates",
+        "final_func", "regular", -1, "text", "none");
+    verifyColumnSchema(results.get(4), "system_schema", "aggregates",
+        "initcond", "regular", -1, "text", "none");
+    verifyColumnSchema(results.get(5), "system_schema", "aggregates",
+        "return_type", "regular", -1, "text", "none");
+    verifyColumnSchema(results.get(6), "system_schema", "aggregates",
+        "state_func", "regular", -1, "text", "none");
+    verifyColumnSchema(results.get(7), "system_schema", "aggregates",
+        "state_type", "regular", -1, "text", "none");
+
+    // Table system_schema.functions
+    results = session.execute("SELECT * FROM system_schema.columns WHERE " +
+        "keyspace_name = 'system_schema' AND table_name = 'functions'").all();
+    assertEquals(8, results.size());
+    verifyColumnSchema(results.get(0), "system_schema", "functions",
+        "keyspace_name", "partition_key", 0, "text", "none");
+    verifyColumnSchema(results.get(1), "system_schema", "functions",
+        "function_name", "clustering", 0, "text", "asc");
+    verifyColumnSchema(results.get(2), "system_schema", "functions",
+        "argument_types", "clustering", 1, "frozen<list<text>>", "asc");
+    verifyColumnSchema(results.get(3), "system_schema", "functions",
+        "argument_names", "regular", -1, "frozen<list<text>>", "none");
+    verifyColumnSchema(results.get(4), "system_schema", "functions",
+        "body", "regular", -1, "text", "none");
+    verifyColumnSchema(results.get(5), "system_schema", "functions",
+        "called_on_null_input", "regular", -1, "boolean", "none");
+    verifyColumnSchema(results.get(6), "system_schema", "functions",
+        "language", "regular", -1, "text", "none");
+    verifyColumnSchema(results.get(7), "system_schema", "functions",
+        "return_type", "regular", -1, "text", "none");
+
+    // TODO: Implement for 'system_schema' tables: indexes, triggers, types, views, etc.
+  }
+
   private void checkContactPoints(String column, Row row) {
     List<InetSocketAddress> contactPoints = miniCluster.getCQLContactPoints();
     boolean found = false;
@@ -405,17 +452,28 @@ public void testSystemKeyspacesAndTables() throws Exception {
     assertEquals(UUID.fromString(uuid), results.get(0).getUUID("id"));
   }
 
-  private void verifyColumnSchema(Row row, String table_name, String column_name, String kind,
-                                  int position, String type, String clustering_order) {
-    assertEquals(DEFAULT_TEST_KEYSPACE, row.getString("keyspace_name"));
+  private void verifyColumnSchema(Row row, String keyspace_name, String table_name,
+                                  String column_name, String kind, int position,
+                                  String type, String clustering_order) {
+    assertEquals(keyspace_name, row.getString("keyspace_name"));
     assertEquals(table_name, row.getString("table_name"));
     assertEquals(column_name, row.getString("column_name"));
     assertEquals(clustering_order, row.getString("clustering_order"));
     assertEquals(kind, row.getString("kind"));
+    // Note: the "position" =
+    //   0,1,2,3... for a hash-key column: index of the column among the "partition_key" columns
+    //   0,1,2,3... for a range-key column: index of the column among the "clustering" columns
+    //   -1         for any non-key column
     assertEquals(position, row.getInt("position"));
     assertEquals(type, row.getString("type"));
   }
 
+  private void verifyColumnSchema(Row row, String table_name, String column_name, String kind,
+                                  int position, String type, String clustering_order) {
+    verifyColumnSchema(row,
+        DEFAULT_TEST_KEYSPACE, table_name, column_name, kind, position, type, clustering_order);
+  }
+
   private void verifyTypeSchema(Row row, String type_name,
                                 List<String> field_names, List<String> field_types) {
     assertEquals(DEFAULT_TEST_KEYSPACE, row.getString("keyspace_name"));

managed/RUNTIME-FLAGS.md

@@ -161,7 +161,6 @@
 | "Enable SSE" | "yb.backup.enable_sse" | "UNIVERSE" | "Enable SSE during backup/restore" | "Boolean" |
 | "Allow Table by Table backups for YCQL" | "yb.backup.allow_table_by_table_backup_ycql" | "UNIVERSE" | "Backup tables individually during YCQL backup" | "Boolean" |
 | "NFS Directry Path" | "yb.ybc_flags.nfs_dirs" | "UNIVERSE" | "Authorised NFS directories for backups" | "String" |
-| "Enable Verbose Logging" | "yb.ybc_flags.enable_verbose" | "UNIVERSE" | "Enable verbose ybc logging" | "Boolean" |
 | "Max Thread Count" | "yb.perf_advisor.max_threads" | "UNIVERSE" | "Max number of threads to support parallel querying of nodes" | "Integer" |
 | "Allow Scheduled YBC Upgrades" | "ybc.upgrade.allow_scheduled_upgrade" | "UNIVERSE" | "Enable Scheduled upgrade of ybc on the universe" | "Boolean" |
 | "Allow User Gflags Override" | "yb.gflags.allow_user_override" | "UNIVERSE" | "Allow users to override default Gflags values" | "Boolean" |
@@ -233,6 +232,8 @@
 | "Network Load balancer health check paths" | "yb.universe.network_load_balancer.custom_health_check_paths" | "UNIVERSE" | "Paths probed by HTTP/HTTPS health checks performed by the network load balancer. Paths are mapped one-to-one with the custom health check ports runtime configuration." | "String List" |
 | "Validate filepath for local release" | "yb.universe.validate_local_release" | "UNIVERSE" | "For certain tasks validates the existence of local filepath for the universe software version." | "Boolean" |
 | "The delay before the next poll of the PITR config creation status" | "yb.pitr.create_poll_delay" | "UNIVERSE" | "It is the delay after which the create PITR config subtask rechecks the status of the PITR config creation in each iteration" | "Duration" |
+| "The delay before the next poll of the PITR config restore status" | "yb.pitr.restore_poll_delay" | "UNIVERSE" | "It is the delay after which the restore PITR config subtask rechecks the status of the restore operation" | "Duration" |
+| "The timeout for restoring a universe using a PITR config" | "yb.pitr.restore_timeout" | "UNIVERSE" | "It is the maximum time that the restore PITR config subtask waits for the restore operation using PITR to be completed; otherwise, it will fail the operation" | "Duration" |
 | "The timeout for creating a PITR config" | "yb.pitr.create_timeout" | "UNIVERSE" | "It is the maximum time that the create PITR config subtask waits for the PITR config to be created; otherwise, it will fail the operation" | "Duration" |
 | "Default PITR retention period for txn xCluster" | "yb.xcluster.transactional.pitr.default_retention_period" | "UNIVERSE" | "The default retention period used to create PITR configs for transactional xCluster replication; it will be used when there is no existing PITR configs and it is not specified in the task parameters" | "Duration" |
 | "Default PITR snapshot interval for txn xCluster" | "yb.xcluster.transactional.pitr.default_snapshot_interval" | "UNIVERSE" | "The default snapshot interval used to create PITR configs for transactional xCluster replication; it will be used when there is no existing PITR configs and it is not specified in the task parameters" | "Duration" |
@@ -242,7 +243,8 @@
 | "Sync user-groups between the Universe DB nodes and LDAP Server" | "yb.security.ldap.ldap_universe_sync" | "UNIVERSE" | "If configured, this feature allows users to synchronise user groups configured on the upstream LDAP Server with user roles in YBDB nodes associated with the universe." | "Boolean" |
 | "Cluster membership check timeout" | "yb.checks.cluster_membership.timeout" | "UNIVERSE" | "Controls the max time to check that there are no tablets assigned to the node" | "Duration" |
 | "Verify current cluster state (from db perspective) before running task" | "yb.task.verify_cluster_state" | "UNIVERSE" | "Verify current cluster state (from db perspective) before running task" | "Boolean" |
-| "Wait time for xcluster/DR replication setup and edit RPCs." | "yb.xcluster.operation_timeout" | "UNIVERSE" | "Wait time for xcluster/DR replication setup and edit RPCs." | "Duration" |
+| "Wait time for xcluster/DR replication setup and edit RPCs" | "yb.xcluster.operation_timeout" | "UNIVERSE" | "Wait time for xcluster/DR replication setup and edit RPCs." | "Duration" |
+| "Maximum timeout for xCluster bootstrap producer RPC call" | "yb.xcluster.bootstrap_producer_timeout" | "UNIVERSE" | "If the RPC call to create the bootstrap streams on the source universe does not return before this timeout, the task will retry with exponential backoff until it fails." | "Duration" |
 | "Leaderless tablets check enabled" | "yb.checks.leaderless_tablets.enabled" | "UNIVERSE" | " Whether to run CheckLeaderlessTablets subtask before running universe tasks" | "Boolean" |
 | "Leaderless tablets check timeout" | "yb.checks.leaderless_tablets.timeout" | "UNIVERSE" | "Controls the max time out when performing the CheckLeaderlessTablets subtask" | "Duration" |
 | "Enable Clock Sync check" | "yb.wait_for_clock_sync.enabled" | "UNIVERSE" | "Enable Clock Sync check" | "Boolean" |

managed/build.sbt

@@ -927,7 +927,7 @@ runPlatform := {
 }
 
 libraryDependencies += "org.yb" % "yb-client" % "0.8.94-SNAPSHOT"
-libraryDependencies += "org.yb" % "ybc-client" % "2.2.0.0-b5"
+libraryDependencies += "org.yb" % "ybc-client" % "2.2.0.0-b6"
 libraryDependencies += "org.yb" % "yb-perf-advisor" % "1.0.0-b33"
 
 libraryDependencies ++= Seq(