[BACKPORT pg15-cherrypicks] all: Bulk port from master - 104

Summary: 6b34652 [doc] Update badge colors (#23985) 6dcfce6 [PLAT-15297][dr] GET /configs fails with java.lang.IllegalStateException: DrConfig iTest-dr(uuid) does not have any corresponding xCluster config 7995311 [#23695] docdb: Improve help strings for tablet replica limit flags. 7659151 [DOC-479] Voyager 1.8.1 updates (#23983) 56c9cc9 [PLAT-15282]: fix NPE on k8s operator backup/restore status update c8776b5 [docs] SSL/TLS configuration with hashicorp vault YSQL plugin (#23979) Excluded: d3dd4f7 Bumping version to 2.25.0.0 on branch master Excluded: 0dbe7d6 [#23312] YSQL: Fix backup/restore for colocated tables without tablespace information. 2beb872 [#23786] YSQL: add yb_make_next_ddl_statement_nonincrementing to YbDbAdminVariables 28d0d8e [#24011] build: fix ./yb_build.sh --gcc13 9feb6e8 [#24012] YSQL: Replace deprecated function shared_ptr::unique to fix macos compiler error 2cc1480 [PLAT-15349] Fix update_lb_config automated NLB updates af4c46d [#23741] docdb: Add parameter to run PITR tests with clone instead 0103dfc [PLAT-14459] Continuous backups for s3 e6bb890 [PLAT-15192] [PLAT-15191] Document node-agent-provision.yaml 1552321 [PLAT-15205] YBA callhome improvements ba6acbd [PLAT-15342] Install Node Agent as root level unit for ynp 2478a50 [PLAT-13058] Remove references to public key filepath, since we don't need ssh keys for node agent based provisioning aa14f76 [#23975] YSQL, ASH: Fix TestYbAsh#testPgAuxInfo 885b72b [#23960] DocDB: Fix WriteBuffer crash in move ctor Excluded: f4a002a [#23314] YSQL: Fix variable name to yb_ for ATPrepSetTableSpace 29c56ed [PLAT-15369][YBA CLI]Backup list-increments doesn't support pretty/json outputs 784160c [PLAT-15370][YBA CLI]backup list table output contains base_backup_uuid and backup_uuid b32222d [PLAT-15280][PLAT-14753] - Pg parity improvements and other ui fixes c6521cf [PLAT-15368][YBA CLI]Backup list print "No more backup present" after printing all the backups in -o pretty/json - pg_catalog_version-test.cc - NonIncrementingDDLMode: YB master commit 2beb872 sets the role to yb_db_admin, which due to new behavior in upstream PG commit b073c3ccd06e4cb845e121387a43faa8c68a7b62 (documented in https://www.postgresql.org/docs/15/ddl-schemas.html#DDL-SCHEMAS-PATTERNS) causes the following unintended test failure: ERROR: permission denied for schema public. Add a workaround to grant create permission to the role now set. Test Plan: Jenkins: urgent, rebase: pg15-cherrypicks Reviewers: jason, jenkins-bot Differential Revision: https://phorge.dev.yugabyte.com/D38220
yugabyte · Sep 20, 2024 · 24a9eac · 24a9eac
1 parent cf701bb
commit 24a9eac
Show file tree

Hide file tree

Showing 81 changed files with 1,359 additions and 546 deletions.
diff --git a/docs/assets/scss/_yb_badge.scss b/docs/assets/scss/_yb_badge.scss
@@ -8,9 +8,9 @@
   border-radius: 6px;
   text-decoration: none;
   margin: 0 10px;
-  background: #7879f1;
-  border: 1px solid #7879f1;
-  color: #fff;
+  background: #E8E9FE;
+  border: 1px solid #E8E9FE;
+  color: #4F4FA4;
   justify-content: center;
   align-items: center;
   position: relative;
@@ -19,17 +19,18 @@
   &:hover,
   &:focus,
   &:active {
-    background: rgba(120, 121, 241, 0.8);
+    background: rgba(232, 233, 254, 0.8);
   }
 
   &.tech-preview {
-    background: #ff603b;
-    border: 1px solid #ff603b;
+    background: #ffeec8;
+    border: 1px solid #ffeec8;
+    color: #9d6c00;
 
     &:hover,
     &:focus,
     &:active {
-      background: rgba(255, 96, 59, 0.8);
+      background: rgba(255, 238, 200, 0.8);
     }
   }
 

diff --git a/docs/content/preview/integrations/hashicorp-vault.md b/docs/content/preview/integrations/hashicorp-vault.md
@@ -160,6 +160,72 @@ vault lease renew <lease-ID>
 vault lease revoke <lease-ID>
 ```
 
+## Configure SSL/TLS
+
+To allow YSQL Hashicorp Vault plugin to communicate securely over SSL with YugabyteDB database, you need the root certificate (`ca.crt`) of the YugabyteDB cluster. To generate these certificates and install them while launching the cluster, follow the instructions in [Create server certificates](../../secure/tls-encryption/server-certificates/).
+
+Because a YugabyteDB Aeon cluster is always configured with SSL/TLS, you don't have to generate any certificate but only set the client-side SSL configuration. To fetch your root certificate, refer to [CA certificate](../../yugabyte-cloud/cloud-secure-clusters/cloud-authentication/#download-your-cluster-certificate).
+
+To start a secure local YugabyteDB cluster using `yugabyted`, refer to [Create a local multi-node cluster](../../reference/configuration/yugabyted/#create-a-local-multi-node-cluster).
+
+For a YugabyteDB Aeon cluster, or a local YugabyteDB cluster with SSL/TLS enabled, set the SSL-related connection parameters along with other connection information while adding the database by either of the following ways:
+
+- Provide the connection information in DSN format:
+
+    ```sh
+    vault write database/config/yugabytedb plugin_name=ysql-plugin  \
+    host="127.0.0.1" \
+    port=5433 \
+    username="yugabyte" \
+    password="yugabyte" \
+    db="yugabyte" \
+    load_balance=true \
+    yb_servers_refresh_interval=0 \
+    sslmode="verify-full" \
+    sslrootcert="path/to/.crt-file" \
+    allowed_roles="*"
+    ```
+
+- Provide the connection information as a connection string:
+
+    ```sh
+    vault write database/config/yugabytedb \
+    plugin_name=ysql-plugin \
+    connection_url="postgres://{{username}}:{{password}}@localhost:5433/yugabyte?sslmode=verify-full&load_balance=true&yb_servers_refresh_interval=0&sslrootcert=path/to/.crt-file" \
+    allowed_roles="*" \
+    username="yugabyte" \
+    password="yugabyte"
+    ```
+
+### SSL modes
+
+The following table summarizes the SSL modes:
+
+| SSL Mode | Client Driver Behavior | YugabyteDB Support |
+| :------- | :--------------------- | ------------------ |
+| disable  | SSL disabled | Supported
+| allow    | SSL enabled only if server requires SSL connection | Supported
+| prefer (default) | SSL enabled only if server requires SSL connection | Supported
+| require | SSL enabled for data encryption and Server identity is not verified | Supported
+| verify-ca | SSL enabled for data encryption and Server CA is verified | Supported
+| verify-full | SSL enabled for data encryption. Both CA and hostname of the certificate are verified | Supported
+
+YugabyteDB Aeon requires SSL/TLS, and connections using SSL mode `disable` will fail.
+
+## Known issues
+
+When executing vault operations, the internal query may fail with the following error:
+
+```output
+ERROR: The catalog snapshot used for this transaction has been invalidated: expected: 2, got: 1: MISMATCHED_SCHEMA (SQLSTATE 40001)
+```
+
+A DML query in YSQL may touch multiple servers, and each server has a Catalog Version which is used to track schema changes. When a DDL statement runs in the middle of the DML query, the Catalog Version is changed and the query has a mismatch, causing it to fail.
+
+For such cases, the database aborts the query and returns a 40001 error code. Operations failing with this code can be safely retried.
+
+For more information, refer to [How to troubleshoot Schema or Catalog version mismatch database errors](https://support.yugabyte.com/hc/en-us/articles/4406287763597-How-to-troubleshoot-Schema-or-Catalog-version-mismatch-database-errors).
+
 ## Learn more
 
 - [Database static roles and credential rotation](https://developer.hashicorp.com/vault/tutorials/db-credentials/database-creds-rotation)

diff --git a/docs/content/preview/yugabyte-voyager/airgapped.md b/docs/content/preview/yugabyte-voyager/airgapped.md
@@ -57,7 +57,7 @@ You can perform an airgapped installation on RHEL 8 and CentOS 8.
 1. Download the airgapped bundle:
 
     ```sh
-    wget https://s3.us-west-2.amazonaws.com/downloads.yugabyte.com/repos/airgapped/yb-voyager-1.8.0-0-rhel-8-x86_64.tar.gz
+    wget https://s3.us-west-2.amazonaws.com/downloads.yugabyte.com/repos/airgapped/yb-voyager-1.8.1-0-rhel-8-x86_64.tar.gz
     ```
 
 1. Extract the bundle.
@@ -171,7 +171,7 @@ You can perform an airgapped installation on Ubuntu 22 and later.
 1. Download the airgapped bundle:
 
     ```sh
-    wget https://s3.us-west-2.amazonaws.com/downloads.yugabyte.com/repos/airgapped/yb-voyager-1.8.0_0_debian.tar.gz
+    wget https://s3.us-west-2.amazonaws.com/downloads.yugabyte.com/repos/airgapped/yb-voyager-1.8.1_0_debian.tar.gz
     ```
 
 1. Extract the bundle.

diff --git a/docs/content/preview/yugabyte-voyager/install-yb-voyager.md b/docs/content/preview/yugabyte-voyager/install-yb-voyager.md
@@ -106,7 +106,7 @@ Install yb-voyager on a machine which satisfies the [Prerequisites](#prerequisit
   </div>
 </div>
 
-### Collect diagnostics
+## Collect diagnostics
 
 By default, yb-voyager captures a [diagnostics report](../diagnostics-report/) using the YugabyteDB diagnostics service that runs each time you use the yb-voyager command. If you don't want to send diagnostics when you run yb-voyager, set the [--send-diagnostics flag](../diagnostics-report/#configuration-flag) to false.
 

diff --git a/docs/content/preview/yugabyte-voyager/migrate/assess-migration.md b/docs/content/preview/yugabyte-voyager/migrate/assess-migration.md
@@ -190,18 +190,18 @@ After the bulk assessment is completed, the top-level directory specified using
 
 ```sh
 /bulk-assessment-dir/
-├── bulkAssessmentReport.html
-├── bulkAssessmentReport.json
+├── bulk_assessment_report.html
+├── bulk_assessment_report.json
 ├── DBNAME-SCHEMA1-export-dir/
 │    └── assessment/
 │          └── reports/
-│                 ├── assessmentReport.html
-│                 └── assessmentReport.json
+│                 ├── migration_assessment_report.html
+│                 └── migration_assessment_report.json
 ├── SID-SCHEMA2-export-dir/
 │    └── assessment/
 │          └── reports/
-│                 ├── assessmentReport.html
-│                 └── assessmentReport.json
+│                 ├── migration_assessment_report.html
+│                 └── migration_assessment_report.json
 └── logs/
      └── yb-voyager-assess-migration-bulk.log
 ```

diff --git a/docs/content/preview/yugabyte-voyager/release-notes.md b/docs/content/preview/yugabyte-voyager/release-notes.md
@@ -13,6 +13,25 @@ type: docs
 
 What follows are the release notes for the YugabyteDB Voyager v1 release series. Content will be added as new notable features and changes are available in the patch releases of the YugabyteDB v1 series.
 
+## v1.8.1 - September 17, 2024
+
+### Enhancements
+
+- The installation methods (including RHEL, Ubuntu, Brew, and the installer script) now install PostgreSQL 16 instead of PostgreSQL 14.
+- Added support in [analyze schema](../reference/schema-migration/analyze-schema/) and [assess-migration](../reference/assess-migration/) to report issues with indexes on columns with complex data types, such as INET, CITEXT, JSONB, TSVECTOR, TSQUERY, and ARRAYs. assess-migration categorizes these issues under the [Index on complex data types](../known-issues/postgresql/#indexes-on-some-complex-data-types-are-not-supported) PostgreSQL unsupported feature.
+- Both [analyze schema](../reference/schema-migration/analyze-schema/) and [assess-migration](../reference/assess-migration/) now detect and report UNLOGGED tables, which are unsupported in YugabyteDB.
+- Changed assessment report file names from "assessmentReport" to "migration_assessment_report", and "bulkAssessmentReport" to "bulk_assessment_report" .
+- Improved UI output of [assess-migration](../reference/assess-migration/) by removing the progress bar from ora2pg when fetching the assessment report.
+
+### Bug Fixes
+
+- Fixed an issue in streaming phase of [import data](../reference/data-migration/import-data/) where conflicts between events that could cause unique constraint errors were not being detected properly.
+- Fixed an issue in [analyze schema](../reference/schema-migration/analyze-schema/) where multi-column GIN indexes were incorrectly reported as an issue due to a regex misidentifying expression indexes that combine two columns.
+- Fixed an issue in [analyze schema](../reference/schema-migration/analyze-schema/) where table or column names containing the keyword "cluster" were incorrectly reported as issues of the type `ALTER TABLE CLUSTER ON`.
+- Fixed an issue in [export-data-from-target](../reference/data-migration/export-data/#export-data-from-target) involving the new YugabyteDB CDC connector from Oracle sources.
+- Fixed a schema registry issue in [import-data](../reference/data-migration/import-data/#import-data) where a "no such file or directory" error occurred.
+- Fixed a bug where Voyager was unable to send details to YugabyteDB UI when upgrading from version 1.7.2 to 1.8 using the same yugabyted cluster.
+
 ## v1.8 - September 3, 2024
 
 ### New Features

diff --git a/docs/content/preview/yugabyte-voyager/rhel.md b/docs/content/preview/yugabyte-voyager/rhel.md
@@ -77,3 +77,11 @@ Install a specific version of `yb-voyager` on your machine using the following c
     ```sh
     yb-voyager version
     ```
+
+### Upgrade yb-voyager
+
+Upgrade yb-voyager using the following command:
+
+```sh
+sudo yum update yb-voyager
+```
diff --git a/docs/content/preview/yugabyte-voyager/ubuntu.md b/docs/content/preview/yugabyte-voyager/ubuntu.md
@@ -19,7 +19,14 @@ Perform the following steps to install yb-voyager using apt for Ubuntu:
 
     This repository contains the `yb-voyager` Debian package and the dependencies required to run `yb-voyager`.
 
-1. Clean the `apt-get` cache and package lists using the following commands::
+1. Install the `postgresql-common` repository to fetch PostgreSQL 16 using the following commands:
+
+    ```sh
+    sudo apt install -y postgresql-common
+    sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
+    ```
+
+1. Clean the `apt-get` cache and package lists using the following commands:
 
     ```sh
     sudo apt-get clean
@@ -55,3 +62,21 @@ Perform the following steps to install yb-voyager using apt for Ubuntu:
     ```sh
     yb-voyager version
     ```
+
+### Upgrade yb-voyager
+
+{{< note title="Note" >}}
+If you are upgrading Voyager from version 1.8.0 or earlier, you need to install the `postgresql-common` repository before the upgrade as follows:
+
+```sh
+sudo apt install -y postgresql-common
+sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
+```
+
+{{< /note >}}
+
+Upgrade yb-voyager using the following command:
+
+```sh
+sudo apt-get upgrade yb-voyager
+```
diff --git a/docs/layouts/shortcodes/badge/ea.html b/docs/layouts/shortcodes/badge/ea.html
@@ -3,7 +3,7 @@
 {{/* version is the first path of the path. eg: preview/stable/v2.12 ...*/}}
 {{- $version := index $path 0 -}}
 <a class="yb-badge"
-  style="background: #7879F1; color:#FFF; border: 0px solid #7879F1; margin: 0px;"
+  style="background: #E8E9FE; color:#4F4FA4; border: 0px solid #E8E9FE; margin: 0px;"
   href="/preview/releases/versioning/#feature-maturity">
   EA
 </a>
diff --git a/docs/layouts/shortcodes/badge/ga.html b/docs/layouts/shortcodes/badge/ga.html
@@ -3,7 +3,7 @@
 {{/* version is the first path of the path. eg: preview/stable/v2.12 ...*/}}
 {{- $version := index $path 0 -}}
 <a class="yb-badge"
-  style="background: #CDEFE1; color:#4E5F6D; border: 0px solid #CDEFE1; margin: 0px;"
+  style="background: #CDEFE1; color:#097345; border: 0px solid #CDEFE1; margin: 0px;"
   href="/preview/releases/versioning/#feature-maturity">
   GA
 </a>
diff --git a/docs/layouts/shortcodes/badge/tp.html b/docs/layouts/shortcodes/badge/tp.html
@@ -3,7 +3,7 @@
 {{/* version is the first path of the path. eg: preview/stable/v2.12 ...*/}}
 {{- $version := index $path 0 -}}
 <a class="yb-badge"
-  style="background: #FF603B; color:#FFF; border: 0px solid #FF603B; margin: 0px;"
+  style="background: #FFEEC8; color:#9D6C00; border: 0px solid #FFEEC8; margin: 0px;"
   href="/preview/releases/versioning/#feature-maturity">
   TP
 </a>
diff --git a/java/yb-pgsql/src/test/java/org/yb/pgsql/TestYbAsh.java b/java/yb-pgsql/src/test/java/org/yb/pgsql/TestYbAsh.java
@@ -37,13 +37,11 @@
 @RunWith(value = YBTestRunner.class)
 public class TestYbAsh extends BasePgSQLTest {
   private static final int ASH_SAMPLING_INTERVAL = 1000;
-
   private static final int ASH_SAMPLE_SIZE = 500;
-
   private static final String ASH_VIEW = "yb_active_session_history";
 
   private void setAshConfigAndRestartCluster(
-      int sampling_interval, int sample_size) throws Exception {
+      int sampling_interval, int sample_size, int circular_buffer_size) throws Exception {
     Map<String, String> flagMap = super.getTServerFlags();
     if (isTestRunningWithConnectionManager()) {
       flagMap.put("allowed_preview_flags_csv",
@@ -55,11 +53,18 @@ private void setAshConfigAndRestartCluster(
     flagMap.put("ysql_yb_enable_ash", "true");
     flagMap.put("ysql_yb_ash_sampling_interval_ms", String.valueOf(sampling_interval));
     flagMap.put("ysql_yb_ash_sample_size", String.valueOf(sample_size));
-    // flagMap.put("create_initial_sys_catalog_snapshot", "true");
+    if (circular_buffer_size > 0) {
+      flagMap.put("ysql_yb_ash_circular_buffer_size", String.valueOf(circular_buffer_size));
+    }
     Map<String, String> masterFlagMap = super.getMasterFlags();
     restartClusterWithFlags(masterFlagMap, flagMap);
   }
 
+  private void setAshConfigAndRestartCluster(
+      int sampling_interval, int sample_size) throws Exception {
+    setAshConfigAndRestartCluster(sampling_interval, sample_size, 0);
+  }
+
   private void executePgSleep(Statement statement, long seconds) throws Exception {
     statement.execute("SELECT pg_sleep(" + seconds + ")");
   }
@@ -252,10 +257,10 @@ public void testNestedQueriesWithAsh() throws Exception {
    */
   @Test
   public void testPgAuxInfo() throws Exception {
-    setAshConfigAndRestartCluster(10, ASH_SAMPLE_SIZE);
+    setAshConfigAndRestartCluster(50, ASH_SAMPLE_SIZE, 1);
     try (Statement statement = connection.createStatement()) {
       statement.execute("CREATE TABLE test_table(k INT, v TEXT)");
-      for (int i = 0; i < 10000; ++i) {
+      for (int i = 0; i < 100; ++i) {
         statement.execute(String.format("INSERT INTO test_table VALUES(%d, 'v-%d')", i, i));
         statement.execute(String.format("SELECT v FROM test_table WHERE k=%d", i));
       }