diff --git a/docs/content/preview/architecture/transactions/read-committed.md b/docs/content/preview/architecture/transactions/read-committed.md index a8ca49d2af7d..54180d0ee261 100644 --- a/docs/content/preview/architecture/transactions/read-committed.md +++ b/docs/content/preview/architecture/transactions/read-committed.md @@ -76,7 +76,7 @@ The recheck steps are as follows: 1. On commit of any conflicting transaction, traverse the chain of updates as described above and re-evaluate the latest version of the row for any conflict. If there is no conflict, `insert` the original row. Else, perform the `do update` part on the latest version of the row. 1. ON CONFLICT DO NOTHING: do nothing if a conflict occurs. -Note that the above methodology in PostgreSQL can lead to two different user visible semantics, one which is the common case and another which is a degenerate situation which can never be seen in practice, but is nevertheless possible and still upholds the semantics of Read Commited isolation. The common case is as follows: +Note that the preceding methodology in PostgreSQL can lead to two different user visible semantics. One is the common case, and the other is a degenerate situation that can never be seen in practice, but is nevertheless possible and still upholds the semantics of Read Committed isolation. The common case is as follows: ```sql create table test (k int primary key, v int); diff --git a/docs/content/preview/contribute/docs/widgets-and-shortcodes.md b/docs/content/preview/contribute/docs/widgets-and-shortcodes.md index 1ae52fdd7aa4..e63306bee3ed 100644 --- a/docs/content/preview/contribute/docs/widgets-and-shortcodes.md +++ b/docs/content/preview/contribute/docs/widgets-and-shortcodes.md @@ -17,10 +17,11 @@ There are a number of display widgets and shortcodes available. All the shortcod ## Admonition boxes Use the note, tip, and warning shortcodes to create admonition boxes. + ### tip {{< tip title="Tip" >}} -A tip box gives a hint or other useful but optional piece of information. +A tip box gives a hint or other helpful but optional piece of information. {{< /tip >}} #### tip source @@ -63,7 +64,7 @@ An inline section switcher lets you switch between content sections **without a ![Inline section switcher](https://raw.githubusercontent.com/yugabyte/docs/master/contributing/inline-section-switcher.png) -The corresponding code for this widget is shown below. Note that the actual content must be placed in a file with the `.md` extension inside a subdirectory whose name is easy to associate with the switcher title. +The corresponding code for this widget is as follows. Note that the actual content must be placed in a file with the `.md` extension inside a subdirectory; name the subdirectory such that it can be associated with the switcher title. ```html -
- -
- -
Security
-
-
- Learn about strategies for securing YugabyteDB. -
- -
+
+
-
- -
- -
Observability
-
-
- Observe metrics at every layer of the database using Prometheus. -
- -
+To run the Explore examples in YugabyteDB Managed, create a single- or multi-node cluster as follows. + +{{< tabpane text=true >}} + + {{% tab header="Single-node cluster" lang="YBM Single" %}} + +Examples requiring a single-node cluster can be run using the free [Sandbox](../yugabyte-cloud/cloud-basics/create-clusters/create-clusters-free/) cluster. + +If you haven't already created your sandbox cluster, log in to YugabyteDB Managed, on the **Clusters** page click **Add Cluster**, and follow the instructions in the **Create Cluster** wizard. + +Save your cluster credentials in a convenient location. You will use them to connect to your cluster. + + {{% /tab %}} + + {{% tab header="Multi-node cluster" lang="YBM Multi" %}} + +Before you can create a multi-node cluster in YugabyteDB Managed, you need to [add your billing profile and payment method](../yugabyte-cloud/cloud-admin/cloud-billing-profile/), or you can [request a free trial](https://support.yugabyte.com/hc/en-us/requests/new?ticket_form_id=360003113431). + +To create a single region multi-node cluster, refer to [Create a single-region cluster](../yugabyte-cloud/cloud-basics/create-clusters/create-single-region/). + + {{% /tab %}} + +{{< /tabpane >}} + +**Connect to your clusters** + +You can run Explore exercises in YugabyteDB Managed using the [Cloud Shell](../yugabyte-cloud/cloud-connect/connect-cloud-shell/): + +1. In YugabyteDB Managed, on the **Clusters** page, select your cluster. +1. Click **Connect**. +1. Click **Launch Cloud Shell**. +1. Enter the user name from the cluster credentials you downloaded when you created the cluster. +1. Select the API to use (YSQL or YCQL) and click **Confirm**. + The shell displays in a separate browser page. Cloud Shell can take up to 30 seconds to be ready. +1. Enter the password from the cluster credentials you downloaded when you created the cluster. + +Note that if your Cloud Shell session is idle for more than 5 minutes, your browser may disconnect you. To resume, close the browser tab and connect again. -
- -
- -
Cluster management
-
-
- Learn about point-in-time recovery. -
-
+
+ +To run the examples in Explore, you'll need to create a single- or multi-node cluster. +For testing and learning YugabyteDB on your computer, use the [yugabyted](../reference/configuration/yugabyted/) cluster management utility to create and manage clusters. + +{{< tabpane text=true >}} + + {{% tab header="Single-node cluster" lang="Single-node cluster" %}} + +You can create a single-node local cluster with a replication factor (RF) of 1 by running the following command: + +```sh +./bin/yugabyted start --advertise_address=127.0.0.1 +``` + +Or, if you are running macOS Monterey: + +```sh +./bin/yugabyted start --advertise_address=127.0.0.1 \ + --master_webserver_port=9999 +``` + +For more information, refer to [Quick Start](../quick-start/linux/#create-a-local-cluster). + +To stop a single-node cluster, do the following: + +```sh +./bin/yugabyted destroy +``` + + {{% /tab %}} + + {{% tab header="Multi-node cluster" lang="Multi-node cluster" %}} + +If a single-node cluster is currently running, first destroy the running cluster as follows: + +```sh +./bin/yugabyted destroy +``` + +Start a local three-node cluster with a replication factor of `3`by first creating a single node cluster as follows: + +```sh +./bin/yugabyted start \ + --advertise_address=127.0.0.1 \ + --base_dir=/tmp/ybd1 \ + --cloud_location=aws.us-east.us-east-1a +``` + +On MacOS and Linux, the additional nodes need loopback addresses configured: + +```sh +sudo ifconfig lo0 alias 127.0.0.2 +sudo ifconfig lo0 alias 127.0.0.3 +``` + +Next, join two more nodes with the previous node. By default, [yugabyted](../reference/configuration/yugabyted/) creates a cluster with a replication factor of `3` on starting a 3 node cluster. + +```sh +./bin/yugabyted start \ + --advertise_address=127.0.0.2 \ + --base_dir=/tmp/ybd2 \ + --cloud_location=aws.us-east.us-east-2a \ + --join=127.0.0.1 +``` + +```sh +./bin/yugabyted start \ + --advertise_address=127.0.0.3 \ + --base_dir=/tmp/ybd3 \ + --cloud_location=aws.us-east.us-east-3a \ + --join=127.0.0.1 +``` + +```sh +./bin/yugabyted configure --fault_tolerance=zone +``` + +To destroy the multi-node cluster, do the following: + +```sh +./bin/yugabyted destroy --base_dir=/tmp/ybd1 +./bin/yugabyted destroy --base_dir=/tmp/ybd2 +./bin/yugabyted destroy --base_dir=/tmp/ybd3 +``` + + {{% /tab %}} + +{{< /tabpane >}} + +**Connect to clusters** + +To run the examples in your cluster, you use either the ysqlsh or ycqlsh CLI to interact with YugabyteDB via the YSQL or YCQL API. + +To start ysqlsh: + +```sh +./bin/ysqlsh +``` + +```output +ysqlsh (11.2-YB-2.0.0.0-b0) +Type "help" for help. + +yugabyte=# +``` + +To start ycqlsh: + +```sh +./bin/ycqlsh +``` + +```output +Connected to local cluster at 127.0.0.1:9042. +[ycqlsh 5.0.1 | Cassandra 3.9-SNAPSHOT | CQL spec 3.4.2 | Native protocol v4] +Use HELP for help. +ycqlsh> +``` + +
+ +## Next step + +Start exploring [SQL features](ysql-language-features/). diff --git a/docs/content/preview/explore/cluster-management/point-in-time-recovery-ycql.md b/docs/content/preview/explore/cluster-management/point-in-time-recovery-ycql.md index 8f3deead8b3e..a28941838332 100644 --- a/docs/content/preview/explore/cluster-management/point-in-time-recovery-ycql.md +++ b/docs/content/preview/explore/cluster-management/point-in-time-recovery-ycql.md @@ -30,10 +30,18 @@ type: docs -Point-in-time recovery (PITR) allows you to restore the state of your cluster's data and some types of metadata from a specific point in time. This can be relative, such as "three hours ago", or an absolute timestamp. For more information, see [Point-in-time recovery](../../../manage/backup-restore/point-in-time-recovery/#features). For details on the `yb-admin` commands, refer to the [Backup and snapshot commands](../../../admin/yb-admin/#backup-and-snapshot-commands) section of the yb-admin documentation. +Point-in-time recovery (PITR) allows you to restore the state of your cluster's data and some types of metadata from a specific point in time. This can be relative, such as "three hours ago", or an absolute timestamp. + +For more information, see [Point-in-time recovery](../../../manage/backup-restore/point-in-time-recovery/#features). For details on the `yb-admin` commands, refer to [Backup and snapshot commands](../../../admin/yb-admin/#backup-and-snapshot-commands). You can try out PITR by creating a namespace and populating it, creating a snapshot schedule, and restoring from that schedule. +{{< note title="Setup" >}} + +Local single-node cluster. See [Set up your YugabyteDB cluster](../../../explore/#set-up-your-yugabytedb-cluster). + +{{< /note >}} + {{< note title="Note" >}} This document contains examples that are deliberately simplified. In many of the scenarios, you could drop the index or table to recover. Consider the examples as part of an effort to undo a larger schema change, such as a database migration, which has performed several operations. diff --git a/docs/content/preview/explore/cluster-management/point-in-time-recovery-ysql.md b/docs/content/preview/explore/cluster-management/point-in-time-recovery-ysql.md index 51465d7436cb..52b8a69b9054 100644 --- a/docs/content/preview/explore/cluster-management/point-in-time-recovery-ysql.md +++ b/docs/content/preview/explore/cluster-management/point-in-time-recovery-ysql.md @@ -35,6 +35,12 @@ For more information, see [Point-in-time recovery](../../../manage/backup-restor You can try out the PITR feature by creating a database and populating it, creating a snapshot schedule, and restoring from a snapshot on the schedule. +{{< note title="Setup" >}} + +Local single-node cluster. See [Set up your YugabyteDB cluster](../../../explore/#set-up-your-yugabytedb-cluster). + +{{< /note >}} + {{< note title="Note" >}} This document contains examples that are deliberately simplified. In many of the scenarios, you could drop the index or table to recover. Consider the examples as part of an effort to undo a larger schema change, such as a database migration, which has performed several operations. diff --git a/docs/content/preview/explore/fault-tolerance/macos.md b/docs/content/preview/explore/fault-tolerance/macos.md index 60e92bec3bb3..e0c34fafdbe2 100644 --- a/docs/content/preview/explore/fault-tolerance/macos.md +++ b/docs/content/preview/explore/fault-tolerance/macos.md @@ -19,33 +19,11 @@ type: docs YugabyteDB can automatically handle failures and therefore provides [high availability](../../../architecture/core-functions/high-availability/). This tutorial demonstrates how YugabyteDB can continue to do reads and writes even in case of node failures. You create YSQL tables with a replication factor (RF) of 3, which allows a [fault tolerance](../../../architecture/docdb-replication/replication/) of 1. This means the cluster remains available for both reads and writes even if one node fails. However, if another node fails (bringing the number of failures to two), writes become unavailable on the cluster to preserve data consistency. -This tutorial uses the [yugabyted](../../../reference/configuration/yugabyted/) cluster management utility. +{{< note title="Setup" >}} -## Create a universe +Local multi-node cluster. See [Set up your YugabyteDB cluster](../../../explore/#set-up-your-yugabytedb-cluster). -To start a local three-node cluster with a replication factor of `3`, first create a single-node cluster as follows: - -```sh -./bin/yugabyted start \ - --listen=127.0.0.1 \ - --base_dir=/tmp/ybd1 -``` - -Next, join two more nodes with the previous node. By default, [yugabyted](../../../reference/configuration/yugabyted/) creates a cluster with a replication factor of `3` on starting a 3 node cluster. - -```sh -./bin/yugabyted start \ - --listen=127.0.0.2 \ - --base_dir=/tmp/ybd2 \ - --join=127.0.0.1 -``` - -```sh -./bin/yugabyted start \ - --listen=127.0.0.3 \ - --base_dir=/tmp/ybd3 \ - --join=127.0.0.1 -``` +{{< /note >}} ## Run the sample key-value app @@ -55,7 +33,7 @@ Download the YugabyteDB workload generator JAR file (`yb-sample-apps.jar`) using wget https://github.com/yugabyte/yb-sample-apps/releases/download/1.3.9/yb-sample-apps.jar?raw=true -O yb-sample-apps.jar ``` -Run the `SqlInserts` workload against the local universe using the following command: +Run the `SqlInserts` workload against the local cluster using the following command: ```sh java -jar ./yb-sample-apps.jar --workload SqlInserts \ @@ -83,7 +61,7 @@ Note how both the reads and the writes are roughly the same across all the nodes ## Remove a node and observe continuous write availability -Remove a node from the universe using the following command: +Remove a node from the cluster using the following command: ```sh ./bin/yugabyted stop \ @@ -92,7 +70,7 @@ Remove a node from the universe using the following command: Refresh the [tablet-servers](http://127.0.0.1:7000/tablet-servers) page to see the status update. -The `Time since heartbeat` value for that node will keep increasing. After that number reaches 60s (1 minute), YugabyteDB changes the status of that node from `ALIVE` to `DEAD`. Note that at this time the universe is running in an under-replicated state for some subset of tablets. +The `Time since heartbeat` value for that node will keep increasing. When the number reaches 60s (1 minute), YugabyteDB changes the status of the removed node from `ALIVE` to `DEAD`. Note that at this time the cluster is running in an under-replicated state for some subset of tablets. ![Read and write IOPS with 3rd node dead](/images/ce/fault_tolerance_dead_node.png) diff --git a/docs/content/preview/explore/indexes-constraints/covering-index-ysql.md b/docs/content/preview/explore/indexes-constraints/covering-index-ysql.md index 7b0c8af7acac..a09a8c47405c 100644 --- a/docs/content/preview/explore/indexes-constraints/covering-index-ysql.md +++ b/docs/content/preview/explore/indexes-constraints/covering-index-ysql.md @@ -32,9 +32,9 @@ CREATE INDEX columnA_columnB_index_name ON table_name(columnA, columnB) INCLUDE ## Example -The following exercise demonstrates how to perform an index-only scan on an [expression (functional) index](../expression-index-ysql/), and further optimize the query performance using a covering index. +{{% explore-setup-single %}} -Create a cluster [locally](../../../quick-start/) or in [YugabyteDB Managed](../../../yugabyte-cloud/cloud-basics/create-clusters-free/) and connect to the cluster using [ysqlsh](../../../admin/ysqlsh/) for local clusters, or [using cloud shell](../../../yugabyte-cloud/cloud-connect/connect-cloud-shell/) for YugabyteDB Managed. +The following exercise demonstrates how to perform an index-only scan on an [expression (functional) index](../expression-index-ysql/), and further optimize the query performance using a covering index. 1. Create and insert some rows into a table `demo` with two columns `id` and `username`. diff --git a/docs/content/preview/explore/indexes-constraints/expression-index-ysql.md b/docs/content/preview/explore/indexes-constraints/expression-index-ysql.md index 0563fe3fc149..97c792f637df 100644 --- a/docs/content/preview/explore/indexes-constraints/expression-index-ysql.md +++ b/docs/content/preview/explore/indexes-constraints/expression-index-ysql.md @@ -34,6 +34,8 @@ Once defined, the index is used when the expression that defines the index is in ## Example +{{% explore-setup-single %}} + A common use case of an expression index is to support case-insensitive text to enable efficient searchability. For example, suppose you have a `users` table with an `email` column to store login email addresses, and you want to maintain case-insensitive authentication. Using the WHERE clause as WHERE LOWER(email) = '' allows you to store the email address as originally entered by the user. diff --git a/docs/content/preview/explore/indexes-constraints/foreign-key-ysql.md b/docs/content/preview/explore/indexes-constraints/foreign-key-ysql.md index 7af91b4807aa..5bb31ea132d5 100644 --- a/docs/content/preview/explore/indexes-constraints/foreign-key-ysql.md +++ b/docs/content/preview/explore/indexes-constraints/foreign-key-ysql.md @@ -54,7 +54,7 @@ Defining the `CONSTRAINT` clause and naming the foreign key is optional. If you ## Examples -Create a cluster [locally](../../../quick-start/) or in [YugabyteDB Managed](../../../yugabyte-cloud/cloud-basics/create-clusters-free/) and connect to the cluster using [ysqlsh](../../../admin/ysqlsh/) for local clusters, or [using cloud shell](../../../yugabyte-cloud/cloud-connect/connect-cloud-shell/) for YugabyteDB Managed. +{{% explore-setup-single %}} The following example creates two tables: diff --git a/docs/content/preview/explore/indexes-constraints/gin.md b/docs/content/preview/explore/indexes-constraints/gin.md index 42799802be3b..1458b49464b7 100644 --- a/docs/content/preview/explore/indexes-constraints/gin.md +++ b/docs/content/preview/explore/indexes-constraints/gin.md @@ -22,21 +22,18 @@ type: docs -In YugabyteDB, tables and secondary indexes are both [key-value stores internally][arch-persistence]. -Loosely speaking: +In YugabyteDB, tables and secondary indexes are both [key-value stores internally](../../../architecture/docdb/persistence/). Loosely speaking: - A _table's_ internal key-value store maps primary keys to the remaining columns. - A _secondary index's_ internal key-value store maps index keys to primary keys. -**Regular indexes index columns.** This makes queries with conditions on the columns more efficient. -For example, if you had a regular index on a single int array column (currently not possible in YSQL), queries like `WHERE myintarray = '{1,3,6}'` would be more efficient when using the index. -However, queries like `WHERE myintarray @> '{3}'` (meaning "is `3` an element of `myintarray`?") would not benefit from the regular index. +**Regular indexes index columns.** + +This makes queries with conditions on the columns more efficient. For example, if you had a regular index on a single int array column (currently not possible in YSQL), queries such as `WHERE myintarray = '{1,3,6}'` would be more efficient when using the index. However, queries like `WHERE myintarray @> '{3}'` (meaning "is `3` an element of `myintarray`?") would not benefit from the regular index. **Generalized inverted indexes (GIN indexes) index elements inside container columns.** -This makes queries with conditions on elements inside the columns more efficient. -The above example would benefit from a GIN index since we can look up the key `3` in the gin index. -[arch-persistence]: ../../../architecture/docdb/persistence/ +This makes queries with conditions on elements inside the columns more efficient. The preceding example would benefit from a GIN index because you can look up the key `3` in the gin index. ## Compatible types @@ -46,8 +43,7 @@ GIN indexes can only be created over a few types: - A GIN index on an _array_ column indexes _array elements_ - A GIN index on a _jsonb_ column indexes _keys/values_ -You can use extensions to support more types. -However, extension support is still in progress: +You can use extensions to support more types. However, extension support is still in progress: - `btree_gin`: unsupported - `hstore`: supported @@ -64,8 +60,7 @@ Create the index using `USING ybgin` to specify the index access method: CREATE INDEX ON mytable USING ybgin (mycol); ``` -The `gin` access method is reserved for temporary relations while `ybgin` is for Yugabyte-backed relations. -You can still specify `USING gin`, and, if `mytable` is not a temporary table, it will be automatically substituted for `ybgin`. +The `gin` access method is reserved for temporary relations while `ybgin` is for Yugabyte-backed relations. You can still specify `USING gin`, and, if `mytable` is not a temporary table, it will be automatically substituted for `ybgin`. GIN indexes can't be unique, so `CREATE UNIQUE INDEX` is not allowed. @@ -75,9 +70,7 @@ Writes are fully supported. ### SELECT -Only [certain SELECTs][operators] use the GIN index. - -[operators]: https://www.postgresql.org/docs/13/gin-builtin-opclasses.html +Only [certain SELECTs](https://www.postgresql.org/docs/13/gin-builtin-opclasses.html) use the GIN index. ## Changes from PostgreSQL @@ -90,7 +83,7 @@ YugabyteDB GIN indexes are somewhat different from PostgreSQL GIN indexes: ## Examples -1. To begin, set up a YugabyteDB cluster. For instance, using `yugabyted`, +1. To begin, set up a local YugabyteDB cluster. For instance, using `yugabyted`, do the following: ```sh ./bin/yugabyted start --base_dir /tmp/gindemo/1 --listen 127.0.0.201 @@ -142,21 +135,21 @@ YugabyteDB GIN indexes are somewhat different from PostgreSQL GIN indexes: ### Timing -Here are some examples to show the speed improvement of queries using GIN index. -GIN indexes currently support IndexScan only, not IndexOnlyScan. -The difference is that IndexScan uses the results of a scan to the index for filtering on the indexed table whereas an IndexOnlyScan need not go to the indexed table since the results from the index are sufficient. -Therefore, a GIN index scan can be more costly than a sequential scan straight to the main table if the index scan does not filter out many rows. -Since cost estimates currently aren't very accurate, the more costly index scan may be chosen in some cases. +The following examples show the speed improvement of queries using GIN indexes. + +GIN indexes currently support IndexScan only, not IndexOnlyScan. The difference is that IndexScan uses the results of a scan to the index for filtering on the indexed table, whereas an IndexOnlyScan need not go to the indexed table because the results from the index are sufficient. Therefore, a GIN index scan can be more costly than a sequential scan straight to the main table if the index scan does not filter out many rows. + +Because cost estimates currently aren't very accurate, the more costly index scan may be chosen in some cases. -The assumption in the following examples is that the user is using the GIN index in ways that take advantage of it. +The assumption in the following examples is that you are using the GIN index in ways that take advantage of it. -1. First, enable timing for future queries. +1. Enable timing for future queries. ```sql \timing on ``` -1. Test GIN index on tsvector: +1. Test GIN index on tsvector as follows: ```sql SET enable_indexscan = off; @@ -186,12 +179,9 @@ The assumption in the following examples is that the user is using the GIN index Time: 2.838 ms ``` - \ - Notice the over 3x timing improvement when using GIN index. - This is on a relatively small table: a little over 1000 rows. - With more and/or bigger rows, the timing improvement should get better. + Notice the over 3x timing improvement when using GIN index. This is on a relatively small table: a little over 1000 rows. With more and/or bigger rows, the timing improvement should get better. -1. Next, try on an int array: +1. Use a GIN index on an int array as follows: ```sql SET enable_indexscan = off; @@ -221,7 +211,7 @@ The assumption in the following examples is that the user is using the GIN index Time: 2.989 ms ``` -1. Finally, try with jsonb: +1. Use a GIN index on with JSONB as follows: ```sql SET enable_indexscan = off; @@ -255,29 +245,17 @@ The assumption in the following examples is that the user is using the GIN index ### Using opclass `jsonb_path_ops` -By default, jsonb GIN indexes use the opclass `jsonb_ops`. -There is another opclass `jsonb_path_ops` that can be used instead. +By default, JSONB GIN indexes use the opclass `jsonb_ops`. Another opclass, `jsonb_path_ops`, can be used instead. -The difference is the way they extract elements out of a jsonb. -`jsonb_ops` extracts keys and values and encodes them as ``. -For example, `'{"abc":[123,true]}'` maps to three GIN keys: `\001abc`, `\004123`, `\003t`. -The flag bytes here indicate the types key, numeric, and boolean, respectively. +The difference is the way they extract elements out of a JSONB. `jsonb_ops` extracts keys and values and encodes them as ``. For example, `'{"abc":[123,true]}'` maps to three GIN keys: `\001abc`, `\004123`, `\003t`. The flag bytes here indicate the types key, numeric, and boolean, respectively. -On the other hand, `jsonb_path_ops` extracts hashed paths. -Using the above example, there are two paths: `"abc" -> 123` and `"abc" -> true`. -Then, there are two GIN keys based on those paths using an internal hashing mechanism: `-1570777299`, `-1227915239`. +On the other hand, `jsonb_path_ops` extracts hashed paths. Using the above example, there are two paths: `"abc" -> 123` and `"abc" -> true`. Then, there are two GIN keys based on those paths using an internal hashing mechanism: `-1570777299`, `-1227915239`. -`jsonb_path_ops` is better suited for queries involving paths, such as the `jsonb @> jsonb` operator. -However, it doesn't support as many operators as `jsonb_ops`. -If write performance and storage aren't an issue, it may be worth creating a GIN index of each jsonb opclass so that reads can choose the faster one. +`jsonb_path_ops` is better suited for queries involving paths, such as the `jsonb @> jsonb` operator. However, it doesn't support as many operators as `jsonb_ops`. If write performance and storage aren't an issue, it may be worth creating a GIN index of each jsonb opclass so that reads can choose the faster one. ### Presplitting -By default, `ybgin` indexes use a single range-partitioned tablet. -Like regular tables and indexes, it is possible to presplit a `ybgin` index to multiple tablets at specified split points. -These split points are for the index, so they need to be represented in the index key format. -This is simple for tsvector and array types, but it gets complicated for jsonb and text (`pg_trgm`). -`jsonb_path_ops` especially should use hash partitioning since the index key is itself a hash, but hash partitioning `ybgin` indexes is currently unsupported. +By default, `ybgin` indexes use a single range-partitioned tablet. Like regular tables and indexes, it is possible to presplit a `ybgin` index to multiple tablets at specified split points. These split points are for the index, so they need to be represented in the index key format. This is simple for tsvector and array types, but it gets complicated for JSONB and text (`pg_trgm`). `jsonb_path_ops` especially should use hash partitioning as the index key is itself a hash, but hash partitioning `ybgin` indexes is currently unsupported. ```sql CREATE INDEX NONCONCURRENTLY vectors_split_idx ON vectors USING ybgin (v) SPLIT AT VALUES (('j'), ('o')); @@ -286,7 +264,7 @@ CREATE INDEX NONCONCURRENTLY jsonbs_split_idx1 ON jsonbs USING ybgin (j) SPLIT A CREATE INDEX NONCONCURRENTLY jsonbs_split_idx2 ON jsonbs USING ybgin (j jsonb_path_ops) SPLIT AT VALUES ((-1000000000), (0), (1000000000)); ``` -Let's focus on just one index for the remainder of this example: `jsonbs_split_idx1`. +The remainder of the example focuses on just one index, `jsonbs_split_idx1`. First, check how the index is partitioned. @@ -303,11 +281,9 @@ c32e1066cefb449cb191ff23d626125f partition_key_start: "S\001some\000\000!" part ba23b657eb5b4bc891ca794bcad06db7 partition_key_start: "S\005jsonb\000\000!" partition_key_end: "" 127.0.0.202:9100 e24423119e734860bb0c3516df948b5c ``` -Then, check the data in each partition. -Flush it to SST files so that we can read them with `sst_dump`. -Ignore lines with "filler" because there are too many of them. -`!!` refers to the previous `list_tablets` command. -Adjust it if it doesn't work in your shell. +Then, check the data in each partition. Flush it to SST files so that we can read them with `sst_dump`. + +Ignore lines with "filler" because there are too many of them. `!!` refers to the previous `list_tablets` command. Adjust it if it doesn't work in your shell. ```sh bin/yb-admin \ @@ -370,29 +346,25 @@ GIN indexes are in active development, and currently have the following limitati - Multi-column GIN indexes are not currently supported. ([#10652](https://github.com/yugabyte/yugabyte-db/issues/10652)) - You can't yet specify `ASC`, `DESC`, or `HASH` sort methods. The default is `ASC` for prefix match purposes, so this can be relaxed in the future. ([#10653](https://github.com/yugabyte/yugabyte-db/issues/10653)) -- UPDATEs may be expensive, since they're currently implemented as DELETE + INSERT. +- UPDATEs may be expensive, as they're currently implemented as DELETE + INSERT. - SELECT operations have the following limitations: - Scans with non-default search mode aren't currently supported. - Scans can't ask for more than one index key. - \ + For example, a request for all rows whose array contains elements 1 or 3 will fail, but one that asks for elements 1 _and_ 3 can succeed by choosing one of the elements for index scan and rechecking the entire condition later. - \ + However, the choice between 1 and 3 is currently unoptimized, so 3 may be chosen even though 1 corresponds to less rows. - Recheck is always done rather than on a case-by-case basis, meaning there can be an unnecessary performance penalty. -If a query is unsupported, you can disable index scan to avoid an error message (`SET enable_indexscan TO off;`) before the query, and re-enable it (`SET enable_indexscan TO on;`) afterwards. -In the near future, cost estimates should route such queries to sequential scan. +If a query is unsupported, you can disable index scan to avoid an error message (`SET enable_indexscan TO off;`) before the query, and re-enable it (`SET enable_indexscan TO on;`) afterwards. In the near future, cost estimates should route such queries to sequential scan. ### Unsupported queries -Sometimes, an unsupported query may be encountered by getting an ERROR. -Here are some workarounds for some of these cases. +Sometimes, an unsupported query may be encountered by getting an ERROR. The following sections describe some workarounds for some of these cases. #### More than one required scan entry -Perhaps the most common issue would be "cannot use more than one required scan entry". -It means that the GIN index scan internally tries to fetch more than one index key. -Since this is currently not supported, it throws an ERROR. +Perhaps the most common issue would be "cannot use more than one required scan entry". This means that the GIN index scan internally tries to fetch more than one index key. Because this is currently not supported, it throws an ERROR. ```sql RESET enable_indexscan; @@ -480,8 +452,7 @@ SELECT * FROM vectors WHERE v @@ to_tsquery('simple', 'lazy'); Time: 5.559 ms ``` -If performance doesn't matter, the universal fix is to disable index scan so that sequential scan is used. -For sequential scan to be chosen, make sure that sequential scan is not also disabled. +If performance doesn't matter, the universal fix is to disable index scan so that sequential scan is used. For sequential scan to be chosen, make sure that sequential scan is not also disabled. ```sql SET enable_indexscan = off; @@ -503,9 +474,8 @@ Notice that the modified query using the index is still 2x faster than the origi #### Non-default search mode All search modes/strategies besides the default one are currently unsupported. -Many of these are best off using a sequential scan. -In fact, the query planner avoids many of these types of index scans by increasing the cost, leading to sequential scan being chosen as the better alternative. -Nevertheless, here are some cases that hit the ERROR. + +Many of these are best off using a sequential scan. In fact, the query planner avoids many of these types of index scans by increasing the cost, leading to sequential scan being chosen as the better alternative. Nevertheless, the following examples show some cases that hit the ERROR. ```sql RESET enable_indexscan; diff --git a/docs/content/preview/explore/indexes-constraints/other-constraints.md b/docs/content/preview/explore/indexes-constraints/other-constraints.md index 85a35c887b71..b17415d37614 100644 --- a/docs/content/preview/explore/indexes-constraints/other-constraints.md +++ b/docs/content/preview/explore/indexes-constraints/other-constraints.md @@ -23,6 +23,8 @@ type: docs +{{% explore-setup-single %}} + ## CHECK Constraint The YSQL `CHECK` constraint allows you to constrain values in columns based on a boolean expression. The values are evaluated with regards to meeting a specific requirement before these values are inserted or updated; if they fail the check, YSQL rejects the changes and displays a constraint violation error. diff --git a/docs/content/preview/explore/indexes-constraints/partial-index-ycql.md b/docs/content/preview/explore/indexes-constraints/partial-index-ycql.md index 44bcf9218aac..8146599bd450 100644 --- a/docs/content/preview/explore/indexes-constraints/partial-index-ycql.md +++ b/docs/content/preview/explore/indexes-constraints/partial-index-ycql.md @@ -50,7 +50,7 @@ CREATE INDEX index_name ON table_name(column_list) WHERE condition; ## Example -Create a cluster [locally](../../../quick-start/) or in [YugabyteDB Managed](../../../yugabyte-cloud/cloud-basics/create-clusters-free/) and connect to the cluster using [ycqlsh](../../../admin/ycqlsh/) for local clusters, or [using cloud shell](../../../yugabyte-cloud/cloud-connect/connect-cloud-shell/) for YugabyteDB Managed. +{{% explore-setup-single %}} Create a keyspace and a table as follows: diff --git a/docs/content/preview/explore/indexes-constraints/partial-index-ysql.md b/docs/content/preview/explore/indexes-constraints/partial-index-ysql.md index 85a851e5b1ab..a3b3f7c0ec94 100644 --- a/docs/content/preview/explore/indexes-constraints/partial-index-ysql.md +++ b/docs/content/preview/explore/indexes-constraints/partial-index-ysql.md @@ -44,9 +44,9 @@ The `WHERE` clause specifies which rows need to be added to the index. ## Example -This example uses the `customers` table from the [Northwind sample database](../../../sample-data/northwind/). +{{% explore-setup-single %}} -Create a cluster [locally](../../../quick-start/) or in [YugabyteDB Managed](../../../yugabyte-cloud/cloud-basics/create-clusters-free/) and [install](../../../sample-data/northwind/#install-the-northwind-sample-database) the sample Northwind database. +This example uses the `customers` table from the [Northwind sample database](../../../sample-data/northwind/#install-the-northwind-sample-database). View the contents of the `customers` table: diff --git a/docs/content/preview/explore/indexes-constraints/primary-key-ycql.md b/docs/content/preview/explore/indexes-constraints/primary-key-ycql.md index 34fdea227d66..bd5845b98898 100644 --- a/docs/content/preview/explore/indexes-constraints/primary-key-ycql.md +++ b/docs/content/preview/explore/indexes-constraints/primary-key-ycql.md @@ -41,7 +41,7 @@ Refer to the Grammar section for [CREATE TABLE](../../../api/ycql/ddl_create_tab ## Examples -Create a cluster [locally](../../../quick-start/) or in [YugabyteDB Managed](../../../yugabyte-cloud/cloud-basics/create-clusters-free/) and connect to the cluster using [ycqlsh](../../../admin/ycqlsh/) for local clusters, or [using cloud shell](../../../yugabyte-cloud/cloud-connect/connect-cloud-shell/) for YugabyteDB Managed. +{{% explore-setup-single %}} ### Column constraint diff --git a/docs/content/preview/explore/indexes-constraints/primary-key-ysql.md b/docs/content/preview/explore/indexes-constraints/primary-key-ysql.md index cc726db7846a..5a95f2a2e43e 100644 --- a/docs/content/preview/explore/indexes-constraints/primary-key-ysql.md +++ b/docs/content/preview/explore/indexes-constraints/primary-key-ysql.md @@ -35,7 +35,8 @@ The Primary Key constraint is a means to uniquely identify a specific row in a t ## Syntax and examples -Create a cluster [locally](../../../quick-start/) or in [YugabyteDB Managed](../../../yugabyte-cloud/cloud-basics/create-clusters-free/) and connect to the cluster using [ysqlsh](../../../admin/ysqlsh/) for local clusters, or [using cloud shell](../../../yugabyte-cloud/cloud-connect/connect-cloud-shell/) for YugabyteDB Managed. +{{% explore-setup-single %}} + ### Primary key for a single column diff --git a/docs/content/preview/explore/indexes-constraints/secondary-indexes.md b/docs/content/preview/explore/indexes-constraints/secondary-indexes.md index 531279b35d71..5717fa52a9ff 100644 --- a/docs/content/preview/explore/indexes-constraints/secondary-indexes.md +++ b/docs/content/preview/explore/indexes-constraints/secondary-indexes.md @@ -67,6 +67,8 @@ For additional information, see [DROP INDEX YCQL API](../../../api/ycql/ddl_drop ## Example scenario using YSQL +{{% explore-setup-single %}} + Suppose you work with a database that includes the following table populated with data: ```sql diff --git a/docs/content/preview/explore/indexes-constraints/unique-index-ycql.md b/docs/content/preview/explore/indexes-constraints/unique-index-ycql.md index 164c431cfb02..9e57463b262c 100644 --- a/docs/content/preview/explore/indexes-constraints/unique-index-ycql.md +++ b/docs/content/preview/explore/indexes-constraints/unique-index-ycql.md @@ -37,6 +37,8 @@ When a unique index is applied to two or more columns, the combined values in th If a table has a primary key defined, a corresponding unique index is created automatically. +{{% explore-setup-single %}} + ## Syntax ```sql @@ -45,8 +47,6 @@ CREATE UNIQUE INDEX index_name ON table_name(column_list); ## Example -Create a cluster [locally](../../../quick-start/) or in [YugabyteDB Managed](../../../yugabyte-cloud/cloud-basics/create-clusters-free/) and connect to the cluster using [ycqlsh](../../../admin/ycqlsh/) for local clusters, or [using cloud shell](../../../yugabyte-cloud/cloud-connect/connect-cloud-shell/) for YugabyteDB Managed. - 1. Create a keyspace and a table as follows: ```cql diff --git a/docs/content/preview/explore/indexes-constraints/unique-index-ysql.md b/docs/content/preview/explore/indexes-constraints/unique-index-ysql.md index 960883a2df6e..041a5a050805 100644 --- a/docs/content/preview/explore/indexes-constraints/unique-index-ysql.md +++ b/docs/content/preview/explore/indexes-constraints/unique-index-ysql.md @@ -44,9 +44,9 @@ CREATE UNIQUE INDEX index_name ON table_name(column_list); ## Example -This example uses the `categories` table from the [Northwind sample database](../../../sample-data/northwind/). +{{% explore-setup-single %}} -Create a cluster [locally](../../../quick-start/) or in [YugabyteDB Managed](../../../yugabyte-cloud/cloud-basics/create-clusters-free/) and [install](../../../sample-data/northwind/#install-the-northwind-sample-database) the sample Northwind database. +This example uses the `categories` table from the [Northwind sample database](../../../sample-data/northwind/#install-the-northwind-sample-database). View the contents of the `categories` table: diff --git a/docs/content/preview/explore/json-support/jsonb-ycql.md b/docs/content/preview/explore/json-support/jsonb-ycql.md index 82021c8c5832..bd06f510d9d5 100644 --- a/docs/content/preview/explore/json-support/jsonb-ycql.md +++ b/docs/content/preview/explore/json-support/jsonb-ycql.md @@ -14,8 +14,6 @@ menu: type: docs --- -JSON data types are for storing JSON (JavaScript Object Notation) data, as specified in [RFC 7159](https://tools.ietf.org/html/rfc7159). Such data can also be stored as `text`, but the JSON data types have the advantage of enforcing that each stored value is valid according to the JSON rules. There are also assorted JSON-specific functions and operators available for data stored in these data types. -
  • @@ -34,24 +32,17 @@ JSON data types are for storing JSON (JavaScript Object Notation) data, as speci
+JSON data types are for storing JSON (JavaScript Object Notation) data, as specified in [RFC 7159](https://tools.ietf.org/html/rfc7159). Such data can also be stored as `text`, but the JSON data types have the advantage of enforcing that each stored value is valid according to the JSON rules. Assorted JSON-specific functions and operators are also available for data stored in these data types. -The JSON data type supported in YCQL is `jsonb`. The JSON functionality in YCQL is **a subset** of the [JSON functionality in PostgreSQL](https://www.postgresql.org/docs/11/datatype-json.html). - -## 1. Prerequisites +{{% explore-setup-single %}} -You need a YugabyteDB cluster to run through the steps below. If do not have a YugabyteDB cluster, you can create one on your local machine as shown below. +JSON functionality in YCQL is **a subset** of the [JSON functionality in PostgreSQL](https://www.postgresql.org/docs/11/datatype-json.html). -```sh -$ ./bin/yugabyted start -``` +YCQL supports the JSONB data type. -Connect to the cluster using `ycqlsh` to run through the examples below. - -```sh -$ ./bin/ycqlsh -``` +## Create a table -Create table with a JSONB column +Create a table with a JSONB column as follows: ```sql ycqlsh> CREATE KEYSPACE store; @@ -61,7 +52,7 @@ ycqlsh> CREATE KEYSPACE store; ycqlsh> CREATE TABLE store.books ( id int PRIMARY KEY, details jsonb ); ``` -Insert JSONB documents +Insert JSONB documents: ```sql INSERT INTO store.books (id, details) VALUES @@ -76,7 +67,7 @@ INSERT INTO store.books (id, details) VALUES (5, '{ "name": "A Brief History of Time", "author": { "first_name": "Stephen", "last_name": "Hawking" }, "year": 1988, "genre": "science", "editors": ["Melisa", "Mark", "John"] }'); ``` -## 2. Query JSON documents +## Query JSON documents You can list all the row inserted using the command below. @@ -84,7 +75,7 @@ You can list all the row inserted using the command below. ycqlsh> SELECT * FROM store.books; ``` -``` +```output id | details ----+------------------------------------------------------------------------------------------------------------------------------------------------------------- 5 | {"author":{"first_name":"Stephen","last_name":"Hawking"},"editors":["Melisa","Mark","John"],"genre":"science","name":"A Brief History of Time","year":1988} @@ -96,61 +87,60 @@ ycqlsh> SELECT * FROM store.books; ### Using `->` and `->>` -- Select with condition on JSONB object value +Select with condition on JSONB object value: -```sql +```cql ycqlsh> SELECT * FROM store.books WHERE details->'author'->>'first_name' = 'William' AND details->'author'->>'last_name' = 'Shakespeare'; ``` -``` +```output id | details ----+---------------------------------------------------------------------------------------------------------------------------------- 1 | {"author":{"first_name":"William","last_name":"Shakespeare"},"editors":["John","Elizabeth","Jeff"],"name":"Macbeth","year":1623} 2 | {"author":{"first_name":"William","last_name":"Shakespeare"},"editors":["Lysa","Mark","Robert"],"name":"Hamlet","year":1603} ``` -- Select with condition on JSONB array element +Select with condition on JSONB array element: -```sql +```cql ycqlsh> SELECT * FROM store.books WHERE details->'editors'->>0 = 'Mark'; ``` -``` +```output id | details ----+------------------------------------------------------------------------------------------------------------------------------------------------- 3 | {"author":{"first_name":"Charles","last_name":"Dickens"},"editors":["Mark","Tony","Britney"],"genre":"novel","name":"Oliver Twist","year":1838} ``` -- Select with condition using on JSONB element +Select with condition using JSONB element: ```sql ycqlsh> SELECT * FROM store.books WHERE CAST(details->>'year' AS integer) = 1950; ``` -``` +```output id | details ----+-------------------------------------------------------------------------------------------------------------------------------------------------------- 4 | {"author":{"first_name":"Charles","last_name":"Dickens"},"editors":["Robert","John","Melisa"],"genre":"novel","name":"Great Expectations","year":1950} ``` +## Update JSON documents -## 3. Update JSON documents - -There are a number of ways to update a JSON document, as shown below. +You can update a JSON document in a number of ways. -### Update entire document +### Update an entire document -You can do this as shown below. +To update an entire document, do the following: -```sql +```cql ycqlsh> UPDATE store.books SET details = '{"author":{"first_name":"Carl","last_name":"Sagan"},"editors":["Ann","Rob","Neil"],"genre":"science","name":"Cosmos","year":1980}' WHERE id = 1; ``` -```sql +```cql ycqlsh> SELECT * FROM store.books WHERE id = 1; ``` -``` +```output id | details ----+----------------------------------------------------------------------------------------------------------------------------------- 1 | {"author":{"first_name":"Carl","last_name":"Sagan"},"editors":["Ann","Rob","Neil"],"genre":"science","name":"Cosmos","year":1980} @@ -158,15 +148,17 @@ ycqlsh> SELECT * FROM store.books WHERE id = 1; ### Update an attribute -```sql +To update an attribute, do the following: + +```cql ycqlsh> UPDATE store.books SET details->'author'->>'first_name' = '"Steve"' WHERE id = 4; ``` -```sql +```cql ycqlsh> SELECT * FROM store.books WHERE id = 4; ``` -``` +```output id | details ----+------------------------------------------------------------------------------------------------------------------------------------------------------ 4 | {"author":{"first_name":"Steve","last_name":"Dickens"},"editors":["Robert","John","Melisa"],"genre":"novel","name":"Great Expectations","year":1950} @@ -174,53 +166,55 @@ ycqlsh> SELECT * FROM store.books WHERE id = 4; ### Update an array element -```sql +To update an array element, do the following: + +```cql ycqlsh> UPDATE store.books SET details->'editors'->>1 = '"Jack"' WHERE id = 4; ``` -```sql +```cql ycqlsh> SELECT * FROM store.books WHERE id = 4; ``` -``` +```output id | details ----+------------------------------------------------------------------------------------------------------------------------------------------------------ 4 | {"author":{"first_name":"Steve","last_name":"Dickens"},"editors":["Robert","Jack","Melisa"],"genre":"novel","name":"Great Expectations","year":1950} ``` -- Update a subdocument. +To update a subdocument: -```sql +```cql ycqlsh> UPDATE store.books SET details->'author' = '{"first_name":"John", "last_name":"Doe"}' WHERE id = 4; ``` -```sql +```cql ycqlsh> SELECT * FROM store.books WHERE id = 4; ``` -``` +```output id | details ----+------------------------------------------------------------------------------------------------------------------------------------------------- 4 | {"author":{"first_name":"John","last_name":"Doe"},"editors":["Robert","Jack","Melisa"],"genre":"novel","name":"Great Expectations","year":1950} ``` -## 4. Upserts +## Upserts ### Add attributes -Update a missing JSONB document resulting in an insert. +Update a missing JSONB document resulting in an insert as follows: -```sql +```cql INSERT INTO store.books (id, details) VALUES (6, '{}'); ycqlsh> UPDATE store.books SET details->'editors' = '["Adam", "Bryan", "Charles"]' WHERE id = 6; ``` -```sql +```cql ycqlsh> SELECT * FROM store.books WHERE id = 6; ``` -``` +```output id | details ----+------------------------------------------------------------------------------------------------------------------------------------------------- 6 | {"editors":["Adam","Bryan","Charles"]} @@ -228,29 +222,22 @@ ycqlsh> SELECT * FROM store.books WHERE id = 6; ### Add subdocuments -- Update a missing JSONB document resulting in an insert of a subdocument. +Update a missing JSONB document resulting in an insert of a subdocument as follows: -```sql +```cql ycqlsh> UPDATE store.books SET details->'author' = '{"first_name":"Jack", "last_name":"Kerouac"}' WHERE id = 6; ``` -```sql +```cql ycqlsh> SELECT * FROM store.books WHERE id = 6; ``` -``` +```output id | details ----+------------------------------------------------------------------------------------------------------------------------------------------------- 6 | {"author":{"first_name":"Jack","last_name":"Kerouac"},"editors":["Adam","Bryan","Charles"]} ``` {{< note title="Note" >}} -JSONB upsert only works for JSON objects and not for other data types like arrays, integers, strings, etc. Additionally, only the leaf property of an object will be inserted if it is missing. Upsert on non-leaf properties is not supported presently. +JSONB upsert only works for JSON objects and not for other data types like arrays, integers, strings, and so on. Additionally, only the leaf property of an object is inserted if it is missing. Upsert on non-leaf properties is not currently supported. {{< /note >}} - -## 5. Clean up (Optional) -Optionally, you can shut down the local cluster you created earlier. - -```sh -$ ./bin/yugabyted destroy -``` diff --git a/docs/content/preview/explore/json-support/jsonb-ysql.md b/docs/content/preview/explore/json-support/jsonb-ysql.md index 613b7ce1b84c..3cc25c3793d4 100644 --- a/docs/content/preview/explore/json-support/jsonb-ysql.md +++ b/docs/content/preview/explore/json-support/jsonb-ysql.md @@ -1,9 +1,9 @@ --- -title: JSON Support -headerTitle: JSON Support +title: JSON support +headerTitle: JSON support linkTitle: JSON support description: JSON Support in YugabyteDB. -headcontent: JSON Support in YugabyteDB. +headcontent: Explore YugabyteDB support for JSON data. image:
menu: preview: @@ -14,8 +14,6 @@ menu: type: docs --- -JSON data types are for storing JSON (JavaScript Object Notation) data, as specified in [RFC 7159](https://tools.ietf.org/html/rfc7159). Such data can also be stored as `text`, but the JSON data types have the advantage of enforcing that each stored value is valid according to the JSON rules. There are also assorted JSON-specific functions and operators available for data stored in these data types. -
  • @@ -34,53 +32,41 @@ JSON data types are for storing JSON (JavaScript Object Notation) data, as speci
-See the section [JSON data types and functionality](../../../api/ysql/datatypes/type_json/) for the YSQL reference documentation on the topic. +JSON data types are for storing JSON (JavaScript Object Notation) data, as specified in [RFC 7159](https://tools.ietf.org/html/rfc7159). Such data can also be stored as `text`, but the JSON data types have the advantage of enforcing that each stored value is valid according to the JSON rules. Assorted JSON-specific functions and operators are also available for data stored in these data types. -{{< note title="Note" >}} -The JSON functionality in YSQL is nearly identical to the [JSON functionality in PostgreSQL](https://www.postgresql.org/docs/11/datatype-json.html). -{{< /note >}} +{{% explore-setup-single %}} -There are two JSON data types supported in YSQL: `json` and `jsonb`. +JSON functionality in YSQL is nearly identical to the [JSON functionality in PostgreSQL](https://www.postgresql.org/docs/11/datatype-json.html). -* **The `jsonb` type** does not preserve white space, does not preserve the order of object keys, and does not keep duplicate object keys. If duplicate keys are specified in the input, only the last value is kept. +YSQL supports the following two JSON data types: -* **The `json` type** stores an exact copy of the input text, and therefore preserves semantically-insignificant white space between tokens, as well as the order of keys within JSON objects. Also, if a JSON object within the value contains the same key more than once, all the key/value pairs are kept. The processing functions consider the last value as the operative one. +* **jsonb** - does not preserve white space, does not preserve the order of object keys, and does not keep duplicate object keys. If duplicate keys are specified in the input, only the last value is kept. +* **json** - stores an exact copy of the input text, and therefore preserves semantically-insignificant white space between tokens, as well as the order of keys in JSON objects. Also, if a JSON object in the value contains the same key more than once, all the key/value pairs are kept. The processing functions consider the last value as the operative one. -{{< tip title="Tip" >}} -**When to use `jsonb` vs `json`?** In general, most applications should prefer to store JSON data as `jsonb`, unless there are quite specialized needs, such as legacy assumptions about ordering of object keys. +{{< tip title="When to use jsonb or json" >}} +In general, most applications should prefer to store JSON data as jsonb, unless there are quite specialized needs, such as legacy assumptions about ordering of object keys. They accept *almost* identical sets of values as input. The major practical difference is one of efficiency: -* The `json` data type stores an exact copy of the input text, which processing functions must reparse on each execution -* The `jsonb` data is stored in a decomposed binary format that makes it slightly slower to input due to added conversion overhead, but significantly faster to process, since no reparsing is needed. `jsonb` also supports indexing, which can be a significant advantage. -{{< /tip >}} - -This section will focus on only the `jsonb` type. - -## 1. Prerequisites - -You need a YugabyteDB cluster to run through the steps below. If do not have a YugabyteDB cluster, you can create one on your local machine as shown below. -```sh -$ ./bin/yugabyted start -``` +* json stores an exact copy of the input text, which processing functions must re-parse on each execution +* jsonb data is stored in a decomposed binary format that makes it slightly slower to input due to added conversion overhead, but significantly faster to process, because no re-parsing is needed. jsonb also supports indexing, which can be a significant advantage. +{{< /tip >}} -Connect to the cluster using `ysqlsh` to run through the examples below. +This section focuses on only the jsonb type. -```sh -$ ./bin/ysqlsh -``` +## Create a table -Next, create a simple table `books` which has a primary key and one `jsonb` column `doc` which contains various details about that book. +Create a basic table `books` with a primary key and one `jsonb` column `doc` that contains various details about each book. ```plpgsql -create table books(k int primary key, doc jsonb not null); +yugabyte=# CREATE TABLE books(k int primary key, doc jsonb not null); ``` Next, insert some rows which contain details about various books. These details are represented as JSON documents, as shown below. ```plpgsql -insert into books(k, doc) values +yugabyte=# INSERT INTO books(k, doc) values (1, '{ "ISBN" : 4582546494267, "title" : "Macbeth", @@ -131,18 +117,17 @@ insert into books(k, doc) values Some of the rows in the example have some of the keys missing (intentional). But the row with "k=6" has every key. {{< /note >}} -## 2. Query JSON documents +## Query JSON documents List all the rows thus: ```plpgsql -select * from books; +yugabyte=# SELECT * FROM books; ``` This is the result: -``` -yugabyte=# select * from books; +```output k | doc ---+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 5 | {"ISBN": 8647295405123, "year": 1988, "genre": "science", "title": "A Brief History of Time", "author": {"given_name": "Stephen", "family_name": "Hawking"}, "editors": ["Melisa", "Mark", "John", "Fred", "Jane"]} @@ -156,23 +141,20 @@ yugabyte=# select * from books; ### Using `->` and `->>` -YSQL has two native operators, the `->` operator (see [this section](../../../api/ysql/datatypes/type_json/functions-operators/subvalue-operators/#the-160-160-160-160-operator)) and the `->>` operator (see [this section](../../../api/ysql/datatypes/type_json/functions-operators/subvalue-operators/#the-160-160-160-160-and-160-160-160-160-operators)), to query JSON documents. The first operator `->` returns a JSON object, while the operator `->>` returns text. These operators work on both `JSON` as well as `JSONB` columns to select a subset of attributes as well as to inspect the JSON document. - +YSQL has two native operators, the [`->` operator](../../../api/ysql/datatypes/type_json/functions-operators/subvalue-operators/#the-160-160-160-160-operator) and the [`->>` operator](../../../api/ysql/datatypes/type_json/functions-operators/subvalue-operators/#the-160-160-160-160-and-160-160-160-160-operators), to query JSON documents. The `->` operator returns a JSON object, while the `->>` operator returns text. These operators work on both `JSON` as well as `JSONB` columns to select a subset of attributes as well as to inspect the JSON document. -The example below shows how to select a few attributes from each document. +The following example shows how to select a few attributes from each document. ```plpgsql -SELECT doc->'title' AS book_title, - CONCAT(doc->'author'->'family_name', +yugabyte=# SELECT doc->'title' AS book_title, + CONCAT(doc->'author'->'family_name', ', ', doc->'author'->'given_name') AS author - FROM books; + FROM books; ``` + This is the result: -``` -yugabyte=# SELECT doc->'title' AS book_title, -yugabyte-# CONCAT(doc->'author'->'family_name', -yugabyte(# ', ', doc->'author'->'given_name') AS author -yugabyte-# FROM books; + +```output book_title | author ---------------------------+-------------------------- "A Brief History of Time" | "Hawking", "Stephen" @@ -184,18 +166,16 @@ yugabyte-# FROM books; (6 rows) ``` -Because the -> operator returns an object, you can chain it to inspect deep into a JSON document, as shown below. +Because the `->` operator returns an object, you can chain it to inspect deep into a JSON document, as follows: ```plpgsql -select '{"title": "Macbeth", "author": {"given_name": "William"}}'::jsonb - -> 'author' -> 'given_name' as first_name; +yugabyte=# SELECT '{"title": "Macbeth", "author": {"given_name": "William"}}'::jsonb + -> 'author' -> 'given_name' as first_name; ``` This is the result: -``` -yugabyte=# select '{"title": "Macbeth", "author": {"given_name": "William"}}'::jsonb - -> 'author' -> 'given_name' as first_name; +```output first_name ------------ "William" @@ -204,20 +184,17 @@ yugabyte=# select '{"title": "Macbeth", "author": {"given_name": "William"}}'::j ### Existence with `?` -The `?` operator (see [this section](../../../api/ysql/datatypes/type_json/functions-operators/key-or-value-existence-operators/#the-160-160-160-160-operator)) can be used to check if a JSON document contains a certain attribute. For example, if you want to find a count of the records where the `doc` column contains a property named *genre*, run the following statement. +The [`?` operator](../../../api/ysql/datatypes/type_json/functions-operators/key-or-value-existence-operators/#the-160-160-160-160-operator) can be used to check if a JSON document contains a certain attribute. For example, if you want to find a count of the records where the `doc` column contains a property named *genre*, run the following statement: ```plpgsql -SELECT doc->'title' AS book_title, - doc->'genre' as genre - FROM books WHERE doc ? 'genre'; +yugabyte=# SELECT doc->'title' AS book_title, + doc->'genre' AS genre + FROM books WHERE doc ? 'genre'; ``` This is the result: -``` -yugabyte=# SELECT doc->'title' AS book_title, -yugabyte-# doc->'genre' as genre -yugabyte-# FROM books WHERE doc ? 'genre'; +```output book_title | genre ---------------------------+----------- "A Brief History of Time" | "science" @@ -229,24 +206,19 @@ yugabyte-# FROM books WHERE doc ? 'genre'; ### Containment with `@>` -The containment operator `@>` (see [this section](../../../api/ysql/datatypes/type_json/functions-operators/containment-operators/)) tests whether one document contains another. If you want to find all books that contain the JSON value `{"author": {"given_name": "William"}}` (in other words, the author of the book has the given name *William*), do this: +The [containment operator `@>`](../../../api/ysql/datatypes/type_json/functions-operators/containment-operators/) tests whether one document contains another. If you want to find all books that contain the JSON value `{"author": {"given_name": "William"}}` (in other words, the author of the book has the given name *William*), do the following: ```plpgsql -SELECT doc->'title' AS book_title, - CONCAT(doc->'author'->'family_name', +yugabyte=# SELECT doc->'title' AS book_title, + CONCAT(doc->'author'->'family_name', ', ', doc->'author'->'given_name') AS author - FROM books - WHERE doc @> '{"author": {"given_name": "William"}}'::jsonb; + FROM books + WHERE doc @> '{"author": {"given_name": "William"}}'::jsonb; ``` This is the result: -``` -yugabyte=# SELECT doc->'title' AS book_title, -yugabyte-# CONCAT(doc->'author'->'family_name', -yugabyte(# ', ', doc->'author'->'given_name') AS author -yugabyte-# FROM books -yugabyte-# WHERE doc @> '{"author": {"given_name": "William"}}'::jsonb; +```output book_title | author ------------+-------------------------- "Macbeth" | "Shakespeare", "William" @@ -254,23 +226,25 @@ yugabyte-# WHERE doc @> '{"author": {"given_name": "William"}}'::jsonb; (2 rows) ``` +## Update JSON documents -## 3. Update JSON documents - -There are a number of ways to update a JSON document, as shown below. +You can update a JSON document in a number of ways, as shown in the following examples. ### Add an attribute -Use the `||` operator (see [this section](../../../api/ysql/datatypes/type_json/functions-operators/concatenation-operator/)) to either update or insert the attribute into the existing JSON document. For example, if you want to add a `stock` attribute to all the books, do this: +Use the [`||` operator](../../../api/ysql/datatypes/type_json/functions-operators/concatenation-operator/) to either update or insert the attribute into the existing JSON document. For example, if you want to add a `stock` attribute to all the books, do the following: ```plpgsql -UPDATE books SET doc = doc || '{"stock": "true"}'; +yugabyte=# UPDATE books SET doc = doc || '{"stock": "true"}'; ``` This is the result: +```sql +yugabyte=# SELECT doc->'title' AS title, doc->'stock' AS stock FROM books; ``` -yugabyte=# SELECT doc->'title' AS title, doc->'stock' as stock FROM books; + +```output title | stock ---------------------------+-------- "A Brief History of Time" | "true" @@ -284,16 +258,19 @@ yugabyte=# SELECT doc->'title' AS title, doc->'stock' as stock FROM books; ### Remove an attribute -Use the `-` opertaor (see [this section](../../../api/ysql/datatypes/type_json/functions-operators/remove-operators/#the-160-160-160-160-operator)) to remove an attribute: +Use the [`-` operator](../../../api/ysql/datatypes/type_json/functions-operators/remove-operators/#the-160-160-160-160-operator) to remove an attribute: ```plpgsql -UPDATE books SET doc = doc - 'stock'; +yugabyte=# UPDATE books SET doc = doc - 'stock'; ``` -This will remove the field from all the documents, as shown below. +This removes the field from all the documents, as shown below. +```sql +yugabyte=# SELECT doc->'title' AS title, doc->'stock' AS stock FROM books; ``` -yugabyte=# SELECT doc->'title' AS title, doc->'stock' as stock FROM books; + +```output title | stock ---------------------------+------- "A Brief History of Time" | @@ -307,7 +284,7 @@ yugabyte=# SELECT doc->'title' AS title, doc->'stock' as stock FROM books; ### Replace a document -To replace an entire document, run the following SQL statement. +To replace an entire document, run the following SQL statement: ```plpgsql UPDATE books @@ -315,30 +292,23 @@ UPDATE books WHERE k=1; ``` -## 4. Built-in functions +## Built-in functions YSQL supports a large number of operators and built-in functions that operate on JSON documents. This section highlights a few of these built-in functions. -{{< note title="Note" >}} -All of the built-in functions supported by PostgreSQL are supported in YSQL. - -Check the reference documentation for a complete [list of JSON functions and operators](../../../api/ysql/datatypes/type_json/functions-operators/). -{{< /note >}} - +YSQL supports all of the built-in functions supported by PostgreSQL. For a complete list, refer to [JSON functions and operators](../../../api/ysql/datatypes/type_json/functions-operators/). ### Expand JSON - `jsonb_each` The [`jsonb_each()`](../../../api/ysql/datatypes/type_json/functions-operators/jsonb-each/) function expands the top-level JSON document into a set of key-value pairs, as shown below. - ```plpgsql -SELECT jsonb_each(doc) FROM books WHERE k=1; +yugabyte=# SELECT jsonb_each(doc) FROM books WHERE k=1; ``` The output is shown below. -``` -yugabyte=# SELECT jsonb_each(doc) FROM books WHERE k=1; +```output jsonb_each ---------------------------------------------------------------------------- (ISBN,4582546494267) @@ -353,13 +323,12 @@ yugabyte=# SELECT jsonb_each(doc) FROM books WHERE k=1; The [`jsonb_object_keys()`](../../../api/ysql/datatypes/type_json/functions-operators/jsonb-object-keys/) function retrieves the keys of the top-level JSON document thus: ```plpgsql -SELECT jsonb_object_keys(doc) FROM books WHERE k=1; +yugabyte=# SELECT jsonb_object_keys(doc) FROM books WHERE k=1; ``` This is the result: -``` -yugabyte=# SELECT jsonb_object_keys(doc) FROM books WHERE k=1; +```output jsonb_object_keys ------------------- ISBN @@ -374,13 +343,12 @@ yugabyte=# SELECT jsonb_object_keys(doc) FROM books WHERE k=1; When you select a `jsonb` (or `json`) value in `ysqlsh`, you see the terse `text` typecast of the value. The [`jsonb_pretty()`](../../../api/ysql/datatypes/type_json/functions-operators/jsonb-pretty/) function returns a more human-readable format: ```plpgsql -SELECT jsonb_pretty(doc) FROM books WHERE k=1; +yugabyte=# SELECT jsonb_pretty(doc) FROM books WHERE k=1; ``` This is the result: -``` -yugabyte=# SELECT jsonb_pretty(doc) FROM books WHERE k=1; +```output jsonb_pretty -------------------------------------- { + @@ -395,11 +363,9 @@ yugabyte=# SELECT jsonb_pretty(doc) FROM books WHERE k=1; (1 row) ``` -## 5. Constraints - -See the [Create indexes and check constraints on JSON columns](../../../api/ysql/datatypes/type_json/create-indexes-check-constraints/) section. +## Constraints -You can create constraint on `jsonb` data types. Here are a couple of examples. +You can create constraints on `jsonb` data types. This section includes some examples. For a fuller discussion, refer to [Create indexes and check constraints on JSON columns](../../../api/ysql/datatypes/type_json/create-indexes-check-constraints/). ### Check JSON documents are objects @@ -429,15 +395,13 @@ check ( ); ``` - -## 6. Indexes on JSON attributes +## Indexes on JSON attributes Indexes are essential to perform efficient lookups by document attributes. Without indexes, queries on document attributes end up performing a full table scan and process each JSON document. This section outlines some of the indexes supported. - ### Secondary index -If you want to support range queries that reference the value for the *year* attribute, do this: +If you want to support range queries that reference the value for the *year* attribute, do the following: ```plpgsql CREATE INDEX books_year @@ -462,7 +426,7 @@ order by 3; You might want to index only those documents that contain the attribute (as opposed to indexing the rows that have a `NULL` value for that attribute). This is a common scenario because not all the documents would have all the attributes defined. This can be achieved using a *partial index*. -In the previous section where you created a secondary index, not all the books may have the `year` attribute defined. Suppose that you want to index only those documents that have a `NOT NULL` `year` attribute create the following partial index: +In the previous section where you created a secondary index, not all the books may have the `year` attribute defined. Suppose that you want to index only those documents that have a `NOT NULL` `year` attribute. Create the following partial index: ```plpgsql CREATE INDEX books_year @@ -472,26 +436,26 @@ CREATE INDEX books_year ### Unique index -You can create a unique index on the "ISBN" key for the books table as shown below. +You can create a unique index on the "ISBN" key for the books table as follows: ```plpgsql -create unique index books_isbn_unq on books((doc->>'ISBN')); +CREATE UNIQUE INDEX books_isbn_unq on books((doc->>'ISBN')); ``` Inserting a row with a duplicate value would fail as shown below. The book has a new primary key `k` but an existing ISBN, `4582546494267`. ```plpgsql -yugabyte=# insert into books values +yugabyte=# INSERT INTO books values (7, '{ "ISBN" : 4582546494267, "title" : "Fake Book with duplicate ISBN" }'); -ERROR: 23505: duplicate key value violates unique constraint "books_isbn_unq" ``` +```output +ERROR: 23505: duplicate key value violates unique constraint "books_isbn_unq" +``` -## 7. Clean up (optional) - -Optionally, you can shut down the local cluster you created earlier. +## Read more -```sh -$ ./bin/yugabyted destroy -``` +* [JSON data types and functionality](../../../api/ysql/datatypes/type_json/) reference +* [JSON functions and operators](../../../api/ysql/datatypes/type_json/functions-operators/) +* [Create indexes and check constraints on JSON columns](../../../api/ysql/datatypes/type_json/create-indexes-check-constraints/) diff --git a/docs/content/preview/explore/linear-scalability/scaling-transactions.md b/docs/content/preview/explore/linear-scalability/scaling-transactions.md index c0fcb2a1828c..7eeafd6d2df2 100644 --- a/docs/content/preview/explore/linear-scalability/scaling-transactions.md +++ b/docs/content/preview/explore/linear-scalability/scaling-transactions.md @@ -83,7 +83,7 @@ Download the YugabyteDB workload generator JAR file (`yb-sample-apps.jar`). $ wget https://github.com/yugabyte/yb-sample-apps/releases/download/1.3.9/yb-sample-apps.jar?raw=true -O yb-sample-apps.jar ``` -Run the `SqlInserts` workload app against the local universe using the following command. +Run the `SqlInserts` workload app against the local cluster using the following command. ```sh $ java -jar ./yb-sample-apps.jar --workload SqlInserts \ @@ -107,7 +107,7 @@ You can check a lot of the per-node stats by browsing to the [tablet-servers](ht ## Add node and observe linear scaling -Add a node to the universe with the same flags. +Add a node to the cluster with the same flags. ```sh $ ./bin/yugabyted start \ @@ -120,7 +120,7 @@ $ ./bin/yugabyted start \ Now you should have 4 nodes. Refresh the [tablet-servers](http://127.0.0.1:7000/tablet-servers) page to see the statistics update. Shortly, you should see the new node performing a comparable number of reads and writes as the other nodes. The 36 tablets will now get distributed evenly across all the 4 nodes, leading to each node having 9 tablets. -The YugabyteDB universe automatically lets the client know to use the newly added node for serving queries. This scaling out of client queries is completely transparent to the application logic, allowing the application to scale linearly for both reads and writes. +The cluster automatically lets the client know to use the newly added node for serving queries. This scaling out of client queries is completely transparent to the application logic, allowing the application to scale linearly for both reads and writes. ![Read and write IOPS with 4 nodes - Rebalancing in progress](/images/ce/transactions_newnode_adding_observe.png) @@ -128,7 +128,7 @@ The YugabyteDB universe automatically lets the client know to use the newly adde ## Remove node and observe linear scale in -Remove the recently added node from the universe. +Remove the recently added node from the cluster as follows: ```sh $ ./bin/yugabyted stop \ diff --git a/docs/content/preview/explore/linear-scalability/sharding-data.md b/docs/content/preview/explore/linear-scalability/sharding-data.md index 9623d1fc8610..d1eff060551e 100644 --- a/docs/content/preview/explore/linear-scalability/sharding-data.md +++ b/docs/content/preview/explore/linear-scalability/sharding-data.md @@ -136,9 +136,9 @@ In this tutorial, you'll explore automatic sharding inside YugabyteDB. First, yo This tutorial uses the [yugabyted](../../../reference/configuration/yugabyted/) cluster management utility. -### Create a universe +### Create a cluster -To create a universe, do the following: +To create a cluster, do the following: 1. Create a single node cluster by running the following command: @@ -321,7 +321,7 @@ Here, the key has been written to one of the tablets. In this example, the table ### Automatic sharding when adding nodes -1. Add one more node to the universe for a total of 4 nodes: +1. Add one more node to the cluster for a total of 4 nodes: ```sh $ ./bin/yugabyted start \ @@ -335,7 +335,7 @@ Here, the key has been written to one of the tablets. In this example, the table ![Auto-sharding when adding one node](/images/ce/sharding_4nodes.png) -1. Add 2 more nodes to the universe, making it a total of 6 nodes: +1. Add 2 more nodes to the cluster, making it a total of 6 nodes: ```sh $ ./bin/yugabyted start \ diff --git a/docs/content/preview/explore/multi-region-deployments/read-replicas-ycql.md b/docs/content/preview/explore/multi-region-deployments/read-replicas-ycql.md index 4c28e03e1cc0..ac2ae0033350 100644 --- a/docs/content/preview/explore/multi-region-deployments/read-replicas-ycql.md +++ b/docs/content/preview/explore/multi-region-deployments/read-replicas-ycql.md @@ -38,13 +38,13 @@ YugabyteDB supports the following types of reads: - [Follower reads](../../ysql-language-features/going-beyond-sql/follower-reads-ycql/) that enable spreading the read workload across all replicas in the primary cluster. - Observer reads that use read replicas. The latter obtain their data via [xCluster replication](../asynchronous-replication-ycql/) which allows for the read workload to be offloaded from the primary cluster. Read replicas are created as a separate cluster that may be located in a different region, possibly closer to the consumers of the data which would result in lower-latency access and enhanced support of analytics workloads. -A datacenter (also known as universe) can have one primary cluster and several read replica clusters. +A data center (also known as a [universe](../../../architecture/concepts/universe/#universe-vs-cluster)) can have one primary cluster and several read replica clusters. -Stale reads are possible with an upper bound on the amount of staleness. Reads are guaranteed to be timeline-consistent. You need to set the consistency level to `ONE` in your application to work with follower reads or observer reads. In addition, you have to set the application's local datacenter to the read replica cluster's region. +Stale reads are possible with an upper bound on the amount of staleness. Reads are guaranteed to be timeline-consistent. You need to set the consistency level to `ONE` in your application to work with follower reads or observer reads. In addition, you have to set the application's local data center to the read replica cluster's region. ## Prerequisites -Ensure that you have downloaded and configured YugabyteDB, as described in [Quick start](/preview/quick-start/). +Ensure that you have downloaded and configured YugabyteDB, as described in [Quick start](../../../quick-start/). {{< note title="Note" >}} @@ -84,9 +84,9 @@ The following command instructs the masters to create three replicas for each ta $ ./bin/yb-admin -master_addresses 127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100 modify_placement_info c.r.z1,c.r.z2,c.r.z3 3 live ``` -The following illustration demonstrates the primary cluster visible via [YugabyteDB Anywhere](/preview/yugabyte-platform/): +The following illustration demonstrates the primary cluster visible via [YugabyteDB Anywhere](../../../yugabyte-platform/): -![img](/images/explore/multi-region-deployments/read-replicas1.png) +![Primary cluster](/images/explore/multi-region-deployments/read-replicas1.png) The following command runs the sample application and starts a YCQL workload: @@ -127,13 +127,13 @@ Output: 11166 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker - Read: 5066.60 ops/sec (0.20 ms/op), 45479 total ops | Write: 1731.19 ops/sec (0.58 ms/op), 16918 total ops | Uptime: 10026 ms | maxWrittenKey: 1 | maxGeneratedKey: 2 | ``` -The following illustration demonstrates the read and write statistics in the primary cluster visible via [YugabyteDB Anywhere](/preview/yugabyte-platform/): +The following illustration demonstrates the read and write statistics in the primary cluster visible via YugabyteDB Anywhere: -![img](/images/explore/multi-region-deployments/read-replicas2.png) +![Primary cluster reads and writes](/images/explore/multi-region-deployments/read-replicas2.png) As per the preceding illustration, using the default workload directs reads and writes to the tablet leader. The arguments in the `java -jar ./yb-sample-apps.jar` command explicitly restrict the number of keys to be written and read to one in order to follow the reads and writes occurring on a single tablet. -The following is a modified command that enables follower reads. Specifying `--local_reads` changes the consistency level to `ONE`. The `--with_local_dc` option defines in which datacenter the application is at any given time. When specified, the read traffic is routed to the same region: +The following is a modified command that enables follower reads. Specifying `--local_reads` changes the consistency level to `ONE`. The `--with_local_dc` option defines in which data center the application is at any given time. When specified, the read traffic is routed to the same region: ```shell $ java -jar ./yb-sample-apps.jar --workload CassandraKeyValue \ @@ -172,11 +172,11 @@ Output: 10778 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker - Read: 4801.54 ops/sec (0.21 ms/op), 44256 total ops | Write: 1637.30 ops/sec (0.61 ms/op), 15635 total ops | Uptime: 10026 ms | maxWrittenKey: 1 | maxGeneratedKey: 2 | ``` -The following illustration demonstrates the reads spread across all the replicas for the tablet visible via [YugabyteDB Anywhere](/preview/yugabyte-platform/): +The following illustration demonstrates the reads spread across all the replicas for the tablet visible via YugabyteDB Anywhere: -![img](/images/explore/multi-region-deployments/read-replicas3.png) +![Read replica reads](/images/explore/multi-region-deployments/read-replicas3.png) -## Using Read Replicas +## Using read replicas The following commands add three new nodes to a read replica cluster in region `r2`: @@ -188,9 +188,9 @@ The following commands add three new nodes to a read replica cluster in region ` ./bin/yb-admin -master_addresses 127.0.0.1:7100,127.0.0.2,127.0.0.3 add_read_replica_placement_info c.r2.z21:1,c.r2.z22:1,c.r2.z23:1 3 rr ``` -The following illustration demonstrates the setup of two clusters, one of which is primary and another one is read replica visible via [YugabyteDB Anywhere](/preview/yugabyte-platform/): +The following illustration demonstrates the setup of the two clusters, one of which is primary and another one is read replica visible via YugabyteDB Anywhere: -![img](/images/explore/multi-region-deployments/read-replicas4.png) +![Primary and read replica](/images/explore/multi-region-deployments/read-replicas4.png) The following command directs `CL.ONE` reads to the primary cluster (as follower reads) in region `r`: @@ -204,9 +204,9 @@ java -jar ./yb-sample-apps.jar --workload CassandraKeyValue \ --value_size 1024 --local_reads --with_local_dc r ``` -The following illustration demonstrates the result of executing the preceding command (visible via [YugabyteDB Anywhere](/preview/yugabyte-platform/):): +The following illustration demonstrates the result of executing the preceding command (visible via YugabyteDB Anywhere): -![img](/images/explore/multi-region-deployments/read-replicas5.png) +![Primary cluster reads](/images/explore/multi-region-deployments/read-replicas5.png) The following command directs the `CL.ONE` reads to the read replica cluster (as observer reads) in region `r2`: @@ -220,13 +220,13 @@ java -jar ./yb-sample-apps.jar --workload CassandraKeyValue \ --value_size 1024 --local_reads --with_local_dc r2 ``` -The following illustration demonstrates the result of executing the preceding command (visible via [YugabyteDB Anywhere](/preview/yugabyte-platform/)): +The following illustration demonstrates the result of executing the preceding command (visible via YugabyteDB Anywhere): -![img](/images/explore/multi-region-deployments/read-replicas6.png) +![Read replica reads](/images/explore/multi-region-deployments/read-replicas6.png) For information on deploying read replicas, see [Read Replica Clusters](../../../deploy/multi-dc/read-replica-clusters/). -## Fault Tolerance +## Fault tolerance In the strong consistency mode (default), more failures can be tolerated by increasing the number of replicas: to tolerate a `k` number of failures, `2k+1` replicas are required in the RAFT group. However, follower reads and observer reads can provide Cassandra-style `CL.ONE` fault tolerance. The `max_stale_read_bound_time_ms` GFlag controls how far behind the followers are allowed to be before they redirect reads back to the RAFT leader (the default is 60 seconds). For "write once, read many times" workloads, this number could be increased. By stopping nodes, you can induce behavior of follower and observer reads such that they continue to read (which would not be possible without follower reads). @@ -248,9 +248,9 @@ The following command stops a node in the read replica cluster: ./bin/yb-ctl stop_node 6 ``` -The following illustration demonstrates the stopped node visible via [YugabyteDB Anywhere](/preview/yugabyte-platform/): +The following illustration demonstrates the stopped node visible via YugabyteDB Anywhere: -![img](/images/explore/multi-region-deployments/read-replicas7.png) +![Read replica with stopped node](/images/explore/multi-region-deployments/read-replicas7.png) Stopping one node redistributes the load onto the two remaining nodes. @@ -260,9 +260,9 @@ The following command stops another node in the read replica cluster: ./bin/yb-ctl stop_node 5 ``` -The following illustration demonstrates two stopped nodes visible via [YugabyteDB Anywhere](/preview/yugabyte-platform/): +The following illustration demonstrates two stopped nodes visible via YugabyteDB Anywhere: -![img](/images/explore/multi-region-deployments/read-replicas8.png) +![Read replica with two stopped nodes](/images/explore/multi-region-deployments/read-replicas8.png) The following command stops the last node in the read replica cluster which causes the reads revert back to the primary cluster and become follower reads: @@ -270,11 +270,11 @@ The following command stops the last node in the read replica cluster which caus ./bin/yb-ctl stop_node 4 ``` -The following illustration demonstrates all stopped nodes in the read replica cluster and activation of nodes in the primary cluster visible via [YugabyteDB Anywhere](/preview/yugabyte-platform/): +The following illustration demonstrates all stopped nodes in the read replica cluster and activation of nodes in the primary cluster visible via YugabyteDB Anywhere: -![img](/images/explore/multi-region-deployments/read-replicas9.png) +![Read replica with three stopped nodes](/images/explore/multi-region-deployments/read-replicas9.png) -This behavior differs from the standard Cassandra behavior. The YCQL interface only honors consistency level `ONE`. All other consistency levels are converted to `QUORUM` including `LOCAL_ONE`. When a local datacenter is specified by the application with consistency level `ONE`, read traffic is localized to the region as long as this region has active replicas. If the application’s local datacenter has no replicas, the read traffic is routed to the primary region. +This behavior differs from the standard Cassandra behavior. The YCQL interface only honors consistency level `ONE`. All other consistency levels are converted to `QUORUM` including `LOCAL_ONE`. When a local data center is specified by the application with consistency level `ONE`, read traffic is localized to the region as long as this region has active replicas. If the application's local data center has no replicas, the read traffic is routed to the primary region. The following command stops one of the nodes in the primary cluster: @@ -282,9 +282,9 @@ The following command stops one of the nodes in the primary cluster: ./bin/yb-ctl stop_node 3 ``` -The following illustration demonstrates the state of nodes, with read load rebalanced to the remaining nodes visible via [YugabyteDB Anywhere](/preview/yugabyte-platform/): +The following illustration demonstrates the state of nodes, with read load rebalanced to the remaining nodes visible via YugabyteDB Anywhere: -![img](/images/explore/multi-region-deployments/read-replicas10.png) +![Primary cluster with one stopped node](/images/explore/multi-region-deployments/read-replicas10.png) The following command stops one more node in the primary cluster: @@ -292,8 +292,8 @@ The following command stops one more node in the primary cluster: ./bin/yb-ctl stop_node 2 ``` -The following illustration demonstrates that the entire read load moved to the one remaining node visible via [YugabyteDB Anywhere](/preview/yugabyte-platform/): +The following illustration demonstrates that the entire read load moved to the one remaining node visible via YugabyteDB Anywhere: -![img](/images/explore/multi-region-deployments/read-replicas11.png) +![Primary cluster with two stopped nodes](/images/explore/multi-region-deployments/read-replicas11.png) For additional information, see [Fault Tolerance](../../fault-tolerance/macos/). diff --git a/docs/content/preview/explore/multi-region-deployments/synchronous-replication-ycql.md b/docs/content/preview/explore/multi-region-deployments/synchronous-replication-ycql.md index 691921991bc0..750755f7f65d 100644 --- a/docs/content/preview/explore/multi-region-deployments/synchronous-replication-ycql.md +++ b/docs/content/preview/explore/multi-region-deployments/synchronous-replication-ycql.md @@ -36,19 +36,19 @@ type: docs YugabyteDB can be deployed in a globally distributed manner to serve application queries from the region closest to the end users with low latencies as well as to survive any outages to ensure high availability. -This page simulates AWS regions on a local machine. First, you deploy YugabyteDB in the `us-west-2` region across multiple availability zones (`a`, `b`, `c`) and start a key-value workload against this universe. Next, you change this setup to run across multiple geographic regions in US East (`us-east-1`) and Tokyo (`ap-northeast-1`), with the workload running uninterrupted during the entire transition. +This page simulates AWS regions on a local machine. First, you deploy YugabyteDB in the `us-west-2` region across multiple availability zones (`a`, `b`, `c`) and start a key-value workload against this cluster. Next, you change this setup to run across multiple geographic regions in US East (`us-east-1`) and Tokyo (`ap-northeast-1`), with the workload running uninterrupted during the entire transition. This page uses the [yb-ctl](../../../admin/yb-ctl/) local cluster management utility. -## Create a multi-zone universe in US West +## Create a multi-zone cluster in US West -If you have a previously running local universe, destroy it using the following command: +If you have a previously running local cluster, destroy it using the following command: ```sh ./bin/yb-ctl destroy ``` -Start a new local universe with a replication factor (RF) of `3`, and each replica placed in different zones (`us-west-2a`, `us-west-2b`, `us-west-2c`) in the `us-west-2` (Oregon) region of AWS. This can be done by running the following command: +Start a new local cluster with a replication factor (RF) of `3`, and each replica placed in different zones (`us-west-2a`, `us-west-2b`, `us-west-2c`) in the `us-west-2` (Oregon) region of AWS. This can be done by running the following command: ```sh ./bin/yb-ctl --rf 3 create --placement_info "aws.us-west-2.us-west-2a,aws.us-west-2.us-west-2b,aws.us-west-2.us-west-2c" @@ -56,11 +56,11 @@ Start a new local universe with a replication factor (RF) of `3`, and each repli In this deployment, the YB-Masters are each placed in a separate zone to allow them to survive the loss of a zone. You can view the masters on the [dashboard](http://localhost:7000/), as per the following illustration: -![Multi-zone universe masters](/images/ce/online-reconfig-multi-zone-masters.png) +![Multi-zone cluster masters](/images/ce/online-reconfig-multi-zone-masters.png) You can view the tablet servers on the [tablet servers page](http://localhost:7000/tablet-servers), as per the following illustration: -![Multi-zone universe tservers](/images/ce/online-reconfig-multi-zone-tservers.png) +![Multi-zone cluster tservers](/images/ce/online-reconfig-multi-zone-tservers.png) ## Start a workload @@ -81,7 +81,7 @@ java -jar ./yb-sample-apps.jar --workload CassandraKeyValue \ You should now see some read and write load on the [tablet servers page](http://localhost:7000/tablet-servers), as per the following illustration: -![Multi-zone universe load](/images/ce/online-reconfig-multi-zone-load.png) +![Multi-zone cluster load](/images/ce/online-reconfig-multi-zone-load.png) ## Add nodes diff --git a/docs/content/preview/explore/multi-region-deployments/synchronous-replication-ysql.md b/docs/content/preview/explore/multi-region-deployments/synchronous-replication-ysql.md index 2e9783497cc7..cdbdd9a98785 100644 --- a/docs/content/preview/explore/multi-region-deployments/synchronous-replication-ysql.md +++ b/docs/content/preview/explore/multi-region-deployments/synchronous-replication-ysql.md @@ -36,19 +36,19 @@ type: docs YugabyteDB can be deployed in a globally distributed manner to serve application queries from the region closest to the end users with low latencies as well as to survive any outages to ensure high availability. -This page simulates AWS regions on a local machine. First, you deploy YugabyteDB in the `us-west-2` region across multiple availability zones (`a`, `b`, `c`) and start a key-value workload against this universe. Next, you change this setup to run across multiple geographic regions in US East (`us-east-1`) and Tokyo (`ap-northeast-1`), with the workload running uninterrupted during the entire transition. +This page simulates AWS regions on a local machine. First, you deploy YugabyteDB in the `us-west-2` region across multiple availability zones (`a`, `b`, `c`) and start a key-value workload against this cluster. Next, you change this setup to run across multiple geographic regions in US East (`us-east-1`) and Tokyo (`ap-northeast-1`), with the workload running uninterrupted during the entire transition. This page uses the [yb-ctl](../../../admin/yb-ctl/) local cluster management utility. -## Create a multi-zone universe in US West +## Create a multi-zone cluster in US West -If you have a previously running local universe, destroy it using the following. +If you have a previously running local cluster, destroy it using the following. ```sh ./bin/yb-ctl destroy ``` -Start a new local universe with a replication factor (RF) of `3`, and each replica placed in different zones (`us-west-2a`, `us-west-2b`, `us-west-2c`) in the `us-west-2` (Oregon) region of AWS. This can be done by running the following: +Start a new local cluster with a replication factor (RF) of `3`, and each replica placed in different zones (`us-west-2a`, `us-west-2b`, `us-west-2c`) in the `us-west-2` (Oregon) region of AWS. This can be done by running the following: ```sh ./bin/yb-ctl --rf 3 create --placement_info "aws.us-west-2.us-west-2a,aws.us-west-2.us-west-2b,aws.us-west-2.us-west-2c" @@ -56,18 +56,18 @@ Start a new local universe with a replication factor (RF) of `3`, and each repli In this deployment, the YB-Masters are each placed in a separate zone to allow them to survive the loss of a zone. You can view the masters on the [dashboard](http://localhost:7000/), as per the following illustration: -![Multi-zone universe masters](/images/ce/online-reconfig-multi-zone-masters.png) +![Multi-zone cluster YB-Masters](/images/ce/online-reconfig-multi-zone-masters.png) -You can view the tablet servers on the [tablet servers page](http://localhost:7000/tablet-servers), as per the following illustration +You can view the tablet servers on the [tablet servers page](http://localhost:7000/tablet-servers), as per the following illustration: -![Multi-zone universe tservers](/images/ce/online-reconfig-multi-zone-tservers.png) +![Multi-zone cluster YB-TServers](/images/ce/online-reconfig-multi-zone-tservers.png) ## Start a workload Download the [YugabyteDB workload generator](https://github.com/yugabyte/yb-sample-apps) JAR file (`yb-sample-apps.jar`) by running the following command: ```sh -wget https://github.com/yugabyte/yb-sample-apps/releases/download/1.3.9/yb-sample-apps.jar?raw=true -O yb-sample-apps.jar +wget https://github.com/yugabyte/yb-sample-apps/releases/download/1.3.9/yb-sample-apps.jar ``` Run a `SqlInserts` workload in a separate shell, as follows: @@ -81,7 +81,7 @@ java -jar ./yb-sample-apps.jar --workload SqlInserts \ You should now see some read and write load on the [tablet servers page](http://localhost:7000/tablet-servers), as per the following illustration: -![Multi-zone universe load](/images/ce/online-reconfig-multi-zone-load.png) +![Multi-zone cluster load](/images/ce/online-reconfig-multi-zone-load.png) ## Add nodes @@ -99,7 +99,7 @@ Add another node in the zone `ap-northeast-1a` of region `ap-northeast-1`, as fo ./bin/yb-ctl add_node --placement_info "aws.ap-northeast-1.ap-northeast-1a" ``` -These two new nodes are added into the cluster but are not taking any read or write IO. This is because YB-Master's initial placement policy of storing data across the zones in `us-west-2` region still applies, as per the following illustration: +These two new nodes are added into the cluster but are not taking any read or write IO. This is because the YB-Master's initial placement policy of storing data across the zones in `us-west-2` region still applies, as per the following illustration: ![Add node in a new region](/images/ce/online-reconfig-add-regions-no-load.png) @@ -112,7 +112,7 @@ Update the placement policy, instructing the YB-Master to place data in the new modify_placement_info aws.us-west-2.us-west-2a,aws.us-east-1.us-east-1a,aws.ap-northeast-1.ap-northeast-1a 3 ``` -You should see that the data as well as the IO gradually moves from the nodes in `us-west-2b` and `us-west-2c` to the newly added nodes. The [tablet servers page](http://localhost:7000/tablet-servers) should soon look similar the the following illustration: +You should see that the data as well as the IO gradually moves from the nodes in `us-west-2b` and `us-west-2c` to the newly added nodes. The [tablet servers page](http://localhost:7000/tablet-servers) should soon look similar to the following illustration: ![Multi region workload](/images/ce/online-reconfig-multi-region-load.png) @@ -120,7 +120,7 @@ You should see that the data as well as the IO gradually moves from the nodes in ### Start new masters -You need to move the YB-Master from the old nodes to the new nodes. In order to do so, first start a new masters on the new nodes, as follows: +You need to move the YB-Master from the old nodes to the new nodes. To do so, first start new masters on the new nodes, as follows: ```sh ./bin/yb-ctl add_node --master --placement_info "aws.us-east-1.us-east-1a" @@ -134,7 +134,7 @@ You need to move the YB-Master from the old nodes to the new nodes. In order to ### Remove old masters -Remove the old masters from the masters Raft group. Assuming nodes with IPs `127.0.0.2` and `127.0.0.3` were the two old nodes, run the following commands: +Remove the old masters from the masters Raft group. Assuming nodes with addresses `127.0.0.2` and `127.0.0.3` were the two old nodes, run the following commands: ```sh ./bin/yb-admin --master_addresses 127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100,127.0.0.4:7100,127.0.0.5:7100 change_master_config REMOVE_SERVER 127.0.0.2 7100 diff --git a/docs/content/preview/explore/observability/prometheus-integration/docker.md b/docs/content/preview/explore/observability/prometheus-integration/docker.md deleted file mode 100644 index 9137b9214579..000000000000 --- a/docs/content/preview/explore/observability/prometheus-integration/docker.md +++ /dev/null @@ -1,244 +0,0 @@ ---- -title: Prometheus integration -headerTitle: Prometheus integration -linkTitle: Prometheus integration -description: Learn about exporting YugabyteDB metrics and monitoring the cluster with Prometheus. -menu: - preview: - identifier: observability-3-docker - parent: explore-observability - weight: 235 -type: docs ---- - - - -You can monitor your local YugabyteDB cluster with a local instance of [Prometheus](https://prometheus.io/), a popular standard for time-series monitoring of cloud native infrastructure. YugabyteDB services and APIs expose metrics in the Prometheus format at the `/prometheus-metrics` endpoint. For details on the metrics targets for YugabyteDB, see [Prometheus monitoring](../../../../reference/configuration/default-ports/#prometheus-monitoring). - -This tutorial uses the [yugabyted](../../../../reference/configuration/yugabyted/) local cluster management utility. - -## 1. Create universe - -Start a new local universe with replication factor of `3`. - -```sh -$ docker network create -d bridge yb-net -``` - -```sh -$ docker run -d --name yugabyte-node1 \ - --network yb-net \ - -p 127.0.0.1:7000:7000 \ - -p 127.0.0.1:9000:9000 \ - -p 127.0.0.1:5433:5433 \ - -p 127.0.0.1:9042:9042 \ - -p 127.0.0.1:6379:6379 \ - yugabytedb/yugabyte:latest bin/yugabyted start --daemon=false --listen=yugabyte-node1 --tserver_flags="start_redis_proxy=true" -``` - -```sh -$ docker run -d --name yugabyte-node2 \ - --network yb-net \ - -p 127.0.0.2:7000:7000 \ - -p 127.0.0.2:9000:9000 \ - -p 127.0.0.2:5433:5433 \ - -p 127.0.0.2:9042:9042 \ - -p 127.0.0.2:6379:6379 \ - yugabytedb/yugabyte:latest bin/yugabyted start --daemon=false --listen=yugabyte-node2 --join=yugabyte-node1 --tserver_flags="start_redis_proxy=true" -``` - -```sh -$ docker run -d --name yugabyte-node3 \ - --network yb-net \ - -p 127.0.0.3:7000:7000 \ - -p 127.0.0.3:9000:9000 \ - -p 127.0.0.3:5433:5433 \ - -p 127.0.0.3:9042:9042 \ - -p 127.0.0.3:6379:6379 \ - yugabytedb/yugabyte:latest bin/yugabyted start --daemon=false --listen=yugabyte-node3 --join=yugabyte-node1 --tserver_flags="start_redis_proxy=true" -``` - -## 2. Run the YugabyteDB workload generator - -Pull the [yb-sample-apps](https://github.com/yugabyte/yb-sample-apps) Docker container image. This container image has built-in Java client programs for various workloads including SQL inserts and updates. - -```sh -$ docker pull yugabytedb/yb-sample-apps -``` - -Run the `CassandraKeyValue` workload application in a separate shell. - -```sh -$ docker run --name yb-sample-apps --hostname yb-sample-apps --net yb-net yugabytedb/yb-sample-apps \ - --workload CassandraKeyValue \ - --nodes yugabyte-node1:9042 \ - --num_threads_write 1 \ - --num_threads_read 4 -``` - -## 3. Prepare Prometheus configuration file - -Copy the following into a file called `yugabytedb.yml`. Move this file to the `/tmp` directory so that you can bind the file to the Prometheus container later on. - -```yaml -global: - scrape_interval: 5s # Set the scrape interval to every 5 seconds. Default is every 1 minute. - evaluation_interval: 5s # Evaluate rules every 5 seconds. The default is every 1 minute. - # scrape_timeout is set to the global default (10s). - -# YugabyteDB configuration to scrape Prometheus time-series metrics -scrape_configs: - - job_name: "yugabytedb" - metrics_path: /prometheus-metrics - relabel_configs: - - target_label: "node_prefix" - replacement: "cluster-1" - metric_relabel_configs: - # Save the name of the metric so we can group_by since we cannot by __name__ directly... - - source_labels: ["__name__"] - regex: "(.*)" - target_label: "saved_name" - replacement: "$1" - # The following basically retrofit the handler_latency_* metrics to label format. - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(.*)" - target_label: "server_type" - replacement: "$1" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(.*)" - target_label: "service_type" - replacement: "$2" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(_sum|_count)?" - target_label: "service_method" - replacement: "$3" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(_sum|_count)?" - target_label: "__name__" - replacement: "rpc_latency$4" - - static_configs: - - targets: ["yugabyte-node1:7000", "yugabyte-node2:7000", "yugabyte-node3:7000"] - labels: - export_type: "master_export" - - - targets: ["yugabyte-node1:9000", "yugabyte-node2:9000", "yugabyte-node3:9000"] - labels: - export_type: "tserver_export" - - - targets: ["yugabyte-node1:12000", "yugabyte-node2:12000", "yugabyte-node3:12000"] - labels: - export_type: "cql_export" - - - targets: ["yugabyte-node1:13000", "yugabyte-node2:13000", "yugabyte-node3:13000"] - labels: - export_type: "ysql_export" - - - targets: ["yugabyte-node1:11000", "yugabyte-node2:11000", "yugabyte-node3:11000"] - labels: - export_type: "redis_export" -``` - -## 4. Start Prometheus server - -Start the Prometheus server as below. The `prom/prometheus` container image will be pulled from the Docker registry if not already present on the localhost. - -```sh -$ docker run \ - -p 9090:9090 \ - -v /tmp/yugabytedb.yml:/etc/prometheus/prometheus.yml \ - --net yb-net \ - prom/prometheus -``` - -Open the Prometheus UI at and then navigate to the Targets page under Status. - -![Prometheus Targets](/images/ce/prom-targets-docker.png) - -## 5. Analyze key metrics - -On the Prometheus Graph UI, you can now plot the read/write throughput and latency for the `CassandraKeyValue` sample app. As you can see from the [source code](https://github.com/yugabyte/yugabyte-db/blob/master/java/yb-loadtester/src/main/java/com/yugabyte/sample/apps/CassandraKeyValue.java) of the app, it uses only SELECT statements for reads and INSERT statements for writes (aside from the initial CREATE TABLE). This means you can measure throughput and latency by simply using the metrics corresponding to the SELECT and INSERT statements. - -Paste the following expressions into the **Expression** box and click **Execute** followed by **Add Graph**. - -### Throughput - -> Read IOPS - -```sh -sum(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-read-iops.png) - -> Write IOPS - -```sh -sum(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-write-iops.png) - -### Latency - -> Read Latency (in microseconds) - -```sh -avg(irate(rpc_latency_sum{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) / -avg(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-read-latency.png) - -> Write Latency (in microseconds) - -```sh -avg(irate(rpc_latency_sum{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) / -avg(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-write-latency.png) - -## 6. Clean up (optional) - -Optionally, you can shut down the local cluster created in Step 1. - -```sh -$ docker stop yugabyte-node1 yugabyte-node2 yugabyte-node3 -$ docker rm yugabyte-node1 yugabyte-node2 yugabyte-node3 -$ docker network remove yb-net -``` - -## What's next? - -Set up [Grafana dashboards](../../grafana-dashboard/grafana/) for better visualization of the metrics being collected by Prometheus. diff --git a/docs/content/preview/explore/observability/prometheus-integration/kubernetes.md b/docs/content/preview/explore/observability/prometheus-integration/kubernetes.md deleted file mode 100644 index 1bfb61c613d0..000000000000 --- a/docs/content/preview/explore/observability/prometheus-integration/kubernetes.md +++ /dev/null @@ -1,79 +0,0 @@ ---- -title: Prometheus integration -headerTitle: Prometheus integration -linkTitle: Prometheus integration -description: Learn about exporting YugabyteDB metrics and monitoring the cluster with Prometheus. -menu: - preview: - identifier: observability-4-kubernetes - parent: explore-observability - weight: 235 -type: docs ---- - - - -You can monitor your local YugabyteDB cluster with a local instance of [Prometheus](https://prometheus.io/), a popular standard for time-series monitoring of cloud native infrastructure. YugabyteDB services and APIs expose metrics in the Prometheus format at the `/prometheus-metrics` endpoint. - -For details on the metrics targets for YugabyteDB, see [Prometheus monitoring](../../../../reference/configuration/default-ports/#prometheus-monitoring). - -If you haven't installed YugabyteDB, follow the [Quick Start](../../../../quick-start/) guide. - -## 1. Create universe - -If you have a previously running local universe, destroy it using the following. - -```sh -$ kubectl delete -f yugabyte-statefulset.yaml -``` - -Start a new local cluster - by default, this will create a three-node universe with a replication factor of `3`. - -```sh -$ kubectl apply -f yugabyte-statefulset.yaml -``` - -## Step 6. Clean up (optional) - -Optionally, you can shut down the local cluster created in Step 1. - -```sh -$ kubectl delete -f yugabyte-statefulset.yaml -``` - -Further, to destroy the persistent volume claims (**you will lose all the data if you do this**), run: - -```sh -kubectl delete pvc -l app=yb-master -kubectl delete pvc -l app=yb-tserver -``` diff --git a/docs/content/preview/explore/observability/prometheus-integration/linux.md b/docs/content/preview/explore/observability/prometheus-integration/linux.md deleted file mode 100644 index abf23eb451d9..000000000000 --- a/docs/content/preview/explore/observability/prometheus-integration/linux.md +++ /dev/null @@ -1,242 +0,0 @@ ---- -title: Prometheus integration -headerTitle: Prometheus integration -linkTitle: Prometheus integration -description: Learn about exporting YugabyteDB metrics and monitoring the cluster with Prometheus. -menu: - preview: - identifier: observability-2-linux - parent: explore-observability - weight: 235 -type: docs ---- - - - -You can monitor your local YugabyteDB cluster with a local instance of [Prometheus](https://prometheus.io/), a popular standard for time-series monitoring of cloud native infrastructure. YugabyteDB services and APIs expose metrics in the Prometheus format at the `/prometheus-metrics` endpoint. For details on the metrics targets for YugabyteDB, see [Prometheus monitoring](../../../../reference/configuration/default-ports/#prometheus-monitoring). - -This tutorial uses the [yugabyted](../../../../reference/configuration/yugabyted/) cluster management utility. - -## Prerequisite - -Prometheus is installed on your local machine. If you have not done so already, follow the links below. - -- [Download Prometheus](https://prometheus.io/download/) -- [Get Started with Prometheus](https://prometheus.io/docs/prometheus/latest/getting_started/) - -## 1. Create universe - -Start a new local three-node universe with a replication factor of `3`. - -```sh -$ ./bin/yugabyted start \ - --base_dir=node-1 \ - --listen=127.0.0.1 \ - --tserver_flags="start_redis_proxy=true" -``` - -```sh -$ ./bin/yugabyted start \ - --base_dir=node-2 \ - --listen=127.0.0.2 \ - --join=127.0.0.1 \ - --tserver_flags="start_redis_proxy=true" -``` - -```sh -$ ./bin/yugabyted start \ - --base_dir=node-3 \ - --listen=127.0.0.3 \ - --join=127.0.0.1 \ - --tserver_flags="start_redis_proxy=true" -``` - -## 2. Run the YugabyteDB workload generator - -Download the [YugabyteDB workload generator](https://github.com/yugabyte/yb-sample-apps) JAR file (`yb-sample-apps.jar`) by running the following command. - -```sh -$ wget https://github.com/yugabyte/yb-sample-apps/releases/download/1.3.9/yb-sample-apps.jar?raw=true -O yb-sample-apps.jar -``` - -Run the `CassandraKeyValue` workload application in a separate shell. - -```sh -$ java -jar ./yb-sample-apps.jar \ - --workload CassandraKeyValue \ - --nodes 127.0.0.1:9042 \ - --num_threads_read 1 \ - --num_threads_write 1 -``` - -## 3. Prepare Prometheus configuration file - -Copy the following into a file called `yugabytedb.yml`. - -```yaml -global: - scrape_interval: 5s # Set the scrape interval to every 5 seconds. Default is every 1 minute. - evaluation_interval: 5s # Evaluate rules every 5 seconds. The default is every 1 minute. - # scrape_timeout is set to the global default (10s). - -# YugabyteDB configuration to scrape Prometheus time-series metrics -scrape_configs: - - job_name: "yugabytedb" - metrics_path: /prometheus-metrics - relabel_configs: - - target_label: "node_prefix" - replacement: "cluster-1" - metric_relabel_configs: - # Save the name of the metric so we can group_by since we cannot by __name__ directly... - - source_labels: ["__name__"] - regex: "(.*)" - target_label: "saved_name" - replacement: "$1" - # The following basically retrofit the handler_latency_* metrics to label format. - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(.*)" - target_label: "server_type" - replacement: "$1" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(.*)" - target_label: "service_type" - replacement: "$2" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(_sum|_count)?" - target_label: "service_method" - replacement: "$3" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(_sum|_count)?" - target_label: "__name__" - replacement: "rpc_latency$4" - - static_configs: - - targets: ["127.0.0.1:7000", "127.0.0.2:7000", "127.0.0.3:7000"] - labels: - export_type: "master_export" - - - targets: ["127.0.0.1:9000", "127.0.0.2:9000", "127.0.0.3:9000"] - labels: - export_type: "tserver_export" - - - targets: ["127.0.0.1:12000", "127.0.0.2:12000", "127.0.0.3:12000"] - labels: - export_type: "cql_export" - - - targets: ["127.0.0.1:13000", "127.0.0.2:13000", "127.0.0.3:13000"] - labels: - export_type: "ysql_export" - - - targets: ["127.0.0.1:11000", "127.0.0.2:11000", "127.0.0.3:11000"] - labels: - export_type: "redis_export" -``` - -## 4. Start Prometheus server - -Go to the directory where Prometheus is installed and start the Prometheus server as below. - -```sh -$ ./prometheus --config.file=yugabytedb.yml -``` - -Open the Prometheus UI at and then navigate to the Targets page under Status. - -![Prometheus Targets](/images/ce/prom-targets.png) - -## 5. Analyze key metrics - -On the Prometheus Graph UI, you can now plot the read/write throughput and latency for the `CassandraKeyValue` sample app. As you can see from the [source code](https://github.com/yugabyte/yugabyte-db/blob/master/java/yb-loadtester/src/main/java/com/yugabyte/sample/apps/CassandraKeyValue.java) of the app, it uses only SELECT statements for reads and INSERT statements for writes (aside from the initial CREATE TABLE). This means you can measure throughput and latency by simply using the metrics corresponding to the SELECT and INSERT statements. - -Paste the following expressions into the **Expression** box and click **Execute** followed by **Add Graph**. - -### Throughput - -> Read IOPS - -```sh -sum(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-read-iops.png) - -> Write IOPS - -```sh -sum(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-write-iops.png) - -### Latency - -> Read Latency (in microseconds) - -```sh -avg(irate(rpc_latency_sum{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) / -avg(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-read-latency.png) - -> Write Latency (in microseconds) - -```sh -avg(irate(rpc_latency_sum{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) / -avg(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-write-latency.png) - -## 6. Clean up (optional) - -Optionally, you can shut down the local cluster created in Step 1. - -```sh -$ ./bin/yugabyted destroy \ - --base_dir=node-1 -``` - -```sh -$ ./bin/yugabyted destroy \ - --base_dir=node-2 -``` - -```sh -$ ./bin/yugabyted destroy \ - --base_dir=node-3 -``` - -## What's next? - -Set up [Grafana dashboards](../../grafana-dashboard/grafana/) for better visualization of the metrics being collected by Prometheus. diff --git a/docs/content/preview/explore/observability/prometheus-integration/macos.md b/docs/content/preview/explore/observability/prometheus-integration/macos.md index 7764b58a6246..846b46c8cbfc 100644 --- a/docs/content/preview/explore/observability/prometheus-integration/macos.md +++ b/docs/content/preview/explore/observability/prometheus-integration/macos.md @@ -11,97 +11,39 @@ menu: type: docs --- - - You can monitor your local YugabyteDB cluster with a local instance of [Prometheus](https://prometheus.io/), a popular standard for time-series monitoring of cloud native infrastructure. YugabyteDB services and APIs expose metrics in the Prometheus format at the `/prometheus-metrics` endpoint. For details on the metrics targets for YugabyteDB, see [Prometheus monitoring](../../../../reference/configuration/default-ports/#prometheus-monitoring). -This tutorial uses the [yugabyted](../../../../reference/configuration/yugabyted/) cluster management utility. - -## Prerequisite - -Prometheus is installed on your local machine. If you have not done so already, follow the links below. +{{< note title="Setup" >}} -- [Download Prometheus](https://prometheus.io/download/) -- [Get Started with Prometheus](https://prometheus.io/docs/prometheus/latest/getting_started/) +Local multi-node cluster. See [Set up your YugabyteDB cluster](../../../../explore/#set-up-your-yugabytedb-cluster). -## 1. Create universe +{{< /note >}} -Start a new local three-node universe with a replication factor of `3`. - -```sh -$ ./bin/yugabyted start \ - --base_dir=node-1 \ - --listen=127.0.0.1 \ - --tserver_flags="start_redis_proxy=true" -``` +## Prerequisites -```sh -$ ./bin/yugabyted start \ - --base_dir=node-2 \ - --listen=127.0.0.2 \ - --join=127.0.0.1 \ - --tserver_flags="start_redis_proxy=true" -``` - -```sh -$ ./bin/yugabyted start \ - --base_dir=node-3 \ - --listen=127.0.0.3 \ - --join=127.0.0.1 \ - --tserver_flags="start_redis_proxy=true" -``` +[Download Prometheus](https://prometheus.io/download/) and refer to [Get Started with Prometheus](https://prometheus.io/docs/prometheus/latest/getting_started/) for installation instructions. -## 2. Run the YugabyteDB workload generator +## Run the YugabyteDB workload generator -Download the [YugabyteDB workload generator](https://github.com/yugabyte/yb-sample-apps) JAR file (`yb-sample-apps.jar`) by running the following command. +Download the [YugabyteDB workload generator](https://github.com/yugabyte/yb-sample-apps) JAR file (`yb-sample-apps.jar`) using the following command: ```sh -$ wget https://github.com/yugabyte/yb-sample-apps/releases/download/1.3.9/yb-sample-apps.jar?raw=true -O yb-sample-apps.jar +wget https://github.com/yugabyte/yb-sample-apps/releases/download/1.3.9/yb-sample-apps.jar?raw=true -O yb-sample-apps.jar ``` Run the `CassandraKeyValue` workload application in a separate shell. ```sh -$ java -jar ./yb-sample-apps.jar \ +java -jar ./yb-sample-apps.jar \ --workload CassandraKeyValue \ --nodes 127.0.0.1:9042 \ --num_threads_read 1 \ --num_threads_write 1 ``` -## 3. Prepare Prometheus configuration file +## Prepare Prometheus configuration file -Copy the following into a file called `yugabytedb.yml`. +From your Prometheus home directory, create a file `yugabytedb.yml` and add the following: ```yaml global: @@ -162,21 +104,21 @@ scrape_configs: export_type: "redis_export" ``` -## 4. Start Prometheus server +## Start Prometheus server -Go to the directory where Prometheus is installed and start the Prometheus server as below. +Start the Prometheus server from the Prometheus home directory as follows: ```sh -$ ./prometheus --config.file=yugabytedb.yml +./prometheus --config.file=yugabytedb.yml ``` -Open the Prometheus UI at and then navigate to the Targets page under Status. +Open the Prometheus UI at and then navigate to the **Targets** page under **Status**. ![Prometheus Targets](/images/ce/prom-targets.png) -## 5. Analyze key metrics +## Analyze key metrics -On the Prometheus Graph UI, you can now plot the read/write throughput and latency for the `CassandraKeyValue` sample app. As you can see from the [source code](https://github.com/yugabyte/yugabyte-db/blob/master/java/yb-loadtester/src/main/java/com/yugabyte/sample/apps/CassandraKeyValue.java) of the app, it uses only SELECT statements for reads and INSERT statements for writes (aside from the initial CREATE TABLE). This means you can measure throughput and latency by simply using the metrics corresponding to the SELECT and INSERT statements. +On the Prometheus Graph UI, you can plot the read or write throughput and latency for the `CassandraKeyValue` sample application. Because the [source code](https://github.com/yugabyte/yugabyte-db/blob/master/java/yb-loadtester/src/main/java/com/yugabyte/sample/apps/CassandraKeyValue.java) of the application uses only SELECT statements for reads and INSERT statements for writes (aside from the initial CREATE TABLE), you can measure throughput and latency by using the metrics corresponding to the SELECT and INSERT statements. Paste the following expressions into the **Expression** box and click **Execute** followed by **Add Graph**. @@ -218,22 +160,22 @@ avg(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcess ![Prometheus Read IOPS](/images/ce/prom-write-latency.png) -## 6. Clean up (optional) +## Clean up (optional) -Optionally, you can shut down the local cluster created in Step 1. +Optionally, you can shut down a local cluster as follows: ```sh -$ ./bin/yugabyted destroy \ +./bin/yugabyted destroy \ --base_dir=node-1 ``` ```sh -$ ./bin/yugabyted destroy \ +./bin/yugabyted destroy \ --base_dir=node-2 ``` ```sh -$ ./bin/yugabyted destroy \ +./bin/yugabyted destroy \ --base_dir=node-3 ``` diff --git a/docs/content/preview/explore/query-1-performance/explain-analyze.md b/docs/content/preview/explore/query-1-performance/explain-analyze.md index 58b5ad23a6c1..5e41b6a80954 100644 --- a/docs/content/preview/explore/query-1-performance/explain-analyze.md +++ b/docs/content/preview/explore/query-1-performance/explain-analyze.md @@ -42,6 +42,8 @@ The *option* and its values are described in the following table. The most impor ### Examples +{{% explore-setup-single %}} + Typically, you start by creating a table in YugabyteDB and inserting rows into it. To create a table called `employees`, execute the following: diff --git a/docs/content/preview/explore/query-1-performance/pg-hint-plan.md b/docs/content/preview/explore/query-1-performance/pg-hint-plan.md index fe0a8d6b53c0..8382ac5c7e0d 100644 --- a/docs/content/preview/explore/query-1-performance/pg-hint-plan.md +++ b/docs/content/preview/explore/query-1-performance/pg-hint-plan.md @@ -24,6 +24,8 @@ To use `pg_hint_plan` effectively, you need thorough knowledge of how your appli {{< /warning >}} +{{% explore-setup-single %}} + ## Configure pg_hint_plan pg_hint_plan is pre-configured, and enabled by default. The following GUC (Grand Unified Configuration) parameters control pg_hint_plan: @@ -39,7 +41,7 @@ pg_hint_plan is pre-configured, and enabled by default. The following GUC (Grand To enable pg_hint_plan, run the following command: ```sql -SET pg_hint_plan.enable_hint=ON; +yugabyte=# SET pg_hint_plan.enable_hint=ON; ``` ### Turn on debug output @@ -84,7 +86,7 @@ yugabyte-# ORDER BY a.aid; ## Using pg_hint_plan -The following table and index definitions are used in the examples that follow to illustrate the features of pg_hint_plan. +The following table and index definitions are used in the examples that follow to illustrate the features of pg_hint_plan: ```sql CREATE TABLE t1 (id int PRIMARY KEY, val int); @@ -156,7 +158,7 @@ Scan method hints enforce the scanning method on tables when specified along wit In the following example, the hint `/*+SeqScan(t2)*/` allows table `t2` to be scanned using `SeqScan`. ``` sql -yugabyte=# /*+SeqScan(t2)*/ +/*+SeqScan(t2)*/ EXPLAIN (COSTS false) SELECT * FROM t1, t2 WHERE t1.id = t2.id; ``` @@ -180,8 +182,8 @@ error hint: In the following example, due to the hint `/*+SeqScan(t1)IndexScan(t2)*/`, `t2` is scanned using IndexScan, and `t1` is scanned using `SeqScan`. ``` sql -yugabyte=# /*+SeqScan(t1)IndexScan(t2)*/ -yugabyte-# EXPLAIN (COSTS false) SELECT * FROM t1, t2 WHERE t1.id = t2.id; +/*+SeqScan(t1)IndexScan(t2)*/ +EXPLAIN (COSTS false) SELECT * FROM t1, t2 WHERE t1.id = t2.id; ``` ```output @@ -205,8 +207,8 @@ error hint: You can also use hint phrases to instruct the query planner not to use a specific type of scan. As shown in the following example, the hint `/*+NoIndexScan(t1)*/` restricts `IndexScan` on table `t1`. ``` sql -yugabyte=# /*+NoIndexScan(t1)*/ -yugabyte-# EXPLAIN (COSTS false) SELECT * FROM t1, t2 WHERE t1.id = t2.id; +/*+NoIndexScan(t1)*/ +EXPLAIN (COSTS false) SELECT * FROM t1, t2 WHERE t1.id = t2.id; ``` ```output @@ -247,8 +249,8 @@ yugabyte=# EXPLAIN (COSTS false) SELECT * FROM t3 WHERE t3.id = 1; **Query using a secondary index**: ``` sql -yugabyte=# /*+IndexScan(t3 t3_id2)*/ -yugabyte-# EXPLAIN (COSTS false) SELECT * FROM t3 WHERE t3.id = 1; +/*+IndexScan(t3 t3_id2)*/ +EXPLAIN (COSTS false) SELECT * FROM t3 WHERE t3.id = 1; ``` ```output @@ -270,8 +272,8 @@ error hint: **Query reverts to `SeqScan` as none of the indices can be used**: ``` sql -yugabyte=# /*+IndexScan(t3 no_exist)*/ -yugabyte-# EXPLAIN (COSTS false) SELECT * FROM t3 WHERE t3.id = 1; +/*+IndexScan(t3 no_exist)*/ +EXPLAIN (COSTS false) SELECT * FROM t3 WHERE t3.id = 1; ``` ```output @@ -293,8 +295,8 @@ error hint: **Query with a selective list of indexes**: ``` sql -yugabyte=# /*+IndexScan(t3 t3_id1 t3_id2)*/ -yugabyte-# EXPLAIN (COSTS false) SELECT * FROM t3 WHERE t3.id = 1; +/*+IndexScan(t3 t3_id1 t3_id2)*/ +EXPLAIN (COSTS false) SELECT * FROM t3 WHERE t3.id = 1; ``` ```output @@ -374,9 +376,9 @@ error hint: In this example, the first query uses a `HashJoin` on tables `t1` and `t2` respectively while the second query uses a `NestedLoop` join for the same. The required join methods are specified in their respective hint phrases. You can use multiple hint phrases to combine join methods and scan methods. ```sql -yugabyte=# /*+NestLoop(t2 t3 t1) SeqScan(t3) SeqScan(t2)*/ -yugabyte-# EXPLAIN (COSTS false) SELECT * FROM t1, t2, t3 -yugabyte-# WHERE t1.id = t2.id AND t1.id = t3.id; +/*+NestLoop(t2 t3 t1) SeqScan(t3) SeqScan(t2)*/ +EXPLAIN (COSTS false) SELECT * FROM t1, t2, t3 +WHERE t1.id = t2.id AND t1.id = t3.id; ``` ```output @@ -409,7 +411,7 @@ In the preceding example, the hint `/*+NestLoop(t2 t3 t4 t1) SeqScan(t3) SeqScan Joining order hints execute joins in a particular order, as enumerated in a hint phrase's parameter list. You can enforce joining in a specific order using the `Leading` hint. ```sql -yugabyte=# /*+Leading(t1 t2 t3)*/ +/*+Leading(t1 t2 t3)*/ EXPLAIN (COSTS false) SELECT * FROM t1, t2, t3 WHERE t1.id = t2.id AND t1.id = t3.id; ``` @@ -427,7 +429,7 @@ EXPLAIN (COSTS false) SELECT * FROM t1, t2, t3 WHERE t1.id = t2.id AND t1.id = t ``` ```sql -yugabyte=# /*+Leading(t2 t3 t1)*/ +/*+Leading(t2 t3 t1)*/ EXPLAIN (COSTS false) SELECT * FROM t1, t2, t3 WHERE t1.id = t2.id AND t1.id = t3.id; ``` @@ -453,7 +455,7 @@ You can leverage the `work_mem` setting in PostgreSQL to improve the performance The following example shows how to enable `work_mem` as a part of a hint plan. ``` sql -yugabyte=# /*+Set(work_mem "1MB")*/ +/*+Set(work_mem "1MB")*/ EXPLAIN (COSTS false) SELECT * FROM t1, t2 WHERE t1.id = t2.id; ``` @@ -478,17 +480,17 @@ error hint: Planner method configuration parameters provide a crude method of influencing the query plans chosen by the query optimizer. If the default plan chosen by the optimizer for a particular query is not optimal, a temporary solution is to use one of these configuration parameters to force the optimizer to choose a different plan. YugabyteDB supports the following configuration parameters: -* enable_hashagg -* enable_hashjoin -* enable_indexscan -* enable_indexonlyscan -* enable_material -* enable_nestloop -* enable_partition_pruning -* enable_partitionwise_join -* enable_partitionwise_aggregate -* enable_seqscan -* enable_sort +- enable_hashagg +- enable_hashjoin +- enable_indexscan +- enable_indexonlyscan +- enable_material +- enable_nestloop +- enable_partition_pruning +- enable_partitionwise_join +- enable_partitionwise_aggregate +- enable_seqscan +- enable_sort pg_hint_plan leverages the planner method configuration by embedding these configuration parameters in each query's comment. Consider the following example: @@ -507,7 +509,7 @@ yugabyte=# EXPLAIN (COSTS false) SELECT * FROM t1, t2 WHERE t1.id = t2.id; ``` ```sql -yugabyte=# /*+Set(enable_indexscan off)*/ +/*+Set(enable_indexscan off)*/ EXPLAIN (COSTS false) SELECT * FROM t1, t2 WHERE t1.id = t2.id; ``` @@ -541,7 +543,7 @@ To enable the hint table, run the following commands: ```sql /* Create the hint_plan.hints table */ - CREATE EXTENSION pg_hint_plan; +CREATE EXTENSION pg_hint_plan; /* * Tell pg_hint_plan to check the hint table for diff --git a/docs/content/preview/explore/query-1-performance/pg-stat-activity.md b/docs/content/preview/explore/query-1-performance/pg-stat-activity.md index f168e000b101..451ab9fc4d28 100644 --- a/docs/content/preview/explore/query-1-performance/pg-stat-activity.md +++ b/docs/content/preview/explore/query-1-performance/pg-stat-activity.md @@ -15,6 +15,8 @@ type: docs YugabyteDB supports the PostgreSQL `pg_stat_activity` view to analyze live queries. This view returns analytic and diagnostic information about active YugabyteDB server processes and queries. The view returns one row per server process, and displays information related to the current status of the database connection. +{{% explore-setup-single %}} + ## Supported fields At a `ysqlsh` prompt, run the following command to return the fields supported by pg_stat_activity: diff --git a/docs/content/preview/explore/query-1-performance/pg-stat-progress-copy.md b/docs/content/preview/explore/query-1-performance/pg-stat-progress-copy.md index d32cd63a2ef0..1bf16c8334a7 100644 --- a/docs/content/preview/explore/query-1-performance/pg-stat-progress-copy.md +++ b/docs/content/preview/explore/query-1-performance/pg-stat-progress-copy.md @@ -68,6 +68,12 @@ This is required for YugabyteDB, because if the COPY command finishes due to an The following examples demonstrate the possible stages (IN PROGRESS, ERROR, SUCCESS) for the copy operation. +{{< note title="Setup" >}} + +Local single-node cluster. See [Set up your YugabyteDB cluster](../../../explore/#set-up-your-yugabytedb-cluster). + +{{< /note >}} + ### Prerequisites - Make sure you have YugabyteDB up and running. If you are new to YugabyteDB, follow the steps in [Quick start](../../../quick-start/) to have YugabyteDB up and running in minutes. diff --git a/docs/content/preview/explore/query-1-performance/pg-stat-statements.md b/docs/content/preview/explore/query-1-performance/pg-stat-statements.md index 96c95e855d19..31faee754d7c 100644 --- a/docs/content/preview/explore/query-1-performance/pg-stat-statements.md +++ b/docs/content/preview/explore/query-1-performance/pg-stat-statements.md @@ -58,6 +58,8 @@ yugabyte=# drop extension pg_stat_statements; ## Examples +{{% explore-setup-single %}} + Describe the columns in the view: ```sql diff --git a/docs/content/preview/explore/transactions/_index.md b/docs/content/preview/explore/transactions/_index.md index db370c11dfc6..b2f2b8712827 100644 --- a/docs/content/preview/explore/transactions/_index.md +++ b/docs/content/preview/explore/transactions/_index.md @@ -14,24 +14,21 @@ type: indexpage --- YugabyteDB is a transactional database that supports distributed transactions. A transaction is a sequence of operations performed as a single logical unit of work. A transaction has four key properties - **Atomicity**, **Consistency**, **Isolation** and **Durability** - commonly abbreviated as ACID. - The table below summarizes the support for transactions across YSQL and YCQL APIs. -| Property | YSQL | YCQL | Comments | -|--------------------------------------------------|-------------|----------|----------| -| [Distributed transactions](distributed-transactions-ysql/) | Yes | Yes | Perform multi-row or multi-table transactions.
Application can connect to any node of the cluster. | -| [Isolation levels](isolation-levels/) | Serializable
Snapshot | Snapshot | *Repeatable read* isolation level in PostgreSQL maps to
*snapshot* isolation in YSQL | -| Set `AUTOCOMMIT = false` | Yes | No | The transaction must be expressed as one statement in YCQL. | - +| Property | YSQL | YCQL | Comments | +| :------- | :--- | :--- | :------- | +| [Distributed transactions](distributed-transactions-ysql/) | Yes | Yes | Perform multi-row or multi-table transactions.
Application can connect to any node of the cluster. | +| [Isolation levels](isolation-levels/) | Serializable
Snapshot | Snapshot | Repeatable read isolation level in PostgreSQL maps to snapshot isolation in YSQL. | +| Set `AUTOCOMMIT = false` | Yes | No | The transaction must be expressed as one statement in YCQL. | -The various features are explained in the sections below. +The various features are explained in the following sections.
@@ -39,7 +36,7 @@ The various features are explained in the sections below.
-
Distributed Transactions
+
Distributed transactions
Understand how distributed transactions work in YugabyteDB. @@ -50,7 +47,7 @@ The various features are explained in the sections below.
-
Isolation Levels
+
Isolation levels
Details about isolation levels in YugabyteDB. @@ -61,7 +58,7 @@ The various features are explained in the sections below.
-
Explicit Locking
+
Explicit locking
Explicit row-level locking in YSQL. diff --git a/docs/content/preview/explore/transactions/distributed-transactions-ycql.md b/docs/content/preview/explore/transactions/distributed-transactions-ycql.md index f60d4e2029d7..a369ee93661b 100644 --- a/docs/content/preview/explore/transactions/distributed-transactions-ycql.md +++ b/docs/content/preview/explore/transactions/distributed-transactions-ycql.md @@ -1,12 +1,12 @@ --- -title: Distributed Transactions -headerTitle: Distributed Transactions -linkTitle: Distributed Transactions -description: Distributed Transactions in YugabyteDB. -headcontent: Distributed Transactions in YugabyteDB. +title: Distributed transactions +headerTitle: Distributed transactions +linkTitle: Distributed transactions +description: Distributed transactions in YugabyteDB. +headcontent: Explore distributed transactions in YugabyteDB. menu: preview: - name: Distributed Transactions + name: Distributed transactions identifier: explore-transactions-distributed-transactions-2-ycql parent: explore-transactions weight: 230 @@ -31,15 +31,19 @@ type: docs -## Creating the table +This example shows how a distributed transaction works in YugabyteDB. -Create a keyspace. +{{% explore-setup-single %}} + +## Create a table + +Create a keyspace as follows: ```sql ycqlsh> CREATE KEYSPACE banking; ``` -The YCQL table should be created with the `transactions` property enabled. The statement should look something as follows. +The YCQL table should be created with the `transactions` property enabled. The statement should be similar to the following: ```sql ycqlsh> CREATE TABLE banking.accounts ( @@ -50,14 +54,14 @@ ycqlsh> CREATE TABLE banking.accounts ( ) with transactions = { 'enabled' : true }; ``` -You can verify that this table has transactions enabled on it by running the following query. +You can verify that this table has transactions enabled by running the following query: ```sql ycqlsh> select keyspace_name, table_name, transactions from system_schema.tables where keyspace_name='banking' AND table_name = 'accounts'; ``` -``` +```output keyspace_name | table_name | transactions ---------------+------------+--------------------- banking | accounts | {'enabled': 'true'} @@ -67,7 +71,7 @@ where keyspace_name='banking' AND table_name = 'accounts'; ## Insert sample data -Let us seed this table with some sample data. +Seed the table with some sample data as follows: ```sql INSERT INTO banking.accounts (account_name, account_type, balance) VALUES ('John', 'savings', 1000); @@ -82,7 +86,7 @@ Here are the balances for John and Smith. ycqlsh> select * from banking.accounts; ``` -``` +```output account_name | account_type | balance --------------+--------------+--------- John | checking | 100 @@ -91,25 +95,25 @@ ycqlsh> select * from banking.accounts; Smith | savings | 2000 ``` -Check John's balance. +Check John's balance as follows: ```sql ycqlsh> SELECT SUM(balance) as Johns_balance FROM banking.accounts WHERE account_name='John'; ``` -``` +```output johns_balance --------------- 1100 ``` -Check Smith's balance. +Check Smith's balance as follows: ```sql ycqlsh> SELECT SUM(balance) as smiths_balance FROM banking.accounts WHERE account_name='Smith'; ``` -``` +```output smiths_balance ---------------- 2050 @@ -118,9 +122,7 @@ ycqlsh> SELECT SUM(balance) as smiths_balance FROM banking.accounts WHERE accoun ## Execute a transaction -Here are a couple of examples of executing transactions. - -Let us say John transfers $200 from his savings account to his checking account. This has to be a transactional operation. This can be achieved as follows. +Suppose John transfers $200 from his savings account to his checking account. This has to be a transactional operation. This can be achieved as follows: ```sql BEGIN TRANSACTION @@ -129,26 +131,26 @@ BEGIN TRANSACTION END TRANSACTION; ``` -If you now selected the value of John's account, you should see the amounts reflected. The total balance should be the same $1100 as before. +If you now select the value of John's account, you should see the amounts reflected. The total balance should be the same $1100 as before. ```sql ycqlsh> select * from banking.accounts where account_name='John'; ``` -``` +```output account_name | account_type | balance --------------+--------------+--------- John | checking | 300 John | savings | 800 ``` -Check John's balance. +Check John's balance as follows: ```sql ycqlsh> SELECT SUM(balance) as Johns_balance FROM banking.accounts WHERE account_name='John'; ``` -``` +```output johns_balance --------------- 1100 @@ -161,14 +163,14 @@ ycqlsh> select account_name, account_type, balance, writetime(balance) from banking.accounts where account_name='John'; ``` -``` +```output account_name | account_type | balance | writetime(balance) --------------+--------------+---------+-------------------- John | checking | 300 | 1517898028890171 John | savings | 800 | 1517898028890171 ``` -Now let us say John transfers the $200 from his checking account to Smith's checking account. We can accomplish that with the following transaction. +Now suppose John transfers the $200 from his checking account to Smith's checking account. Run the following transaction: ```sql BEGIN TRANSACTION @@ -179,13 +181,13 @@ END TRANSACTION; ## Verify -We can verify the transfer was made as we intended, and also verify that the time at which the two accounts were updated are identical by performing the following query. +Verify that the transfer was made as intended, and that the time at which the two accounts were updated are identical by performing the following query: ```sql ycqlsh> select account_name, account_type, balance, writetime(balance) from banking.accounts; ``` -``` +```output account_name | account_type | balance | writetime(balance) --------------+--------------+---------+-------------------- John | checking | 100 | 1517898167629366 @@ -200,19 +202,19 @@ The net balance for John should have decreased by $200 which that of Smith shoul ycqlsh> SELECT SUM(balance) as Johns_balance FROM banking.accounts WHERE account_name='John'; ``` -``` +```output johns_balance --------------- 900 ``` -Check Smith's balance. +Check Smith's balance as follows: ```sql ycqlsh> SELECT SUM(balance) as smiths_balance FROM banking.accounts WHERE account_name='Smith'; ``` -``` +```output smiths_balance ---------------- 2250 diff --git a/docs/content/preview/explore/transactions/distributed-transactions-ysql.md b/docs/content/preview/explore/transactions/distributed-transactions-ysql.md index 756c442c63ba..9ee044ce9053 100644 --- a/docs/content/preview/explore/transactions/distributed-transactions-ysql.md +++ b/docs/content/preview/explore/transactions/distributed-transactions-ysql.md @@ -1,12 +1,12 @@ --- -title: Distributed Transactions -headerTitle: Distributed Transactions -linkTitle: Distributed Transactions -description: Distributed Transactions in YugabyteDB. -headcontent: Distributed Transactions in YugabyteDB. +title: Distributed transactions +headerTitle: Distributed transactions +linkTitle: Distributed transactions +description: Distributed transactions in YugabyteDB. +headcontent: Explore distributed transactions in YugabyteDB. menu: preview: - name: Distributed Transactions + name: Distributed transactions identifier: explore-transactions-distributed-transactions-1-ysql parent: explore-transactions weight: 230 @@ -31,9 +31,13 @@ type: docs -## Overview of transaction control +This example shows how a distributed transaction works in YugabyteDB. -This section explains how a distributed transaction works in YugabyteDB. We will use the example table below to describe the control flow of a simple transaction. +{{% explore-setup-single %}} + +## Create a table + +Create the following table: ```sql CREATE TABLE accounts ( @@ -55,8 +59,11 @@ INSERT INTO accounts VALUES ('Smith', 'checking', 50); The table should look as follows: +```sql +yugabyte=# SELECT * FROM accounts; ``` -yugabyte=# select * from accounts; + +```output account_name | account_type | balance --------------+--------------+--------- John | checking | 100 @@ -66,7 +73,9 @@ yugabyte=# select * from accounts; (4 rows) ``` -Now, we will run the following transaction and explain what happens at each step. +## Run a transaction + +Run the following transaction: ```sql BEGIN TRANSACTION; @@ -77,6 +86,7 @@ BEGIN TRANSACTION; COMMIT; ``` +The following table explains what happens at each step. @@ -104,7 +114,7 @@ UPDATE accounts SET balance = balance - 200 @@ -128,20 +138,23 @@ COMMIT;
- The transaction coordinator writes a *provisional record* to the tablet that contains this row. The provisional record consists of the transaction id, so the state of the transaction can be determined. If there already exists a provisional record written by another transaction, then the current transaction would use the transaction id that is present in the provisional record to fetch details and check if there is a potential conflict. + The transaction coordinator writes a provisional record to the tablet that contains this row. The provisional record consists of the transaction ID, so the state of the transaction can be determined. If a provisional record written by another transaction already exists, then the current transaction would use the transaction ID that is present in the provisional record to fetch details and check if there is a potential conflict.
- Note that in order to COMMIT, all the provisional writes must have successfully completed. The COMMIT statement causes the transaction coordinator to update the transaction status in the transaction status table to COMMITED, at which point it is assigned the commit timestamp (which is a *hybrid timestamp* to be precise). At this point, the transaction is completed. In the background, the COMMIT record along with the commit timestamp is applied to each of the rows that participated to make future lookups of these rows efficient. + Note that to COMMIT, all the provisional writes must have successfully completed. The COMMIT statement causes the transaction coordinator to update the transaction status in the transaction status table to COMMITED, at which point it is assigned the commit timestamp (which is a hybrid timestamp to be precise). At this point, the transaction is completed. In the background, the COMMIT record along with the commit timestamp is applied to each of the rows that participated to make future lookups of these rows efficient.
-This is shown diagrammatically below. -![distributed_txn_write_path](/images/architecture/txn/distributed_txn_write_path.svg) +This is shown diagrammatically in the following illustration. +![Distributed transaction write path](/images/architecture/txn/distributed_txn_write_path.svg) -After the above transaction succeeds, the table should look as follows. +After the above transaction succeeds, the table should look as follows: +```sql +yugabyte=# SELECT * FROM accounts; ``` -yugabyte=# select * from accounts; + +```output account_name | account_type | balance --------------+--------------+--------- John | checking | 300 @@ -151,10 +164,9 @@ yugabyte=# select * from accounts; (4 rows) ``` - ### Scalability -Since all nodes of the cluster can process transactions by becoming transaction coordinators, horizontal scalability can simply be achieved by distributing the queries evenly across the nodes of the cluster. +Because all nodes of the cluster can process transactions by becoming transaction coordinators, horizontal scalability can be achieved by distributing the queries evenly across the nodes of the cluster. ### Resilience @@ -166,15 +178,17 @@ Each update performed as a part of the transaction is replicated across multiple {{< note title="Note" >}} YugabyteDB currently supports optimistic concurrency control, with pessimistic concurrency control being worked on actively. -{{}} - +{{}} -## Transaction Options +## Transaction options -You can see the various options supported by transactions by running the `\h BEGIN` statement, as shown below. +You can see the various options supported by transactions by running the following `\h BEGIN` meta-command: -``` +```sql yugabyte=# \h BEGIN +``` + +```output Command: BEGIN Description: start a transaction block Syntax: @@ -196,21 +210,28 @@ The `transaction_mode` can be set to one of the following options: As an example, trying to do a write operation such as creating a table or inserting a row in a `READ ONLY` transaction would result in an error as shown below. -``` +```sql yugabyte=# BEGIN READ ONLY; +``` + +```output BEGIN +``` +```sql yugabyte=# CREATE TABLE example(k INT PRIMARY KEY); -ERROR: 25P02: current transaction is aborted, commands ignored until end of - transaction block +``` + +```output +ERROR: cannot execute CREATE TABLE in a read-only transaction ``` ### `DEFERRABLE` transactions The `DEFERRABLE` transaction property in YSQL is similar to PostgreSQL in that has no effect unless the transaction is also `SERIALIZABLE` and `READ ONLY`. -When all three of these properties (`SERIALIZABLE`, `DEFERRABLE` and `READ ONLY`) are set for a transaction, the transaction may block when first acquiring its snapshot, after which it is able to run without the normal overhead of a `SERIALIZABLE` transaction and without any risk of contributing to or being canceled by a serialization failure. +When all three of these properties (`SERIALIZABLE`, `DEFERRABLE`, and `READ ONLY`) are set for a transaction, the transaction may block when first acquiring its snapshot, after which it is able to run without the typical overhead of a `SERIALIZABLE` transaction and without any risk of contributing to or being canceled by a serialization failure. {{< tip title="Tip" >}} -This mode is well suited for long-running reports or backups without being impacting or impacted by other transactions. +This mode is well-suited for long-running reports or backups without being impacting or impacted by other transactions. {{< /tip >}} diff --git a/docs/content/preview/explore/transactions/explicit-locking.md b/docs/content/preview/explore/transactions/explicit-locking.md index f80676ca2e19..0319d3b56fd1 100644 --- a/docs/content/preview/explore/transactions/explicit-locking.md +++ b/docs/content/preview/explore/transactions/explicit-locking.md @@ -1,13 +1,11 @@ --- -title: Explicit Locking -headerTitle: Explicit Locking -linkTitle: Explicit Locking -description: Explicit Locking in YugabyteDB. -headcontent: Explicit Locking in YugabyteDB. -image:
+title: Explicit locking +headerTitle: Explicit locking +linkTitle: Explicit locking +description: Explicit locking in YugabyteDB. +headcontent: Explore row locking in YugabyteDB menu: preview: - name: Explicit Locking identifier: explore-transactions-explicit-locking-1-ysql parent: explore-transactions weight: 245 @@ -30,8 +28,8 @@ This section describes how explicit locking works in YugabyteDB. YugabyteDB supports most row-level locks, similar to PostgreSQL. Explicit row-locks use transaction priorities to ensure that two transactions can never hold conflicting locks on the same row. To do this, the query layer acquires the row lock by assigning a very high value for the priority of the transaction that is being run. This causes all other transactions that conflict with the current transaction to fail, because they have a lower transaction priority. {{< note title="Note" >}} -Explicit locking is an area of active development in YugabyteDB. A number of enhancements are planned in this area. Unlike PostgreSQL, YugabyteDB uses optimistic concurrency control and does not block / wait for currently held locks, instead opting to abort the conflicting transaction with a lower priority. Pessimistic concurrency control is currently under development. -{{}} +Explicit locking is an area of active development in YugabyteDB. A number of enhancements are planned. Unlike PostgreSQL, YugabyteDB uses optimistic concurrency control and does not block or wait for currently held locks, instead opting to abort the conflicting transaction with a lower priority. Pessimistic concurrency control is currently under development. +{{}} The types of row locks currently supported are: @@ -40,8 +38,12 @@ The types of row locks currently supported are: * `FOR SHARE` * `FOR KEY SHARE` +## Example + The following example uses the `FOR UPDATE` row lock. First, a row is selected for update, thereby locking it, and subsequently updated. A concurrent transaction should not be able to abort this transaction by updating the value of that row after the row is locked. +{{% explore-setup-single %}} + To try out this scenario, first create an example table with sample data, as follows: ```sql diff --git a/docs/content/preview/explore/transactions/isolation-levels.md b/docs/content/preview/explore/transactions/isolation-levels.md index 54d08ea4540f..16b9f855ef4e 100644 --- a/docs/content/preview/explore/transactions/isolation-levels.md +++ b/docs/content/preview/explore/transactions/isolation-levels.md @@ -1,13 +1,13 @@ --- -title: Isolation Levels -headerTitle: Isolation Levels -linkTitle: Isolation Levels +title: Isolation levels +headerTitle: Isolation levels +linkTitle: Isolation levels description: Isolation Levels in YugabyteDB. -headcontent: Isolation Levels in YugabyteDB. +headcontent: Explore isolation levels in YugabyteDB. image:
menu: preview: - name: Isolation Levels + name: Isolation levels identifier: explore-transactions-isolation-levels-1-ysql parent: explore-transactions weight: 235 @@ -32,46 +32,47 @@ type: docs --> +YugabyteDB supports three isolation levels in the transactional layer - Serializable, Snapshot, and Read Committed. PostgreSQL (and the SQL standard) have four isolation levels - Serializable, Repeatable read, Read Committed, and Read uncommitted. The mapping between the PostgreSQL isolation levels in YSQL, along with which transaction anomalies can occur at each isolation level, are shown in the following table. -Yugabyte supports three isolation levels in the transactional layer - Serializable, Snapshot and Read Committed. PostgreSQL (and the SQL standard) have four isolation levels - `Serializable`, `Repeatable read`, `Read Committed` and `Read uncommitted`. The mapping between the PostgreSQL isolation levels in YSQL, along with which transaction anomalies can occur at each isolation level are shown below. - -PostgreSQL Isolation | YugabyteDB Equivalent | Dirty Read | Nonrepeatable Read | Phantom Read | Serialization Anomaly ------------------|--------------|------------------------|---|---|--- -Read
uncommitted | Read Committed$ | Allowed, but not in YSQL | Possible | Possible | Possible -Read
committed | Read Committed$ | Not possible | Possible | Possible | Possible -Repeatable
read | Snapshot | Not possible | Not possible | Allowed, but not in YSQL | Possible +| PostgreSQL Isolation | YugabyteDB Equivalent | Dirty Read | Non-repeatable Read | Phantom Read | Serialization Anomaly | +| :------------------- | :-------------------- | :--------- | :------------------ | :----------- | :-------------------- | +Read uncommitted | Read Committed$ | Allowed, but not in YSQL | Possible | Possible | Possible +Read committed | Read Committed$ | Not possible | Possible | Possible | Possible +Repeatable read | Snapshot | Not possible | Not possible | Allowed, but not in YSQL | Possible Serializable | Serializable | Not possible | Not possible | Not possible | Not possible -$ Read Committed Isolation is supported only if the tserver gflag `yb_enable_read_committed_isolation` is set to `true`. By default this gflag is `false` and in this case the Read Committed isolation level of Yugabyte's transactional layer falls back to the stricter Snapshot Isolation (in which case `READ COMMITTED` and `READ UNCOMMITTED` of YSQL also in turn use Snapshot Isolation). Read Committed support is currently in [Beta](/preview/faq/general/#what-is-the-definition-of-the-beta-feature-tag). +$ Read Committed Isolation is supported only if the YB-Tserver flag `yb_enable_read_committed_isolation` is set to `true`. By default this flag is `false` and in this case the Read Committed isolation level of the YugabyteDB transactional layer falls back to the stricter Snapshot Isolation (in which case `READ COMMITTED` and `READ UNCOMMITTED` of YSQL also in turn use Snapshot Isolation). Read Committed support is currently in [Beta](/preview/faq/general/#what-is-the-definition-of-the-beta-feature-tag). {{< note title="Note" >}} -The default isolation level for the YSQL API is essentially Snapshot (i.e., same as PostgreSQL's `REPEATABLE READ`) because `READ COMMITTED`, which is the YSQL API's (and also PostgreSQL's) syntactic default, maps to Snapshot Isolation (unless the tserver gflag `yb_enable_read_committed_isolation` is set to `true`). +The default isolation level for the YSQL API is essentially Snapshot (that is, the same as PostgreSQL's `REPEATABLE READ`) because `READ COMMITTED`, which is the YSQL API's (and also PostgreSQL's) syntactic default, maps to Snapshot Isolation (unless the YB-Tserver flag `yb_enable_read_committed_isolation` is set to `true`). To set the transaction isolation level of a transaction, use the command `SET TRANSACTION`. {{< /note >}} -As seen from the table above, the most strict isolation level is `Serializable`, which requires that any concurrent execution of a set of `Serializable` transactions is guaranteed to produce the same effect as running them in some serial (one transaction at a time) order. The other levels are defined by which anomalies must not occur as a result of interactions between concurrent transactions. Due to the definition of Serializable isolation, none of these anomalies are possible at that level. For reference, the various transaction anomalies are described briefly below: +As seen from the preceding table, the most strict isolation level is `Serializable`, which requires that any concurrent execution of a set of `Serializable` transactions is guaranteed to produce the same effect as running them in some serial (one transaction at a time) order. The other levels are defined by which anomalies must not occur as a result of interactions between concurrent transactions. Due to the definition of Serializable isolation, none of these anomalies are possible at that level. For reference, the various transaction anomalies are described briefly below: * `Dirty read`: A transaction reads data written by a concurrent uncommitted transaction. -* `Nonrepeatable read`: A transaction re-reads data it has previously read and finds that data has been modified by another transaction (that committed since the initial read). +* `Nonrepeatable read`: A transaction re-reads data it has previously read and finds that data has been modified by another transaction (that committed after the initial read). * `Phantom read`: A transaction re-executes a query returning a set of rows that satisfy a search condition and finds that the set of rows satisfying the condition has changed due to another recently-committed transaction. * `Serialization anomaly`: The result of successfully committing a group of transactions is inconsistent with all possible orderings of running those transactions one at a time. -Let us now look at how Serializable, Snapshot and Read Committed isolation works in YSQL. - -## Serializable Isolation +## Serializable isolation The *Serializable* isolation level provides the strictest transaction isolation. This level emulates serial transaction execution for all committed transactions; as if transactions had been executed one after another, serially, rather than concurrently. Serializable isolation can detect read-write conflicts in addition to write-write conflicts. This is accomplished by writing *provisional records* for read operations as well. -Let's use a bank overdraft protection example to illustrate this case. The hypothetical case is that there is a bank which allows depositors to withdraw money up to the total of what they have in all accounts. The bank will later automatically transfer funds as needed to close the day with a positive balance in each account. Within a single transaction they check that the total of all accounts exceeds the amount requested. +### Example + +The following bank overdraft protection example illustrates this case. The hypothetical case is that there is a bank which allows depositors to withdraw money up to the total of what they have in all accounts. The bank will later automatically transfer funds as needed to close the day with a positive balance in each account. Within a single transaction they check that the total of all accounts exceeds the amount requested. -Let's say someone tries to withdraw $900 from two of their accounts simultaneously, each with $500 balances. At the `REPEATABLE READ` transaction isolation level, that could work; but if the `SERIALIZABLE` transaction isolation level is used, a read/write conflict will be detected and one of the transactions will be rejected. +{{% explore-setup-single %}} -The example can be set up with these statements: +Suppose someone tries to withdraw $900 from two of their accounts simultaneously, each with $500 balances. At the `REPEATABLE READ` transaction isolation level, that could work; but if the `SERIALIZABLE` transaction isolation level is used, a read/write conflict is detected and one of the transactions rejected. + +Set up the example with the following statements: ```sql create table account @@ -86,98 +87,125 @@ insert into account values ('kevin','checking', 500); ``` - +Next, connect to the cluster using two independent ysqlsh instances, referred to as session #1 and session #2. + +
- - + + - - + - - +``` + + - - + - - + +``` + +```output +ERROR: 40001: Operation failed. + Try again.: Transaction aborted: XXXX +``` + + - - +
session #1session #2session #1session #2
- Begin a transaction in session #1 with the Serializable isolation level. The account total is $1000, so a $900 withdrawal is OK. - 

-begin isolation level serializable;

+
+ +Begin a transaction in session #1 with the Serializable isolation level. The account total is $1000, so a $900 withdrawal is OK. + +```sql +begin isolation level serializable; select type, balance from account - where name = 'kevin';
+ where name = 'kevin'; +``` + +```output type | balance ----------+--------- saving | $500.00 checking | $500.00 (2 rows) - - 		+``` + +
+ - Begin a transaction in session #2 with the Serializable isolation level as well. Once again, the account total is $1000, so a $900 withdrawal is OK. - 

-begin isolation level serializable;

+
+ +Begin a transaction in session #2 with the Serializable isolation level as well. Once again, the account total is $1000, so a $900 withdrawal is OK. + +```sql +begin isolation level serializable; select type, balance from account - where name = 'kevin';
+ where name = 'kevin'; +``` + +```output type | balance ----------+--------- saving | $500.00 checking | $500.00 (2 rows) - -
- Withdraw $900 from the savings account, given the total is $1000 this should be OK. - 

+
+ +Withdraw $900 from the savings account, given the total is $1000 this should be OK. + +```sql update account set balance = balance - 900::money where name = 'kevin' and type = 'saving'; - - +``` + +
- Simultaneously, withdrawing $900 from the checking account is going to be a problem. This cannot co-exist with the other transaction's activity. This transaction would fail immediately. - 

+
+ +Simultaneously, withdrawing $900 from the checking account is going to be a problem. This can't co-exist with the other transaction's activity. This transaction would fail immediately. + +```sql update account set balance = balance - 900::money where name = 'kevin' and type = 'checking'; -  -ERROR: 40001: Operation failed. Try again.: Transaction aborted: XXXX - -
- This transaction can now be committed. - 

-commit;

+
+ +This transaction can now be committed. + +```sql +commit; select type, balance from account - where name = 'kevin';
+ where name = 'kevin'; +``` + +```output type | balance ----------+---------- checking | $500.00 saving | -$400.00 (2 rows) - - 		+``` + +
- - -## Snapshot Isolation +## Snapshot isolation The Snapshot isolation level only sees data committed before the transaction began (or in other words, it works on a "snapshot" of the table). Transactions running under Snapshot isolation do not see either uncommitted data or changes committed during transaction execution by other concurrently running transactions. Note that the query does see the effects of previous updates executed within its own transaction, even though they are not yet committed. This is a stronger guarantee than is required by the SQL standard for the `REPEATABLE READ` isolation level. @@ -190,205 +218,261 @@ Snapshot isolation detects only write-write conflicts, it does not detect read-w Applications using this level must be prepared to retry transactions due to serialization failures. {{< /note >}} -Let's run through the scenario below to understand how transactions behave under the snapshot isolation level (which PostgreSQL's *Repeatable Read* maps to). +### Example + +The following scenario shows how transactions behave under the snapshot isolation level (which PostgreSQL's *Repeatable Read* maps to). -First, create an example table with sample data. +Create an example table with sample data as follows: ```sql CREATE TABLE IF NOT EXISTS example (k INT PRIMARY KEY); TRUNCATE TABLE example; ``` -Next, connect to the cluster using two independent `ysqlsh` instances called *session #1* and *session #2* below. +Next, connect to the cluster using two independent `ysqlsh` instances, referred to as *session #1* and *session #2*. -{{< note title="Note" >}} -You can connect the session #1 and session #2 `ysqlsh` instances to the same server, or to different servers. -{{< /note >}} - - +
- - + + - - + - - + - - +``` + + - - + - - + -
session #1session #2session #1session #2
- Begin a transaction in session #1. This will be snapshot isolation by default, meaning it will work against a snapshot of the database as of this point. - 

+
+ +Begin a transaction in session #1. This is snapshot isolation by default, meaning it will work against a snapshot of the database as of this point. + +```sql BEGIN TRANSACTION; - - +``` + +
- Insert a row, but let's not commit the transaction. This row should be visible only to this transaction. - 

-INSERT INTO example VALUES (1);

+
+ +Insert a row, but don't commit the transaction. This row should be visible only to this transaction. + +```sql +INSERT INTO example VALUES (1); SELECT * FROM example; +``` + +```output k --- 1 (1 row) - - +``` + +
+ - Insert a different row here. Verify that the row inserted in the transaction in session #1 is not visible in this session. - 

-INSERT INTO example VALUES (2);

+
+ +Insert a different row. Verify that the row inserted in the transaction in session #1 is not visible in this session as follows: + +```sql +INSERT INTO example VALUES (2); SELECT * FROM example; +``` + +```output k --- 2 (1 row) - -
- The row inserted in the other session would not be visible here, because we're working against an older snapshot of the database. Let's verify that. - 

+
+ +The row inserted in the other session is not visible here, because you're working against an older snapshot of the database. Verify that as follows: + +```sql SELECT * FROM example; +``` + +```output k --- 1 (1 row) - - +``` + +
- Now let's commit this transaction. As long as the row(s) we're writing as a part of this transaction are not modified during the lifetime of the transaction, there would be no conflicts. Let's verify we can see all rows after the commit. - 

-COMMIT; 

+
+ +Now commit this transaction. As long as the row(s) you're writing as a part of this transaction are not modified during the lifetime of the transaction, there would be no conflicts. Verify you can see all rows after the commit as follows: + +```sql +COMMIT; SELECT * FROM example; +``` + +```output k --- 1 2 (2 rows) - - +``` + +
-## Read Committed Isolation +## Read committed isolation -This is same as Snapshot Isolation except that every statement in the transaction will see all data that has been committed before it is issued (note that this implicitly also means that the statement will see a consistent snapshot). In other words, each statement works on a new "snapshot" of the database that includes everything that is committed before the statement is issued. Conflict detection is the same as in Snapshot Isolation. +Read committed isolation is the same as Snapshot isolation except that every statement in the transaction will see all data that has been committed before it is issued (note that this implicitly also means that the statement will see a consistent snapshot). In other words, each statement works on a new "snapshot" of the database that includes everything that is committed before the statement is issued. Conflict detection is the same as in Snapshot isolation. - - - - - +### Example - - - +``` + +Next, connect to the cluster using two independent ysqlsh instances, referred to as session #1 and session #2. + +
session #1session #2
- Create a sample table. - 

+Create a sample table as follows:
+
+```sql
 CREATE TABLE test (k int PRIMARY KEY, v int);
 INSERT INTO test VALUES (1, 2);
-
- 		-
+ + + - - + - - + - - - +(1 row) +``` - - - + - - + - - + diff --git a/docs/content/preview/explore/ysql-language-features/_index.md b/docs/content/preview/explore/ysql-language-features/_index.md index 41d536fec353..5267c801b207 100644 --- a/docs/content/preview/explore/ysql-language-features/_index.md +++ b/docs/content/preview/explore/ysql-language-features/_index.md @@ -24,7 +24,7 @@ The following diagram shows how YugabyteDB reuses the PostgreSQL query layer, sp ![Reusing the PostgreSQL query layer in YSQL](/images/section_icons/architecture/Reusing-PostgreSQL-query-layer.png) -## SQL Features in YSQL +## SQL features in YSQL The following table lists the most important YSQL features which you would find familiar if you have worked with PostgreSQL. diff --git a/docs/content/preview/explore/ysql-language-features/advanced-features/collations.md b/docs/content/preview/explore/ysql-language-features/advanced-features/collations.md index 447b8371741b..a73f0733ba6d 100644 --- a/docs/content/preview/explore/ysql-language-features/advanced-features/collations.md +++ b/docs/content/preview/explore/ysql-language-features/advanced-features/collations.md @@ -79,7 +79,7 @@ select count(collname) from pg_collation where collprovider = 'i'; (1 row) ``` -## Collation Creation +## Collation creation In addition to predefined collations, you can define new collations. For example, @@ -139,7 +139,7 @@ select * from coll_tab4 order by name; (2 rows) ``` -## Index Collation +## Index collation When a table column has an explicit collation, an index built on the column will be sorted according to the column collation. YSQL also allows the index to have its own explicit collation that is different from that of the table column. For example: @@ -148,9 +148,9 @@ create table coll_tab5(name text collate "en-US-x-icu"); create index name_idx on coll_tab5(name collate "C" asc); ``` -This can be useful to speed up queries that involve pattern matching operators such as LIKE because a regular index will be sorted according to collation "en-US-x-icu" and such an index cannot be used by pattern matching operators. +This can speed up queries that involve pattern matching operators such as LIKE because a regular index will be sorted according to collation "en-US-x-icu" and such an index cannot be used by pattern matching operators. -## Collation Strength +## Collation strength YSQL uses the same rules as in PostgreSQL to determine which collation is used in sorting character strings. An explicitly specified collation has more _strength_ then a referenced column, which has more strength than a text expression without an explicit collation. For example: @@ -255,7 +255,7 @@ There are a number of YSQL limitations on collation due to the internal implemen yugabyte=# ``` -* Libc collations are very limited: +* libc collations are very limited: ```sql select collname from pg_collation where collprovider = 'c'; diff --git a/docs/content/preview/explore/ysql-language-features/advanced-features/cursor.md b/docs/content/preview/explore/ysql-language-features/advanced-features/cursor.md index 0d9ff8c4e5d6..e8ccc3d6c653 100644 --- a/docs/content/preview/explore/ysql-language-features/advanced-features/cursor.md +++ b/docs/content/preview/explore/ysql-language-features/advanced-features/cursor.md @@ -45,15 +45,15 @@ Operations involving cursors must be performed inside transactions. You need to declare a cursor before you can open and use it. There are two ways to declare a cursor: -- As a variable of type `refcursor` placed within the YSQL block's declaration section, as demonstrated by the following syntax: +- As a variable of type `refcursor` placed in the YSQL block's declaration section, as demonstrated by the following syntax: - ``` + ```sql DECLARE new_cursor refcursor; ``` - As an element bound to a query, based on the following syntax: - ``` + ```sql new_cursor CURSOR [( arguments )] FOR a_query; ``` @@ -197,7 +197,7 @@ For more information and examples, refer to the "Returning Cursors" section in [ You can iterate through the result set of a bound cursor using a certain form of the `FOR` statement, as per the following syntax: -``` +```sql FOR rec_var IN bound_cursor_var [ ( [ argument_name := ] argument_value [, ...] ) ] LOOP @@ -221,6 +221,8 @@ CLOSE employees_cursor_2; ## Examples +{{% explore-setup-single %}} + Suppose you work with a database that includes the following table populated with data: ```sql diff --git a/docs/content/preview/explore/ysql-language-features/advanced-features/partitions.md b/docs/content/preview/explore/ysql-language-features/advanced-features/partitions.md index 0180467e5f06..1cdb32d6a98c 100644 --- a/docs/content/preview/explore/ysql-language-features/advanced-features/partitions.md +++ b/docs/content/preview/explore/ysql-language-features/advanced-features/partitions.md @@ -1,7 +1,7 @@ --- -title: Table Partitioning -linkTitle: Table Partitioning -description: Table Partitioning in YSQL +title: Table partitioning +linkTitle: Table partitioning +description: Table partitioning in YSQL image: /images/section_icons/secure/create-roles.png menu: preview: @@ -15,11 +15,13 @@ type: docs This section describes how to partition tables in YugabyteDB using YSQL. +{{% explore-setup-single %}} + ## Overview Partitioning is another term for physically dividing large tables in YugabyteDB into smaller, more manageable tables to improve performance. Typically, tables with columns containing timestamps are subject to partitioning because of the historical and predictable nature of their data. -Since partitioned tables do not appear nor act differently from the original table, applications accessing the database are not always aware of the fact that partitioning has taken place. +Because partitioned tables do not appear nor act differently from the original table, applications accessing the database are not always aware of the fact that partitioning has taken place. YSQL supports the following types of partitioning: @@ -27,9 +29,9 @@ YSQL supports the following types of partitioning: - List partitioning, when a table is partitioned via listing key values to appear in each partition. - Hash partitioning, when a table is partitioned by specifying a modulus and remainder for each partition. -For supplementary information on partitioning, see [Row-Level Geo-Partitioning](../../../multi-region-deployments/row-level-geo-partitioning/). +For supplementary information on partitioning, see [Row-level geo-partitioning](../../../multi-region-deployments/row-level-geo-partitioning/). -## Declarative Table Partitioning +## Declarative table partitioning YSQL allows you to specify how exactly to divide a table. You provide a partitioning method and partition key consisting of a list of columns or expressions. The divided table is called a partitioned table, and the resulting tables are called partitions. When you insert rows into a partitioned table, they are redirected to a partition depending on the value of the partition key. You can also directly insert rows into the partition table itself, and those rows can be fetched by querying the parent table. @@ -89,8 +91,7 @@ CREATE TABLE order_changes_2021_01 PARTITION OF order_changes FOR VALUES FROM ('2021-01-01') TO ('2021-02-01'); ``` -Partitioning ranges are inclusive at the lower ( `FROM` ) bound and exclusive at the upper ( `TO` ) bound. -Each month range in the preceding examples includes the start of the month, but does not include the start of the following month. +Partitioning ranges are inclusive at the lower ( `FROM` ) bound and exclusive at the upper ( `TO` ) bound. Each month range in the preceding examples includes the start of the month, but does not include the start of the following month. To create a new partition that contains only the rows that don't match the specified partitions, add a default partition as follows: @@ -106,8 +107,11 @@ yugabyte=# CREATE INDEX ON order_changes (change_date); This automatically creates indexes on each partition, as demonstrated by the following output: -```output +```sql yugabyte=# \d order_changes_2019_02 +``` + +```output Table "public.order_changes_2019_02" Column | Type | Collation | Nullable | Default -------------+------+-----------+----------+--------- @@ -117,10 +121,13 @@ yugabyte=# \d order_changes_2019_02 Partition of: order_changes FOR VALUES FROM ('2019-02-01') TO ('2019-03-01') Indexes: "order_changes_2019_02_change_date_idx" lsm (change_date HASH) +``` -... - +```sql yugabyte=# \d order_changes_2021_01 +``` + +```output Table "public.order_changes_2021_01" Column | Type | Collation | Nullable | Default -------------+------+-----------+----------+--------- @@ -179,7 +186,7 @@ SELECT count(*) FROM order_changes WHERE change_date >= DATE '2020-01-01'; If the `order_changes` table is partitioned by `change_date`, there is a big chance that only a subset of partitions needs to be queried. When enabled, both partition pruning and constraint exclusion can provide significant performance improvements for such queries by filtering out partitions that do not satisfy the criteria. -Even though partition pruning and constraint exclusion target the same goal, the underlying mechanisms are different. Specifically, constraint exclusion is applied during query planning, and therefore only works if the `WHERE` clause contains constants or externally supplied parameters. For example, a comparison against a non-immutable function such as `CURRENT_TIMESTAMP` cannot be optimized, since the planner cannot know which child table the function's value might fall into at run time. On the other hand, partition pruning is applied during query execution, and therefore can be more flexible. However, it is only used for `SELECT` queries. Updates can only benefit from constraint exclusion. +Even though partition pruning and constraint exclusion target the same goal, the underlying mechanisms are different. Specifically, constraint exclusion is applied during query planning, and therefore only works if the `WHERE` clause contains constants or externally supplied parameters. For example, a comparison against a non-immutable function such as `CURRENT_TIMESTAMP` cannot be optimized, because the planner cannot know which child table the function's value might fall into at run time. On the other hand, partition pruning is applied during query execution, and therefore can be more flexible. However, it is only used for `SELECT` queries. Updates can only benefit from constraint exclusion. Both optimizations are enabled by default, which is the recommended setting for the majority of cases. However, if you know for certain that one of your queries will have to scan all the partitions, you can consider disabling the optimizations for that query: @@ -191,4 +198,4 @@ SELECT count(*) FROM order_changes WHERE change_date >= DATE '2019-01-01'; To re-enable partition pruning, set the `enable_partition_pruning` setting to `on`. -For constraint exclusion, the recommended (and default) setting is neither `off` nor `on`, but rather an intermediate value `partition`, which means that it’s applied only to queries that are executed on partitioned tables. +For constraint exclusion, the recommended (and default) setting is neither `off` nor `on`, but rather an intermediate value `partition`, which means that it's applied only to queries that are executed on partitioned tables. diff --git a/docs/content/preview/explore/ysql-language-features/advanced-features/savepoints.md b/docs/content/preview/explore/ysql-language-features/advanced-features/savepoints.md index c7e21bdd2cfc..4f0c00a668f0 100644 --- a/docs/content/preview/explore/ysql-language-features/advanced-features/savepoints.md +++ b/docs/content/preview/explore/ysql-language-features/advanced-features/savepoints.md @@ -13,7 +13,7 @@ aliases: type: docs --- -This document provides an overview of YSQL savepoints, and demonstrates how to use them to checkpoint your progress within a transaction. The `SAVEPOINT` command establishes a new savepoint within the current transaction. +This document provides an overview of YSQL savepoints, and demonstrates how to use them to checkpoint your progress within a transaction. The `SAVEPOINT` command establishes a new savepoint in the current transaction. ## Overview @@ -31,6 +31,8 @@ The relevant savepoint commands are: ## Example +{{% explore-setup-single %}} + 1. Create a sample table. ```plpgsql diff --git a/docs/content/preview/explore/ysql-language-features/advanced-features/views.md b/docs/content/preview/explore/ysql-language-features/advanced-features/views.md index 629ac8f9039e..6a1afb4d5533 100644 --- a/docs/content/preview/explore/ysql-language-features/advanced-features/views.md +++ b/docs/content/preview/explore/ysql-language-features/advanced-features/views.md @@ -15,11 +15,11 @@ type: docs This document describes how to create, use, and manage views in YSQL. -## Overview - Regular views allow you to present data in YugabyteDB tables by using a different variety of named queries. In essence, a view is a proxy for a complex query to which you assign a name. In YSQL, views do not store data. However, YSQL also supports materialized views which _do_ store the results of the query. -## Creating Views +{{% explore-setup-single %}} + +## Create views You create views based on the following syntax: @@ -27,7 +27,7 @@ You create views based on the following syntax: CREATE VIEW view_name AS query_definition; ``` -*query_definition* can be a simple `SELECT` statement or a `SELECT` statement with joins. +*query_definition* can be a basic `SELECT` statement or a `SELECT` statement with joins. Suppose you work with a database that includes the following table populated with data: @@ -74,7 +74,7 @@ employee_no | name If you create a view based on multiple tables with joins, using this view in your queries would significantly simplify the process. -## Modifying Views +## Modify views You can modify the query based on which a view was created by combining the `CREATE VIEW` statement with `OR REPLACE`, as demonstrated by the following syntax: @@ -106,7 +106,7 @@ The preceding query produces the following output: 1222 | Bette Davis | Sales ``` -## Deleting Views +## Delete views You can remove (drop) an existing view by using the `DROP VIEW` statement, as demonstrated by the following syntax: @@ -124,7 +124,7 @@ DROP VIEW IF EXISTS employees_view; You can also remove more than one view by providing a comma-separated list of view names. -## Using Updatable Views +## Use updatable views Some YSQL views are updatable. The defining query of such views (1) must have only one entry (either a table or another updatable view) in its `FROM` clause; and (2) cannot contain `DISTINCT`, `GROUP BY`, `HAVING`, `EXCEPT`, `INTERSECT`, or `LIMIT` clauses at the top level. In addition, the view's selection list cannot contain window functions, set-returning or aggregate functions. @@ -162,7 +162,7 @@ DELETE FROM employees_view WHERE employee_no = 1227; ``` -## Materialized Views +## Materialized views Materialized views are relations that persist the results of a query. They can be created using the `CREATE MATERIALIZED VIEW` command, and their contents can be updated using the `REFRESH MATERIALIZED VIEW` command. @@ -216,8 +216,10 @@ employee_no | name 1222 | Bette Davis ``` -For detailed documentation on materialized views please refer to the following links: +## Read more + +For detailed documentation on materialized views, refer to the following topics: -- [`CREATE MATERIALIZED VIEW`](../../../../api/ysql/the-sql-language/statements/ddl_create_matview/) -- [`REFRESH MATERIALIZED VIEW`](../../../../api/ysql/the-sql-language/statements/ddl_refresh_matview/) -- [`DROP MATERIALIZED VIEW`](../../../../api/ysql/the-sql-language/statements/ddl_drop_matview/) +- [CREATE MATERIALIZED VIEW](../../../../api/ysql/the-sql-language/statements/ddl_create_matview/) +- [REFRESH MATERIALIZED VIEW](../../../../api/ysql/the-sql-language/statements/ddl_refresh_matview/) +- [DROP MATERIALIZED VIEW](../../../../api/ysql/the-sql-language/statements/ddl_drop_matview/) diff --git a/docs/content/preview/explore/ysql-language-features/data-manipulation.md b/docs/content/preview/explore/ysql-language-features/data-manipulation.md index 3bd2ffc4850e..eb82a6171435 100644 --- a/docs/content/preview/explore/ysql-language-features/data-manipulation.md +++ b/docs/content/preview/explore/ysql-language-features/data-manipulation.md @@ -13,6 +13,8 @@ type: docs This section describes how to manipulate data in YugabyteDB using the YSQL `INSERT`, `UPDATE`, and `DELETE` statements. +{{% explore-setup-single %}} + ## Insert rows Initially, database tables are not populated with data. Using YSQL, you can add one or more rows containing complete or partial data by inserting one row at a time. diff --git a/docs/content/preview/explore/ysql-language-features/data-types.md b/docs/content/preview/explore/ysql-language-features/data-types.md index 7cbbd0c3d5ee..30c138ebb1dd 100644 --- a/docs/content/preview/explore/ysql-language-features/data-types.md +++ b/docs/content/preview/explore/ysql-language-features/data-types.md @@ -15,6 +15,8 @@ This document describes the data types supported in YSQL, from the basic data ty The [JSONB document data type](../../json-support/jsonb-ysql/) is described in a separate section. +{{% explore-setup-single %}} + ## Strings The following character types are supported: diff --git a/docs/content/preview/explore/ysql-language-features/databases-schemas-tables.md b/docs/content/preview/explore/ysql-language-features/databases-schemas-tables.md index 39f66aa66074..40be74bf5e80 100644 --- a/docs/content/preview/explore/ysql-language-features/databases-schemas-tables.md +++ b/docs/content/preview/explore/ysql-language-features/databases-schemas-tables.md @@ -13,6 +13,8 @@ type: docs This section covers basic topics including how to connect to your cluster using the YSQL shell, and use the shell to manage databases, schemas, and tables. +{{% explore-setup-single %}} + ## YSQL shell Use the [ysqlsh shell](../../../admin/ysqlsh/) to interact with a Yugabyte database cluster using the [YSQL API](../../../api/ysql/). Because `ysqlsh` is derived from the PostgreSQL shell `psql` code base, all `psql` commands work as is in `ysqlsh`. Some default settings such as the database default port and the output format of some of the schema commands have been modified for YugabyteDB. diff --git a/docs/content/preview/explore/ysql-language-features/expressions-operators.md b/docs/content/preview/explore/ysql-language-features/expressions-operators.md index ad4586dbb3b3..2f1ed1afe4de 100644 --- a/docs/content/preview/explore/ysql-language-features/expressions-operators.md +++ b/docs/content/preview/explore/ysql-language-features/expressions-operators.md @@ -13,6 +13,8 @@ type: docs This document describes how to use boolean, numeric, and date expressions, as well as basic operators. In addition, it provides information on conditional expression and operators. +{{% explore-setup-single %}} + ## Basic operators A large number of YSQL types have corresponding mathematical operators that are typically used for performing comparisons and mathematical operations. Operators allow you to specify conditions in YSQL statements and create links between conditions. diff --git a/docs/content/preview/explore/ysql-language-features/going-beyond-sql/follower-reads-ycql.md b/docs/content/preview/explore/ysql-language-features/going-beyond-sql/follower-reads-ycql.md index 94837e8ace27..90cbffa00bfe 100644 --- a/docs/content/preview/explore/ysql-language-features/going-beyond-sql/follower-reads-ycql.md +++ b/docs/content/preview/explore/ysql-language-features/going-beyond-sql/follower-reads-ycql.md @@ -51,15 +51,15 @@ You can specify the maximum staleness of data when reading from tablet followers In this tutorial, you update a single key-value over and over, and read it from the tablet leader. While that workload is running, you start another workload to read from a follower and verify that you are able to read from a tablet follower. -### Create universe +### Create a cluster -If you have a previously running local universe, destroy it by executing the following command: +If you have a previously running cluster, destroy it by executing the following command: ```sh $ ./bin/yb-ctl destroy ``` -Start a new local universe with three nodes and a replication factor (RF) of `3`, as follows: +Start a new local cluster with three nodes and a replication factor (RF) of `3`, as follows: ```sh $ ./bin/yb-ctl --rf 3 create @@ -79,9 +79,9 @@ Download the [YugabyteDB workload generator](https://github.com/yugabyte/yb-samp $ wget https://github.com/yugabyte/yb-sample-apps/releases/download/1.3.9/yb-sample-apps.jar?raw=true -O yb-sample-apps.jar ``` -By default, the YugabyteDB workload generator runs with strong read consistency, where all data is read from the tablet leader. Note that the `yb-sample-apps.jar` sets the [consistency](../../../../admin/ycqlsh/#consistency) level to ONE by default. You can populate exactly one key with a `10KB` value into the system. Because the replication factor is `3`, this key is replicated to only three of the four nodes in the universe. +By default, the YugabyteDB workload generator runs with strong read consistency, where all data is read from the tablet leader. Note that the `yb-sample-apps.jar` sets the [consistency](../../../../admin/ycqlsh/#consistency) level to ONE by default. You can populate exactly one key with a `10KB` value into the system. Because the replication factor is `3`, this key is replicated to only three of the four nodes in the cluster. -Run the `CassandraKeyValue` workload application to constantly update this key-value, as well as perform reads with strong consistency against the local universe, as follows: +Run the `CassandraKeyValue` workload application to constantly update this key-value, as well as perform reads with strong consistency against the local cluster, as follows: ```sh $ java -jar ./yb-sample-apps.jar --workload CassandraKeyValue \ @@ -154,6 +154,5 @@ $ ./bin/yb-ctl destroy ## Read more -- [Read replica deployment](../../../../deploy/multi-dc/read-replica-clusters/). - -- [Read replicas](../../../multi-region-deployments/read-replicas-ycql/) in YCQL. +- [Read replica deployment](../../../../deploy/multi-dc/read-replica-clusters/) +- [Read replicas](../../../multi-region-deployments/read-replicas-ycql/) diff --git a/docs/content/preview/explore/ysql-language-features/queries.md b/docs/content/preview/explore/ysql-language-features/queries.md index 5c8a2a947d18..c337b767b623 100644 --- a/docs/content/preview/explore/ysql-language-features/queries.md +++ b/docs/content/preview/explore/ysql-language-features/queries.md @@ -39,6 +39,8 @@ The following `SELECT` statement clauses provide flexibility and allow you to fi ### SELECT examples +{{% explore-setup-single %}} + Suppose you work with a database that includes the following table populated with data: ```sql diff --git a/docs/content/preview/explore/ysql-language-features/sql-feature-support.md b/docs/content/preview/explore/ysql-language-features/sql-feature-support.md index 0eda32b67ba4..213b282f387a 100644 --- a/docs/content/preview/explore/ysql-language-features/sql-feature-support.md +++ b/docs/content/preview/explore/ysql-language-features/sql-feature-support.md @@ -18,23 +18,23 @@ To understand which standard SQL features we support, refer to the following tab | Data type | Supported | Documentation | | :-------- | :-------: | :------------ | -| `ARRAY` | ✓ | [Array documentation](../../../api/ysql/datatypes/type_array/) | -| `BINARY` | ✓ | [Binary documentation](../../../api/ysql/datatypes/type_binary/) | +| `ARRAY` | ✓ | [Array data types](../../../api/ysql/datatypes/type_array/) | +| `BINARY` | ✓ | [Binary data types](../../../api/ysql/datatypes/type_binary/) | | `BIT`,`BYTES` | ✓ | | -| `BOOLEAN` | ✓ | [Boolean documentation](../../../api/ysql/datatypes/type_bool/) | -| `CHAR`, `VARCHAR`, `TEXT` | ✓ | [Character data types documentation](../../../api/ysql/datatypes/type_character/) | -| `COLLATE` | ✓ | [Collate documentation](../../ysql-language-features/advanced-features/collations/#root) | -| `DATE`, `TIME`, `TIMESTAMP`, `INTERVAL` | ✓ | [Date and time data types documentation](../../../api/ysql/datatypes/type_datetime/) | -| `DEC`, `DECIMAL`, `NUMERIC` | ✓ | [Fixed point numbers documentation](../../../api/ysql/datatypes/type_numeric/#fixed-point-numbers) | -| `ENUM` | ✓ |[ENUM documentation](../../ysql-language-features/data-types/#enumerations-enum-type) | -| `FLOAT`, `REAL`, `DOUBLE PRECISION` | ✓ | [Floating point numbers documentation](../../../api/ysql/datatypes/type_numeric/) | -| `JSON`, `JSONB` | ✓ | [JSON data types documentation](../../../api/ysql/datatypes/type_json/) | -| `MONEY` | ✓ | [Money data type documentation](../../../api/ysql/datatypes/type_money/) | -| `SERIAL`, `SMALLSERIAL`, `BIGSERIAL`| ✓ | [Serial documentation](../../../api/ysql/datatypes/type_serial/) | +| `BOOLEAN` | ✓ | [Boolean data types](../../../api/ysql/datatypes/type_bool/) | +| `CHAR`, `VARCHAR`, `TEXT` | ✓ | [Character data types](../../../api/ysql/datatypes/type_character/) | +| `COLLATE` | ✓ | [Collations](../../ysql-language-features/advanced-features/collations/#root) | +| `DATE`, `TIME`, `TIMESTAMP`, `INTERVAL` | ✓ | [Date and time data types](../../../api/ysql/datatypes/type_datetime/) | +| `DEC`, `DECIMAL`, `NUMERIC` | ✓ | [Fixed point numbers](../../../api/ysql/datatypes/type_numeric/#fixed-point-numbers) | +| `ENUM` | ✓ |[Enumerations](../../ysql-language-features/data-types/#enumerations-enum-type) | +| `FLOAT`, `REAL`, `DOUBLE PRECISION` | ✓ | [Floating-point numbers](../../../api/ysql/datatypes/type_numeric/#floating-point-numbers) | +| `JSON`, `JSONB` | ✓ | [JSON data types](../../../api/ysql/datatypes/type_json/) | +| `MONEY` | ✓ | [Money data types](../../../api/ysql/datatypes/type_money/) | +| `SERIAL`, `SMALLSERIAL`, `BIGSERIAL`| ✓ | [Serial data types](../../../api/ysql/datatypes/type_serial/) | | `SET`| ✗ | | -| `SMALLINT, INT, INTEGER, BIGINT` | ✓ | [Integers documentation](../../../api/ysql/datatypes/type_numeric/) | -| `INT4RANGE`, `INT8RANGE`, `NUMRANGE`, `TSRANGE`, `TSTZRANGE`, `DATERANGE` | ✓ | [Range data types documentation](../../../api/ysql/datatypes/type_range/) | -| `UUID` | ✓ | [UUID documentation](../../../api/ysql/datatypes/type_uuid/) | +| `SMALLINT, INT, INTEGER, BIGINT` | ✓ | [Integers](../../../api/ysql/datatypes/type_numeric/#integers) | +| `INT4RANGE`, `INT8RANGE`, `NUMRANGE`, `TSRANGE`, `TSTZRANGE`, `DATERANGE` | ✓ | [Range data types](../../../api/ysql/datatypes/type_range/) | +| `UUID` | ✓ | [UUID data type](../../../api/ysql/datatypes/type_uuid/) | | `XML`| ✗ | | | `TSVECTOR` | ✓ | | | UDT(Base, Enumerated, Range, Composite, Array, Domain types) | ✓ | | @@ -43,19 +43,19 @@ To understand which standard SQL features we support, refer to the following tab | Operation | Supported | Documentation | | :-------- | :-------: | :------------ | -| Altering tables | ✓ | [`ALTER TABLE` documentation](../../../api/ysql/the-sql-language/statements/ddl_alter_table/) | -| Altering databases | ✓ | [`ALTER DATABASE` documentation](../../../api/ysql/the-sql-language/statements/ddl_alter_db/) | +| Altering tables | ✓ | [ALTER TABLE](../../../api/ysql/the-sql-language/statements/ddl_alter_table/) | +| Altering databases | ✓ | [ALTER DATABASE](../../../api/ysql/the-sql-language/statements/ddl_alter_db/) | | Altering columns | ✗ | | | Altering a column's data type | ✗ | | -| Adding columns | ✓ | [`ADD COLUMN` documentation](../../../api/ysql/the-sql-language/statements/ddl_alter_table/#add-column-column-name-data-type-constraint-constraints) | -| Removing columns | ✓ | [`DROP COLUMN` documentation](../../../api/ysql/the-sql-language/statements/ddl_alter_table/#drop-column-column-name-restrict-cascade) | -| Adding constraints | ✓ | [`ADD CONSTRAINT` documentation](../../../api/ysql/the-sql-language/statements/ddl_alter_table/#add-alter-table-constraint-constraints) | -| Removing constraints | ✓ | [`DROP CONSTRAINT` documentation](../../../api/ysql/the-sql-language/statements/ddl_alter_table/#drop-constraint-constraint-name-restrict-cascade) | +| Adding columns | ✓ | [ADD COLUMN](../../../api/ysql/the-sql-language/statements/ddl_alter_table/#add-column-column-name-data-type-constraint-constraints) | +| Removing columns | ✓ | [DROP COLUMN](../../../api/ysql/the-sql-language/statements/ddl_alter_table/#drop-column-column-name-restrict-cascade) | +| Adding constraints | ✓ | [ADD CONSTRAINT](../../../api/ysql/the-sql-language/statements/ddl_alter_table/#add-alter-table-constraint-constraints) | +| Removing constraints | ✓ | [DROP CONSTRAINT](../../../api/ysql/the-sql-language/statements/ddl_alter_table/#drop-constraint-constraint-name-restrict-cascade) | | Altering indexes | ✗ | | -| Adding indexes | ✓ | [`CREATE INDEX` documentation](../../../api/ysql/the-sql-language/statements/ddl_create_index/) | +| Adding indexes | ✓ | [CREATE INDEX](../../../api/ysql/the-sql-language/statements/ddl_create_index/) | | Removing indexes | ✗ | | | Altering a primary key | ✗ | | -| Adding user-defined schemas | ✓ | [`CREATE SCHEMA` documentation](../../../api/ysql/the-sql-language/statements/ddl_create_schema/) | +| Adding user-defined schemas | ✓ | [CREATE SCHEMA](../../../api/ysql/the-sql-language/statements/ddl_create_schema/) | | Removing user-defined schemas | ✗ | | | Altering user-defined schemas | ✗ | | @@ -63,11 +63,11 @@ To understand which standard SQL features we support, refer to the following tab | Feature | Supported | Documentation | | :------ | :-------: | :------------ | -| Check | ✓ | [Check documentation](../../indexes-constraints/other-constraints/#check-constraint) | -| Unique | ✓ | [Unique documentation](../../indexes-constraints/other-constraints/#unique-constraint) | -| Not Null | ✓ | [Not Null documentation](../../indexes-constraints/other-constraints/#not-null-constraint) | -| Primary Key | ✓ | [Primary Key documentation](../../indexes-constraints/primary-key-ysql/) | -| Foreign Key | ✓ | [Foreign Key documentation](../../indexes-constraints/foreign-key-ysql/) | +| Check | ✓ | [Check constraint](../../indexes-constraints/other-constraints/#check-constraint) | +| Unique | ✓ | [Unique constraint](../../indexes-constraints/other-constraints/#unique-constraint) | +| Not Null | ✓ | [Not Null constraint](../../indexes-constraints/other-constraints/#not-null-constraint) | +| Primary Key | ✓ | [Primary keys](../../indexes-constraints/primary-key-ysql/) | +| Foreign Key | ✓ | [Foreign keys](../../indexes-constraints/foreign-key-ysql/) | | Default Value | ✗ | | | Deferrable Foreign Key constraints | ✓ | | | Deferrable Primary Key and Unique constraints | ✗ | | @@ -77,9 +77,9 @@ To understand which standard SQL features we support, refer to the following tab | Component | Supported | Documentation | | :-------- | :-------: | :------------ | -| Indexes | ✓ | [Indexes documentation](../../indexes-constraints/) | -| GIN indexes | ✓ | [GIN Indexes documentation](../../indexes-constraints/gin/) | -| Partial indexes | ✓ | [Partial indexes documentation](../../indexes-constraints/partial-index-ysql/) | +| Indexes | ✓ | [Indexes and constraints](../../indexes-constraints/) | +| GIN indexes | ✓ | [GIN indexes](../../indexes-constraints/gin/) | +| Partial indexes | ✓ | [Partial indexes](../../indexes-constraints/partial-index-ysql/) | | Expression indexes | ✓ | [Expression indexes](../../indexes-constraints/expression-index-ysql/) | | Multi-column indexes | ✓ | | | Covering indexes | ✓ | [Covering indexes](../../indexes-constraints/covering-index-ysql/) | @@ -94,50 +94,50 @@ To understand which standard SQL features we support, refer to the following tab | Feature | Supported | Documentation | | :------ | :-------: | :------------ | -| Transactions | ✓ | [Transactions documentation](../../transactions/) | -| `BEGIN` | ✓ | [`BEGIN` documentation](../../../api/ysql/the-sql-language/statements/txn_begin/) | -| `COMMIT` | ✓ | [`COMMIT` documentation](../../../api/ysql/the-sql-language/statements/txn_commit/) | -| `ROLLBACK` | ✓ | [`ROLLBACK` documentation](../../../api/ysql/the-sql-language/statements/txn_rollback/) | -| `SAVEPOINT` | ✓ | [`SAVEPOINT` documentation](../../../api/ysql/the-sql-language/statements/savepoint_create/) | -| `ROLLBACK TO SAVEPOINT` | ✓ | [`ROLLBACK TO SAVEPOINT` documentation](../../../api/ysql/the-sql-language/statements/savepoint_create/) | +| Transactions | ✓ | [Transactions](../../transactions/) | +| `BEGIN` | ✓ | [BEGIN](../../../api/ysql/the-sql-language/statements/txn_begin/) | +| `COMMIT` | ✓ | [COMMIT](../../../api/ysql/the-sql-language/statements/txn_commit/) | +| `ROLLBACK` | ✓ | [ROLLBACK](../../../api/ysql/the-sql-language/statements/txn_rollback/) | +| `SAVEPOINT` | ✓ | [SAVEPOINT](../../../api/ysql/the-sql-language/statements/savepoint_create/) | +| `ROLLBACK TO SAVEPOINT` | ✓ | [ROLLBACK TO SAVEPOINT](../../../api/ysql/the-sql-language/statements/savepoint_rollback/) | ### Roles and Permissions | Component | Supported | Details | | :-------- | :-------: | :------ | -| Users | ✓ | | -| Roles | ✓ | | +| Users | ✓ | [Manage users and roles](../../../secure/authorization/create-roles/) | +| Roles | ✓ | [Manage users and roles](../../../secure/authorization/create-roles/) | | Object ownership | ✓ | | -| Privileges | ✓ | | +| Privileges | ✓ | [Grant privileges](../../../secure/authorization/ysql-grant-permissions/) | | Default privileges | ✗ | | ### Queries | Component | Supported | Details | | :-------- | :-------: | :------ | -| FROM, WHERE, GROUP BY, HAVING, DISTINCT, LIMIT/OFFSET, WITH queries| ✓ | | -| EXPLAIN query plans| ✓ | | -| JOINs (INNER/OUTER, LEFT/RIGHT) | ✓ | | -| Expressions and Operators| ✓ | | -| Common Table Expressions (CTE) and Recursive Queries| ✓ | | -| Upserts (INSERT ... ON CONFLICT DO NOTHING/UPDATE) | ✓ | | +| FROM, WHERE, GROUP BY, HAVING, DISTINCT, LIMIT/OFFSET, WITH queries| ✓ | [Group data](../queries/#group-data) | +| EXPLAIN query plans| ✓ | [Analyze queries with EXPLAIN](../../query-1-performance/explain-analyze/) | +| JOINs (INNER/OUTER, LEFT/RIGHT) | ✓ | [Join columns](../queries/#join-columns) | +| Expressions and Operators| ✓ | [Expressions and operators](../expressions-operators/) | +| Common Table Expressions (CTE) and Recursive Queries| ✓ | [Recursive queries and CTEs](../queries/#recursive-queries-and-ctes) | +| Upserts (INSERT ... ON CONFLICT DO NOTHING/UPDATE) | ✓ | [Upsert](../data-manipulation/#upsert) | ### Advanced SQL | Component | Supported | Details | | :-------- | :-------: | :------ | -| Stored procedures | ✓ | | -| User-defined functions| ✓ | | -| Cursors | ✓ | | +| Stored procedures | ✓ | [Stored procedures](../stored-procedures/) | +| User-defined functions| ✓ | [Functions](../../../api/ysql/user-defined-subprograms-and-anon-blocks/#functions) | +| Cursors | ✓ | [Cursors](../advanced-features/cursor/) | | Row-level triggers (BEFORE, AFTER, INSTEAD OF) | ✓ | | | Statement-level triggers (BEFORE, AFTER, INSTEAD OF) | ✓ | | | Deferrable triggers | ✗ | | | Transition tables (REFERENCING clause for triggers) | ✗ | | -| Sequences | ✓ | | +| Sequences | ✓ | [Auto-Increment column values](../data-manipulation/#auto-increment-column-values) | | Identity columns | ✓ | | -| Views | ✓ | | -| Materialized views | ✓ | | -| Window functions | ✓ | | +| Views | ✓ | [Views](../advanced-features/views/) | +| Materialized views | ✓ | [Materialized views](../advanced-features/views/#materialized-views)| +| Window functions | ✓ | [Window functions](../../../api/ysql/exprs/window_functions/)| | Common table expressions | ✓| | -| Extensions| ✓| | -| Foreign data wrappers| ✓| | +| Extensions| ✓| [PostgreSQL extensions](../pg-extensions/) | +| Foreign data wrappers| ✓| [Foreign data wrappers](../advanced-features/foreign-data-wrappers/) | diff --git a/docs/content/preview/explore/ysql-language-features/stored-procedures.md b/docs/content/preview/explore/ysql-language-features/stored-procedures.md index 8c457b6f4546..32c775107bfe 100644 --- a/docs/content/preview/explore/ysql-language-features/stored-procedures.md +++ b/docs/content/preview/explore/ysql-language-features/stored-procedures.md @@ -13,13 +13,9 @@ type: docs This section describes how to use stored procedures to perform transactions. -## Overview - -Stored procedures, in large part, are just functions that support transactions. PostgreSQL 11 introduced stored procedures, and Yugabyte supports them as well. - ## Create a stored procedure -To create a stored procedure in YSQL, use the [`CREATE PROCEDURE`](../../../api/ysql/the-sql-language/statements/ddl_create_procedure/) statement, which has the following syntax: +Stored procedures, in large part, are just functions that support transactions. To create a stored procedure in YSQL, use the [`CREATE PROCEDURE`](../../../api/ysql/the-sql-language/statements/ddl_create_procedure/) statement, which has the following syntax: ```sql CREATE [OR REPLACE] PROCEDURE procedure_name(parameter_list) @@ -74,6 +70,8 @@ If the name of the stored procedure is not unique (for example, if you had two ` ## Example workflow +{{% explore-setup-single %}} + In the following example, you create a new table and a stored procedure to perform operations on that table. Finally, you clean up by removing the procedure and the table. 1. Create an `accounts` table with two users, and set the balance of both accounts to $10,000: diff --git a/docs/content/preview/explore/ysql-language-features/triggers.md b/docs/content/preview/explore/ysql-language-features/triggers.md index 228799b0bf73..37fe423c04ab 100644 --- a/docs/content/preview/explore/ysql-language-features/triggers.md +++ b/docs/content/preview/explore/ysql-language-features/triggers.md @@ -13,6 +13,8 @@ type: docs This document describes how to use triggers when performing data manipulation and definition. +{{% explore-setup-single %}} + ## Overview In YSQL, a function invoked automatically when an event associated with a table occurs is called a trigger. The event is typically caused by modification of data during `INSERT`, `UPDATE`, and `DELETE`. The even can also be caused by schema changes. @@ -52,6 +54,8 @@ ON tbl_name [FOR [EACH] { ROW | STATEMENT }] The trigger *tr_name* fires before or after *event* which can be set to `INSERT` , `DELETE`, `UPDATE`, or `TRUNCATE`. *tbl_name* represents the table associated with the trigger. If you use the `FOR EACH ROW` clause, the scope of the trigger would be one row. If you use the `FOR EACH STATEMENT` clause, the trigger would be fired for each statement. *trigger_function* represents the procedure to be performed when the trigger is fired. +### Example + Suppose you work with a database that includes the following table populated with data: ```sql @@ -159,6 +163,8 @@ DROP TRIGGER [IF EXISTS] tr_name *tr_name* represents the trigger to be deleted if it exists. If you try to delete a non-existing trigger without using the `IF EXISTS` statement, the `DROP TRIGGER` statement results in an error, whereas using `IF EXISTS` to delete a non-existing trigger results in a notice. *tbl_name* represents the table associated with the trigger. The `CASCADE` option allows you to automatically delete objects that depend on the trigger and the `RESTRICT` option (default) allows you to refuse to delete the trigger if it has dependent objects. +### Example + The following example demonstrates how to delete the `dept_changes` trigger used in the examples from [Create triggers](#create-triggers): ```sql @@ -176,6 +182,8 @@ ALTER TABLE tbl_name *tbl_name* represents the table whose trigger represented by *tr_name* you are disabling. Using the `ALL` option allows you to disable all triggers associated with the table. A disabled trigger, even though it exists in the database, cannot fire on an event associated with this trigger. +### Examples + The following example shows how to disable a trigger on the `employees` table: ```sql @@ -217,7 +225,7 @@ ALTER TABLE employees The main difference between regular triggers and event triggers is that the former capture data manipulation events on a single table, whereas the latter can capture data definition events on a database. -The `CREATE EVENT TRIGGER` statement has he following syntax: +The `CREATE EVENT TRIGGER` statement has the following syntax: ```sql CREATE EVENT TRIGGER tr_name ON event @@ -227,6 +235,8 @@ CREATE EVENT TRIGGER tr_name ON event *tr_name*, which is unique in the database, represents the new trigger. *event* represents the event that triggers a call to the function *function_name* whose return type is `event_trigger` (optional). You can define more than one trigger for the same event, in which case the triggers fire in alphabetical order based on the name of the trigger. If a `WHEN` condition is included in the `CREATE EVENT TRIGGER` statement, then the trigger is fired for specific commands. *filter_variable* needs to be set to`TAG`, as this is the only supported variable, and *filter_value* represents a list of values for *filter_variable*. +### Example + The following example is based on examples from [Create triggers](#create-triggers), except that the `record_dept_changes` function returns an event trigger instead of a regular trigger. The example shows how to create an `sql_drop` trigger for one of the events currently supported by YSQL: ```sql diff --git a/docs/content/stable/deploy/multi-dc/async-replication.md b/docs/content/stable/deploy/multi-dc/async-replication.md index 201c91836ce4..7311411e5496 100644 --- a/docs/content/stable/deploy/multi-dc/async-replication.md +++ b/docs/content/stable/deploy/multi-dc/async-replication.md @@ -132,7 +132,7 @@ Replication lag is computed at the tablet level as follows: *hybrid_clock_time* is the hybrid clock timestamp on the source's tablet-server, and *last_read_hybrid_time* is the hybrid clock timestamp of the latest record pulled from the source. -To obtain information about the overall maximum lag, you should check `/metrics` or `/prometheus-metrics` for `async_replication_sent_lag_micros` or `async_replication_committed_lag_micros` and take the maximum of these values across each source's T-Server. For information on how to set up the node exporter and Prometheus manually, see [Prometheus integration](../../../explore/observability/prometheus-integration/linux/). +To obtain information about the overall maximum lag, you should check `/metrics` or `/prometheus-metrics` for `async_replication_sent_lag_micros` or `async_replication_committed_lag_micros` and take the maximum of these values across each source's T-Server. For information on how to set up the node exporter and Prometheus manually, see [Prometheus integration](../../../explore/observability/prometheus-integration/macos/). ## Set up replication with TLS diff --git a/docs/content/stable/explore/observability/prometheus-integration/docker.md b/docs/content/stable/explore/observability/prometheus-integration/docker.md deleted file mode 100644 index b7e037207e2f..000000000000 --- a/docs/content/stable/explore/observability/prometheus-integration/docker.md +++ /dev/null @@ -1,244 +0,0 @@ ---- -title: Prometheus integration -headerTitle: Prometheus integration -linkTitle: Prometheus integration -description: Learn about exporting YugabyteDB metrics and monitoring the cluster with Prometheus. -menu: - stable: - identifier: observability-3-docker - parent: explore-observability - weight: 235 -type: docs ---- - - - -You can monitor your local YugabyteDB cluster with a local instance of [Prometheus](https://prometheus.io/), a popular standard for time-series monitoring of cloud native infrastructure. YugabyteDB services and APIs expose metrics in the Prometheus format at the `/prometheus-metrics` endpoint. For details on the metrics targets for YugabyteDB, see [Prometheus monitoring](../../../../reference/configuration/default-ports/#prometheus-monitoring). - -This tutorial uses the [yugabyted](../../../../reference/configuration/yugabyted/) local cluster management utility. - -## 1. Create universe - -Start a new local universe with replication factor of `3`. - -```sh -$ docker network create -d bridge yb-net -``` - -```sh -$ docker run -d --name yugabyte-node1 \ - --network yb-net \ - -p 127.0.0.1:7000:7000 \ - -p 127.0.0.1:9000:9000 \ - -p 127.0.0.1:5433:5433 \ - -p 127.0.0.1:9042:9042 \ - -p 127.0.0.1:6379:6379 \ - yugabytedb/yugabyte:latest bin/yugabyted start --daemon=false --listen=yugabyte-node1 --tserver_flags="start_redis_proxy=true" -``` - -```sh -$ docker run -d --name yugabyte-node2 \ - --network yb-net \ - -p 127.0.0.2:7000:7000 \ - -p 127.0.0.2:9000:9000 \ - -p 127.0.0.2:5433:5433 \ - -p 127.0.0.2:9042:9042 \ - -p 127.0.0.2:6379:6379 \ - yugabytedb/yugabyte:latest bin/yugabyted start --daemon=false --listen=yugabyte-node2 --join=yugabyte-node1 --tserver_flags="start_redis_proxy=true" -``` - -```sh -$ docker run -d --name yugabyte-node3 \ - --network yb-net \ - -p 127.0.0.3:7000:7000 \ - -p 127.0.0.3:9000:9000 \ - -p 127.0.0.3:5433:5433 \ - -p 127.0.0.3:9042:9042 \ - -p 127.0.0.3:6379:6379 \ - yugabytedb/yugabyte:latest bin/yugabyted start --daemon=false --listen=yugabyte-node3 --join=yugabyte-node1 --tserver_flags="start_redis_proxy=true" -``` - -## 2. Run the YugabyteDB workload generator - -Pull the [yb-sample-apps](https://github.com/yugabyte/yb-sample-apps) Docker container image. This container image has built-in Java client programs for various workloads including SQL inserts and updates. - -```sh -$ docker pull yugabytedb/yb-sample-apps -``` - -Run the `CassandraKeyValue` workload application in a separate shell. - -```sh -$ docker run --name yb-sample-apps --hostname yb-sample-apps --net yb-net yugabytedb/yb-sample-apps \ - --workload CassandraKeyValue \ - --nodes yugabyte-node1:9042 \ - --num_threads_write 1 \ - --num_threads_read 4 -``` - -## 3. Prepare Prometheus configuration file - -Copy the following into a file called `yugabytedb.yml`. Move this file to the `/tmp` directory so that you can bind the file to the Prometheus container later on. - -```yaml -global: - scrape_interval: 5s # Set the scrape interval to every 5 seconds. Default is every 1 minute. - evaluation_interval: 5s # Evaluate rules every 5 seconds. The default is every 1 minute. - # scrape_timeout is set to the global default (10s). - -# YugabyteDB configuration to scrape Prometheus time-series metrics -scrape_configs: - - job_name: "yugabytedb" - metrics_path: /prometheus-metrics - relabel_configs: - - target_label: "node_prefix" - replacement: "cluster-1" - metric_relabel_configs: - # Save the name of the metric so we can group_by since we cannot by __name__ directly... - - source_labels: ["__name__"] - regex: "(.*)" - target_label: "saved_name" - replacement: "$1" - # The following basically retrofit the handler_latency_* metrics to label format. - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(.*)" - target_label: "server_type" - replacement: "$1" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(.*)" - target_label: "service_type" - replacement: "$2" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(_sum|_count)?" - target_label: "service_method" - replacement: "$3" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(_sum|_count)?" - target_label: "__name__" - replacement: "rpc_latency$4" - - static_configs: - - targets: ["yugabyte-node1:7000", "yugabyte-node2:7000", "yugabyte-node3:7000"] - labels: - export_type: "master_export" - - - targets: ["yugabyte-node1:9000", "yugabyte-node2:9000", "yugabyte-node3:9000"] - labels: - export_type: "tserver_export" - - - targets: ["yugabyte-node1:12000", "yugabyte-node2:12000", "yugabyte-node3:12000"] - labels: - export_type: "cql_export" - - - targets: ["yugabyte-node1:13000", "yugabyte-node2:13000", "yugabyte-node3:13000"] - labels: - export_type: "ysql_export" - - - targets: ["yugabyte-node1:11000", "yugabyte-node2:11000", "yugabyte-node3:11000"] - labels: - export_type: "redis_export" -``` - -## 4. Start Prometheus server - -Start the Prometheus server as below. The `prom/prometheus` container image will be pulled from the Docker registry if not already present on the localhost. - -```sh -$ docker run \ - -p 9090:9090 \ - -v /tmp/yugabytedb.yml:/etc/prometheus/prometheus.yml \ - --net yb-net \ - prom/prometheus -``` - -Open the Prometheus UI at and then navigate to the Targets page under Status. - -![Prometheus Targets](/images/ce/prom-targets-docker.png) - -## 5. Analyze key metrics - -On the Prometheus Graph UI, you can now plot the read/write throughput and latency for the `CassandraKeyValue` sample app. As you can see from the [source code](https://github.com/yugabyte/yugabyte-db/blob/master/java/yb-loadtester/src/main/java/com/yugabyte/sample/apps/CassandraKeyValue.java) of the app, it uses only SELECT statements for reads and INSERT statements for writes (aside from the initial CREATE TABLE). This means you can measure throughput and latency by simply using the metrics corresponding to the SELECT and INSERT statements. - -Paste the following expressions into the **Expression** box and click **Execute** followed by **Add Graph**. - -### Throughput - -> Read IOPS - -```sh -sum(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-read-iops.png) - -> Write IOPS - -```sh -sum(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-write-iops.png) - -### Latency - -> Read Latency (in microseconds) - -```sh -avg(irate(rpc_latency_sum{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) / -avg(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-read-latency.png) - -> Write Latency (in microseconds) - -```sh -avg(irate(rpc_latency_sum{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) / -avg(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-write-latency.png) - -## 6. Clean up (optional) - -Optionally, you can shut down the local cluster created in Step 1. - -```sh -$ docker stop yugabyte-node1 yugabyte-node2 yugabyte-node3 -$ docker rm yugabyte-node1 yugabyte-node2 yugabyte-node3 -$ docker network remove yb-net -``` - -## What's next? - -Set up [Grafana dashboards](../../grafana-dashboard/grafana/) for better visualization of the metrics being collected by Prometheus. diff --git a/docs/content/stable/explore/observability/prometheus-integration/kubernetes.md b/docs/content/stable/explore/observability/prometheus-integration/kubernetes.md deleted file mode 100644 index 06714ee9ebec..000000000000 --- a/docs/content/stable/explore/observability/prometheus-integration/kubernetes.md +++ /dev/null @@ -1,79 +0,0 @@ ---- -title: Prometheus integration -headerTitle: Prometheus integration -linkTitle: Prometheus integration -description: Learn about exporting YugabyteDB metrics and monitoring the cluster with Prometheus. -menu: - stable: - identifier: observability-4-kubernetes - parent: explore-observability - weight: 235 -type: docs ---- - - - -You can monitor your local YugabyteDB cluster with a local instance of [Prometheus](https://prometheus.io/), a popular standard for time-series monitoring of cloud native infrastructure. YugabyteDB services and APIs expose metrics in the Prometheus format at the `/prometheus-metrics` endpoint. - -For details on the metrics targets for YugabyteDB, see [Prometheus monitoring](../../../../reference/configuration/default-ports/#prometheus-monitoring). - -If you haven't installed YugabyteDB yet, do so first by following the [Quick Start](../../../../quick-start/install/macos/) guide. - -## 1. Create universe - -If you have a previously running local universe, destroy it using the following. - -```sh -$ kubectl delete -f yugabyte-statefulset.yaml -``` - -Start a new local cluster - by default, this will create a three-node universe with a replication factor of `3`. - -```sh -$ kubectl apply -f yugabyte-statefulset.yaml -``` - -## Step 6. Clean up (optional) - -Optionally, you can shut down the local cluster created in Step 1. - -```sh -$ kubectl delete -f yugabyte-statefulset.yaml -``` - -Further, to destroy the persistent volume claims (**you will lose all the data if you do this**), run: - -```sh -kubectl delete pvc -l app=yb-master -kubectl delete pvc -l app=yb-tserver -``` diff --git a/docs/content/stable/explore/observability/prometheus-integration/linux.md b/docs/content/stable/explore/observability/prometheus-integration/linux.md deleted file mode 100644 index 9f28f757a640..000000000000 --- a/docs/content/stable/explore/observability/prometheus-integration/linux.md +++ /dev/null @@ -1,242 +0,0 @@ ---- -title: Prometheus integration -headerTitle: Prometheus integration -linkTitle: Prometheus integration -description: Learn about exporting YugabyteDB metrics and monitoring the cluster with Prometheus. -menu: - stable: - identifier: observability-2-linux - parent: explore-observability - weight: 235 -type: docs ---- - - - -You can monitor your local YugabyteDB cluster with a local instance of [Prometheus](https://prometheus.io/), a popular standard for time-series monitoring of cloud native infrastructure. YugabyteDB services and APIs expose metrics in the Prometheus format at the `/prometheus-metrics` endpoint. For details on the metrics targets for YugabyteDB, see [Prometheus monitoring](../../../../reference/configuration/default-ports/#prometheus-monitoring). - -This tutorial uses the [yugabyted](../../../../reference/configuration/yugabyted/) cluster management utility. - -## Prerequisite - -Prometheus is installed on your local machine. If you have not done so already, follow the links below. - -- [Download Prometheus](https://prometheus.io/download/) -- [Get Started with Prometheus](https://prometheus.io/docs/prometheus/latest/getting_started/) - -## 1. Create universe - -Start a new local three-node universe with a replication factor of `3`. - -```sh -$ ./bin/yugabyted start \ - --base_dir=node-1 \ - --listen=127.0.0.1 \ - --tserver_flags="start_redis_proxy=true" -``` - -```sh -$ ./bin/yugabyted start \ - --base_dir=node-2 \ - --listen=127.0.0.2 \ - --join=127.0.0.1 \ - --tserver_flags="start_redis_proxy=true" -``` - -```sh -$ ./bin/yugabyted start \ - --base_dir=node-3 \ - --listen=127.0.0.3 \ - --join=127.0.0.1 \ - --tserver_flags="start_redis_proxy=true" -``` - -## 2. Run the YugabyteDB workload generator - -Download the [YugabyteDB workload generator](https://github.com/yugabyte/yb-sample-apps) JAR file (`yb-sample-apps.jar`) by running the following command. - -```sh -$ wget https://github.com/yugabyte/yb-sample-apps/releases/download/1.3.9/yb-sample-apps.jar?raw=true -O yb-sample-apps.jar -``` - -Run the `CassandraKeyValue` workload application in a separate shell. - -```sh -$ java -jar ./yb-sample-apps.jar \ - --workload CassandraKeyValue \ - --nodes 127.0.0.1:9042 \ - --num_threads_read 1 \ - --num_threads_write 1 -``` - -## 3. Prepare Prometheus configuration file - -Copy the following into a file called `yugabytedb.yml`. - -```yaml -global: - scrape_interval: 5s # Set the scrape interval to every 5 seconds. Default is every 1 minute. - evaluation_interval: 5s # Evaluate rules every 5 seconds. The default is every 1 minute. - # scrape_timeout is set to the global default (10s). - -# YugabyteDB configuration to scrape Prometheus time-series metrics -scrape_configs: - - job_name: "yugabytedb" - metrics_path: /prometheus-metrics - relabel_configs: - - target_label: "node_prefix" - replacement: "cluster-1" - metric_relabel_configs: - # Save the name of the metric so we can group_by since we cannot by __name__ directly... - - source_labels: ["__name__"] - regex: "(.*)" - target_label: "saved_name" - replacement: "$1" - # The following basically retrofit the handler_latency_* metrics to label format. - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(.*)" - target_label: "server_type" - replacement: "$1" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(.*)" - target_label: "service_type" - replacement: "$2" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(_sum|_count)?" - target_label: "service_method" - replacement: "$3" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(_sum|_count)?" - target_label: "__name__" - replacement: "rpc_latency$4" - - static_configs: - - targets: ["127.0.0.1:7000", "127.0.0.2:7000", "127.0.0.3:7000"] - labels: - export_type: "master_export" - - - targets: ["127.0.0.1:9000", "127.0.0.2:9000", "127.0.0.3:9000"] - labels: - export_type: "tserver_export" - - - targets: ["127.0.0.1:12000", "127.0.0.2:12000", "127.0.0.3:12000"] - labels: - export_type: "cql_export" - - - targets: ["127.0.0.1:13000", "127.0.0.2:13000", "127.0.0.3:13000"] - labels: - export_type: "ysql_export" - - - targets: ["127.0.0.1:11000", "127.0.0.2:11000", "127.0.0.3:11000"] - labels: - export_type: "redis_export" -``` - -## 4. Start Prometheus server - -Go to the directory where Prometheus is installed and start the Prometheus server as below. - -```sh -$ ./prometheus --config.file=yugabytedb.yml -``` - -Open the Prometheus UI at and then navigate to the Targets page under Status. - -![Prometheus Targets](/images/ce/prom-targets.png) - -## 5. Analyze key metrics - -On the Prometheus Graph UI, you can now plot the read/write throughput and latency for the `CassandraKeyValue` sample app. As you can see from the [source code](https://github.com/yugabyte/yugabyte-db/blob/master/java/yb-loadtester/src/main/java/com/yugabyte/sample/apps/CassandraKeyValue.java) of the app, it uses only SELECT statements for reads and INSERT statements for writes (aside from the initial CREATE TABLE). This means you can measure throughput and latency by simply using the metrics corresponding to the SELECT and INSERT statements. - -Paste the following expressions into the **Expression** box and click **Execute** followed by **Add Graph**. - -### Throughput - -> Read IOPS - -```sh -sum(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-read-iops.png) - -> Write IOPS - -```sh -sum(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-write-iops.png) - -### Latency - -> Read Latency (in microseconds) - -```sh -avg(irate(rpc_latency_sum{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) / -avg(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-read-latency.png) - -> Write Latency (in microseconds) - -```sh -avg(irate(rpc_latency_sum{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) / -avg(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-write-latency.png) - -## 6. Clean up (optional) - -Optionally, you can shut down the local cluster created in Step 1. - -```sh -$ ./bin/yugabyted destroy \ - --base_dir=node-1 -``` - -```sh -$ ./bin/yugabyted destroy \ - --base_dir=node-2 -``` - -```sh -$ ./bin/yugabyted destroy \ - --base_dir=node-3 -``` - -## What's next? - -Set up [Grafana dashboards](../../grafana-dashboard/grafana/) for better visualization of the metrics being collected by Prometheus. diff --git a/docs/content/stable/explore/observability/prometheus-integration/macos.md b/docs/content/stable/explore/observability/prometheus-integration/macos.md index 5a8b46b4ef68..f53d404c2304 100644 --- a/docs/content/stable/explore/observability/prometheus-integration/macos.md +++ b/docs/content/stable/explore/observability/prometheus-integration/macos.md @@ -11,38 +11,6 @@ menu: type: docs --- - - You can monitor your local YugabyteDB cluster with a local instance of [Prometheus](https://prometheus.io/), a popular standard for time-series monitoring of cloud native infrastructure. YugabyteDB services and APIs expose metrics in the Prometheus format at the `/prometheus-metrics` endpoint. For details on the metrics targets for YugabyteDB, see [Prometheus monitoring](../../../../reference/configuration/default-ports/#prometheus-monitoring). This tutorial uses the [yugabyted](../../../../reference/configuration/yugabyted/) cluster management utility. diff --git a/docs/content/v2.12/explore/observability/prometheus-integration/docker.md b/docs/content/v2.12/explore/observability/prometheus-integration/docker.md deleted file mode 100644 index 639ee21bde78..000000000000 --- a/docs/content/v2.12/explore/observability/prometheus-integration/docker.md +++ /dev/null @@ -1,244 +0,0 @@ ---- -title: Prometheus integration -headerTitle: Prometheus integration -linkTitle: Prometheus integration -description: Learn about exporting YugabyteDB metrics and monitoring the cluster with Prometheus. -menu: - v2.12: - identifier: observability-3-docker - parent: explore-observability - weight: 235 -type: docs ---- - - - -You can monitor your local YugabyteDB cluster with a local instance of [Prometheus](https://prometheus.io/), a popular standard for time-series monitoring of cloud native infrastructure. YugabyteDB services and APIs expose metrics in the Prometheus format at the `/prometheus-metrics` endpoint. For details on the metrics targets for YugabyteDB, see [Prometheus monitoring](../../../../reference/configuration/default-ports/#prometheus-monitoring). - -This tutorial uses the [yugabyted](../../../../reference/configuration/yugabyted) local cluster management utility. - -## 1. Create universe - -Start a new local universe with replication factor of `3`. - -```sh -$ docker network create -d bridge yb-net -``` - -```sh -$ docker run -d --name yugabyte-node1 \ - --network yb-net \ - -p 127.0.0.1:7000:7000 \ - -p 127.0.0.1:9000:9000 \ - -p 127.0.0.1:5433:5433 \ - -p 127.0.0.1:9042:9042 \ - -p 127.0.0.1:6379:6379 \ - yugabytedb/yugabyte:latest bin/yugabyted start --daemon=false --listen=yugabyte-node1 --tserver_flags="start_redis_proxy=true" -``` - -```sh -$ docker run -d --name yugabyte-node2 \ - --network yb-net \ - -p 127.0.0.2:7000:7000 \ - -p 127.0.0.2:9000:9000 \ - -p 127.0.0.2:5433:5433 \ - -p 127.0.0.2:9042:9042 \ - -p 127.0.0.2:6379:6379 \ - yugabytedb/yugabyte:latest bin/yugabyted start --daemon=false --listen=yugabyte-node2 --join=yugabyte-node1 --tserver_flags="start_redis_proxy=true" -``` - -```sh -$ docker run -d --name yugabyte-node3 \ - --network yb-net \ - -p 127.0.0.3:7000:7000 \ - -p 127.0.0.3:9000:9000 \ - -p 127.0.0.3:5433:5433 \ - -p 127.0.0.3:9042:9042 \ - -p 127.0.0.3:6379:6379 \ - yugabytedb/yugabyte:latest bin/yugabyted start --daemon=false --listen=yugabyte-node3 --join=yugabyte-node1 --tserver_flags="start_redis_proxy=true" -``` - -## 2. Run the YugabyteDB workload generator - -Pull the [yb-sample-apps](https://github.com/yugabyte/yb-sample-apps) Docker container image. This container image has built-in Java client programs for various workloads including SQL inserts and updates. - -```sh -$ docker pull yugabytedb/yb-sample-apps -``` - -Run the `CassandraKeyValue` workload application in a separate shell. - -```sh -$ docker run --name yb-sample-apps --hostname yb-sample-apps --net yb-net yugabytedb/yb-sample-apps \ - --workload CassandraKeyValue \ - --nodes yugabyte-node1:9042 \ - --num_threads_write 1 \ - --num_threads_read 4 -``` - -## 3. Prepare Prometheus configuration file - -Copy the following into a file called `yugabytedb.yml`. Move this file to the `/tmp` directory so that you can bind the file to the Prometheus container later on. - -```yaml -global: - scrape_interval: 5s # Set the scrape interval to every 5 seconds. Default is every 1 minute. - evaluation_interval: 5s # Evaluate rules every 5 seconds. The default is every 1 minute. - # scrape_timeout is set to the global default (10s). - -# YugabyteDB configuration to scrape Prometheus time-series metrics -scrape_configs: - - job_name: "yugabytedb" - metrics_path: /prometheus-metrics - relabel_configs: - - target_label: "node_prefix" - replacement: "cluster-1" - metric_relabel_configs: - # Save the name of the metric so we can group_by since we cannot by __name__ directly... - - source_labels: ["__name__"] - regex: "(.*)" - target_label: "saved_name" - replacement: "$1" - # The following basically retrofit the handler_latency_* metrics to label format. - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(.*)" - target_label: "server_type" - replacement: "$1" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(.*)" - target_label: "service_type" - replacement: "$2" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(_sum|_count)?" - target_label: "service_method" - replacement: "$3" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(_sum|_count)?" - target_label: "__name__" - replacement: "rpc_latency$4" - - static_configs: - - targets: ["yugabyte-node1:7000", "yugabyte-node2:7000", "yugabyte-node3:7000"] - labels: - export_type: "master_export" - - - targets: ["yugabyte-node1:9000", "yugabyte-node2:9000", "yugabyte-node3:9000"] - labels: - export_type: "tserver_export" - - - targets: ["yugabyte-node1:12000", "yugabyte-node2:12000", "yugabyte-node3:12000"] - labels: - export_type: "cql_export" - - - targets: ["yugabyte-node1:13000", "yugabyte-node2:13000", "yugabyte-node3:13000"] - labels: - export_type: "ysql_export" - - - targets: ["yugabyte-node1:11000", "yugabyte-node2:11000", "yugabyte-node3:11000"] - labels: - export_type: "redis_export" -``` - -## 4. Start Prometheus server - -Start the Prometheus server as below. The `prom/prometheus` container image will be pulled from the Docker registry if not already present on the localhost. - -```sh -$ docker run \ - -p 9090:9090 \ - -v /tmp/yugabytedb.yml:/etc/prometheus/prometheus.yml \ - --net yb-net \ - prom/prometheus -``` - -Open the Prometheus UI at and then navigate to the Targets page under Status. - -![Prometheus Targets](/images/ce/prom-targets-docker.png) - -## 5. Analyze key metrics - -On the Prometheus Graph UI, you can now plot the read/write throughput and latency for the `CassandraKeyValue` sample app. As you can see from the [source code](https://github.com/yugabyte/yugabyte-db/blob/master/java/yb-loadtester/src/main/java/com/yugabyte/sample/apps/CassandraKeyValue.java) of the app, it uses only SELECT statements for reads and INSERT statements for writes (aside from the initial CREATE TABLE). This means you can measure throughput and latency by using the metrics corresponding to the SELECT and INSERT statements. - -Paste the following expressions into the **Expression** box and click **Execute** followed by **Add Graph**. - -### Throughput - -> Read IOPS - -```sh -sum(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-read-iops.png) - -> Write IOPS - -```sh -sum(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-write-iops.png) - -### Latency - -> Read Latency (in microseconds) - -```sh -avg(irate(rpc_latency_sum{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) / -avg(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-read-latency.png) - -> Write Latency (in microseconds) - -```sh -avg(irate(rpc_latency_sum{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) / -avg(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-write-latency.png) - -## 6. Clean up (optional) - -Optionally, you can shut down the local cluster created in Step 1. - -```sh -$ docker stop yugabyte-node1 yugabyte-node2 yugabyte-node3 -$ docker rm yugabyte-node1 yugabyte-node2 yugabyte-node3 -$ docker network remove yb-net -``` - -## What's next? - -Set up [Grafana dashboards](../../grafana-dashboard/grafana/) for better visualization of the metrics being collected by Prometheus. diff --git a/docs/content/v2.12/explore/observability/prometheus-integration/kubernetes.md b/docs/content/v2.12/explore/observability/prometheus-integration/kubernetes.md deleted file mode 100644 index 18e3c454e4cf..000000000000 --- a/docs/content/v2.12/explore/observability/prometheus-integration/kubernetes.md +++ /dev/null @@ -1,79 +0,0 @@ ---- -title: Prometheus integration -headerTitle: Prometheus integration -linkTitle: Prometheus integration -description: Learn about exporting YugabyteDB metrics and monitoring the cluster with Prometheus. -menu: - v2.12: - identifier: observability-4-kubernetes - parent: explore-observability - weight: 235 -type: docs ---- - - - -You can monitor your local YugabyteDB cluster with a local instance of [Prometheus](https://prometheus.io/), a popular standard for time-series monitoring of cloud native infrastructure. YugabyteDB services and APIs expose metrics in the Prometheus format at the `/prometheus-metrics` endpoint. - -For details on the metrics targets for YugabyteDB, see [Monitoring with Prometheus](../../../reference/configuration/default-ports/#monitoring-with-prometheus). - -If you haven't installed YugabyteDB yet, do so first by following the [Quick Start](../../../quick-start/install/) guide. - -## 1. Create universe - -If you have a previously running local universe, destroy it using the following. - -```sh -$ kubectl delete -f yugabyte-statefulset.yaml -``` - -Start a new local cluster - by default, this will create a three-node universe with a replication factor of `3`. - -```sh -$ kubectl apply -f yugabyte-statefulset.yaml -``` - -## Step 6. Clean up (optional) - -Optionally, you can shut down the local cluster created in Step 1. - -```sh -$ kubectl delete -f yugabyte-statefulset.yaml -``` - -Further, to destroy the persistent volume claims (**you will lose all the data if you do this**), run: - -```sh -kubectl delete pvc -l app=yb-master -kubectl delete pvc -l app=yb-tserver -``` diff --git a/docs/content/v2.12/explore/observability/prometheus-integration/linux.md b/docs/content/v2.12/explore/observability/prometheus-integration/linux.md deleted file mode 100644 index 4d036a9bcb1f..000000000000 --- a/docs/content/v2.12/explore/observability/prometheus-integration/linux.md +++ /dev/null @@ -1,242 +0,0 @@ ---- -title: Prometheus integration -headerTitle: Prometheus integration -linkTitle: Prometheus integration -description: Learn about exporting YugabyteDB metrics and monitoring the cluster with Prometheus. -menu: - v2.12: - identifier: observability-2-linux - parent: explore-observability - weight: 235 -type: docs ---- - - - -You can monitor your local YugabyteDB cluster with a local instance of [Prometheus](https://prometheus.io/), a popular standard for time-series monitoring of cloud native infrastructure. YugabyteDB services and APIs expose metrics in the Prometheus format at the `/prometheus-metrics` endpoint. For details on the metrics targets for YugabyteDB, see [Prometheus monitoring](../../../../reference/configuration/default-ports/#prometheus-monitoring). - -This tutorial uses the [yugabyted](../../../../reference/configuration/yugabyted) cluster management utility. - -## Prerequisite - -Prometheus is installed on your local machine. If you have not done so already, follow the links below. - -- [Download Prometheus](https://prometheus.io/download/) -- [Get Started with Prometheus](https://prometheus.io/docs/prometheus/latest/getting_started/) - -## 1. Create universe - -Start a new local three-node universe with a replication factor of `3`. - -```sh -$ ./bin/yugabyted start \ - --base_dir=node-1 \ - --listen=127.0.0.1 \ - --tserver_flags="start_redis_proxy=true" -``` - -```sh -$ ./bin/yugabyted start \ - --base_dir=node-2 \ - --listen=127.0.0.2 \ - --join=127.0.0.1 \ - --tserver_flags="start_redis_proxy=true" -``` - -```sh -$ ./bin/yugabyted start \ - --base_dir=node-3 \ - --listen=127.0.0.3 \ - --join=127.0.0.1 \ - --tserver_flags="start_redis_proxy=true" -``` - -## 2. Run the YugabyteDB workload generator - -Download the [YugabyteDB workload generator](https://github.com/yugabyte/yb-sample-apps) JAR file (`yb-sample-apps.jar`) by running the following command. - -```sh -$ wget https://github.com/yugabyte/yb-sample-apps/releases/download/1.3.9/yb-sample-apps.jar?raw=true -O yb-sample-apps.jar -``` - -Run the `CassandraKeyValue` workload application in a separate shell. - -```sh -$ java -jar ./yb-sample-apps.jar \ - --workload CassandraKeyValue \ - --nodes 127.0.0.1:9042 \ - --num_threads_read 1 \ - --num_threads_write 1 -``` - -## 3. Prepare Prometheus configuration file - -Copy the following into a file called `yugabytedb.yml`. - -```yaml -global: - scrape_interval: 5s # Set the scrape interval to every 5 seconds. Default is every 1 minute. - evaluation_interval: 5s # Evaluate rules every 5 seconds. The default is every 1 minute. - # scrape_timeout is set to the global default (10s). - -# YugabyteDB configuration to scrape Prometheus time-series metrics -scrape_configs: - - job_name: "yugabytedb" - metrics_path: /prometheus-metrics - relabel_configs: - - target_label: "node_prefix" - replacement: "cluster-1" - metric_relabel_configs: - # Save the name of the metric so we can group_by since we cannot by __name__ directly... - - source_labels: ["__name__"] - regex: "(.*)" - target_label: "saved_name" - replacement: "$1" - # The following basically retrofit the handler_latency_* metrics to label format. - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(.*)" - target_label: "server_type" - replacement: "$1" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(.*)" - target_label: "service_type" - replacement: "$2" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(_sum|_count)?" - target_label: "service_method" - replacement: "$3" - - source_labels: ["__name__"] - regex: "handler_latency_(yb_[^_]*)_([^_]*)_([^_]*)(_sum|_count)?" - target_label: "__name__" - replacement: "rpc_latency$4" - - static_configs: - - targets: ["127.0.0.1:7000", "127.0.0.2:7000", "127.0.0.3:7000"] - labels: - export_type: "master_export" - - - targets: ["127.0.0.1:9000", "127.0.0.2:9000", "127.0.0.3:9000"] - labels: - export_type: "tserver_export" - - - targets: ["127.0.0.1:12000", "127.0.0.2:12000", "127.0.0.3:12000"] - labels: - export_type: "cql_export" - - - targets: ["127.0.0.1:13000", "127.0.0.2:13000", "127.0.0.3:13000"] - labels: - export_type: "ysql_export" - - - targets: ["127.0.0.1:11000", "127.0.0.2:11000", "127.0.0.3:11000"] - labels: - export_type: "redis_export" -``` - -## 4. Start Prometheus server - -Go to the directory where Prometheus is installed and start the Prometheus server as below. - -```sh -$ ./prometheus --config.file=yugabytedb.yml -``` - -Open the Prometheus UI at and then navigate to the Targets page under Status. - -![Prometheus Targets](/images/ce/prom-targets.png) - -## 5. Analyze key metrics - -On the Prometheus Graph UI, you can now plot the read/write throughput and latency for the `CassandraKeyValue` sample app. As you can see from the [source code](https://github.com/yugabyte/yugabyte-db/blob/master/java/yb-loadtester/src/main/java/com/yugabyte/sample/apps/CassandraKeyValue.java) of the app, it uses only SELECT statements for reads and INSERT statements for writes (aside from the initial CREATE TABLE). This means you can measure throughput and latency by using the metrics corresponding to the SELECT and INSERT statements. - -Paste the following expressions into the **Expression** box and click **Execute** followed by **Add Graph**. - -### Throughput - -> Read IOPS - -```sh -sum(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-read-iops.png) - -> Write IOPS - -```sh -sum(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-write-iops.png) - -### Latency - -> Read Latency (in microseconds) - -```sh -avg(irate(rpc_latency_sum{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) / -avg(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="SelectStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-read-latency.png) - -> Write Latency (in microseconds) - -```sh -avg(irate(rpc_latency_sum{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) / -avg(irate(rpc_latency_count{server_type="yb_cqlserver", service_type="SQLProcessor", service_method="InsertStmt"}[1m])) -``` - -![Prometheus Read IOPS](/images/ce/prom-write-latency.png) - -## 6. Clean up (optional) - -Optionally, you can shut down the local cluster created in Step 1. - -```sh -$ ./bin/yugabyted destroy \ - --base_dir=node-1 -``` - -```sh -$ ./bin/yugabyted destroy \ - --base_dir=node-2 -``` - -```sh -$ ./bin/yugabyted destroy \ - --base_dir=node-3 -``` - -## What's next? - -Set up [Grafana dashboards](../../grafana-dashboard/grafana/) for better visualization of the metrics being collected by Prometheus. diff --git a/docs/content/v2.12/explore/observability/prometheus-integration/macos.md b/docs/content/v2.12/explore/observability/prometheus-integration/macos.md index 6213a2ddf889..914a18d2955f 100644 --- a/docs/content/v2.12/explore/observability/prometheus-integration/macos.md +++ b/docs/content/v2.12/explore/observability/prometheus-integration/macos.md @@ -11,38 +11,6 @@ menu: type: docs --- - - You can monitor your local YugabyteDB cluster with a local instance of [Prometheus](https://prometheus.io/), a popular standard for time-series monitoring of cloud native infrastructure. YugabyteDB services and APIs expose metrics in the Prometheus format at the `/prometheus-metrics` endpoint. For details on the metrics targets for YugabyteDB, see [Prometheus monitoring](../../../../reference/configuration/default-ports/#prometheus-monitoring). This tutorial uses the [yugabyted](../../../../reference/configuration/yugabyted) cluster management utility. diff --git a/docs/content/v2.6/explore/observability/prometheus-integration/docker.md b/docs/content/v2.6/explore/observability/prometheus-integration/docker.md index 3fd1366494f6..e332942d46ac 100644 --- a/docs/content/v2.6/explore/observability/prometheus-integration/docker.md +++ b/docs/content/v2.6/explore/observability/prometheus-integration/docker.md @@ -14,21 +14,21 @@ type: docs
  • - + macOS
  • - + Linux
  • - + Docker diff --git a/docs/content/v2.6/explore/observability/prometheus-integration/kubernetes.md b/docs/content/v2.6/explore/observability/prometheus-integration/kubernetes.md index 2db2cf6b6445..d72c3c0eabe3 100644 --- a/docs/content/v2.6/explore/observability/prometheus-integration/kubernetes.md +++ b/docs/content/v2.6/explore/observability/prometheus-integration/kubernetes.md @@ -14,21 +14,21 @@ type: docs
    • - + macOS
    • - + Linux
    • - + Docker diff --git a/docs/content/v2.6/explore/observability/prometheus-integration/linux.md b/docs/content/v2.6/explore/observability/prometheus-integration/linux.md index b4b3c7654421..52eeada09700 100644 --- a/docs/content/v2.6/explore/observability/prometheus-integration/linux.md +++ b/docs/content/v2.6/explore/observability/prometheus-integration/linux.md @@ -14,21 +14,21 @@ type: docs
      • - + macOS
      • - + Linux
      • - + Docker diff --git a/docs/content/v2.6/explore/observability/prometheus-integration/macos.md b/docs/content/v2.6/explore/observability/prometheus-integration/macos.md index 85d671a6ae88..1f40e52ad8e0 100644 --- a/docs/content/v2.6/explore/observability/prometheus-integration/macos.md +++ b/docs/content/v2.6/explore/observability/prometheus-integration/macos.md @@ -14,21 +14,21 @@ type: docs
        • - + macOS
        • - + Linux
        • - + Docker diff --git a/docs/content/v2.8/explore/observability/prometheus-integration/kubernetes.md b/docs/content/v2.8/explore/observability/prometheus-integration/kubernetes.md index 1871b73483ff..1d76827b4123 100644 --- a/docs/content/v2.8/explore/observability/prometheus-integration/kubernetes.md +++ b/docs/content/v2.8/explore/observability/prometheus-integration/kubernetes.md @@ -14,21 +14,21 @@ type: docs
          • - + macOS
          • - + Linux
          • - + Docker diff --git a/docs/layouts/shortcodes/explore-setup-single.html b/docs/layouts/shortcodes/explore-setup-single.html new file mode 100644 index 000000000000..96dfce8a4b33 --- /dev/null +++ b/docs/layouts/shortcodes/explore-setup-single.html @@ -0,0 +1,4 @@ +
            +

            Setup

            + Local single-node cluster or use the Sandbox cluster in YugabyteDB Managed. See Set up your YugabyteDB cluster. +
            \ No newline at end of file diff --git a/docs/netlify.toml b/docs/netlify.toml index 2e6731332399..a605a300cbfb 100644 --- a/docs/netlify.toml +++ b/docs/netlify.toml @@ -459,6 +459,12 @@ from = "/preview/explore/fault-tolerance/*" to = "/preview/explore/fault-tolerance/macos/" +# Prometheus integration redirects + +[[redirects]] + from = "/preview/explore/observability/prometheus-integration/*" + to = "/preview/explore/observability/prometheus-integration/macos/" + # Hugo resource caching plugin configuration # https://github.com/cdeleeuwe/netlify-plugin-hugo-cache-resources#readme diff --git a/docs/static/images/section_icons/architecture/Reusing-PostgreSQL-query-layer.png b/docs/static/images/section_icons/architecture/Reusing-PostgreSQL-query-layer.png index 068000533b05..fd6519586b6c 100644 Binary files a/docs/static/images/section_icons/architecture/Reusing-PostgreSQL-query-layer.png and b/docs/static/images/section_icons/architecture/Reusing-PostgreSQL-query-layer.png differ
session #1session #2
- By default, the tserver gflag yb_enable_read_committed_isolation=false. In this case, Read Committed maps to Snapshot Isolation at the transactional layer. So, READ COMMITTED of YSQL API in turn maps to Snapshot Isolation. - 
BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;
-SELECT * FROM test;

+
+ +By default, the YB-TServer flag `yb_enable_read_committed_isolation` is false. In this case, Read Committed maps to Snapshot Isolation at the transactional layer. So, READ COMMITTED of YSQL API in turn maps to Snapshot Isolation. + +```sql +BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED; +SELECT * FROM test; +``` + +```output k | v ---+--- 1 | 2 -(1 row) - +(1 row) +``` + +
- - Insert a new row. - 
INSERT INTO test VALUES (2, 3);
+ 		+ +Insert a new row. + +```sql +INSERT INTO test VALUES (2, 3); +``` + +
- Perform read again in the same transaction. Note that the recently inserted row (2, 3) isn't - visible to the statement because Read Committed is disabled at the transactional layer and maps to - Snapshot (in which the whole transaction sees a consistent snapshot of the database). - 
SELECT * FROM test;
-COMMIT;

+
+ +Perform the read again in the same transaction. + +```sql +SELECT * FROM test; +COMMIT; +``` + +```output k | v ---+--- 1 | 2 -(1 row) - -
- Set tserver gflag yb_enable_read_committed_isolation=true - 
BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;
-SELECT * FROM test;

+The inserted row (2, 3) isn't visible because Read Committed is disabled at the transactional layer and maps to Snapshot (in which the whole transaction sees a consistent snapshot of the database).
+
+Set the YB-Tserver flag yb_enable_read_committed_isolation=true.
+
+```sql
+BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;
+SELECT * FROM test;
+```
+
+```output
  k | v
 ---+---
  1 | 2
  2 | 3
-(2 rows)
- 		+(2 rows) +``` + +
- - In another session, insert a new row. - 
INSERT INTO test VALUES (3, 4);
+ 		+ +Insert a new row. + +```sql +INSERT INTO test VALUES (3, 4); +``` + +
- Perform read again in the same transaction. This time, the statement will be able to see the - row (3, 4) that was committed after this transaction was started but before the statement was issued. - 

+
+ +Perform the read again in the same transaction. + +```sql SELECT * FROM test; +``` + +```output k | v ---+--- 1 | 2 2 | 3 3 | 4 (3 rows) - - +``` + +This time, the statement can see the row (3, 4) that was committed after this transaction was started but before the statement was issued. + +