Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs][core] new quickstart tutorial #11620

Closed
wants to merge 29 commits into from

Conversation

ddhodge
Copy link
Contributor

@ddhodge ddhodge commented Mar 1, 2022

Update the quick start with the new YSQL dataset developed for YB Cloud; move the old tutorial to the Data Analytics data set page.

@netlify /latest/quick-start/explore/ysql/

@ddhodge ddhodge self-assigned this Mar 1, 2022
@ddhodge ddhodge added the area/documentation Documentation needed label Mar 1, 2022
@netlify
Copy link

netlify bot commented Mar 1, 2022

✔️ Deploy Preview for infallible-bardeen-164bc9 ready!

🔨 Explore the source changes: b6fe515

🔍 Inspect the deploy log: https://app.netlify.com/sites/infallible-bardeen-164bc9/deploys/6220f36710ec860007f97b79

😎 Browse the preview: https://deploy-preview-11620--infallible-bardeen-164bc9.netlify.app/latest/quick-start/explore/ysql/

jharveysmith and others added 25 commits March 1, 2022 19:45
Summary:
Rename packages to include OS version number.  This is really a continuation of D15600, we
have the version naming logic spread between two files.

Test Plan: Create some releases and see their names.

Reviewers: devops, steve.varnau

Reviewed By: steve.varnau

Subscribers: jenkins-bot, devops

Differential Revision: https://phabricator.dev.yugabyte.com/D15736
Summary: - Modified list backups API

Test Plan: - Verified List backups API response

Reviewers: vbansal, vpatibandla

Reviewed By: vpatibandla

Subscribers: jenkins-bot

Differential Revision: https://phabricator.dev.yugabyte.com/D15720
…tificates

Summary:
When generating node certificates we go with the default of 4 years as expiry date
But when user uploaded a self signed certificate the rootca expiry can be different
This diff updates the node certificate creation code such that it always uses root CA start date and
end date

Test Plan:
Created universe with TLS on. Verified certs are created without any issues and with expected expiry
dates

Reviewers: sanketh, arnav

Reviewed By: arnav

Subscribers: yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D15659
Summary:
**Issue:**
cqlsh not able to connect when self-signed certs are rotated with only client-to-node enabled

**Context:**
For this particular case of cert rotation. following is the context:
- When enabling only client-to-node while universe creation, rootAndClientRootCASame is set to true and certs will be present in yugabyte-tls-config and client certs in .yugabytedb directories
- During cert rotation from UI, rootAndClientRootCASame will be sent as false and clientRootCA will beset. This results in rotation of certificates from yugabyte-tls-config to yugabyte-client-tls-config. And we don't update .yugabytedb directory as we don't create client certs when clientRootCA is set
- During cert rotation we don't clean up old certs. So in this case certs in yugabyte-tls-config and .yugabytedb are untouched

**Cause:**
- UPDATE_CERT_DIRS case got deleted by mistake in this commit: (yugabyte@0a5e0d0#diff-283fac6bc255eee9a87e254e9de15ebc4de4b14146ec5ca2a83f20cb69c9985aL922). This handles updating cert_dir and cert_for_client_dir gflags during cert rotation. This was main root cause of cqlsh failure as DB is still referring to yugabyte-tls-config after cert rotation
- Noticed another bug in non-rolling cert rotate flow where gflags are updated after restart. Because of this even after adding back UPDATE_CERT_DIRS logic gflags are not getting updated
- After fixing the above two causes, cqlsh will connect fine but ysqlsh fails. This is because of the traces of old certs in .yugabytedb. In ysqlsh if no certs are provided it will default to the certs present in the home directory

**Fixes:**
- Revert the changes in this commit: yugabyte@cd48b7a as they are causing itest failures
- Add the accidentally removed code of updating gflags during cert rotation in NodeManager.java
- Move the gflags update before the restart in case of non-rolling cert rotation flow
- Add function to cleanup client certs in .yugabytedb directory before any cert rotation
- Remove checkForHashicorpVaultCerts function and refactor the logic to prevent NPEs in case rootCA is null
- Code cleanup in EncryptionInTransitUtil.java and VaultPKI.java

Test Plan:
Tested cert rotation flow with only client-to-node enabled
Verified ysqlsh and ycqlsh connectivity
Verified the same for the case when both node-to-node and client-to-node enabled
Tested both rolling and non-rolling upgrade flows

Reviewers: sanketh, arnav

Reviewed By: arnav

Subscribers: jenkins-bot, yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D15727
Summary:
The bootscript can fail when we use two NICs, so we remove the verification step if their is a

Test Plan: Create a dual NIC cluster, and verify that we don't fail in case the bootscript does not run.

Reviewers: muthu, arnav, amindrov

Reviewed By: amindrov

Subscribers: jenkins-bot, yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D15737
Summary: - Modified list Backup API response to include per-keyspace and total backup size

Test Plan: - Verified abckups size were returned in API response

Reviewers: achauhan

Reviewed By: achauhan

Subscribers: jenkins-bot, yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D15747
Summary:
It turns out that 2.9 release was actually branched before 2.8, and does not have upgrade_ysql command.
Made changes to not execute it during 2.9.x upgrade.

Test Plan:
Create 2.7 universe.
Upgrade it to 2.9 - upgrade is successful.
Create 2.7 universe.
Upgrade it to 2.8 - upgrade is successful and upgrade_ysql command was executed during upgrade.

Reviewers: spotachev

Reviewed By: spotachev

Subscribers: jenkins-bot, yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D15754
Summary:
In YBCPrepareAlterTableCmd() of ybccmds.c, if YBCPgAlterTableAddColumn() returns error, call to ReleaseSysCache() is skipped.
This revision moves the ReleaseSysCache() call earlier so that the tuple is always released.

Test Plan: Build yugabyte db and run test suite via Jenkins

Reviewers: myang, amartsinchyk

Reviewed By: myang, amartsinchyk

Subscribers: yql

Differential Revision: https://phabricator.dev.yugabyte.com/D14831
Summary:
It does not look like there is currently any convenient way to pass
extra options to the mvn subcommand in yb_build.sh --java-test.  Have
yb_build read from a new environment variable
YB_EXTRA_MVN_OPTIONS_IN_TESTS and pass those options to the mvn
subcommand used when running java tests.  Note that there is another mvn
subcommand for collecting tests: that one is unaffected.

One way this can be used is setting system properties, and future Java
tests can read custom properties and change behavior accordingly.

Use the same strategy for passing options as done for
YB_EXTRA_GTEST_FLAGS.

Test Plan:
Jenkins: java only, build platform: alma8, build type: release

    ./yb_build.sh --java-test TestPgRegressPgMisc
    # Following should show nothing:
    grep -E '(abc|bar)' java/yb-pgsql/target/surefire-reports_org.yb.pgsql.TestPgRegressPgMisc/TEST-org.yb.pgsql.TestPgRegressPgMisc.xml

    YB_EXTRA_MVN_OPTIONS_IN_TESTS='-Dabc=123 -Dfoo=bar' ./yb_build.sh --java-test TestPgRegressPgMisc
    # Following should show both properties set:
    grep -E '(abc|bar)' java/yb-pgsql/target/surefire-reports_org.yb.pgsql.TestPgRegressPgMisc/TEST-org.yb.pgsql.TestPgRegressPgMisc.xml

Reviewers: mbautin, jharveysmith, steve.varnau

Reviewed By: steve.varnau

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D15739
Summary:
We were using the static config to fetch if the cloud is enabled, but itest uses runtime
configs to set that. So, we fetch the runtime config per customer instead.

Test Plan:
Run itest for dual NIC universes D15504
https://jenkins.dev.yugabyte.com/job/dev-itest-pipeline/701/

Reviewers: anabaria, hkandala, daniel

Reviewed By: daniel

Subscribers: jenkins-bot, yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D15762
Summary:
What you enter in this field gets appended to list of cors origin whitelist.
It needs to have protocol host and port

Test Plan: NA

Reviewers: spotachev

Reviewed By: spotachev

Subscribers: yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D15767
…blespace update

Summary:
Changed the tablespace update procedure to start transaction table creation for tablespaces
which have placements set and at least one table in them (to account for tablespaces with bad
placements). This should handle both tablespaces from previous versions with no associated transaction
table, and tablespaces whose first table was through ALTER TABLE SET TABLESPACE. The transaction
table creation hook in CreateTable is left in as an optimization.

Test Plan: `ybd --cxx-test pgwrapper_geo_transactions-test --gtest_filter GeoTransactionsTest.TestAutomaticLocalTransactionTableCreationWithAlter`

Reviewers: dsrinivasan

Reviewed By: dsrinivasan

Subscribers: rthallam, bogdan, ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D15601
…g_temp_xxxx after non-concurrent refresh

Summary:
A non-concurrent refresh of the matview creates a new relation (with the name pg_temp_xxxx) and swaps the storage of the matview with that of the new relation.
Currently, after a refresh, the matview's corresponding docdb table has the name pg_temp_xxxx. This is not only confusing for the user, but it also causes problems during backup/restore. This is because the restore functionality relies on the docdb table name to reconcile the relation with the pg metadata.

This diff fixes the issue as follows:

a) When we create the new relation during a non-concurrent refresh, we pass down the matview's OID to the catalog manager and maintain the information in `matview_pg_table_ids_map_`.
b) After creating the new relation, we also change the name of the docdb table to the matview's name.

Note: when we backup/restore a matview, we will backup/restore it `WITH DATA`.

Test Plan:
Jenkins: urgent
`yb_feature_matview` in `TestPgRegressFeature`, and `TestYbBackup`

Reviewers: alex, myang, oleg

Reviewed By: myang, oleg

Subscribers: alex, jenkins-bot, smishra, yql

Differential Revision: https://phabricator.dev.yugabyte.com/D15461
(With a LOT of broken-link fixes, too)
Summary:
Two things fixed:

1. Alter SB bundleDetails column from type json -> json_alias.
2. Convert SupportBundleUtil to a singleton class without static methods. Will be useful for mocking.

Test Plan: Manually tested.

Reviewers: hkandala

Reviewed By: hkandala

Subscribers: jenkins-bot, sanketh

Differential Revision: https://phabricator.dev.yugabyte.com/D15753
…niverse of a multi-region universe

Summary: Revert ui changes from [PLAT-2449]

Test Plan: Edit universe -> change preferred zone -> save

Reviewers: kkannan

Reviewed By: kkannan

Subscribers: jenkins-bot, yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D15770
Summary: Adding support for YugabyteDB on Oracle 7 nodes.

Test Plan:
testing on gcp RHCK custom image
verify universe creation succeeds, sample apps run
verify universe creation oracle 8 still succeeds
run itest pipeline

Reviewers: hsu, sanketh

Reviewed By: sanketh

Subscribers: jenkins-bot, yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D13645
…lush_rate_limit_ flags

Summary:
Changing default values for flags `rocksdb_compact_flush_rate_limit_bytes_per_sec=1GB` and
`rocksdb_compact_flush_rate_limit_sharing_mode=tserver` on base of result of test runs for
sample apps and heavy workloads.

Test Plan:
ybd --cxx-test tserver_ts_tablet_manager-test --gtest_filter TsTabletManagerTest.RateLimiterSharing
ybd --cxx-test docdb_docdb_rocksdb_util-test --gtest_filter DocDBRocksDBUtilTest.RocksDBRateLimiter
ybd --cxx-test integration-tests_tablet-split-itest --gtest_filter TabletSplitITest.PostSplitCompactionDoesntBlockTabletCleanup
ybd --cxx-test client_ql-stress-test --gtest_filter QLStressTest.DynamicCompactionPriority

Reviewers: bogdan

Reviewed By: bogdan

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D15564
Summary:
In CopyFrom(), currently we regard non-YB table as TEMP table.

However, this produces confusing message for FOREIGN table:
```
ts1|pid52548|:30139 2022-03-01 09:16:20.854 UTC [52764] WARNING:  Batched COPY is not supported on temporary tables. Defaulting to using one transaction for the entire copy.
```
This revision checks the Relation type and uses proper table type in the message.

Test Plan: ybd --java-test 'org.yb.pgsql.TestPgRegressContribPostgresFdw#schedule'

Reviewers: ena, myang

Reviewed By: myang

Subscribers: yql

Differential Revision: https://phabricator.dev.yugabyte.com/D15734
@CLAassistant
Copy link

CLAassistant commented Mar 3, 2022

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
11 out of 18 committers have signed the CLA.

✅ ddhodge
✅ jharveysmith
✅ jaki
✅ tedyu
✅ Sahith02
✅ sb-yb
✅ aishwarya24
✅ fizaaluthra
✅ es1024
✅ lizayugabyte
✅ polarweasel
❌ kv83821-yb
❌ hkandala
❌ anmalysh-yb
❌ Arnav15
❌ arybochkin
❌ mchiddy
❌ Yury Shchetinin


Yury Shchetinin seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@ddhodge ddhodge closed this Mar 3, 2022
@ddhodge ddhodge deleted the docs_core_quickstart branch January 3, 2023 21:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/documentation Documentation needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.