Skip to content

Latest commit

 

History

History
92 lines (60 loc) · 2 KB

Usage.md

File metadata and controls

92 lines (60 loc) · 2 KB
Raw

Usage

Run query on data

qv s3://tpc-h-parquet/1/customer -q 'select c_custkey, UPPER(c_name) from tbl'

View schema of data

qv ./datasets/tpc-h-parquet/1/customer -s

View data on GCS.

Configuration

QV expects the environment variable 'GOOGLE_APPLICATION_CREDENTIALS' to exist and point to a file which contains google credentials.

qv gs://datafusion-delta-testing/data/delta/COVID-19_NYT

View data on S3

Configuration

Usually Credential loading works out of the box when using the AWS SDK for Rust.

The following environment variables are needed for credentials:

  • AWS_REGION
  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

In case you have AWS SSO credentials you need to set the following:

  • AWS_PROFILE

In case you have a custom endpoint in place (eg: minio) you also need to set: #* AWS_ENDPOINT_URL AWS_ENDPOINT AWS_ALLOW_HTTP https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3Builder.html

qv s3://tpc-h-parquet/1/customer

Specify AWS (SSO) profile to use

qv s3://tpc-h-parquet/1/customer --profile my-user

This is the same as:

AWS_PROFILE=my-user qv s3://tpc-h-parquet/1/customer

View data from S3 console URL

qv https://s3.console.aws.amazon.com/s3/buckets/datafusion-delta-testing?region=eu-central-1&prefix=simple_table/&showversions=false

View data which matches a globbing pattern:

qv "s3://datafusion-parquet-testing/data/alltypes_pla*n.parquet"

View delta table (no need for a manifest)

qv /Users/timvw/src/github/delta-rs/rust/tests/data/COVID-19_NYT

View delta table at specific point in time

qv /Users/timvw/src/github/delta-rs/rust/tests/data/COVID-19_NYT --at "2022-01-01T16:39:00+01:00"

View glue table

qv glue://mydb.table1