Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

#180 fix for destinationTable not returned #187

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

#180 fix for destinationTable not returned #187

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion server/handler.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1485,7 +1485,7 @@ func (h *jobsInsertHandler) Handle(ctx context.Context, r *jobsInsertRequest) (*
func (h *jobsInsertHandler) addQueryResultToDynamicDestinationTable(ctx context.Context, tx *connection.Tx, r *jobsInsertRequest, response *internaltypes.QueryResponse) error {
projectID := r.project.ID
jobID := r.job.JobReference.JobId
datasetID := jobID
datasetID := "ds_" + jobID
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this is needed?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

datasetId must be unique although tableId does not need to. That's why we needed.

reference:
https://cloud.google.com/ruby/docs/reference/google-cloud-bigquery/latest/Google-Cloud-Bigquery-Project#Google__Cloud__Bigquery__Project_create_dataset_instance_
https://cloud.google.com/ruby/docs/reference/google-cloud-bigquery/latest/Google-Cloud-Bigquery-Dataset#Google__Cloud__Bigquery__Dataset_create_table_instance_

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the jobID is already unique for each request. So if you just want to make datasetID unique, jobID seems fine.

Copy link

@kromiii kromiii Sep 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems fine, but actually, after deleting this ds_ I got an error below when using bigquery ruby client.
Do you know what causes this erorr? @goccy

2023-09-21 21:49:45 bq-emulator-test-bq-1  | 2023-09-21T12:49:45.555Z   ERROR   server/handler.go:2355   internalError   {"error": "internalError: failed to query SELECT * FROM `job_JSsjGMduSIBf5L_poPWkkL2vZdkd`: no such table: test_job_JSsjGMduSIBf5L_poPWkkL2vZdkd"}

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kromiii Sorry, I don't known by this information. I'd like to have reproducible test code. Go is preferred, but other languages are also acceptable. Also, please attach the logs obtained at that time.

Copy link

@kromiii kromiii Sep 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created a repo to test this. https://github.com/kromiii/bq-emulator-test
This issue depends on the ruby client, so I attached a simple ruby script for testing bigquery-emulator.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@prabhav007 Let us know the reason of ds_ if you are available.

Copy link
Author

@prabhav007 prabhav007 Oct 2, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case of jobs with anonymous tables, when the dataset name and the table name are the same, an error response is received. When they are different, it works fine. PFB the error response I got locally.

http://localhost:9050/bigquery/v2/projects/demo-0d168064/datasets/0ffd9b42-7541-4317-ad16-fad5ec3beb55/tables/0ffd9b42-7541-4317-ad16-fad5ec3beb55/data?maxResults=10&prettyPrint=false

{"error":{"errors":[{"reason":"internalError","location":"","debugInfo":"","message":"failed to query SELECT * FROM 0ffd9b42-7541-4317-ad16-fad5ec3beb55: no such table: demo-0d168064_0ffd9b42-7541-4317-ad16-fad5ec3beb55"}],"code":500,"message":"failed to query SELECT * FROM 0ffd9b42-7541-4317-ad16-fad5ec3beb55: no such table: demo-0d168064_0ffd9b42-7541-4317-ad16-fad5ec3beb55"}}

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@prabhav007 Thank you. I could reproduce the error. However, the quetion is that it works well in the python clinent even if the dataset name and the table name is the same. What makes this difference?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure, I was using the java client for spring.

tableID := jobID

tableDef, err := h.tableDefFromQueryResponse(tableID, response)
Expand Down Expand Up @@ -1521,6 +1521,11 @@ func (h *jobsInsertHandler) addQueryResultToDynamicDestinationTable(ctx context.
if err := r.server.contentRepo.AddTableData(ctx, tx, projectID, datasetID, tableDef); err != nil {
return fmt.Errorf("failed to add table data: %w", err)
}
r.job.Configuration.Query.DestinationTable = &bigqueryv2.TableReference{
DatasetId: datasetID,
ProjectId: projectID,
TableId: tableID,
}
return nil
}

Expand Down