Skip to content

Commit

Permalink
Schema Viewer Drawer (getredash#3291)
Browse files Browse the repository at this point in the history
* Process extra column metadata for a few sql-based data sources.

* Add Table and Column metadata tables.

* Periodically update table and column schema tables in a celery task.

* Fetching schema returns data from table and column metadata tables.

* Add tests for backend changes.

* Front-end shows extra table metadata and uses new schema response.

* Delete datasource schema data when deleting a data source.

* Process and store data source schema when a data source is first created or after a migration.

* Tables should have a unique name per datasource.

* Addressing review comments.

* Update migration file for mixins.

* Appease PEP8

* Upgrade migration file for rebase.

* Cascade delete.

* Adding org_id

* Remove redundant column and table prefixes.

* Non-existing tables and columns should be filtered out on the server side not client side.

* Fetching table samples should be optional and should happen in a separate task per table.

* Allow users to force a schema refresh.

* Use updated_at to help prune old schema metadata periodically.

* Using settings.SCHEMAS_REFRESH_QUEUE

* fix for getredash#2426 test

* more stable test_interactive_new

* Closes #927, #928: Schema refresh improvements.

* Closes #934, #935: Remove type from schema browser and don't show empty example column in schema drawer (#936)

* Speed up schema fetch requests with fewer postgres queries.

* Add column metadata to Athena glue processing.

* Fix bug assuming 'metadata' exists for every table.

* Closes #939: Persisted, existing table metadata should be updated.

* Sample processing should be rate-limited.

* Add cli command for refreshing data samples.

* Schema refreshes should not overwrite column 'example' field.

* refresh_samples() should filter tables_to_sample on the datasource's id being sampled

* Correctly wrap long text in schema drawer.

Schema Improvements Part 2: Add data source config options.

Adding BigQuery schema drawer with data types and samples.

Add empty migration to replace the removed schedule_until migration

Add merge migration.

Fix spacing issue with data scanned value in query execution metadata.

Increase schema refresh timeout.

Remove old migrations.

Co-authored-by: Alison <github@bankofknowledge.net>
Co-authored-by: Jannis Leidel <jannis@leidel.info>
  • Loading branch information
3 people authored and robhudson committed Jun 11, 2020
1 parent ae10e64 commit 6dd763f
Show file tree
Hide file tree
Showing 55 changed files with 2,736 additions and 237 deletions.
4 changes: 4 additions & 0 deletions client/app/assets/less/inc/base.less
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,10 @@ strong {
transition: height 0s, width 0s !important;
}

.admin-schema-editor {
padding: 50px 0;
}

.bg-ace {
background-color: fade(@redash-gray, 12%) !important;
}
Expand Down
4 changes: 3 additions & 1 deletion client/app/assets/less/inc/popover.less
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
.popover {
box-shadow: fade(@redash-gray, 25%) 0px 0px 15px 0px;
color: #000000;
z-index: 1000000001; // So that it can popover a dropdown menu
}

.popover-title {
Expand All @@ -19,4 +21,4 @@
p {
margin-bottom: 0;
}
}
}
14 changes: 9 additions & 5 deletions client/app/assets/less/inc/schema-browser.less
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,14 @@ div.table-name {
border-radius: @redash-radius;
position: relative;

.copy-to-editor {
.copy-to-editor, .info {
display: none;
}

&:hover {
background: fade(@redash-gray, 10%);

.copy-to-editor {
.copy-to-editor, .info {
display: flex;
}
}
Expand All @@ -38,7 +38,7 @@ div.table-name {
background: transparent;
}

.copy-to-editor {
.copy-to-editor, .info {
color: fade(@redash-gray, 90%);
cursor: pointer;
position: absolute;
Expand All @@ -51,21 +51,25 @@ div.table-name {
justify-content: center;
}

.info {
right: 20px
}

.table-open {
padding: 0 22px 0 26px;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
position: relative;

.copy-to-editor {
.copy-to-editor, .info {
display: none;
}

&:hover {
background: fade(@redash-gray, 10%);

.copy-to-editor {
.copy-to-editor, .info {
display: flex;
}
}
Expand Down
14 changes: 14 additions & 0 deletions client/app/assets/less/redash/query.less
Original file line number Diff line number Diff line change
Expand Up @@ -508,3 +508,17 @@ nav .rg-bottom {
padding-right: 0;
}
}

.ui-select-choices-row .info {
display: none;
}

.ui-select-choices-row {
&:hover {
.info {
cursor: pointer;
width: 20px;
display: inline;
}
}
}
8 changes: 8 additions & 0 deletions client/app/components/dynamic-form/dynamicFormHelper.js
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,13 @@ function getFields(type = {}, target = { options: {} }) {
placeholder: `My ${type.name}`,
autoFocus: isNewTarget,
},
{
name: "description",
title: "Description",
type: "text",
required: false,
initialValue: target.description,
},
...orderedInputs(configurationSchema.properties, configurationSchema.order, target.options),
];

Expand All @@ -108,6 +115,7 @@ function getFields(type = {}, target = { options: {} }) {

function updateTargetWithValues(target, values) {
target.name = values.name;
target.description = values.description;
Object.keys(values).forEach(key => {
if (key !== "name") {
target.options[key] = values[key];
Expand Down
17 changes: 16 additions & 1 deletion client/app/components/proptypes.js
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,16 @@ export const DataSource = PropTypes.shape({
type_name: PropTypes.string,
});

export const DataSourceMetadata = PropTypes.shape({
key: PropTypes.number,
name: PropTypes.string,
type: PropTypes.string,
example: PropTypes.string,
description: PropTypes.string,
});

export const Table = PropTypes.shape({
columns: PropTypes.arrayOf(PropTypes.string).isRequired,
columns: PropTypes.arrayOf(PropTypes.object).isRequired,
});

export const Schema = PropTypes.arrayOf(Table);
Expand All @@ -31,6 +39,13 @@ export const RefreshScheduleDefault = {
until: null,
};

export const TableMetadata = PropTypes.shape({
key: PropTypes.number.isRequired,
name: PropTypes.string.isRequired,
description: PropTypes.string,
visible: PropTypes.bool.isRequired,
});

export const Field = PropTypes.shape({
name: PropTypes.string.isRequired,
title: PropTypes.string,
Expand Down
8 changes: 4 additions & 4 deletions client/app/components/queries/QueryEditor/ace.js
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,9 @@ function buildTableColumnKeywords(table) {
const keywords = [];
table.columns.forEach(column => {
keywords.push({
caption: column,
name: `${table.name}.${column}`,
value: `${table.name}.${column}`,
caption: column.name,
name: `${table.name}.${column.name}`,
value: `${table.name}.${column.name}`,
score: 100,
meta: "Column",
className: "completion",
Expand All @@ -56,7 +56,7 @@ function buildKeywordsFromSchema(schema) {
});
tableColumnKeywords[table.name] = buildTableColumnKeywords(table);
table.columns.forEach(c => {
columnKeywords[c] = "Column";
columnKeywords[c.name] = "Column";
});
});

Expand Down
2 changes: 1 addition & 1 deletion client/app/components/queries/QueryEditor/index.jsx
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ QueryEditor.propTypes = {
PropTypes.shape({
name: PropTypes.string.isRequired,
size: PropTypes.number,
columns: PropTypes.arrayOf(PropTypes.string).isRequired,
columns: PropTypes.arrayOf(PropTypes.object).isRequired,
})
),
onChange: PropTypes.func,
Expand Down
141 changes: 141 additions & 0 deletions client/app/components/queries/SchemaData.jsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
import { some } from "lodash";
import React from "react";
import PropTypes from "prop-types";
import Drawer from "antd/lib/drawer";
import Table from "antd/lib/table";

import { DataSourceMetadata, Query } from "@/components/proptypes";

function textWrapRenderer(text) {
return <div style={{ wordWrap: "break-word", wordBreak: "break-all" }}>{text}</div>;
}

export default class SchemaData extends React.PureComponent {
static propTypes = {
show: PropTypes.bool.isRequired,
onClose: PropTypes.func.isRequired,
tableName: PropTypes.string,
tableDescription: PropTypes.string,
tableMetadata: PropTypes.arrayOf(DataSourceMetadata),
sampleQueries: PropTypes.arrayOf(Query),
};

static defaultProps = {
tableName: "",
tableDescription: "",
tableMetadata: [],
sampleQueries: [],
};

render() {
const tableDataColumns = [
{
title: "Metadata",
dataIndex: "metadata",
width: 400,
key: "metadata",
},
{
title: "Value",
dataIndex: "value",
width: 400,
key: "value",
render: text => {
if (typeof text === "string") {
return text;
}
return (
<ul style={{ margin: 0, paddingLeft: "15px" }}>
{Object.values(text).map(query => (
<li>
<a target="_blank" rel="noopener noreferrer" href={`queries/${query.id}/source`}>
{query.name}
</a>
</li>
))}
</ul>
);
},
},
];

const columnDataColumns = [
{
title: "Column Name",
dataIndex: "name",
width: 400,
key: "name",
render: textWrapRenderer,
},
{
title: "Column Type",
dataIndex: "type",
width: 400,
key: "type",
render: textWrapRenderer,
},
];

const hasDescription = some(this.props.tableMetadata, columnMetadata => columnMetadata.description);

const hasExample = some(this.props.tableMetadata, columnMetadata => columnMetadata.example);

if (hasDescription) {
columnDataColumns.push({
title: "Description",
dataIndex: "description",
width: 400,
key: "description",
render: textWrapRenderer,
});
}

if (hasExample) {
columnDataColumns.push({
title: "Example",
dataIndex: "example",
width: 400,
key: "example",
render: textWrapRenderer,
});
}
const tableData = [
{
metadata: "Table Description",
value: this.props.tableDescription || "N/A",
key: "description",
},
{
metadata: "Sample Usage",
value: this.props.sampleQueries.length > 0 ? this.props.sampleQueries : "N/A",
key: "sample",
},
];

return (
<Drawer closable={false} placement="bottom" height={500} onClose={this.props.onClose} visible={this.props.show}>
<h4 style={{ margin: 0 }}>{this.props.tableName}</h4>
<hr />
<h5>Table Data</h5>
<Table
dataSource={tableData}
pagination={false}
scroll={{ y: 350 }}
size="small"
showHeader={false}
columns={tableDataColumns}
/>
<br />
<h5>Column Data</h5>
<Table
dataSource={this.props.tableMetadata}
pagination={false}
scroll={{ y: 175 }}
size="small"
rowKey="id"
columns={columnDataColumns}
/>
</Drawer>
);
}
}
12 changes: 12 additions & 0 deletions client/app/pages/data-sources/EditDataSource.jsx
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import navigateTo from "@/components/ApplicationArea/navigateTo";
import notification from "@/services/notification";
import LoadingState from "@/components/items-list/components/LoadingState";
import DynamicForm from "@/components/dynamic-form/DynamicForm";
import SchemaTable from "@/pages/data-sources/schema-table-components/SchemaTable";
import helper from "@/components/dynamic-form/dynamicFormHelper";
import HelpTrigger, { TYPES as HELP_TRIGGER_TYPES } from "@/components/HelpTrigger";
import wrapSettingsTab from "@/components/SettingsWrapper";
Expand All @@ -26,6 +27,7 @@ class EditDataSource extends React.Component {
dataSource: null,
type: null,
loading: true,
schema: null,
};

componentDidMount() {
Expand All @@ -34,6 +36,7 @@ class EditDataSource extends React.Component {
const { type } = dataSource;
this.setState({ dataSource });
DataSource.types().then(types => this.setState({ type: find(types, { type }), loading: false }));
DataSource.fetchSchema({ id: this.props.dataSourceId }).then(data => this.setState({ schema: data.schema, loading: false }));
})
.catch(error => this.props.onError(error));
}
Expand Down Expand Up @@ -75,6 +78,12 @@ class EditDataSource extends React.Component {
});
};

updateSchema = (schema, tableId, columnId) => {
const { dataSource } = this.state;
const data = { tableId, columnId, schema };
DataSource.updateSchema({ id: dataSource.id, data: data });
};

testConnection = callback => {
const { dataSource } = this.state;
DataSource.test({ id: dataSource.id })
Expand Down Expand Up @@ -127,6 +136,9 @@ class EditDataSource extends React.Component {
<div className="col-md-4 col-md-offset-4 m-b-10">
<DynamicForm {...formProps} />
</div>
<div className="col-md-12 admin-schema-editor">
<SchemaTable schema={this.state.schema} updateSchema={this.updateSchema} />
</div>
</div>
);
}
Expand Down
Loading

0 comments on commit 6dd763f

Please sign in to comment.