Skip to content

Commit

Permalink
Schema Viewer Drawer (getredash#3291)
Browse files Browse the repository at this point in the history
* Process extra column metadata for a few sql-based data sources.

* Add Table and Column metadata tables.

* Periodically update table and column schema tables in a celery task.

* Fetching schema returns data from table and column metadata tables.

* Add tests for backend changes.

* Front-end shows extra table metadata and uses new schema response.

* Delete datasource schema data when deleting a data source.

* Process and store data source schema when a data source is first created or after a migration.

* Tables should have a unique name per datasource.

* Addressing review comments.

* Update migration file for mixins.

* Appease PEP8

* Upgrade migration file for rebase.

* Cascade delete.

* Adding org_id

* Remove redundant column and table prefixes.

* Non-existing tables and columns should be filtered out on the server side not client side.

* Fetching table samples should be optional and should happen in a separate task per table.

* Allow users to force a schema refresh.

* Use updated_at to help prune old schema metadata periodically.

* Using settings.SCHEMAS_REFRESH_QUEUE

* fix for getredash#2426 test

* more stable test_interactive_new

* Closes #927, #928: Schema refresh improvements.

* Closes #934, #935: Remove type from schema browser and don't show empty example column in schema drawer (#936)

* Speed up schema fetch requests with fewer postgres queries.

* Add column metadata to Athena glue processing.

* Fix bug assuming 'metadata' exists for every table.

* Closes #939: Persisted, existing table metadata should be updated.

* Sample processing should be rate-limited.

* Add cli command for refreshing data samples.

* Schema refreshes should not overwrite column 'example' field.

* refresh_samples() should filter tables_to_sample on the datasource's id being sampled

* Correctly wrap long text in schema drawer.

Schema Improvements Part 2: Add data source config options.

Adding BigQuery schema drawer with data types and samples.

Add empty migration to replace the removed schedule_until migration

Add merge migration.

Co-authored-by: Alison <github@bankofknowledge.net>
  • Loading branch information
2 people authored and jezdez committed Jan 22, 2020
1 parent febc003 commit 1442cec
Show file tree
Hide file tree
Showing 57 changed files with 2,759 additions and 165 deletions.
1 change: 1 addition & 0 deletions client/app/assets/less/ant.less
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
@import '~antd/lib/radio/style/index';
@import '~antd/lib/time-picker/style/index';
@import '~antd/lib/pagination/style/index';
@import '~antd/lib/drawer/style/index';
@import '~antd/lib/table/style/index';
@import '~antd/lib/popover/style/index';
@import '~antd/lib/icon/style/index';
Expand Down
4 changes: 4 additions & 0 deletions client/app/assets/less/inc/base.less
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,10 @@ strong {
transition: height 0s, width 0s !important;
}

.admin-schema-editor {
padding: 50px 0;
}

// Ace Editor
.ace_editor {
border: 1px solid fade(@redash-gray, 15%) !important;
Expand Down
4 changes: 3 additions & 1 deletion client/app/assets/less/inc/popover.less
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
.popover {
box-shadow: fade(@redash-gray, 25%) 0px 0px 15px 0px;
color: #000000;
z-index: 1000000001; // So that it can popover a dropdown menu
}

.popover-title {
Expand All @@ -19,4 +21,4 @@
p {
margin-bottom: 0;
}
}
}
14 changes: 9 additions & 5 deletions client/app/assets/less/inc/schema-browser.less
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,14 @@ div.table-name {
border-radius: @redash-radius;
position: relative;

.copy-to-editor {
.copy-to-editor, .info {
display: none;
}

&:hover {
background: fade(@redash-gray, 10%);

.copy-to-editor {
.copy-to-editor, .info {
display: flex;
}
}
Expand All @@ -36,7 +36,7 @@ div.table-name {
background: transparent;
}

.copy-to-editor {
.copy-to-editor, .info {
color: fade(@redash-gray, 90%);
cursor: pointer;
position: absolute;
Expand All @@ -49,21 +49,25 @@ div.table-name {
justify-content: center;
}

.info {
right: 20px
}

.table-open {
padding: 0 22px 0 26px;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
position: relative;

.copy-to-editor {
.copy-to-editor, .info {
display: none;
}

&:hover {
background: fade(@redash-gray, 10%);

.copy-to-editor {
.copy-to-editor, .info {
display: flex;
}
}
Expand Down
14 changes: 14 additions & 0 deletions client/app/assets/less/redash/query.less
Original file line number Diff line number Diff line change
Expand Up @@ -698,3 +698,17 @@ nav .rg-bottom {
}
}
}

.ui-select-choices-row .info {
display: none;
}

.ui-select-choices-row {
&:hover {
.info {
cursor: pointer;
width: 20px;
display: inline;
}
}
}
8 changes: 8 additions & 0 deletions client/app/components/dynamic-form/dynamicFormHelper.js
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,13 @@ function getFields(type = {}, target = { options: {} }) {
placeholder: `My ${type.name}`,
autoFocus: isNewTarget,
},
{
name: "description",
title: "Description",
type: "text",
required: false,
initialValue: target.description,
},
...orderedInputs(configurationSchema.properties, configurationSchema.order, target.options),
];

Expand All @@ -108,6 +115,7 @@ function getFields(type = {}, target = { options: {} }) {

function updateTargetWithValues(target, values) {
target.name = values.name;
target.description = values.description;
Object.keys(values).forEach(key => {
if (key !== "name") {
target.options[key] = values[key];
Expand Down
8 changes: 4 additions & 4 deletions client/app/components/keywordBuilder.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ function buildTableColumnKeywords(table) {
const keywords = [];
table.columns.forEach(column => {
keywords.push({
caption: column,
name: `${table.name}.${column}`,
value: `${table.name}.${column}`,
caption: column.name,
name: `${table.name}.${column.name}`,
value: `${table.name}.${column.name}`,
score: 100,
meta: "Column",
className: "completion",
Expand All @@ -29,7 +29,7 @@ function buildKeywordsFromSchema(schema) {
});
tableColumnKeywords[table.name] = buildTableColumnKeywords(table);
table.columns.forEach(c => {
columnKeywords[c] = "Column";
columnKeywords[c.name] = "Column";
});
});

Expand Down
17 changes: 16 additions & 1 deletion client/app/components/proptypes.js
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,16 @@ export const DataSource = PropTypes.shape({
type_name: PropTypes.string,
});

export const DataSourceMetadata = PropTypes.shape({
key: PropTypes.number,
name: PropTypes.string,
type: PropTypes.string,
example: PropTypes.string,
description: PropTypes.string,
});

export const Table = PropTypes.shape({
columns: PropTypes.arrayOf(PropTypes.string).isRequired,
columns: PropTypes.arrayOf(PropTypes.object).isRequired,
});

export const Schema = PropTypes.arrayOf(Table);
Expand All @@ -31,6 +39,13 @@ export const RefreshScheduleDefault = {
until: null,
};

export const TableMetadata = PropTypes.shape({
key: PropTypes.number.isRequired,
name: PropTypes.string.isRequired,
description: PropTypes.string,
visible: PropTypes.bool.isRequired,
});

export const Field = PropTypes.shape({
name: PropTypes.string.isRequired,
title: PropTypes.string,
Expand Down
144 changes: 144 additions & 0 deletions client/app/components/queries/SchemaData.jsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
import React from "react";
import PropTypes from "prop-types";
import { react2angular } from "react2angular";
import Drawer from "antd/lib/drawer";
import Table from "antd/lib/table";

import { DataSourceMetadata, Query } from "@/components/proptypes";

function textWrapRenderer(text) {
return <div style={{ wordWrap: "break-word", wordBreak: "break-all" }}>{text}</div>;
}

class SchemaData extends React.PureComponent {
static propTypes = {
show: PropTypes.bool.isRequired,
onClose: PropTypes.func.isRequired,
tableName: PropTypes.string,
tableDescription: PropTypes.string,
sampleQueries: PropTypes.arrayOf(Query),
tableMetadata: PropTypes.arrayOf(DataSourceMetadata),
};

static defaultProps = {
tableName: "",
tableDescription: "",
tableMetadata: [],
sampleQueries: [],
};

render() {
const tableDataColumns = [
{
title: "Metadata",
dataIndex: "metadata",
width: 400,
key: "metadata",
},
{
title: "Value",
dataIndex: "value",
width: 400,
key: "value",
render: text => {
if (typeof text === "string") {
return text;
}
return (
<ul style={{ margin: 0, paddingLeft: "15px" }}>
{Object.values(text).map(query => (
<li>
<a target="_blank" rel="noopener noreferrer" href={`queries/${query.id}/source`}>
{query.name}
</a>
</li>
))}
</ul>
);
},
},
];

const columnDataColumns = [
{
title: "Column Name",
dataIndex: "name",
width: 400,
key: "name",
render: textWrapRenderer,
},
{
title: "Column Type",
dataIndex: "type",
width: 400,
key: "type",
render: textWrapRenderer,
},
];

const hasDescription = this.props.tableMetadata.some(columnMetadata => columnMetadata.description);

const hasExample = this.props.tableMetadata.some(columnMetadata => columnMetadata.example);

if (hasDescription) {
columnDataColumns.push({
title: "Description",
dataIndex: "description",
width: 400,
key: "description",
render: textWrapRenderer,
});
}

if (hasExample) {
columnDataColumns.push({
title: "Example",
dataIndex: "example",
width: 400,
key: "example",
render: textWrapRenderer,
});
}
const tableData = [
{
metadata: "Table Description",
value: this.props.tableDescription || "N/A",
},
{
metadata: "Sample Usage",
value: this.props.sampleQueries.length > 0 ? this.props.sampleQueries : "N/A",
},
];

return (
<Drawer closable={false} placement="bottom" height={500} onClose={this.props.onClose} visible={this.props.show}>
<h4 style={{ margin: 0 }}>{this.props.tableName}</h4>
<hr />
<h5>Table Data</h5>
<Table
dataSource={tableData}
pagination={false}
scroll={{ y: 350 }}
size="small"
showHeader={false}
columns={tableDataColumns}
/>
<br />
<h5>Column Data</h5>
<Table
dataSource={this.props.tableMetadata}
pagination={false}
scroll={{ y: 175 }}
size="small"
columns={columnDataColumns}
/>
</Drawer>
);
}
}

export default function init(ngModule) {
ngModule.component("schemaData", react2angular(SchemaData, null, []));
}

init.init = true;
17 changes: 14 additions & 3 deletions client/app/components/queries/schema-browser.html
Original file line number Diff line number Diff line change
Expand Up @@ -19,22 +19,33 @@
</div>

<div class="schema-browser" vs-repeat vs-size="$ctrl.getSize(table)">
<div ng-repeat="table in $ctrl.schema | filter:$ctrl.schemaFilterObject | filter: {name: '!'+$ctrl.versionFilter}">
<div ng-repeat="table in $ctrl.schema | filter:$ctrl.schemaFilterObject | filter: {name: '!'+$ctrl.versionFilter} | filter:$ctrl.itemExists track by table.name">
<div class="table-name" ng-click="$ctrl.showTable(table)">
<i class="fa fa-table"></i>
<strong>
<span title="{{table.name}}">{{table.name}}</span>
<span ng-if="table.size !== undefined"> ({{table.size}})</span>
</strong>
<i ng-if="table.column_metadata" class="fa fa-question-circle info" title="More Info" aria-hidden="true"
ng-click="openSchemaInfo($event, table)"></i>
<i class="fa fa-angle-double-right copy-to-editor" aria-hidden="true"
ng-click="$ctrl.itemSelected($event, [table.name])"></i>
</div>
<div uib-collapse="table.collapsed">
<div ng-repeat="column in table.columns | filter:$ctrl.schemaFilterColumn track by column" class="table-open">{{column}}
<div ng-repeat="column in table.columns | filter:$ctrl.schemaFilterColumn track by column.id" class="table-open">
{{column.name}}
<i class="fa fa-angle-double-right copy-to-editor" aria-hidden="true"
ng-click="$ctrl.itemSelected($event, [column])"></i>
ng-click="$ctrl.itemSelected($event, [column.name])"></i>
</div>
</div>
</div>
</div>
<schema-data
show="showSchemaInfo"
table-name="tableName"
table-description="tableDescription"
table-metadata="tableMetadata"
sample-queries="sampleQueries"
on-close="closeSchemaInfo"
></schema-data>
</div>
Loading

0 comments on commit 1442cec

Please sign in to comment.