Improve examples & related tests #7773
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mistercrunch
force-pushed
the
improve_load_examples
branch
from
c96cc2d
to
d12747a
Compare
mistercrunch
force-pushed
the
improve_load_examples
branch
14 times, most recently
from
c11a5f4
to
1b249b5
Compare
Codecov Report
@@ Coverage Diff @@
## master #7773 +/- ##
==========================================
- Coverage 65.82% 65.77% -0.05%
==========================================
Files 461 461
Lines 22105 22167 +62
Branches 2425 2425
==========================================
+ Hits 14551 14581 +30
- Misses 7433 7465 +32
Partials 121 121
Continue to review full report at Codecov.
|
mistercrunch
changed the title
mistercrunch
commented
Jul 6, 2019
mistercrunch
commented
Jul 6, 2019
mistercrunch
commented
Jul 6, 2019
mistercrunch
commented
Jul 6, 2019
12 tasks
mistercrunch
force-pushed
the
improve_load_examples
branch
from
8a5d82d
to
f01b8ff
Compare
villebro
approved these changes
Jul 9, 2019
villebro
reviewed
Jul 9, 2019
| "path_json": Text, | ||
| }, | ||
| index=False, | ||
| ) |
There was a problem hiding this comment.
Oh, one more thing; when I added csv import functionality for BigQuery, I refactored db_engine_specs so that one can call db_engine_spec.df_to_sql(df, *kwargs) in place of df.to_sql(*kwargs) for engines that don't support df.to_sql(). So to make this work universally here, one would write
database.db_engine_spec.df_to_sql(
tbl_name,
database.get_sqla_engine(),
if_exists="replace",
chunksize=500,
dtype={
"color": String(255),
"name": String(255),
"polyline": Text,
"path_json": Text,
},
index=False,
)This doesn't necessarily have to be addressed in this PR; I can do that later, too, as I have a good test rig for that.
related to apache#7472, longer term we will generate the examples by exporting them into tarball as in apache#7472. In the meantime, we need this subset of the features: * allowing specifying an alternate database connection for examples * allowing a --only-metadata flag to `load_examples` to load only dashboard and chart definitions, no actual data is loaded
mistercrunch
force-pushed
the
improve_load_examples
branch
from
f01b8ff
to
146f599
Compare
mistercrunch
force-pushed
the
improve_load_examples
branch
from
135a85f
to
4ca819d
Compare
6 tasks
smacker
added a commit
to smacker/incubator-superset
that referenced
this pull request
Jul 18, 2019
The bug was introduced in apache#7773 It uses filter by `cls.table_name == datasource_name`: https://github.com/apache/incubator-superset/pull/7773/files#diff-a8dd5ec8d8decda2e3c5571d1ec0cdb6R740 But export puts `slc.datasource.name` into exported json: https://github.com/apache/incubator-superset/pull/7773/files#diff-ceeb7eee8d573333109e0037299c9711L673 `slc.datasource.name` in case of `SqlaTable` is `"{}.{}".format(self.schema, self.table_name)`
6 tasks
mistercrunch
pushed a commit
that referenced
this pull request
Jul 23, 2019
The bug was introduced in #7773 It uses filter by `cls.table_name == datasource_name`: https://github.com/apache/incubator-superset/pull/7773/files#diff-a8dd5ec8d8decda2e3c5571d1ec0cdb6R740 But export puts `slc.datasource.name` into exported json: https://github.com/apache/incubator-superset/pull/7773/files#diff-ceeb7eee8d573333109e0037299c9711L673 `slc.datasource.name` in case of `SqlaTable` is `"{}.{}".format(self.schema, self.table_name)`
alex-mark
pushed a commit
to alex-mark/incubator-superset
that referenced
this pull request
Jul 29, 2019
* [WiP] improve load_examples related to apache#7472, longer term we will generate the examples by exporting them into tarball as in apache#7472. In the meantime, we need this subset of the features: * allowing specifying an alternate database connection for examples * allowing a --only-metadata flag to `load_examples` to load only dashboard and chart definitions, no actual data is loaded * Improve logging * Rename data->examples * Load only if not exist * By default do not load, add a force flag * fix build * set published to true
alex-mark
pushed a commit
to alex-mark/incubator-superset
that referenced
this pull request
Jul 29, 2019
The bug was introduced in apache#7773 It uses filter by `cls.table_name == datasource_name`: https://github.com/apache/incubator-superset/pull/7773/files#diff-a8dd5ec8d8decda2e3c5571d1ec0cdb6R740 But export puts `slc.datasource.name` into exported json: https://github.com/apache/incubator-superset/pull/7773/files#diff-ceeb7eee8d573333109e0037299c9711L673 `slc.datasource.name` in case of `SqlaTable` is `"{}.{}".format(self.schema, self.table_name)`
6 tasks
mistercrunch
added
🏷️ bot
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
CATEGORY
Choose one
SUMMARY
related to #7478
This enables setting up an alternative URI for loading/storing/serving the examples data. New URI is defined with config key
SQLALCHEMY_EXAMPLES_URI. If nothing not set explicitly, will default to point to the metadata database (SQLALCHEMY_DATABASE_URI). This in turn adds a new default entry to theDatabasemodel namedexamplesAs a result, examples data tables can now be reused across Superset environments, and the tables don't get mixed in with the metadata database, leaving it cleaner.
Approach
load_examplesto load onlydashboard and chart definitions, no actual data is loaded
Longer term we will generate the examples by exporting
them into tarball as in #7472.