Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix errors on Windows tests in new tests/functional #4767

Merged
merged 3 commits into from
Feb 25, 2022
Merged

Fix errors on Windows tests in new tests/functional #4767

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6f90cc3
[#4781] Convert reads and writes in project fixture to text/utf-8 enc…
gshank Feb 23, 2022
ad7a4e2
Switch to using write_file and read_file functions
gshank Feb 25, 2022
9a2e1a1
Add comment
gshank Feb 25, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
- Re-format codebase (except tests) using pre-commit hooks ([#3195](https://github.com/dbt-labs/dbt-core/issues/3195), [#4697](https://github.com/dbt-labs/dbt-core/pull/4697))
- Add deps module README ([#4686](https://github.com/dbt-labs/dbt-core/pull/4686/))
- Initial conversion of tests to pytest ([#4690](https://github.com/dbt-labs/dbt-core/issues/4690), [#4691](https://github.com/dbt-labs/dbt-core/pull/4691))
- Fix errors in Windows for tests/functions ([#4781](https://github.com/dbt-labs/dbt-core/issues/4781), [#4767](https://github.com/dbt-labs/dbt-core/pull/4767))

Contributors:
- [@NiallRees](https://github.com/NiallRees) ([#4447](https://github.com/dbt-labs/dbt-core/pull/4447))
Expand Down
17 changes: 11 additions & 6 deletions core/dbt/tests/fixtures/project.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ def dbt_profile_data(unique_schema, database_host):
def profiles_yml(profiles_root, dbt_profile_data):
os.environ["DBT_PROFILES_DIR"] = str(profiles_root)
path = os.path.join(profiles_root, "profiles.yml")
with open(path, "w") as fp:
with open(path, "w", encoding="utf-8") as fp:
fp.write(yaml.safe_dump(dbt_profile_data))
yield dbt_profile_data
del os.environ["DBT_PROFILES_DIR"]
Expand All @@ -138,7 +138,8 @@ def dbt_project_yml(project_root, project_config_update, logs_dir):
if project_config_update:
project_config.update(project_config_update)
runtime_config_file = project_root.join("dbt_project.yml")
runtime_config_file.write(yaml.safe_dump(project_config))
with open(runtime_config_file, "w", encoding="utf-8") as fp:
fp.write(yaml.safe_dump(project_config))


# Fixture to provide packages as either yaml or dictionary
Expand All @@ -155,7 +156,8 @@ def packages_yml(project_root, packages):
data = packages
else:
data = yaml.safe_dump(packages)
project_root.join("packages.yml").write(data)
with open(project_root.join("packages.yml"), "w", encoding="utf-8") as fp:
fp.write(data)


# Fixture to provide selectors as either yaml or dictionary
Expand All @@ -172,7 +174,8 @@ def selectors_yml(project_root, selectors):
data = selectors
else:
data = yaml.safe_dump(selectors)
project_root.join("selectors.yml").write(data)
with open(project_root.join("selectors.yml"), "w", encoding="utf-8") as fp:
fp.write(data)


# This creates an adapter that is used for running test setup and teardown,
Expand Down Expand Up @@ -205,13 +208,15 @@ def write_project_files(project_root, dir_name, file_dict):
def write_project_files_recursively(path, file_dict):
for name, value in file_dict.items():
if name.endswith(".sql") or name.endswith(".csv") or name.endswith(".md"):
path.join(name).write(value)
with open(path.join(name), "w", encoding="utf-8") as fp:
fp.write(value)
elif name.endswith(".yml") or name.endswith(".yaml"):
if isinstance(value, str):
data = value
else:
data = yaml.safe_dump(value)
path.join(name).write(data)
with open(path.join(name), "w", encoding="utf-8") as fp:
fp.write(data)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this is happening in multiple places. I don't want to lose why we're doing it this way (in 6 months). Could we pull it out to centralize this and add a comment as to why we do this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent idea, and much more readable. Have created 'read_file' and 'write_file' functions.

else:
write_project_files_recursively(path.mkdir(name), value)

Expand Down
2 changes: 1 addition & 1 deletion tests/fixtures/jaffle_shop.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -395,7 +395,7 @@ def seeds():
dir_path = os.path.dirname(os.path.realpath(__file__))
for file_name in ("raw_customers.csv", "raw_orders.csv", "raw_payments.csv"):
path = os.path.join(dir_path, "jaffle_shop_data", file_name)
with open(path, "rb") as fp:
with open(path, "r") as fp:
seeds[file_name] = fp.read()
return seeds

Expand Down
2 changes: 1 addition & 1 deletion tests/functional/basic/test_copy_uppercase.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ def models():
def seeds(test_data_dir):
# Read seed file and return
path = os.path.join(test_data_dir, "seed-initial.csv")
with open(path, "rb") as fp:
with open(path, "r") as fp:
seed_csv = fp.read()
return {"seed.csv": seed_csv}
return {}
Expand Down
2 changes: 1 addition & 1 deletion tests/functional/basic/test_simple_copy.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ def models():
def seeds(test_data_dir):
# Read seed file and return
path = os.path.join(test_data_dir, "seed-initial.csv")
with open(path, "rb") as fp:
with open(path, "r") as fp:
seed_csv = fp.read()
return {"seed.csv": seed_csv}
return {}
Expand Down
4 changes: 2 additions & 2 deletions tests/functional/basic/test_simple_reference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,11 +141,11 @@ def seeds(test_data_dir):
# Read seed file and return
seeds = {"properties.yml": properties_yml}
path = os.path.join(test_data_dir, "seed-initial.csv")
with open(path, "rb") as fp:
with open(path, "r") as fp:
seed_csv = fp.read()
seeds["users.csv"] = seed_csv
path = os.path.join(test_data_dir, "summary_expected.csv")
with open(path, "rb") as fp:
with open(path, "r") as fp:
summary_csv = fp.read()
seeds["summary_expected.csv"] = summary_csv
return seeds
Expand Down