Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump minor version 1.1.0 #98

Merged
merged 4 commits into from
Mar 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,11 @@ stat_client.set_publish_date(date)

## Saving and restoring Uttrekksbeskrivelser and Transfers as json

From `stat_client.transfer()` you will recieve a StatbankTransfer object, from `stat_client.get_description` a StatbankUttrekksBeskrivelse-object. These can be serialized and saved to disk, and later be restored, maybe this can be a form of logging on which transfers were done? This is also a way of having one notebook get all the descriptions, then have a notebook for each table, which can actually run .validate, withoug setting up a client with username and password. Then have another notebook that sends all the tabels, which requires username and password...
From `stat_client.transfer()` you will recieve a StatbankTransfer object, from `stat_client.get_description()` a StatbankUttrekksBeskrivelse-object. These can be serialized and saved to disk, and later be restored, maybe this can be a form of logging on which transfers were done?
This can also be used to:
1. have one notebook get all the descriptions of all tables produced from the pipeline (requires password).
1. then have a notebook for each table, restoring the description from the local jsos, which can actually run .validate (without typing in password).
1. then have a notebook at the end, that sends all the tables (requiring password-entry a second time).

```python
filbesk_06339 = stat_client.get_description("06339")
Expand All @@ -157,6 +161,18 @@ Some deeper data-structures, like the dataframes in the transfer will not be ser

---


## The logger
Statbank-package makes its own logger using the python logging package. The logger is available at `statbank.logger`.
A lot of the validations are logged as the level "info", if they seem ok.
Or on the level "warning" if things are not ok. The levels are colorized with colorama, green for INFO, magenta for WARNING.
If you *dont* want to see the info-parts of the validate-method, you can change the loggers level before calling validate, like this:
```python
import statbank
import logging
statbank.logger.setLevel(logging.WARNING)
```

## Version history
- 1.1.0 Migrating to "new template" for Pypi-packages at SSB, with full typing support, a reference website, logging instead of print etc.
- 1.0.6 fixing new functionality on "IRkodelister"
Expand Down
2 changes: 1 addition & 1 deletion src/statbank/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
from statbank.apidata import apidata_all
from statbank.apidata import apidata_rotate
from statbank.client import StatbankClient
from statbank.logger import logger
from statbank.statbank_logger import logger

__all__ = ["StatbankClient", "apidata", "apidata_all", "apidata_rotate"]

Expand Down
2 changes: 1 addition & 1 deletion src/statbank/apidata.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
from statbank.api_types import QueryWholeType


from statbank.logger import logger
from statbank.statbank_logger import logger

# Getting data from Statbank

Expand Down
2 changes: 1 addition & 1 deletion src/statbank/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
from statbank.globals import OSLO_TIMEZONE
from statbank.globals import STATBANK_TABLE_ID_LEN
from statbank.globals import TOMORROW
from statbank.logger import logger
from statbank.statbank_logger import logger
from statbank.transfer import StatbankTransfer
from statbank.uttrekk import StatbankUttrekksBeskrivelse

Expand Down
File renamed without changes.
69 changes: 39 additions & 30 deletions src/statbank/transfer.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
from statbank.auth import StatbankAuth
from statbank.globals import OSLO_TIMEZONE
from statbank.globals import SSB_TBF_LEN
from statbank.logger import logger
from statbank.statbank_logger import logger

if TYPE_CHECKING:
from statbank.api_types import TransferResultType
Expand Down Expand Up @@ -62,7 +62,7 @@ class StatbankTransfer(StatbankAuth):
response (requests.Response): The resulting response from the transfer-request. Headers might be deleted without warning.
"""

def __init__( # noqa: PLR0913, PLR0912
def __init__( # noqa: PLR0913
self,
data: dict[str, pd.DataFrame],
tableid: str = "",
Expand All @@ -81,36 +81,10 @@ def __init__( # noqa: PLR0913, PLR0912

May run the validations from the StatbankValidation class before the transfer.
"""
self._set_user_attrs(loaduser=loaduser, shortuser=shortuser, cc=cc, bcc=bcc)
self._set_date(date=date)
self.data = data
self.tableid = tableid

if isinstance(loaduser, str) and loaduser != "":
self.loaduser = loaduser
else:
error_msg = "You must set loaduser as a parameter"
raise ValueError(error_msg)

if shortuser:
self.shortuser = shortuser
else:
self.shortuser = os.environ["JUPYTERHUB_USER"].split("@")[0]
if cc:
self.cc = cc
else:
self.cc = self.shortuser
if bcc:
self.bcc = bcc
else:
self.bcc = self.cc

# At this point we want date to be a string?
if date is None:
date = dt.now().astimezone(OSLO_TIMEZONE) + td(days=1)
if isinstance(date, str):
self.date: str = date
else:
self.date = date.strftime("%Y-%m-%d")

self.overwrite = overwrite
self.approve = approve
self.validation = validation
Expand Down Expand Up @@ -179,6 +153,41 @@ def delay(self) -> bool:
"""Obfuscate the delay a bit from the user. We dont want transfers transferring again without recreating the object."""
return self.__delay

def _set_user_attrs(
self,
loaduser: str = "",
shortuser: str = "",
cc: str = "",
bcc: str = "",
) -> None:
if isinstance(loaduser, str) and loaduser != "":
self.loaduser = loaduser
else:
error_msg = "You must set loaduser as a parameter"
raise ValueError(error_msg)

if shortuser:
self.shortuser = shortuser
else:
self.shortuser = os.environ["JUPYTERHUB_USER"].split("@")[0]
if cc:
self.cc = cc
else:
self.cc = self.shortuser
if bcc:
self.bcc = bcc
else:
self.bcc = self.cc

def _set_date(self, date: dt | str | None = None) -> None:
# At this point we want date to be a string?
if date is None:
date = dt.now().astimezone(OSLO_TIMEZONE) + td(days=1)
if isinstance(date, str):
self.date: str = date
else:
self.date = date.strftime("%Y-%m-%d")

def to_json(self, path: str = "") -> str | None:
"""Store a copy of the current state of the transfer-object as a json.

Expand Down
2 changes: 1 addition & 1 deletion src/statbank/uttrekk.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
import requests as r

from statbank.auth import StatbankAuth
from statbank.logger import logger
from statbank.statbank_logger import logger
from statbank.uttrekk_validations import StatbankUttrekkValidators
from statbank.uttrekk_validations import StatbankValidateError

Expand Down
Loading