Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[REVIEW] Correct performance regression with 0.7 groupby. #1689

Merged
merged 3 commits into from
May 9, 2019
Merged

[REVIEW] Correct performance regression with 0.7 groupby. #1689

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d81f3cb
Improve performance
thomcom May 9, 2019
0f4bc52
CHANGELOG
thomcom May 9, 2019
c6690dd
Merge branch 'branch-0.7' into bug-ext-0.7-performance-regression
raydouglass May 9, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@
- PR #1648 ORC reader: fix non-deterministic output when skiprows is non-zero
- PR #1676 Fix groupby `as_index` behaviour with `MultiIndex`
- PR #1659 Fix bug caused by empty groupbys and multiindex slicing throwing exceptions

- PR #1689 Fix groupby performance regression


# cuDF 0.6.1 (25 Mar 2019)
Expand Down
3 changes: 2 additions & 1 deletion python/cudf/dataframe/multiindex.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
from collections.abc import Sequence

from cudf.dataframe import columnops
from cudf.dataframe.series import Series
from cudf.comm.serialize import register_distributed_serializer
from cudf.dataframe.index import Index, StringIndex
from cudf.utils import utils
Expand Down Expand Up @@ -63,7 +64,7 @@ def __init__(self, levels, codes=None, labels=None, names=None):

# converting levels to numpy array will produce a Float64Index
# (on empty levels)for levels mimicking the behavior of Pandas
self.levels = np.array(levels)
self.levels = np.array([Series(level).to_array() for level in levels])
self._validate_levels_and_codes(self.levels, self.codes)
self.name = None
self.names = names
Expand Down
7 changes: 2 additions & 5 deletions python/cudf/groupby/groupby.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,12 +206,9 @@ def apply_multiindex_or_single_index(self, result):
# time the groupby is calculated.
for by in self._by:
level = result[by].unique()
code = result[by]
for idx, value in enumerate(level):
level_mask = code == value
code = code.masked_assign(idx, level_mask)
replaced = result[by].replace(level, range(len(level)))
levels.append(level)
codes[by] = code
codes[by] = Series(replaced, dtype="int32")
names.append(by)
multi_index = MultiIndex(levels=levels,
codes=codes,
Expand Down