Improve nonunique #1824
Improve nonunique #1824
Conversation
| @inbounds for g_row in gslots | ||
| (g_row > 0) && (res[g_row] = false) | ||
| end | ||
| return res | ||
| end | ||
|
|
||
| nonunique(df::AbstractDataFrame, cols::Union{Integer, Symbol}) = nonunique(df[[cols]]) | ||
| nonunique(df::AbstractDataFrame, cols::Any) = nonunique(df[cols]) | ||
| nonunique(df::AbstractDataFrame, cols) = |
There was a problem hiding this comment.
The old approach looked more Julian to me (use dispatch). Also, better use long function form for multi-line functions.
| @@ -1567,3 +1567,5 @@ end | |||
| function permutecols!(df::DataFrame, p::AbstractVector{Symbol}) | |||
| permutecols!(df, index(df)[p]) | |||
| end | |||
|
|
|||
| nonunique(df::DataFrame, cols) = nonunique(select(df, cols, copycols=false)) | |||
There was a problem hiding this comment.
Maybe better implement select for all existing AbstractDataFrame types? Then we wouldn't need this special case.
| @@ -871,15 +871,22 @@ function nonunique(df::AbstractDataFrame) | |||
| gslots = row_group_slots(ntuple(i -> df[i], ncol(df)), Val(true))[3] | |||
| # unique rows are the first encountered group representatives, | |||
| # nonunique are everything else | |||
| res = fill(true, nrow(df)) | |||
| res = trues(nrow(df)) | |||
There was a problem hiding this comment.
Isn't that a bit slower? There may be a speed-memory tradeoff, and in general we tend to favor speed since we assume the data frame stores many columns so it doesn't really matter to allocate another one temporarily.
|
TODO note: This will be added after |
|
I will close this once I move all functionality to #1847 |
|
This PR is now redundant. It is covered by #1847. |
A small PR with the following changes in
nonunique:BitVectorinstead ofVector{Bool}trueascols(it should error)DataFrame(as in the old implementation we were performing an unnecessary copy)