Add a generic distributed pairwise(!) function?

[dkarrasch]

I was wondering if something along the following lines would be of general interest:

function distributedpairwise!(r::SharedMatrix, metric::SemiMetric, a::AbstractMatrix)
    n = size(a, 2)
    size(r) == (n, n) || throw(DimensionMismatch("Incorrect size of r."))
    @inbounds for j = 1:n
        aj = view(a, :, j)
        @sync @distributed for i = (j + 1):n # have parallel computation here to avoid uneven work load
            r[i, j] = evaluate(metric, view(a, :, i), aj)
        end
        r[j, j] = 0
        for i = 1:(j - 1)
            r[i, j] = r[j, i]   # leveraging the symmetry of SemiMetric
        end
    end
    r
end

and similarly for pairwise(::SemiMetric, a, b)? Or the other way around, is there a performance loss to be expected if the above was to replace the current pairwise! implementation, which would neatlessly work even if only one worker was available? Note that even the latter would not affect specialized pairwise instances like for Euclidean().

[nalimilan]

Or the other way around, is there a performance loss to be expected if the above was to replace the current pairwise! implementation, which would neatlessly work even if only one worker was available? Note that even the latter would not affect specialized pairwise instances like for Euclidean().

AFAIK distributing operations across workers will only increase performance if the operations are really costly. Have you done some benchmarking?

OTOH using several threads when possible should be a net gain everywhere.

[dkarrasch]

I have a metric type which is much costlier than the usual Euclidean calculations, that's why I considered doing it in a distributed fashion, even more so since generally the distance computations are independent. When I find the time, I'll do some benchmarking and report the results. I just wanted to check whether there are obvious reasons to not have a distributed version.