Using STL algorithms with JSON containers with expected results?

[bandzaw]

I'd like to use JSON as any other STL container with the same expected results. For example, given this code snippet:

#include <iostream>
#include <map>
#include <algorithm>
#include "json.hpp"


int main()
{
    {
        std::map<std::string,int> diffs;
        std::map<std::string,int> m1{{"key1",42}};
        std::map<std::string,int> m2{{"key2",42}};
        std::set_difference(
            m1.begin(), m1.end(),
            m2.begin(), m2.end(),
        std::inserter(diffs, diffs.end()));
        std::cout << "map diffs: " << diffs.size() << std::endl;
    }
    {
        nlohmann::json diffs = nlohmann::json::array();
        nlohmann::json m1{{"key1",42}};
        nlohmann::json m2{{"key2",42}};
        std::set_difference(
            m1.begin(), m1.end(),
            m2.begin(), m2.end(),
        std::inserter(diffs, diffs.end()));
        std::cout << "json diffs: " << diffs.size() << std::endl;
    }
}

the output is:

map diffs: 1
json diffs: 0

which is not what I expected. I'd really want to be able to compare JSON containers, including their keys.

The reason is that JSON iterators, when dereferenced, only return a reference to the value pointed to by the iterator (compare with std::map returning a std::pair of its key&value).
But then I thought I could use the proxy iterator: iteration_proxy and its key and value methods to work around my problem by writing a custom comparator and use it with std::set_difference:

    {
        nlohmann::json diffs = nlohmann::json::array();
        nlohmann::json m1{{"key1",42}};
        nlohmann::json m2{{"key2",42}};
        auto p1 = m1.items();
        auto p2 = m2.items();
        std::set_difference(
            p1.begin(), p1.end(),
            p2.begin(), p2.end(),
        std::inserter(diffs, diffs.end()),
        [&](const auto& e1, const auto& e2) -> bool {
            return (e1.key() < e2.key()) && (e1.value() < e2.value());
        });
        std::cout << "json diffs: " << diffs.size() << std::endl;
    }

This however, fails to compile. The compiler (g++ 7.2 (on Ubuntu 16.04)) complains that there is no iterator_category or value_type in iteration_proxy_internal. Hmm. From its name, and from the compiler message, I guess the iterator proxy is not an iterator. But, could it easily be extended to support my use case? Or can I solve my problem in another way? Or do I have to resort to writing my own algorithms for the JSON containers?

[theodelrieu]

Indeed, the iteration_proxy class does not fulfill the requirements of Iterator.

It should expose the 5 typedefs needed by iterator_traits, plus other functions to be 100% compliant.

[nlohmann]

The iteration wrapper was originally designed to be used in a range for. That's why it's no wonder it lacks the required functions. But I think this is all fixable.

[nlohmann]

I gave this issue a first try (see commit 8d8f890), but got stuck with this error message

In file included from ../test/src/unit-regression.cpp:32:
../single_include/nlohmann/json.hpp:1544:9: error: static_assert failed "could not find to_json() method in T's namespace"
        static_assert(sizeof(BasicJsonType) == 0,
        ^             ~~~~~~~~~~~~~~~~~~~~~~~~~~
../single_include/nlohmann/json.hpp:1560:16: note: in instantiation of function template specialization 'nlohmann::detail::to_json_fn::call<nlohmann::basic_json<std::map, std::vector, std::__1::basic_string<char>, bool, long long, unsigned long long, double, std::allocator, adl_serializer>, nlohmann::detail::iteration_proxy<nlohmann::detail::iter_impl<nlohmann::basic_json<std::map, std::vector, std::__1::basic_string<char>, bool, long long, unsigned long long, double, std::allocator, adl_serializer> > >::iteration_proxy_internal &>' requested here
        return call(j, std::forward<T>(val), priority_tag<1> {});
               ^
../single_include/nlohmann/json.hpp:9723:9: note: in instantiation of function template specialization 'nlohmann::detail::to_json_fn::operator()<nlohmann::basic_json<std::map, std::vector, std::__1::basic_string<char>, bool, long long, unsigned long long, double, std::allocator, adl_serializer>, nlohmann::detail::iteration_proxy<nlohmann::detail::iter_impl<nlohmann::basic_json<std::map, std::vector, std::__1::basic_string<char>, bool, long long, unsigned long long, double, std::allocator, adl_serializer> > >::iteration_proxy_internal &>' requested here
        ::nlohmann::to_json(j, std::forward<ValueType>(val));
        ^
../single_include/nlohmann/json.hpp:10900:28: note: in instantiation of function template specialization 'nlohmann::adl_serializer<nlohmann::detail::iteration_proxy<nlohmann::detail::iter_impl<nlohmann::basic_json<std::map, std::vector, std::__1::basic_string<char>, bool, long long, unsigned long long, double, std::allocator, adl_serializer> > >::iteration_proxy_internal, void>::to_json<nlohmann::basic_json<std::map, std::vector, std::__1::basic_string<char>, bool, long long, unsigned long long, double, std::allocator, adl_serializer>, nlohmann::detail::iteration_proxy<nlohmann::detail::iter_impl<nlohmann::basic_json<std::map, std::vector, std::__1::basic_string<char>, bool, long long, unsigned long long, double, std::allocator, adl_serializer> > >::iteration_proxy_internal &>' requested here
        JSONSerializer<U>::to_json(*this, std::forward<CompatibleType>(val));
                           ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/algorithm:5651:25: note: in instantiation of function template specialization 'nlohmann::basic_json<std::map, std::vector, std::__1::basic_string<char>, bool, long long, unsigned long long, double, std::allocator, adl_serializer>::basic_json<nlohmann::detail::iteration_proxy<nlohmann::detail::iter_impl<nlohmann::basic_json<std::map, std::vector, std::__1::basic_string<char>, bool, long long, unsigned long long, double, std::allocator, adl_serializer> > >::iteration_proxy_internal &, nlohmann::detail::iteration_proxy<nlohmann::detail::iter_impl<nlohmann::basic_json<std::map, std::vector, std::__1::basic_string<char>, bool, long long, unsigned long long, double, std::allocator, adl_serializer> > >::iteration_proxy_internal, 0>' requested here
            *__result = *__first1;
                        ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/algorithm:5677:12: note: in instantiation of function template specialization 'std::__1::__set_difference<(lambda at ../test/src/unit-regression.cpp:1632:48) &, nlohmann::detail::iteration_proxy<nlohmann::detail::iter_impl<nlohmann::basic_json<std::map, std::vector, std::__1::basic_string<char>, bool, long long, unsigned long long, double, std::allocator, adl_serializer> > >::iteration_proxy_internal, nlohmann::detail::iteration_proxy<nlohmann::detail::iter_impl<nlohmann::basic_json<std::map, std::vector, std::__1::basic_string<char>, bool, long long, unsigned long long, double, std::allocator, adl_serializer> > >::iteration_proxy_internal, std::__1::insert_iterator<nlohmann::basic_json<std::map, std::vector, std::__1::basic_string<char>, bool, long long, unsigned long long, double, std::allocator, adl_serializer> > >' requested here
    return __set_difference<_Comp_ref>(__first1, __last1, __first2, __last2, __result, __comp);
           ^
../test/src/unit-regression.cpp:1629:14: note: in instantiation of function template specialization 'std::__1::set_difference<nlohmann::detail::iteration_proxy<nlohmann::detail::iter_impl<nlohmann::basic_json<std::map, std::vector, std::__1::basic_string<char>, bool, long long, unsigned long long, double, std::allocator, adl_serializer> > >::iteration_proxy_internal, nlohmann::detail::iteration_proxy<nlohmann::detail::iter_impl<nlohmann::basic_json<std::map, std::vector, std::__1::basic_string<char>, bool, long long, unsigned long long, double, std::allocator, adl_serializer> > >::iteration_proxy_internal, std::__1::insert_iterator<nlohmann::basic_json<std::map, std::vector, std::__1::basic_string<char>, bool, long long, unsigned long long, double, std::allocator, adl_serializer> >, (lambda at ../test/src/unit-regression.cpp:1632:48)>' requested here
        std::set_difference(
             ^

[theodelrieu]

I think the issue's root comes from the fact that iterator_proxy is the one returned by begin/end.

Thus json does not model the Container concept on that point, since iterator_proxy returns itself when dereferenced.

I would rather go for a different iterator that should be created explicitly by users:

for (const auto& key_value : nlohmann::whatever_iterator{j}) {
  key_value.key();
}

IIRC that was iterator_wrapper's role before, what was the rationale behind that change?

[gregmarr]

iterator_wrapper was replaced by .items().

[nlohmann]

I am stuck here. Any ideas?

[theodelrieu]

Couldn't iterator_proxy_internal return a std::pair instead of itself when dereferenced? (This would mirror what happens in @bandzaw 's first snippet with std::map.

What is the rationale behind iterator_proxy_internal being its own value_type?

[nlohmann]

To be honest: the code in develop has one reason to be as it is: it works... It is hard to implement iterators that work with std::map, std::vector, and primitive values alike. And the step to support a range for was also difficult. I never expected them to be perfect, but since nobody complained so far, I did not touch them...

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Daniel599]

Hi @nlohmann ,
Sorry I'm late to the party, I saw the "help needed" label and wanted to try to help :)
I manged to make the code compile, using your branch of-course; the problem was that a to_json(BasicJsonType, iteration_proxy_internal) was needed:

template<typename IteratorType> class iteration_proxy; // TODO: Forward decl needed, maybe move it somewhere else
template<typename BasicJsonType, typename T,
         enable_if_t<std::is_same<T, typename iteration_proxy<typename BasicJsonType::iterator>::iteration_proxy_internal>::value, int> = 0>
void to_json(BasicJsonType& j, T b) noexcept
{
    typename BasicJsonType::object_t tmp_obj;
    tmp_obj[b.key()] = b.value(); // TODO: maybe there is a better way?
    external_constructor<value_t::object>::construct(j, std::move(tmp_obj));
}

side note: need to make iteration_proxy_internal a friend of to_json (or make it public).
also the test was a bit different from the example in the issue, I changed it as follows (hope it is still correct):

    SECTION("issue #1045 - Using STL algorithms with JSON containers with expected results?")
    {
        json diffs = nlohmann::json::array();
        json m1{{"key1", 42}};
        json m2{{"key2", 42}};
        auto p1 = m1.items();
        auto p2 = m2.items();

        using it_type = decltype(p1.begin());

        std::set_difference(
            p1.begin(), p1.end(),
            p2.begin(), p2.end(),
            std::inserter(diffs, diffs.end()), [&](const it_type & e1, const it_type & e2) -> bool
        {
            using comper_pair = std::pair<std::string, decltype(e1.value())>; // Trying to avoid unneeded copy
            return comper_pair(e1.key(), e1.value()) < comper_pair(e2.key(), e2.value()); // Using pair comper
        });

        CHECK(diffs.size() == 1); // Note the change here, was 2
    }

Sadly I don't have time to make it as a pull-request, but I hope it helps.

[nlohmann]

Fixed with #1134.