How to parse json file with type composition of std::optional and std::variant

[chakpongchung]

@dubnde @nlohmann

This is related to the following issues, perhaps think of this question and the json file as a test case for parsing compositional type of optional and variant (see secondary field from the full json input below):

#1281
#1261 (comment)

How I can parse the input json inside this file ?
https://github.com/smogon/pokemon-showdown/blob/master/data/moves.js

For the secondary and flags properties? They are type composition of std::optional and std::variant .

Though only the full example can surface the problem in a clearer way, a minimal starting point example would be this one for the impatient:

[
  {
    "secondary": false
  },
  {

    "secondary": {
      "chance": 10,
      "boosts": {
        "spd": -1
      }
    }
  },
  {
    "secondary": {
      "chance": 30,
      "volatileStatus": "flinch"
    }
  },
  {
    "secondary": {
      "chance": 30
    }
  },
  {
    "secondary": {
      "chance": 10,
      "self": {
        "boosts": {
          "atk": 1,
          "def": 1,
          "spa": 1,
          "spd": 1,
          "spe": 1
        }
      }
    }
  },
  {
    "secondary": {
      "chance": 10,
      "status": "brn"
    }
  },
  {
    "secondary": {
      "chance": 50,
      "self": {
        "boosts": {
          "def": 2
        }
      }
    }
  },
  {
    "secondary": {
      "chance": 100,
      "self": {}
    }
  },
  {
    "secondary": {
      "chance": 50,
      "boosts": {
        "accuracy": -1
      }
    }
  }
]

The expected output can be but not limited to:

1. parsed object which has more structure, such that we can access by key(if it exists)
2. it has dynamic structure so we can check key existence during run time. For example, "boosts" has only a finite number(<10) of possible keys.

For your convenience, you can attach this snippet to the end of the js file and run it using node move.js. Two valid json files will be saved to your disk. One is a list of json objects while the other is an object with string as key.

var fs = require('fs');
fs.writeFile("moves_object.json", JSON.stringify(BattleMovedex), function(err) {}); // 1. save a json object with string key

var jsonList = []
for (var key of Object.keys(BattleMovedex)) {
    jsonList.push(BattleMovedex[key]);
}
fs.writeFile("moves.json", JSON.stringify(jsonList), function(err) { // 2. save as a list of json object
    if (err) {
        console.log(err);
    }
});

In c++(17/20), there are previous attempts here:
#1281
#1910
#1261 (comment)

Here is a boiler plate with the type move_t I would like to use that you can begin with:

#include <nlohmann/json.hpp>
#include <bits/stdc++.h>
#include <variant>

using namespace std;
using nlohmann::json;

struct Boosts {
    optional<int> atk;
    optional<int> def;
};

struct Self {
    optional<Boosts> boosts;
};
struct Secondary {
    optional<int> chance;
    optional<string> volatileStatus;
    optional<Boosts> boosts;
    optional<Self> self;
    optional<string> status;
};

struct move_t {
    int num;
    variant<bool, int> accuracy;
    int basePower;
    string category;
    optional<string> desc = std::nullopt;
    string shortDesc;
    string id;
    string name;
    optional<variant<bool, Secondary>> secondary;

};


void from_json(const nlohmann::json &j, move_t &p) {
    p.num = j.at("num").get<int>();
}

void to_json(nlohmann::json &j, const move_t &p) {
}

int main() {

    std::ifstream moves("./moves.json");
    json moves_json;
    moves >> moves_json;
    std::vector<move_t> v1 = moves_json;
    for (move_t m : v1) {
        std::cout << std::setw(2)
                  << m.num
                  << ", " << m.basePower
                  << ", " << m.category
                  << std::endl;
    }
}

Could you provide an example to parse this full json file?

[nlohmann]

I do not fully understand your goal. What do you want to achieve after parsing the JSON above? Convert it to a certain type?

[chakpongchung]

there is an interesting take using variant from c++17 as in this example for a json parser:
https://gist.github.com/willkill07/76268e7a88136705f7c2ea9177897cf1

Unfortunately I have not found a way to make it work in my local.

[nlohmann]

How should the C++ type where you want to parse to look like?

[chakpongchung]

@nlohmann The c++ type should look like this:

https://github.com/chakpongchung/postman-parser/blob/master/src/main/scala/catchThemAll/Move.scala

https://github.com/chakpongchung/catchThemAll/blob/master/Main.hs

Or as c++ code in the following:

struct Boosts {
    optional<int> atk;
    optional<int> def;
};

struct Self {
    optional<Boosts> boosts;
};
struct Secondary {
    optional<int> chance;
    optional<string> volatileStatus;
    optional<Boosts> boosts;
    optional<Self> self;
    optional<string> status;
};

struct move_t {
    int num;
    variant<bool, int> accuracy;
    int basePower;
    string category;
    optional<string> desc = std::nullopt;
    string shortDesc;
    string id;
    string name;
    optional<variant<bool, Secondary>> secondary;

};

@dota17 Please take a closer look of this part optional<variant<bool, Secondary>> secondary; which is quite different from the comment you cited. So the problem is not solved yet.

I can do it in Haskell and Scala but still wanna know whether I can find a json library in C++ supporting this.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dota17]

@chakpongchung The data types you mentioned like struct move_t or the secondary example are similar to the types mentioned earlier. This comment is a very good example for how to parse the composition of std::optional and std::variant data. If you fully understand the sample code, it is not difficult to parse the json data you refered.

[chakpongchung]

I looked at the comment suggested. I dont see why it is solving the problem of mixing optional and variant.

[nlohmann]

@chakpongchung Do you still need assistance on this?

[chakpongchung]

Yes, my question is still not addressed properly. The other issue mentioned is not close to what I am after. I am looking for support of std::optional<variant<a,b,c>> to parse the list of jsons which can contain different types in certain properties. @nlohmann

[nlohmann]

What about

#include <iostream>
#include <optional>
#include <variant>
#include "json.hpp"

using json = nlohmann::json;

// partial specialization (full specialization works too)
namespace nlohmann {
    template <typename T>
    struct adl_serializer<std::optional<T>> {
        static void to_json(json& j, const std::optional<T>& opt) {
            if (opt == std::nullopt) {
                j = nullptr;
            } else {
              j = *opt;
            }
        }

        static void from_json(const json& j, std::optional<T>& opt) {
            if (j.is_null()) {
                opt = std::nullopt;
            } else {
                opt = j.get<T>();
            }
        }
    };
}

struct Boosts {
    std::optional<int> atk;
    std::optional<int> def;
};

void to_json(json& j, const Boosts& b)
{
    j["atk"] = b.atk;
    j["def"] = b.def;
}

struct Self {
    std::optional<Boosts> boosts;
};

void to_json(json& j, const Self& s)
{
    j["boosts"] = s.boosts;
}

struct Secondary {
    std::optional<int> chance;
    std::optional<std::string> volatileStatus;
    std::optional<Boosts> boosts;
    std::optional<Self> self;
    std::optional<std::string> status;
};

void to_json(json& j, const Secondary& s)
{
    j["chance"] = s.chance;
    j["volatileStatus"] = s.volatileStatus;
    j["boosts"] = s.boosts;
    j["self"] = s.self;
    j["status"] = s.status;
}

struct move_t {
    int num;
    std::variant<bool, int> accuracy;
    int basePower;
    std::string category;
    std::optional<std::string> desc = std::nullopt;
    std::string shortDesc;
    std::string id;
    std::string name;
    std::optional<std::variant<bool, Secondary>> secondary;
};

void to_json(json& j, const move_t& m)
{
    j["num"] = m.num;

    switch(m.accuracy.index())
    {
        case 0:
            j["accuracy"] = std::get<bool>(m.accuracy);
            break;
        case 1:
            j["accuracy"] = std::get<int>(m.accuracy);
            break;
    }

    j["basePower"] = m.basePower;
    j["category"] = m.category;
    j["desc"] = m.desc;
    j["shortDesc"] = m.shortDesc;
    j["id"] = m.id;
    j["name"] = m.name;

    if (m.secondary.has_value())
    {
        switch(m.secondary.value().index())
        {
            case 0:
                j["secondary"] = std::get<bool>(m.secondary.value());
                break;
            case 1:
                j["secondary"] = std::get<Secondary>(m.secondary.value());
                break;
        }
    }
    else
    {
        j["secondary"] = nullptr;
    }
}

int main() {
    Boosts b;
    b.atk = 1;
    Self s;
    s.boosts = b;
    
    Secondary sec;
    sec.boosts = b;
    sec.self = s;
    
    move_t m;
    m.secondary = sec;
    m.accuracy = 10;
    
    std::cout << json(m).dump(2) << std::endl;
}

prints

{
  "accuracy": 10,
  "basePower": 0,
  "category": "",
  "desc": null,
  "id": "",
  "name": "",
  "num": 0,
  "secondary": {
    "boosts": {
      "atk": 1,
      "def": null
    },
    "chance": null,
    "self": {
      "boosts": {
        "atk": 1,
        "def": null
      }
    },
    "status": null,
    "volatileStatus": null
  },
  "shortDesc": ""
}

[chakpongchung]

I think it is much closer now, except that you are hand crafting some inputs.

https://github.com/pokeai/pokeai
To make the conversation more concrete, I modified your program a little bit to read from a file and pasted part of the json input here as a test case. The goal is to partially or fully, depending on how complete the move_t is, parse the whole json. The optional variant is the key obstacle.

And I also wonder why in c++ we need to add switch case in the to_json compared to other languages

[nlohmann]

Is there anything left to do here?

[chakpongchung]

Yes, I tried your code and use it to parse the json which is a list of move_t as shown in the github repo link above. It shows parsing error. Here is the way I am trying to use it:

int main() {

  std::ifstream moves("./moves.json");
  json moves_json;
  moves >> moves_json;

  std::cout << moves_json << std::endl;

  std::vector<move_t> v1 = moves_json;
  for (move_t m : v1) {
    std::cout << std::setw(2)
              << m.num
              << m.accuracy.index()
              << ", " << m.basePower
              << ", " << m.category
              << std::endl;
  }

}

[nlohmann]

Where does the error occur? What does it say?

[chakpongchung]

The error message is not so indicative and it starts from this line:

  std::vector<move_t> v1 = moves_json;

/usr/include/nlohmann/json.hpp:1201:9: error: static assertion failed: could not find from_json() method in T's namespace
         static_assert(sizeof(BasicJsonType) == 0,
         ^~~~~~~~~~~~~

[nlohmann]

In the code above, only the conversion to JSON was defined. If you want to read from a JSON value, you need to define the respective from_json functions, see https://github.com/nlohmann/json#arbitrary-types-conversions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.