Ingest: foreach processor not accepting dot-notation for nested fields

[fkelbert]

Elasticsearch version (bin/elasticsearch --version): 7.5 Elastic Cloud

The following does not work as expected:

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "foreach": {
          "field": "mail.domains",
          "processor": {
            "gsub": {
              "field": "_ingest._value",
              "pattern": "^[^@]*@(.*)$",
              "replacement": "$1"
            }
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "mail.domains": [
          "foo@bar.com"
        ]
      }
    }
  ]
}

The above results in:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "field [mail] not present as part of path [mail.domains]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "field [mail] not present as part of path [mail.domains]"
  },
  "status": 400
}

This does work, instead:

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "foreach": {
          "field": "mail.domains",
          "processor": {
            "gsub": {
              "field": "_ingest._value",
              "pattern": "^[^@]*@(.*)$",
              "replacement": "$1"
            }
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "mail": {
          "domains": [
            "foo@bar.com"
          ]
        }
      }
    }
  ]
}

Both should actually work.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Ingest)

[fkelbert]

The same issue applies to the enrich processor, and possibly others.

[gaobinlong]

Maybe none of the processors can accept dot-notation for nested fields, but we can use Dot Expander Processor to resolve the problem.

[fkelbert]

Sure, the Dot Expander can resolve the issue.

I would very much welcome an initiative to make this easier for users, though. Elasticsearch seamlessly accepts documents using the dot notation, so it is kind of unexpected that Ingest processors would not accept the same notation.

I suppose there are solid technical reasons for this behaviour. Yet, I am wondering if we can make things easier for users, very much in line with #15951

[dakrone]

This has been open for quite a while, and we haven't made much progress on this due to focus in other areas. For now I'm going to close this as something we aren't planning on implementing. We can re-open it later if needed.