Cant index FileBeats data through Logstash due to "source" field mapping problem with ES template

[sempervictus]

Beats, ES, LS, and Kibana 6.1.2 in use on ubuntu 14
Filebeat -> Logstash -> ES is failing with:

 "caused_by"=>{"type"=>"mapper_parsing_exception", "reason"=>"Mapping definition for [source] has unsupported parameters:  [properties : {outer_ipv6_location={type=geo_point}, port={ignore_above=1024, type=keyword}, stats={properties={net_bytes_total={type=long}, net_packets_total={type=long}}}, ipv6={ignore_above=1024, type=keyword}, ip={ignore_above=1024, type=keyword}, outer_ip={ignore_above=1024, type=keyword}, outer_ipv6={ignore_above=1024, type=keyword}, ip_location={type=geo_point}, ipv6_location={type=geo_point}, mac={ignore_above=1024, type=keyword}, outer_ip_location={type=geo_point}}]"}}}}}

The messages look like this:

{
    "prospector" => {
        "type" => "log"
    },
      "@version" => "1",
          "host" => "svl-elk00-local",
    "@timestamp" => 2018-01-28T08:08:45.333Z,
       "fileset" => {
          "name" => "auth",
        "module" => "system"
    },
       "message" => "Jan 28 08:08:45 ubuntu1404-64 su[1505]: pam_unix(su:session): session closed for user root",
        "offset" => 18061,
          "beat" => {
            "name" => "svl-elk00-local",
        "hostname" => "svl-elk00-local",
         "version" => "6.1.2"
    },
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
        "source" => "/var/log/auth.log"
}

The FB config is:

cat /etc/filebeat/filebeat.yml 
---
filebeat.config.modules.path: "${path.config}/modules.d/*.yml"
filebeat.config.modules.reload.enabled: false
filebeat.modules:
-   module: system
    syslog:
    -   enabled: true
    auth:
    -   enabled: false
filebeat.prospectors:
-   input_type: log
    paths:
    - "/var/log/*log"
    - "/var/log/dmesg"
    - "/var/log/secure"
    enabled: false
    fields_under_root: false
output.logstash:
    ssl.enabled: true
    ssl.verification_mode: none
    hosts: "127.0.0.1:9778"

Metricbeat on the same setup works, mostly, but kibana in those dashboards complains about

Saved "field" parameter is now invalid. Please select a new field.
Visualize: "field" is a required parameter

The ES templates were loaded via filebeat setup --template, also tried manually doing it by having FB export and pushing via curl (after setting everything up clean):

curl -s -XPUT -H 'Content-Type: application/json' localhost:9200/_template/filebeat-6.1.2 -d"$(  /usr/share/filebeat/bin/filebeat export template --path.config /etc/filebeat)"

All produce the same result - LS cant index filebeats data into ES.
Am i doing something very wrong, or is this a side effect of the push to try and have beats speak directly to ES? Exposing elasticsearch to every potential beats collector sounds about as safe as streaking through Kabul, and we use LS for a ton of other data types, so i'd like to figure out a way to keep running what appears to be (according to the docs) a supported data pipeline once we move to 6. Thanks.

[jasonlang80]

I can confirm similar behavior with a slightly different error.

running a straight text logfile produces:

"log"
2018-08-02T20:15:47.020Z WARN elasticsearch/client.go:502 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbed0f77bd23e324a, ext:12511128886, loc:(*time.Location)(0x1f483e0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"source":"/mnt/log/webservers/apache/access.log", "offset":3681282, "message":"Jul 29 01:03:30 web303 webship[1200]: ip,ip [29/Jul/2018:01:03:30 -0600] 230680 - \"-/-\" \"-\" \"-\" \"GET /ec/api/v3/prepayStatus HTTP/1.1\" 200 158 \"https://endpoint.com/ec/\" \"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36\"", "prospector":common.MapStr{"type":"log"}, "input":common.MapStr{"type":"log"}, "beat":common.MapStr{"hostname":"logcatcher", "version":"6.3.0", "name":"logcatcher"}, "host":common.MapStr{"name":"logcatcher"}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc420423040), Source:"/mnt/log/webservers/apache/webship-access.log", Offset:3681282, Timestamp:time.Time{wall:0xbed0f778b046486e, ext:14975314, loc:(*time.Location)(0x1f483e0)}, TTL:-1, Type:"log", FileStateOS:file.StateOS{Inode:0x2081ea, Device:0xca50}}}, Flags:0x1} (status=400): {"type":"mapper_parsing_exception","reason":"Failed to parse mapping [doc]: Mapping definition for [error] has unsupported parameters: [properties : {code={type=long}, message={norms=false, type=text}, type={ignore_above=1024, type=keyword}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Mapping definition for [error] has unsupported parameters: [properties : {code={type=long}, message={norms=false, type=text}, type={ignore_above=1024, type=keyword}}]"}}

config:
`#------------------------------- System Module -------------------------------
#- module: system

Syslog

#syslog:
#enabled: true

# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false

# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:

Authorization logs

#auth:
#enabled: true

# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:

# Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
#var.convert_timezone: false

# Input configuration (advanced). Any input configuration option
# can be added under this section.
#input:`

metric beats are working, but not filebeat.

[ruflin]

It seems there is a mapping conflict here with the error field. Do you have an index for each beat version? Do you also have other data going into the same index? For some reason error is probably a text field in your index but should be an object.

For the Kibana issue: Seems to be related to #6489

[andrewkroh]

I'm closing this due to inactivity. If anyone is still having issues please reach out on https://discuss.elastic.co/c/beats and we'll help you.