Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I'm using elasticsearch, kibana and logstash 6.0.1.

I wish to upload csv data to elasticsearch by logstash and removing fields (path, @timestamp, @version, host and message). I'm showing logstash.conf and emp.csv files below. The upload will work if I don't use the remove_field instruction but I need to. Furthermore, the index was not created.

logstash.conf:

input {
  file {
      path => "e:\emp.csv"
      start_position => "beginning"
filter {
  csv {
      separator => ","
      columns => ["code","color"]
      remove_field => ["path", "@timestamp", "@version", "host", "message"]
  mutate {convert => ["code", "string"]}
  mutate {convert => ["color", "string"]}
output {
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "emp5"
    user => "elastic"
    password => "password"
  stdout {}
emp.csv:
1,blue
2,red
What is missing in this case?
In your csv file the data is not available that you are trying to delete.
Instead try this to delete for example path and host field:
(...)
filter {
  csv {
      separator => ","
      columns => ["code","color"]
    mutate {
      remove_field => ["path", "host"]
(...)
And for information, if field path and/or host doesn't exist, there's no problem. The plugin will remove field if field exists, and just do nothing if field does not exist.
Edit:
I have tested it on fresh elastic stack:
You can delete index with:
curl -X DELETE "localhost:9200/emp5"
Also note that in your current config logstash will read the file only once.
You can change that behaviour by adding sincedb_path => "/dev/null"
or in Windows case: sincedb_path => "NUL" inside:
input {
    file {
           (...) # here
section.
Then after logstash work verify result with:
curl -X GET "localhost:9200/emp5?pretty"
  "emp5" : {
    "aliases" : { },
    "mappings" : {
      "doc" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          "@version" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
          "code" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
          "color" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
          "message" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
    "settings" : {
      "index" : {
        "number_of_shards" : "5",
        "blocks" : {
          "read_only_allow_delete" : "true"
        "provided_name" : "emp5",
        "creation_date" : "1576099826712",
        "number_of_replicas" : "1",
        "uuid" : "reXYzqPgQryYcASoov9l5A",
        "version" : {
          "created" : "6080599"
As you can see there is no host and path field.
                Yes, you're right. It's working removing path and host. Then I added message, timestamp and version to remove_field. Adding sincedb_path => "NUL" was necessary. Thanks.
– user3637971
                Dec 12, 2019 at 22:07
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.