Skip to content

mathzdev/elastic-logstash-csv-importer-sample-elk

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

9 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

A simple model to import your csv into elasticsearch using logstash

First Step

Certify if you have logstash in your machine, or download here https://www.elastic.co/pt/downloads/logstash

Second Step

You need to see the logstash.conf file and its structure, and adapt it if necessary.

input {
  file {
    path => "cities.csv"
    start_position => "beginning"
  }
}

filter {  
  csv {
    separator => ","
    columns => ["LatD", "LatM", "LatS", "NS", "LonD", "LonM", "LonS", "EW", "City", "State"]
  }
}

output {
  elasticsearch { 
    hosts => ["localhost:9200"] 
    index => "cities"
  }
}

Input

The input section is about your file and respective configurations, in this case only setting the start position for the logstash start reading. Sometimes in path need to put real path like "D:/PATH/TO/YOUR/CITIES/cities.csv".

Filter

The filter section is about your file structure, and the separation of columns and values. And have de mutate additional parameters, to mutate your data, in this case we using strip to remove whitespaces after and before any word, and removing dumb columns ("path", "host", "message").

Output

The output section is to set your index (the collection to save the data) and your host destiny (address to elasticsearch).

More about config file

See more at https://www.elastic.co/guide/en/logstash/master/configuration-file-structure.html

Third Step

Now is ready to run the simple command logstash -f logstash.conf and the magic starts! Do you need to wait the process (maybe to take a time :D)...

Kibana showing your data

See in kibana your data after import:

All data

Using

And Filtering

Using

Enjoy the ELK stack !

Credits

Me @mathzdev to create this simple tuto, and @WilsonLucas Wilson's Linkedin for teach me about logstash.

Elk Stack

My website https://mathz.dev

About

A simple model to import your csv into elasticsearch using logstash.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published