tsharkVM - Tshark + ELK Analytics Virtual Machine

This project builds virtual machine which can be used for analytics of tshark -T ek (ndjson) output. The virtual appliance is built using vagrant, which builds Debian 10 with pre-installed and pre-configured ELK stack. After the VM is up, the process is simple: decoded pcaps (tshark -T ek output / ndjson) are sent over TCP/17570 to the VM ELK stack in VM will process and index the data Kibana is running in VM and can be accessed on http://127.0.0.1:15601/app/kibana#/dashboards Instuctions to build VM from Ubuntu desktop Clone source code git clone https://github.com/H21lab/tsharkVM.git Build tshark VM virtualbox vagrant bash ./build.sh ">sudo apt updatesudo apt install tshark virtualbox vagrantbash ./build.sh Upload pcaps to VM # copy your pcaps into ./Trace# run following script bash upload_pcaps.sh # or use tshark directly towards 127.0.0.1 17570/tcptshark -r trace.pcapng -x -T ek > /dev/tcp/localhost/17570 Open Kibana with browser firefox http://127.0.0.1:15601/app/kibana#/dashboards Open Main Dashboard and increase time window to e.g. last 100 years to see there the sample pcaps.     SSH to VM cd ./VMvagrant ssh Delete VM cd ./VMvagrant destroy default Start VM cd ./VMvagrant up Stop VM cd ./VMvagrant halt SSH into VM and check if ELK is running correctly cd ./VMvagrant sshsudo systemctl status kibana.servicesudo systemctl status elasticsearch.servicesudo systemctl status logstash.service Elasticsearch mapping template In the project is included simple Elasticseacrh mapping template generated for the frame,eth,ip,udp,tcp,dhcp protocols. To handle additional protocols efficiently it can be required to update the mapping template in the following way: Elasticsearch version ruby ./Public/process_tshark_mapping_json.rb # 3. Upload file to vagrant VM cd VM vagrant upload ../Kibana/custom_tshark_mapping_deduplicated.json /home/vagrant/tsharkVM/Kibana/custom_tshark_mapping_deduplicated.json cd .. # 4. Connect to VM and upload template in the Elasticsearch cd VM vagrant ssh cd tsharkVM/Kibana curl -X PUT "localhost:9200/_index_template/packets_template" -H 'Content-Type: application/json' -d@custom_tshark_mapping_deduplicated.json "># 1. Create custom mapping, by selecting required protocolstshark -G elastic-mapping --elastic-mapping-filter frame,eth,ip,udp,tcp,dns > ./Kibana/custom_tshark_mapping.json# 2. Deduplicate and post-process the mapping to fit current Elasticsearch versionruby ./Public/process_tshark_mapping_json.rb# 3. Upload file to vagrant VMcd VMvagrant upload ../Kibana/custom_tshark_mapping_deduplicated.json /home/vagrant/tsharkVM/Kibana/custom_tshark_mapping_deduplicated.jsoncd ..# 4. Connect to VM and upload template in the Elasticsearchcd VMvagrant sshcd tsharkVM/Kibanacurl -X PUT "localhost:9200/_index_template/packets_template" -H 'Content-Type: application/json' -d@custom_tshark_mapping_deduplicated.json Alternative can be using the dynamic mapping. See template ./Kibana/template_tshark_mapping_dynamic.json. And consider setting the numeric_detection parameter true/false depending on the mapping requirements and pcaps used. Upload the template into Elasticsearch in similar way as described above. Limitations tshark -G elastic-mapping --elastic-mapping-filter mapping could be outdated, it is not following properly the Elasticsearch changes and the output can be duplicated. The manual configuration and post-processing of the mapping template is required. Program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY. License The default license of source codes provided inside this project is the Apache License v2.0. Additionally refer to individual licenses and terms of used of installed software (see licenses for Wireshark, Elastic and other). Attribution Special thanks to people who helped with the Wireshark development or otherwise contributed to this work: Anders Broman Alexis La Goutte Christoph Wurm Dario Lombardo Vic Hargrave Example pcap in ./Traces subfolder was downloaded from https://wiki.wireshark.org/SampleCaptures Created by Martin Kacer Copyright 2021 H21 lab, All right reserved, https://www.h21lab.com Download tsharkVM

tsharkVM - Tshark + ELK Analytics Virtual Machine

This project builds virtual machine which can be used for analytics of tshark -T ek (ndjson) output. The virtual appliance is built using vagrant, which builds Debian 10 with pre-installed and pre-configured ELK stack.

After the VM is up, the process is simple:

  • decoded pcaps (tshark -T ek output / ndjson) are sent over TCP/17570 to the VM
  • ELK stack in VM will process and index the data
  • Kibana is running in VM and can be accessed on http://127.0.0.1:15601/app/kibana#/dashboards

Instuctions to build VM from Ubuntu desktop

Clone source code
git clone https://github.com/H21lab/tsharkVM.git

Build tshark VM
virtualbox vagrant bash ./build.sh ">
sudo apt update
sudo apt install tshark virtualbox vagrant
bash ./build.sh

Upload pcaps to VM
# copy your pcaps into ./Trace
# run following script
bash upload_pcaps.sh

# or use tshark directly towards 127.0.0.1 17570/tcp
tshark -r trace.pcapng -x -T ek > /dev/tcp/localhost/17570

Open Kibana with browser
firefox http://127.0.0.1:15601/app/kibana#/dashboards

Open Main Dashboard and increase time window to e.g. last 100 years to see there the sample pcaps.

 

 


SSH to VM
cd ./VM
vagrant ssh

Delete VM
cd ./VM
vagrant destroy default

Start VM
cd ./VM
vagrant up

Stop VM
cd ./VM
vagrant halt

SSH into VM and check if ELK is running correctly
cd ./VM
vagrant ssh
sudo systemctl status kibana.service
sudo systemctl status elasticsearch.service
sudo systemctl status logstash.service

Elasticsearch mapping template

In the project is included simple Elasticseacrh mapping template generated for the frame,eth,ip,udp,tcp,dhcp protocols. To handle additional protocols efficiently it can be required to update the mapping template in the following way:

Elasticsearch version ruby ./Public/process_tshark_mapping_json.rb # 3. Upload file to vagrant VM cd VM vagrant upload ../Kibana/custom_tshark_mapping_deduplicated.json /home/vagrant/tsharkVM/Kibana/custom_tshark_mapping_deduplicated.json cd .. # 4. Connect to VM and upload template in the Elasticsearch cd VM vagrant ssh cd tsharkVM/Kibana curl -X PUT "localhost:9200/_index_template/packets_template" -H 'Content-Type: application/json' -d@custom_tshark_mapping_deduplicated.json ">
# 1. Create custom mapping, by selecting required protocols
tshark -G elastic-mapping --elastic-mapping-filter frame,eth,ip,udp,tcp,dns > ./Kibana/custom_tshark_mapping.json

# 2. Deduplicate and post-process the mapping to fit current Elasticsearch version
ruby ./Public/process_tshark_mapping_json.rb

# 3. Upload file to vagrant VM
cd VM
vagrant upload ../Kibana/custom_tshark_mapping_deduplicated.json /home/vagrant/tsharkVM/Kibana/custom_tshark_mapping_deduplicated.json
cd ..

# 4. Connect to VM and upload template in the Elasticsearch
cd VM
vagrant ssh
cd tsharkVM/Kibana
curl -X PUT "localhost:9200/_index_template/packets_template" -H 'Content-Type: application/json' -d@custom_tshark_mapping_deduplicated.json

Alternative can be using the dynamic mapping. See template ./Kibana/template_tshark_mapping_dynamic.json. And consider setting the numeric_detection parameter true/false depending on the mapping requirements and pcaps used. Upload the template into Elasticsearch in similar way as described above.


Limitations

tshark -G elastic-mapping --elastic-mapping-filter mapping could be outdated, it is not following properly the Elasticsearch changes and the output can be duplicated. The manual configuration and post-processing of the mapping template is required.

Program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY.


License

The default license of source codes provided inside this project is the Apache License v2.0. Additionally refer to individual licenses and terms of used of installed software (see licenses for Wireshark, Elastic and other).


Attribution

Special thanks to people who helped with the Wireshark development or otherwise contributed to this work:

Example pcap in ./Traces subfolder was downloaded from https://wiki.wireshark.org/SampleCaptures

Created by Martin Kacer

Copyright 2021 H21 lab, All right reserved, https://www.h21lab.com