Bank - LizardFS with Elasticsearch


SimonH
a month ago 0 SimonH 2

LizardFS is being used for storing and aggregating data from many applications before it gets to ELK.

Thanks to a configuration of Elasticsearch and LizardFS we get a storage with searching capabilities that are compliant with GDPR.

Data aggregation due to security precautions is done from flat files. These files are later processed by parsers in Logstash. The whole process is done this way so data can return to a canonical state of a log.

LizardFS is mounted on each Elasticsearch node which makes scalability really easy on both layers (just add a node for Elasticsearch). LizardFS can be scaled both up and down simply by adding a drive or a node – the system will auto-balance itself.

Hardware agnosticism enables you to use existing infrastructure and gives you the possibility to exit vendor lock-in if you are buying new commodity components.

LizardFS and Elasticsearch are open source products, you can install them on existing infrastructure. They enable you to create a platform for content management and storage in a cost-effective way. High availability and extreme scalability are just a few added value features provided by LizardFS.

LizardFS Stack

“At the moment our LizardFS stack is deployed on cheap and almost obsolete servers (not dedicated for DDS) and even still we are more and more impressed every day.”

“In the near future, we will migrate to the new servers with 10G network and 10TB per disk.

This is cheap storage with HA and has huge potential.”

Compute

14x HP DL 380 (Generation: 5, 6, 7)

Network

now 1G (10G in the near future)

Storage:

82x disks 10k


Reading this thread: