Hardware Requirements

The architecture of Moloch enables it to be distributed on multiple devices. For small networks, demonstrations or home deployment, it is possible to host all the tools necessary on a single device; however, for capturing large volumes of data at high transfer rates, it is recommended not to run Capture and Elasticsearch on the same machine. Moloch allows for software demo version testing directly on the website. In case of storage space shortage, Moloch replaces the oldest data with the new. Moloch can also perform replications, effectively doubling storage space usage. We advise to thoroughly think through the use of this feature.

Elasticsearch and amount of nodes

Amount of nodes(servers) to be used depends on:

• The amount of RAM available to each node
• For how many days will the metadata(SPI data) be stored
• Disk speed
• Size of the HTTP portion of traffic
• Average transfer rate of all interfaces
• Whether the connections are short-term or long-term
• Required reaction speed of requests
• Estimated number of users requesting service at the same time

It must be taken into account, that to store one day’s worth of Elasticsearch module metadata (SPI data) at 1Gbit/s, roughly 200GB of disk space is needed. For example, to store 14 days’ worth of traffic at average network traffic of 2.5Gbit/s, we can easily calculate the amount of storage needed is 14 * 2.5 * 200, which amounts to roughly 7TB.

The formula to approximately calculate the amount of nodes needed for Elasticsearch is: ¼ * [average network traffic in Gbit/s] * [number of days to be archived]. For example, to archive 20 days’ worth of traffic at 1Gbit/s, 5 nodes would be needed. If Moloch is to be deployed on higher performance machines, multiple Elasticsearch nodes can be run on a single device. Since the deployment of additional nodes is a simple task, we recommend starting with fewer nodes and adding new ones until the required reaction speed of requests is reached.

Capture

It has to be remarked that while capturing at 1Gbit/s of traffic, 11TB of disk space is required for archiving of pcap files alone. For example, to store 7 days’ worth of traffic at average speed of 2.5 Gbit/s, the amount of storage needed is [ 7 * 2.5 * 11 ] TB, which amounts to 192.5TB. Total bandwidth size must include both directions of transfer, therefore a 10G uplink is capable of generating 20Gbit of capture data (10Gbit for each direction). Considering this, it is recommended to have multiple uplinks connected to Moloch. For example, for 10G uplink with 4Gbit/s traffic in both directions, it would be advisable to use two 10G uplinks for capture, since using a single 10G uplink runs a risk of packet loss.

To capture large amounts of data (several Gbit/s) we advise using the following hardware :

• RAM: 64 GB to 96 GB
• OS disks: RAID5 works best. SSDs are not required
• CAPTURE disks: 20+x 4TB disks or 6 TB SATA.
• RAID: Hardware RAID card with at least 1GB of cache.
• NIC: New Intel NICs are recommended, but most NICs should work fine.
• CPU: at least 2*6 cores. The amount of cores needed grows with average uplink traffic. Moloch allows for device load balancing through mirroring.

When considering purchase of additional SSDs or NICs, considering adding another monitoring device instead is advised.

Sources

• CRZP Komplexný systém pre detekciu útokov a archiváciu dát – Moloch