- Author : Tomáš Mokoš, Marek Brodec
Considering the fact that the formulas that we used to calculate for how many days can Moloch archive network traffic and what hardware should we use were only approximate, we have decided to measure some statistics to help us clear up these values.
From the Elasticsearch node quantity calculation formula: ¼ * [average network traffic in Gbit/s] * [number of days to be archived], we get that at 2 Mbit/s, one node should suffice.
Using the formula 86400 * [average network traffic per second] * [number of days to be archived], we can calculate that 1Gbit/s of traffic takes up 11TB of disk space daily, therefore 2Mbit/s of traffic will take up 22GB per day. At this rate we can archive approximately 113 days’ worth of raw data.
Since our lab server is not under heavy load, only 7GB (22%) of RAM is used on average. This is due to the the fact that during the night the network traffic is minimal. Non-uniform network traffic creates distortions, therefore long-term observation would be desirable.
Moloch by itself uses about 5% of total CPU utilization and 1.0 to 1.3 GB of RAM (3-3.5%).
Utilization of disk capacity was 340 GB (17%) on the first week, 220 GB (11%) on second week and 140 GB (7%) on third week.
Thanks to our use of data trimming we have managed to archive 6 months’ worth of traffic, although the actual value is closer to 4 months, since during the two months of the exam period, network traffic was very low. The captured data took up 52% (1.3 TB) of storage.
Sources
- Report Projekt 1-2 – Marek Brodec