Menu Close

Category: Network security

Moloch – Components and architecture

Tomáš Mokoš 0 Comments

Components

Moloch consists of three components:

  • Elasticsearch – search engine powering the Moloch system. It is distributed under the terms of Apache license. Requests are handled using HTTP and results are returned in JSON file format. Elasticsearch supports database sharding, making it fast and scalable.
  • Capture – C language based application for real-time network traffic monitoring. Captured data is written to disk in PCAP format. Alternatively, it can be used to import PCAP files for analysis and archiving manually through command line. The application analyzes protocols of OSI layers three through seven and creates SPI data which it sends to the Elasticsearch cluster for indexing.
  • Viewer – The viewer uses a number of node.js tools. Node.js is an event-based, server-side Javascript platform with its own HTTP and JSON communication. Viewer runs on each device with running Capture module and it provides a web UI for searching, displaying and exporting of PCAP files. GUI/API calls are carried out using URIs, enabling integration with security information and event management (SIEM) systems, consoles or command line for PCAP file obtaining.

Architecture

All of the components can be located and run on a single node, however this is not recommended for processing of larger data flows. Whether the data flow is too large can be determined by requests taking too long to respond, in that case, transition to multi-node architecture is advised. The individual components have distinct requirements, Capture requires large amounts of disk space to store received PCAP files, by contrast, Elasticsearch requires large amount of RAM for idexing and searching. The viewer has the smallest requirements of the three, allowing it to be located anywhere.

Read more →

Installation of Scirius CE

Miroslav Kohútik 0 Comments

Installation of Scirius CE

  • Author: Miroslav Kohútik
  • Operating system : Ubuntu 16.04

Scirius Community Edition is a web interface dedicated to Suricata ruleset management. It handles the rules file and updates of the associated files.

This guide will walk you through the installation of Scirius Community Edition on Ubuntu 16.04 operating system.
Before proceeding with installation of Scirius CE, you need to have IDS Suricata installed. Installation guide for Suricata can be found here.

Read more →

Installation of Suricata

Miroslav Kohútik 1 Comment

Installation and basic setup of Suricata

First, add the latest stable Suricata repository to APT:

sudo add-apt-repository ppa:oisf/suricata-stable
sudo apt-get update

Now you can either install Suricata with:

sudo apt-get install suricata

or the Suricata package with built-in (enabled) debugging

sudo apt-get install suricata-dbg

Basic setup

Start with creating a directory for Suricata’s log information.

sudo mkdir /var/log/suricata

To prepare the system for using it, enter:

sudo mkdir /etc/suricata

The next step is to copy classification.config, reference.config and suricata.yaml from the base build/installation directory (ex. from git it will be the oisf directory) to the /etc/suricata directory. Do so by entering the following:

sudo cp classification.config /etc/suricata
sudo cp reference.config /etc/suricata
sudo cp suricata.yaml /etc/suricata

Auto setup

You can also use the available auto setup features of Suricata:

The make install-conf option will do the regular “make install” and then automatically create/setup all the necessary directories and suricata.yaml.

 ./configure && make && make install-conf

The make install-rules option will do the regular “make install” and it automatically downloads and sets up the latest ruleset from Emerging Threats available for Suricata.

./configure && make && make install-rules

The make install-full option combines everything mentioned above (install-conf and install-rules) – and will present you with a ready to run (configured and set up) Suricata

./configure && make && make install-full

Source:

Suricata – Ubuntu installation

Read more →

Moloch – Web interface specification

Tomáš Mokoš 0 Comments

Viewer is a feature of Moloch that makes processing of captured data easier with the use of web browser GUI. Viewer offers access to numerous services, most notably :

  • Sessions
  • SPI View
  • SPI Graph
  • Connections
  • Upload
  • Files
  • Users
  • Stats

Sessions

The main tab illustrated below contains:

  • Search field – here you can enter different search filtering rules. Being similar to Wireshark, it enables, for example, to search for results containing source/destination IP address (ip.dst== X.X.X.X && ip.src== X.X.X.X), port(port.dst 80 || port.src 80), web URL (http.uri == www.foo.com), time of delivery, etc.
  • Country map – results are cross-referenced with GeoIP database and the number of occurrences from a given country is displayed on a map using shades of the color purple.
  • Occurrence frequency chart – this bar chart represents traffic in a given time period. The y-axis can be set to represent the number of sessions, packets or bytes of data.

Okno vyhladavania

  • Record window – displays individual connections and their parameters, including times of beginning and end of session, source and destination IP addresses, source and destination port, packet count and session size in bytes. After unpacking, the session can be exported to a PCAP file, where you can add your own tag and see detailed information about the nature and origin of the packet. A DNS request is Illustrated below

Okno zaznamov

SPI View

SPI (Session Profile Information) View is used to take a deeper look at connection metrics. Instead of writing queries manually, queries can be expanded by clicking on the respective actions in SPI view, which adds the requested item to a query by using AND or AND NOT operators. This tab also provides a quick view of occurrence frequency of all items requested by the user. Furthermore, SPI view also offers a quick summary of monitored IP addresses in a given time period, HTTP response codes, IRC NICKs/channels, etc.

SPI View

SPI Graph

SPI Graph provides user with visualization of any given item contained in SPI View. This tab is very useful for displaying activity for a type of SPI, as well as in-depth analysis. The illustration below displays the demand for HTTP with http.method view selected. The displayed bar charts represent the amount of sessions/packets/databytes captured. To the right of each chart, after clicking View Map button, a map containing origins of the sessions is displayed. The methods GET, POST, HEAD OPTIONS and CONNECT are also displayed below. Maximum number of charts displayed can be set by the Max Elements field, with the default value being 20 and the maximum value being 500. Charts can also be sorted by number of sessions or by name. Moloch also offers a refresh rate setting from 5 to 60 seconds.

SPI Graph

Connections

The connections tab provides the user with a tree graph rooted in a source or destination node of the user’s own selection. The illustration below displays the relations between source IP address and destination IP address and port, with 500 sessions displayed using the query size setting. The minimum query size is 100 and the maximum size is 100000. Furthermore, the minimum number of connections can be set to values from 1(like in the case below) to 5. Node dist defines the distance of nodes in pixels. Any node in the graph can be moved after clicking the Unlock button. Source nodes are marked in violet and destination nodes are marked in yellow. By clicking a node, information about the node is displayed: Type (source/destination), Links (number of connected nodes), number of sessions, bytes, databytes and packets. This view also allows for nodes to be added with AND/OR logic as well as hiding nodes by clicking the Hide Node tab. This tab is appropriate for users who prefer node data analysis over visualization.

Connections

Upload

This tab is used to import PCAP files selected by clicking on Search button and clicking upload. Tags (separated by commas) can be appended to the imported packets, however, this option must be enabled in data/moloch/etc/config.ini. The Upload option is in experimental state and can be enabled by the following command

/data/moloch/bin/moloch-capture --copy -n {NODE} -r {TMPFILE}

NODE is the node name, TMPFILE is the file to be imported, CONFIG is the configuration file and TAGS are the tags to be appended.

Files

The files tab displays the table of archived PCAP files. Details include: file number, node, file name, whether the file is locked, upload date and file size. File size (in GB) is defined in data/moloch/etc/config.ini using the maxFileSizeG parameter. If a file is locked, Moloch cannot delete it and it must be deleted manually.

Users

The users tab defines access rights to individual users. It allows to add/remove user accounts and to edit their passwords.

Stats

The stats tab provides visual representation and table display of metrics for each node. Other display options can also be used:

Packets/Sec Sessions/Sec Active TCP Sessions Total Dropped/Sec Free Space (MB)
Free Space (%) Fragments Queue Active UDP Sessions Input Dropped/Sec Bytes/Sec
Memory Active Fragments Active ICMP Sessions Active Sessions Bits/Sec
ES Queue Overload Dropped/Sec Disk Queue CPU Memory (%)
Waiting Queue Fragments Dropped/Sec Closing Queue Packet Queue ES Dropped/Sec

Statistics for each node display: current time on the node, number of sessions, remaining free storage, CPU usage, RAM usage, number of packets in queue, number of packets per second, number of bytes per second, number of lost packets per second, number of packets dropped due to congestion and number of packets dropped by Elasticsearch per second. Charts depicting visualization of these stats can be displayed by clicking the plus(+) icon. Elasticsearch statistics contain number of files saved under unique ID, HDD size, Heap size, OS usage, CPU usage, bytes read per second, bytes written per second and number of searches per second.

Sources:

  • CRZP Komplexný systém pre detekciu útokov a archiváciu dát – Moloch
  • Report Projekt 1-2 – Marek Brodec

Read more →

Port-Mirroring

Tomáš Mokoš 0 Comments

Port-Mirroring

Port mirroring is used on a network switch to send a copy of network packets seen on one switch port (or an entire VLAN) to a network monitoring connection on another switch port. This is commonly used for network appliances that require monitoring of network traffic such as an intrusion detection system, passive probe or real user monitoring (RUM) technology that is used to support application performance management (APM).

In our particular case, the faculty provided us with a Cisco Catalyst 2960 switch. We have configured this switch to mirror all internet-bound data traffic traversing the interface connected to network gateway, to the interface connected to Moloch server. As a result, we can now monitor all inbound and outbound lab traffic.

Switch(config)#monitor session 1 source fa0/1 both

– This command specifies source interface as fa 0/1. The parameter “both” specifies both directions to be monitored.

Switch(config)#monitor session 1 destination interface fa0/24

– This command defines the destination interface of mirrored traffic

Port-Mirror

Sources

  • CRZP Komplexný systém pre detekciu útokov a archiváciu dát – Moloch

Read more →

Moloch – Usage possibilities of Moloch

Tomáš Mokoš 0 Comments

Usage possibilities of Moloch

  • Author : Tomáš Mokoš

Moloch offers many distinct usage possibilities, the set of which is not limited to the ones mentioned down below and can be expanded by individual users, provided they can find other applications of this service:

    • DOS attacks – Analysis of connections suspected of originating DOS attacks.
    • Geolocation – Identification of connection’s country of origin.
    • Access Intelligence – Helps with the analysis of authorized/non-authorized access to system resources, applications, servers, system operation and different functions. You can also perform depth analysis (with the use of tagging) of a particular system, application or service running in the network
    • Port connection usage – amount of connections on a particular port.
    • URL connection usage – amount of connections tied to a particular URL by requests.
    • Data volumes

    As an example, we will show you the use of Moloch for analysis of the CICIDS 2017 dataset, where we analyze a DDoS Hulk attack. First, we filter the traffic. Using the command tags == CICIDS2017_WEDNESDAY && ip.dst == 192.168.10.50 we extract the traffic from the day of the attack with the webserver’s IP as the destination address.

    Moloch1

    Afterwards, in the  SPI Graph tab, we can look up the source IP addresses that communicated with this web server by setting SPI Graph to ip.src.

    As we can see, the IP address 172.16.0.1 generated 84315 of the 85268 sessions, making it likely to be the address of the attacker.

    Moloch2

    In the SPI View tab, we can see that the network communication did not originate from just one port, but several thousands and almost all of these were bound for the port 80. Furthermore, we can see that most of the communication was bound for miscellaneous URIs, which is characteristic of a Hulk attack. By generating random URIs, Hulk attack causes resource depletion of the web server, making the server inaccessible.Moloch3
    Moloch4

    Sources

    • CRZP Komplexný systém pre detekciu útokov a archiváciu dát – Moloch

Read more →

Moloch – Network interface configuration

Tomáš Mokoš 0 Comments

Considering the possibility of packet loss at high traffic flows, it is recommended for the packet capture interface to NOT be the same as the interface connected to the internet, in this case, the interface assigned with static IP address. On the server in our lab there are two interfaces, one for packet capture and one for “outside” communication. To prevent packet loss, it is recommended to increase the Moloch-side interface’s buffer to maximum and turn off most of the NIC’s services by using the following commands:

ethtool –G enp0s9 rx 4096 tx 4096 
ethtool –K enp0s9 rx off tx off gs off tso off gso off

You can find out the maximum buffer size using the ethool -g command, to check NIC’s services use the ethtool -k command. Disable most of NIC’s services, since you want to capture network traffic instead of what the OS can see, they are not going to be used anyway.

Sources

  • CRZP Komplexný systém pre detekciu útokov a archiváciu dát – Moloch

Read more →

Moloch – Load Testing

Tomáš Mokoš 0 Comments

  • Author : Tomáš Mokoš, Marek Brodec

In our topology, the server running Moloch was connected to a 100Mbps switch, therefore, even though the generated network traffic reached 140Mbps, the flow was subsequently limited on switch.

Single source to single destination test

At first, while generating packets with a generated IP address from cloud to a lab PC, we have had a problem with the cloud’s security policies. These policies prevented the sending of packets with source IP address different from the one assigned to the hosting cloud instance, therefore we have only generated traffic from a single source IP address to a single destination IP address.

We have observed the volume of incoming data and its effect on performance using network monitoring tools mentioned in the previous chapters. The graphs below show that in the moment of traffic spike, CPU usage spiked as well, however it did not exceed 20%. The graph titled “Mem” shows the RAM usage of all running services including the instances of Moloch, Snort and many others with the addition of cache memory.

Bigdesk1

The following graph shows utilization of RAM allocated exclusively to Elasticsearch. Having been allocated with 25 GB of RAM, Elasticsearch uses 4.2 GB in this instance. This test revealed that Elasticsearch uses unnecessary amount of RAM and after the test we have decreased the amount of allocated RAM to 18 GB.

Bigdesk2

N sources to single destination test

The previous packet generation was not ideal because the relations had many shared characteristics. All test cases had identical IP addresses, number of packets and payload, which made indexing much easier. To approximate real network traffic as precisely as possible, we have tried to address the issue with source IP address generation. The solution was packet generation from our own laptop with OS Kali Linux running in VirtualBox as a source of attacks. The laptop was connected to our switch and generated traffic towards the lab. The packets passed through the interface which was mirrored towards our server.

Test results showed CPU utilization rise in the range from 20 to 30%.

Bigdesk3

“Heap Mem” graph shows that, of the allocated 19 GB of RAM, 6.3 GB is used .

Bigdesk4

The results revealed that the amount of allocated RAM can be further decreased, freeing up space for other processes. Graph titled “Mem” shows that all running processes use 31 GB of RAM, therefore it would be advisable to stop all unnecessary processes before testing. This, however, was not an option for us, because in parallel with us, other students have been working on their respective bachelor’s theses.

Testing evaluation

Restart of services was performed before the single source to single destination test, but service instances were not restarted before other tests. We have concluded that higher network traffic mainly influences CPU utilization. Graphs of both Elasticsearch RAM usage and overall RAM usage do not show any significant spikes during the arrival of generated test traffic. The table shows that during normal traffic (4.5 Mbps at the time), RAM usage is higher in both cases above than before the first test, where traffic was 20 times higher than normal. These two graphs are mainly affected by the time elapsed from service start and possibly the amount of captured data. These tests were performed in the same order as they were mentioned above, with one week between the individual tests. Even though the graphs indicate utilization of 31 out of 32 GB of RAM, from the 18 GB allocated to Elasticsearch, only one third is used. This can be solved by reducing the amount of RAM allocated to Elasticsearch, thus freeing up space for other processes.

Cache can be cleared by the following command:

free && sync && echo 3 > /proc/sys/vm/drop_caches && free

This drops RAM usage from 31 GB to 20 GB.

Sources

  • Report Project 1-2 – Marek Brodec

Read more →

Moloch – CPU, RAM and HDD usage

Tomáš Mokoš 0 Comments

  • Author : Tomáš Mokoš, Marek Brodec

Considering the fact that the formulas that we used to calculate for how many days can Moloch archive network traffic and what hardware should we use were only approximate, we have decided to measure some statistics to help us clear up these values.

From the Elasticsearch node quantity calculation formula: ¼ * [average network traffic in Gbit/s] * [number of days to be archived], we get that at 2 Mbit/s, one node should suffice.

Using the formula 86400 * [average network traffic per second] * [number of days to be archived], we can calculate that 1Gbit/s of traffic takes up 11TB of disk space daily, therefore 2Mbit/s of traffic will take up 22GB per day. At this rate we can archive approximately 113 days’ worth of raw data.

Since our lab server is not under heavy load, only 7GB (22%) of RAM is used on average. This is due to the the fact that during the night the network traffic is minimal. Non-uniform network traffic creates distortions, therefore long-term observation would be desirable.

RAM

Moloch by itself uses about 5% of total CPU utilization and 1.0 to 1.3 GB of RAM (3-3.5%).

Utilization of disk capacity was 340 GB (17%) on the first week, 220 GB (11%) on second week and 140 GB (7%) on third week.

GB

Thanks to our use of data trimming we have managed to archive 6 months’ worth of traffic, although the actual value is closer to 4 months, since during the two months of the exam period, network traffic was very low. The captured data took up 52% (1.3 TB) of storage.

HDD

Sources

  • Report Projekt 1-2 – Marek Brodec

Read more →

Forensic analytic tools

Tomáš Mokoš 0 Comments

Forensic analytic tools

  • Author : Tomáš Mokoš

NetworkMiner

NetworkMiner is a Network forensic analysis tool (NFAT) for Windows operating systems. NetworkMiner can be used as a passive network sniffer/packet capturing tool in order to detect operating systems, sessions, hostnames, open ports etc. NetworkMiner’s primary purpose is collection of data regarding network hosts, rather than data regarding network traffic. In addition to direct file capture, NetworkMiner can also parse PCAP files for off-line analysis and to regenerate/reassemble transmitted files and certificates from PCAP files. This function can be used for extraction and archiving of media files transferred through the network. Supported file extraction protocols are FTP, SMB and HTTP. Extracted user credentials (username and password) for supported protocols can be found in the Credentials tab. Other useful features include keyword search in the captured/archived data and Nmap MAC vendor lookup.

Xplico

Xplico is an open-source NFAT. The goal of Xplico is the extraction of application data contained in a capture sample of Internet traffic. For example, Xplico can export all e-mails (POP, IMAP and SMTP), HTTP contents, VoIP calls, FTP and TFTP files, etc.

Elastic Stack

Elastic Stack provides reliable and safe transfer of data of any format from any source and real-time searching analysis and visualization. Elastic Stack consists of Kibana, Elasticsearch, Beats and Logstash. Elasticsearch is a search and analytics engine. Beats is a dta gathering platform. Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a “stash” like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch.

Sguil

Sguil is built by network security analysts for network security analysts. Sguil’s main component is an intuitive GUI that provides access to realtime events, session data, and raw packet captures. Sguil facilitates the practice of Network Security Monitoring and event driven analysis. The Sguil client is written in tcl/tk and can be run on any operating system that supports tcl/tk (including Linux, BSD, Solaris, MacOS, and Win32).

Sources:

  • CRZP Komplexný systém pre detekciu útokov a archiváciu dát – Moloch

Read more →