Hi, I am Nick, and I work for UDV Technologies as a junior researcher. Today, I want to share some interesting information concerning DNS tunnels and ways to detect them. The article will be on a technical side, so some background in ML would definitely help.

In the modern era, digital technologies are infiltrating every aspect of our lives, emphasizing the importance of data security. That is why special attention is paid to the detection of hidden information transmission channels that adversaries could use to bypass security systems. One of the methods used by adversaries involves DNS tunneling. This article focuses on explaining what DNS tunnels are, how adversaries create them, and how to use machine learning methods for effective tunnel detection.

DNS and DNS tunneling methods

The DNS (Domain Name System) protocol was originally designed as a tool for converting domain names to the corresponding IP addresses and vice versa. However, adversaries got creative and began to use the protocol both to detect hosts, and to transfer data via special channels. These channels are known as DNS tunnels.

DNS tunnels create hidden channels that allow transferring data through standard DNS transactions, embedding information in protocol command fields that are not normally used for data transfer. Their main purpose is to bypass the usual detection and blocking mechanisms. Why DNS? Because it is one of the most widely used protocols on the Internet.

A DNS tunnel usually consists of a special DNS client that integrates data into requests, and a special DNS server that understands such requests and generates special responses. This may include generating a large number of subdomains with encoded data, reordering requests and responses to mask the tunnel, and embedding information in DNS packet parameters.

Figure 1 DNS tunnel diagram

The DNS packet contains several fields that can be analyzed to detect DNS tunnels:

• DNS packet header includes a query ID, control flags (QR - query/response definition, Opcode - operation code, AA - authoritative server response, TC - connection fault, RD - recursion,   RA - recursion availability), number of queries, responses, authoritative and additional queries, responses, and other test parameters.

• DNS queries include data on DNS queries, including domain names and query types (for example, A - IPv4 address query, MX - mail server query, CNAME - canonical name query, etc.).

• DNS responses contain information on records that match DNS queries. Each response can include a response type (for example, A, CNAME, MX), an IP address record and a time-to-live value (TTL).

• Authoritative DNS records can contain information on servers that have authority for a given domain name.

• Additional DNS records can contain additional records that may be useful for the response interpretation.

Figure 2 Regular DNS query example

The DNS packet analysis for DNS tunnel identification can include checking DNS query and response length, query frequency, use of unusual subdomains, analyzing query types (for example, TXT-type queries), detecting abnormal query and response sequences, and comparing to known DNS tunnel signatures. These analysis methods can help detecting possible DNS tunnels and other malicious activity that exploits the DNS protocol.

Figure 3 Tunneled DNS query example

Tunnel detection method overview

Researchers and cybersecurity professionals have been drawn to the topic of detecting DNS tunnels, which resulted in multiple papers, studies and tools. Existing detection methods range from statistical traffic analysis and monitoring network behavior to developing signatures that detect DNS tunnel features.

Figure 4 The variety of the detection methods

The figure above shows some of the well-known ways to solve this issue. Let’s have a closer look at them.

• Signature-based technology: analyzing signature features that are specific to objects or phenomena to profile or detect.

• Threshold-based technology: profiling or detecting based on a threshold value for selected characteristics.

• Unsupervised learning: analyzing unlabeled data to detect structure, clustering, or dimensionality reduction.

• Supervised learning: training a model on labeled data, where each example has a known label, to predict new data labels for the data previously unseen.

• Deep learning-based technology: using deep neural networks with multiple layers to extract high-level patterns from data.

In order to select a method, a thorough study of existing approaches should be performed. A few detailed studies quite worth reviewing are available on the Internet.

1) DNS Tunneling Detection by Cache-Property-Aware Features.

The authors focus on the trace left behind by DNS tunneling, which is challenging to conceal. In situations involving data leakage through DNS tunneling, malware establishes a direct connection to the caching DNS server, leading to cache misses due to the DNS tunneling queries generated. Given that typical DNS transactions typically result in successful cache hits, the cache hit/miss ratio can serve as an abnormal activity indicator.

2) A Hybrid Method of Genetic Algorithm and Support Vector Machine for DNS Tunneling Detection. This research focuses on a hybrid method that combines a genetic algorithm method for optimal feature selection with a support vector method (SVM) classifier. The authors achieved an F-measure of 0.946 using this method.

3) DNS Tunneling: A Deep Learning based Lexicographical Detection Approach. This paper describes a detection method based on a convolutional neural network (CNN) with minimal architectural complexity to enhance performance. Despite its simple architecture, the CNN model detects over 92% of the total number of DNS tunneling domains with a false positive rate close to 0.8%.

Dataset overview

Developing and evaluating DNS tunnel detection methods through machine learning, like any other machine learning task, requires high-quality datasets. In our case, a few diverse data sources were used to create training and testing datasets. Data obtained from our lab environment served as the basis for our own training dataset. To generate this data, we deployed two virtual machines (client and server), registered a domain name, and configured an NS record to facilitate communication between the client and server VMs. We generated both conventional traffic and traffic using DNS tunnels with the help of tcp2dns, iodine, and tuns. This resulted in a range of scenarios and variations of tunnel traffic, which were used to enhance model training.

Supplementary datasets to validate and evaluate the model performance were used, specifically, pcap-files sourced from open repositories on GitHub (see here, here and here), as well as the data from Canadian Institute for Cybersecurity (see here). These datasets provided more traffic scenarios than the model could ever encounter in real-world environments and enabled us to thoroughly assess its performance.

This approach has also enhanced the generalization performance of the studied models and their accuracy in detecting DNS tunnels.

LightGBM

A machine learning algorithm using LightGBM (Light Gradient Boosting Machine) was used as a guiding model for efficient DNS tunnel detection. This algorithm performs gradient boosting and is renowned for its high performance and accuracy. It is also well-known for efficient processing of the extensive datasets and detecting complex dependencies between properties, making it suitable for detecting subtle and invisible patterns within DNS traffic.

An important note on the strategy chosen to detect DNS tunnels: it’s a good idea to split the traffic into DNS queries and DNS responses, and then run two separate models to analyze each message type. This enables a better precision in detection of the features and differences between queries and responses, which is crucial for detecting potential DNS tunnels.

Figure 5 DNS traffic processing diagram

Another point to consider is that only the statistical characteristics of the traffic were used to ensure models universality and generalization. These attributes can be extracted from the traffic without disclosing the content of the queries or responses. Here’s the list of the attributes and their impact:

Request:

• 'qd_qname_len’ / 'qd_qname_shannon’ - requested domain length / entropy

• 'ancount’ / 'nscount’ - number of records in the answer / name server block

• 'chars’ - number of characters in the domain name

• 'digits’ – number of digits in the domain name

• 'ratio’ – vowels to consonants ratio in the domain name

Response

• 'qd_qname_len’ / 'qd_qname_shannon’ - requested domain length / entropy

• 'an_rdata_len','an_rdata_shannon’ - response data length / entropy

• 'an_rrname_len','an_rrname_shannon’ - domain length/entropy in the response

• 'ancount’ / 'nscount’ - number of records in the answer / name Server block

These statistical features enable models to detect patterns unique to DNS tunnels with minimal reliance on the specific content of the transmitted information.

Metrics and results obtained

Three different datasets were used to evaluate the efficiency of the model. Let’s look at the model performance for each of them.

GitHub dataset:

The dataset sourced from GitHub repositories offered numerous test cases developed by the research community. This dataset was used to evaluate the model's ability to generalize across various tunnel traffic variations.

Request model:

Response model:

The results show that the model has successfully learned to detect typical tunnels.

CIC dataset:

The dataset obtained from the CIC provided more complex and realistic scenarios created with contemporary threats in mind. This dataset was used to test how the model handles sophisticated DNS tunnel bypass and detection methods, and evaluate its ability to detect new and unknown patterns.

Request model:

Response model:

The model exhibited sufficient performance on the CIC dataset, indicating that it has not been overtrained and is capable of performing in unfamiliar situations.

These trained models were successfully integrated into a Network Traffic Analysis system currently in development. The prospective NTA is designed as an advanced solution for network traffic analysis, offering robust detection capabilities for anomalies and malicious activities. Implemented as a vital component of the security infrastructure, the NTA will ensure reliable protection against potential threats on the network.

Are you interested in prospective cybersecurity solutions? Contact UDV Technologies team to find out more on how AI and machine learning can help in protecting your business!