heiDGAF - Domain Generation Algorithms Finder

Machine learning-based DNS classifier for detecting Domain Generation Algorithms (DGAs), tunneling, and data exfiltration by malicious actors.
Explore the docs »

Report Bug · Request Feature

Continuous Integration

HOST_IP=127.0.0.1 docker compose -f docker/docker-compose.yml up

python -m venv .venv
source .venv/bin/activate

sh install_requirements.sh

Alternatively, you can use pip install and enter all needed requirements individually with -r requirements.*.txt.

Now, you can start each stage, e.g. the inspector:

python src/inspector/inspector.py

(back to top)

To configure heiDGAF according to your needs, use the provided config.yaml.

The most relevant settings are related to your specific log line format, the model you want to use, and possibly infrastructure.

The section pipeline.log_collection.collector.logline_format has to be adjusted to reflect your specific input log line format. Using our adjustable and flexible log line configuration, you can rename, reorder and fully configure each field of a valid log line. Freely define timestamps, RegEx patterns, lists, and IP addresses. For example, your configuration might look as follows:

- [ "timestamp", Timestamp, "%Y-%m-%dT%H:%M:%S.%fZ" ]
- [ "status_code", ListItem, [ "NOERROR", "NXDOMAIN" ], [ "NXDOMAIN" ] ]
- [ "client_ip", IpAddress ]
- [ "dns_server_ip", IpAddress ]
- [ "domain_name", RegEx, '^(?=.{1,253}$)((?!-)[A-Za-z0-9-]{1,63}(?<!-)\.)+[A-Za-z]{2,63}$' ]
- [ "record_type", ListItem, [ "A", "AAAA" ] ]
- [ "response_ip", IpAddress ]
- [ "size", RegEx, '^\d+b$' ]

The options pipeline.data_inspection and pipeline.data_analysis are relevant for configuring the model. The section environment can be fine-tuned to prevent naming collisions for Kafka topics and adjust addressing in your environment.

For more in-depth information on your options, have a look at our official documentation, where we provide tables explaining all values in detail.

To monitor the system and observe its real-time behavior, multiple Grafana dashboards have been set up.

Have a look at the following pictures showing examples of how these dashboards might look at runtime.

(back to top)

To train and test our and possibly your own models, we currently rely on the following datasets:

• CICBellDNS2021
• DGTA Benchmark
• DNS Tunneling Queries for Binary Classification
• UMUDGA - University of Murcia Domain Generation Algorithm Dataset
• DGArchive

We compute all features separately and only rely on the domain and class for binary classification.

For testing purposes, we provide multiple scripts in the scripts directory. Use real_logs.dev.py to send data from the datasets into the pipeline. After downloading the dataset and storing it under <project-root>/data, run

python scripts/real_logs.dev.py

to start continuously inserting dataset traffic.

We feature two trained models:

1. XGBoost (src/train/model.py#XGBoostModel) and
2. RandomForest (src/train/model.py#RandomForestModel).

After installing the requirements, use src/train/train.py:

> python -m venv .venv
> source .venv/bin/activate

> pip install -r requirements/requirements.train.txt

> python src/train/train.py
Usage: train.py [OPTIONS] COMMAND [ARGS]...

Options:
  -h, --help  Show this message and exit.

Commands:
  explain
  test
  train

Setting up the dataset directories (and adding the code for your model class if applicable) lets you start the training process by running the following commands:

> python src/train/train.py train  --dataset <dataset_type> --dataset_path <path/to/your/datasets> --model <model_name>

The results will be saved per default to ./results, if not configured otherwise.

> python src/train/train.py test  --dataset <dataset_type> --dataset_path <path/to/your/datasets> --model <model_name> --model_path <path_to_model_version>

> python src/train/train.py explain  --dataset <dataset_type> --dataset_path <path/to/your/datasets> --model <model_name> --model_path <path_to_model_version>

This will create a rules.txt file containing the innards of the model, explaining the rules it created.

(back to top)

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

(back to top)

Distributed under the EUPL License. See LICENSE.txt for more information.

(back to top)