Design and Evaluation of a Custom Isolated Virtual Malware Lab: Automated Artifact Generation and Payload Recovery for Packed and Obfuscated Malware
Zoom Link: https://umassd.zoom.us/j/97089567317?pwd=p2ZghinLHjhbCp9bhueDAT2kmidSaW.1
Dr. Gokhan Kul
gkul@umassd.edu
https://umassd.zoom.us/j/97089567317?pwd=p2ZghinLHjhbCp9bhueDAT2kmidSaW.1
Thesis Advisor: Dr. Gokhan Kul - CIS/DSC, University of Massachusetts, Dartmouth
Committee Members:
- Dr. Adnan El-Nasan, CIS/DSC, University of Massachusetts, Dartmouth
- Dr. Asif Kamal Turzo, CIS/DSC, University of Massachusetts, Dartmouth
Abstract
Traditional malware reverse engineering relies heavily on static analysis, often producing decompiled pseudo-source that lacks meaningful semantic information and behavioral context, particularly for packed or obfuscated binaries. Although retrieval-augmented generation (RAG) has shown promise for enhancing binary analysis, existing approaches primarily retrieve static code artifacts and rarely integrate complementary evidence collected before and after payload recovery and during runtime execution. To address these limitations, prior work proposed an autonomous multimodal hybrid RAG framework for reverse engineering packed and obfuscated malware. The broader proposed framework is designed to integrate static artifacts before and after payload recovery, as well as runtime execution evidence from the target binary to support semantic reconstruction and binary-level behavioral analysis.
This work implements the first stage of the previously proposed autonomous multimodal hybrid RAG framework. This thesis presents the design, implementation, and evaluation of a custom isolated virtual malware laboratory and an automated artifact generation pipeline that integrates static analysis, controlled execution, and automated payload recovery for packed and obfuscated malware. The proposed system enables the systematic collection of static artifacts before and after payload recovery, alongside complementary dynamic artifacts collected during controlled execution. Automated payload recovery techniques are implemented for UPX-packed payloads with anti-static-unpacking modifications, as well as Hyperion-encrypted payloads, allowing the pipeline to generate decompiled code and other static artifacts relevant to the actual malware logic.
The proposed system is evaluated across three dimensions: laboratory security, artifact generation, and payload recovery. Laboratory security is evaluated through systematic validation of network isolation, inter-VM communication restrictions, and access-control policies, while environmental realism is examined as an additional design consideration. Artifact generation is evaluated in terms of correctness and completeness. Payload recovery is evaluated through reconstruction success, executable validity, original entry point (OEP) correctness, and comparisons of static artifacts generated from the unobfuscated, obfuscated, and recovered executables, including decompiled code, disassembly, and control-flow graphs (CFGs).
Together, these contributions establish a controlled and automated foundation for generating heterogeneous malware reverse-engineering artifacts to support future retrieval-augmented semantic reconstruction and binary-level behavioral analysis.
For further information please contact Dr. Gokhan Kul at gkul@umassd.edu. All CIS MS students are encouraged to attend.