BFOR - 418/618 - Reverse Engineering Malware

Basic Dynamic Analysis

Katta Sreenivasulu, Tarun Sai
September 13, 2023

Dr. Prinkle Sharma

University at Albany, SUNY

1
Contents
1 Abstract 3

2 Architecture of X86 Processor 4

2.1 Registers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1.1 General Purpose Registers . . . . . . . . . . . . . . . . . . 4
2.1.2 Segment Registers . . . . . . . . . . . . . . . . . . . . . . 4
2.1.3 Flag Registers . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Control Unit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 Bus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.4 ALU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3 Software Used 5
3.1 Oracle Virtual Box VM . . . . . . . . . . . . . . . . . . . . . . . 5
3.2 Ubuntu or Kali Linux . . . . . . . . . . . . . . . . . . . . . . . . 6
3.3 NASM Shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

4 Programs 7
4.1 Program 1 – Hello World . . . . . . . . . . . . . . . . . . . . . . 7
4.1.1 CODE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.1.2 Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.2 Program 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.2.1 Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.3 Program 3 – Display User Input Number . . . . . . . . . . . . . 9
4.3.1 Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.4 Program 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.4.1 Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
4.5 Program 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.5.1 Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.6 Program 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.6.1 Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2
1 Abstract
Assembly is a low-level programming language that acts as an intermediary be-
tween a high-level programming language and processor-specific machine code.
Assembly language provides a human-readable representation of a computer’s
Instruction Set Architecture, allowing programmers to write efficient code that
directly interacts with computer hardware like CPU, main memory, and helps
programmers optimize for specific hardware, resulting in faster and resource-
efficient code.
In this lab, we will be working on understanding x86 architecture of the
processor, different hardware, functionality of these hardware, and how we use
assembly on these hardware to utilize them effectively for generating machine-
specific object code.

3
2 Architecture of X86 Processor
2.1 Registers
2.1.1 General Purpose Registers
Data Registers There are four data registers in x86 processor: EAX, EBX,
ECX, and EDX. They are 32-bit in size, and their significant 16 bits are named
AX, BX, CX, and DX respectively.
EAX: Primary accumulator used for I/O and most arithmetic instructions.
EBX: Base register used in indexed addressing.
ECX: Count register used to store loop count in iterative operations.
EDX: Data register used in I/O operations, mainly used with EAX for multi-
plication and division operations involving large numbers.

Pointer Registers There are three categories of pointer registers in x86 pro-
cessor: EIP, ESP, and EBP for 32-bit, and the corresponding 16-bit portions
are IP, SP, and BP.
EIP: Instruction Pointer used to store the offset address of the next instruction
to be executed.
ESP: Stack Pointer provides the offset value within the program stack, pointing
to the top of the stack.
EBP: Base pointer stores the base address of the memory.

Index Registers There are two Index registers: Source Index (ESI) and Des-
tination Index (EDI).
ESI: Used as Source Index for string operations.
EDI: Used as Destination Index for string operations.

2.1.2 Segment Registers

There are six segment registers: CS (Code Segment), DS (Data Segment), SS
(Stack Segment), ES (Extra Segment), FS, and GS (extra segments after E
Segment).
ECS: Code Segment contains all instructions to be executed, and ECS stores
the starting address of the code segment.
EDS: Data Segment contains data, constants, and work areas. EDS stores the
starting address of the data segment.
ESS: Stack Segment contains data and return addresses of procedures or sub-
routines. ESS register stores the starting address of the stack.

4
2.1.3 Flag Registers
The EFLAGS is a 32-bit register used as a collection of bits representing Boolean
values to store the results of operations and the state of the processor.

Figure 1: EFLAGS Register

2.2 Control Unit

The control unit is responsible for fetching the instruction from the Instruction
Queue (Main Memory) using the stack pointer or Program Counter, which refers
to the top of the stack containing the next instruction to be executed. It also
controls the flow of data between registers, ALU, and main memory.

2.3 Bus
The data bus transfers instructions and data between the CPU and memory.
The I/O bus transfers data between the CPU and the devices in the computer
system that have input/output commands.

2.4 ALU
The Arithmetic Logic Unit (ALU) operates on data stored in general-purpose
registers. ALU operations take one or two inputs from registers and store the
result back into the register. It supports Arithmetic, Logical, Relational, and
Shift operations.

3 Software Used
Here we are installing Debian-based Linux OS in a virtualized manner. So, to
accomplish that, we need two software: Oracle Virtual Box and Ubuntu or Kali
Linux. Inside these guest OS, we need to install NASM shell to execute our
assembly code by generating object code.

3.1 Oracle Virtual Box VM

1. Download Oracle Virtual Box from Downloads -> Oracle VM VirtualBox.
2. Follow the steps proceeding as the installation guide displays.

5
3.2 Ubuntu or Kali Linux
1. Download the Ubuntu ISO file from Ubuntu Virtual Machine Images
for VirtualBox and VMware (osboxes.org) or directly pre-installed VM
from Get Kali | Kali Linux.

2. Dynamically allocate HDD or SDD memory for the guest OS to work on

and install its packages in.
3. For Kali Linux, after downloading the pre-installed VM, extract it to see
two files of .vdi and .vbox format.

4. Click on Add and provide the installation guide the above two mentioned
files when it requests.
5. Use the default username – osboxes.org and password osboxes.org if Ubuntu
ISO is used.
6. Use default username kali and password kali if proceeded with Kali Linux.

3.3 NASM Shell

NASM Shell is pre-installed in Kali Linux pre-built VM; however, it must be
manually installed in Ubuntu Xenial from osboxes.org.

1. After logging into Ubuntu Xenial, open the link provided to download the
https://www.nasm.us/ tool:
2. Select the latest stable version.
3. Download the gzip file, which is in the tar.x.xxx.x.gz file format.

4. Once downloaded, extract the zip file to a convenient directory.

5. Change the directory into nasm (cd to nasm) location and execute config-
ure file by ./configure command.
6. Make the binaries using make to build the nasm and ndisasm binaries.

7. Install the final binaries using make install to install nasm and ndisasm
in /usr/local/bin and to install the manual in man page.

6
4 Programs
4.1 Program 1 – Hello World
In this program, we will display the message “Hello World!” using assembly
language. The code is stored in section .text, and section .data is for initialized
data or constants. MOV eax, 4 is used to store the direct value 4 into EAX
(accumulator), and int is an interrupt call used to call the kernel and interrupt
the execution.

4.1.1 CODE
section .data
hello db ’Hello, world!’,0xa
len equ $-hello

section .text
global _start

_start:
; write to stdout
mov eax, 4
mov ebx, 1
mov ecx, hello
mov edx, len
int 0x80

; exit
mov eax, 1
xor ebx, ebx
int 0x80

4.1.2 Result

Figure 2: Hello World Output

7
4.2 Program 2
In this program, we are printing a message “Displaying 9 stars” and 9 stars in a
new line. Here we are printing output by using sys write system call by loading
eax with 4.

CODE
section .data
stars db ’*********’, 0xa
len equ $-stars

section .text
global _start

_start:
; write to stdout
mov eax, 4
mov ebx, 1
mov ecx, stars
mov edx, len
int 0x80

; exit
mov eax, 1
xor ebx, ebx
int 0x80

4.2.1 Result

Figure 3: Displaying 9 Stars

8
4.3 Program 3 – Display User Input Number
In the example code below, we are moving the value of the userMsg instruction
to the edx data register using the mov data transfer instruction. The int system
call instruction is used to call the kernel to an interrupt.

CODE
section .data
userMsg db ’Enter a number: ’, 0
userLen equ $-userMsg
userIn resb 5

section .text
global _start

_start:
; write user prompt
mov eax, 4
mov ebx, 1
mov ecx, userMsg
mov edx, userLen
int 0x80

; read user input

mov eax, 3
mov ebx, 0
mov ecx, userIn
mov edx, 5
int 0x80

; exit
mov eax, 1
xor ebx, ebx
int 0x80

9
4.3.1 Result

Figure 4: Return the user entered number

10
4.4 Program 4
In this program, we will display a sequence of “*********”. So here in the
below code, we are implementing double byte “db” in the data segment. (mov
ebx,1) is the file descriptor for stdout which will print output using sys write
system call by using the file descriptor of 4 (mov eax,4) and called the kernel
with interrupt function defined as 80h, which is equivalent to 0x80.

CODE
section .data
stars db ’*********’, 0xa
len equ $-stars

section .text
global _start

_start:
; write to stdout
mov eax, 4
mov ebx, 1
mov ecx, stars
mov edx, len
int 0x80

; exit
mov eax, 1
xor ebx, ebx
int 0x80

11
4.4.1 Result

Figure 5: Displaying 9 stars

12
4.5 Program 5
This program will display the output of printing a message “The sum is:” and
sum by accessing the value located at the address in the variable res. In the
below code, section.text is the actual code section, section.data is for ini-
tialized data or constants, section.bss is a static variable created, but a tem-
porary value is assigned to it and that value will be changed during runtime.

CODE
section .data
msg db ’The sum is: ’, 0
msgLen equ $-msg

section .bss
res resd 1

section .text
global _start

_start:
; calculate the sum
mov eax, 5
add eax, 7
mov [res], eax

; write message
mov eax, 4
mov ebx, 1
mov ecx, msg
mov edx, msgLen
int 0x80

; write the sum

mov eax, 4
mov ebx, 1
mov ecx, [res]
mov edx, 4
int 0x80

; exit
mov eax, 1
xor ebx, ebx
int 0x80

13
4.5.1 Result

Figure 6: MUL operation in x86

14
4.6 Program 6
MUL is used to perform multiplication in assembly. There are two instructions
for multiplying binary data. The MUL (Multiply) instruction handles unsigned
data and the IMUL (Integer Multiply) handles signed data. Both instructions
affect the Carry and Overflow flag.

15
CODE

Figure 7: MUL and IMUL in x86

16
4.6.1 Result

Figure 8: Program to perform MUL operation in x86 Assembly

17