GPU Programming

Course Code: CSGG3018

Instructor: AMIT GURUNG
Email: [email protected]

Welcomes

12 programmes accredited Ranked 501-600 in Ranked 801-850 in Ranked 46th in

Rankings 2025 World Rankings 2025 Rankings 2024*
A Grade
* University Category

Jan – June, 2025

1
CUDA
 Overview
1.GPU Architecture
2.Data Parallelism
3.CUDA Program Structure
4.Vector Addition
5.Device Global Memory and Data Transfer
6.Kernel Functions and Threading
GPU Architecture
 GPU Architecture

Some general terminology:

A device refers to a GPU in the computer.
A host refers to a CPU in the computer.
 GPUs vs CPUs .
.
.

Control A A A A A A
L L L L L L …
ALU ALU Cache U U U U U U

Control A A A A A A
Control
L L L L L L …
ALU ALU Cache U U U U U U

Control A A A A A A
L L L L L L …
Cache U U U U U U
Cache
Control A A A A A A
L L L L L L …
Cache U U U U U U

DRAM DRAM

CPUs are designed to minimize the execution time of single

threads.
GPUs are designed to maximize the throughput of many
identical threads working on different memory.
 CUDA Capable GPU Architecture
Host In a modern CUDA capable GPU, multiple
streaming units (SMs) contain multiple
Input Assembler streaming processors (SPs) that share the
same control logic and instruction cache.
Thread Execution Manager

SP SP SP SP SP SP SP SP SP SP SP SP SP SP SP SP

SP SP SP SP SP SP SP SP SP SP SP SP SP SP SP SP
SMs …
SP SP SP SP SP SP SP SP SP SP SP SP SP SP SP SP

Cache Cache Cache Cache Cache Cache Cache Cache

Texture Texture Texture Texture Texture Texture Texture Texture

Load/Store Load/Store Load/Store Load/Store Load/Store Load/Store Load/Store

Global Memory
 CUDA Capable GPU Architecture
Host

Input Assembler

Thread Execution Manager

Host:

In CUDA terminology, the host refers to the central processing unit
(CPU) and its associated memory.

The host is responsible for managing the overall execution of a
CUDA program, including:
a) Memory Management: Allocating and transferring data between
the host (CPU) memory and the device (GPU) memory.
b) Kernel Launching: Initiating functions (kernels) that execute on
the GPU.
c) Synchronization: Coordinating the execution flow between the
CPU and GPU to ensure correct program behavior.

The host prepares data, transfers it to the device, launches kernels
on the GPU, and retrieves the results after computation.
 CUDA Capable GPU Architecture
Host

Input Assembler

Thread Execution Manager

Input Assembler:

The input assembler is a component of the GPU's
graphics pipeline (see Reference), primarily involved in
the initial stages of rendering graphics.

Its main functions include:

Vertex Fetching: Retrieving vertex data from memory.

Primitive Assembly: Organizing vertices into geometric
primitives such as points, lines, and triangles.

In the context of general-purpose computing with CUDA,
the input assembler is not directly utilized, as CUDA
focuses on computation rather than graphics rendering.
 CUDA Capable GPU Architecture
Host

Input Assembler

Thread Execution Manager

Thread Execution Manager:


The thread execution manager is a crucial component of
the GPU architecture that oversees the scheduling and
execution of threads on the GPU.

Its responsibilities include:

Thread Scheduling: Managing the concurrent execution of
thousands of threads across multiple streaming multiprocessors
(SMs).

Resource Allocation: Distributing computational resources such
as registers and shared memory among active threads.

Latency Hiding: Switching between threads to mask memory
access latencies, ensuring efficient utilization of the GPU's
computational units.
 CUDA Capable GPU Architecture
CUDA programs define kernels which are
Host loaded into the instruction caches on each SM.
The kernel allows for hundreds or thousands
of threads to perform the same instructions
Input Assembler
over different data.

Thread Execution Manager

SP SP SP SP SP SP SP SP SP SP SP SP SP SP SP SP

SP SP SP SP SP SP SP SP SP SP SP SP SP SP SP SP
SMs
…
SP SP SP SP SP SP SP SP SP SP SP SP SP SP SP SP

Cache Cache Cache Cache Cache Cache Cache Cache

Texture Texture Texture Texture Texture Texture Texture Texture

Load/Store Load/Store Load/Store Load/Store Load/Store Load/Store Load/Store

Global Memory
Data Parallelism
 Task Parallelism vs Data
Parallelism
*Vector processing units and GPUs take
care of data parallelism, where the same
operation or set of operations need to
be performed over a large set of data.
*Most parallel programs utilize data
parallelism to achieve performance
improvements and scalability.
 Task Parallelism vs Data
Parallelism

Task parallelism, however, is when a large

set of independent (or mostly independent)
tasks need to be performed.
The tasks are generally much more complex
than the operations performed by data
parallel programs.
 Task Parallelism vs Data
Parallelism

It is worth mentioning that both can be combined

— the tasks can contain data parallel elements.

Further, in many ways systems are getting closer to

do both. GPU hardware and programming
languages like CUDA are allowing more
operations to be performed in parallel for the
data parallel tasks.
Data Processing Example

For example, vector addition is a data parallel:

Vector A A[0] A[1] A[2] A[3] … A[N]

Vector B B[0] B[1] B[2] B[3]

… B[N]

+ + + + +

Vector C C[0] C[1] C[2] C[3]

… C[N]
CUDA Program
Structure
 CUDA Program Structure
CUDA programs are C/C++ programs containing code that uses CUDA
extensions (.cu).

The CUDA compiler (nvcc, nvcxx) is essentially a wrapper around another C/C++
compiler (gcc, g++, llvm). The CUDA compiler compiles the parts for the GPU
and the regular compiler compiles for the CPU:

Integrated C programs with CUDA extensions

NVCC Compiler
Host Code Device Code
Host C preprocessor,
Device just-in-time Compiler
compiler, linker

Heterogeneous Computing Platform With CPUs and GPUs

 CUDA Program Structure
When CUDA programs run, they alternate between CPU serial code
and GPU parallel kernels (which execute many threads in parallel).

The program will block waiting for the CUDA kernel to complete
executing all its parallel threads.

CPU serial code

GPU parallel kernel Block of Block of Block of Block of Block of Block of

KernelA<<<nBlocks, nThreads>>>(args); Threads Threads Threads Threads Threads Threads

CPU serial code

GPU parallel kernel Block of Block of Block of Block of Block of Block of

KernelA<<<nBlocks, nThreads>>>(args); Threads Threads Threads Threads Threads Threads
Vector Addition
 Vector Addition in C
//Compute vector sum h_C = h_A + h_B
void vecAdd(float* h_A, float* h_B, float* h_C, int n) {
for (int i = 0; i < n; i++) {
h_C[i] = h_A[i] + h_B[i];
}
}

int main() {
float* h_A = new float[100];
float* h_B = new float[100];
float* h_C = new float[100];
…
//assign elements into h_A, h_b
…
vecAdd(h_A, h_B, h_C, 100);
}

The above is a simple C program which performs vector

addition. The arrays h_A and h_b are added with the
results being stored into h_C.
 Skeleton Vector Add in
CUDA
It’s important to note that GPUs and CPUs do not
share the same memory, so in a CUDA program
you have to move the memory back and forth.

Host (CPU) Device (GPU)

Host Memory (RAM) Global Memory
 Skeleton Vector Add in
CUDA
//We want to parallelize this!
//Compute vector sum h_C = h_A + h_B
void vecAdd(float* h_A, float* h_B, float* h_C, int n) {
for (int i = 0; i < n; i++) {
h_C[i] = h_A[i] + h_B[i];
}
}

int main() {
float* h_A = new float[100];
float* h_B = new float[100];
float* h_C = new float[100];
…
//assign elements into h_A, h_b

//allocate enough memory for h_A, h_B, h_C on the GPU

//copy the memory of h_A and h_B onto the GPU
vecAddCUDA(h_A, h_B, h_C, 100); //perform this on the GPU!
//copy the memory of h_C on the GPU back into h_C on the CPU
}
Device Global
Memory and
Data Transfer
 Memory Operations in
CUDA
CUDA provides three methods for allocating on and moving
memory to GPUs.

cudaMalloc(); //allocates memory on the GPU (like malloc)

cudaFree(); //frees memory on the GPU (like free)

cudaMemcpy(); //copies memory from the host to the

//device (like memcpy)
 Memory Operations in
CUDA
cudaMalloc(); //allocates memory on the GPU (like malloc)
cudaFree(); //frees memory on the GPU (like free)

cudaMemcpy(); //copies memory from the host to the

//device (like memcpy)

All three methods can return a cudaError_t type, which

can be used to test for error conditions, eg:
cudaError_t err = cudaMalloc((void**) &d_A, size);

(void**)&d_A is the address of the pointer that will point to

the allocated device memory.

size is the total size in bytes to allocate.

 Memory Operations in
CUDA

Syntax of cudaMemcpy:
cudaError_t cudaMemcpy(void* dst, const void* src, size_t count,
cudaMemcpyKind kind);

dst: Destination pointer (d_A in device memory).

src: Source pointer (h_A in host memory).

count: Number of bytes to copy (size).

kind: Direction of copy (Eg., cudaMemcpyHostToDevice indicates

host to device transfer).
 Memory Operations in
CUDA

All three methods can return a cudaError_t type, which

can be used to test for error conditions, eg:
cudaError_t err = cudaMalloc((void**) &d_A, size);

if (err != cudaSuccess) {
printf(“%s in file %s at line %d\n”,
cudaGetErrorString(err),
__FILE__, __LINE__);
exit(EXIT_FAILURE);
}
 Skeleton Vector Add in
int main() { CUDA
int size = 100;
float* h_A = new float[size];
float* h_B = new float[size];
float* h_C = new float[size];
//assign elements into h_A, h_b
…

//allocate enough memory for h_A, h_B, h_C as d_A, d_B, d_C
//on the GPU
float *d_A, *d_B, *d_C;
cudaMalloc((void**) &d_A, size);
cudaMalloc((void**) &d_B, size);
cudaMalloc((void**) &d_C, size);
//copy the memory of h_A and h_B onto the GPU
cudaMemcpy(d_A, h_A, size, cudaMemcpyHostToDevice);
cudaMemcpy(d_B, h_B, size, cudaMemcpyHostToDevice);

vecAddCUDA(d_A, d_B, d_C); //perform this on the GPU!

//copy the memory of d_C on the GPU back into h_C on the CPU
cudaMemcpy(h_C, d_C, size, cudaMemcpyDeviceToHost);

//free the memory on the device

cudaFree(d_A); cudaFree(d_B); cudaFree(d_C);
}
Kernel Functions
and Threading
 CUDA Grids
CUDA assigns groups of threads to blocks, and groups of blocks
to grids. For now, we’ll just worry about blocks of threads.

Using the blockIdx, blockDim, and threadIdx keywords, you

can determine which thread is being run in the kernel, from
which block.

For our GPU vector add, this will work something like this:

Block 0 Block 1 Block N-1

Thread 0 1 2 … 256 Thread 0 1 2 …
256 Thread 0 1 2 …
256

i = (blockIdx.x * i = (blockIdx.x * blockDim.x) … i = (blockIdx.x *

blockDim.x) + threadIdx.x; + threadIdx.x; blockDim.x) + threadIdx.x;

d_C = d_A[i] + d_B[i]; d_C = d_A[i] + d_B[i]; d_C = d_A[i] + d_B[i];

… … …
 Vector Add Kernel

//Computes the vector sum on the GPU.

__global__ void vecAddKernel(float* A, float* B, float* C, int n) {
int i = threadIdx.x + (blockDim.x * blockIdx.x);
if (i < n) C[i] = A[i] + B[i];
}

int main() {
…
//this will create ceil(size/256.0) blocks, each with 256 threads
//that will each run the vecAddKernel.
vecAddKernel<<<ceil(size/256.0), 256>>>(d_A, d_B, d_C, size);
…
}
 Vector Add Kernel

BlockIdx: this variable is a constant for all blocks and stores the number of
threads along each dimension of the block. For one-dimentional block, we
refer only to blockDim.x.

BlockDim: The dimension (total number of threads) of the thread block is

accessible within the kernel through this variable

ThreadIdx: The index of the current thread within its block.

 CUDA keywords
Function Qualifiers Executed Only callable
on the: from the:
__device__ float deviceFunc() device device

__global__ void KernelFunc() device host

__host__ float HostFunc() host host

By default, all functions are __host__ functions, so you

typically do not see this frequently in CUDA code.

__device__ functions are for use within kernel functions, they

can only be called and used on the GPU.

__global__ functions are accessible from the CPU, however they

are executed on the GPU. This function can not return any value
and so must be void.
 Code for Vector Addition in GPU
#include <iostream>
#include <cuda_runtime.h>
// CUDA kernel for vector addition
__global__ void vecAddKernel(float* d_A, float* d_B, float* d_C, int n) {
int i = blockIdx.x * blockDim.x + threadIdx.x;
if (i < n) {
// Define block and grid sizes
d_C[i] = d_A[i] + d_B[i];
int blockSize = 256;
}
} int gridSize = (n + blockSize - 1) / blockSize;

int main() { // Launch the vector addition kernel

int n = 4; vecAddKernel<<<gridSize, blockSize>>>(d_A, d_B, d_C, n);
size_t size = n * sizeof(float);
// Allocate host memory // Copy result from device to host
float* h_A = new float[n]; cudaMemcpy(h_C, d_C, size, cudaMemcpyDeviceToHost);
float* h_B = new float[n];
float* h_C = new float[n]; // Display the result
for (int i = 0; i < n; ++i) {
// Initialize host arrays
std::cout << "h_C[" << i << "] = " << h_C[i] << std::endl;
for (int i = 0; i < n; ++i) {
}
h_A[i] = 5.0f;
h_B[i] = 5.0f;
// Free device memory
}
cudaFree(d_A);
// Allocate device memory cudaFree(d_B);
float *d_A, *d_B, *d_C; cudaFree(d_C);
cudaMalloc((void**)&d_A, size);
cudaMalloc((void**)&d_B, size); // Free host memory
cudaMalloc((void**)&d_C, size); delete[] h_A;
// Copy data from host to device delete[] h_B;
cudaMemcpy(d_A, h_A, size, cudaMemcpyHostToDevice); delete[] h_C;
cudaMemcpy(d_B, h_B, size, cudaMemcpyHostToDevice);
return 0;
 A few simple exercises

1) Write a CUDA program that takes an array of integer and negates each
element.

2) Write a CUDA program to initialize an array with a constant value. Pass the
value from the CPU code.

3) Write a CUDA program that compute element-wise squaring of a given

array.
References

1 . GPU Programming, IBM Publication.

2 .https://learn.microsoft.com/en-us/windows/win32/direct3d11/
overviews-direct3d-11-graphics-pipeline.

Watch List:
https://www.youtube.com/watch?v=usY0643pYs8

https://www.youtube.com/watch?v=cRY5utouJzQ&t=60s

https://www.youtube.com/watch?v=OJuA3DZNfz8

https://www.youtube.com/watch?v=uTyYNPU4mGQ