International Journal of Control, Automation and Systems 16(6) (2018) 3013-3025 ISSN:1598-6446 eISSN:2005-4092

http://dx.doi.org/10.1007/s12555-018-0017-x http://www.springer.com/12555

Autonomous Vision-based Target Detection and Safe Landing for UAV

Mohammed Rabah*, Ali Rohan, Muhammad Talha, Kang-Hyun Nam, and Sung Ho Kim

Abstract: Target detection is crucial for many applications of Unmanned Aerial Vehicles (UAVs) such as search
and rescue, object transportation, object detection, inspection, and mapping. One of the considerable applications
of target detection is the safe landing of UAV to the drone station for battery charging and its maintenance. For
this, vision-based target detection methods are utilized. Generally, high-cost cameras and expensive CPU’s were
used for target detection. With the recent development of Raspberry Pi (RPi), it is possible to use the embedded
system with cheap price for such applications. In the current research, RPi based drone target detection and safe
landing system are proposed with the integration of PID controller for target detection, and Fuzzy Logic controller
for safe landing. The proposed system is equipped with a USB camera which is connected to RPi for detecting the
target and a laser rangefinder (LIDAR) for measuring the distance for safe landing. To verify the performance of
the developed system, a practical test bench based on a quadcopter and a target drone station is developed. Several
experiments were conducted under different scenarios. The result shows that the proposed system works well for
the target finding and safe landing of the quadcopter.

Keywords: Fuzzy logic, ground effect, quadcopter, safe landing, target detection.

1. INTRODUCTION Detecting the ground target and safe landing is the most
challenging part, as any mistake in any of them can lead
Quardcopter, also known as a quadrotor, is one type of the quadcopter to tip over, which will cause the destruction
UAV, which is lifted and propelled by four rotors [1, 2]. of the quadcopter or even harm any human in its range due
The quadcopter has high maneuverability, as it can hover, to the big size of the propellers and high rpm used in it.
take off, cruise and land in narrow areas. Quadcopters There have been a lot of research regarding target de-
have simpler control mechanism compared to the other tection and safe landing. In [9], a target detection algo-
UAVs [3]. rithm based on GPS and camera is proposed. Other re-
Recently, there have been increasing interests in the search presents an image processing based target detec-
UAVs applications such as surveillance, search and res- tion algorithm using high specification CPU’s [10]. In
cue, and object detection [4–8]. Especially, target detec- [11, 12], several ways to control the drone using cloud
tion is an important pre-function in the UAVs applications, computing-based vision is proposed. Also, there are num-
as it is required for a safe landing to drone station for bat- bers of works that focus on the communication between
tery charging or some other tasks. However, detecting a the drone and the ground target [13]. Some researchers
target, and landing is not an easy task due to the lack of were trying to detect and track shaped targets cite14,15,
sensitivity of sensors used for this application. For these while others tried to detect people and vehicle [16]. In
applications, image processing techniques are generally [17, 18], an image processing algorithm, that works using
used. color detection by a camera is proposed.
The autonomous quadcopter system can be divided into One of the most challenging parts of target detection is
four processes: the timing of the image processing. So, the best way is to
use a graphical processing unit (GPU) which is optimized
1) Taking off. for imaging algorithms.
2) Performing the task. Most of the prior research shows good performance.
3) Detection of the ground target. However, they are either using a normal navigator GPS
to detect the position of the target, which is not accurate,
4) Safe landing. as the used GPS accuracy is limited to 7-8 m and GPS

Manuscript received January 10, 2018; revised April 28, 2018; accepted June 5, 2018. Recommended by Associate Editor Son-cheol Yu
under the direction of Editor Jessie (Ju H.) Park.

Mohammed Rabah, Ali Rohan, and Muhammad Talha are with the School of Electronics and Information Engineering, Kunsan National
University, Korea (e-mails: [email protected], [email protected], [email protected]). Kang-Hyun
Nam and Sung Ho Kim are with Kunsan National University, Korea (e-mails: {khnam, shkim}@kunsan.ac.kr).
* Corresponding author.

⃝ICROS,
c KIEE and Springer 2018
3014 Mohammed Rabah, Ali Rohan, Muhammad Talha, Kang-Hyun Nam, and Sung Ho Kim

cannot work indoors, or they are using a higher specifica-

tion CPUs, which is very expensive and consumes more
power.
One problem with quadcopter safe landing is the ground
effect. Ground effect is a nonlinear effect generated near
the ground while landing. It causes the increase in thrust
of the rotors that will cause floating of the quadcopter
above the ground. This will make landing difficult and
lead to high power consumption.
The current research focuses on the problems of the pre-
vious research works. It focuses on replacing the high
specification and expensive CPUs with a much cheaper
one. It also focuses on overcoming the ground effect.
In order to overcome the cost problem, a vision-based
target detection algorithm based on PID controller using
RPi is proposed. A novel, Fuzzy logic based safe land-
Fig. 1. Block diagram of the overall system.
ing algorithm is developed to overcome the ground effect
which exists near to the ground while quadcopter is land-
ing. The proposed system is equipped with a USB cam- ARM Cortex M4 CPU with a clock frequency of 168
era connected to RPI for detecting the target and a laser MHz. It is equipped with a 10 DOF-IMU, to measure roll,
rangefinder (LIDAR) for measuring the distance for safe pitch, yaw, and altitude. It also has eight PWM outputs
landing. which can support up to eight BLDC motors. It also has
To verify the system performance, a practical test bench several connectivity options for additional peripherals like
based on a quadcopter and a target drone station was de- UART, I2C, CAN, SPI, and ADC, etc.
veloped. Several experiments were conducted under dif-
The reason for utilizing the Pixhawk flight controller is
ferent scenarios and results were obtained. The practical
because it can be interfaced easily with the ArduCopter,
configuration of the proposed system is discussed in de-
which is an open source code that is written in C++ and is
tail in Section 2. Section 3 presents the ground effect phe-
free for modification.
nomenon. The proposed target detection and safe landing
algorithm are explained in Section 4. Section 5 includes
2.2. Raspberry Pi (RPI)
the simulation results of the proposed safe landing algo-
rithm. Section 6 demonstrates the experimental results of The RPI is a low cost, small size computer with a wire-
the proposed system. Finally, the conclusion is drawn in less LAN and Bluetooth connectivity. It can be plugged
Section 7. into a computer monitor using an HDMI cable, and it uses
a standard mouse and keyboard. It is capable of doing ev-
erything that can be done with a desktop computer. It can
2. CONFIGURATION OF THE PROPOSED
also help to learn various programming languages such as
VISION-BASED TARGET DETECTION AND
Python and C++.
SAFE LANDING SYSTEM FOR
QUADCOPTER In the current work, RPi 3 is used, because it has higher
CPU performance than the previous versions. RPi 3 is
The proposed vision-based target detection and safe good for vision-based image processing which requires
landing system consist of the following components: fast execution time. Generally, the image processing algo-
flight controller, RPi, USB camera, ESC module, RC re- rithm for target detection based on PID controller and the
ceiver/transmitter, BLDC motor, a LIDAR sensor, and fuzzy logic algorithm for a safe landing is implemented
camera gimbal with quadcopter mainframe. A block di- inside of RPi. The USB camera is connected to RPi via
agram of the overall system is shown in Fig. 1. USB port which continuously sends the captured images
to RPi. Inside of RPi, two types of control algorithms,
2.1. Flight controller image processing for target detection and Fuzzy logic for
The flight controller is the brain of the quadcopter. It safe landing process the data received from the USB cam-
reads all data coming from the sensors and calculates the era and LIDAR. This data is sent to the Pixhawk flight
best commands and finally sends it to ESC module. ESC controller which generates the flight control commands.
module can receive the rpm data from the flight controller
and RC receiver to control the speed of each BLDC mo- 2.3. Camera and gimbal
tors. The flight controller used in this work is Pixhawk The camera used in this work is a 720p USB camera.
flight controller. The Pixhawk flight controller has an It’s a cheap camera which can be directly connected to
 Autonomous Vision-based Target Detection and Safe Landing for UAV 3015

Raspberry Pi. Furthermore, a 2D-gimbal is used for al-

lowing the camera to always face the ground while the
quadcopter is hovering around.

2.4. LIDAR-Lite rangefinder

LIDAR-Lite is a low power and lightweight 40 m
rangefinder. This rangefinder is used due to its charac-
teristics of low noise, high efficiency, and high range. It is
used to calculate the altitude of the quadcopter that is used
in the RPi for a safe landing.
Fig. 2. Quadcopter thrust effect under IGE.
3. GROUND EFFECT

In all types of UAVs, ground effect is the increased

force near the ground in comparison to high altitude.
UAV’s lifting force can be divided into two parts:
1) IGE (In Ground Effect),
2) OGE (Out of Ground Effect).
IGE is a condition where the downwash of air from the
main rotor can react with a hard surface (the ground) and Fig. 3. Quadcopter thrust effect under OGE.
give a useful reaction to the UAV in the form of more lift
force available with less power required. OGE is the op-
posite of IGE, where there are no hard surfaces for the
downwash to react against. For example, a UAV hovering
45m above the ground will be in an OGE condition and
will require more power to maintain a constant altitude
than if it was hovering at 4 m. Hence, a UAV will always
have a lower OGE ceiling than IGE due to the amount of
available power [19].
IGE and OGE effect in quadcopter UAV is shown in
Figs. 2 and 3. IGE effect is the most important issue to
be considered when designing an auto landing controller
mechanism. Therefore, the designed controller must be
intelligent enough to overcome its nonlinear effect during
the landing process.
Fig. 4. Color-based target detection and safe landing algo-
rithm.
4. AUTONOMOUS VISION-BASED TARGET
DETECTION AND SAFE LANDING
ALGORITHM
is applied to the output image. The thresholding works by
4.1. Vision-based target detection algorithm selecting a pixel value. If the pixel value is greater than
A color based image processing algorithm is proposed the threshold value which is fixed in the code, the output
in the current study and implemented by using Open CV will be white color, otherwise, it will be black color. Then,
library. USB camera is installed on the quadcopter and morphology transformers are used on the output image to
it keeps on taking snapshots continuously, which is pro- get rid of any noise in it. Finally, in order to calculate the
cessed by the color based target detection algorithm as centroid of the image, a 1st order spatial moments around
shown in Fig. 4. The color-based target detection and safe the x-axis, y-axis and the 0th order central moments of the
landing algorithm can be divided into three main parts: binary image is calculated. 0th order central moments of
1) Target detection part(based on image processing). the binary image are equal to the white area of the image
2) Safe landing part (based on Fuzzy logic). in pixels. The center of the detected white area can be
3) Stabilization control part. stated in pixels as in (1) and (2).
In target detection part, the captured image is converted
xtarget = m10/m00, (1)
from RGB (Red, Green, and Blue) format to HSV (Hue,
Saturation, and Value) format. Afterward, a thresholding ytarget = m01/m00, (2)
3016 Mohammed Rabah, Ali Rohan, Muhammad Talha, Kang-Hyun Nam, and Sung Ho Kim

(a) (b)

Fig. 6. (a) Flowchart of the position control algorithm. (b)

Flowchart of the velocity control algorithm.
Fig. 5. Vision-based PID controller for target detection.

where m10 is the 1st order spatial moments around x-axis,

m01 is the 1st order spatial moments around y-axis, and
m00 is the 0th order central moment.
After that, the centroid of the target is compared to the
centroid of the frame image. The deviation of the cen-
troids between them is used to calculate the required roll Fig. 7. Block diagram of the proposed fuzzy logic con-
and pitch angle for stabilization of quadcopter. troller.
In stabilization control part, the distance error is used
as an input to the PID controller. A PID controller is a
feedback control mechanism commonly used in industrial landing velocity is too fast, quadcopter might get crashed.
control systems [20]. In this work, PID controller which Otherwise, if the landing speed is too slow, quadcopter
is shown in Fig. 5 is utilized for generating the required can’t land at a ground target and just hovers above the
roll and pitch angle commands to make the quadcopter ground target. Whereas, in velocity control technique, the
hovering above the detected ground target. controller continuously generates the proper throttle value
As it is depicted in Fig. 5, image processing algorithm to land quadcopter at a constant speed. Similarly, almost
keeps on calculating the centroid of the detected target. same drawbacks are present in this method such as safety
These centroids are subtracted from the reference (0, 0), issues and slow landing. The drawbacks of the aforemen-
which is the centroid of the frame image. Finally, the out- tioned techniques are caused by the ground effect which
put of PID controller is given to the flight controller. exists near the ground while quadcopter is landing.
Simultaneously, LIDAR sensor keeps on measuring the Generally, some advanced auto landing algorithm
altitude of the quadcopter. This data is used in the fuzzy should be required to compensate for the ground effect.
logic controller to calculate the required throttle value for So, a fuzzy logic based auto landing algorithm, illustrated
a safe landing. Next section will cover this part in details. in Fig. 7, is proposed. The proposed fuzzy logic controller
has two inputs (altitude and speed) and one output (throt-
4.2. Fuzzy logic based safe landing algorithm tle value). Rule base in the fuzzy logic controller is very
When the drone detects the colored target on the ground important. In the current research, the above two control
at a certain altitude, safe landing algorithm is activated. In algorithms (position and velocity control) are considered
this work, we utilize Fuzzy logic based safe landing algo- together to build up the rule base.
rithm which can be thought of a mixture of two kinds of Generally, the fuzzy logic controller is made up of,
landing algorithms, a position and velocity control algo- fuzzification, rule base, the defuzzification process [24].
rithm [21–23]. The flowcharts of each control algorithms The fuzzy input, output membership functions, and a rule
are shown in Fig. 6. base are designed in MATLAB/ SIMULINK.
Position control technique utilizes the current position
information of the quadcopter. It continuously generates a 4.2.1 Fuzzification and designing the rule base
slightly smaller throttle value than the previous one until Fuzzy input-output membership functions and rule-
the quadcopter lands at a ground. This control technique base is designed in MATLAB/SIMULINK using the
is safe. However, the response of this algorithm is too Simulink fuzzy toolbox. The universe of each input and
slow and it does not consider velocity information. Fur- output is carefully selected according to desired ranges.
thermore, some safety issues are present because there is This fuzzy logic controller has two inputs and one out-
no way of considering landing speed. For example, if the put, 30 rules are made for optimum landing control. In-
 Autonomous Vision-based Target Detection and Safe Landing for UAV 3017

put and output membership functions are shown in Fig. 8.

The universe of discourse of each membership function
defines the operation range of that specific fuzzy linguis-
tic variables. These ranges are adjustable and act sim-
ilar to PID gains. These fuzzy linguistic variables are
defined as IGE_NG (IGE range Near Ground), IGE_FG
(IGE range Far Ground), OGE_Small, OGE_Medium, (a)
OGE_Big and OGE_VBig for the corresponding distance
input as in Fig. 8(a). Fuzzy linguistic variables for the sec-
ond input in vertical velocity is defined as NB (Negative
Big), NS (Negative Small), Normal, PS(Positive Small),
PB(Positive Big) as shown in Fig. 8(b). Velocity input
parameters are the most important parameter in this con-
troller which controls the landing velocity and direction
(b)
at the same time. The fuzzy set ‘Normal’ indicates the
normal landing velocity for safe landing. Whereas in
Fig. 8(c), NB, NM, NS and PS, PM, PB fuzzy linguis-
tic functions for landing velocity input. Negative veloc-
ity like NB and NS indicate high-speed landing velocities
and Positive velocity indicates the upward moment of the
quadcopter. Equation (3) explains landing velocity calcu-
lation formula. (c)
Altcurrent − Alt pre
Velland = , (3)
∆t Fig. 8. (a) Fuzzy input (distance). (b) Fuzzy input (veloc-
where Velland is the landing velocity, Altcurrent is the cur- ity). (c) Fuzzy output (throttle Adjustment).
rent altitude, Alt pre is the previous altitude, and ∆t is the
time difference between the current altitude and the previ-
ous altitude. These ranges for each parameter can be changed in
Distance range is taken from 0-250 cm, any value above fuzzy logic or gains can be used for more precise tuning.
250 cm will be taken as 250cm for fuzzy logic input. Making the rules-base is most crucial part of the fuzzy
IGE range is fixed to 100 cm for this quadcopter sys- logic controller, the whole function of this controller de-
tem. IGE range depends on quadcopter size and value of pends on the rule base. Table 1 shows the 30 rules de-
thrust produced by its propellers, the IGE range will in- signed to control the auto-landing operation. The ordinary
crease with high thrust producing quadcopters. The IGE fuzzy logic controller works similar to PID controller with
range is adjustable and it can be set according to the quad- fixed gains. Therefore, to implement the proposed landing
copter specification. In this designed landing controller, algorithm, the rule base is modified to overcome the PID
IGE is divided into two separate operating regions. First controller drawbacks. This rule-base works in two differ-
is IN_IGE_FG and second is IN_IGE_NG, IGE starts get- ent Modes, Mode A and Mode B.
ting effective from the boundary region of IN_IGE_FG Mode A: This is normal operation mode when quad-
and this effect keeps on increasing exponentially as alti- copter altitude is more than IGE range. Here, vertical
tude decreases. Two-step division of IGE expands the con- velocity and direction of the quadcopter are monitored
trol option to make the controller precise and reliable over and controlled to land with a constant velocity (−0.2 m/s)
different scenarios. Landing velocity ranges from −0.7 to within the normal range. In this case, throttle adjustment
0.5 m/s, the “±” sign indicates moving direction, negative values work between NS to PS unless until any uncertain
means moving downward and positive means moving up- change in velocity happens such as quadcopter is landing
ward. Throttle adjustments are throttle percentage values with very high velocity or quadcopter is going upward as
need to be added or subtracted from actual throttle per- shown in Table.1 (Mode A). NM and PM will be used to
centage in flight controller ranges from −0.5 to 0.5. change the landing velocity and direction in this case.
Equation (4) gives the relation between actual and ad- Mode B: The second scenario is when quadcopter is
justed throttle. within IGE range, throttle adjustments are then shifted to
NB and PB for a quick change in throttle values to keep the
Trottleinput = Trottle pre ± Trottlead j , (4)
landing velocity constant to −0.2 m/s as thrust gets more
where Trottleinput is the throttle value of flight controller, effective within IGE range as shown in Table 1 (Mode
Trottle pre is the previous throttle input, and Trottlead j is B). NS and PS don’t provide enough thrust changes to
the throttle adjustments values from landing controller. keep the landing velocity constant under IGE range. To
3018 Mohammed Rabah, Ali Rohan, Muhammad Talha, Kang-Hyun Nam, and Sung Ho Kim

Table 1. Fuzzy logic rule base for safe landing.

Altitude Mode B Mode A
Velocity IGE_NG IGE_FG OGE_Small OGE_Medium OGE_Big OGE_VBig
NB PB PB PM PM PM PM
NS PB PM PS PS PS PS
Normal Zero Zero Zero Zero Zero Zero
PS NB NM NS NS NS NS
PB NB NB NM NM NM NM

improve the controller performance to next level, landing

controller operation under IGE range is divided into two
steps. First is IGE_FG, quadcopter starts entering inside
IGE range in this region. Therefore, Throttle adjustment
commands are shifted towards higher gain values to pro-
vide enough throttle adjustments to overcome IGE. The
second region is IGE_NG, IGE gets more effective in this
region which delays the landing time and keeps the quad-
copter hovering for a long time before complete landing.
To overcome this problem, gains are sifted more towards
high values which make the throttle adjustment commands
to work between NB and PB values. PB and NB land-
ing velocity inputs will get invalid due to very low alti-
tude therefore only PB and NB throttle adjustment com-
mands will perform rest landing operation. The transition Fig. 9. Operating regions of proposed fuzzy logic landing
region between Mode A and Mode B needs to be smooth controller.
to avoid any kind of abrupt speed change. This transition
response depends on the overlap region between IGE_FG
and OGE_Small fuzzy linguistic variables. Fig. 9 explains
operation regions for Mode A, Mode B, and a transition
region between Mode A and Mode B.

4.2.2 Defuzzification
When the quadcopter is landing at a certain altitude,
corresponding rules are used to generate the practical
throttle adjustment value for a safe landing. There are
many defuzzification methods for this. In this work, we
utilize the Center of Gravity (CoG) method [25]. Throttle
adjustment values calculated from CoG method are added
or subtracted from throttle percentage value as in (4). Fi-
nally, the output of the fuzzy logic controller is given to
flight controller as shown in Fig. 6.

4.3. Proposed target detection and safe landing algo-

rithm
To get better performance, the aforementioned target de- (a) (b)
tection and safe landing algorithm are combined in the
proposed algorithm. The combined algorithm can be Fig. 10. (a) Flowchart of target detection and safe landing
thought as two algorithms working simultaneously with algorithm. (b) Flowchart of the received data pro-
each other, where the safe landing algorithm is the main cessing inside of the flight controller.
algorithm, and the target detection algorithm is used to de-
crease the distance between the quadcopter and the center
of the detected target as much as possible. scenarios according to the quadcopter altitude. In Sce-
For this, the proposed algorithm can be divided into two nario 1, since there is no ground effect on the quadcopter,
 Autonomous Vision-based Target Detection and Safe Landing for UAV 3019

PID controller for target detection keeps on calculating the the landing under IGE threshold. Also, in Fig. 12(b), it
roll and pitch values to make both centroids coincide with can be seen that fuzzy logic based landing controller keeps
each other. Simultaneously altitude information from the landing velocity constant at the desired value, and chang-
LIDAR sensor is used for safe landing algorithm. In Sce- ing the gain value will affect the settling time. In both
nario 2, to avoid any effect from the ground effect, both figures, the gain value of 1.05 shows the best response,
roll and pitch values from the PID controller are zero. and it shows that the Fuzzy logic controller that combines
From this moment on, only safe landing algorithm is acti- both speed and position control strategy covers the IGE
vated in this mode. efficiently according to the designed rule base. Moreover,
Fig. 10 shows the flowchart of the proposed target de- fuzzy logic landing controller provides steady state land-
tection and safe landing algorithm. The first flow chart ing velocity which makes the landing process smooth and
is for the target detection and safe landing. During this safe.
process, LIDAR sensor keeps on measuring the altitude
of the quadcopter and RPi calculates the required throttle
values for a safe landing. Simultaneously, target detection 6. EXPERIMENTS
algorithm works continuously until the target is detected
at the altitude of 1 m (Scenario 1). When the quadcopter
is under 1 m, the Fuzzy logic based safe landing algorithm To verify the performance of the proposed target detec-
is triggered. During this process, roll and pitch values are tion and safe landing algorithm, several experiments were
zero (Scenario 2). performed indoor and outdoor. The experimental system
The second flowchart shows the data packet processing is composed of quadcopter frame with flight controller and
inside of the flight controller. Whenever the flight con- RPi controller which executes target detection and safe
troller receives the data packet, it will check whether the landing. Even though the RPi is not powerful enough for
target is detected or not by checking the flag. If the target real-time target detection and safe landing, we tried to use
is detected, it will send commands to quadcopter to start the RPi owing to its low cost.
stabilizing the quadcopter and safe landing. Furthermore,
two emergency switches on the RC transmitter are pro-
vided to support the safe operation of the quadcopter. One 6.1. Target detection algorithm
is for the manual trigger of target detection, and the other To perform the target detection, USB camera with a res-
is for the manual trigger of a safe landing. olution of 160 × 120 is used for guaranteeing the proper
execution time that will not affect the performance of the
5. SIMULATION STUDIES whole quadcopter system. Fig. 13 shows a top view and a
side view of the quadcopter system.
MATLAB/SIMULINK is used to test the proposed safe
landing algorithm based on fuzzy logic to overcome the As it is shown in Fig. 13, a USB camera and a gimbal
ground effect. The used simulation system is available on are connected to the RPi. Gimbal is used to make the cam-
MATLAB file exchange and it is free to use. The purpose era facing the ground while the quadcopter is maneuver-
of this system is to study the behavior of a quadcopter. A ing. Generally, the quadcopter can come to the vicinity of
GUI is provided to show real moments of quadcopter in the ground target with the help of GPS information. When
3D space. The full system is shown in Fig. 11(a). the quadcopter is hovering in the vicinity of the target at
As mentioned before, the output of Fuzzy logic con- an altitude of about 2.5 m or higher, the camera starts to
troller can be multiplied by a gain value to tune the best re- keep on taking snapshots continuously. Each snapshot is
sponse from fuzzy logic. Equation (5) describes the fuzzy processed by image processing algorithm to calculate the
logic output calculation by COG technique. centroid of the detected target inside of the frame image.
Fig. 14 shows the sequential results of the vision-based
∑ni=1 µi µ (i) target detection algorithm.
Throttleadj = K( ) (5)
∑ni=1 µ (i)
As it is indicated in Fig. 14, the captured RGB image is
where Throttleadj is the Fuzzy logic output value, K is the converted into HSV image. After applying the threshold-
throttle adjustment gain, i is the input number, µi is the ing technique to the HSV image, an image can be obtained
corresponding membership value, µ is the fuzzy input, with lots of black spots inside of the detected target. In
and n is the total number of inputs. order to remove these spots, the morphological transfor-
Fig. 12(a) shows the Fuzzy logic safe landing controller mation is used. Finally, it is possible to calculate the cen-
response at different gain values, while Fig. 12(b) shows troid of the detected target. If the centroids of the detected
the landing velocity response at different gains. As shown target are available, this information is given to the PID
in Fig. 12(a), higher gain value improves the response controller to calculate the required roll and pitch angle for
time. However, when the gain is very high, it will disturb stabilizing the quadcopter.
3020 Mohammed Rabah, Ali Rohan, Muhammad Talha, Kang-Hyun Nam, and Sung Ho Kim

(a)

(b)

Fig. 11. (a) Quadcopter simulation system. (b) Attitude command block.

6.2. Generation of a lookup table for the fuzzy logic logic controller. Additionally, the Simulink model is uti-
controller lized as shown in Fig. 15 to pre-calculate the output of
Generally, the Fuzzy logic controller is a complicated con- Fuzzy logic controller corresponding to different input
troller that takes long computation time. In the current ap- values.
plication, RPI is responsible for performing two tasks at Landing velocity signal is made ranging from −0.7 to
the same time. Especially, image processing takes a large 0.5 m/s using signal builder block, and output throttle ad-
execution time due to its huge data size. Therefore, im- justment values are saved for 0-250 cm altitude, with a
plementing Fuzzy logic controller in parallel with image 5cm difference. The simulation program is executed for
processing will cause processing delay which can cause all landing velocity inputs at each altitude point and out-
catastrophe. To overcome this problem, a pre-calculated put of the Fuzzy logic controller is recorded. For example,
lookup table method is used instead of a real-time Fuzzy altitude input of quadcopter is fixed to 250 cm and land-
 Autonomous Vision-based Target Detection and Safe Landing for UAV 3021

(a)

(a)

(b)

Fig. 12. (a) Fuzzy logic landing controller gain tuning. (b)
Landing velocity response for fuzzy logic landing
controller at different gains.

ing velocity input is provided from −0.7 to 0.5 m/s with

the step size of 0.048 m/s, corresponding output throttle
adjustment values are recorded. The same process is re-
peated for each altitude point from 0-250cm with a 5cm
difference. Conclusive look-up table has been created
with 25 different velocity values aligned in rows with 52
different altitude values as columns which made look-up (b)
table of 1300 values. This look-up table is written inside
the RPi. The landing controller measures the altitude and
Fig. 13. (a) Side view of the quadcopter system. (b) Top
velocity using LIDAR and then round off the data to ex-
view of the quadcopter system.
actly match the look-up table inputs.
When the safe landing process is triggered, the real-
time altitude and velocity measurements can be calculated
using LIDAR sensor. These measured values are then the quadcopter detects the target, it starts the stabilization
rounded off to match the actual look-up table entries. The process to make the quadcopter hovering right above the
rounded off values of velocity and altitude are used as in- ground target. At this point, safe landing process is initi-
put for the look-up table, which generates the correspond- ated. From this point, safe landing and quadcopter stabi-
ing throttle adjustment output for these specific inputs. Fi- lization work simultaneously to overcome any destabiliza-
nally, this data is further sent to flight controller via UART tion due to environmental conditions. This process contin-
communication to update the attitude commands for a safe ues until quadcopter reaches 1m altitude. From now on,
landing. These instructions are updated every 200 ms in quadcopter stabilization is turned off and only safe land-
the flight controller. ing algorithm works for faster landing process. Altitude
Initially, the quadcopter is hovering at an altitude of is measured in cm unit via LIDAR rangefinder and data is
2.5 m and target detection algorithm is triggered. Once recorded every 500 ms using RF transceiver.
3022 Mohammed Rabah, Ali Rohan, Muhammad Talha, Kang-Hyun Nam, and Sung Ho Kim

ing algorithm every 100 ms.

When the quadcopter reaches an altitude of about 1 m
(Scenario 2), roll and pitch PID controller stops working.
From this moment on, safe landing algorithm is only ac-
tivated. The whole response of the proposed algorithm
is shown in Fig. 16. Fig. 16 represent the centroid de-
viation between detected target’s centroid and the frame
image’s centroid while testing the target detection algo-
rithm indoor and outdoor. As it can be seen in the figures,
PID1 shows better response and performance than PID2.
Table 4, shows the response time of the target detection al-
gorithm indoor and outdoor. As it can be seen in Table 4,
the outdoor test took more time because of the environ-
Fig. 14. Sequential results of image processing algorithm ment effect.
indoor. Fig. 17 represents the trajectory of the centroid of the
detected target with different PID gains indoor and out-
door. In Fig. 17, the Start point represents the centroids
of the detected target at an altitude of 2.5 m, and the End-
point represents the centroids of the detected target at an
altitude of 1 m. As can be seen in the figures, PID1 shows
a better response than that of PID2.
Fig. 18(a) and (b) show the response of the proposed
target detection and safe landing algorithm indoor and out-
door, while Fig. 19 shows the response of the auto landing
function that is implemented inside the pixhawk. Table 5
shows the response time of the proposed algorithm indoor,
Fig. 15. Simulink block for generation of lookup table. outdoor and the auto landing function response time of
pixhawk.
As shown in Table 5, when the drone is less than 1 m,
6.3. Experimental results the proposed safe landing algorithm shows a much bet-
To verify the feasibility of the proposed target detection ter response to overcome the ground effect and land more
and safe landing algorithm, several experiments are per- smoothly than the one that is implemented in the pixhawk.
formed. Initially, the quadcopter is hovering at an altitude Fig. 20 shows photos taken during testing the proposed
of above 2.5 meters (Scenario 1). During this period, tar- algorithm indoor and outdoor. Photo 1, shows the quad-
get detection and safe landing algorithms are simultane- copter hovering at an altitude of 2.5 meters, photo 2 shows
ously activated. For the target detection, a PID controller the stabilization and safe landing process. Photo 3, shows
is used. The roll and pitch PID controllers used the gains the quadcopter at an altitude of below 1 meter where only
listed in Tables 2 and 3. safe landing is working, and photo 4 shows the final land-
PID controller for target detection keeps on calculating ing to the ground target.
the roll and pitch values every 10 ms to make both cen-
troids coincide with each other. Simultaneously, altitude 7. CONCLUSION
information from the LIDAR sensor is used for safe land-
In this paper, a vision based target detection and safe
landing algorithm have been proposed. Target detection
Table 2. PID gains for pitch control. algorithm based on color is developed using PID con-
troller for quadcopter stabilization. Also, a safe landing
Y P I D algorithm based on the fuzzy logic controller has been de-
PID1 3.8 0.075 0.084 veloped to overcome the ground effect. Furthermore, the
PID2 4.0 0.05 0.07

Table 3. PID gains for roll control. Table 4. Target detection response time.
X P I D PID Indoor (sec) Outdoor (sec)
PID1 3.5 0.05 0.07 PID1 14.8 28.8
PID2 3.0 0.01 0.1 PID2 20.5 30.2
 Autonomous Vision-based Target Detection and Safe Landing for UAV 3023

(a)
(a)

(b)
(b)

Fig. 17. The trajectory of the centroid of the detected tar-

get (a) indoor, (b) outdoor.

(c)

(a)

(d)

Fig. 16. Response of proposed algorithm in (a) ‘x’ posi-

tion indoor, (b) ‘y’ position indoor, (c) ‘x’ posi-
tion outdoor, (d) ‘y’ position outdoor.
(b)

Table 5. The response time of the proposed algorithm. Fig. 18. Experimental results of fuzzy logic landing con-
troller (a) indoor, (b) outdoor.
Altitude Indoor (sec) Outdoor Pixhawk
(sec) (sec)
2.5 to 1 m 15 28 27
where the UAV can safely land after detecting target for
<1m 5 5.5 18
wireless charging then continue its task. In order to verify
Total time 20 33.5 45
the performance of the proposed algorithm, several exper-
iments are performed indoor and outdoor. The obtained
relatively cheap embedded controller was used for real- results show that the proposed system works well indoor
time applications such as target detection and safe landing. and outdoor for the target finding and safe landing of the
The proposed target detection and safe landing algorithm quadcopter. Furthermore, this system is currently being
can be used in many application such as mining detection, improved to safe land on a moving target.
3024 Mohammed Rabah, Ali Rohan, Muhammad Talha, Kang-Hyun Nam, and Sung Ho Kim

[11] J. Chudoba, M. Saska, T. Baca, and L. Preucil, “Local-

ization and stabilization of micro aerial vehicles based on
visual features tracking,” Proc. of the International Confer-
ence on Unmanned Aircraft Systems, pp. 611-616, 2014.
[12] H. An, J. Liu, C. Wang, and L. Wu, “Approximate back-
stepping fault-tolerant control of the flexible air-breathing
hypersonic vehicle,” IEEE/ASME Transactions on Mecha-
tronics, vol. 21, no. 3, pp. 1680-1691, 2016.
[13] E. E. Nyein, H. M. Tun, Z. M. Naing, and W. K. Moe,
Fig. 19. Experimental result of Pixhawk auto landing.
“Implementation of vision-based landing target detection
for VTOL UAV using raspberry Pi,” International Journal
of Scientific & Technology Research, vol. 4, no. 8, pp. 184-
REFERENCES 188, 2015.

[1] G. M. Hoffmann, D. G. Rajnarayan, S. L. Waslander, D. [14] T. Muskardin, G. Balmer, S. Wlach, K. Kondak, M. La-
Dostal, J. S. Jang, and C. J. Tomlin, “The Stanford testbed iacker, and A. Ollero, “Landing of a fixed-wing uav on
of autonomous rotorcraft for multi agent control (STAR- a mobile ground vehicle,” IEEE Int. Conf. Robot. Autom.
MAC),” Proc. of the 23rd Digital Avionics System Confer- (ICRA), pp. 1237-1242, May 2016.
ence, vol. 2, pp. 12-E, 2004. [15] J. Gleason, A. V. Nefian, X. Bouyssounousse, T. Fong, and
[2] ICAO, “Unmanned aircraft systems (UAS) circular: UAS G. Bebis, “Vehicle detection from aerial imagery,” Proc. of
328,” Cir 328 AN/190. IEEE International Conference on Robotics and Automa-
tion, pp. 2065-2070, 2011.
[3] J. Stafford, How a Quadcopter Works | Clay Allen, Univer-
sity of Alaska, Fairbanks, Retrieved 2015-01-20, Spring [16] A. Gaszczak, T. P. Breckon, and J. Han, “Real-time peo-
2014. ple and vehicle detection from UAV imagery,” Intelligent
Robots and Computer Vision XXVIII: Algorithms and Tech-
[4] M. Bhaskaranand and J. D. Gibson, “Low-complexity niques, vol. 7878, International Society for Optics and Pho-
video encoding for UAV reconnaissance and surveillance,” tonics, 2011.
Proc. of the IEEE Military Communications Conference,
pp. 1633-1638, 2011. [17] F. Arifin, R. Arifandi Daniel, and D. Widiyanto, “Au-
tonomous detection and tracking of an object au-
[5] P. Doherty and P. Rudol, “A UAV search and rescue sce- tonomously using AR. drone quadrotor,” Journal of Com-
nario with human body detection and geolocalization,” puter Science and Information, vol. 7, no. 1, pp. 11-17,
Australasian Joint Conference on Artificial Intelligence, 2014.
pp. 1-13, 2007.
[18] K. Boudjit and C. Larbes, “Detection and target tracking
[6] T. Tomic, K. Schmid, P. Lutz, A. Domel, M. Kassecker, with a quadrotor using fuzzy logic. In Modelling,” Proc.
E. Mair, I. L. Grixa, F. Ruess, M. Suppa, and D. Burschka, of the 8th International Conference of Identification and
“Toward a fullyautonomous UAV: research platform for in- Control, pp. 127-132, 2015.
door and outdoor urbansearch and rescue,” Robotics & Au-
tomation Magazine, IEEE, vol. 19, no. 3, pp. 46-56, 2012. [19] T. G. Wilson and P. H. Trickey, “D. C. Machine. With solid
state commutation,” Electrical Engineering, vol. 81, no.
[7] L. Merino, F. Caballero, J. R. Martinez-de Dios, J. Fer- 11, pp.879-884.
ruz, and A. Ollero, “A cooperative perception system for
[20] A. Radhakrishnan, “An experimental investigation of
multiple UAVs: application to automatic detection of for-
ground effect on a quad tilt-rotor in hover and low-velocity
est fires,” Journal of Field Robotics, vol. 23, no. 3.4, pp.
forward flight,” Ph.D. Thesis, University of Maryland, Col-
165-184, 2006.
lege Park, 2006.
[8] J. Liu, C. Wu, Z. Wang, and L. Wu, “Reliable filter design
[21] M. Barr. Introduction to Closed-Loop Control, Embedded
for sensor networks using type-2 fuzzy framework,” IEEE
Systems Programming, CRC Press, 2012.
Trans. Ind. Electron, vol. 13, no. 4, pp. 1742-1752, Aug.
2017. [22] S. Bouabdallah, “Design and control of quadrotors with ap-
plication to autonomous flying,” Ph.D Thesis, Ecole Poly-
[9] S. Lange, N. Sunderhauf, and P. Protzel, “A vision based
technique Federale de Lausanne, pp.129-136, 2007.
onboard approach for landing and position control of
an autonomous multirotor UAV in GPS-denied environ- [23] H. An, J. Liu, C. Wang, and L. Wu, “Disturbance observer-
ments,” Proc. of the International Conference on Advanced based antiwindup control for air-breathing hypersonic ve-
Robotics, pp. 1-6, 2009. hicles,” IEEE Transactions on Industrial Electronics, vol.
63, no. 5, pp. 3038-3049, 2016.
[10] T. Vladimir, D.-H. Kim, Y.-G. Ha, and D. Jeon, “Fast
multi-line detection and tracking with CUDA for vision- [24] B. Yingcai and D. Haibin, “Implementation of autonomous
based UAV autopilot,” Proc. of the 8th International Con- visual tracking and landing for a low-cost quadrotor,” Inter-
ference on Innovative Mobile and Internet Services in national Journal for Light and Electron Optics, vol. 124,
Ubiquitous Computing, pp. 96-101, 2014. no. 18, pp.3296-3300, September 2013.
 Autonomous Vision-based Target Detection and Safe Landing for UAV 3025

(a)

(b)
Fig. 20. Photos taken during practical test (a) indoor, (b) outdoor.

[25] L. X. Wang and J. M. Mendel, “Fuzzy basis functions, uni- Muhammad Talha received his B.S. de-
versal approximation, and orthogonal least-squares learn- gree in Electrical Engineering from The
ing,” IEEE Transaction on Neural Networks, vol. 3, no. 5, University of Faisalabad in 2012, an M.S.
pp. 807-814, 1992. degree in Control System Engineering
[26] Y. Bai and D. Wang, “Fundamentals of fuzzy logic con- from Kunsan National University in 2015,
trol - fuzzy sets, fuzzy rules and defuzzification,” Advanced and a Ph.D. degree in Electrical, Electron-
Fuzzy Logic Technologies in Industrial Applications, pp. ics and Control Engineering from Kunsan
17-36, 2006. National University in 2018. His research
interests include renewable energy sys-
tems, power converters, UAV’s, fuzzy logic.
Mohammed Rabah received his B.S de-
gree in Electronics and Telecommunica- Kang-Hyun Nam is a professor at Kunsan
tion Engineering from the AL-SAFWA National University working in collabora-
High Institute of Engineering, Cairo, tion with the Industry-University Coopera-
Egypt in 2015. He completed his MS in tion. His research interests include sensor
Electrical, Electronics and Control Engi- networks, IoT platform, big data, intelli-
neering from Kunsan National University, gent networking system, 5G slice service
Kunsan, Korea in Dec. 2017. Currently, logic.
pursuing his Ph.D. in Electrical, Electron-
ics and Control Engineering from Kunsan National University,
Korea. His research interests includes UAV’s, fuzzy logic sys-
tems and machine learning. Sung-Ho Kim received his B.S. degree in
Electrical Engineering from Korea Univer-
sity in 1984, an M.S. degree from Korea
Ali Rohan received his B.S. degree in University in 1986, and a Ph.D. degree
Electrical Engineering from The Univer- from Korea University in 1991. He com-
sity of Faisalabad, Pakistan in 2012. Cur- pleted POST-DOC from Hiroshima Uni-
rently, pursuing his MS & Ph.D. in Elec- versity (Japan). Current, he is a profes-
trical, Electronics and Control Engineer- sor at Kunsan National University. His re-
ing from Kunsan National University, Ko- search interests include wind turbine sys-
rea. His research interests includes re- tem, sensor networks, neural network and fuzzy logic, intelligent
newable energy system, power electronics, control systems.
fuzzy logic, neural network, EV system,
flywheel energy storage system.