• Overview
• Requirements
• Preparation
• Quick start
• Services
  • DetectObject: A ROS service which receives an image and returns a result of object detection.
    • Service definition
    • Example client
  • CaptureAndDetect: A ROS service which receives a topic name and returns a result of object detection for an image retrieved from the topic.
    • Service definition
    • Example client
• Scripts
  • detect_once.py: A one-time detection script which works without ROS.
  • detect.py: A script which provides both a DetectObject service and a CaptureAndDetect service.
  • capture_and_detect.py: A script which provides a CaptureAndDetect service.
  • monitor_and_detect.py A script which makes a topic for publishing a visualized result of detection of an image retrieved from an input topic.
• Tools

This package includes scripts for providing object detection services on ROS. In this package, the two ROS services, DetectObject and CaptureAndDetect are defined as abstract interfaces of object detection. They can work with any models defined in ChainerCV.

This package also includes scripts for preparing pre-trained models of YOLOv2, YOLOv3 and Faster R-CNN.

• Chainer

• ChainerCV

• OpenCV-Python

• ROS cvbridge package

• A trained model file and a label file.
  A trained model file is a NPZ file and a label file is a JSON file which includes a map between a class name string and an integer specifying the class.

  Scripts for providing ROS services in this package assumes that a file of a trained model has been saved by the function save_npz_with_structure() in chainer_npz_with_structure.py, which saves how to construct a model being saved as well as weights. Since the function encodes object construction and saves it into a file, a model object implemented by a newer class can be loaded from the file without changing the scripts in this package. (chainer.serializers.load_npz() requires to prepare a model object in a script before loading weights. )

  A model file generated by chainer.serializers.save_npz() can be converted for this package by the script prepare.py.

To use this package, a model file and a label file are required. Some types of pre-trained models and corresponding label files can be retrieved by the following procedures.

The script prepare_model_for_yolo_coco.sh can generate model files and label files for YOLOv2 and YOLOv3 which are trained with the COCO dataset (80 classes).

The script retrieves the pre-trained models, yolov2.weights and yolov3.weights from https://pjreddie.com/darknet/yolo/ and converts them so that they can be used with this package. To convert the pre-trained models, you need libdarknet.so (or libdarknet_ros_lib.so if you use the darknet_ros package).

An example command is following;

rosrun detect_object prepare_model_for_yolo_coco.sh libdarknet.so

The command generates coco.json, yolo-v2-coco.npz and yolo-v3-coco.npz in scripts/. coco.json is a label file for the COCO dataset with 80 classes. yolo-v2-coco.npz is a YOLOv2 model file converted from yolov2.weights. yolo-v3-coco.npz is a YOLOv3 model file converted from yolov3.weights.

The above command and the script prepare_model_for_yolo_coco.sh use rosrun to find the script, but they do not use functions of ROS essentially. So you can prepare models by manually executing the contents of the script.

The script prepare.py can generate a model file and a label file for Faster R-CNN based on VGG-16.

The script can retrieve the pre-trained model of the class chainercv.links.model.faster_rcnn.FasterRCNNVGG16 and converts them so that they can be used with this package.

An example command is following;

cd scripts/
python prepare.py --model faster_rcnn --output_model faster-rcnn-vgg16-voc07.npz --output_json faster-rcnn-vgg16-voc07.json

The command generates faster-rcnn-vgg16-voc07.json and faster-rcnn-vgg16-voc07.npz in scripts/. The former is a label file for the PASCAL VOC2007 dataset with 20 classes. The latter is a Faster R-CNN model trained with the dataset.

A trained model must be generated by the function save_npz_with_structure() in chainer_npz_with_structure.py, which saves how to construct a model being saved as well as weights.

If you have a model as an instance of chainer.Link, it can be used with this package by saving the model with the function save_npz_with_structure().

An example procedure is following. In this procedure, it is assumed that the catkin workspace is ~/catkin_ws.

1. cd ~/catkin_ws

2. Download this package.

   cd src/
   git clone https://github.com/cvmat/ros_detect_object
   

3. Build this package.

   cd ~/catkin_ws
   catkin_make
   

   This package includes only Python scripts which do not require compilation, but catkin_make is required for installing service definition files.

4. Prepare a model.

   roscd detect_object/scripts
   python prepare.py --model faster_rcnn --output_model faster-rcnn-vgg16-voc07.npz --output_json faster-rcnn-vgg16-voc07.json
   cd ~/catkin_ws
   

5. Start a node which provides a service.

   rosrun detect_object detect.py --label_file faster-rcnn-vgg16-voc07.json --model faster-rcnn-vgg16-voc07.npz --gpu 0
   

6. Call the service.

   rosrun detect_object detect_client.py target1.png target2.png target3.png
   

   If you have a ROS topic of an image sensor, it can be used as a source as follows;

   • Using rosservice:

     rosservice call /capture_and_detect '{topic: "/camera/image", capture_timeout: {secs: 1, nsecs: 0}}'
     

   • Using capture_and_detect_client.py:

     rosrun detect_object capture_and_detect_client.py --topic /camera/image --display
     

A ROS service which receives an image and returns a result of object detection.

detect.py provides this service.

srv/DetectObject.srv

• INPUT

  sensor_msgs/Image image
  

  image means an input image.

• OUTPUT

  sensor_msgs/RegionOfInterest[] regions
  int32[] labels
  float32[] scores
  string[] names
  

  This service returns information of detected objects. All of regions, labels, scores and names have the same number of elements. Each index corresponds to one of detected objects.

  • regions[i] means the rectangle surrounding the i-th detected object in the input image.
  • labels[i] means the integer label of the detected object.
  • scores[i] means the degree of confidence of detection.
  • names[i] means the name of the detected object.

  An integer label is uniquely bound to a name and their relation is defined in a label file given when invoking detect.py.

• detect_client.py
  This script calls a DetectObject service with a request for each image given as arguments and then display returned information.

  Example command:

  rosrun detect_object detect_client.py target1.png target2.png target3.png
  

  Usage:

  rosrun detect_object detect_client.py [-h]
    [--detection_service_name DETECTION_SERVICE_NAME]
    [--display] FILE [FILE ...]
  

  • --detection_service_name DETECTION_SERVICE_NAME: (default: detect_object ) Specify a name of a service that the script calls.
  • --display: If given, the script displays results graphically.
  • FILE [FILE ...]: input images.

• detect_client_save.py
  This script calls a DetectObject service with a request for an input image and writes a visualized result into a specified output image file.

  Example command:

  rosrun detect_object detect_client_save.py --input input.png --output output.png
  

  Usage:

  rosrun detect_object detect_client_save.py
    [--detection_service_name DETECTION_SERVICE_NAME]
    [--output OUTPUT_FILE] [--input INPUT_FILE]
  

  • --detection_service_name DETECTION_SERVICE_NAME: (default: detect_object ) Specify a name of a service that the script calls.
  • --input INPUT_FILE: Specify an input image.
  • --output OUTPUT_FILE: Specify a filename of an output image.

• detect_snapshot.py
  This script calls a DetectObject service with a request for an image retrieved from an input topic and writes a visualized result into a specified output image file.

  Example command:

  rosrun detect_object detect_snapshot.py --input /camera/image --output output.png
  

  Usage:

  rosrun detect_object detect_snapshot.py
    [--detection_service_name DETECTION_SERVICE_NAME]
    [--output OUTPUT_FILE] [--input INPUT_TOPIC]
  

  • --detection_service_name DETECTION_SERVICE_NAME: (default: detect_object) Specify a name of a service that the script calls.
  • --input INPUT_TOPIC: Specify an input topic.
  • --output OUTPUT_FILE: Specify a filename of an output image.

A ROS service which receives a topic name and returns a result of object detection for an image retrieved from the topic.

detect.py and capture_and_detect.py provide this service.

srv/CaptureAndDetect.srv

• INPUT

  string topic
  duration capture_timeout
  bool return_image_data
  

  topic means an input topic which an input image will be retrieved from.
  capture_timeout means a timeout. If the detection does not finish in the specified time, the request will fail and return a response.
  return_image_data specifies whether a response includes an input or not. If return_image_data is true and the request succeeds, the response includes the input image. If return_image_data is false or the request fails, the response does not include the input image.

• OUTPUT

  bool success
  string error_msg
  sensor_msgs/Image image
  sensor_msgs/RegionOfInterest[] regions
  int32[] labels
  float32[] scores
  string[] names
  

  This service returns information of detected objects. If the request fails, success will be false and error_msg will be a string describing the error. If return_image_data in a request is true and the request succeeds, the response includes the corresponding input image.

  All of regions, labels, scores and names have the same number of elements. Each index corresponds to one of detected objects.

  • regions[i] means the rectangle surrounding the i-th detected object in the input image.
  • labels[i] means the integer label of the detected object.
  • scores[i] means the degree of confidence of detection.
  • names[i] means the name of the detected object.

  An integer label is uniquely bound to a name and their relation is defined in a label file given when invoking detect.py or capture_and_detect.py .

• A CaptureAndDetect service can be called by the rosservice command-line tool as below.

  rosservice call /capture_and_detect '{topic: "/camera/image", capture_timeout: {secs: 1, nsecs: 0}}'
  

• capture_and_detect_client.py
  This script calls a CaptureAndDetect service and then display returned information.

  Example command:

  rosrun detect_object capture_and_detect_client.py --topic /camera/image --display
  

  Usage:

  rosrun detect_object capture_and_detect_client.py [-h]
    [--service_name SERVICE_NAME] [--topic INPUT_TOPIC]
    [--label_format LABEL_FORMAT] [--display]
  

  • --service_name SERVICE_NAME: (default: capture_and_detect )
    Specify a name of a service that the script calls.
  • --topic INPUT_TOPIC: Specify an input topic.
  • --label_format LABEL_FORMAT: Specify a format of a label text in a visualized result displayed if --display is given.
  • --display:
    If given, the script displays results graphically.

This is a script for detecting objects in given images with a single command. This script can be executed without ROS.

The below command will detect objects in input1.png, input2.png and input3.png with the model stored in faster-rcnn-vgg16-voc07.npz.

python detect_once.py --label_file faster-rcnn-vgg16-voc07.json --model faster-rcnn-vgg16-voc07.npz --gpu 0 input1.png input2.png input3.png

This script provides both a DetectObject service and a CaptureAndDetect service.

Example command:

rosrun detect_object detect.py --label_file faster-rcnn-vgg16-voc07.json --model faster-rcnn-vgg16-voc07.npz --gpu 0

Usage:

rosrun detect_object detect.py [-h]
  [--detection_service_name DETECTION_SERVICE_NAME]
  [--service_name_of_detect_object SERVICE_NAME_OF_DETECT_OBJECT]
  [--service_name_of_capture_and_detect SERVICE_NAME_OF_CAPTURE_AND_DETECT]
  [--gpu GPU] [--label_file LABEL_FILE] [--model MODEL]
  [--node_name NODE_NAME] [--tcpros_port TCPROS_PORT]
  [--xmlrpc_port XMLRPC_PORT]

• --detection_service_name DETECTION_SERVICE_NAME: Specify a name of a service that the script provides. This is equivalent to --service_name_of_detect_object.
• --service_name_of_detect_object SERVICE_NAME_OF_DETECT_OBJECT: (default: detect_object ) Specify a name of a service that the script provides for DetectObject. This is equivalent to --detection_service_name.
• --service_name_of_capture_and_detect SERVICE_NAME_OF_CAPTURE_AND_DETECT: (default: capture_and_detect ) Specify a name of a service that the script provides for CaptureAndDetect.
• --gpu GPU: (default: -1) Specify an index of a GPU used for computation. A negative number means that no GPUs will be used. The index is based on the CUDA library and it starts from 0.
• --label_file LABEL_FILE: Specify a label file.
• --model MODEL: Specify a model file.
• --node_name NODE_NAME: (default: detect_object_server) Specify a ROS node name.
• --tcpros_port TCPROS_PORT: (default: 0) Specify a TCP port which this node listens. If 0 is given, the port will be automatically determined (default behavior).
• --xmlrpc_port XMLRPC_PORT: (default: 0) Specify a XML-RPC port which this node listens. If 0 is given, the port will be automatically determined (default behavior).

This script provides a CaptureAndDetect service.

Example command:

rosrun detect_object capture_and_detect.py --label_file faster-rcnn-vgg16-voc07.json --model faster-rcnn-vgg16-voc07.npz --gpu 0

Usage:

rosrun detect_object capture_and_detect.py [-h]
  [--service_name SERVICE_NAME]
  [--gpu GPU] [--label_file LABEL_FILE] [--model MODEL]
  [--node_name NODE_NAME] [--tcpros_port TCPROS_PORT]
  [--xmlrpc_port XMLRPC_PORT]

• --service_name SERVICE_NAME: (default: capture_and_detect ) Specify a name of a service that the script provides for CaptureAndDetect.
• --gpu GPU: (default: -1) Specify an index of a GPU used for computation. A negative number means that no GPUs will be used. The index is based on the CUDA library and it starts from 0.
• --label_file LABEL_FILE: Specify a label file.
• --model MODEL: Specify a model file.
• --node_name NODE_NAME: (default: capture_and_detect_server) Specify a ROS node name.
• --tcpros_port TCPROS_PORT: (default: 0) Specify a TCP port which this node listnes. If 0 is given, the port will be automatically determined (default behavior).
• --xmlrpc_port XMLRPC_PORT: (default: 0) Specify a XML-RPC port which this node listnes. If 0 is given, the port will be automatically determined (default behavior).

This script publishes visualized result of detection of an image retrieved from a topic repeatedly.

This script subscribes a topic for retrieving an input image and calls the DetectObject service, and then publishes visualized result of detection.

Example command:

rosrun detect_object detect.py --label_file faster-rcnn-vgg16-voc07.json --model faster-rcnn-vgg16-voc07.npz --gpu 0
rosrun topic_tools throttle messages /camera/image 2
rosrun detect_object monitor_and_detect.py input:=/camera/image_throttle output:=/visualized_result
rosrun image_view image_view image:=/visualized_result

Usage:

rosrun detect_object monitor_and_detect.py [-h]
  [input:=INPUT_TOPIC] [output:=OUTPUT_TOPIC]

• input:=INPUT_TOPIC: Specify an input topic.
• output:=OUTPUT_TOPIC: Specify an output topic.

• convert_darknet_metadata_to_json.py
• convert_darknet_to_npz.py
• prepare.py
• prepare_model_for_yolo_coco.sh
• update_darknet_model_format.py