Skip to content
/ bpycv Public
forked from DIYer22/bpycv

Computer vision utils for Blender (generate instance annoatation, depth and 6D pose by one line code)

License

Notifications You must be signed in to change notification settings

3d-py/bpycv

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

bpycv: computer vision and deep learning utils for Blender

Contents: Features | Install | Demo | Tips

0
Figure.1 Render instance annoatation, RGB image and depth

▮ Features

  • Render annotations for semantic segmentation, instance segmentation and panoptic segmentation
  • Generate 6DoF pose ground truth
  • Render depth ground truth
  • Pre-defined domain randomization:
  • Very easy to install and run demo
  • Support docker: docker run -v /tmp:/tmp diyer22/bpycv (see Dockerfile)
  • A Python Codebase for building synthetic datasets
  • To Cityscapes format

News: We win 🥈2nd place in IROS 2020 Open Cloud Robot Table Organization Challenge (OCRTOC)

▮ Install

bpycv support Blender 2.8+, 2.9+

# Get pip: equl to /blender-path/2.xx/python/bin/python3.7m -m ensurepip
blender -b --python-expr "__import__('ensurepip')._bootstrap()" 
# Update pip toolchain
blender -b --python-expr "__import__('pip._internal')._internal.main(['install', '-U', 'pip', 'setuptools', 'wheel'])"
# pip install bpycv
blender -b --python-expr "__import__('pip._internal')._internal.main(['install', '-U', 'bpycv'])"
# Check bpycv ready
blender -b -E CYCLES --python-expr 'import bpycv,boxx;boxx.tree(bpycv.render_data())'

▮ Demo

1. Fast Instance Segmentation and Depth Demo

Copy-paste this code to Scripting/Text Editor and click Run Script button(or Alt+P)

import cv2
import bpy
import bpycv
import random
import numpy as np

# remove all MESH objects
[bpy.data.objects.remove(obj) for obj in bpy.data.objects if obj.type == "MESH"]

for index in range(1, 20):
    # create cube and sphere as instance at random location
    location = [random.uniform(-2, 2) for _ in range(3)]
    if index % 2:
        bpy.ops.mesh.primitive_cube_add(size=0.5, location=location)
        categories_id = 1
    else:
        bpy.ops.mesh.primitive_uv_sphere_add(radius=0.5, location=location)
        categories_id = 2
    obj = bpy.context.active_object
    # set each instance a unique inst_id, which is used to generate instance annotation.
    obj["inst_id"] = categories_id * 1000 + index

# render image, instance annoatation and depth in one line code
# result["ycb_meta"] is 6d pose GT
result = bpycv.render_data()

# save result
cv2.imwrite(
    "demo-rgb.jpg", result["image"][..., ::-1]
)  # transfer RGB image to opencv's BGR

# save instance map as 16 bit png
# the value of each pixel represents the inst_id of the object to which the pixel belongs
cv2.imwrite("demo-inst.png", np.uint16(result["inst"]))

# convert depth units from meters to millimeters
depth_in_mm = result["depth"] * 1000
cv2.imwrite("demo-depth.png", np.uint16(depth_in_mm))  # save as 16bit png

# visualization inst_rgb_depth for human
cv2.imwrite("demo-vis(inst_rgb_depth).jpg", result.vis()[..., ::-1])

Open ./demo-vis(inst_rgb_depth).jpg:

demo-vis(inst_rgb_depth)

2. YCB Demo

mkdir ycb_demo
cd ycb_demo/

# prepare code and example data
git clone https://github.com/DIYer22/bpycv
git clone https://github.com/DIYer22/bpycv_example_data

cd bpycv/example/

blender -b -P ycb_demo.py

cd dataset/vis/
ls .  # visualize result here
# 0.jpg

Open visualize result ycb_demo/bpycv/example/dataset/vis/0.jpg:
0
instance_map | RGB | depth

example/ycb_demo.py Inculding:

  • Domain randomization for background, light and distractor (from ShapeNet)
  • Codebase for building synthetic datasets base on YCB dataset

3. 6DoF Pose Demo

Generate and visualize 6DoF pose GT: example/6d_pose_demo.py

▮ Tips

Blender may can't direct load .obj or .dea file from YCB and ShapeNet dataset.
It's better to transefer and format using meshlabserver by run meshlabserver -i raw.obj -o for_blender.obj -m wt



suggestion and pull request are welcome 😊

About

Computer vision utils for Blender (generate instance annoatation, depth and 6D pose by one line code)

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 99.0%
  • Other 1.0%