Skip to content

Commit f0e8e2e

Browse files
agrechnevagrechnev
committed
2_3, readme
1 parent eb87207 commit f0e8e2e

File tree

3 files changed

+253
-4
lines changed

3 files changed

+253
-4
lines changed

‎README.md

Lines changed: 92 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,99 @@
1-
First steps with Google Mediapipe: An Informal tutorial
1+
First steps with Google MediaPipe: An Informal tutorial
22
========
33

44
By Oleksiy Grechnyev, IT-JIM (https://www.it-jim.com/)
55

6-
Development in progress, more detailed readme is coming soon ...
6+
Development in progress, more is coming soon hopefully ...
77

8-
1. Install bazel and mediapipe, check that 'C++ hello world works'
9-
2. Copy `first_steps` to `mediapipe/mediapipe/examples`, run just like google examples
8+
What is this?
9+
-----------
10+
11+
This unofficial tutorial gives a gentle introduction into the Google MediaPipe C++ API.
12+
13+
Google MediaPipe (MP) is fun, but in my opinion the official docs are insufficient,
14+
especially if you are interested in the core C++ API. But the C++ API is what MP is
15+
really about, the "Solutions" and language bindings are just thin wrappers.
16+
17+
What you will learn:
18+
* How to write MP pipelines in C++, send and receive data packets
19+
* How to write your own MP calculators
20+
* How to use standard MP calculators (only a few examples)
21+
22+
What you will NOT learn from this tutorial:
23+
* MediaPipe solutions
24+
* Languages and platforms other than C++/desktop
25+
* Audio processing
26+
* GPU usage
27+
* Neural network inference in TF/TF Lite (but maybe I'll come back to that later)
28+
* GLog, Google Test, and other Google goodies not crucial for understanding MediaPipe
29+
30+
31+
Prerequisites:
32+
* Fluent C++ knowledge
33+
* Minimal knowledge of Bazel, OpenCV, Protobuf
34+
* Read official MP docs (Sections "MediaPipe in C++" and "Framework Concepts")
35+
* Install MP and try the MP C++ "Hello world" example
1036

1137
Enjoy!
38+
39+
Installation
40+
------------
41+
42+
This has been tested on Kubuntu 20.10, but hopefully it works on other OSes as well.
43+
44+
1. Install Bazel and MediaPipe as per official MP docs.
45+
2. Check that hello world works. In the MP repo root directory, type:
46+
`export GLOG_logtostderr=1`
47+
`bazel run --define MEDIAPIPE_DISABLE_GPU=1 //mediapipe/examples/desktop/hello_world`
48+
Important: On modern Linux you will see weird errors if you don't have `python` in your path, `python3` will not do!
49+
This is fixed by `sudo apt install python-is-python2`.
50+
If the standard hellow world does not work for you, I cannot help you further.
51+
3. Copy the directory `first_steps` from this repo to `mediapipe/examples` in the MP repo (in will sit next to `android`, `coral`, `desktop` and `ios`).
52+
53+
To run an example `1_1` (for instance) type:
54+
`bazel run --define MEDIAPIPE_DISABLE_GPU=1 //mediapipe/examples/first_steps/1_1`
55+
56+
To run an example with additional files needed at runtime (curently it's only 1_3) type:
57+
`bazel build --define MEDIAPIPE_DISABLE_GPU=1 //mediapipe/examples/first_steps/1_3`
58+
`bazel-bin/mediapipe/examples/first_steps/1_3/1_3`
59+
60+
Examples
61+
---------
62+
63+
Each example introduces some new mediapipe concept. They have detailed comments. Please follow them in order, as
64+
example `1_2` assumes you are already familiar with `1_1`. Currently the following examples are available:
65+
66+
1.1: Simplest mediapipe pipeline
67+
1.2: Our first custom calculator
68+
1.3: Joining strings, synchronization, source
69+
1.4: Calculator options
70+
1.5: Calculator with side packets
71+
72+
2.1: Video pipeline
73+
2.2: Video pipeline with ImageCroppingCalculator and ScaleImageCalculator
74+
2.3: Video pipeline with ImageCroppingCalculator (dynamic crop)
75+
76+
Why Bazel?
77+
--------
78+
79+
You might have noticed that all this is very different from what usual C++ programming look like.
80+
Where is "sudo apt install libmediapipe-dev" (sarcasm) ? Where is CMake?
81+
82+
They say Google technologies do not work outside of Google, and having written this tutorial I totally agree.
83+
84+
There is currently no way to use MP without Bazel (at least for beginners). In particular, you canot
85+
create an "MP release" as a .SO library + headers. Using MP as an external
86+
Bazel dependency is also *very* tricky (although you can find examples on the net).
87+
88+
This, of course, makes using MediaPipe *very*, *very* inconvenient. For all practical purposes (again, at least for beginners),
89+
the only way to use MP in your own software project, is to build your project *inside the MediaPipe tree*, just
90+
as we did with our `first_steps`. Naturally, this means that your entire project must be a Bazel project,
91+
and you will have to look for any other (system-wide) external libraries (e.g. Boost or Qt) on your own (hardcoded paths or
92+
custom shell scripts), as the usual tools like CMake find_package or pkgconfig are not available in Bazel.
93+
94+
Finally, is there any IDE for Bazel C++ projects? CLion plugin is advertized, but it does not work with modern CLion versions.
95+
VS code also has Bazel plugins, which did not work for me either. However, if I open the entire MP tree as a single
96+
VS Code project, VS Code can (mostly) find MP headers, and thus the "Show symbol definition" function works.
97+
To me this is the main reason to use IDE (as opposed to a text editor). So I edited the code in VS Code, and executed it from the terminal.
98+
99+
I wonder what IDE do they use in Google?

‎first_steps/2_3/BUILD

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
cc_binary(
2+
name="2_3",
3+
srcs=["main.cpp"],
4+
deps=[
5+
"//mediapipe/framework:calculator_framework",
6+
"//mediapipe/calculators/core:pass_through_calculator",
7+
"//mediapipe/calculators/image:image_cropping_calculator",
8+
"//mediapipe/calculators/image:scale_image_calculator",
9+
"//mediapipe/framework/formats:image_frame",
10+
"//mediapipe/framework/formats:image_frame_opencv",
11+
"//mediapipe/framework/port:opencv_highgui",
12+
"//mediapipe/framework/port:opencv_imgproc",
13+
"//mediapipe/framework/port:parse_text_proto",
14+
"//mediapipe/framework/port:status",
15+
],
16+
)

‎first_steps/2_3/main.cpp

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
/// Example 2.3 : Video pipeline with ImageCroppingCalculator (dynamic crop)
2+
/// By Oleksiy Grechnyev, IT-JIM
3+
/// Here I show how to use ImageCroppingCalculator with a dynamic crop
4+
/// Translation: the cropping rect is no longer supplied in calculator options,
5+
/// But cropping rects for each frame come from a separate input stream in_rect
6+
/// ACHTUNG! This pipeline is still NOT real-time!
7+
8+
9+
#include <iostream>
10+
#include <string>
11+
#include <memory>
12+
#include <atomic>
13+
#include <mutex>
14+
#include <cmath>
15+
16+
#include "mediapipe/framework/calculator_framework.h"
17+
#include "mediapipe/framework/formats/image_frame.h"
18+
#include "mediapipe/framework/formats/image_frame_opencv.h"
19+
#include "mediapipe/framework/port/parse_text_proto.h"
20+
#include "mediapipe/framework/port/status.h"
21+
22+
#include "mediapipe/framework/port/opencv_highgui_inc.h"
23+
#include "mediapipe/framework/port/opencv_imgproc_inc.h"
24+
25+
#include "mediapipe/framework/formats/rect.pb.h"
26+
27+
//==============================================================================
28+
mediapipe::Status run() {
29+
using namespace std;
30+
using namespace mediapipe;
31+
32+
// A graph with ImageCroppingCalculator
33+
// Here we have a second inpt stream in_rect
34+
// Which contains the cropping rect for each frame, note the tag RECT
35+
string protoG = R"(
36+
input_stream: "in"
37+
input_stream: "in_rect"
38+
output_stream: "out"
39+
node {
40+
calculator: "ImageCroppingCalculator"
41+
input_stream: "IMAGE:in"
42+
input_stream: "RECT:in_rect"
43+
output_stream: "IMAGE:out"
44+
}
45+
)";
46+
47+
// Parse config and create graph
48+
CalculatorGraphConfig config;
49+
if (!ParseTextProto<mediapipe::CalculatorGraphConfig>(protoG, &config)) {
50+
return absl::InternalError("Cannot parse the graph config !");
51+
}
52+
CalculatorGraph graph;
53+
MP_RETURN_IF_ERROR(graph.Initialize(config));
54+
55+
// Mutex protecting imshow() and the stop flag
56+
mutex mutexImshow;
57+
atomic_bool flagStop(false);
58+
59+
// Add observer to "out", then start the graph
60+
// This callback displays the frame on the screen
61+
auto cb = [&mutexImshow, &flagStop](const Packet &packet)->Status{
62+
63+
// Get cv::Mat from the packet
64+
const ImageFrame & outputFrame = packet.Get<ImageFrame>();
65+
cv::Mat ofMat = formats::MatView(&outputFrame);
66+
cv::Mat frameOut;
67+
cvtColor(ofMat, frameOut, cv::COLOR_RGB2BGR);
68+
cout << packet.Timestamp() << ": RECEIVED VIDEO PACKET size = " << frameOut.size() << endl;
69+
70+
{
71+
lock_guard<mutex> lock(mutexImshow);
72+
// Display frame on screen and quit on ESC
73+
cv::imshow("frameOut", frameOut);
74+
if (27 == cv::waitKey(1)){
75+
cout << "It's time to QUIT !" << endl;
76+
flagStop = true;
77+
}
78+
}
79+
return OkStatus();
80+
};
81+
MP_RETURN_IF_ERROR(graph.ObserveOutputStream("out", cb));
82+
graph.StartRun({});
83+
84+
// Start the camera and check that it works
85+
cv::VideoCapture cap(cv::CAP_ANY);
86+
if (!cap.isOpened())
87+
return absl::NotFoundError("CANNOT OPEN CAMERA !");
88+
cv::Mat frameIn, frameInRGB;
89+
90+
// Camera loop, runs until we get flagStop == true
91+
for (int i=0; !flagStop ; ++i){
92+
// Read next frame from camera
93+
cap.read(frameIn);
94+
if (frameIn.empty())
95+
return absl::NotFoundError("CANNOT OPEN CAMERA !");
96+
97+
cout << "SIZE_IN = " << frameIn.size() << endl;
98+
{
99+
lock_guard<mutex> lock(mutexImshow);
100+
cv::imshow("frameIn", frameIn);
101+
}
102+
103+
// Convert it to a packet and send
104+
cv::cvtColor(frameIn, frameInRGB, cv::COLOR_BGR2RGB);
105+
ImageFrame *inputFrame = new ImageFrame(
106+
ImageFormat::SRGB, frameInRGB.cols, frameInRGB.rows, ImageFrame::kDefaultAlignmentBoundary
107+
);
108+
frameInRGB.copyTo(formats::MatView(inputFrame));
109+
Timestamp ts(i);
110+
MP_RETURN_IF_ERROR(graph.AddPacketToInputStream("in",
111+
Adopt(inputFrame).At(ts)
112+
));
113+
114+
// Create a crop rect (center+width+height) for each frame
115+
// Let's move the rect up and down for fun
116+
int yc = int(frameIn.rows * (0.5 + 0.2 * cos(0.3 * i)));
117+
Rect rect;
118+
rect.set_width(0.8*frameIn.cols);
119+
rect.set_height(0.4*frameIn.rows);
120+
rect.set_x_center(frameIn.cols / 2);
121+
rect.set_y_center(yc);
122+
MP_RETURN_IF_ERROR(graph.AddPacketToInputStream("in_rect", MakePacket<Rect>(rect).At(ts)));
123+
}
124+
// Don't forget to close both input streams !
125+
graph.CloseInputStream("in");
126+
graph.CloseInputStream("in_rect");
127+
// Wait for the graph to finish
128+
MP_RETURN_IF_ERROR(graph.WaitUntilDone());
129+
return OkStatus();
130+
}
131+
132+
//==============================================================================
133+
int main(int argc, char** argv){
134+
using namespace std;
135+
136+
FLAGS_alsologtostderr = 1;
137+
google::SetLogDestination(google::GLOG_INFO, ".");
138+
google::InitGoogleLogging(argv[0]);
139+
140+
cout << "Example 2.3 : Video pipeline with ImageCroppingCalculator (dynamic crop)" << endl;
141+
mediapipe::Status status = run();
142+
cout << "status =" << status << endl;
143+
cout << "status.ok() = " << status.ok() << endl;
144+
return 0;
145+
}

0 commit comments

Comments
 (0)