[Feature Request] Deployment in c++ project

[x10000year]

In production environment, deep learning models have to be deployed in applications developed in languages beyond python. ONNX is never a viable solution for complex projects in which there are many custom ops/extensions. It is terrible to develop and maintain the same custom ops/extensions for both pytorch and caffe2. c/c++ interfaces for model deployment is needed, even if it may require running python interpreter in c/c++.

Another feature request is better support for custom op. In tensorflow, custom ops can be built into a .so with the toolchain of the user's choice. The custom ops can be used in training after loading the .so file in python. The ffi way provided by pytorch is too limited and is not suitable for big c/c++ projects.

[ezyang]

CC @zdevito

Although we have not ben advertising this widely yet (as there are some BC-breaking changes coming down the pipe), ATen is intended to be a C++ library for working with Torch operations. PyTorch already heavily uses ATen internally to implement a lot of functionality, and we are actively exploring using ATen to make it easy to move custom C++ ops written for PyTorch easily into Caffe2. (Why still Caffe2? ATen is just a library, it is not a runtime, and short term, we don't plan to be in the business of building a new, production-ready runtime on top of Torch.)

[x10000year]

@ezyang cool

[ahkarami]

Dear @ezyang & @x10000year,
Suppose I have a pre-trained classic CNN model (e.g., ResNet which is trained on a dataset) in PyTorch. Now, I want to convert this model to C++ (fro some reasons like better performance). What is the best, fastest, and easiest way for doing this task?
Is the PyTorch ATen (A TENsor library) solution of this work?
What is your opinion about Pytorch C++ Library?

[ezyang]

For a classic CNN, your language choice doesn't really matter at all, they will all perform basically the same. The more important factor is what environment you need to integrate the model in.

[ahkarami]

Dear @ezyang,
Thank you for your answer. I agree with you about classic models. However, suppose one must convert some pre-trained models (e.g., detection based models, 3D ConvNets, etc.) from PyTorch into C++ (e.g., for production & deployment in a large and commercial project).
To the best of my knowledge, In this cases, the PyTorch team offers ONNX to convert models into Caffe2. But, In my opinion, the syntax of caffe2 framework is so hard and its tutorials & docs are really bad (especially for its C++ API). Now, Is another better solution exists for doing this task?
As I have searched about this task, I have found that ATen Library can be a great solution for doing such tasks. However, as you have mentioned, ATen is just a library, it is not a runtime. Also you have mentioned that we don't plan to be in the business of building a new, production-ready runtime on top of Torch. Has the plan of PyTorch team changed about designing a new C++ production-ready runtime on top of PyTorch? (If your answer is no, Why not? I think it would be really great that the PyTorch has a C++ API like its current fantastic syntax.) The Pytorch C++ Library actually done it, and I think it is a really fantastic project, but unfortunately it seems that its contributor hasn't any plan for progress it. Maybe it would be great that PyTorch team think about designing and releasing a nice C++ API for PyTorch via ATen library.

[apaszke]

@ahkarami We are thinking of adding PyTorch-like C++ interfaces, but it's not a top priority for us at the moment. ATen is still evolving, and as @ezyang there will be some breaking changes along the way. Think of it as an alpha version.

[ahkarami]

Dear @apaszke,
Thank you for your time and response. I hope that the PyTorch-like C++ interface will release as soon as possible.

[samhodge]

One of the issues I have had with converting CNNs to a production ready C++ solution is the fact that the dimensions are often fixed in the serialized model. But if you want to process an image of any arbitrary dimension you need more of a dynamic symbol based representation of the network rather than a fixed list of operations for a given known input . While this is possible for Python with the whole framework at your disposal, when all you have is a serialized network and something like ATen to run the feed forward it’s not the same as having the full functionality of PyTorch in C++.

[ebetica]

Shameless plug https://github.com/ebetica/autogradpp

[thuyen]

@ezyang At the moment, is there any way to use custom ATen ops in caffe2? The docs currently only show how to use built-in ATen Ops.
https://github.com/pytorch/pytorch/blob/master/caffe2/contrib/aten/README.md

Thank you

[ezyang]

@thuyen I don't think this is possible at this moment. C10 will hopefully make this better but don't expect it in the next month.