Support for images

[dberenbaum]

Image files can be important in dvc projects because:

• It's often easier to make custom plots as image files rather than using dvc plots.
• Especially in fields like computer vision, there may be other image types that need to be compared between commits.

I initially thought this was a straightforward feature to implement, but I think there are questions to discuss:

1. Should there be dvc-specific support for image diffs? What should the UI and output look like (for example, dvc images diff)? Should it support comparison of two images like metrics diff or more than two like plots diff?
2. Does dvc need a way to specify whether outputs are images (for example, adding an image: true option under a dvc.yaml output)? This might be helpful to show/diff all images in a project, and to enable support for apps like viewer to show all images. Could image files instead be inferred from file extensions?

Some examples of diffing images:
https://docs.github.com/en/github/managing-files-in-a-repository/rendering-and-diffing-images
https://github.com/ewanmellor/git-diff-image
https://oscarnajera.com/2017/11/git-diff-images-and-pdfs/

@pared @mnrozhkov @JIoJIaJIu @dmpetrov

[pared]

Does dvc need a way to specify whether outputs are images

Currently, we are determining the file type by extension. I think we could do the same for images. Let's start with supporting some popular formats, and implement new ones per users requests.

Should there be dvc-specific support for image diffs? What should the UI and output look like (for example, dvc images diff)? Should it support comparison of two images like metrics diff or more than two like plots diff?

I guess that depends on what we are actually expecting from dvc plots show for images. If we do expect it to embed the target image into HTML, then we could probably solve the diff problem by just showing the targets from different branches one near another, just like we do with the confusion matrix template in the current version of plots.

[skshetry]

Currently, we are determining the file type by extension. I think we could do the same for images. Let's start with supporting some popular formats, and implement new ones per users requests.

We could also try to detect if they are image using image headers and then fallback to using filename or just assume it to be a non-image.
https://docs.python.org/3/library/imghdr.html
https://github.com/h2non/filetype.py

P.S: This does mean that we'll have to read first few bytes, so it's not fast and easy to determine compared to the extension which we'll always have on hand, even if dvc doesn't have a file on hand.

[dberenbaum]

After thinking about it, I lean towards forcing users to be explicit about images. Thinking about a use case like computer vision, there may be lots of image files in the raw data, and then a small number of images used for reporting, plots, etc. It might be important to have dvc only manage the specified images.

I guess that depends on what we are actually expecting from dvc plots show for images.

Do you see this as being part of dvc plots show? I was thinking it was more likely to be a different command, like maybe dvc images show/diff.

[pared]

Do you see this as being part of dvc plots show? I was thinking it was more likely to be a different command, like maybe dvc images show/diff.

That might be a better idea. The question is, do we want to expand, already big, amount of commands. Maybe we should pack them into a subcommand? Like dvc vis plots and dvc vis images?

[dberenbaum]

That's a good idea, and maybe even combine metrics (and params?) into the same subcommand. I'm thinking through proposals to unify the various show/diff commands.

Introducing the subcommand would be a breaking change, though, right? Maybe it's better to implement as a separate command first and then reorganize commands later.

[pared]

Sure, reorganization should probably go for next major release.

[pmrowla]

I don't really see why this needs to be core DVC functionality. DVC should try to be data-type agnostic, and I'm not sure that setting a precedent for handling specific file types is a good idea.

To me, this sounds like a use case where a user should be using api.open()/api.read() to get file handles for the 2 (or more) versions of a file they want to binary diff, and then feed that data into their preferred diffing tool. (In this case, any one of the widely available image processing/diffing tools that are out there, like imagemagick or PIL/Pillow)

[pared]

Maybe what do we need is a way for users to api.get_url but in an aggregated way?
What I mean, is that we don't know what user will want to do with the binary target for diff - it can be image, it can be audio file, whatever.
We could however provide a method that would allow users to get all the required URLs from remote/cache. For example, user could ask "give me file_x and file_y for revisions rev1, rev2, rev3" and API could return structured data so that user can process that later in a way is suitable for the specific use case.

[dberenbaum]

@pmrowla That makes sense to me, and I think we are mostly in agreement? I tried to address some of these issues in the proposal for a unified diff (iterative/enhancement-proposals#4). We can continue discussion in either place.

@pared That seems pretty close to the diff driver suggestion in iterative/enhancement-proposals#4. There might need to be some minimal additional parsing for each type of output, like converting all metrics file types (json, yaml, etc.) to one consistent data schema.

[pared]

@dberenbaum I agree that it looks like a driver specification. Though I would argue that this use case is a little bit different: when we are talking about iterative/enhancement-proposals#4 we talk about diff command. Here, I am thinking about including some abstract driver into dvc.api so that users could handle their specific use cases by themselves if metrics/plots/params(?)/images(?) don't fill their needs.

[dberenbaum]

@pared Maybe I'm missing your point. I agree that it makes sense to have an abstract driver to handle custom needs outside of metrics/plots/params(?)/images(?). What is special about those other than support for diff (and show)?

[pared]

Well I guess the main point here is to let user implement own driver. I imagined it as a code-related task, and something user would be handling in his environment rather than cli but looking at the discussion under diff issue, my view on user- implemented driver is what you call difftool - so it seems to me we agree in that matter.

[pared]

It seems that subject raised in this discussion is also discussed in iterative/enhancement-proposals#4

[mnrozhkov]

Thanks for starting this discussion! Let me share my thoughts and expectations about this feature.

First, short thoughts on discussion above
1 - dvc-specific support for image diffs like dvc images diff seems unnecessary (images is just one type of content DVC may visualize with dvc plots) and it's enough to embed them into current plots functionality
2 - for me, the most natural way to work with images - dvc.yaml where I would specify some plot as an image

train_model:
 cmd: python train_model.py --base_config=config/params.yaml
 deps:
   - traintest.csv
   - train_model.py
 params:
   - config/params.yaml:
       - base
       - train
 outs:
   - model.pickle
 plots:
   - reports/grid_search.png:
       cache: true
       image: true
       title: Grid Search results

3 - with such dvc.yaml I could run dvc plots show/diff and get my images in generated plots.html file (let me share this slide below)

4 - I don't know about use case for computer vision though.

[dberenbaum]

Thanks, @mnrozhkov! I think this makes sense for your use case, but I worry that it may not generalize well for a few reasons:

• Other users may want to do the same for images that are not plots (see the image below for an example of what a computer vision diff might look like). In that case, would it be a separate section in dvc.yaml, or should plots still be used?
• In terms of implementation, the currently supported plots and the images would be quite different, so it might be hard to combine them. For example, the plots have a template option that wouldn't make much sense for images.
• DVC is primarily a command-line tool, and plots are already an anomaly in that they generate HTML output. It might make more sense for the slides you show above to be part of Studio or some other front-end extension and for DVC to only manage the data to generate those visualizations.

[dberenbaum]

How should image plots be identified?

Here are a few proposals (feel free to suggest others):

1. They must all be added explicitly to dvc.yaml (see the example above from @mnrozhkov).
2. Any file added to a specified plots or images directory path will be read like a plot regardless of whether it's in dvc.yaml at all.
3. Some kind of automatic heuristic can be used (any file with the appropriate extension stored in the Git repo).

These could also be combined so that there is a default behavior but also ways to be more flexible (similar to params reading automatically from params.yaml but also allowing custom params files).