Share Images (high-level abstraction over Blob)

[mgiuca]

In #7 I proposed sharing Blobs. The primary use case for this is sharing images, but using this API to share images is a bit hard. If you wanted to share the contents of a <canvas> you would do:

var canvas;  // HTMLCanvasElement representing a <canvas> element
canvas.toBlob(blob => navigator.share({blob: blob, mimeType: 'image/png'}),
              'image/png');

A little unwieldy, but manageable.

To share an <img> element, things get a bit hairy:

var image;  // HTMLImageElement representing an <img> element
var canvas = document.createElement('canvas');
canvas.width = image.width;
canvas.height = image.height;
canvas.getContext('2d').drawImage(image, 0, 0);
canvas.toBlob(blob => navigator.share({blob: blob, mimeType: 'image/png'}),
              'image/png');

(Note: I haven't tested this code, just wrote it on the fly. There may be an easier way but that's the best way I can find to convert an image into a blob.)

So maybe we should be providing a share attribute called 'image' that takes a CanvasImageSource -- basically lets you directly share an Image, Video or Canvas object without having to convert to a blob. The usage would be:

var image;  // HTMLImageElement representing an <img> element
navigator.share({image: image, mimeType: 'image/png'}),

As shown above, this isn't strictly necessary (can be polyfilled). So it would be a policy decision whether we want to provide unnecessary-but-helpful abstractions, or whether we should save that for wrapper libraries.

[PaulKinlan]

+1 to this, this seems like a really nice simplification of the API. Do we need mimeType if the image is already an ImageElement - I suspect it already has the data type.

I was also thinking in the future you might want to pass in a MediaElement too for sharing audio/movies (think snapchat client etc)

[mgiuca]

Do we need mimeType if the image is already an ImageElement - I suspect it already has the data type.

It depends on what type of image source it is. For example if it's a Canvas element then it doesn't have a MIME type because it's just raw pixel data, and the mimeType attribute you set there would specify what format to write it out to.

Even for an Image element, I don't think the element will retain the original file; it will have loaded it into raw pixels, and this will be re-encoding the output. It's possible we want a special case if the element is in fact an <img> with a source, that it should be writing out the same file as the source (rather than exporting to a new file, which for JPEG means lossy re-encoding), but this is a special case, not the general case.

[mgiuca]

There are some design issues here:

What does mimeType mean in the context of multiple fields? For example if you have:

navigator.share({text: 'My cat', image: catPic, mimeType: 'image/jpg'});

It isn't quite clear that mimeType should apply to the image rather than the text, or the whole object. Maybe text has an implied MIME type of 'text/plain', but what if we introduce more fields, like blob? Perhaps a better design is nested:

navigator.share({text: 'My cat',
                 image: {source: catPic, mimeType: 'image/jpg'}});

The dict would be optional (image could either be a dict or a CanvasImageSource). If mimeType is not supplied it would either be the type of the image source, or default to 'image/png' if it's a canvas.

Thoughts?

[mgiuca]

There's a further complication with CSP / CORS: a website may not be able to convert an image into a blob due to cross-origin protections (and we would have to enforce those rules too, if we included the ability to send image data from an image object).

Take the code I mentioned above:

var image;  // HTMLImageElement representing an <img> element
var canvas = document.createElement('canvas');
canvas.width = image.width;
canvas.height = image.height;
canvas.getContext('2d').drawImage(image, 0, 0);
canvas.toBlob(blob => navigator.share({blob: blob, mimeType: 'image/png'}),
              'image/png');

If image is not origin-clean, this fails with a SecurityError, because it would give the site access to the pixels of an image from another origin. However, it may still be desirable to share images that the sending site does not have pixel access to (e.g., served from a CDN).

This brings us to the realisation that there are two distinct use cases for sharing images:

1. Share an image by URL (potentially from a different origin).
2. Share an image as data (possibly one created dynamically in a canvas, or from an image private to the sending site that requires cookies to access).

These two use cases are overlapping but neither subsumes the other, so we may have to provide both.

Before we proceed, I think we need to better understand the use cases that we want this for. Currently, the receivers we're looking at (native Android apps) pretty much all ignore any attached images. Social networking apps pretty much universally ignore the attached image and fetch their own image from the shared URL. So there may not be a lot of value in doing this anyway.

@PaulKinlan You said this would be useful: what use cases are you thinking of?

[robdodson]

The dict would be optional (image could either be a dict or a CanvasImageSource). If mimeType is not supplied it would either be the type of the image source, or default to 'image/png' if it's a canvas.

That sounds ok to me.

Currently, the receivers we're looking at (native Android apps) pretty much all ignore any attached images. Social networking apps pretty much universally ignore the attached image and fetch their own image from the shared URL. So there may not be a lot of value in doing this anyway.

Do you know why Android and social networking sites ignore attached images?

[mgiuca]

Do you know why Android and social networking sites ignore attached images?

I don't actually know why, but I suspect there's a chicken-and-egg problem: most senders don't include an image and receivers want one, so instead of looking in the intent for an image, they just fetch the page and pull one there. They would already have code for this since they do it when users paste a URL.

Also this way guarantees that the thumbnail attached to the post is actually from the linked site; if they respected the attached image, it would allow users to share a link to a site with any image they like.

[PaulKinlan]

Google keep shares a screen shot of the page, and the app has to know how to pull it in via a stream extra (at least if it is handling a normal share) https://paul.kinlan.me/parsing-screenshot-from-Chrome-for-Android-send-intent/

…

On Mon, Nov 28, 2016, 11:02 PM Matt Giuca ***@***.***> wrote: Do you know why Android and social networking sites ignore attached images? I don't actually know why, but I suspect there's a chicken-and-egg problem: most senders don't include an image and receivers want one, so instead of looking in the intent for an image, they just fetch the page and pull one there. They would already have code for this since they do it when users paste a URL. Also this way guarantees that the thumbnail attached to the post is actually from the linked site; if they respected the attached image, it would allow users to share a link to a site with any image they like. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACxxir9KbYctSEqatZXgj2jM7yJYku8ks5rC11zgaJpZM4Kr3QN> .

[mgiuca]

@PaulKinlan "Google keep shares a screen shot of the page" -- I think you mean Google Chrome shares a screenshot of the page, and Keep knows how to pull it in, right?

At least that was my go-to example of the image sharing working between two apps on Android. Sadly, I tried it yesterday and Keep ignored the image; later it fetched its own image from the page just like G+ or Facebook does. :( Therefore, I don't know of any Android apps that will actually use an image sent in a SEND intent.

[justinfagnani]

Pardons for my ignorance - I don't know much about web-share, but I was asking @robdodson some naive questions and he just dropped me in here :)

My biggest question is: is, or if not why isn't, the share API using a DataTransfer object? If it did then it would have multi-format capabilities by default and share some API with clipboard and drag and drop. Not sure what the API would look like, since DataTransfer isn't currently constructible.

[mgiuca]

Hi justinfagnani@. To be honest, I hadn't thought about that. I looked it up (MDN: DataTransfer).

That's a good idea. I'd be a bit hesitant to tie Web Share to the existing drag-and-drop spec, because a) it might tie us into the specific shape of drag-and-drop / clipboard data which isn't quite like a social share, and b) it might limit us if we want to add a feature that isn't supported in DataTransfer (it would be much harder to argue to extend DataTransfer just for sharing, than to extend our own ShareData class).

I'm really confused (from reading both MDN and the spec) what a DataTransfer really is... it has an "old interface" which appears to be a mapping from MIME type to data (implying that there is only one piece of data associated with a DataTransfer but it can be expressed in many formats, perhaps a "text/plain" and a "text/html"). And it has a "new interface" which is a read-only list of items, which can be files and multiple items can have the same format, implying a collection of multiple different objects (perhaps different files that have all been dragged together).

The old interface feels designed for clipboard while the new interface feels designed for drag-and-drop. It's weird that they're both in the same object because clipboard feels very different to drag-and-drop. Rolling share into that might complicate things even more, since we have a different set of constraints: we have specific fields with fixed semantics, like "title" and "text" which wouldn't easily fit into either of those models.

Still, something to think about. I will think some more and ask around.

[mgiuca]

I was contacted by a developer about this who wants to use it in an app that has an image editor in a canvas, to share the contents of the canvas.

I think this is a totally legitimate use case and we should support it, despite my previous reservations (that it won't let you share static images from your own page if they are blocked by cross-origin protections). I think the sharing static images use case is "holding it wrong". The real use case for sharing images is to share dynamic images being edited or created by the page, and there is currently no way to do that.

[ipeychev]

Since I am this developer, here is my input too :)

The application I'm working on is a collaborative whiteboard, canvas based. I wanted to allow sharing the canvas as an image, leveraging the native functionality in Android. This is indeed a legitimate and a quite realistic use case.
It was said it could be done via Blob and I think this should be fine for the developers.

[cdata]

I share a similar feeling to @mgiuca about static images and "holding it wrong." This API seems especially useful for cases where images are being created in place by the user of the web site in question. If the image is static, why not just share the URL for it instead?

Diving a little deeper, I'm curious to know if this API will eventually support sharing videos like those that can be generated with MediaRecorder. While it may be easy enough to temporarily host an image for the sake of sharing the URL, it is more challenging to use this flow to support sharing a video. Let me know if you think this should be a separate issue.

[mgiuca]

I think video and image sharing should be captured in the same way. We can just think of a video as an image with a different MIME type. (The big difference is that some apps may accept images but not videos and vice versa; we can differentiate those by filtering based on MIME type.)

[domenic]

Why not (ImageBitmapSource or ImageBitmap) instead of CanvasImageSource? The more general type, the better.

I'm not sure whether such an image type would obviate the need for a blob option at all, or whether you want to keep blob and image distinct.

[mgiuca]

Why not (ImageBitmapSource or ImageBitmap) instead of CanvasImageSource? The more general type, the better.

Not sure what ImageBitmapSource is (a quick Google doesn't show an MDN or anything that explains what it is). CanvasImageSource is a super-type of HTMLImageElement, HTMLVideoElement, HTMLCanvasElement, and ImageBitmap. So that's the most general thing I could find.

That was an alternative to accepting an image blob.

[domenic]

https://html.spec.whatwg.org/multipage/webappapis.html#imagebitmapsource

[mgiuca]

Oh I see, it's a superclass of that. Yes, that makes sense.

[mgiuca]

We could also just allow (ImageBitmapSource or ImageBitmap or HTMLImageElement) to be shared, which would mean you can either share your own pixel data or the pixel data from an image that you are unable to read due to CORS.

We would have to carefully consider the security of this, however (since it would mean the site is able to share data from a remote site, potentially using the user's cookie jar, that it doesn't itself have read-access to). It might be OK because it's only being shared with a target of the user's choice, not the site's choice.

[domenic]

ImageBitmapSource includes CanvasImageSource which includes HTMLImageElement, so I'm a bit confused.

Also I just realized ImageBitmapSource already includes ImageBitmap since it's part of CanvasImageSource.

For a bit of clarity, as these nested typedefs are getting confusing for me, let's list everything...

• HTMLImageElement (*)
• SVGImageElement (*)
• HTMLVideoElement (**)
• HTMLCanvasElement (**)
• ImageBitmap (**)
• OffscreenCanvas (**)
• Blob (***)
• ImageData (***)

Here:

• * = HTMLOrSVGImageElement
• * + ** = CanvasImageSource
• * + ** + *** = ImageBitmapSource

I think this is correct... We should probably centralize these in HTML and add a nice table or something.

[mgiuca]

Hah OK, thanks for the clarifications.

I wasn't so much concerned with the types as the ability to use an HTMLImageElement which has additional utility but also potential security ramifications. Alternatively, we could explicitly state that it must be origin-clean or it's a SecurityError.

[domenic]

Yeah, that makes sense, and is probably better than rejecting HTMLImageElement entirely, since you could bypass such a type check by just using createImageBitmap. Being conservative with regard to cross-origin data, at least to start, seems good.

I think you'd have a few different checks, e.g. origin-clean flag for ImageBitmap and the image's origin for HTMLImageElement, but those details can be figured out.

[wibblymat]

Canvas elements already have cross-origin logic. If you call drawImage on a canvas context, the source image will be checked. Cross origin images without CORS headers cause the canvas to become 'tainted'. Similarly, using a tainted canvas as the source will propagate the taint to the next canvas.

Seems like this already specced behaviour can be reused directly.

[mgiuca]

@wibblymat Yes, I'm aware of that logic and I'm reusing it in the Web Share spec. What I was considering up above was whether it was necessary to restrict sharing of images with CORS enabled. I suppose the simplest way to express this is to consider whether a tainted canvas (or equivalently, an image source that would taint the canvas) can be shared. I could go either way on this but it's a good idea to start conservative as Dom suggests.

I've put together a draft spec for sharing images using this concept. Don't need to review it yet, but just an FYI.

[rdtandel]

is there any working code for image share? please share over here.
I've tried below codes:
1.await navigator.share({text: text,url: url,image: document.querySelector('#img'), mimeType: 'image/png'});
2.var image1 = new Image();
image1.src = linkToShare;
image1.onload = function () {
var canvas = document.createElement('canvas');
canvas.width = this.naturalWidth; // or 'width' if you want a special/scaled size
canvas.height = this.naturalHeight; // or 'height' if you want a special/scaled size

		canvas.getContext('2d').drawImage(this, 0, 0);
                    toDataURL = canvas.toDataURL('image/png');
      await navigator.share({text: text,url: url,image: toDataURL , mimeType: 'image/png'});

3. blob example above explained.

[mgiuca]

No, you cannot currently share images with navigator.share.

[YKingslayer]

I don't knew why cannot currently share images with navigator.share.

That's so sad about now.

[anik587]

you need https for using navigator.share

[bogdyak]

No, you cannot currently share images with navigator.share.

Any changes at 2020 ?
Thanks

[mgiuca]

This is fixed in the Level 2 specification: https://w3c.github.io/web-share/level-2/

Note that it still needs to be condensed into the base spec (so it's technically considered a draft, though it works in Chrome). Closing this issue; follow #7 for updates on merging that work into the base spec.