4.8.10.13. User interface

If the attribute is present, or if scripting is disabled for the media element, then the user agent should expose a user interface to the user. This user interface should include features to begin playback, pause playback, seek to an arbitrary position in the content (if the content supports arbitrary seeking), change the volume, change the display of closed captions or embedded sign-language tracks, select different audio tracks or turn on audio descriptions, and show the media content in manners more suitable to the user (e.g. full-screen video or in an independent resizable window). Other controls may also be made available.

If the media element has a current media controller, then the user agent should expose audio tracks from all the slaved media elements (although avoiding duplicates if the same media resource is being used several times). If a media resource's audio track exposed in this way has no known name, and it is the only audio track for a particular media element, the user agent should use the element's title attribute, if any, as the name (or as part of the name) of that track.

Even when the attribute is absent, however, user agents may provide controls to affect playback of the media resource (e.g. play, pause, seeking, and volume controls), but such features should not interfere with the page's normal rendering. For example, such features could be exposed in the media element's context menu. The user agent may implement this simply by exposing a user interface to the user as described above (as if the controls attribute was present).

Where possible (specifically, for starting, stopping, pausing, and unpausing playback, for seeking, for changing the rate of playback, for fast-forwarding or rewinding, for listing, enabling, and disabling text tracks, and for muting or changing the volume of the audio), user interface features exposed by the user agent must be implemented in terms of the DOM API described above, so that, e.g., all the same events fire.

When a media element has a current media controller, the user agent's user interface for pausing and unpausing playback, for seeking, for changing the rate of playback, for fast-forwarding or rewinding, and for muting or changing the volume of audio of the entire group must be implemented in terms of the MediaController API exposed on that current media controller. When a media element has a current media controller, and all the slaved media elements of that MediaController are paused, the user agent should also unpause all the slaved media elements when the user invokes a user agent interface control for beginning playback.

The "play" function in the user agent's interface must set the playbackRate attribute to the value of the defaultPlaybackRate attribute before invoking the play() method. When a media element has a current media controller, the attributes and method with those names on that MediaController object must be used. Otherwise, the attributes and method with those names on the media element itself must be used.

Features such as fast-forward or rewind must be implemented by only changing the playbackRate attribute (and not the defaultPlaybackRate attribute). Again, when a media element has a current media controller, the attributes with those names on that MediaController object must be used; otherwise, the attributes with those names on the media element itself must be used.

When a media element has a current media controller, seeking must be implemented in terms of the currentTime attribute on that MediaController object. Otherwise, the user agent must directly seek to the requested position in the media element's media timeline. For media resources where seeking to an arbitrary position would be slow, user agents are encouraged to use the approximate-for-speed flag when seeking in response to the user manipulating an approximate position interface such as a seek bar.

When a media element has a current media controller, user agents may additionally provide the user with controls that directly manipulate an individual media element without affecting the MediaController, but such features are considered relatively advanced and unlikely to be useful to most users.

For the purposes of listing chapters in the media resource, only text tracks in the media element's list of text tracks that are showing and whose text track kind is chapters should be used. Such tracks must be interpreted according to the rules for constructing the chapter tree from a text track. When seeking in response to a user maniplating a chapter selection interface, user agents should not use the approximate-for-speed flag.

The controls IDL attribute must reflect the content attribute of the same name.

The volume attribute must return the playback volume of any audio portions of the media element, in the range 0.0 (silent) to 1.0 (loudest). Initially, the volume should be 1.0, but user agents may remember the last set value across sessions, on a per-site basis or otherwise, so the volume may start at other values. On setting, if the new value is in the range 0.0 to 1.0 inclusive, the playback volume of any audio portions of the media element must be set to the new value. If the new value is outside the range 0.0 to 1.0 inclusive, then, on setting, an IndexSizeError exception must be thrown instead.

The muted attribute must return true if the audio output is muted and false otherwise. Initially, the audio output should not be muted (false), but user agents may remember the last set value across sessions, on a per-site basis or otherwise, so the muted state may start as muted (true). On setting, if the new value is true then the audio output should be muted and if the new value is false it should be unmuted.

Whenever either of the values that would be returned by the volume and muted attributes change, the user agent must queue a task to fire a simple event named volumechange at the media element.

An element's effective media volume is determined as follows:

1. If the user has indicated that the user agent is to override the volume of the element, then the element's effective media volume is the volume desired by the user. Abort these steps.

2. If the element's audio output is muted, the element's effective media volume is zero. Abort these steps.

3. If the element has a current media controller and that MediaController object's media controller mute override is true, the element's effective media volume is zero. Abort these steps.

4. Let volume be the playback volume of the audio portions of the media element, in range 0.0 (silent) to 1.0 (loudest).

5. If the element has a current media controller, multiply volume by that MediaController object's media controller volume multiplier.

6. The element's effective media volume is volume, interpreted relative to the range 0.0 to 1.0, with 0.0 being silent, and 1.0 being the loudest setting, values in between increasing in loudness. The range need not be linear. The loudest setting may be lower than the system's loudest possible setting; for example the user could have set a maximum volume.

When a media element is created, if it has a muted attribute specified, the user agent must mute the media element's audio output, overriding any user preference.

The defaultMuted IDL attribute must reflect the muted content attribute.

The length IDL attribute must return the number of ranges represented by the object.

The start(index) method must return the position of the start of the indexth range represented by the object, in seconds measured from the start of the timeline that the object covers.

The end(index) method must return the position of the end of the indexth range represented by the object, in seconds measured from the start of the timeline that the object covers.

These methods must throw IndexSizeError exceptions if called with an index argument greater than or equal to the number of ranges represented by the object.

When a TimeRanges object is said to be a normalized TimeRanges object, the ranges it represents must obey the following criteria:

• The start of a range must be greater than the end of all earlier ranges.
• The start of a range must be less than the end of that same range.

In other words, the ranges in such an object are ordered, don't overlap, aren't empty, and don't touch (adjacent ranges are folded into one bigger range).

Ranges in a TimeRanges object must be inclusive.

Thus, the end of a range would be equal to the start of a following adjacent (touching but not overlapping) range. Similarly, a range covering a whole timeline anchored at zero would have a start equal to zero and an end equal to the duration of the timeline.

The timelines used by the objects returned by the buffered, seekable and played IDL attributes of media elements must be that element's media timeline.

The track attribute must return the value it was initialized to. When the object is created, this attribute must be initialized to null. It represents the context information for the event.

4.8.10.17 Security and privacy considerations

The main security and privacy implications of the video and audio elements come from the ability to embed media cross-origin. There are two directions that threats can flow: from hostile content to a victim page, and from a hostile page to victim content.

If a victim page embeds hostile content, the threat is that the content might contain scripted code that attempts to interact with the Document that embeds the content. To avoid this, user agents must ensure that there is no access from the content to the embedding page. In the case of media content that uses DOM concepts, the embedded content must be treated as if it was in its own unrelated top-level browsing context.

For instance, if an SVG animation was embedded in a video element, the user agent would not give it access to the DOM of the outer page. From the perspective of scripts in the SVG resource, the SVG file would appear to be in a lone top-level browsing context with no parent.

If a hostile page embeds victim content, the threat is that the embedding page could obtain information from the content that it would not otherwise have access to. The API does expose some information: the existence of the media, its type, its duration, its size, and the performance characteristics of its host. Such information is already potentially problematic, but in practice the same information can more or less be obtained using the img element, and so it has been deemed acceptable.

However, significantly more sensitive information could be obtained if the user agent further exposes metadata within the content such as subtitles or chapter titles. Such information is therefore only exposed if the video resource passes a CORS resource sharing check. The crossorigin attribute allows authors to control how this check is performed. [CORS]

Without this restriction, an attacker could trick a user running within a corporate network into visiting a site that attempts to load a video from a previously leaked location on the corporation's intranet. If such a video included confidential plans for a new product, then being able to read the subtitles would present a serious confidentiality breach.

4.8.10.19 Best practices for implementors of media elements

This section is non-normative.

How accurately various aspects of the media element API are implemented is considered a quality-of-implementation issue.

For example, when implementing the buffered attribute, how precise an implementation reports the ranges that have been buffered depends on how carefully the user agent inspects the data. Since the API reports ranges as times, but the data is obtained in byte streams, a user agent receiving a variable-bit-rate stream might only be able to determine precise times by actually decoding all of the data. User agents aren't required to do this, however; they can instead return estimates (e.g. based on the average bit rate seen so far) which get revised as more information becomes available.

As a general rule, user agents are urged to be conservative rather than optimistic. For example, it would be bad to report that everything had been buffered when it had not.

Another quality-of-implementation issue would be playing a video backwards when the codec is designed only for forward playback (e.g. there aren't many key frames, and they are far apart, and the intervening frames only have deltas from the previous frame). User agents could do a poor job, e.g. only showing key frames; however, better implementations would do more work and thus do a better job, e.g. actually decoding parts of the video forwards, storing the complete frames, and then playing the frames backwards.

Similarly, while implementations are allowed to drop buffered data at any time (there is no requirement that a user agent keep all the media data obtained for the lifetime of the media element), it is again a quality of implementation issue: user agents with sufficient resources to keep all the data around are encouraged to do so, as this allows for a better user experience. For example, if the user is watching a live stream, a user agent could allow the user only to view the live video; however, a better user agent would buffer everything and allow the user to seek through the earlier material, pause it, play it forwards and backwards, etc.

When multiple tracks are synchronised with a MediaController, it is possible for scripts to add and remove media elements from the MediaController's list of slaved media elements, even while these tracks are playing. How smoothly the media plays back in such situations is another quality-of-implementation issue.

When a media element that is paused is removed from a document and not reinserted before the next time the event loop spins, implementations that are resource constrained are encouraged to take that opportunity to release all hardware resources (like video planes, networking resources, and data buffers) used by the media element. (User agents still have to keep track of the playback position and so forth, though, in case playback is later restarted.)

The size of the coordinate space does not necessarily represent the size of the actual bitmap that the user agent will use internally or during rendering. On high-definition displays, for instance, the user agent may internally use a bitmap with two device pixels per unit in the coordinate space, so that the rendering remains at high quality throughout.

When the canvas element is created, and subsequently whenever the width and height attributes are set (whether to a new value or to the previous value), the bitmap and any associated contexts must be cleared back to their initial state and reinitialized with the newly specified coordinate space dimensions.

When the canvas is initialized, its bitmap must be cleared to transparent black.

When a canvas element does not represent its fallback content, it provides a paint source whose width is the element's intrinsic width, whose height is the element's intrinsic height, and whose appearance is the element's bitmap.

The width and height IDL attributes must reflect the respective content attributes of the same name, with the same defaults.

A canvas element can have a primary context, which is the first context to have been obtained for that element. When created, a canvas element must not have a primary context.

The most commonly used primary context is the HTML Canvas 2D Context. [CANVAS2D]

The getContext(contextId, arguments...) method of the canvas element, when invoked, must run the following steps:

1. Let contextId be the first argument to the method.

2. If contextId is not the name of a context supported by the user agent, return null and abort these steps.

   An example of this would be a user agent that theoretically supports the "webgl" 3D context, in the case where the platform does not have hardware support for OpenGL and the user agent does not have a software OpenGL implementation. Despite the user agent recognising the "webgl" name, it would return null at this step because that context is not, in practice, supported at the time of the call.

3. If the element has a primary context and that context's entry in the WHATWG Wiki CanvasContexts page does not list contextId as a context with which it is compatible, return null and abort these steps. [WHATWGWIKI]

4. If the element does not have a primary context, let the element's primary context be contextId.

5. If the getContext() method has already been invoked on this element for the same contextId, return the same object as was returned that time, and abort these steps. The additional arguments are ignored.

6. Return a new object for contextId, as defined by the specification given for contextId's entry in the WHATWG Wiki CanvasContexts page. [WHATWGWIKI]

The supportsContext(contextId, arguments...) method of the canvas element, when invoked, must return false if calling getContext() on the same object and with the same arguments would definitely return null at this time, and true otherwise.

New context types may be registered in the WHATWG Wiki CanvasContexts page. [WHATWGWIKI]

Anyone is free to edit the WHATWG Wiki CanvasContexts page at any time to add a new context type. These new context types must be specified with the following information:

Keyword

The value of contextID that will return the object for the new API.

Specification

A link to a formal specification of the context type's API. It could be another page on the Wiki, or a link to an external page. If the type does not have a formal specification, an informal description can be substituted until such time as a formal specification is available.

Compatible with

The list of context types that are compatible with this one (i.e. that operate on the same underlying bitmap). This list must be transitive and symmetric; if one context type is defined as compatible with another, then all types it is compatible with must be compatible with all types the other is compatible with.

Vendors may also define experimental contexts using the syntax vendorname-context, for example, moz-3d. Such contexts should be registered in the WHATWG Wiki CanvasContexts page.

The toDataURL() and toDataURLHD() methods must run the following steps:

1. If the canvas element's origin-clean flag is set to false, throw a SecurityError exception and abort these steps.

2. If the canvas has no pixels (i.e. either its horizontal dimension or its vertical dimension is zero) then return the string "data:," and abort these steps. (This is the shortest data: URL; it represents the empty string in a text/plain resource.)

3. Let file be a serialization of the image as a file, using the method's arguments (if any) as the arguments, and with the native flag set if the method is the toDataURLHD() method, and not otherwise.

4. Return a data: URL representing file. [RFC2397]

The toBlob() and toBlobHD() methods must run the following steps:

1. If the canvas element's origin-clean flag is set to false, throw a SecurityError exception and abort these steps.

2. Let callback be the first argument.

3. Let arguments be the second and subsequent arguments to the method, if any.

4. If the canvas has no pixels (i.e. either its horizontal dimension or its vertical dimension is zero) then let result be null.

   Otherwise, let result be a Blob object representing a serialization of the image as a file, using arguments, and with the native flag set if the method is the toBlobHD() method, and not otherwise. [FILEAPI]

5. Return, but continue running these steps asynchronously.

6. If callback is null, abort these steps.

7. Queue a task to invoke the FileCallback callback with result as its argument. The task source for this task is the canvas blob serialization task source. [FILESYSTEMAPI]

When a user agent is to create a serialization of the image as a file, optionally with some given arguments, and optionally with a native flag set, it must create an image file in the format given by the first value of arguments, or, if there are no arguments, in the PNG format. [PNG]

If the native flag is set, or if the canvas bitmap has one pixel per coordinate space unit, then the image file must have the same pixel data (before compression, if applicable) as the canvas bitmap, and if the file format used supports encoding resolution metadata, the resolution of the bitmap (device pixels per coordinate space units being interpreted as image pixels per CSS pixel) must be given as well.

Otherwise, the image file's pixel data must be the canvas bitmap's pixel data scaled to one image pixel per coordinate space unit, and if the file format used supports encoding resolution metadata, the resolution must be given as 96dpi (one image pixel per CSS pixel).

If arguments is not empty, the first value must be interpreted as a MIME type giving the format to use. If the type has any parameters, it must be treated as not supported.

For example, the value "image/png" would mean to generate a PNG image, the value "image/jpeg" would mean to generate a JPEG image, and the value "image/svg+xml" would mean to generate an SVG image (which would probably require that the implementation actually keep enough information to reliably render an SVG image from the canvas).

User agents must support PNG ("image/png"). User agents may support other types. If the user agent does not support the requested type, it must create the file using the PNG format. [PNG]

User agents must convert the provided type to ASCII lowercase before establishing if they support that type.

For image types that do not support an alpha channel, the serialized image must be the canvas image composited onto a solid black background using the source-over operator.

If the first argument in arguments gives a type corresponding to one of the types given in the first column of the following table, and the user agent supports that type, then the subsequent arguments, if any, must be treated as described in the second cell of that row.

For the purposes of these rules, an argument is considered to be a number if it is converted to an IDL double value by the rules for handling arguments of type any in the Web IDL specification. [WEBIDL]

Other arguments must be ignored and must not cause the user agent to throw an exception. A future version of this specification will probably define other parameters to be passed to these methods to allow authors to more carefully control compression settings, image metadata, etc.

4.8.11.1 Color spaces and color correction

The canvas APIs must perform color correction at only two points: when rendering images with their own gamma correction and color space information onto the canvas, to convert the image to the color space used by the canvas (e.g. using the 2D Context's drawImage() method with an HTMLImageElement object), and when rendering the actual canvas bitmap to the output device.

Thus, in the 2D context, colors used to draw shapes onto the canvas will exactly match colors obtained through the getImageDataHD() method.

The toDataURL() and toDataURLHD() methods must not include color space information in the resources they return. Where the output format allows it, the color of pixels in resources created by toDataURL() must match those returned by the getImageData() method, and the color of pixels in resources created by toDataURLHD() must match those returned by the getImageDataHD() method.

In user agents that support CSS, the color space used by a canvas element must match the color space used for processing any colors for that element in CSS.

The gamma correction and color space information of images must be handled in such a way that an image rendered directly using an img element would use the same colors as one painted on a canvas element that is then itself rendered. Furthermore, the rendering of images that have no color correction information (such as those returned by the toDataURL() method) must be rendered with no color correction.

Thus, in the 2D context, calling the drawImage() method to render the output of the toDataURLHD() method to the canvas, given the appropriate dimensions, has no visible effect.

4.8.11.2 Security with canvas elements

Information leakage can occur if scripts from one origin can access information (e.g. read pixels) from images from another origin (one that isn't the same).

To mitigate this, canvas elements are defined to have a flag indicating whether they are origin-clean. All canvas elements must start with their origin-clean set to true. The flag must be set to false if any of the following actions occur:

• The element's 2D context's drawImage() method is called with an HTMLImageElement or an HTMLVideoElement whose origin is not the same as that of the Document object that owns the canvas element.

• The element's 2D context's drawImage() method is called with an HTMLCanvasElement whose origin-clean flag is false.

• The element's 2D context's fillStyle attribute is set to a CanvasPattern object that was created from an HTMLImageElement or an HTMLVideoElement whose origin was not the same as that of the Document object that owns the canvas element when the pattern was created.

• The element's 2D context's fillStyle attribute is set to a CanvasPattern object that was created from an HTMLCanvasElement whose origin-clean flag was false when the pattern was created.

• The element's 2D context's strokeStyle attribute is set to a CanvasPattern object that was created from an HTMLImageElement or an HTMLVideoElement whose origin was not the same as that of the Document object that owns the canvas element when the pattern was created.

• The element's 2D context's strokeStyle attribute is set to a CanvasPattern object that was created from an HTMLCanvasElement whose origin-clean flag was false when the pattern was created.

• The element's 2D context's fillText() or strokeText() methods are invoked and consider using a font that has an origin that is not the same as that of the Document object that owns the canvas element. (The font doesn't even have to be used; all that matters is whether the font was considered for any of the glyphs drawn.)

The toDataURL(), toDataURLHD(), toBlob(), getImageData(), and getImageDataHD() methods check the flag and will throw a SecurityError exception rather than leak cross-origin data.

Even resetting the canvas state by changing its width or height attributes doesn't reset the origin-clean flag.