GB2572947A

GB2572947A - Method and apparatus for encapsulating images in a file

Info

Publication number: GB2572947A
Application number: GB1805711.7A
Authority: GB
Inventors: Ouedraogo Naël; Maze Frédéric; Denoual Franck; Ruellan Hervé; Takaku Masahiko
Original assignee: Canon Inc
Current assignee: Canon Inc
Priority date: 2018-04-05
Filing date: 2018-04-05
Publication date: 2019-10-23
Also published as: GB202110996D0; GB2572660B; GB2572660A; GB2603225A; GB2603225B; GB201805711D0; GB201815558D0

Abstract

A first method involves determining a grouping type 402 based on a capture mode of the images 404-406. A grouping structure is then generated and comprises image identifiers of the images of the group. The grouping data structure, grouping type and the images are then embedded in the file. The capture mode may be a bracketing mode of different exposures or focal lengths, or panorama, time lapse, or image burst. A second method involves determining a label parameter describing information to annotate images and generating a grouping data structure comprising image identifiers of the images of the group. The grouping data structure, label parameter and images are then embedded in the file. A third method involves determining a grouping type based on a capture mode of the images, and generating a reference data structure associated with a first image and referencing a second image. The reference data structure comprises the grouping type. The reference data structure and the images are then embedded in the file.

Description

METHOD AND APPARATUS FOR ENCAPSULATING IMAGES IN A FILE

The present disclosure concerns a method and a device for encapsulating multiple images in a file.

Modern cameras provide different capture modes to capture images. Some of these capture modes result in capturing series of images. For example, they offer bracketing modes where several images are captured, the value of one parameter of the capture varying from one capture image to another. The parameter may be the exposure time, the value of the white, or the focus for example. The image burst mode provides the ability to take a series of images with no delay. It can be used to capture a fast event in sport for example. Panorama mode allows obtaining a series of overlapping images to reconstitute a large view of a scene.

Images captured by a camera are stored on a storage device like a memory card for example. The images are typically encoded to reduce the size of data on the storage device. Many encoding standard may be used, like JPEG or the more recent HEVC standard.

The HEVC standard defines a profile for the encoding of still images and describes specific tools for compressing single still images or bursts of still images. An extension of the ISO Base Media File Format (ISOBMFF) used for such kind of image data has been proposed for inclusion into the ISO/IEC 23008 standard, in Part 12, under the name: “HEIF or High Efficiency Image File Format”.

HEIF (High Efficiency Image File Format) is a standard developed by the Moving Picture Experts Group (MPEG) for storage and sharing of images and image sequences.

The MIAF (Multi-Image Application Format) is a standard developed by MPEG into ISO/IEC 23000 standard part 22 that defines a set of constraints on HEIF specification to simplify the format.

While providing limited grouping mechanisms, the HEIF and MIAF file formats do not provide grouping mechanism adapted to gather images resulting of a capture according to one of the cited capture modes.

The present invention has been devised to address one or more of the foregoing concerns. It concerns the extension of the grouping mechanisms in HEIF adapted to capture modes resulting in a plurality of captured images.

According to another aspect of the invention, there is provided a method of encapsulating images in a file, wherein the method comprises:

- determining a grouping type based on a capture mode of the images;

- generating a grouping data structure describing the images as a group, the grouping data structure comprising image identifiers of the images of the group;

- embedding the grouping data structure, the grouping type and the images in the file.

In an embodiment, the grouping type is comprised within the grouping data structure.

In an embodiment:

- the grouping data structure comprises a generic capture mode type; and

- the grouping type is comprised within a property data structure associated with the images of the group to characterize the generic capture mode type.

In an embodiment, the property data structure further comprises parameters associated with the grouping type.

In an embodiment:

the grouping data structure comprises a generic capture mode type; and the grouping type is comprised as an additional parameter within the grouping data structure.

In an embodiment, a property data structure associated with the grouping data structure is further embedded within the file.

- determining a label parameter describing information to annotate images; and

- embedding the grouping data structure, the label parameter and the images in the file.

- determining a grouping type based on a capture mode of the images;

- generating a reference data structure associated with a first image and referencing a second image, the reference data structure comprising the grouping type;

- embedding the reference data structure and the images in the file.

According to another aspect of the invention, there is provided a method of reading images in a file, wherein the method comprises:

- reading a grouping type based on a capture mode of the images;

- reading a grouping data structure describing the images as a group, the grouping data structure comprising image identifiers of the images of the group;

- reading the images identified in the grouping data structure according to the grouping type.

In an embodiment:

- the grouping data structure comprises a generic capture mode type; and

In an embodiment:

- the grouping data structure comprises a generic capture mode type; and

- the grouping type is comprised as an additional parameter within the grouping data structure.

- reading a label parameter describing information to annotate images; and

- reading the images identified in the grouping data structure according to the label parameter.

- reading a grouping type based on a capture mode of the images;

- reading a reference data structure associated with a first image and referencing a second image, the reference data structure comprising the grouping type;

- reading the first and second images in the file.

In an embodiment, data structures are boxes as defined in ISO Base Media File Format.

In an embodiment, a grouping type is defined for at least one of the capture modes comprising auto exposure bracketing mode, white balance bracketing mode, focus bracketing mode, flash exposure bracketing mode, depth of field bracketing mode, iso bracketing mode, time lapse mode, panorama mode, image burst mode, and user collection mode.

In an embodiment, the grouping data structure is one of an EntityToGroup box, a SampleToGroup box, or a TrackGroup box as defined in ISO Base Media File Format.

In an embodiment, the property data structure is one of a SampleEntry box, a SampleGroupEntry box, or a ItemProperty box as defined in ISO Base Media File Format.

According to another aspect of the invention, there is provided a device for encapsulating images in a file, wherein the device comprises circuitry configured for:

- determining a grouping type based on a capture mode of the images;

- determining a label parameter describing information to annotate images; and

- determining a grouping type based on a capture mode of the images;

- embedding the reference data structure and the images in the file.

According to another aspect of the invention, there is provided a device for reading images in a file, wherein the device comprises circuitry configured for:

- reading a grouping type based on a capture mode of the images;

- reading a label parameter describing information to annotate images; and

- reading a grouping type based on a capture mode of the images;

- reading the first and second images in the file.

According to another aspect of the invention, there is provided a computer program product for a programmable apparatus, the computer program product comprising a sequence of instructions for implementing a method according to the invention, when loaded into and executed by the programmable apparatus.

According to another aspect of the invention, there is provided a computerreadable storage medium storing instructions of a computer program for implementing a method according to the invention.

According to another aspect of the invention, there is provided a computer program which upon execution causes the method according to the invention to be performed.

According to another aspect of the invention, there is provided a method for encapsulating an encoded bitstream comprising entities (for example image items or tracks), the encapsulated bitstream comprising a data part and a metadata part. The method comprises:

providing in the metadata part, entity information identifying one or more entities, providing in the metadata part, a grouping information chosen among a plurality of grouping information, signaling that the one or more entities identified by the entity information, form a group based on a predetermined criteria indicated by said grouping information, each grouping information indicating a different predetermined criteria, outputting said bitstream together with said provided information as an encapsulated data file.

In an embodiment, the encoded bitstream comprises encoded captured images or encoded samples representing captured images.

In an embodiment, the entity information identifies at least two entities.

In an embodiment, the method further comprising capturing said images by a capturing device based on capture features, wherein the predetermined criteria is related to at least one of the capture features of the capturing step.

In an embodiment, the grouping information is a four-letter code.

According to another aspect of the invention, there is provided a method for encapsulating an encoded bitstream representing entities (for example image items or tracks), the encapsulated bitstream comprising a data part and a metadata part.

The method comprises:

Providing in the metadata part, entity information identifying one or more entities,

Providing in the metadata part, a grouping information signaling that the at least two entities identified by the entity information, form a group based on at least one common property,

Providing property information about said a least one common property,

Providing linking information for respectively linking each one of the one or more entities identified by the entity information to the common property, and

Outputting said bitstream together with said provided information as an encapsulated data file.

In an embodiment, entity information identifies at least two entities and the linking information links each one of the at least two entities identified by the entity information to the common property.

In an embodiment, the property information signals for said at least one common property, at least two different values, and the linking information links each one of the at least two entities identified by the entity information to at least two different values of the common property.

In an embodiment, the linking information links each one of the at least two entities identified by the entity information to the at least two different values of the common property.

In an embodiment, the method further comprising capturing said images by a capturing device based on capture features, wherein the common property is related to at least one of the capture features of the capturing step.

In an embodiment, the linking information is a parameter comprised in the grouping structure, said parameter taking different values for representing the properties.

In an embodiment, the linking information is a structure (for instance a box), associating property information to the group.

At least parts of the methods according to the invention may be computer implemented. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a circuit, module or system. Furthermore, the present invention may take the form of a computer program product embodied in any tangible medium of expression having computer usable program code embodied in the medium.

Since the present invention can be implemented in software, the present invention can be embodied as computer readable code for provision to a programmable apparatus on any suitable carrier medium. A tangible, nontransitory carrier medium may comprise a storage medium such as a floppy disk, a CD-ROM, a hard disk drive, a magnetic tape device or a solid state memory device and the like. A transient carrier medium may include a signal such as an electrical signal, an electronic signal, an optical signal, an acoustic signal, a magnetic signal or an electromagnetic signal, e.g. a microwave or RF signal.

Embodiments of the invention will now be described, by way of example only, and with reference to the following drawings in which:

Figure 1 illustrates an example of an HEIF file that contains several still images;

Figure 2 illustrates the main steps of a process for encapsulating a series of images in one file using HEIF format;

Figure 3 illustrates the main steps of a parsing process of and HEIF file generated by the encapsulating process of Figure 2;

Figure 4 illustrates an example according to a first embodiment of the invention including a grouping data structure;

Figure 5 is a schematic block diagram of a computing device for implementation of one or more embodiments of the invention.

The HEVC standard defines a profile for the encoding of still images and describes specific tools for compressing single still images or bursts of still images. An extension of the ISO Base Media File Format (ISOBMFF) used for such kind of image data has been proposed for inclusion into the ISO/IEC 23008 standard, in Part 12, under the name: “HEIF or High Efficiency Image File Format.

The HEIF and MIAF standards cover two forms of storage corresponding to different use cases:

• the storage of image sequences, with timing that is optionally used at the decoder, and in which the images may be dependent on other images, and • the storage of single images, and collections of independently coded images.

In the first case, the encapsulation is close to the encapsulation of the video tracks in the ISO Base Media File Format (see document « Information technology — Coding of audio-visual objects — Part 12: ISO base media file format», ISO/IEC 14496-12:2014, Fifth edition, Avril 2015), and the similar tools and concepts are used, such as the ‘trak’ boxes and the sample grouping for description. The ‘trak’ box is a file format box that contains sub boxes for describing a track, that is to say, a timed sequence of related samples.

Boxes, also called containers, are data structures provided to describe the data in the files. Boxes are object oriented building block for describing meta data in an image file.

In the second case, a set of ISOBMFF boxes, the ‘meta’ boxes are used. These boxes and their hierarchy offer less description tools than the ‘track related’ boxes (‘trak’ box hierarchy) and relate to “information items” or “items” instead of related samples. It is to be noted that the wording ‘box’ and the wording ‘container’ may be both used with the same meaning to refer to data structures that contain metadata describing the organization or/and properties of the image data in the file.

Figure 1 illustrates an example of an HEIF file 101 that contains several still images. This file contains a first 'moov' box 102 that describes several tracks

121 and 122. Typically, the track 121 is a 'pict' track designed to describe a set of pictures for which the temporal information is not necessarily meaningful and

122 is a 'vide' track designed to describe video content. Both these tracks describes a series of image samples, an image sample being a set of pixels captured at the same time, for example a frame of a video sequence. Main difference between the two tracks is that in 'pict' tracks the timing information is not necessarily meaningful whereas for 'vide' track the timing information is intended to constraint the timing of the display of the samples. The data corresponding to these samples is stored in the container for media data: the ‘mdat’ box 104.

In a second container, the HEIF file 101 describes several single images as different items 131 and 132 of the 'meta' box 103. The 'mdat' container 104 stores the encoded images corresponding to these items as represented by the data portion 141 and 142. The ‘item’ boxes aim at describing the organization of the encoded data in the ‘mdat’ box and providing some properties of the encoded images.

The purpose of HEIF file 101 is to describe the different alternatives available to store multiple samples in one HEIF file. Of course, there is no requirements to use 'pict' and 'video' tracks with items in the same HEIF file. For instance, we may store the multiple images either as items or as a track of samples that can be a ‘pict’ track or a ‘vide’ track. The actual choice is typically made by the application generating the file according to the type of images and the contemplated usage of the file.

Considering the storage issue by a camera when shooting images, several capture modes lead to the capture of a plurality of related images. These capture modes comprise, as an example, the bracketing mode. The bracketing consists in taking a series of images, each image of the series being captured using a different value of a camera parameter. Different kinds of bracketing are possible depending on the camera parameter used as the varying value for the capture. Bracketing based on the variation of the exposure is known as AE (Auto Exposure) bracketing. Some example of alternatives that may be used with HEIF to store two series of three shots in AE bracketing mode will be described, three shots being one example configuration. Each series comprises the capture of three images, a first one with a low exposure time, the second one with an intermediate exposure time, and a third one with a high exposure time.

According to a first example, several items may be used. The camera is setup to encapsulate each shot in one image item. In this example, the resulting HEIF file then contains six items.

According to a second example, several samples, namely the six images, are stored in one ‘pict’ track. The camera encapsulates each shot as a sample of a ‘pict’ track. In HEIF, the ‘pict’ track contains samples for which the temporal information is not necessarily relevant in contrary to ‘vide’ tracks.

According to a third example, the six images are stored as samples in a ‘vide’ track using ‘moov’/’moof’ boxes for storing the metadata describing the samples.

ISO Base Media File Format specifies several alternatives to group samples or items depending on the container that holds the samples or items to group. These alternatives can be considered as grouping data structures or grouping mechanism, i.e. boxes or data structures providing metadata describing a grouping criterion and/or group properties and/or group entities.

A first grouping mechanism is adapted for the grouping of items or tracks. In this mechanism, the wording ‘entity’ is used to refer to items or tracks. This mechanism specifies the grouping of entities. An EntityToGroupBox is defined according to the following syntax:

aligned(8) class EntityToGroupBox(grouping_type, version, flags) extends FullBox(grouping_type, version, flags) { unsigned int(32) group_id;

unsigned int(32) num entities in group; for(i=0; i<num_entities_in_group; i++) unsigned int(32) entity_id;

// the remaining data may be specified for a particular grouping_type }

The grouping_type is used to specify the type of grouping. Only two values of grouping_type are specified in HEIF. A first grouping_type ‘altr’ specifies that the different entities are alternatives that may alternatively be used in an application. A second grouping_type ‘ster’ specifies that two entities are a stereo pair, typically left and right views, in a stereoscopic application. No other grouping type of entities is specified. The groupjd is a unique identifier of the group of entities. Then, the list of entityjd gives all the entities pertaining to the group.

It is to be noted that this mechanism is very limited as there are only two types of groups specified. Moreover, nothing is specified to provide some further potential information on the group, especially for the ‘altr’ grouping_type, no indication is given on why an entity is an alternative to another entity.

A second grouping mechanism is specified for grouping samples of ‘pict’ or ‘vide’ tracks. This mechanism relies on two dedicated boxes. A first box named SampleToGroupBox allows the specification of the samples pertaining to one or more groups. A second box named SampleGroupDescriptionBox comprises one or more sample group entries for each sample group to describe properties of the group of samples. Their syntax is given by:

aligned(8) class SampleToGroupBox extends FullBox('sbgp', version, 0) {

unsigned int(32) grouping_type;

if (version == 1) { unsigned int(32) grouping_type_parameter;

} unsigned int(32) entry_count; for (i=l; i <= entry_count; i++)

{	unsigned int(32)	sample_count;
	unsigned int(32)	group_description_index;
}
}	// Sequence Entry

abstract class SampleGroupDescriptionEntry (unsigned int(32) grouping_type) {

} abstract class VisualSampleGroupEntry (unsigned int(32) grouping_type) extends SampleGroupDescriptionEntry (grouping_type) {

} abstract class AudioSampleGroupEntry (unsigned int(32) grouping_type) extends SampleGroupDescriptionEntry (grouping_type) {

} abstract class HintSampleGroupEntry (unsigned int(32) grouping_type) extends SampleGroupDescriptionEntry (grouping_type) {

} abstract class SubtitleSampleGroupEntry (unsigned int(32) grouping_type) extends SampleGroupDescriptionEntry (grouping_type) {

} abstract class TextSampleGroupEntry (unsigned int(32) grouping_type) extends SampleGroupDescriptionEntry (grouping_type) {

} aligned(8) class SampleGroupDescriptionBox (unsigned int(32) handler_type) extends FullBox('sgpd', version, 0){ unsigned int(32) grouping_type;

if (version>=l) { unsigned int(32) default_length; } if (version>=2) { unsigned int(32) default_group_description_index;

} unsigned int(32) entry_count;

int i;

for (i = 1 ; i <= entry_count ; i++){ if (version>=l) { if (default_length==0) { unsigned int(32) description_length;

} }

SampleGroupDescriptionEntry (grouping_type);

// an instance of a class derived from SampleGroupDescriptionEntry // that is appropriate and permitted for the media type } }

The SampleToGroupBox specifies a list of samples pertaining to the group. Each group of samples, and then each sample, is associated with a SampleGroupDescriptionEntry by a group_deschption_index. It may be noted that several SampleGroupDescriptionEntry are defined for different types of samples, which can be one of the following: visual, audio, hint, subtitle or text.

The SampleGroupDescriptionBox comprises the type of group (the grouping_type parameter) and for each type of group an instance of a box derived from SampleGroupDescriptionEntry to specify some properties associated to the group of samples. The specific SampleGroupDescriptionEntry can be considered as a data structure to describe or store one or more properties for group of samples.

Moreover, the properties are associated to each sample of the group in the SampleToGroupBox. No similar mechanism is provided to easily associate a property to all the items of a group of items. Entity grouping and sample grouping consist in associating a grouping type which identifies the reason of the grouping with a set of samples, items or tracks. In this document, it is referred to Grouping Information as information in one of the EntityToGroup, or SampleToGroup Boxes which convey information to group a set of images.

The HEIF standard provides some mechanisms designed to specify properties associated to images, in particular some data structures to declare or store properties for images and more generally for items (of any kind of media types).

Regarding items, inside the ItemPropertiesBox (‘iprp’), an ItemPropertyContainerBox is provided to store or describe properties of an item, for example an image item. The ItemPropertyAssociation box associates one or more properties with a given item. It is possible to associate a same property with several items. A list of items sharing the same properties may be seen as a kind of group described by the associated properties.

The associated syntax is as follow:

aligned(8) class ItemProperty(property_type) extends Box(property_type) {

} aligned(8) class ItemFullProperty(property_type, version, flags) extends FullBox(property_type, version, flags) }

aligned(8) class ItemPropertyContainerBox extends Box('ipco') {

properties ItemProperty() []; // boxes derived from // ItemProperty or ItemFullProperty, to fill box }

aligned(8) class ItemPropertyAssociation extends FullBox('ipma', version, flags) {

unsigned int(32) entry_count;

for(i = 0; i < entry_count; i++) { if (version < 1) unsigned int(16) item_ID;

else unsigned int(32) item_ID;

unsigned int(8) association_count;

for (i=0; i<association_count; i++) { bit(l) essential;

if (flags & 1) unsigned int(15) property_index;

else unsigned int(7) property_index;

} }

} aligned(8) class ItemPropertiesBox extends Box('iprp') {

ItemPropertyContainerBox property_container;

ItemPropertyAssociation association[];

The Item Property and ItemFullProperty boxes are designed for the description of a property. ItemFullProperty allows several versions of signalling of the syntax and contains one or more parameter present conditionally to the value of the flags parameter.

The ItemPropertyContainerBox is designed for describing a set of properties as an array of ItemProperty boxes.

The ItemPropertyAssociation box is designed to describe the association between items and their properties. It provides the description of a list of item identifiers, each item identifier (item_ID) being associated with a list of property index referring to a property in the ItemPropertyContainerBox.

Finally, the ItemPropertyContainerBox and the ItemPropertyAssociation boxes are gathered within an ItemPropertiesBox.

Regarding samples in tracks, a SampleEntry box is provided to describe properties of samples. Similarly, to the ItemProperty box, each sample then refers to a SampleEntry. Several samples may refer to a same SampleEntry box. All the samples sharing a same SampleEntry may be seen as forming a group, the semantic of which being described by the type of the SampleEntry. The wording Property information is used to refer to data described in one of the SampleEntry, SampleGroupEntry or ItemProperty or ItemFullProperty boxes.

The invention provides a mechanism that provides a mean for describing a group of images captured according to a given capture mode. It is provided a mean to describe the capture mode that has been used. According to some embodiments, some additional information regarding the capture may be described in relation with the group of images.

Figure 2 illustrates the main steps of a process for encapsulating a series of images in one file using HEIF format. A camera may for instance apply this processing. The given example applies to the grouping of images captured by a camera according to different capture modes.

First, the capture mode of the series of photo is determined in a step 201. The capture mode of the series of images describes the kind of relationship between the images of the series. For instance, the capture modes are one of the following:

Bracketing capture mode includes auto exposure, white balance, focus, flash bracketing modes. All these bracketing modes consist in performing several shots of the same content with different values of one or more parameters of the shooting. These different bracketing modes differ in the parameter whose value is varying in the series of capture. The capture system changes one capture parameter to generate the different versions. For example, in auto exposure bracketing the time of exposure is modified for each image capture.

Image burst is a capture mode consisting in capturing successively a series of images with a small interval of time between two image captures.

Panorama is a capture mode where several images are captured with an overlap between each capture. The principle is then to stitch each captured image to form a panorama of higher resolution.

Time-lapse is a capture mode consisting in capturing several images with the same device with a predetermined timing between each shot.

User collection, also called photo series, is a capture mode where a user associates images in a series that shares the same context. For instance, a photograph makes several photos of the same product and wants to store all the images he made in the same file. He starts the user collection capture mode at the beginning of the session. Once he finishes his shooting session, he stops the capture mode.

Once a capture mode has been selected in step 201, the processing loop composed of steps 202, 203, and 204 is applied. Until the end of the capture mode (for example by activating a specific options or buttons in the graphical or physical user interface of the device), the capturing device first captures an image in a step 203 and then encapsulates the encoded image in file format in a step 204. The capture step 203 includes the acquisition of the image and the encoding of the image in using a video or still picture codec. For instance, in this example, the codec is H.264 or HEVC/H.265 format.

For the storage of images, two main alternative possibilities are available.

In the first alternative, images are encoded independently and stored in the file as HEIF items. During this encapsulation step, additional information on the condition of capture may be provided in the file. For example, for auto exposure bracketing mode, the exposure data used for the capture of each image may be provided. This description is provided using properties in an Item Property box.

In a second alternative, images are stored in a ‘pict’ or ‘vide’ track. Additional information may be provided using SampleEntry or SampleGroupEntry boxes. The encoding of images may depend on previous images using an HEVC encoder similarly to video encoding. Previous images in the track are available as reference image for predictive encoding.

In a step 205, the encapsulation of the series of images is finalized. During this step, the storage of the captured images as a group of images is signalled in the file using one of the available grouping mechanism previously described. The type of the capture mode may be signalled in the grouping mechanism or alternatively in additional information stored for each image during step 204.

In order to be able to perform the storage of images as described in Figure 2, new types of grouping are necessary along with new methods to signal these types of grouping in a file with associated properties.

One grouping type per capture mode.

According to a first embodiment, a grouping type is defined for each capture mode. This grouping type is described within the grouping information.

Figure 4 illustrates an example according to this first embodiment.

Images are described as items 404, 405 and 406 in an ItemlnfoBox 403.

A grouping information box, for example an EntityToGroupBox 401 comprises the grouping type based on the capture mode. Then the grouping information box 401 comprises a list 402 of item identifiers of the item pertaining to the group.

For example, a generic ‘brak’ grouping type may be defined for bracketing capture mode (the ‘brak’ code here is an example). The particular type of bracketing mode, namely auto exposure bracketing, white balance bracketing, focus bracketing, flash exposure bracketing, depth of field bracketing and iso bracketing, may be signalled using a parameter of the ‘brak’ grouping type. For instance, with grouping of entities mechanism, the EntityToGroup contains a bracketing_type parameter which determines the type of the bracketing. Typically, the bracketing_type parameter is a reserved four-character code. One value of four-character code is predetermined for each bracketing type. In another example, sample grouping mechanism is used and the parameter of the ‘brak’ grouping type is described in a specific kind of (inheriting from) VisualSampleGroupEntry .

Alternatively, a specific grouping type may be defined for each particular bracketing capture mode: ‘aebr’ for auto exposure bracketing, ‘wbbr’ for white balance bracketing, ‘fobr’ for focus bracketing, ‘afbr’ for flash exposure bracketing, ‘dobr’ for depth of field bracketing and ‘isbr’ for iso bracketing.

A ‘tila’ grouping type may be defined for the time lapse capture mode. This type of grouping covers all the usages where several photos are taken, and where time is important. The main usage is for time lapse, with or without regular interval between photos. The parameters of the time lapse (e.g. the total duration of the time laps, the offset between each images) may be signaled using a parameter of the grouping (e.g.: as EntityToGroup or VisualSampleGroupEntry parameter).

A ‘pano’ grouping type may be defined for panorama grouping type. This includes all the usages where several photos are taken, and where the spatial dimension is important. This includes panoramas, but also different views of a given item. The parameters of the panorama (e.g. the capture pattern for instance from left to right, right to left, z-scan or spiral scan) may be signaled using a parameter of the grouping (e.g.: as EntityToGroup or VisualSampleGroupEntry parameter).

An ‘imbu’ grouping type may be defined for image burst grouping type. This covers all the cases when several images are captured successively without interruption. The parameters of the image burst (e.g. the time offset between each capture and the total duration of the capture) may be signaled using a parameter of the grouping (e.g.: as EntityToGroup or VisualSampleGroupEntry parameter).

An ‘ucol’, ‘pser’, ‘labl’ or ‘udcs’ grouping type may be defined for user collection or photo series or a set of photos with the same tag(s) or label(s) or user defined capture series capture mode. An automatic process may for instance groups the photo shot in the same location or/and within a predetermined interval of time. For instance, the capture device uses the location information (for instance from a GPS sensor) of the capture image to determine the name of the town corresponding to the location. All the images taken in the same town form one group of images. The parameters of the user collection (e.g. the location of the capture or a description of the content) may be signaled using a parameter of the grouping (e.g.: as EntityToGroup or VisualSampleGroupEntry parameter).

For example, a photograph makes two photos using AE bracketing. It results in 3x2 samples. Each photo is embedded in one image item.

The HEIF File thus contains six items as described below:

• Item #1: ltem_ld=1; // captured with short exposure time e.g. exposure_stop=-2 • Item #2: ltem_ld=2; // captured with medium exposure time e.g. exposure_stop=0 • Item #3: ltem_ld=3; // captured with long exposure time e.g. exposure_stop=2 • Item #4: ltem_ld=4; // captured with short exposure time e.g. exposure_stop=-2 • Item #5: ltem_ld=5; // captured with medium exposure time e.g. exposure_stop=0 • Item #6: ltem_ld=6; // captured with long exposure time e.g. exposure_stop=2

An EntityToGroupBox with 'aebr' grouping type value is defined. There are two entries (one for each AE series of three samples):

GoupsListBox{

EntityToGroupBox('aebr') { group_id='1234' // Identifier of the first bracketing set num_entities_in_group=3 // bracketing set size equal to 3 entity_id=l // refer the 1st item entity_id=2 // refer the 2nd item entity_id=3 // refer the 3rd item }

EntityToGroupBox('aebr') { group_id='1235' // Identifier of the second bracketing set num_entities_in_group=3 // bracketing set size equal to 3 entity_id=4 // refer the 1st item of the 2nd set i.e. 4th item entity_id=5 // refer the 1st item of the 3rd set i.e. 5th item entity_id=6 // refer the 1st item of the 4th set i.e. 6th item }

}

Generic capture series grouping type.

In a second embodiment, a generic grouping type ‘case’ for “capture series” is defined (the four character code ‘case’ being one example, any reserved non already used four character code might be used). This grouping type is intended to be used for all the capture modes previously described. When the images of the series are described as items, the Grouping Information is an EntityToGroup information box with a grouping_type parameter equal to 'case'. When, the images of the series are described as samples in a track (either 'vide' or 'pict' tracks), the Grouping Information is for instance a SampleToGroup and SampleGroupDeschptionBox information box with a grouping_type equal to 'case'.

When the HEIF contains several images with different capture modes, they all belong to the same 'case' group. When a series of items or samples belongs to a capture series group, the encapsulation shall specify the 'case' grouping type. To distinguish between the different capture, the type of the capture mode may be signalled in the Property Information. For instance, SampleEntry, SampleGroupDescriptionEntry, Item Property or ItemFullProperty may be used to precise the capture mode or/and parameter related to the series for some items of the group.

For example, the images being stored as items, an ItemPropertyContainerBox contains a set of Item Property boxes that describe the capture mode for a series of images. For example, one property describes an auto exposure bracketing capture mode, while another property describes the actual exposure value used for the capture.

In another embodiment, the Property Information is specified as a VisualSampleEntry information. In this embodiment, a SampleToChunkBox 'stsc' is present in the moov/trak/stbl box hierarchy of the multi-image file recorded on the camera. A chunk is a contiguous set of samples. The SampleToChunkBox::sample_deschption_index provides the index in the 'stsd' box of the SampleEntry applying to the samples of this chunk. A specific visual sample entry is defined containing an extra box compared to usual visual sample entries (for example of type 'hvcT or'avcT as in ISO/IEC 14496-15). This extra box provides the parameters for the grouping type, for example description of the bracketing mode, or a tag or label provided by the user (or any specific property as suggested by the non-limitative list of this invention).

Regarding the auto exposure bracketing capture mode, in a first embodiment, the value of the exposure is described in property information as an ItemProperty or a VisualSampleEntry as follows:

1. Syntax for Item Property aligned(8) class AutoExposureBracketingProperty extends ItemProperty('aebr') { int (8) exposure_stop;

2. Syntax for Sample Entry

Box type: 'aebr'

Container: VisualSampleEntry

Mandatory: No

Quantity: Zero or one aligned(8) class AutoExposureBracketing extends Box ('aebr') { int (8) exposure_stop;

}

3. Syntax for Sample Group Entry aligned(8) class AutoExposureBracketingEntry extends VisualSampleGroupEntry('aebr') { int (8) exposure_stop }

4. Semantic • exposure_stop is an integer value specifying the EV stop of the item.

According to another embodiment, the exposure value of each image is computed from the information specified in the property information, typically an Item Property. For instance, the syntax may be:

aligned(8) class AutoExposureBracketingProperty extends ItemProperty('aebr') { int (8) exposure_step; /*full=l, half=2, third=3, quarter=4*/ int (8) exposure_stop;

} where exposure_step is an integer value that specifies the increment steps used for the exposure bracketing. For example, when equal to 1, full stop increment is used, when equal to 2, half stop increment is used, when equal to 3, third stop increment is used, and when equal to 4, a quarter stop increment is used. The exposure_stop parameter is the maximum stop value.

As a result, the first image in the order of declaration in the Grouping Information is the one with the lowest exposure time. The number of images in the Grouping Information is odd. The exposure stop of this image is equal to the result of the formula: -1 *exposure_stop/exposure_step.

The last image has an exposure stop equal to exposure_stop/exposure_step. The image with index equal to the number of images in the group divided by two (i.e. the image at the middle position) has an exposure_stop equal to 0. When the number of images is greater than three, the exposure_stop interval [-exposure_stop/exposure_step, exposure_stop/exposure_step] is equally distributed among all the images described in the group.

According to yet another embodiment, the order of the images in the bracketing set (the order of description in the Grouping Information i.e. in the EntityToGroup or in the SampleToGroup boxes) is different than the natural order (ascending order). The Property Information includes a new parameter that describes the order of the bracketing set. The syntax may be as follows:

aligned(8) class AutoExposureBracketingProperty extends ItemProperty('aebr') { int (8) exposure_order;

int (8) exposure_step;

int (8) exposure_stop;

}

The new exposure_order parameter specifies the order of the image in the bracketing set:

- equal to 0, the ordering is by ascending order of exposure_stop in the exposure_stop interval. For example, the order is {-2EV, 0EV, 2EV}

- equal to 1, the ordering is by descending order of exposure_stop in the exposure_stop interval. For example, the order is {2EV, 0EV, 2EV}

- equal to 2, the ordering start first by the image of exposure_stop equal to 0EV, followed by the other images in ascending order. For example, the order is {0EV, -2EV, 2EV}

- equal to 3, the ordering start first by the image of exposure_stop equal to 0EV, followed by the other images in descending order. For example, the order is {0EV, 2EV, -2EV}

For White Balance Bracketing the Property Information signals the color temperature of the image in either ItemProperty, ItemFullProperty or SampleEntry or SampleGroupEntry. The syntax may be as follows:

1. Syntax for Item Entry aligned(8) class WhiteBalanceBracketingProperty extends ItemProperty('wbbr') { unsigned int (16) color_temperature }

2. Syntax for Sample Entry

Box type: 'wbbr'

Container: VisualSampleEntry

Mandatory: No

Quantity: Zero or one aligned(8) class WhiteBalanceBracketingEntry extends Box ('wbbr') { unsigned int (32) color_temperature }

3. Syntax for Sample Group Entry aligned(8) class WhiteBalanceBracketing extends VisualSampleGroupEntry('wbbr') { unsigned int (32) color_temperature }

4. Semantic

- color_temperature represents the color temperature value, for example in Kelvin. In another alternative, the color temperature is computed from the color_temperature. For instance, each value of the parameter corresponds to a predetermined color temperature. For instance, the color temperature in Kelvin is equal to color_temperature*10+2200.

For Focus Bracketing the Property Information signals the focus distance used when capturing the image in either ItemProperty, ItemFullProperty or SampleEntry or SampleGroupEntry. The syntax may be as follows:

1. Syntax for ItemProperty aligned(8) class FocusBracketingltemProperty extends ItemProperty('fobr') { usigned int (32) focus_distance_numerator usigned int (32) focus_distance_denominator }

2. Syntax for Sample Entry

Box type: 'fobr'

Container: VisualSampleEntry

Mandatory: No

Quantity: Zero or one aligned(8) class FocusBracketing extends Box ('fobr') { usigned int (32) focus_distance_numerator usigned int (32) focus_distance_denominator }

3. Semantic

The focus distance is expressed for instance in meter as the ratio between focus_distance_numerator and focus_distance_denominator fields. In another embodiment, these two parameters are replaced by a focus_distance parameter which expresses the distance in millimeters.

For Image burst capture mode, the Property Information signals the index of the capture in either ItemProperty, ItemFullProperty or SampleEntry or SampleGroupEntry boxes. Below is a possible syntax of the Property Information as an Item Property. The syntax of the box for VisualSampleEntry and SampleGroupEntry contains the same parameters and defines the same four character codes. The difference is that, for VisualSampleEntry, the new class extends the Box class and shall be defined in one VisualSampleEntry container.

For SampleGroupEntry, the new class extends the VisualSampleGroupEntry box with same four character coded as defined for the Item Property.

In a first embodiment the syntax may be:

aligned(8) class ImageBurstProperty extends ItemProperty('imbu') { unsigned int (32) burst_index }

with the semantic:

• burstjndex represents the index of the item in the burst of images.

In a second embodiment, the index of the images in the burst is inferred from the order of declaration of the items. This reduces the number of properties to describe and thus the HEIF file is more compact. Below is the syntax of the Property Information as an Item Property. The syntax of the box for VisualSampleEntry and SampleGroupEntry contains the same parameters and defines the same four character codes. The difference is that for VisualSampleEntry, the new class extends the Box class and shall be defined in one VisualSampleEntry container. For SampleGroupEntry, the new class extends the VisualSampleGroupEntry box with same four character codes as defined for the Item Property.

The syntax may be:

aligned(8) class ImageBurstProperty extends ItemProperty('imbu') { unsigned int (1) infer_index_value;

if (infer_index_value == 0) unsigned int (32) burst_index }

with the following semantics:

• infer_index_value is a boolean value which indicates that the index of the item in the burst of image is equal to the position index of the item in the 'iinf' box.

• burstjndex represents the index of the item in the burst of images. In another embodiment, the inferjndex_value indicates that the index of the item in the burst of images is equal to the index of the item in the

Grouping Information. This embodiment advantageously permits to define several orderings (e.g. a normal and reverse order) by defining two groups.

In another embodiment, the ImageBurst Propertyinformation includes the timing interval between two images of the burst as an unsigned integer value expressed in milliseconds.

The Photo Series or User Collection or Property Information signals the index of the captured image in set used when capturing the image in either ItemProperty, ItemFullProperty or SampleEntry or SampleGroupEntry. Below is the syntax of the Property Information as an Item Property. The syntax of the box for VisualSampleEntry and SampleGroupEntry contains the same parameters and defines the same four character codes. The difference is that for VisualSampleEntry, the new class extends the Box class and shall be defined in one VisualSampleEntry container. For SampleGroupEntry, the new class extends the VisualSampleGroupEntry box with same four character codes as defined for the Item Property.

The syntax may be:

aligned(8) class PhotoSeriesProperty extends ItemProperty('pser') { unsigned int (32) seriesjndex }

with the following semantics:

- seriesjndex represents the index of the item in the burst of images.

In another embodiment, the index of the photo series is inferred as done for Image Burst Property Information, thus saving description bytes.

The Flash Exposure Bracketing Information signals the flash exposure value of the captured image in set used when capturing the image in either ItemProperty, ItemFullProperty or SampleEntry or SampleGroupEntry. Below is the syntax of the Property Information as an Item Property. The syntax of the box for VisualSampleEntry and SampleGroupEntry contains the same parameters and defines the same four character codes. The difference is that for

VisualSampleEntry, the new class extends the Box class and shall be defined in one VisualSampleEntry container. For SampleGroupEntry, the new class extends the VisualSampleGroupEntry box with same four character codes as defined for the Item Property.

The syntax may be:

aligned(8) class FlashExposureBracketingProperty extends ItemProperty('afbr') { int (8) flash_exposure_numerator;

int (8) flash_exposure_denominator;

} with the following semantics:

flash_exposure_numerator and flash_exposure_denominator are integers representing the flash exposure value of the item or sample expressed in a number of f-stops as the ratio of flash_exposure_numerator and flash_exposure_denom inator.

The Depth of Field Bracketing Information signals the aperture change value of the captured image in set used when capturing the image in either ItemProperty, ItemFullProperty or SampleEntry or SampleGroupEntry. Below is the syntax of the Property Information as an Item Property. The syntax of the box for VisualSampleEntry and SampleGroupEntry contains the same parameters and defines the same four character codes. The difference is that for VisualSampleEntry, the new class extends the Box class and shall be defined in one VisualSampleEntry container. For SampleGroupEntry, the new class extends the VisualSampleGroupEntry box with same four character codes as defined for the Item Property.

The syntax may be:

aligned(8) class DepthOfFieldBracketingProperty extends ItemProperty('dobr') { int (8) f_stop_numerator;

int (8) f_stop_denominator;

with the following semantics:

where the aperture change is expressed in a number off-stops as the ratio of f_stop_numerator and f_stop_denominator which are integer values.

In the following example, two different captures have been done using auto exposure bracketing capture mode. Each capture results in capturing three different images. The two captures result in six images. Each image being embedded in one item.

The file structure would be:

• Item #1: ltemld=1; exposure_stop=-2 // exposure=low • Item #2: ltemld=2; exposure_stop=0 // exposure=mid • Item #3: ltemld=3; exposure_stop=2 11 exposure=high • Item #4: ltemld=4; exposure_stop=-1 //exposure=low • Item #5: ltemld=5; exposure_stop=011 exposure=mid • Item #6: ltemld=6; exposure_stop=-1 //exposure=high

An EntityToGroupBox with 'case' grouping type value is defined. There are two entries (one for each AE series of 3 samples):

GoupsListBox{

EntityToGroupBox('case') { group_id='1234' // 1^st bracketing set num_entities_in_group=3 entity_id=l // Refer to item #1 entity_id=2 // Refer to item #2 entity_id=3// Refer to item #3 }

EntityToGroupBox('case') { group_id='1235' // 2^nd bracketing set num_entities_in_group=3 entity_id=4 // Refer to item #4 entity_id=5// Refer to item #5 entity_id=6// Refer to item #6

ItemPropertyContainerBox { // List all the bracketing properties used by all the items AutoExposureBracketingProperty( ^faebr·’) { // property_index=l exposure_stop = -2 }

AutoExposureBracketingProperty( ^faebr·’) { // property_index=2 exposure_stop = -1 }

AutoExposureBracketingProperty( ^faebr·’) { // property_index=3 exposure_stop = 0 }

AutoExposureBracketingProperty( ^faebr·’) { // property_index=4 exposure_stop = 1 }

AutoExposureBracketingProperty( ^faebr·’) { // property_index=5 exposure_stop = 2 }

}

ItemPropertyAssociation{ // Associate each item with properties describing the exposure entry_count =6//6 items // Item #1 item_id = 1 association_count = 1 essential =0// The property is not essential property_index =1// Exposure in property of property_index=l // Item #2 item_id = 2 association_count = 1 essential = 0 property_index = 3// Exposure in property of property_index=3 // Item #3 item_id = 3 association_count = 1 essential = 0 property_index =5// Exposure in property of property_index=5 // Item #4 item_id = 4 association_count = 1 essential = 0 property_index =2// Exposure in property of property_index=2 // Item #5 item_id = 5 association_count = 1 essential = 0 property_index = 3// Exposure in property of property_index=3 // Item #6 item_id = 6 association_count = 1 essential = 0 property_index = 4// Exposure in property of property_index=4 }

In another embodiment, the camera is configured to group the images of a shot consisting in a series of three images at different exposure levels. In this example, the camera specifies the Property Information in a VisualSampleEntry, i.e. metadata describing one or more samples of a track. In the resulting multiimage file, a chunk will contain a number of samples equal to the number of images, three in the example, taken during the shot. A new specific VisualSampleEntry and corresponding chunk of three samples, according to the example, are created in the ‘stbl’ box of the recorded multi-image file. The

SampleToChunkBox::sample_description_index for this new shot is set equal to the index of the specific VisualSampleEntry in the 'stbl' box. At the end of the shooting sequence, the user gets a multi-image file made up of one track containing and describing all the pictures. Through the camera Ul, he can see the sequences of shots as a series of three images.

In another example, the images organized in a different ways may be accessed per exposure level. If the camera is configured to group the images per exposure level, then each chunk contains one image. One specific visual sample entry is declared per exposure level. If N specific VisualSampleEnthes are declared and if the camera always captures the pictures in the same order (e.g. under exposed, normal, over exposed), then the SampleToChunkBox::sample_description_index can be set equal to (picture index in the shot) modulo N. At the end of the shooting sequence, the user gets a multi-image file made up of one track containing and describing all the pictures. Through the camera Ul, the recorded pictures may be seen organized according to their exposure level. To describe the samples per exposure level, the use of specific sample groups as described in previous embodiments is preferred, especially the use of CompactSampleGroup that relies on patterns of samples. It is the case here, when the exposure levels occur in the same order. A pattern is identified and the sample group is defined as a pattern of samples, each sample group being associated to a sample group entry in the corresponding SampleGroupDescriptionBox. The use of classical sample groups, i.e. without patterns, is also possible and is preferred over VisualSampleEntry since changing the sample entry might lead to decoder reset in some implementations, which is not the case with sample groups.

Generic capture series grouping type associated with properties

This third embodiment is a variant of the second embodiment regarding generic capture series grouping type.

In this variant, additional parameters are defined for the generic capturetime grouping type (e.g. noted 'case' for capture series) to provide parameters that are common to all items in the group (e.g. exposure, white-balance, panorama...) and to link the group with item properties that precise parameters specific to an item within the group. In particular, a 'capture_mode' parameter signals the four characters code that identifies the Property Information (Property’s property_type) that further describes the parameters specific to an item (for example the images described as image items). The principle is that one HEIF parser may determine the type of Property Information (i.e. the type of the ItemProperty or SampleGroupDeschptionBox or SampleEntry) from the 'capture_mode' parameter of the generic capture-time grouping type.

The principle is that one HEIF reader may easily retrieve the Item Property among all Item Properties associated to a given group of items.

The syntax of the Grouping Information when described as EntityToGroup is described below:

aligned(8) class EntityToGroupBox('case', version, flags) extends FullBox('case', version, flags) { unsigned int(32) group_id;

unsigned int(32) num_entities_in_group; for(i=0; i<num_entities_in_group; i++) unsigned int(32) entity_id;

// Parameters below provide common parameters for the grouping_type 'case' unsigned int(32) capture_mode; // 4CC identifying the capture mode }

where capture_mode identifies the capture mode. Item properties with same four characters code provide item-specific parameters for a given capture_mode.

For each item in the group, an item property with the same 4CC as the 'capture_mode' defines the item-specific parameters for this 'capture_mode'.

e.g.: The following 4CC codes can be defined for capture_mode:

Capture mode type	4cc code
Auto Exposure bracketing	aebr
White balance bracketing	wbbr
Focus bracketing	fobr
Flash Exposure bracketing	afbr
Depth of field bracketing	dobr
Panorama	pano
User-defined capture series	udcs

Examples of Item Property to define for above capture_mode.

aligned(8) class AutoExposureBracketingProperty extends ItemProperty('aebr') { int (8) exposure_step;

int (8) exposure_numerator }

where exposure_step is an integer value that specifies the increment steps used for the exposure bracketing. When equals to 1, full stop increment is used, when equals to 2, half stop increment is used, when equals to 3, third stop increment is used, and when equals to 4, a quarter stop increment is used. exposure_numerator is an integer value specifying the exposure numerator used to compute the exposure value stop of the item.

The exposure value (EV) stop of the associated item is then equal to the result of the following formula: exposure_numerator/exposure_step.

aligned(8) class WhiteBalanceBracketingProperty extends ItemProperty('wbbr') { unsigned int (32) color_temperature }

where color_temperature represents the color temperature value in Kelvin.

aligned(8) class FocusBracketingProperty extends ItemProperty('fobr') { usigned int (32) focus_distance_numerator usigned int (32) focus_distance_denominator }

where the focus distance is expressed in meter as the ratio of focus_distance_numerator and focus_distance_denominator.

int (8) flash_exposure_denominator;

} where the flash exposure compensation is expressed in a number of fstops as the ratio of flash_exposure_numerator and flash_exposure_denom inator.

int (8) f_stop_denominator;

} where the aperture change is expressed in a number off-stops as the ratio of f_stop_numerator and f_stop_denominator.

aligned(8) class PanoramaBracketingProperty extends ItemProperty('pano') { usigned int (1) panorama_direction unsigned int(15) frame_number;

} where panorama_direction is a 1 -bit flag signaling the panorama direction (0: left-to-right, 1: right-to-left) and frame_number an integer representing the frame number according to the panorama direction (lower to higher).

aligned(8) class UserDefinedCaptureSeriesProperty extends ItemProperty('udcs') { utf8string label }

where label a null-terminated string in UTF-8 characters which gives a human-readable name for the user-defined capture series.

An example of this embodiment of a capture using auto exposure bracketing mode resulting in three images stored as items.

The file structure may be:

Item	#1:	Itemld=l;	exposure_stop=-2 //	exposure=low
Item	#2:	Itemld=2;	exposure_stop=0 //	exposure=mid
Item	#3:	Itemld=3;	exposure_stop=2 //	exposure=high

An EntityToGroupBox with 'case' grouping type value and 'aebr' capture mode is defined:

GroupsListBox{

EntityToGroupBox('case') { group_id='1234' num_entities_in_group=3 entity_id=l entity_id=2 entity_id=3 capture_mode='aebr' }

}

ItemPropertyContainerBox {

AutoExposureBracketingProperty( “'aebr·’) { exposure_step = 1;

exposure_numerator = -2;

}

AutoExposureBracketingProperty( ^faebr·’) { exposure_step = 1; exposure_numerator = 0;

}

AutoExposureBracketingProperty( ^faebr·’) { exposure_step = 1;

exposure_numerator = 2 }

}

ItemPropertyAssociation{ entry_count = 3 item_id = 1 association_count = 1 essential = 0 property_index = 1 item_id = 2 association_count = 1 essential = 0 property_index = 2 item_id = 3 association_count = 1 essential = 0 property_index = 3 }

Groups of images can also be encapsulated into one or more ‘pict’ tracks. Each image can be a sync sample (all intra) or may use inter prediction.

For example, for images sequences the principle is to rely on sample grouping and to define a dedicated grouping_type and sample group description for each kind of capture modes. For example, ‘aebr’ indicates group of image samples for an automatic exposure bracketing. A corresponding VisualSampleGroupEntry would be defined as follows:

aligned(8) class AutoExposureBracketingEntry extends VisualSampleGroupEntry('aebr') {

int (8) exposure_order;

int (8) exposure_step;

int (8) exposure_stop;

}

where exposure_order parameter specifies the order of the image in the bracketing set. exposure_step is an integer value that specifies the increment steps used for the exposure bracketing. When equals to 1, full stop increment is used, when equals to 2, half stop increment is used, when equals to 3, third stop increment is used, and when equals to 4, a quarter stop increment is used. exposure_stop is an integer value specifying the maximum exposure value (EV) stop of the sample.

The exposure stop of the associated image is then equal to the result of the following formula: -1*exposure_stop/exposure_step.

Similar definitions can be defined for all other capture modes

In case the property related to the capture series does not vary over time, default sample grouping mechanism can be used or Item Property can be defined in VisualSampleEntry (as for ‘pasp’, ‘coir’ or ‘clap’ Item Properties).

In an alternative of this embodiment, the capture mode refers to a Propertyinformation described in Proprietary box. As a result, the capture_mode parameter refers to a four characters code of a Proprietary box (e.g. a 'uuid' box). In such case, the Grouping Information further describes data that identifies precisely which proprietary information pertains to the images in the 'case' group. For instance, when capture_mode is equal to 'uuid' the Property Information specifies the usertype field of the box of the 'uuid' type. A HEIF reader could advantageously determine the Proprietary box that further describe the capture mode. This embodiment applies also to any other Proprietary box extension system that includes a four characters code (generally referred as sub_type or proprietary_type) that permits to distinguish the different Proprietary boxes.

In another alternative of this embodiment, the grouping type listed in the first embodiment may further specify Property Information with a four characters code equal to the grouping type. For instance, the capture device declares an Auto Exposure Bracketing group with grouping_type equal to 'aebr' and associates an ItemProperties 'aebr' with each Item pertaining to the group.

For instance, for image items the principle is to define several specific grouping types for EntityToGroup corresponding to the different grouping purposes (capture modes). The same 4CC is used for the grouping_type of a given EntityToGroup and for the Item Property that provides the parameters specific to an item within the corresponding group. For example, the parameters of items pertaining to an EntityToGroup('aebr') are provided by ItemProperty('aebr') where ‘aebr’ is the FourCC for auto-exposure bracketing set.

VisualSampleEntry

In a fourth embodiment, the information common to all the images in the capture series is stored in a new box in VisualSampleEntry of the ‘pict’ or ‘vide’ track. This new box signals that the samples of the track associated to this sample entry pertains to same capture mode (e.g. bracketing set, image burst, photo series...). The sample-specific information, (described for example as Property Information in previous embodiments) are stored in a dedicated metadata track. For example, the VisualSampleEntry indicates that the capture mode is auto exposure bracketing. The parameters of the bracketing (i.e. exposure configuration) are described in the samples of the metadata track.

The capture device may associate the track describing the image samples with one or more metadata tracks describing the sample-specific information via a track reference of type 'cdsc'.

New item property and sample entry for tagging.

In a fifth embodiment, the Property Information describes information to annotate a series of images. This Property Information includes a label parameter as a null-terminated string in UTF-8 characters that gives a human-readable name for the tag. In one alternative, this property is declared as other item properties in the 'ipco' box and can be associated to items in 'ipma' box. For instance, the table below describes the syntax of the ItemProperty (or ItemFullProperty) that contains the label parameter. The camera device fills the content of this label parameter with a string of UTF-8 characters typically defined by the user. In one alternative, the label contains information describing the context of the shooting. For instance, the user may enter a string or the capture device may generate a string that contains the date and the location of the capture. For example, this string may contain Captured in May 2013 at Paris, Visit of Eiffel Tower. In particular, the capture device may use image recognition program available on the device to automatically generate the label.

aligned(8) class Label extends ItemProperty('labl') { utf8string label }

In one alternative, the Property Information is described as one SampleEntry or SampleGroupEntry for a series of images described in one ‘trak’ container. The syntax of the Property Information in this context is the following. In such a case, the syntax is close to ItemProperty syntax as represented below.

Box type: 'aebr'

Container: VisualSampleEntry

Mandatory: No

Quantity: Zero or one aligned(8) class Label extends Box('labl') { utf8string label }

As a result, the label is an UTF-8 string describing the content of a sample in a track.

In another embodiment, the Grouping Information indicates that a series of images are related (without precise purpose) through a specific string. For Entity grouping, the grouping type is set equal to 'labl' or'tag' four character code.

In another alternative, the tag or label property can also be associated to a group of items with a different grouping type than 'labl' or 'tag '.

In another embodiment, the Photo Series or User Collection Property information includes a label parameter as an additional parameter.

The syntax is for example as follows: aligned(8) class UserDefinedCaptureSeriesProperty extends ItemProperty('udcs') { utf8string label }

Item Property association to refer to a group.

In a sixth embodiment, the Property Information applies to several images (i.e. either items or samples) which are gathered in one group by the Grouping Information (for instance, through signaling in one EntityToGroup or SampleToGroup box). In such a case, the Property Information is associated with one group instead of one image. For example, the ItemPropertyAssociation box allows referring to an identifier of group (for example to the EntityToGroup::group_id parameter) that groups a series of items. The itemJD field of the ‘ipma‘ box is then replaced by an entityjd which may refer to either an identifier of Item (itemJD) or to a group identifier (groupjd). For instance, the Grouping Information is an EntityToGroup. The identifier that one ItemPropertyAssociation box may refer to is the groupjd field. The advantage of this embodiment is that the description of the properties of each image is more compact since repetition of associations is avoided.

The Table below is the new syntax of the ItemPropertyAssociation box: aligned(8) class ItemPropertyAssociation extends FullBox('ipma', version, flags) {

unsigned int(32) entry_count;

for(i = 0; i < entry_count; i++) { if (version < 1) unsigned int(16) entity_ID;

else unsigned int(32) entity_ID;

unsigned int(8) association_count;

for (i=0; i<association_count; i++) { bit(l) essential;

if (flags & 1) unsigned int(15) property_index; else unsigned int(7) property_index;

} }

}

The semantics of the different parameters of ItemPropertyAssociation remain unchanged except item_ID field which is renamed to entityJD. The entity_ID parameter may refer either to one item or to one EntityToGroup. Thus, the value of entity_ID should be equal to one groupjd value or to one item_ID value.

For example, the capture device employs the user collection capture mode and describes each image as one Item. For instance, the capture device stores four images. The Property Information shared with each Item corresponds to the label set by the photographer as in previous embodiment. The HEIF boxes describes thus four Items (one for each image of the user collection) • Item #1: Item ld=1;

• Item #2: ltemld=2;

• Item #3: ltemld=3;

• Item #4: ltemld=4;

The Grouping Information is described as an EntityToGroupBox with 'case' grouping type value, which gathers the set of four items. Then, a label is described which is common to all items. This corresponding property, identified by the ‘labl’ code, is for example, a label set by the user such as a product reference or a predetermined value for instance the date of capture of the first image in the set. The association of the property with the item is described in one ItemPropertyAssociation box. In this embodiment, the Item Property Association box contains only a single entry for the four items. The HEIF writer associates the property with the identifier of the group defined in the 'case' group defined in the EntityToGroup box. This mutualisation can be applied to other properties, for example the sizes (‘ispe’) or the decoding configuration (‘hvcC’). The table is a pseudo-code of the description of the HEIF file for this example:

GroupsListBox{

EntityToGroupBox('case') { group_id='1234' num_entities_in_group=4 entity_id=l entity_id=2 entity_id=3 entity_id=4 }

}

ItemPropertyContainerBox { ^fispe^J {1920 1080} ^fhvcC·’ {...} ^flabl^J { label = Product ref. 32184103

ItemPropertyAssociation{ entry_count = 1 entity_id = 1234 association_count = 3 essential = 0, property_index=l; // sizes essential = 1, property_index=2; // decoder configuration essential = 0, property_index =3// label }

A writer may keep on associating sizes and decoder configuration on an image basis and only share the property specific to the grouping type (the ‘labl’ in the example below):

ItemPropertyAssociation{ entry_count = 5 // Properties associated to the group entity_id = 1234 association_count = 1 essential = 0, property_index =3// label // Properties associated to the image item #1 entity_id = 1 association_count = 2 essential = 0, property_index =1// sizes essential = 1, property_index =2// decoder configuration // Properties associated to the image item#2 // Properties associated to the image item#3 // Properties associated to the image item#4 entity_id = 4 association_count = 2 essential = 0, property_index =1// sizes essential = 1, property_index =2// decoder configuration

As can be seen from the above example, having the Image property association map allowing association of one or more property to a group of images or entities leads to a less verbose description of the image file.

Item Reference.

In a seventh embodiment, the relationships between the series of images captured are described in one capture series as Item Reference. The ItemReferenceBox describes the reference between two items of an HEIF file and associates a type of Reference to each association through reference_type parameter. In this embodiment, one reference type is defined for each capture mode. The four character code of the reference_type to use for a given capture mode is the same as the one described for grouping_type.

The principle is to set one item reference between each image captured by the device in the capture order. For example, the second image of the series of images thus describes a reference to the first image of the series, and the third image describes a reference to the second image and so on.

Depending on the reference_type value, the HEIF parser determines the relationship between a first and a second item.

For example, when reference_type is equal to:

• ‘pano’: the second item is the image that follows the first item in the capture pattern of the panorama. If the capture pattern is from left to right, the second item is at the right of the first item. By default, the left to right pattern is assumed.

• ‘imbu’ or ‘pser’ or ‘tila’: the second item is the image that follows the first item in the order of capture of the set of images.

• ‘brak’: the second item is the image in the bracketing set that follows the first item.

• ‘aebr’: the second item is the image in the exposure bracketing set that follows the first item. As a result, the second item as a longer exposure time than the first item when the bracketing is done in ascending order of exposure stop.

• ‘wbbr’: the second item is the image in the white balance bracketing set that follows the first item. As a result, the second item as a higher color temperature than the first item when the bracketing is done in ascending order of color temperature.

• ‘fobr’: the second item is the image in the focus bracketing set that follows the first item. As a result, the second item as a farther focus distance than the first item when the bracketing is done in ascending order of focus distance.

The principle is the same for the other bracketing types.

In another embodiment, the set of images from the capture series is further processed by the capturing device to generate a new image. For instance, the capture device may generate an HDR image from a series of images captured with auto exposure bracketing. In such a case, an item reference is signaled between the HDR item and the identifier of the group of the series of images in the auto exposure bracketing. As a result, the ItemReferenceBox syntax is the following:

aligned(8) class SingleltemTypeReferenceBox(referenceType) extends Box(referenceType) { unsigned int(16) from_item_ID;

unsigned int(16) reference_count;

for (j=0; j<reference_count; j++) { unsigned int(16) to_entity_ID;

} }

aligned(8) class SingleltemTypeReferenceBoxLarge(referenceType) extends Box(referenceType) { unsigned int(32) from_item_ID;

unsigned int(16) reference_count; for (j=0; j<reference_count; j++) { unsigned int(32) to_entity_ID;

} }

aligned(8) class ItemReferenceBox extends FullBox('iref', version, 0) { if (version==0) {

SingleltemTypeReferenceBox references[];

} else if (version==l) { SingleltemTypeReferenceBoxLarge references[];

} }

The new parameter to_entity_ID is an identifier of Item or group. The advantage is that a HEIF parser is able to determine that one item is computed from a group of image.

Figure 3 illustrates the main steps of a parsing process of and HEIF file generated by the encapsulating process of Figure 2. The decoding process starts by the parsing of an HEIF file with a series of images. In a step 301 the Grouping Information is parsed. The capture mode of the series of images is determined in a step 303 when the Grouping Information is present i.e. the HEIF file includes an EntityToGroup or TrackGroup or SampleToGroup box with a grouping_type equal to one of the value previously described. In a first alternative, the grouping_type parameter specifies directly the capture mode. In a second alternative, the Grouping Information signals that the set of images belong to a capture series group (the grouping type is equal to ‘case’). In this case, the Property Information is parsed in a step 302 in either ItemProperty, ItemFullProperty or VisualSampleEntry or VisualSampleGroupEntry box to determine in a step 303 the capture mode of the group of images and the parameter associated to the set and to each image of the capture series.

When the capture mode corresponds to a bracketing capture mode, the decoder notifies in a step 304 the player that the HEIF file contains a series of bracketing images. In such a case, the application provides a GUI interface that permits to view the different bracketing alternatives. In one embodiment, the interface provides the information provided in the Property Information such as the Item Properties to extract the characteristics of the capture associated to each image. In particular, for auto exposure bracketing the exposure stop of each shot are displayed in a step 305 in order to allow a user to select the appropriate shot.

Upon selection of the preferred exposure, the decoding device may modify the HEIF file to mark the selected image as primary item.

When the capture mode corresponds to a Panorama image, the decoder notifies in step 304 the player that HEIF file contains a series of images in which the user may navigate. The GUI interface may indicate specific GUI elements to allow a user to navigate between the images as a spatial composition. The player parses the Property Information to extract the pattern of capture of the set images (for example from left to right) in order to generate a navigation interface adapted to the pattern. For example, if the pattern of capture is from left to right, the GUI interface provides horizontal navigation arrows to navigate between the items of the HEIF file.

When the capture mode corresponds to a Photo Series or an Image Burst, the decoder notifies in step 304 the player, for example, to start a diaporama between all the images of the Photo Series or Image Burst group. In one embodiment, the display time of each image in the diaporama is a function of the timing Interval specified in Property Information of Image burst group.

In one embodiment, the player displays the label information provided in the Property Information at the beginning of the diaporama or as a watermarking in each image to allow the user to identify rapidly the content of the Photo Series. In another embodiment, the user may select one image as the preferred image from the series of images of the Photo series group. In such a case, the preferred image is marked as the Primary Item. In another embodiment, the user may select several images as preferred images from the Photo Series. In such a case, the player creates a new Photo Series group with the selected images and associates the same label Property Information. A new boolean parameter in Property Information indicates that the Photo Series is the Primary group of the Photo Series.

Figure 5 is a schematic block diagram of a computing device 500 for implementation of one or more embodiments of the invention. The computing device 500 may be a device such as a micro-computer, a workstation or a light portable device. The computing device 500 comprises a communication bus connected to:

- a central processing unit 501, such as a microprocessor, denoted CPU;

- a random access memory 502, denoted RAM, for storing the executable code of the method of embodiments of the invention as well as the registers adapted to record variables and parameters necessary for implementing the method according to embodiments of the invention, the memory capacity thereof can be expanded by an optional RAM connected to an expansion port for example;

- a read only memory 503, denoted ROM, for storing computer programs for implementing embodiments of the invention;

- a network interface 504 is typically connected to a communication network over which digital data to be processed are transmitted or received. The network interface 504 can be a single network interface, or composed of a set of different network interfaces (for instance wired and wireless interfaces, or different kinds of wired or wireless interfaces). Data packets are written to the network interface for transmission or are read from the network interface for reception under the control of the software application running in the CPU 501;

- a user interface 505 may be used for receiving inputs from a user or to display information to a user;

- a hard disk 506 denoted HD may be provided as a mass storage device;

- an I/O module 507 may be used for receiving/sending data from/to external devices such as a video source or display.

The executable code may be stored either in read only memory 503, on the hard disk 506 or on a removable digital medium such as for example a disk. According to a variant, the executable code of the programs can be received by means of a communication network, via the network interface 504, in order to be stored in one of the storage means of the communication device 500, such as the hard disk 506, before being executed.

The central processing unit 501 is adapted to control and direct the execution of the instructions or portions of software code of the program or programs according to embodiments of the invention, which instructions are stored in one of the aforementioned storage means. After powering on, the CPU 501 is capable of executing instructions from main RAM memory 502 relating to a software application after those instructions have been loaded from the program ROM 503 or the hard-disc (HD) 506 for example. Such a software application, when executed by the CPU 501, causes the steps of the flowcharts of the invention to be performed.

Any step of the algorithms of the invention may be implemented in software by execution of a set of instructions or program by a programmable computing machine, such as a PC (“Personal Computer”), a DSP (“Digital Signal Processor”) or a microcontroller; or else implemented in hardware by a machine or a dedicated component, such as an FPGA (“Field-Programmable Gate Array”) or an ASIC (“Application-Specific Integrated Circuit”).

Although the present invention has been described hereinabove with reference to specific embodiments, the present invention is not limited to the specific embodiments, and modifications will be apparent to a skilled person in the art which lie within the scope of the present invention.

Many further modifications and variations will suggest themselves to those versed in the art upon making reference to the foregoing illustrative embodiments, which are given by way of example only and which are not intended to limit the scope of the invention, that being determined solely by the appended claims. In particular the different features from different embodiments may be interchanged, where appropriate.

Each of the embodiments of the invention described above can be implemented solely or as a combination of a plurality of the embodiments. Also, features from different embodiments can be combined where necessary or where the combination of elements or features from individual embodiments in a single embodiment is beneficial.

Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving 5 the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features.

In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. The mere 10 fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be advantageously used.

Claims

1. A method of encapsulating images in a file, wherein the method comprises:

- determining a grouping type based on a capture mode of the images;

2. The method of claim 1, wherein the grouping type is comprised within the grouping data structure.

3. The method of claim 1, wherein:

- the grouping data structure comprises a generic capture mode type; and

4. The method of claim 3, wherein the property data structure further comprises parameters associated with the grouping type.

5. The method of claim 1, wherein:

- the grouping data structure comprises a generic capture mode type; and

6. The method of claim 1, wherein a property data structure associated with the grouping data structure is further embedded within the file.

7. A method of encapsulating images in a file, wherein the method comprises:

- determining a label parameter describing information to annotate images; and

8. A method of encapsulating images in a file, wherein the method comprises:

- determining a grouping type based on a capture mode of the images;

- embedding the reference data structure and the images in the file.

9. A method of reading images in a file, wherein the method comprises:

- reading a grouping type based on a capture mode of the images;

10. The method of claim 9, wherein the grouping type is comprised within the grouping data structure.

11. The method of claim 9, wherein:

- the grouping data structure comprises a generic capture mode type; and

12. The method of claim 11, wherein the property data structure further comprises parameters associated with the grouping type.

13. The method of claim 9, wherein:

- the grouping data structure comprises a generic capture mode type; and

14. The method of claim 9, wherein a property data structure associated with the grouping data structure is further embedded within the file.

15. A method of reading images in a file, wherein the method comprises:

- reading a label parameter describing information to annotate images; and

16. A method of reading images in a file, wherein the method comprises:

- reading a grouping type based on a capture mode of the images;

- reading the first and second images in the file.

17. The method according to any one claim 1 to 16, wherein data structures are boxes as defined in ISO Base Media File Format.

18. The method of any one claim 1 to 6 and 9 to 14, wherein a grouping type is defined for at least one of the capture modes comprising auto exposure bracketing mode, white balance bracketing mode, focus bracketing mode, flash exposure bracketing mode, depth of field bracketing mode, iso bracketing mode, time lapse mode, panorama mode, image burst mode, and user collection mode.

19. The method of any one claim 1 to 6, wherein the grouping data structure is one of an EntityToGroup box, a SampleToGroup box, or a TrackGroup box as defined in ISO Base Media File Format.

20. The method of any one claim 3, 4, 6, 11, 12, or 14 wherein the property data structure is one of a SampleEntry box, a SampleGroupEntry box, or a Item Property box as defined in ISO Base Media File Format.

21. A device for encapsulating images in a file, wherein the device comprises circuitry configured for:

- determining a grouping type based on a capture mode of the images;

22. A device for encapsulating images in a file, wherein the device comprises circuitry configured for:

- determining a label parameter describing information to annotate images; and

23. A device for encapsulating images in a file, wherein the device comprises circuitry configured for:

- determining a grouping type based on a capture mode of the images;

- embedding the reference data structure and the images in the file.

24. A device for reading images in a file, wherein the device comprises circuitry configured for:

- reading a grouping type based on a capture mode of the images;

25. A device for reading images in a file, wherein the device comprises circuitry configured for:

- reading a label parameter describing information to annotate images; and

26. A device for reading images in a file, wherein the device comprises circuitry configured for:

- reading a grouping type based on a capture mode of the images;

- reading the first and second images in the file.

27. A computer program product for a programmable apparatus, the computer program product comprising a sequence of instructions for implementing a method according to any one of claims 1 to 20, when loaded into and executed by the programmable apparatus.

28. A computer-readable storage medium storing instructions of a computer program for implementing a method according to any one of claims 1 to 20.

29. A computer program which upon execution causes the method of any one of claims 1 to 20 to be performed.