JP7146407B2 - IMAGING DEVICE, DISPLAY CONTROL DEVICE, AND CONTROL METHOD OF DISPLAY CONTROL DEVICE - Google Patents

Description

The present invention relates to an imaging device, a display control device, and a control method for a display control device.

An imaging device is known as an image processing device that compresses and encodes image data. Such an image processing apparatus acquires a moving image signal by an imaging unit, compresses and encodes the acquired moving image signal, and records the compressed and encoded image file on a recording medium. Conventionally, image data before compression encoding is represented by SDR (Standard Dynamic Range) with a luminance level of 100 nits as the upper limit. However, in recent years, image data expressed in HDR (High Dynamic Range) with the upper limit of the luminance level extended to about 10000 nits and having a luminance range close to the luminance range perceivable by humans has been provided.

Patent Document 1 describes an image data recording apparatus that generates and records image data that allows an HDR image to be checked in detail even by a device that is not HDR compatible when shooting and recording an HDR image.

JP 2017-139618 A

Japanese Patent Application Laid-Open No. 2002-200000 describes recording of an HDR image, but does not consider an optimum recording method for recording an HDR image.

Accordingly, the present invention provides an apparatus for recording an image in a recording format suitable for recording and playback when recording an image with a large amount of data, such as an HDR image, and for reproducing an image recorded in that recording format. It is an object of the present invention to provide a display control device for

Further, the present invention provides an apparatus for encoding and recording an image, in which an image is recorded in a recording form suitable for recording and reproduction even if there are various restrictions at the time of encoding, and an image is recorded in the recording form. An object of the present invention is to provide a display control device for reproducing an image.

In order to solve the above problems, the imaging device of the present invention includes:
An imaging device that captures an image with an imaging sensor and records the captured image, comprising: encoding means for encoding image data obtained by imaging with the imaging sensor; An area of the image data that is larger than the reproduction area to be reproduced is set as an encoding area, the image data in the encoding area is encoded by the encoding means, and the encoded image data is recorded in a predetermined recording format. and a control means for controlling recording on a medium, wherein the control means sets a part of a sensor image area from which image data can be acquired by the imaging sensor as an encoding area, and The image data of the area is controlled to be encoded by the encoding means, and there is a restriction on the position where encoding can be started/finished in the sensor image area, and the control means controls the reproduction area and the encoding area in the sensor image area. It is characterized by setting the start position and end position of the coding region based on the possible start/end positions.

Further, the display control device of the present invention is
The image data of the encoding area larger than the playback area to be reproduced is divided into a plurality of divided image data, each of the plurality of divided image data is encoded, and the encoded plurality of divided image data is stored in a predetermined recording format. reading means for reading out an image file from a recording medium recorded as one image file in; combining means for combining a plurality of divided image data contained in the read image file; and display control means for combining the divided image data into one image data, and controlling the display means to display the image data of the reproduction area among the combined image data.

According to the present invention, an imaging apparatus for recording an image in a recording format suitable for recording and reproduction when recording a high-resolution image with a large amount of data, and an image recorded in the recording format. A display controller can be provided for playback.

According to the present invention, in an apparatus for encoding and recording an image, even if there are various restrictions at the time of encoding, an apparatus for recording an image in a recording form suitable for recording and reproduction, and an apparatus for recording an image in the recording form. It is possible to provide a display control device for reproducing the image.

FIG. 2 is a block diagram showing the configuration of the imaging device 100; The figure which shows the structure of a HEIF file. 4 is a flowchart showing processing in HDR shooting mode; 4 is a flowchart showing processing for determining a method of dividing image data in HDR shooting mode; FIG. 4 is a diagram showing an encoding region of image data and a division method when HDR image data is recorded; 4 is a flow chart showing the process of building a HEIF file. 4 is a flowchart showing display processing when reproducing HDR image data recorded as an HEIF file. 4 is a flowchart showing overlay image creation processing; 6 is a flowchart showing image item property acquisition processing. 4 is a flowchart showing image item data acquisition processing. 6 is a flowchart showing item ID acquisition processing of an image that constitutes a main image; 4 is a flowchart showing image creation processing for one image item. FIG. 4 is a diagram showing the positional relationship between an overlay image and divided images;

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings, taking an imaging apparatus 100 as an example, but the present invention is not limited to the following embodiments.

<Structure of Imaging Device>
FIG. 1 is a block diagram showing an imaging device 100. As shown in FIG. As shown in FIG. 1, the imaging apparatus 100 includes a CPU 101, a memory 102, a nonvolatile memory 103, an operation unit 104, an imaging unit 112, an image processing unit 113, an encoding processing unit 114, and a display control unit. It has a portion 115 and a display portion 116 . Furthermore, the imaging apparatus 100 has a communication control section 117 , a communication section 118 , a recording medium control section 119 and an internal bus 130 . The imaging apparatus 100 forms an optical image of a subject on the pixel array of the imaging unit 112 using the imaging lens 111 , but the imaging lens 111 is not removable from the body of the imaging apparatus 100 . It may be detachable. The imaging apparatus 100 also writes and reads image data to and from the recording medium 120 via the recording medium control unit 119 . The recording medium 120 may or may not be detachable from the imaging device 100 .

The CPU 101 controls the operation of each unit (each functional block) of the imaging apparatus 100 via the internal bus 130 by executing a computer program stored in the nonvolatile memory 103 .

Memory 102 is a rewritable volatile memory. The memory 102 temporarily records a computer program for controlling the operation of each unit of the imaging apparatus 100, information such as parameters related to the operation of each unit of the imaging apparatus 100, information received by the communication control unit 117, and the like. In addition, the memory 102 temporarily records the image acquired by the imaging unit 112, the image processed by the image processing unit 113, the encoding processing unit 114, and the like, and information. The memory 102 has sufficient storage capacity to temporarily record them.

The non-volatile memory 103 is an electrically erasable and recordable memory, and for example, an EEPROM or the like is used. The non-volatile memory 103 stores information such as a computer program that controls the operation of each unit of the imaging apparatus 100 and parameters related to the operation of each unit of the imaging apparatus 100 . Various operations performed by the imaging apparatus 100 are realized by the computer program.

An operation unit 104 provides a user interface for operating the imaging device 100 . The operation unit 104 includes various buttons such as a power button, a menu button, and a shooting button, and the various buttons are configured by switches, touch panels, and the like. The CPU 101 controls the imaging apparatus 100 according to user instructions input via the operation unit 104 . Although the case where the CPU 101 controls the imaging apparatus 100 based on the operation input via the operation unit 104 has been described as an example, the present invention is not limited to this. For example, the CPU 101 may control the imaging device 100 based on a request input via the communication unit 118 from a remote controller (not shown), a portable terminal (not shown), or the like.

A photographing lens (lens unit) 111 includes a lens group (not shown) including a zoom lens, a focus lens, etc., a lens control section (not shown), an aperture (not shown), and the like. The taking lens 111 can function as zoom means for changing the angle of view. The lens control unit adjusts the focus and controls the aperture value (F value) according to control signals sent from the CPU 101 . The imaging unit 112 can function as an acquisition unit that sequentially acquires a plurality of images forming a moving image. As the imaging unit 112, for example, an area image sensor such as a CCD (charge-coupled device) or a CMOS (complementary metal oxide semiconductor) device is used. The imaging unit 112 has a pixel array (not shown) in which photoelectric conversion units (not shown) that convert an optical image of a subject into electrical signals are arranged in a matrix, that is, two-dimensionally. An optical image of a subject is formed on the pixel array by the imaging lens 111 . The imaging unit 112 outputs the captured image to the image processing unit 113 or the memory 102 . Note that the imaging unit 112 can also acquire a still image.

The image processing unit 113 performs predetermined image processing on image data output from the imaging unit 112 or image data read from the memory 102 . Examples of the image processing include interpolation processing, reduction processing (resize processing), color conversion processing, and the like. Also, the image processing unit 113 uses the image data acquired by the imaging unit 112 to perform predetermined arithmetic processing for exposure control, ranging control, and the like. The CPU 101 performs exposure control, distance measurement control, and the like based on the calculation result obtained by the calculation processing by the image processing unit 113 . Specifically, the CPU 101 performs AE (auto exposure) processing, AWB (auto white balance) processing, AF (auto focus) processing, and the like.

The encoding processing unit 114 compresses the size of the image data by performing intra-frame predictive encoding (intra-screen predictive encoding), inter-frame predictive encoding (inter-screen predictive encoding), etc. on the image data. . The encoding processing unit 114 is, for example, an encoding device configured by a semiconductor element or the like. The encoding processing unit 114 may be an encoding device provided outside the imaging device 100 . The encoding processing unit 114, for example, uses H.264. 265 (ITU H.265 or ISO/IEC23008-2).

The display control section 115 controls the display section 116 . The display unit 116 has a display screen (not shown). The display control unit 115 generates an image that can be displayed on the display screen of the display unit 116 by performing resizing processing, color conversion processing, and the like on the image data, and outputs the image, that is, the image signal to the display unit 116. Output. The display unit 116 displays an image on the display screen based on the image signal sent from the display control unit 115 . The display unit 116 has an OSD (On Screen Display) function that displays a setting screen such as a menu on the display screen. The display control unit 115 can superimpose the OSD image on the image signal and output the image signal to the display unit 116 . The display unit 116 is composed of a liquid crystal display, an organic EL display, or the like, and displays image signals sent from the display control unit 115 . The display unit 116 may be, for example, a touch panel. If display unit 116 is a touch panel, display unit 116 can also function as operation unit 104 .

The communication control unit 117 is controlled by the CPU 101 . Communication control section 117 generates a modulated signal conforming to a wireless communication standard predetermined by IEEE802.11 or the like, and outputs the modulated signal to communication section 118 . Communication control section 117 also receives a modulated signal conforming to the wireless communication standard via communication section 118 , decodes the received modulated signal, and outputs a signal corresponding to the decoded signal to CPU 101 . The communication control unit 117 has a register for storing communication settings. The communication control unit 117 can adjust the transmission/reception sensitivity during communication under the control of the CPU 101 . The communication control unit 117 can perform transmission and reception using a predetermined modulation method. The communication unit 118 outputs the modulated signal supplied from the communication control unit 117 to an external device 127 such as an information communication device outside the imaging device 100, and has an antenna for receiving the modulated signal from the external device 127. ing. Further, the communication unit 118 is provided with circuits for communication and the like. Although a case where wireless communication is performed by the communication unit 118 has been described as an example here, communication performed by the communication unit 118 is not limited to wireless communication. For example, the communication unit 118 and the external device 127 may be connected by electrical connection using wiring or the like.

A recording medium control unit 119 controls the recording medium 120 . Recording medium control unit 119 outputs a control signal for controlling recording medium 120 to recording medium 120 based on a request from CPU 101 . As the recording medium 120, for example, a nonvolatile memory, a magnetic disk, or the like is used. The recording medium 120 may be removable or non-removable as described above. The recording medium 120 records encoded image data and the like. Image data and the like are saved as files in a format compatible with the file system of the recording medium 120 . Examples of files include MP4 files (ISO/IEC 14496-14:2003), MXF (Material eXchange Format) files, and the like. Functional blocks 101 to 104, 112 to 115, 117, and 119 are accessible to each other via internal bus .

Here, a normal operation of the imaging apparatus 100 of this embodiment will be described.

When the user operates the power button of the operation unit 104 , the imaging apparatus 100 issues an activation instruction from the operation unit 104 to the CPU 101 . Upon receiving this instruction, the CPU 101 controls a power supply unit (not shown) to supply power to each block of the imaging apparatus 100 . When power is supplied, the CPU 101 confirms, based on an instruction signal from the operation unit 102, which mode, for example, a still image shooting mode or a playback mode, the mode switching switch of the operation unit 104 is set to.

In a normal still image shooting mode, the imaging apparatus 100 performs shooting processing when the user operates the still image recording button of the operation unit 104 in a shooting standby state. In the shooting process, image data of a still image shot by the imaging unit 112 is subjected to image processing by the image processing unit 113, encoding processing by the encoding processing unit 114, and encoded image data by the recording medium control unit 119. is recorded on the recording medium 120 as an image file. In the shooting standby state, the image capturing unit 112 captures an image at a predetermined frame rate, the image processing unit performs image processing for display, and the display control unit 115 displays the live view image on the display unit 116 . display.

In the reproduction mode, the image file recorded on the recording medium 120 is read by the recording medium control unit 119, and the image data of the read image file is decoded by the encoding processing unit 114. FIG. In other words, the encoding processing unit 114 also has a decoder function. Then, the image processing unit 113 performs processing for display, and the display control unit 115 causes the display unit 116 to display the image.

Shooting and playback of normal still images are performed as described above, but the imaging apparatus of this embodiment has an HDR shooting mode for shooting not only normal still images but also HDR still images. Also, it is possible to play back HDR still images that have been taken.

Processing for capturing and reproducing an HDR still image will be described below.

<File structure>
First, the file structure for recording HDR still images will be described.

Recently, a still image file format called High Efficiency Image File Format (hereinafter referred to as HEIF) has been established. (ISO/IEC 23008-12:2017)

This has the following features compared to conventional still image file formats such as JPEG.
- The file format conforms to the ISO base media file format (hereinafter referred to as ISOBMFF). (ISO/IEC 14496-14:2003)
・Not only a single still image but also multiple still images can be stored.
・HEVC/H. 265 and AVC/H. 264 or other compression format used for compressing moving images.

In this embodiment, HEIF is adopted as the HDR still image recording file.

First, data stored by the HEIF will be described.

HEIF manages individual data to be stored in units called items.

Each item has, in addition to the data itself, an integer item ID (item_ID) that is unique within the file, and an item type (item_type) that indicates the type of item.

Items can be divided into image items, where the data represents an image, and metadata items, where the data is metadata.

Image items include a coded image item, which is image data whose data is encoded, and a derived image item, which represents an image that is the result of manipulating one or more other image items. ).

An example of a derived image item is an overlay image, which is a derived image item in overlay form. This is an overlay image resulting from synthesizing an arbitrary number of image items at arbitrary positions using the ImageOverlay structure (overlay information).

Exif data can be stored as an example of a metadata item.

As mentioned above, HEIF can store multiple image items.

If there are relationships between images, those relationships can be described.

Examples of the relationship between multiple images are the relationship between a derived image item and its constituent image items, the relationship between a main image and a thumbnail image, and the like.

Also, the relationship between image items and metadata items can be similarly described.

The HEIF format complies with the ISOBMFF format. Therefore, first, ISOBMFF will be briefly described.

The ISOBMFF format manages data in a structure called a box.

A box is a data structure that begins with a 4-byte data length field and a 4-byte data type field, followed by data of arbitrary length.

The structure of the data part is determined by the data type. The ISOBMFF specifications and HEIF specifications define several data types and the structure of their data portions.

A box may also have another box in its data. That is, boxes can be nested. Here, a box nested in the data part of a certain box is called a subbox.

A box that is not a subbox is called a file-level box. This is a box accessible sequentially from the beginning of the file.

A file in HEIF format will be described with reference to FIG.

First, let's talk about file-level boxes.

A file type box whose data type is 'ftyp' stores information about file compatibility. A file specification conforming to ISOBMFF declares the structure of the file defined by the specification and the data stored in the file with a 4-byte code called brand, and stores them in the file type box. Placing the file type box at the beginning of the file allows the reader of the file to know the structure of the file by checking the contents of the file type box without further reading and interpreting the contents of the file.

The HEIF specification uses the brand 'mif1' to represent its file structure. Also, when the encoded image data to be stored is HEVC-compressed data, it is represented by the brand 'heic' or 'heix' according to the HEVC compression profile.

A metadata box of data type 'meta' contains various sub-boxes and stores data about each item. The breakdown will be described later.

A media data box whose data type is 'mdat' stores the data of each item. For example, encoded image data for encoded image items and Exif data for metadata items.

Next, the sub-boxes of the metadata box will be explained.

A handler box whose data type is 'hdlr' stores information representing the type of data managed by the metadata box. In the HEIF specification, the handler_type of a handler box is 'pict'.

A data information box whose data type is 'dinf' specifies the file in which the data targeted by this file exists. In ISOBMFF, it is possible to store data intended for a certain file in a file other than that file. In that case, a data entry URL box describing the URL of the file in which the data exists is stored in the data reference in the data information box. If the data of interest exists in the same file, store the data entry URL box with only a flag indicating that.

A primary item box of data type 'pitm' stores the item ID of the image item representing the main image.

The item information box whose data type is 'iinf' is a box for storing the following item information entries.

The item information entry whose data type is 'infe' stores the item ID, item type, and flag of each item.

The item type of an image item whose encoded image data is HEVC compressed data is 'hvc1' and is an image item.

The item type of the derived image item in overlay format, that is, the ImageOverlay structure (overlay information) is 'iovl' and is classified as an image item.

The item type of a metadata item in Exif format is 'Exif' and is a metadata item.

A graphical item can also be specified to be a hidden image if the least significant bit of the flag field is set. When this flag is set, the image item is not treated as a display target during playback and is hidden.

The item reference box whose data type is 'iref' stores the reference relationship between each item by the reference relationship type, the item ID of the referencing item, and the item ID of one or more referencing items.

In the case of an overlay-format derived image item, the reference relationship type is 'dimg', the item ID of the referencing item is the item ID of the derived image item, the item ID of the reference destination item is the item ID of each image item that composes the overlay, to store

In the case of a thumbnail image, 'thmb' is stored as the type of reference relationship, the item ID of the thumbnail image is stored as the item ID of the referencing item, and the item ID of the main image is stored as the item ID of the referencing item.

The item property box whose data type is 'iprp' is a box for storing the following item property container box and item property related box.

An item property container box whose data type is 'ipco' is a box that stores individual property data boxes.

Each image item can have property data representing characteristics and attributes of the image.

Property data boxes include:

Decoder configuration and initialization data (for HEVC, the type is 'hvcC') is data used to initialize the decoder. HEVC parameter set data (VideoParameterSet, SequenceParameterSet, PictureParameterSet) are stored here.

Image spatial extents (of type 'ispe') are the dimensions (width, height) of the image.

Color space information (Colour information, type 'colr') is the color space information of the image.

Image rotation information (Image rotation, type 'irot') is the direction of rotation when an image is rotated and displayed.

Image pixel information (type is 'pixi') is information indicating the number of bits of data forming an image.

Further, in this embodiment, as HDR metadata, master display color volume information (Mastering display color volume; type is 'MDCV') and content light level information (Contents light level information; type is 'CLLI') are stored as property It is stored as a data box.

Properties other than the above exist, but they are not specified here.

The item property association box of data type 'ipma' stores the association between each item and property in the form of an item ID and an array of indices in 'ipco' of the associated property.

An item data box whose data type is 'idat' is a box that stores item data with a small data size.

As for the data of the derived image item in overlay format, the ImageOverlay structure (overlay information) storing the position information of the component image can be stored in 'idat'. The overlay information includes a canvas_fill_value parameter, which is background color information, and output_width and output_height parameters, which are the final size of the reproduced image after overlay synthesis. Further, there are horizontal_offset and vertical_offset parameters indicating the horizontal position coordinates and vertical position coordinates of the arrangement position in the composite image for each image to be the composition element. By using these parameters included in the overlay information, it is possible to reproduce a composite image in which a plurality of images are arranged at arbitrary positions within one image with a specified background color and size.

The item location box whose data type is 'iloc' stores location information of data of each item in the form of offset reference (construction_method), offset value from the offset reference, and length.

The offset reference is the beginning of the file, or 'idat'.

Note that there are other boxes in the metadata box than the above, but they are not specified here.

FIG. 2 illustrates a structure in which two coded image items constitute an image in overlay format. The number of encoded image items forming an overlay format image is not limited to two. As the number of encoded image items that make up an overlay format image increases, the following boxes and items increase accordingly.
- Added item info entry for coded image items in the item info box.
- The item ID of the encoded image item is added to the reference item ID of the reference relationship type 'dimg' in the item reference box.
• Decoder configuration/initialization data, image space extents, etc., in the item properties container box.
- Added item for index of coded image item and its related property to item property related box.
- Added location information items for encoded image items to the item location box.
・Encoded image data is added to the media data box.

<Photography of HDR image>
Next, processing in the imaging device 100 when capturing and recording an HDR image will be described.

The width and height of images to be recorded are increasing these days, but if too large a size is encoded, compatibility may be lost when decoding with other models, and the scale of the system may increase. Specifically, in this embodiment, H.264 is used for encoding/decoding. An example using H.265 will be described. H. The H.265 standard has parameters such as Profile and Level, and these parameters change as the image encoding method and image size increase. These values are parameters for judging whether or not the data can be reproduced by the equipment at the time of decoding. In this embodiment, when recording a large image size, a method will be described in which the image is divided into a plurality of image display areas, the overlay method stipulated by the HEIF standard is used, and the encoded data is stored in a HEIF file and managed. By dividing into a plurality of image display areas in this way, the size of each image display area is reduced, and the reproduction compatibility with other models is improved. In the following description, the size of one side of the encoding area is defined as 4096 or less. The encoding format is H.264. Any size other than 265 may be used, and the size of one side may be defined as a size other than 4096.

Alignment constraints on the starting position of the encoding area, the width and height of the encoding area, and the starting position of the reproduction area and the width and height of the reproduction area will now be described. As an imaging device, there are generally hardware restrictions such as vertical and horizontal encoding start alignments in encoding regions, playback width and height alignments during playback, and the like when encoding divided images. In other words, it is necessary to calculate the encoding start position of each of one or more encoding regions and the start position, width, and height of the reproduction region before encoding. Since the image capturing apparatus records by switching between a plurality of image sizes according to instructions from the user, it is necessary to switch between them for each image size.

The encoding processing unit 114 described in the present embodiment has alignment restrictions on the start position of 64 pixels in the horizontal direction (hereinafter referred to as the x direction) and 1 pixel in the vertical direction (hereinafter referred to as the y direction) at the time of encoding. Alignment restrictions for the width and height to be encoded are 32-pixel units and 16-pixel units, respectively. The starting position of the reproduction area has alignment constraints of two pixels in the x direction and one pixel in the y direction. Further, the width and height alignment restrictions of the reproduction area are restricted in units of 2 pixels and 1 pixel, respectively. Although an example of alignment constraints has been described here, other alignment constraints may generally be used.

Next, a flow from shooting HDR image data in the HDR recording mode (HDR shooting mode) to recording the shot HDR image data as an HEIF file will be described with reference to FIG. In the HDR mode, the color gamut BT. 2020 and processing for expressing the HDR color space using the PQ gamma curve. Recording for HDR is omitted in this embodiment.

In S101, the CPU 101 determines whether or not the user has pressed SW2. When the user catches the subject and presses up to SW2 (YES in S101), the CPU 101 detects that SW2 has been pressed, and transitions to S102. The CPU 101 continues this detection until SW2 is pressed (NO in S101).

In S102, the CPU 101 determines how many divisions the coding area is to be divided vertically and horizontally (division number N) and how to divide the coding area in order to save the image of the angle of view captured by the user in the HEIF file. This result is stored in memory 102 . S102 will be described later with reference to FIG. Proceed to S103.

In S103, the CPU 101 initializes the variable M in the memory 102 to M=1. After that, the process proceeds to S104.

In S104, the CPU 101 monitors whether the loop has been performed for the number of divisions N determined in S102. If the number of loops reaches the number of divisions N (YES in S104), proceed to S109; otherwise (NO in S104), proceed to S105.

In S105, the CPU 101 reads the information on the code start area and the reproduction area of the divided image M from the memory 102 described above. Proceed to S106.

In S106, the CPU 101 informs the encoding processing unit 114 which area of the entire image arranged in the memory 102 is to be encoded, and causes the encoding processing unit 114 to encode the area. Proceed to S107.

In S107, the CPU 101 temporarily stores in the memory 102 the encoded data resulting from encoding by the encoding processing unit 114 and accompanying information generated during encoding. Specifically, H. In the example of H.265, H.265 files such as VPS, SPS, PPS, etc., which must later be stored in HEIF files. H.265 standard information. Since it is not directly related to this embodiment, it is omitted here. H.265 VPS, SPS, and PPS are various pieces of information necessary for decoding, such as the size of encoded data, bit depth, specification of display/non-display areas, and frame information. After that, the process proceeds to S108.

In S108, the CPU 101 increments the variable M and returns to S104.

In S109, the CPU 101 generates metadata to be stored in the HEIF file. The CPU 101 extracts information necessary for reproduction from the imaging unit 112, image processing unit 113, encoding processing unit 114, etc., and temporarily stores it in the memory 102 as metadata. Specifically, it is data formed in the Exif format. The details of the Exif data are already common, so the description in this embodiment is omitted. Proceed to S110.

At S110, the CPU 101 encodes the thumbnail. The image processing unit 113 reduces the entire image in the memory 102 to a thumbnail size and temporarily stores it in the memory 102 . The CPU 101 instructs the encoding processing unit 114 to encode the thumbnail images in the memory 102 . Here, the thumbnails are coded with a size that satisfies the above-described alignment restrictions, but a method in which the coding area and the reproduction area are different may be used. This thumbnail is also H. H.265 encoded. Proceed to S111.

In S111, the CPU 101 temporarily saves in the memory 102 the coded data resulting from the coding of the thumbnail by the coding processing unit 114 and the associated information generated during coding. H.264 such as VPS, SPS, PPS, etc., as in the previous description of S107. H.265 standard information. Next, the process proceeds to S112.

In S112, the CPU 101 has stored various data in the memory 102 in the steps up to this point. The CPU 101 combines the data stored in the memory 102 in order to complete the HEIF file and stores it in the memory 102 . A flow for completing this HEIF file will be described later with reference to FIG. Proceed to S113.

In S 113 , the CPU 101 instructs the recording medium control unit 119 to write the HEIF file in the memory 102 to the recording medium 120 . After that, the process returns to S101.

The HEIF file is recorded on the recording medium 120 after the image is captured in these steps.

Next, the division method determination in S102 described above will be described with reference to FIGS. 4 and 5A. Below, the upper left corner of the sensor image area is defined as the origin (0, 0), the coordinates are (x, y), and the width and height are represented by w and h, respectively. The region is expressed as [x, y, w, h] by combining the start coordinates (x, y) and the width and height w, h.

In S121, the CPU 101 acquires from the memory 102 the recording mode at the time of shooting. A recording mode is a recording method in which an image size, an image aspect ratio, an image compression ratio, etc. are determined, and the user selects one of these recording modes for photographing. For example, setting such as L size and 3:2 aspect ratio. Since the shooting angle of view differs depending on the recording mode, the shooting angle of view is determined. Proceed to S122.

In S122, the start coordinate position (x0, y0) of the reproduced image is obtained from the sensor image area [0, 0, H, V]. Next, the process proceeds to S123.

In S123, the area [x0, y0, Hp, Vp] of the reproduced image is obtained from the memory 102. FIG. Next, the process proceeds to S124.

In S124, it is necessary to encode so as to include the area of the reproduced image. As described above, the alignment of the code start position is 64 pixels in the x direction, so if encoding is to include the area of the reproduced image, the encoding must start from the position (Ex0, Ey0). Thus, the size of Lex is obtained. If this Lex offset is 0 (NO in S124), the process proceeds to S126, and if necessary (YES in S124), the process proceeds to S125.

At S125, Lex is obtained. Lex is determined by the alignment of the reproduced image area and the encoding start position, as described above. The CPU 101 determines Lex as follows. It is obtained by dividing x0 by the pixel alignment 64 pixels of the horizontal encoding start position and multiplying the quotient by the previous 64 pixels. Since the calculation is performed without including the remainder, the coordinates positioned to the left of the start position of the reproduced image area are obtained. Therefore, Lex is obtained by the difference between the x-coordinate of the reproduction image area and the x-coordinate of the encoding start position obtained by the previous calculation. Proceed to S126.

In S126, the coordinates of the final position of the right end of the uppermost line of the reproduced image are calculated. Let this be (xN, y0). xN is obtained by adding Hp to x0. Proceed to S127.

In S127, the right end of the encoded area is obtained so as to include the right end of the reproduced image. In order to obtain the right end w of the coding area, alignment constraints on the coding width are necessary. As described above, the restrictions on the width and height of encoding are each restricted to multiples of 32 pixels. Based on the relationship between this constraint and (xN, y0), the CPU 101 calculates whether or not an offset of Rex is required prior to the right end of the reproduced image. If Rex is required (YES in S127), the process proceeds to S128; otherwise (NO in S127), the process proceeds to S129.

In S128, the CPU 101 adds Rex to the right side of the right edge coordinates (xN, y0) of the reproduced image so as to have a 32-pixel alignment, and obtains the encoding end position (ExN, Ey0). Proceed to S129.

In S129, the CPU 101 obtains the lower right coordinates of the display area. Since the size of the reproduced image is self-explanatory, the coordinates are (xN, yN). Proceed to S130.

In S130, the lower end of the encoded region is obtained so as to include the lower end of the reproduced image. To find the bottom edge of the encoded region, an encoding height alignment constraint is required. As mentioned earlier, the encoding height constraints are each a multiple of 16 pixels. The CPU 101 calculates from the relationship between this constraint and the lower right coordinate (xN, yN) of the reproduced image whether or not the Vex offset is required prior to the lower end of the reproduced image. If Vex is required (YES in S130), the process proceeds to S131, and if not (NO in S130), the process proceeds to S132.

In S131, the CPU 101 calculates Vex from the relationship between the above constraints and the lower right coordinates (xN, yN) of the reproduced image, and obtains the lower right coordinates (ExN, EyN) of the encoding area. The Vex calculation method is set to a multiple of 16 pixels, which is the height alignment of encoding, so as to include y0 to yN, which are the encoding start positions in the vertical direction. Specifically, assuming that y0=0, Vp is divided by 16 pixels of height alignment for encoding, and the quotient and remainder are obtained. If there is a remainder, add 1 to the quotient and multiply by the previous 16 pixels, since the encoded region must be taken to include the reconstructed region. This gives EyN of the y-coordinate of the lower end of the encoded region which is a multiple of 16 pixels but still contains Vp. Vex is obtained by the difference of yN from EyN. EyN is a value obtained by offsetting yN downward by Vex, and ExN is a value already obtained in S128. Proceed to S132.

In S132, the CPU 101 obtains the size Hp'xVp' of the encoding region as follows. Hp' is obtained by adding Lex and Rex obtained from the alignment constraint to the horizontal size Hp. Vex obtained from the alignment constraint is added to the vertical size Vp to obtain Vp'. From Hp' and Vp', the size of the coding region Hp'.times.Vp' is obtained. Proceed to S133.

In S133, the CPU 101 determines whether or not the horizontal size Hp' to be encoded exceeds the 4096 pixels to be divided. proceed to

In S134, the horizontal size Hp' to be encoded is divided into two or more regions so as to be 4096 pixels or less. For example, 6000 pixels are divided into two, and 9000 pixels are divided into three.

In the case of division into two, division is performed at an approximate center position that satisfies the alignment of 32 pixels with respect to the width of the previous encoding. If it is not uniform, increase the left split image. Also, if the division into three is not uniform, divide the divided areas into A, B, and C while paying attention to the alignment regarding the width of the previous encoding so that the divided areas A and B have the same size, The divided area C is divided so as to be slightly smaller than that. The division position and size are determined by the same algorithm even if the division is three or more. Proceed to S135.

In S135, the CPU 101 determines whether or not the vertical size Vp' to be encoded exceeds the 4096 pixels to be divided. exit.

In S136, as in S134, Vp' is divided into a plurality of divided regions, and the process ends.

An example of division in S102 will be described with specific numerical values using FIGS. FIG. 5 shows not only how the image data is divided, but also the relationship between the encoding area and the reproduction area.

FIG. 5B will be described. FIG. 5B shows a case in which an image is divided into left and right areas, and Lex, Rex, and Vex are assigned to the left, right, and bottom ends of the reproduction area. First, the reproduced image area is [256+48, 0, 4864, 3242], which is the area of the final image to be recorded by the imaging device. The upper left coordinates of this reconstructed image area are (256+48, 0), not a multiple of 64 pixel units which is the starting alignment of the encoding mentioned above. Therefore, Lex is provided 48 pixels to the left of this point, and the left end of the encoding start position is at x=256. Next, the width of the reproduction area is w = 48 + 4864, but this is not a multiple of 32 pixels of the alignment of the width of the encoding, and considering the constraint alignment of the width at the time of encoding, the right end of the reproduction image area is is given 16 pixels Rex. Similarly, when the vertical direction is calculated, Vex=4 pixels. Since the size of this reproduced image in the horizontal direction exceeds 4096, the horizontal direction is divided into two at approximately the center position while considering the alignment of the width of the encoding. 0, 2496, 3242+8], and the encoded image area of the divided image 2 is obtained as [256+2496, 0, 2432, 3242+8].

FIG. 5(c) will be described. FIG. 5C shows a case in which an image is divided into left and right regions, and Rex and Vex are added to the right and bottom ends of the reproduction region. Since the x-position of the reproduced image area is x=0, it matches the 64-pixel starting alignment of the coding area, so Lex=0. Thereafter, the calculation method is the same as described above, and as a result, the encoded image area of divided image 1 is [0, 0, 2240, 32456+8], and the encoded image area of divided image 2 is [2240, 0, 2144, 2456+8]. ] is found.

FIG. 5(d) will be described. FIG. 5D shows a case in which an image is divided into left and right areas, and Lex and Rex are given to the left and right of the reproduction area. The vertical size of the reproduced image area is h=3648, which is a multiple of 16 pixels which is the constraint on the image height at the time of encoding, so Vex=0. Thereafter, the calculation method is the same as described above, and as a result, the encoded image area of divided image 1 is [256+48, 0, 2240, 3648], and the encoded image area of divided image 2 is [256+2240, 0, 2144, 3648]. ] is found.

FIG. 5(d) will be described. FIG. 5D shows a case in which an image is divided into four areas vertically and horizontally, and Lex and Rex are given to the left and right of the reproduction area. In this case, since the encoded area exceeds 4096 in both the vertical and horizontal directions, it is divided into squares. Thereafter, the calculation method is the same as described above, and as a result, the encoded image area of divided image 1 is [0, 0, 4032, 3008], and the encoded image area of divided image 2 is [4032, 0, 4032, 3008]. ], the encoded image area of divided image 3 is [0, 3008, 4032, 2902], and the encoded image area of divided image 4 is [4032, 3008, 4032, 2902].

In this manner, the CPU 101 must determine the area of the reproduced image according to the recording mode, and calculate the divided area for each determined area. As another method, these calculations are performed in advance and the results are stored in memory 102 . The CPU 101 may read the information of the divided area and the reproduction area corresponding to the recording mode from the memory 102 and set it in the encoding processing section 114 .

When a photographed image is divided into a plurality of divided images and recorded as in this embodiment, it is necessary to synthesize a plurality of display areas in order to restore the photographed image from the recorded image. When generating a composite image, it is more efficient if each divided image does not have an overlapping area, because there is no need to encode or record extra data. Therefore, in this embodiment, each reproduction region and each coding region are set so that there is no overlapping portion (overlapping region) at the boundary between divided images. However, overlapping areas may be provided between divided images depending on conditions such as hardware alignment restrictions.

Next, the method of constructing the HEIF file described in S112 will be described with reference to FIG. Since the HEIF file has the structure shown in FIG. 2, the CPU 101 constructs the file in the memory 102 in order from the beginning.

A 'ftyp' box is generated in S141. This box is placed at the top of the file to know the compatibility of the file. The CPU 101 stores 'heic' as the major brand of this box, and 'mif1' and 'heic' as the compatible brands. Although these values are used in this embodiment, other values may be used. Proceed to S142.

A 'meta' box is generated in S142. This box is a box that stores multiple boxes described below. The CPU 101 generates this box and proceeds to S143.

At S143, a 'hdlr' box to be stored in the 'iinf' box is generated. This box shows the attributes of the 'meta' box mentioned earlier. The CPU 101 stores 'pict' in this type and proceeds to S144. The 'pict' is information representing the type of data managed by the metadata box, and the HEIF specification defines the handler_type of the handler box as 'pict'.

At S144, a 'dinf' box to be stored in the 'iinf' box is generated. This box indicates the location of the data targeted by this file. The CPU 101 generates this box and advances to S145.

At S145, a 'pitm' box to be stored in the 'iinf' box is generated. This box stores the item ID of the image item that represents the main image. Since the main image is the overlay image synthesized by the overlay, the CPU 101 stores the overlay information item ID as the item ID of the main image. Proceed to S146.

At S146, a 'iinf' box to be stored in the 'meta' box is generated. This box is for managing the list of items. Here, an initial value is entered in the data length field, 'iinf' is saved in the data type field, and the process proceeds to S147.

At S147, an 'infe' box to be stored in the 'iinf' box is generated. This 'infe' box is a box for registering item information of each item stored in the file, and an 'infe' box is generated for each item. As for split images, each split image is registered in this box as one item. In addition, overlay information for constructing a main image from a plurality of divided images, Exif information, and thumbnail images are registered as items. At this time, as described above, overlay information, divided images, and thumbnail images are registered as image items. For image items, it is possible to add a flag indicating a hidden image, and by adding this flag, the image will not be displayed during playback. In other words, by adding this hidden image flag to the image item, it is possible to designate an image to be hidden during playback. Therefore, a flag indicating a hidden image is set for the split image, and the flag is not set for the item of the overlay information. As a result, the individual divided images themselves are excluded from the display target, and only the image obtained by overlay-combining the plurality of divided images according to the overlay information becomes the display target. Create separate 'infe' boxes for each item and store them in the previous 'iinf'. Proceed to S148.

At S148, an 'iref' box to be stored in the 'iinf' box is generated. This box stores information indicating the relationship between an image composed of an overlay (main image) and the divided images that compose the image. Since the dividing method has been determined in S102 described above, the CPU 101 generates this box based on the determined dividing method. Proceed to S149.

At S149, an 'iprp' box to be stored in the 'iinf' box is generated. This box is a box for storing item properties, and is a box for storing the 'ipco' box generated in S150 and the 'ipma' box generated in S151. Proceed to S150.

At S150, an 'ipco' box to be stored in the 'iprp' box is generated. This box is the property container box for the item and stores various properties. There are multiple properties, and the CPU 101 generates the following property container box and stores it in the 'ipco' box. There are properties generated for each image item and properties generated common to multiple image items. A 'colr' box is generated as information common to the overlay image and the split image configured by the overlay information of the main image. The 'colr' box stores color information such as an HDR gamma curve as color space information of the main image (overlay image) and divided images. For split images, 'hvcC' box and 'ispe' box are generated respectively. In S107 and S111 described above, the CPU 101 reads out the accompanying information at the time of encoding stored in the memory 102, and generates a property container box of 'hvcC'. The accompanying information stored in this 'hvcC' includes not only information when the encoding area is encoded, the size (width, height) of the encoding area, but also the size (width, height) of the reproduction area, It also includes the position information of the reproduction area in the coding area. These accompanying information are Golomb-compressed and recorded in the property container box of 'hvcC'. Since the Golomb compression is a general compression method, the explanation is omitted here. The 'ispe' box stores the size (width, height) information (for example, 2240×2450, etc.) of the reproduction area of the divided image. A 'irot' box storing information indicating the rotation of the overlay image that is the main image and a 'pixi' box indicating the number of bits of image data are generated. The 'pixi' box may be generated separately for the main image (overlay image) and the divided image. generate only It also creates 'CLLI' and 'MDCV' boxes in which supplementary information for HDR is stored. Furthermore, as properties of the thumbnail image, apart from the main image (overlay image), a 'colr' box that stores color space information, a 'hvcC' box that stores information at the time of encoding, and an 'ispe' box that stores image size. A box, a 'pixi' box storing information on the number of bits of image data, and a 'CLLI' box storing HDR supplementary information are generated. Create these property container boxes and store them inside the 'ipco' box. Proceed to S151.

An 'ipma' box is generated in S151. This box is a box that indicates the relationship between items and properties, and indicates which property is associated with each item. The CPU 101 determines the relationship between items and properties from various data stored in the memory 102 and generates this box. Proceed to S152.

An 'idat' box is generated in S152. This box stores overlay information indicating how to arrange the reproduction area of each divided image to generate an overlay image. The overlay information includes a canvas_fill_value parameter, which is background color information, and output_width and output_height parameters, which are the size of the entire overlay image. Further, there are horizontal_offset and vertical_offset parameters indicating horizontal position coordinates and vertical position coordinates for synthesizing the split images for each split image serving as a synthesis element. The CPU 101 writes information on these parameters based on the division method determined in S102. Specifically, the size of the entire playback area is described in the output_width and output_height parameters indicating the size of the overlay image. Then, the horizontal_offset and vertical_offset parameters indicating the position information of each divided image are the offset values in the width direction and the height direction from the start coordinate position (x0, y0) at the upper left of the reproduction area to the upper left position of the divided image. Describe. By generating the overlay information in this way, the image before division can be reproduced by arranging the divided images based on the horizontal_offset and vertical_offset parameters and synthesizing the divided images when reproducing the image. become. In this embodiment, since no overlapping area is provided in the divided images, the position information is described so that the divided images are arranged so as not to overlap each other. By specifying a region to be displayed in the image obtained by synthesizing the divided images with the output_width and output_height parameters, only the reproduction region can be reproduced as a display target. An 'idat' box in which the generated overlay information is stored is generated. Proceed to S153.

A 'iloc' box is generated in S153. This box is a box showing where various data are placed in the file. Since various information is stored in the memory 102, this box is generated from these sizes. Specifically, the information indicating the overlay is stored in the above 'idat', and the position and size information inside this 'idat'. Also, the thumbnail data and code data 12 are stored in the 'mdat' box, and the position and size information are stored. Proceed to S154.

A 'mdat' box is generated in S154. This box is a box containing multiple boxes described below. The CPU 101 generates an 'mdat' box and advances to S155.

At S155, the Exif data is stored in the 'mdat' box. Since the metadata Exif is stored in the memory 102 in S109, the CPU 101 reads it out from the memory 102 and adds it to the 'mdat' box. Proceed to S156.

At S156, the thumbnail data is stored in the 'mdat' box. Since the thumbnail data was saved in the memory 102 in the previous S110, the CPU 101 reads it from the memory 102 and adds it to the 'mdat' box. Proceed to S157.

In S157, data of encoded image 1 is stored in the 'mdat' box. Since the data of the divided image 1 has been saved in the memory 102 in the first loop of the previous S106, the CPU 101 reads this from the memory 102 and adds it to the 'mdat' box. This is repeated up to encoded image N, and all encoded images 1-N are added to the 'mdat' box. The CPU 101 constructs the HEIF file in such steps.

Through the above processing, the HDR image can be recorded on the recording medium 120 in the HEIF format as shown in FIG.

<Playback of HDR image>
Next, processing in the imaging device 100 when reproducing an HDR image file recorded in the HEIF format on the recording medium 120 will be described. In the present embodiment, the case of reproduction by the imaging apparatus 100 will be described, but the case of reproducing an HDR image recorded in the recording medium 120 in HEIF format in an image processing apparatus that does not have an imaging unit. Alternatively, similar processing may be implemented.

HEIF playback (display) processing when the main image is an overlay image will be described with reference to FIG.

In step S<b>701 , the CPU 101 uses the recording medium control unit 119 to read the head portion of the specified file existing on the recording medium 120 into the memory 102 . Then, it checks whether there is a file type box with a correct structure at the beginning of the read file, and whether 'mif1' representing HEIF exists in the brand therein.

Also, if a brand corresponding to a unique file structure is recorded, the presence of that brand is checked. As long as the brand guarantees a specific structure, this brand check may also omit some further structure checks, eg steps S703, S704.

In step S<b>702 , the CPU 101 reads the metadata box of the specified file from the recording medium 120 to the memory 102 .

In step S703, the CPU 101 searches for the handler box from the metadata box read in step S702 and checks the structure. For HEIF, the handler type must be 'pict'.

At step S704, the CPU 101 searches the data information box from the metadata box read at step S702 and checks the structure. In this embodiment, it is assumed that the data exists in the same file, so it is checked whether the data entry URL box is flagged.

In step S705, the CPU 101 searches for the primary item box from the metadata box read in step S702, and acquires the item ID of the main image.

In step S706, the CPU 101 searches for an item information box from the metadata box read out in step S702, and obtains an item information entry corresponding to the item ID of the main image obtained in step S705. In the case of a HEIF format image file recorded by the above-described HDR image capturing process, overlay information is specified as the main image. That is, in the item information entry corresponding to the item ID of the main image, the item type is 'iovl' representing overlay.

In step S707, the CPU 101 executes overlay image creation processing. This will be explained later using FIG.

In step S<b>708 , the CPU 101 displays the overlay image created in step S<b>707 on the display unit 116 via the display control unit 115 .

When displaying the created overlay image, it may be necessary to specify the color space information of the image. For the color space information of the image, since the color gamut is specified by color_primaries of the color space property 'colr' and the conversion characteristics (equivalent to gamma) are specified by transfer_characteristics, those values are used. As an example of an HDR image, if the color gamut is Rec. ITU-R BT. 2020, the conversion characteristic is Rec. ITU-R BT. 2100 (PQ), etc. are used.

Also, if HDR metadata such as 'MDCV' and 'CLLI' exists in the item property container box, it is also possible to use it.

Next, overlay image creation processing will be described using FIG.

In step S801, the CPU 101 executes overlay image property acquisition processing. This will be explained later with reference to FIG.

In step S802, the CPU 101 executes overlay information acquisition processing for an overlay image. Since the overlay information is recorded in the 'idat' box, the data stored in the 'idat' box is obtained.

The overlay information includes the image size of the overlay image (output_width, output_height parameters), position information (horizontal_offset, vertical_offset) of individual image items (divided image items) that make up the overlay image, and the like.

The overlay information is image item data handling of an image item whose item type is overlay 'iovl'.

This acquisition processing will be described later with reference to FIG.

In step S803, the CPU 101 executes item ID acquisition processing for all divided image items. This will be explained later using FIG.

In step S804, the CPU 101 uses the overlay image size of the overlay information acquired in step S802 to secure memory for storing image data of that size. This memory area is called a canvas.

In step S805, the CPU 101 initializes a divided image counter n to zero.

In step S806, the CPU 101 checks whether the split image counter n is equal to the number of split image items. If so, proceed to step S2210. If not, proceed to step S807.

In step S807, the CPU 101 executes image creation processing for one image item with respect to the n-th divided image item. This will be explained later using FIG.

In step S808, the CPU 101 arranges the image (divided image) of the n-th divided image item created in step S807 on the canvas secured in step S804 according to the overlay information acquired in step S802.

In the overlay, the split images that make up the overlay can be placed at arbitrary positions on the canvas, and the positions are described in the overlay information. However, the outside of the overlay image is out of display. Therefore, here, when arranging the divided images on the canvas, only the area overlapping the area of the overlay image, ie, the reproduction area of the divided image, is arranged among the coding areas of the divided images.

Description will be made below with reference to FIG. 13 .

The coordinate system assumes that the upper left corner of the overlay image is the origin (0, 0), the X coordinate increases rightward, and the Y coordinate increases downward.

The size of the overlay image is width Wo and height Ho (0<Wo, 0<Ho).

Therefore, the overlay image has coordinates (0, 0) in the upper left and coordinates (Wo-1, Ho-1) in the lower right.

The size of the divided image is assumed to be width Wn and height Hn (0<Wn, 0<Hn).

Let (Xn, Yn) be the upper left position of the divided image.

Therefore, the divided image has upper left coordinates (Xn, Yn) and lower right coordinates (Xn+Wn-1, Yn+Hn-1).

The overlapping area of the divided image with the overlay image (canvas) can be obtained by the following method.

The following case is a case where there is no overlapping area.
Wo-1<Xn (the left edge of the split image is to the right of the right edge of the overlay image)
Xn+Wn-1<0 (the right edge of the split image is left of the left edge of the overlay image)
Ho-1<Yn (the top edge of the split image is below the bottom edge of the overlay image)
Yn+Hn-1<0 (the bottom edge of the divided image is above the top edge of the overlay image)
In this case, this image is excluded from processing.

Also, in the following cases, the entire divided image becomes the overlapping area.
0<=Xn and (Xn+Wn-1)<=(Wo-1) and 0<=Yn and (Yn+Hn-1)<=(Ho-1)
In this case, the entire divided image is arranged at the specified position (Xn, Yn) on the canvas.

In cases other than the above, part of the divided image becomes the layout target.
Assume that the upper left coordinates of the overlapping area are (Xl, Yt) and the lower right coordinates are (Xr, Yb).
The left end Xl is as follows.
if (0<=Xn)
Xl=Xn;
else
Xl = 0;
The right end Xr is as follows.
if (Xn+Wn-1<=Wo-1)
Xr=Xn+Wn−1;
else
Xr = Wo-1;
The upper end Yt is as follows.
if (0<=Yn)
Yt=Yn;
else
Yt = 0;
The lower end Yb is as follows.
if (Yn+Hn-1<=Ho-1)
Yb = Yn + Hn-1;
else
Yb = Ho-1;
The size of this overlapping region is Xr-Xl+1 in width and Yb-Yt+1 in height.

The upper left coordinates (Xl, Yt) and lower right coordinates (Xr, Yb) of this overlapping area are, as described above, a coordinate system with the upper left corner of the canvas as the origin (0, 0).

The upper left coordinates (Xl, Yt) of the overlapping area are expressed in a coordinate system with the origin at the upper left of the divided image as follows.
Xl′=Xl−Xn;
Yt′=Yt−Yn;

In summary, it is as follows.

A rectangle with a width of (Xr-Xl+1) and a height of (Yb-Yt+1) is cut out from a position (Xl', Yt') away from the upper left of the divided image, and placed at a position of (Xl, Yt) on the canvas.

Thereby, it is possible to arrange the reproduction area of the divided image at an appropriate position on the canvas.

In step S809, the CPU 101 adds 1 to the divided image counter n, and returns to step S806.

In step S810, the CPU 101 checks whether the rotation property 'irot' exists in the properties of the overlay image acquired in step S801, and if it exists, checks the rotation angle. If the rotation property does not exist, or if the rotation property exists but the rotation angle is 0, the process ends. If the rotation property exists and the rotation angle is other than 0, the process proceeds to step S811.

In step S811, the CPU 101 rotates the canvas image created in steps S806 to S809 by the angle obtained in step S810, and uses it as a created image.

This makes it possible to create an overlay image.

In the above description, the processing of steps S806 to S809 is repeated in a loop for the number of divided images. good.

Next, image item property acquisition processing will be described with reference to FIG.

In step S901, the CPU 101 searches for the entry of the designated image item ID from the item property related box within the item property box within the metadata box, and obtains the property index array stored therein.

At step S902, the CPU 101 initializes an array counter n to zero.

At step S903, the CPU 101 checks whether the array counter n is equal to the number of array elements. If equal, exit. If not, proceed to step S904.

In step S904, the CPU 101 acquires the index property of the n-th element of the array from the item property container box within the item property box within the metadata box.

In step S905, the CPU 101 adds 1 to the array counter n, and returns to step S903.

Next, image item data acquisition processing will be described with reference to FIG.

In step S1001, CPU 101 searches the entry for the specified image item ID from the item location box within the metadata box and obtains the offset criterion (construction_method), offset, and length.

In step S1002, CPU 101 checks the offset reference acquired in step S1001. The offset reference has a value of 0 representing an offset from the beginning of the file and a value of 1 representing an offset within the item data box. If the offset reference is 0, the process proceeds to step S1003. If the offset reference is 1, go to step S1004.

In step S1003, the CPU 101 reads the length bytes from the head of the file to the memory 102 from the offset byte position.

In step S1004, the CPU 101 reads into the memory 102 the length bytes from the offset byte position from the beginning of the data portion of the item data box in the metadata box.

Next, with reference to FIG. 11, the item ID acquisition processing of the images forming the main image will be described.

In step S1101, the CPU 101 searches the item reference box in the metadata box for an entry whose reference type is 'ding' and whose referencing item ID is the item ID of the main image.

In step S1102, the CPU 101 obtains an array of reference item IDs of the entry obtained in step S1101.

Next, image creation processing for one encoded image item will be described with reference to FIG.

In step S1201, CPU 101 acquires properties of the image item. This has already been explained using FIG.

In step S1202, CPU 101 acquires the data of the image item. This has already been explained using FIG. Image item data is encoded image data.

In step S1203, the CPU 101 initializes the decoder using the decoder configuration/initialization data among the properties acquired in step S1201.

In step S1204, the CPU 101 decodes the encoded image data obtained in step S1202 using a decoder to obtain the decoding result.

In step S1205, CPU 101 checks whether pixel format conversion is required.

Exit if not needed. If necessary, go to step S1206.

Pixel format conversion is required when the pixel format of the output data of the decoder and the pixel format of the image supported by the display device are different.

For example, if the pixel format of the output data of the decoder is YCbCr (luminance-chrominance) format and the pixel format of the image of the display device is RGB format, pixel format conversion from YCbCr format to RGB format is required. Pixel format conversion is also required for the same YCbCr format, but with different bit depths (8 bits, 10 bits, etc.) and color difference samples (4:2:0, 4:2:2, etc.).

The matrix_coefficients of the color space information property 'colr' specifies the coefficients for conversion from RGB format to YCbCr format. good.

In step S1206, CPU 101 converts the decoder output data obtained in step S1204 into a desired pixel format.

This makes it possible to create an image of the specified encoded image item.

Through the above processing, the HDR image recorded in the HEIF format as shown in FIG. 2 on the recording medium 120 can be reproduced.

<Other embodiments>
Although the present invention has been described above based on the embodiments, the present invention is not limited to these embodiments, and various forms within the scope of the present invention are also included in the present invention. .

Further, the function of the above embodiment may be used as a control method, and the control method may be executed by the image processing apparatus. Alternatively, a program having the functions of the above embodiments may be used as a control program, and the control program may be executed by a computer provided in the image processing apparatus. Note that the control program is recorded in, for example, a computer-readable storage medium.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or storage medium. It can also be realized by processing in which one or more processors in the computer of the system or apparatus read and execute the program. It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

Claims (22)

An imaging device that captures an image with an imaging sensor and records the captured image,
encoding means for encoding image data captured by the imaging sensor;
An area larger than a reproduction area to be reproduced is set as an encoding area in the image data obtained by shooting with the imaging sensor, and the image data in the encoding area is encoded by the encoding means, a control means for controlling recording of encoded image data on a recording medium in a predetermined recording format ;
The control means sets a part of a sensor image area from which image data can be acquired by the imaging sensor as the encoding area, and encodes the image data of the encoding area by the encoding means. control like
In the sensor image area, there are restrictions on the positions where encoding can be started/finished,
The control means sets a start position and an end position of the coding region based on the reproduction region and a position where the coding can be started/finished in the sensor image region.
An imaging device characterized by: The control means divides the image data of the encoding region into a plurality of divided image data, encodes the plurality of divided image data by the encoding means, and converts the encoded divided images into a plurality of divided images. 2. The imaging apparatus according to claim 1, wherein control is performed to record data as one image file on said recording medium in a predetermined recording format. Having setting means for setting the recording size,
the control means determines the reproduction area and the encoding area according to the recording size set by the setting means;
2. The imaging apparatus according to claim 1, wherein the size of said reproduction area is an area corresponding to said set recording size. the predetermined recording format is a recording format capable of designating an area to be played back from image data to be recorded;
2. The imaging apparatus according to claim 1, wherein said control means designates said reproduction area smaller than said encoded area to be recorded as an area to be reproduced. The control means encodes the thumbnail image data corresponding to the image data in the reproduction area by the encoding means, and encodes the encoded thumbnail image data together with the encoded plurality of divided image data in the predetermined recording. 2. The imaging apparatus according to claim 1, wherein control is performed to record as one image file in a format. 2. The imaging apparatus according to claim 1, wherein said encoding means compression-encodes said image data according to the HEVC (High Efficiency Video Coding) standard. 2. The imaging apparatus according to claim 1, wherein the predetermined recording format is HEIF (High Efficiency Image File Format). The predetermined recording format is HEIF (High Efficiency Image File Format), and the control means records the plurality of divided image data as image items, respectively, and records the plurality of divided image data. 3. The imaging device according to claim 2, wherein the image is recorded as a derived image item . 9. The imaging apparatus according to claim 8 , wherein said control means records said plurality of divided image data as derived image items in an overlay format. 10. The imaging apparatus according to claim 8 , wherein the control means designates the reproduction area so that only the reproduction area is reproduced in the derived image item. The control means controls position information of each of the plurality of divided image data so that the plurality of divided image data are arranged in the derived image item so as to have a positional relationship corresponding to the encoding region. 11. The imaging apparatus according to any one of claims 8 to 10 , wherein control is performed so as to record . 12. The imaging apparatus according to claim 11 , wherein said control means records the position information of each of said plurality of divided image data so that said plurality of divided image data do not overlap each other. 13. The imaging apparatus according to any one of claims 8 to 12, wherein said control means records said derived image item as a main image in said predetermined recording format. 14. The restriction according to any one of claims 1 to 13, wherein the position where encoding can be started in the sensor image area is a position in units of a predetermined number of pixels in the width or height direction of the sensor image area. 1. The imaging device according to item 1. In the constraint, a position in units of a predetermined number of pixels in each of the width and height directions of the sensor image area is predetermined as the position where the encoding can be started;
The predetermined number of pixels is different in the width or height direction of the sensor image area,
15. The imaging apparatus according to claim 14, characterized by: 16. The imaging apparatus according to claim 15, wherein the predetermined number of pixels is larger in the width direction than in the height direction. The control means sets a position obtained by multiplying a quotient obtained by dividing a start position of the reproduction area in the sensor image area by the predetermined number of pixels by the predetermined number of pixels as a start position of the encoding area. 15. The imaging device according to claim 14, characterized by: When the start position of the reproduction area in the sensor image area is a position where encoding can be started in the sensor image area, the control means changes the start position of the reproduction area to the start position of the encoding area. 18. The imaging apparatus according to any one of claims 1 to 17, characterized in that it is set to . wherein the constraints include a start position constraint that is a constraint on a position where encoding can be started in the sensor image area, and an encoding size constraint that is a constraint on the width/height of the encoding area. 19. An imaging device according to any one of claims 1 to 18. The control means sets the start position of the encoding region based on the start position of the reproduction region and the position where encoding can be started, and sets the start position of the encoding region and the encoding size constraint. 20. The imaging apparatus according to claim 19, wherein the end position of said encoding region is set from . A control method for a recording device that records an image captured by an imaging sensor, comprising:
an encoding step of encoding the image data obtained by the photographing;
An area larger than a reproduction area to be reproduced is set as an encoding area in the image data obtained by shooting, and the image data in the encoding area is encoded in the encoding step, and the encoded image data is encoded. a control step of controlling the image data to be recorded on a recording medium in a predetermined recording format;
In the control step, a part of a sensor image region from which image data can be acquired by the imaging sensor is set as the encoding region, and the image data of the encoding region is encoded in the encoding step. control like
In the sensor image area, there are restrictions on the positions where encoding can be started/finished,
In the control step, in the sensor image area, a start position and an end position of the encoding area are set based on the reproduction area and a position where the encoding can be started/finished.
A method of controlling a recording apparatus, characterized by: A program that causes a computer to function as each means of the imaging apparatus according to any one of claims 1 to 20 .

Images