JP2019145918A - Imaging apparatus, display control device and control method for display control device - Google Patents

Abstract

To record an image in a recording form suitable for recording and reproduction.SOLUTION: An imaging apparatus comprises; encoding means that encodes image data obtained by photographing with an imaging sensor; and recording control means by which image data in an encoding area larger than a reproduction area, as a reproduction target, among the image data obtained by photographing with the imaging sensor are encoded with the encoding means and the plurality of encoded image data are recorded on a recording medium in a predetermined recording format.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an imaging device and a display control device display control device control method.

  An imaging apparatus is known as an image processing apparatus that compresses and encodes image data. Such an image processing apparatus acquires a moving image signal by the imaging unit, compresses and encodes the acquired moving image signal, and records the image file subjected to the compression encoding on a recording medium. Conventionally, image data before compression encoding has been expressed in SDR (Standard Dynamic Range) with a luminance level of 100 nits as an upper limit. However, in recent years, image data having a luminance range that is expressed by HDR (High Dynamic Range) in which the upper limit of the luminance level is expanded to about 10,000 nits and that is close to the luminance range that can be perceived by humans has been provided.

  Patent Document 1 describes an image data recording apparatus that generates and records image data capable of confirming an HDR image in detail even when an HDR image is captured and recorded when an HDR image is captured and recorded.

JP 2017-139618 A

  Patent Document 1 describes recording an HDR image, but does not consider an optimum recording method for recording an HDR image.

  Therefore, the present invention provides an apparatus for recording an image in a recording format suitable for recording and reproduction when recording an image having a large amount of data such as an HDR image, and also reproduces an image recorded in the recording format. An object of the present invention is to provide a display control device for performing the above.

  The present invention also relates to an apparatus for encoding and recording an image, and an apparatus for recording an image in a recording form suitable for recording and reproduction, even if there are various restrictions at the time of encoding. An object of the present invention is to provide a display control device for reproducing a recorded image.

In order to solve the above-described problem, an imaging apparatus according to the present invention includes:
An imaging device that captures and records an image, the imaging sensor, an encoding unit that encodes image data obtained by imaging with the imaging sensor, and image data obtained by imaging with the imaging sensor A recording control means for encoding image data in an encoding area larger than a reproduction area to be reproduced by an encoding means, and controlling the encoded plural image data to be recorded on a recording medium in a predetermined recording format; It is characterized by having.

The display control device of the present invention is
The image data of the encoding area larger than the reproduction area to be reproduced is divided into a plurality of divided image data, and the plurality of divided image data is encoded by the encoding means, respectively. Decoding means for reading out an image file from a recording medium recorded as one image file in a predetermined recording format, a composite means for combining a plurality of divided image data contained in the read image file, and decoding by a decoding means Display control means for combining the plurality of divided image data into one image data and controlling the display means to display the image data in the reproduction area among the combined image data. .

  According to the present invention, when an image with a large amount of data and a high resolution is recorded, an image pickup apparatus that records an image in a recording form suitable for recording and reproduction, and an image recorded in the recording form are recorded. A display control device for reproduction can be provided.

  Alternatively, according to the present invention, in an apparatus for encoding and recording an image, even if there are various restrictions during encoding, the apparatus for recording an image in a recording form suitable for recording and reproduction, and the recording form A display control device for reproducing a recorded image can be provided.

1 is a block diagram illustrating a configuration of an imaging apparatus 100. FIG. The figure which shows the structure of a HEIF file. The flowchart which shows the process in HDR imaging | photography mode. 9 is a flowchart showing processing for determining a method for dividing image data in the HDR shooting mode. The figure which shows the encoding area | region and division | segmentation method of image data at the time of HDR image data recording. The flowchart which shows the process which builds a HEIF file. The flowchart which shows the display process at the time of reproducing | regenerating the HDR image data recorded as a HEIF file. The flowchart which shows an overlay image creation process. The flowchart which shows the property acquisition process of an image item. The flowchart which shows the data acquisition process of an image item. The flowchart which shows the item ID acquisition process of the image which comprises a main image. The flowchart which shows the image creation process of 1 image item. The figure which shows the positional relationship of an overlay image and a divided image.

  Embodiments of the present invention will be described below in detail with reference to the accompanying drawings, taking the imaging apparatus 100 as an example, but the present invention is not limited to the following embodiments.

<Configuration of imaging device>
FIG. 1 is a block diagram illustrating the imaging apparatus 100. As illustrated in FIG. 1, the imaging device 100 includes a CPU 101, a memory 102, a nonvolatile memory 103, an operation unit 104, an imaging unit 112, an image processing unit 113, an encoding processing unit 114, and display control. Part 115 and a display part 116. Furthermore, the imaging apparatus 100 includes a communication control unit 117, a communication unit 118, a recording medium control unit 119, and an internal bus 130. The imaging device 100 forms an optical image of a subject on the pixel array of the imaging unit 112 using the photographing lens 111, but the photographing lens 111 is not removable from the body (housing, main body) of the imaging device 100. It may be detachable. In addition, the imaging apparatus 100 writes and reads image data to and from the recording medium 120 via the recording medium control unit 119. The recording medium 120 may be detachable from the imaging apparatus 100 or may not be detachable.

  The CPU 101 controls the operation of each unit (each functional block) of the imaging apparatus 100 via the internal bus 130 by executing a computer program stored in the nonvolatile memory 103.

  The memory 102 is a rewritable volatile memory. The memory 102 temporarily records a computer program for controlling the operation of each unit of the imaging apparatus 100, information such as parameters relating to the operation of each unit of the imaging apparatus 100, information received by the communication control unit 117, and the like. The memory 102 temporarily records the image acquired by the imaging unit 112, the image processed by the image processing unit 113, the encoding processing unit 114, and the like and information. The memory 102 has a sufficient storage capacity for temporarily recording these.

  The nonvolatile memory 103 is an electrically erasable and recordable memory, and for example, an EEPROM or the like is used. The nonvolatile memory 103 stores information such as a computer program that controls the operation of each unit of the imaging apparatus 100 and parameters related to the operation of each unit of the imaging apparatus 100. Various operations performed by the imaging apparatus 100 are realized by the computer program.

  The operation unit 104 provides a user interface for operating the imaging device 100. The operation unit 104 includes various buttons such as a power button, a menu button, and a shooting button, and the various buttons include switches, a touch panel, and the like. The CPU 101 controls the imaging device 100 in accordance with a user instruction input via the operation unit 104. Here, the case where the CPU 101 controls the imaging apparatus 100 based on an operation input via the operation unit 104 has been described as an example, but the present invention is not limited to this. For example, the CPU 101 may control the imaging device 100 based on a request input via a communication unit 118 from a remote controller (not shown), a portable terminal (not shown), or the like.

  The photographing lens (lens unit) 111 includes a lens group (not shown) including a zoom lens, a focus lens, and the like, a lens control unit (not shown), a diaphragm (not shown), and the like. The photographing lens 111 can function as a zoom unit that changes the angle of view. The lens control unit performs focus adjustment and aperture value (F value) control by a control signal transmitted from the CPU 101. The imaging unit 112 can function as an acquisition unit that sequentially acquires a plurality of images constituting a moving image. As the imaging unit 112, for example, an area image sensor such as a CCD (charge coupled device), a CMOS (complementary metal oxide semiconductor) device or the like is used. The imaging unit 112 has a pixel array (not shown) in which photoelectric conversion units (not shown) that convert an optical image of a subject into an electrical signal are arranged in a matrix, that is, two-dimensionally. An optical image of the subject is formed on the pixel array by the photographing lens 111. The imaging unit 112 outputs the captured image to the image processing unit 113 or the memory 102. Note that the imaging unit 112 can also acquire a still image.

  The image processing unit 113 performs predetermined image processing on the image data output from the imaging unit 112 or the image data read from the memory 102. Examples of the image processing include interpolation processing, reduction processing (resizing processing), color conversion processing, and the like. Further, the image processing unit 113 uses the image data acquired by the imaging unit 112 to perform predetermined calculation processing for exposure control, distance measurement control, and the like. Based on the calculation result obtained by the calculation processing by the image processing unit 113, the CPU 101 performs exposure control, distance measurement control, and the like. Specifically, the CPU 101 performs AE (automatic exposure) processing, AWB (auto white balance) processing, AF (autofocus) processing, and the like.

  The encoding processing unit 114 compresses the size of the image data by performing intra-frame prediction encoding (intra-screen prediction encoding), inter-frame prediction encoding (inter-frame prediction encoding), and the like on the image data. . The encoding processing unit 114 is an encoding device configured with, for example, a semiconductor element. The encoding processing unit 114 may be an encoding device provided outside the imaging device 100. The encoding processing unit 114 is, for example, H.264. The encoding process is performed according to the H.265 (ITU H.265 or ISO / IEC 23008-2) method.

  The display control unit 115 controls the display unit 116. The display unit 116 includes a display screen (not shown). The display control unit 115 generates an image that can be displayed on the display screen of the display unit 116 by performing resizing processing, color conversion processing, and the like on the image data, and the image, that is, the image signal is displayed on the display unit 116. Output. The display unit 116 displays an image on the display screen based on the image signal sent from the display control unit 115. The display unit 116 has an OSD (On Screen Display) function that is a function for displaying a setting screen such as a menu on the display screen. The display control unit 115 can output the image signal to the display unit 116 by superimposing the OSD image on the image signal. The display unit 116 is configured by a liquid crystal display, an organic EL display, or the like, and displays the image signal sent from the display control unit 115. The display unit 116 may be a touch panel, for example. When the display unit 116 is a touch panel, the display unit 116 can also function as the operation unit 104.

  The communication control unit 117 is controlled by the CPU 101. The communication control unit 117 generates a modulation signal conforming to a wireless communication standard determined in advance by IEEE 802.11 or the like, and outputs the modulation signal to the communication unit 118. In addition, the communication control unit 117 receives a modulated signal conforming to the wireless communication standard via the communication unit 118, decodes the received modulated signal, and outputs a signal corresponding to the decoded signal to the CPU 101. The communication control unit 117 includes a register for storing communication settings. The communication control unit 117 can adjust the transmission / reception sensitivity during communication under the control of the CPU 101. The communication control unit 117 can perform transmission / reception with a predetermined modulation method. The communication unit 118 includes an antenna that outputs a modulation signal supplied from the communication control unit 117 to an external device 127 such as an information communication device outside the imaging apparatus 100 and receives a modulation signal from the external device 127. ing. The communication unit 118 includes a communication circuit and the like. Here, the case where wireless communication is performed by the communication unit 118 has been described as an example, but the communication performed by the communication unit 118 is not limited to wireless communication. For example, the communication unit 118 and the external device 127 may be connected by electrical connection using wiring or the like.

  The recording medium control unit 119 controls the recording medium 120. The recording medium control unit 119 outputs a control signal for controlling the recording medium 120 to the recording medium 120 based on a request from the CPU 101. As the recording medium 120, for example, a nonvolatile memory or a magnetic disk is used. As described above, the recording medium 120 may be detachable or non-detachable. The recording medium 120 records encoded image data and the like. Image data or the like is saved as a file in a format compatible with the file system of the recording medium 120. Examples of the file include an MP4 file (ISO / IEC 14496-14: 2003), an MXF (Material eXchange Format) file, and the like. Each of the functional blocks 101 to 104, 112 to 115, 117, and 119 can access each other via the internal bus 130.

  Here, the normal operation of the imaging apparatus 100 of the present embodiment will be described.

  When the user operates the power button of the operation unit 104, the imaging apparatus 100 issues an activation instruction to the CPU 101 from the operation unit 104. In response to this instruction, the CPU 101 controls a power supply unit (not shown) to supply power to each block of the imaging apparatus 100. When the power is supplied, the CPU 101 checks, for example, which mode the mode selector switch of the operation unit 104 is in, for example, a still image shooting mode, a playback mode, or the like by an instruction signal from the operation unit 102.

  In the normal still image shooting mode, the imaging apparatus 100 performs shooting processing when the user operates the still image recording button of the operation unit 104 in a shooting standby state. In the shooting process, image data of a still image shot by the imaging unit 112 is subjected to image processing by the image processing unit 113, encoded by the encoding processing unit 114, and encoded by the recording medium control unit 119. Is recorded on the recording medium 120 as an image file. In the imaging standby state, an image is captured at a predetermined frame rate by the imaging unit 112, image processing for display is performed by the image processing unit, and the display control unit 115 displays the image on the display unit 116, thereby displaying the live view image. Display.

  In the reproduction mode, the image file recorded on the recording medium 120 is read by the recording medium control unit 119, and the image data of the read image file is decoded by the encoding processing unit 114. That is, the encoding processing unit 114 also has a decoder function. Then, the image processing unit 113 performs display processing, and the display control unit 115 causes the display unit 116 to display.

  Although normal still image shooting and playback are performed as described above, the imaging apparatus of the present embodiment has an HDR shooting mode for shooting not only normal still images but also HDR still images. In addition, the captured HDR still image can be reproduced.

  Hereinafter, HDR still image shooting and playback processing will be described.

<File structure>
First, the file structure for recording an HDR still image will be described.

  Recently, a still image file format called High Efficiency Image File Format (hereinafter referred to as HEIF) has been developed. (ISO / IEC 23008-12: 2017)

This has the following characteristics as compared with a conventional still image file format such as JPEG.
A file format compliant with the ISO base media file format (hereinafter referred to as ISOBMFF). (ISO / IEC 14496-14: 2003)
-Not only a single still image but also a plurality of still images can be stored.
・ HEVC / H. H.265 and AVC / H. Still images compressed in a compression format used for compressing moving images such as H.264 can be stored.

  In this embodiment, HEIF is adopted as a recording file for HDR still images.

  First, data stored in the HEIF will be described.

  HEIF manages individual data to be stored in units of items.

  Each item has, in addition to the data itself, an integer item ID (item_ID) that is unique within the file and an item type (item_type) that represents the item type.

  Items can be divided into image items in which data represents images, and metadata items in which data is metadata (metadata items).

  The image item includes a coded image item (coded image item) that is image data obtained by encoding data, and a derived image item that represents an image obtained as a result of operating one or more other image items. )

  An example of the derived image item is an overlay image that is a derived image item in an overlay format. This is an overlay image obtained as a result of combining an arbitrary number of image items at an arbitrary position using an ImageOverlay structure (overlay information).

  Exif data can be stored as an example of a metadata item.

  As described above, the HEIF can store a plurality of image items.

  If there are relationships between multiple images, they can be described.

  Examples of the relationship between the plurality of images include a relationship between the derived image item and the image item constituting the derived image item, a relationship between the main image and the thumbnail image, and the like.

  Further, the relationship between the image item and the metadata item can be similarly described.

  The HEIF format conforms to the ISOBMFF format. Therefore, first, ISOBMFF will be briefly described.

  The ISOBMFF format manages data in a structure called a box.

  The box is a data structure that starts with a 4-byte data length field and a 4-byte data type field, and thereafter has arbitrary length data.

  The structure of the data part is determined by the data type. In the ISOBMFF specification and HEIF specification, several data types and structures of data portions thereof are defined.

  A box may have another box as data. That is, boxes can be nested. Here, a box nested in the data portion of a certain box is called a sub-box.

  A box that is not a sub-box is called a file-level box. This is a box that can be accessed sequentially from the beginning of the file.

  The HEIF format file will be described with reference to FIG.

  First, the file level box will be described.

  Information regarding file compatibility is stored in a file type box whose data type is 'ftyp'. The file specification conforming to ISOBMFF declares the structure of the file defined by the specification and the data stored in the file with a 4-byte code called a brand, and stores them in the file type box. By arranging the file type box at the beginning of the file, the reader of the file can check the contents of the file type box and know the structure of the file without further reading and interpreting the contents of the file.

  In the HEIF specification, the file structure is represented by the brand 'mif1'. In addition, when the encoded image data to be stored is HEVC compressed data, it is represented by the brand 'heic' or 'heix' according to the HEVC compression profile.

  The metadata box whose data type is 'meta' includes various sub boxes and stores data regarding each item. The breakdown will be described later.

  In the media data box whose data type is 'mdat', data of each item is stored. For example, encoded image data of an encoded image item and Exif data of a metadata item.

  Next, sub-boxes of the metadata box will be described.

  The handler box whose data type is “hdlr” stores information indicating the type of data managed by the metadata box. In the HEIF specification, handler_type of the handler box is “pict”.

  The data information box whose data type is “dinf” designates a file in which data targeted by this file exists. In ISOBMFF, data targeted by a certain file can be stored in a file other than that file. In that case, the data entry URL box describing the URL of the file in which the data exists is stored in the data reference in the data information box. When the target data exists in the same file, a data entry URL box having only a flag representing it is stored.

  The primary item box whose data type is 'pitm' stores the item ID of the image item representing the main image.

  The item information box whose data type is 'iinf' is a box for storing the following item information entry.

  The item information entry whose data type is “inf” stores the item ID, item type, and flag of each item.

  The item type of an image item whose encoded image data is HEVC compressed data is 'hvc1', which is an image item.

  The derived image item in the overlay format, that is, the item type of the ImageOverlay structure (overlay information) is 'iovl', and is classified as an image item.

  The item type of the metadata item in the Exif format is “Exif”, which is a metadata item.

  Further, when the least significant bit of the flag field is set in the image item, it can be specified that the image item is a hidden image. When this flag is set, the image item is not treated as a display target during reproduction and is not displayed.

  The item reference box whose data type is “iref” stores the reference relationship between the items by the type of the reference relationship, the item ID of the reference source item, and the item ID of one or more reference destination items.

  In the case of the derived image item in the overlay format, the type of reference relationship is “dimg”, the item ID of the reference source item is the item ID of the derived image item, the item ID of the reference destination item is the item ID of each image item constituting the overlay, Is stored.

  In the case of a thumbnail image, 'thmb' is stored as the type of reference relationship, the item ID of the thumbnail image is stored as the item ID of the reference source item, and the item ID of the main image is stored as the item ID of the reference destination item.

  The item property box whose data type is 'iprp' is a box for storing the following item property container box and item property related box.

  The item property container box whose data type is 'ipco' is a box for storing individual property data boxes.

  Each image item can have property data representing the characteristics and attributes of the image.

  The property data box includes the following.

  Decoder configuration and initialization data (Decoder configuration and initialization. In the case of HEVC, the type is 'hvcC') is data used for initialization of the decoder. HEVC parameter set data (VideoParameterSet, SequenceParameterSet, PictureParameterSet) is stored here.

  Image spatial extents (type is 'ispe') is the size (width, height) of the image.

  Color space information (Color information, type “color”) is color space information of an image.

  Image rotation information (Image rotation, type “irot”) is the rotation direction when rotating and displaying an image.

  Image pixel information (type is 'pixi') is information indicating the number of bits of data constituting the image.

  In this embodiment, master display color volume information (Mastering display color volume. Type is 'MDCV') and content light level information (Contents light level information. Type is 'CLLI') are used as HDR metadata. Stored as a data box.

  There are other properties, but they are not specified here.

  The item property association box whose data type is 'ipma' stores the association between each item and the property in the form of an item ID and an array of indices in the associated property 'ipco'.

  The item data box whose data type is 'idat' is a box for storing item data having a small data size.

  The data of the derived image item in the overlay format can store an ImageOverlay structure (overlay information) that stores the position information of the component image and the like in 'idat'. The overlay information includes a canvas_fill_value parameter that is background color information, and an output_width, output_height parameter that is a final reproduced image size after composition by overlay. Furthermore, for each image that is a composite element, there are horizontal_offset and vertical_offset parameters that indicate the arrangement position in the composite image and the horizontal position coordinate and vertical position coordinate. By using these parameters included in the overlay information, it is possible to reproduce a composite image in which a plurality of images are arranged at arbitrary positions in one image having a designated background color and size.

  The item location box whose data type is 'iloc' stores the position information of the data of each item in the form of an offset reference (construction_method), an offset value from the offset reference, and a length.

  The offset reference is the beginning of the file or 'idat'.

  There are other boxes in the metadata box, but they are not specified here.

FIG. 2 illustrates a structure in which two encoded image items constitute an overlay image. The number of encoded image items constituting an overlay format image is not limited to two. When the number of encoded image items constituting an overlay image increases, the following boxes and items increase accordingly.
・ Item information entry of encoded image item is added to item information box.
The item ID of the encoded image item is increased to the reference item ID of the reference relation type “dimg” in the item reference box.
Decoder configuration / initialization data, image space range, etc. in item property container box.
-Added item of encoded image item and related property index to item property related box.
-The item location information item is added to the item location box.
-Encoded image data is added to the media data box.

<Shooting HDR images>
Next, processing in the imaging apparatus 100 when shooting and recording an HDR image will be described.

  The width and height of images to be recorded are increasing recently, but if an excessively large size is encoded, compatibility is lost when decoding with other models, and the scale of the system increases. Specifically, in this embodiment, H.264 is used for encoding / decoding. An example using H.265 will be described. H. H.265 has parameters such as Profile and Level as standards, and these parameters change as the image encoding method and the image size increase. These values are parameters for determining whether or not playback is possible in the device at the time of decoding. In some cases, this parameter is determined and played as being unplayable. In this embodiment, when a large image size is recorded, a method of dividing the image into a plurality of image display areas, storing the code data in a HEIF file and managing them using an overlay method according to the HEIF standard will be described. By dividing the image display area into a plurality of image display areas in this way, the size of each image display area is reduced, and reproduction compatibility with other models is improved. In the following description, it is assumed that the size of one side of the coding area is 4096 or less. The encoding format is H.264. It may be other than 265, and the size of one side may be defined to a size other than 4096.

  Next, alignment restrictions regarding the start position of the encoding area and the width / height of the encoding area, and the start position of the reproduction area and the width / height of the reproduction area will be described. As an imaging device, when encoding a divided image, vertical / horizontal encoding start alignment of an encoding region, reproduction width / height alignment at the time of reproduction, and the like generally exist as hardware constraints. That is, it is necessary to calculate and encode the encoding start position of each of one or more encoding areas, the start position, width, and height of the reproduction area. Since the imaging apparatus switches and records a plurality of image sizes in response to an instruction from the user, it is necessary to switch them for each image size.

  The encoding processing unit 114 described in the present embodiment has alignment restrictions on the start position in units of 64 pixels in the horizontal direction (hereinafter referred to as x direction) and in units of 1 pixel in the vertical direction (hereinafter referred to as y direction) during encoding. In addition, there are restrictions on the width and height alignment restrictions for encoding in units of 32 pixels and 16 pixels, respectively. The start position of the reproduction area has an alignment constraint of a start position of 2 pixels in the x direction and 1 pixel in the y direction. In addition, the alignment restrictions on the width and height of the reproduction area are limited in units of two pixels and one pixel, respectively. In addition, although the example about the alignment constraint was described here, generally other alignment constraints may be used.

  Next, a flow from shooting HDR image data in the HDR recording mode (HDR shooting mode) to recording the shot HDR image data as a HEIF file will be described with reference to FIG. In the HDR mode, the image capturing unit 112 and the image processing unit 113 use the color gamut BT. 2020 and processing for expressing the HDR color space using the PQ gamma curve. In this embodiment, recording for HDR is omitted.

  In S101, the CPU 101 determines whether or not the user has pressed SW2. When the user captures the subject and presses down to SW2 (YES in S101), the CPU 101 detects that SW2 has been pressed, and proceeds to S102. The CPU 101 continues this detection until SW2 is pressed (NO in S101).

  In S <b> 102, the CPU 101 determines the number of divisions of the encoding area in the vertical and horizontal directions (number of divisions N) and how to divide the image so that the image of the angle of view captured by the user is saved in the HEIF file. This result is stored in the memory 102. S102 will be described later with reference to FIG. The process proceeds to S103.

  In S103, the CPU 101 initializes a variable M in the memory 102 to M = 1. Thereafter, the process proceeds to S104.

  In S104, the CPU 101 monitors whether the number of divisions N determined in S102 has been looped. If the number of loops reaches the division number N (YES in S104), the process proceeds to S109, otherwise (NO in S104), the process proceeds to S105.

  In S <b> 105, the CPU 101 reads out information on the code start area and the reproduction area of the divided image M from the previous memory 102. The process proceeds to S106.

  In S <b> 106, the CPU 101 informs the encoding processing unit 114 which region of the entire image arranged in the memory 102 is to be encoded and performs encoding. The process proceeds to S107.

  In S <b> 107, the CPU 101 temporarily stores in the memory 102 code data resulting from encoding by the encoding processing unit 114 and accompanying information generated during encoding. Specifically H. In the example of H.265, an H.264 such as VPS, SPS, or PPS that must be stored in the HEIF file later. It is H.265 standard information. This is omitted because it is not directly related to the present embodiment. VPS, SPS, and PPS of H.265 are various pieces of information necessary for decoding such as code data size, bit depth, display / non-display area designation, and frame information. Thereafter, the process proceeds to S108.

  In S108, the CPU 101 increments the variable M and returns to S104.

  In S109, the CPU 101 generates metadata to be stored in the HEIF file. The CPU 101 extracts information necessary for reproduction from the imaging unit 112, the image processing unit 113, the encoding processing unit 114, and the like, and temporarily stores the information in the memory 102 as metadata. Specifically, it is data formed in the Exif format. Since the details of Exif data are already general, the description in this embodiment will be omitted. Proceed to S110.

  In S110, the CPU 101 encodes the thumbnail. The entire image in the memory 102 is reduced by the image processing unit 113 to a thumbnail size and temporarily arranged in the memory 102. The CPU 101 instructs the encoding processing unit 114 to encode the thumbnail images in the memory 102. Here, the thumbnail is encoded with a size that satisfies the alignment constraint described above, but a method in which the encoding area and the reproduction area are different may be used. This thumbnail is also H.264. It is encoded with H.265. Proceed to S111.

  In S <b> 111, the CPU 101 temporarily stores in the memory 102 code data obtained as a result of encoding the thumbnail by the encoding processing unit 114 and accompanying information generated at the time of encoding. Similar to the description of S107 above, H.P. such as VPS, SPS, and PPS. It is H.265 standard information. Next, the process proceeds to S112.

  In S112, the CPU 101 stores various data in the memory 102 through the steps so far. The CPU 101 combines the data stored in the memory 102 in order, completes the HEIF file, and stores it in the memory 102. The flow for completing the HEIF file will be described later with reference to FIG. The process proceeds to S113.

  In step S <b> 113, the CPU 101 instructs the recording medium control unit 119 to write the HEIF file in the memory 102 to the recording medium 120. Thereafter, the process returns to S101.

  After shooting in such a step, the HEIF file is recorded on the recording medium 120.

  Next, the division method determination in S102 described above will be described with reference to FIGS. 4 and 5A. Hereinafter, the upper left of the sensor image area is defined as the origin (0, 0), the coordinates are (x, y), and the width and height are represented by w and h, respectively. The region is expressed as [x, y, w, h] by combining the start coordinate (x, y) and the width height w, h.

  In step S <b> 121, the CPU 101 acquires the recording mode at the time of shooting from the memory 102. The recording mode is a recording method in which, for example, the image size, the image aspect ratio, the image compression rate, and the like are determined. The user selects one of the recording modes and performs shooting. For example, it is a setting such as L size, 3: 2 aspect. Since the shooting angle of view varies depending on the recording mode, the shooting angle of view is determined. The process proceeds to S122.

  In S122, the start coordinate position (x0, y0) of the reproduced image is acquired from the sensor image area [0, 0, H, V]. Next, the process proceeds to S123.

  In S123, the area [x0, y0, Hp, Vp] of the reproduced image is acquired from the memory 102. Next, the process proceeds to S124.

  In S124, it is necessary to encode so as to include the area of the reproduced image. As described above, since the alignment of the code start position is an alignment of 64 pixels in the x direction, if encoding is performed so as to include the region of the reproduced image, the encoding must be performed from the position (Ex0, Ey0). In this way, the size of Lex is obtained. If this Lex offset is 0 (NO in S124), the process proceeds to S126, and if necessary (YES in S124), the process proceeds to S125.

  In S125, Lex is obtained. As described above, Lex is determined by the alignment between the reproduction image area and the encoding start position. The CPU 101 determines Lex as follows. x0 is divided by 64 pixels of the pixel alignment at the encoding start position in the horizontal direction, and the quotient is multiplied by the previous 64 pixels. Since the calculation is performed without including the remainder, the coordinates located on the left side of the start position of the reproduction image area are obtained. Therefore, Lex is obtained by the difference in the x coordinate of the encoding start position obtained by the previous calculation from the x coordinate of the reproduced image area. The process proceeds to S126.

  In S126, the coordinates of the final position of the right end of the uppermost line of the reproduced image are calculated. Let this be (xN, y0). xN is obtained by adding Hp to x0. The process proceeds to S127.

  In S127, the right end of the coding area is obtained so as to include the right end of the reproduced image. In order to obtain the right end w of the encoding region, an alignment constraint on the encoding width is necessary. As described above, the encoding width / height constraint is a multiple of 32 pixels. From the relationship between this restriction and (xN, y0), the CPU 101 calculates whether or not the Rex offset is necessary before the right end of the reproduced image. If Rex is necessary (YES in S127), the process proceeds to S128, and if not (NO in S127), the process proceeds to S129.

  In S128, the CPU 101 adds Rex to the right side of the right end coordinates (xN, y0) of the reproduced image so as to align 32 pixels, and obtains the encoding end position (ExN, Ey0). The process proceeds to S129.

  In S129, the CPU 101 obtains the lower right coordinate of the display area. Since the size of the reproduced image is self-explanatory, the coordinates are (xN, yN). Proceed to S130.

  In S130, the lower end of the encoding area is obtained so as to include the lower end of the reproduced image. In order to obtain the lower end of the encoding region, an alignment constraint on the encoding height is necessary. As described above, the encoding height constraint is a multiple of 16 pixels. From the relationship between this restriction and the lower right coordinates (xN, yN) of the reproduced image, the CPU 101 calculates whether an offset of Vex is necessary before the lower end of the reproduced image. If Vex is necessary (YES in S130), the process proceeds to S131, and if not required (NO in S130), the process proceeds to S132.

  In S131, the CPU 101 calculates Vex from the relationship between the previous restriction and the lower right coordinates (xN, yN) of the reproduced image, and obtains the lower right coordinates (ExN, EyN) of the coding area. The Vex calculation method is set to a position that is a multiple of 16 pixels as the encoding height alignment so as to include y0 to yN that are the encoding start positions in the vertical direction. Specifically, assuming that y0 = 0, Vp is divided by 16 pixels of the encoding height alignment to obtain the quotient and the remainder. Since the coding area must be taken so as to include the reproduction area, if there is a remainder, 1 is added to the quotient and the previous 16 pixels are multiplied. As a result, EyN of the y coordinate of the lower end of the encoding region including Vp is obtained while being a multiple of 16 pixels. Vex is determined by the difference between EyN and yN. Note that EyN is a value that is offset downward by Vex from yN, and ExN is a value that has already been obtained in S128. Proceed to S132.

  In S132, the CPU 101 calculates the size Hp'xVp 'of the coding area as follows. Hp ′ is obtained by adding Lex and Rex obtained from the alignment constraint to the horizontal size Hp. Further, Vp ′ obtained from the alignment constraint is added to the vertical size Vp to obtain Vp ′. The size Hp ′ × Vp ′ of the coding region is obtained from Hp ′ and Vp ′. The process proceeds to S133.

  In S133, the CPU 101 determines whether or not the horizontal size Hp ′ to be encoded exceeds 4096 pixels to be divided. If it exceeds (YES in S133), the process proceeds to S134, and if it does not exceed (NO in S133), S135. Proceed to

  In S134, the horizontal size Hp ′ to be encoded is divided into two or more regions so as to be 4096 pixels or less. For example, if it is 6000 pixels, it is divided into two, and if it is 9000 pixels, it is divided into three.

  In the case of two divisions, division is performed at an approximate center position that satisfies the alignment 32 pixels related to the previous encoding width. If it is not uniform, enlarge the left split image. Also, in the case of division into three divisions, if they are not equal, the division regions A, B, and C are separated from each other while paying attention to the alignment relating to the previous encoding width so that the sizes of the division regions A and B are equal. The divided area C is divided so as to be slightly smaller than that. The division position and size are determined by the same algorithm even if there are three or more divisions. The process proceeds to S135.

  In S135, the CPU 101 determines whether or not the vertical size Vp ′ to be encoded exceeds 4096 pixels to be divided. If it exceeds (YES in S135), the process proceeds to S136, and if not (NO in S135), the process is performed. Exit.

  In S136, Vp 'is divided into a plurality of divided areas, as in S134, and the process ends.

  An example of the division in S102 will be described with specific numerical values using FIGS. 5B, 5C, 5D, and 5E. FIG. 5 shows not only the image data dividing method but also the relationship between the encoding area and the reproduction area.

  FIG. 5B will be described. FIG. 5B shows a case in which an image is divided into two left and right regions, and Lex, Rex, and Vex are given to the left and right and lower ends of the reproduction region. First, the reproduction image area [256 + 48, 0, 4864, 3242] is an area of a final image that is to be recorded by the imaging apparatus. The upper left coordinate of the reproduced image area is (256 + 48, 0), and is not a multiple of 64 pixel units, which is the start alignment of encoding described above. Therefore, Lex is provided on the left side of 48 pixels from here, and the left end of the encoding start position is a position of x = 256. Next, the width of the reproduction area is w = 48 + 4864, but this is not a multiple of 32 pixels of the alignment of the encoding width, and the right end of the reproduction image area is considered in consideration of the width alignment at the time of encoding. 16 pixels Rex is given to. Similarly, when the vertical direction is calculated, Vex = 4 pixels. Since this reproduced image has a size in the horizontal direction exceeding 4096, the horizontal direction is divided into two at approximately the center position in consideration of the alignment of the encoding width. As a result, the encoding region of the divided image 1 is [256, 0, 2496, 3242 + 8], and [256 + 2496, 0, 2432, 3242 + 8] is obtained as the encoded image area of the divided image 2.

  FIG. 5C will be described. FIG. 5C shows a case where an image is divided into two left and right areas, and Rex and Vex are given to the right and bottom edges of the reproduction area. Since the x position of the reproduced image area is x = 0, it matches the 64 pixels that are the start alignment of the encoding area, and therefore Lex = 0. Thereafter, the calculation method is the same as described above. As a result, the encoded area of the divided image 1 is [0, 0, 2240, 32456 + 8], and the encoded image area of the divided image 2 is [2240, 0, 2144, 2456 + 8]. ] Is obtained.

  FIG. 5D will be described. FIG. 5D shows a case where an image is divided into two left and right areas, and Lex and Rex are given to the left and right of the reproduction area. The vertical size h of the reproduced image area is h = 3648, which is a multiple of 16 pixels, which is a restriction on the image height at the time of encoding, so Vex = 0. Thereafter, the calculation method is the same as that described above. As a result, the encoded area of the divided image 1 is [256 + 48, 0, 2240, 3648], and the encoded image area of the divided image 2 is [256 + 2240, 0, 2144, 3648]. ] Is obtained.

  FIG. 5D will be described. FIG. 5D shows a case where an image is divided into four areas on the top, bottom, left, and right, and Lex and Rex are given to the left and right of the reproduction area. In this case, since the coding area exceeds 4096 in both the vertical and horizontal directions, division is performed into a square shape. Thereafter, the calculation method is the same as that described above. As a result, the encoded region of the divided image 1 is [0, 0, 4032, 3008], and the encoded image region of the divided image 2 is [4032, 0, 4032, 3008]. ], [0, 3008, 4032, 2902] is obtained for the encoded image area of the divided image 3, and [4032, 3008, 4032, 2902] is obtained for the encoded image area of the divided image 4.

  In this way, the CPU 101 needs to determine the area of the reproduced image in accordance with the recording mode, and calculate the divided area for each area. As another method, these calculations are performed in advance and the results are stored in the memory 102. The CPU 101 may read information on the divided area and the reproduction area corresponding to the recording mode from the memory 102 and set the information in the encoding processing unit 114.

  When the captured image is divided into a plurality of divided images and recorded as in this embodiment, it is necessary to combine a plurality of display areas in order to restore the captured image from the recorded image. When generating a composite image, it is more efficient that there is no overlapping area in each divided image, because it is not necessary to encode and record extra data. For this reason, in this embodiment, each reproduction region and each coding region are set so that there is no overlapping portion (overlapping region) at the boundary between the divided images. However, depending on conditions such as hardware alignment constraints, an overlapping area may be provided between the divided images.

  Next, the HEIF file construction method described in S112 will be described with reference to FIG. Since the HEIF file has the structure shown in FIG. 2, the CPU 101 constructs it on the memory 102 in order from the beginning of the file.

  In step S141, the 'ftyp' box is generated. This box is used to check file compatibility and is placed at the top of the file. The CPU 101 stores 'heic' as the major brand of this box, and 'mif1' and 'heic' as the compatible brand. In this embodiment, these values are used, but other values may be used. The process proceeds to S142.

  In step S142, a 'meta' box is generated. This box is a box for storing a plurality of boxes described below. The CPU 101 generates this box and proceeds to S143.

  In S143, an 'hdlr' box to be stored in the 'iinf' box is generated. This box shows the attributes of the 'meta' box described above. The CPU 101 stores 'pict' in this type and proceeds to S144. 'Pict' is information indicating the type of data managed by the metadata box. In the HEIF specification, handler_type of the handler box is defined as 'pict'.

  In S144, a 'dinf' box to be stored in the 'iinf' box is generated. This box indicates the location of data targeted by this file. The CPU 101 generates this box and proceeds to S145.

  In S145, a 'pitm' box to be stored in the 'iinf' box is generated. This box stores the item ID of the image item representing the main image. Since the main image is an overlay image synthesized by overlay, the CPU 101 stores the overlay information item ID as the item ID of the main image. The process proceeds to S146.

  In S146, an 'iinf' box to be stored in the 'meta' box is generated. This box is a box for managing a list of items. Here, an initial value is entered in the data length field, 'iinf' is stored in the data type field, and the process proceeds to S147.

  In S147, an 'infe' box to be stored in the 'iinf' box is generated. This 'infe' box is a box for registering item information of each item stored in the file, and an 'infe' box is generated for each item. For the divided images, each divided image is registered in this box as one item. In addition, overlay information, Exif information, and thumbnail images for forming a main image from a plurality of divided images are registered as items. At this time, as described above, overlay information, divided images, and thumbnail images are registered as image items. For an image item, a flag indicating a hidden image can be added. By adding this flag, an image is not displayed during reproduction. That is, by adding this hidden image flag to the image item, it is possible to designate an image that is not displayed during reproduction. Therefore, for the divided image, a flag indicating the hidden image is set, and the flag is not set for the item of overlay information. As a result, the individual divided images themselves are excluded from the display target, and only the image resulting from the overlay synthesis of the plurality of divided images based on the overlay information is the display target. Create an 'infe' box for each item individually and store them in the previous 'iinf'. The process proceeds to S148.

  In S148, an 'iref' box to be stored in the 'iinf' box is generated. This box stores information indicating the relationship between an image (main image) composed of an overlay and divided images constituting the image. Since the division method is determined in S102 described above, the CPU 101 generates this box based on the determined division method. The process proceeds to S149.

  In S149, an 'iprp' box to be stored in the 'iinf' box is generated. This box is a box for storing the property of the item, and is a box for storing the 'ipco' box generated in S150 and the 'ipma' box generated in S151. Proceed to S150.

  In S150, an 'ipco' box to be stored in the 'iprp' box is generated. This box is an item property container box and stores various properties. There are a plurality of properties, and the CPU 101 generates the following property container box and stores it in the 'ipco' box. Some properties are generated for each image item, and some properties are generated in common for a plurality of image items. A 'colr' box is generated as information common to the overlay image and the divided image constituted by the overlay information serving as the main image. In the 'colr' box, color information such as an HDR gamma curve is stored as color space information of the main image (overlay image) and the divided image. For the divided images, an 'hvcC' box and an 'ispe' box are generated. In S107 and S111 described above, the CPU 101 reads accompanying information stored in the memory 102 at the time of encoding, and generates a property container box of 'hvcC'. The accompanying information stored in this 'hvcC' includes not only the information when the encoding area is encoded, the size (width and height) of the encoding area, but also the size (width and height) of the reproduction area, The position information of the reproduction area in the encoding area is also included. The accompanying information is Golomb-compressed and recorded in the property container box of 'hvcC'. Since Golomb compression is a general compression method, description thereof is omitted here. The 'ispe' box stores size (width, height) information (for example, 2240 × 2450) of the playback area of the divided image. An 'irot' box storing information representing rotation of the overlay image serving as the main image and a 'pixi' box indicating the number of bits of the image data are generated. The 'pixi' box may be generated separately for the main image (overlay image) and the divided image. However, in this embodiment, the overlay image and the divided image have the same number of bits, that is, 10 bits. Only generate. Also, a 'CLLI' or 'MDCV' box in which supplementary information of HDR is stored is generated. In addition to the main image (overlay image), as a thumbnail image property, a 'colr' box storing color space information, an 'hvcC' box storing information at the time of encoding, and 'ispe' storing an image size. A box, a 'pixi' box storing information on the number of bits of image data, and a 'CLLI' box storing HDR supplementary information are generated. These property container boxes are generated and stored in the 'ipco' box. The process proceeds to S151.

  In S151, an 'ipma' box is generated. This box is a box indicating the relationship between items and properties, and indicates which property is related to each item. The CPU 101 determines the relationship between items and properties from various data stored in the memory 102, and generates this box. The process proceeds to S152.

  In S152, an 'idat' box is generated. This box stores overlay information indicating how the reproduction area of each divided image is arranged to generate an overlay image. The overlay information includes a canvas_fill_value parameter that is background color information, and an output_width and output_height parameters that are the size of the entire overlay image. Further, for each divided image serving as a synthesis element, there are horizontal_offset and vertical_offset parameters indicating the horizontal position coordinates and vertical position coordinates for synthesizing the divided images. For these parameters, the CPU 101 describes these pieces of information based on the division method determined in S102. Specifically, the size of the entire playback area is described in output_width and output_height parameters indicating the size of the overlay image. In the horizontal_offset and vertical_offset parameters indicating the position information of each divided image, offset values in the width direction and the height direction from the start coordinate position (x0, y0) at the upper left of the reproduction area to the upper left position of the divided image are respectively set. Describe. By generating the overlay information in this manner, when the image is reproduced, the divided image is arranged based on the horizontal_offset and vertical_offset parameters, and the divided image is synthesized, thereby reproducing the image before the division. It becomes like this. In the present embodiment, since the overlapping area is not provided in the divided image, the position information is described so that the divided images are arranged so as not to overlap. Then, by designating the area to be displayed by the output_width and output_height parameters from the images obtained by combining the divided images, it is possible to reproduce only the reproduction area as a display target. An 'idat' box in which the overlay information generated in this way is stored is generated. The process proceeds to S153.

  In S153, an 'iloc' box is generated. This box is a box indicating where in the file various data is to be arranged. Since various information is stored in the memory 102, this box is generated from these sizes. Specifically, the information indicating the overlay is stored in the previous 'idat', which is the position and size information inside this 'idat'. The thumbnail data and the code data 12 are stored in the 'mdat' box, and this position and size information are stored. Proceed to S154.

  In S154, an 'mdat' box is generated. This box is a box including a plurality of boxes described below. The CPU 101 generates an 'mdat' box and proceeds to S155.

  In step S155, the Exif data is stored in the 'mdat' box. In step S109, since the metadata Exif is stored in the memory 102, the CPU 101 reads it from the memory 102 and adds it to the 'mdat' box. The process proceeds to S156.

  In S156, the thumbnail data is stored in the 'mdat' box. In the previous S110, since the thumbnail data is stored in the memory 102, the CPU 101 reads it from the memory 102 and adds it to the 'mdat' box. The process proceeds to S157.

  In step S157, the encoded image 1 data is stored in the 'mdat' box. In the first loop of S106, since the data of the divided image 1 is stored in the memory 102, the CPU 101 reads it from the memory 102 and adds it to the 'mdat' box. This is repeated up to the encoded image N, and all the encoded images 1-N are added to the 'mdat' box. Through such steps, the CPU 101 constructs a HEIF file.

  Through the above processing, the HDR image can be recorded on the recording medium 120 in the HEIF format as shown in FIG.

<Reproduction of HDR image>
Next, processing in the imaging apparatus 100 when reproducing an HDR image file recorded in the HEIF format on the recording medium 120 will be described. In the present embodiment, the case of reproducing with the imaging apparatus 100 will be described. However, in the case of reproducing the HDR image recorded in the HEIF format recorded on the recording medium 120 in the image processing apparatus having no imaging unit. In addition, the same processing may be realized.

  A HEIF reproduction (display) process when the main image is an overlay image will be described with reference to FIG.

  In step S <b> 701, the CPU 101 uses the recording medium control unit 119 to read the top portion of the designated file existing on the recording medium 120 into the memory 102. Then, it is checked whether a file type box having a correct structure exists in the head portion of the read file, and whether “mif1” representing HEIF exists in the brand in the file.

  If a brand corresponding to a unique file structure is recorded, the existence of the brand is checked. As long as the brand guarantees a specific structure, it is possible to omit some subsequent structural checks, eg, step S703, step S704, by checking this brand.

  In step S <b> 702, the CPU 101 reads out the metadata box of the designated file from the recording medium 120 to the memory 102.

  In step S703, the CPU 101 searches for the handler box from the metadata box read in step S702, and checks the structure. For HEIF, the handler type must be 'pict'.

  In step S704, the CPU 101 searches for the data information box from the metadata box read in step S702, and checks the structure. In this embodiment, since it is premised that data exists in the same file, it is checked that a flag indicating the time is set in the data entry URL box.

  In step S705, the CPU 101 searches for the primary item box from the metadata box read in step S702, and acquires the item ID of the main image.

  In step S706, the CPU 101 searches for the item information box from the metadata box read in step S702, and acquires an item information entry corresponding to the item ID of the main image acquired in step S705. In the case of an HEIF format image file recorded by the above-described HDR image capturing process, overlay information is designated as the main image. That is, in the item information entry corresponding to the item ID of the main image, the item type is “iovl” representing overlay.

  In step S707, the CPU 101 executes overlay image creation processing. This will be described later with reference to FIG.

  In step S708, the CPU 101 displays the overlay image created in step S707 on the display unit 116 via the display control unit 115.

  When displaying the created overlay image, it may be necessary to specify color space information of the image. As the color space information of the image, the color gamut is specified by color_primaries of the color space property 'color', and the conversion characteristics (equivalent to gamma) are specified by transfer_characteristics. As an example of an HDR image, the color gamut is Rec. ITU-R BT. 2020, the conversion characteristic is Rec. ITU-R BT. 2100 (PQ) or the like is used.

  In addition, when there is HDR metadata such as 'MDCV', 'CLLI' in the item property container box, it is also possible to use it.

  Next, overlay image creation processing will be described with reference to FIG.

  In step S801, the CPU 101 executes overlay image property acquisition processing. This will be described later with reference to FIG.

  In step S802, the CPU 101 executes overlay information acquisition processing for an overlay image. Since the overlay information is recorded in the 'idat' box, the data stored in the 'idat' box is acquired.

  The overlay information includes an image size of the overlay image (output_width, output_height parameter), position information (horizontal_offset, vertical_offset) of individual image items (divided image items) constituting the overlay image, and the like.

  The overlay information is image item data handling of an image item whose item type is overlay 'iovl'.

  This acquisition process will be described later with reference to FIG.

  In step S803, the CPU 101 executes all item ID acquisition processing for the divided image items. This will be described later with reference to FIG.

  In step S804, the CPU 101 uses the overlay image size of the overlay information acquired in step S802 to secure a memory for storing image data of that size. This memory area is called a canvas.

  In step S805, the CPU 101 initializes the divided image counter n to 0.

  In step S806, the CPU 101 checks whether the divided image counter n is equal to the number of divided image items. If equal, the process proceeds to step S2210. If not equal, the process proceeds to step S807.

  In step S807, the CPU 101 executes an image creation process for one image item regarding the n-th divided image item. This will be described later with reference to FIG.

  In step S808, the CPU 101 arranges the image (divided image) of the nth divided image item created in step S807 on the canvas secured in step S804 according to the overlay information acquired in step S802.

  In the overlay, the divided images constituting the overlay can be arranged at an arbitrary position on the canvas, and the position is described in the overlay information. However, the outside of the overlay image is not displayed. Therefore, here, when the divided image is arranged on the canvas, only the area overlapping with the area of the overlay image, that is, the reproduction area of the divided image, is arranged among the encoded areas of the divided image.

  This will be described below with reference to FIG.

  The coordinate system assumes that the upper left of the overlay image is the origin (0, 0), the X coordinate increases in the right direction, and the Y coordinate increases in the downward direction.

  The size of the overlay image is defined as width Wo and height Ho (0 <Wo, 0 <Ho).

  Therefore, the overlay image has an upper left coordinate (0, 0) and a lower right coordinate (Wo-1, Ho-1).

  Further, the size of the divided image is defined as a width Wn and a height Hn (0 <Wn, 0 <Hn).

  The upper left position of the divided image is (Xn, Yn).

  Therefore, the divided image has an upper left coordinate (Xn, Yn) and a lower right coordinate (Xn + Wn-1, Yn + Hn-1).

  The overlapping area of the divided image with the overlay image (canvas) can be obtained by the following method.

In the following cases, there is no overlapping area.
Wo-1 <Xn (the left edge of the divided image is to the right of the right edge of the overlay image)
Xn + Wn-1 <0 (the right edge of the divided image is to the left of the left edge of the overlay image)
Ho-1 <Yn (the upper end of the divided image is below the lower end of the overlay image)
Yn + Hn-1 <0 (the lower end of the divided image is higher than the upper end of the overlay image)
In this case, this image is not processed.

In the following cases, the entire divided image becomes an overlapping region.
0 <= Xn and (Xn + Wn-1) <= (Wo-1) and 0 <= Yn and (Yn + Hn-1) <= (Ho-1)
In this case, the entire divided image is arranged at a designated position (Xn, Yn) on the canvas.

In cases other than the above, a part of the divided image is an arrangement target.
Let the upper left coordinates of the overlapping region be (Xl, Yt) and the lower right coordinates be (Xr, Yb).
The left end Xl is as follows.
if (0 <= Xn)
Xl = Xn;
else
Xl = 0;
The right end Xr is as follows.
if (Xn + Wn-1 <= Wo-1)
Xr = Xn + Wn-1;
else
Xr = Wo-1;
The upper end Yt is as follows.
if (0 <= Yn)
Yt = Yn;
else
Yt = 0;
The lower end Yb is as follows.
if (Yn + Hn-1 <= Ho-1)
Yb = Yn + Hn−1;
else
Yb = Ho-1;
The size of this overlapping region is a width Xr−Xl + 1 and a height Yb−Yt + 1.

  As described above, the upper left coordinates (Xl, Yt) and lower right coordinates (Xr, Yb) of this overlapping area are a coordinate system with the upper left corner of the canvas as the origin (0, 0).

When the upper left coordinates (Xl, Yt) of the overlapping area are expressed in a coordinate system with the upper left corner of the divided image as the origin, the following is obtained.
Xl ′ = Xl−Xn;
Yt ′ = Yt−Yn;

  In summary:

  A rectangle having a width (Xr−Xl + 1) and a height (Yb−Yt + 1) is cut out from a position (Xl ′, Yt ′) away from the upper left of the divided image, and is arranged at a position (Xl, Yt) on the canvas.

  Thereby, it is possible to arrange the reproduction area of the divided image at an appropriate position on the canvas.

  In step S809, the CPU 101 increments the divided image counter n by 1, and returns to step S806.

  In step S810, the CPU 101 checks whether the rotation property 'irot' exists in the properties of the overlay image acquired in step S801, and if so, checks the rotation angle. If the rotation property does not exist, or if the rotation property exists but the rotation angle is 0, the process ends. If the rotation property exists and the rotation angle is other than 0, the process proceeds to step S811.

  In step S811, the CPU 101 rotates the canvas image created in steps S806 to S809 by the angle acquired in step S810, and uses it as a created image.

  Thereby, it is possible to create an overlay image.

  In the above description, the processing from step S806 to step S809 is repeated in a loop for the number of divided images. However, in an environment where parallel processing is possible, the processing from step S806 to step S809 may be performed in parallel for the number of divided images. good.

  Next, image item property acquisition processing will be described with reference to FIG.

  In step S901, the CPU 101 searches for an entry of the designated image item ID from the item property related box in the item property box in the metadata box, and acquires an array of property indexes stored therein.

  In step S902, the CPU 101 initializes the array counter n to 0.

  In step S903, the CPU 101 checks whether the array counter n is equal to the number of array elements. If equal, exit. If not equal, the process proceeds to step S904.

  In step S904, the CPU 101 acquires the property of the index of the n-th element of the array from the item property container box in the item property box in the metadata box.

  In step S905, the CPU 101 adds 1 to the array counter n and returns to step S903.

  Next, image item data acquisition processing will be described with reference to FIG.

  In step S1001, the CPU 101 searches the item location box in the metadata box for an entry for the designated image item ID, and acquires an offset reference (construction_method), offset, and length.

  In step S1002, the CPU 101 checks the offset reference acquired in step S1001. In the offset reference, a value 0 represents an offset from the beginning of the file, and a value 1 represents an offset in the item data box. If the offset reference is 0, the process proceeds to step S1003. If the offset reference is 1, the process proceeds to step S1004.

  In step S <b> 1003, the CPU 101 reads the length byte from the position of the offset byte from the beginning of the file to the memory 102.

  In step S <b> 1004, the CPU 101 reads out the length byte from the position of the offset byte from the head of the data portion of the item data box in the metadata box to the memory 102.

  Next, an item ID acquisition process for an image constituting the main image will be described with reference to FIG.

  In step S1101, the CPU 101 searches the item reference box in the metadata box for an entry whose reference type is ‘ding’ and whose reference source item ID is the item ID of the main image.

  In step S1102, the CPU 101 acquires an array of reference destination item IDs of the entries acquired in step S1101.

  Next, an image creation process for one encoded image item will be described with reference to FIG.

  In step S1201, the CPU 101 acquires the property of the image item. This has been described with reference to FIG.

  In step S1202, the CPU 101 acquires the data of the image item. This has been described with reference to FIG. The image item data is encoded image data.

  In step S1203, the CPU 101 initializes the decoder using the decoder configuration / initialization data among the properties acquired in step S1201.

  In step S1204, the CPU 101 decodes the encoded image data acquired in step S1202 with a decoder, and acquires a decoding result.

  In step S1205, the CPU 101 checks whether pixel format conversion is necessary.

  If not needed, exit. If necessary, the process proceeds to step S1206.

  If the pixel format of the output data of the decoder is different from the pixel format of the image supported by the display device, pixel format conversion is necessary.

  For example, when the pixel format of the output data of the decoder is the YCbCr (luminance color difference) format and the pixel format of the image of the display device is the RGB format, the pixel format conversion from the YCbCr format to the RGB format is necessary. Even in the same YCbCr format, pixel format conversion is also required when the bit depth (8 bits, 10 bits, etc.) and color difference samples (4: 2: 0, 4: 2: 2, etc.) are different.

  Since the coefficient for converting from the RGB format to the YCbCr format is specified in matrix_coefficients of the color space information property 'colr', in order to convert from the YCbCr format to the RGB format, the reciprocal of the coefficient is used Good.

  In step S1206, the CPU 101 converts the decoder output data acquired in step S1204 into a desired pixel format.

  Thereby, it is possible to create an image of the specified encoded image item.

  Through the above processing, an HDR image recorded in the HEIF format as shown in FIG. 2 on the recording medium 120 can be reproduced.

<Other embodiments>
As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to these embodiment, Various forms of the range which does not deviate from the summary of this invention are also contained in this invention. .

  Further, the function of the above embodiment may be used as a control method, and this control method may be executed by the image processing apparatus. In addition, a program having the functions of the above-described embodiments may be used as a control program, and the control program may be executed by a computer included in the image processing apparatus. The control program is recorded on a computer-readable storage medium, for example.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium. It can also be realized by a process in which one or more processors in the computer of the system or apparatus read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (17)

An imaging device that captures and records an image,
An imaging sensor;
Encoding means for encoding image data obtained by imaging by the imaging sensor;
Of the image data obtained by photographing with the image sensor, image data in an encoding area larger than the reproduction area to be reproduced is encoded by the encoding means, and the encoded plural image data is predetermined. An image pickup apparatus comprising: a recording control unit that controls recording on a recording medium in a recording format.   The recording control unit divides the image data of the coding area into a plurality of divided image data, the plurality of divided image data is encoded by the coding unit, respectively, and the plurality of encoded divided image data The image pickup apparatus according to claim 1, wherein the image pickup device is controlled to be recorded on the recording medium as one image file in a predetermined recording format.   When the HDR image data is captured and recorded, the recording control unit includes HDR image data in an encoding area larger than a reproduction area to be reproduced among HDR image data obtained by imaging from the imaging sensor. Is divided into a plurality of divided HDR image data, each of the plurality of divided HDR image data is encoded by the encoding means, and the plurality of divided divided HDR image data are converted into one image in the predetermined recording format. The imaging apparatus according to claim 2, wherein control is performed so as to record the file as a file on the recording medium. Having setting means for setting the recording size;
The recording control means determines the reproduction area and the encoding area according to the recording size set by the setting means,
2. The imaging apparatus according to claim 1, wherein the size of the reproduction area is an area corresponding to the set recording size.   The recording control means determines the encoding area from the reproduction area and a read start possible position and a read end possible position among image data obtained by photographing with the imaging sensor. The imaging device according to claim 4. The predetermined recording format is a recording format in which an area to be reproduced can be specified in the recorded image data,
The imaging apparatus according to claim 1, wherein the recording control unit designates the reproduction area smaller than the encoded area to be recorded as an area to be reproduced.   The recording control means encodes thumbnail image data corresponding to the image data in the reproduction area by the encoding means, and the encoded thumbnail image data together with the plurality of encoded divided image data and the predetermined recording format. The image pickup apparatus according to claim 1, wherein control is performed so as to record as one image file.   The imaging apparatus according to claim 1, wherein the encoding unit compresses and encodes the 8 image data in accordance with a HEVC (High Efficiency Video Coding) standard.   The imaging apparatus according to claim 1, wherein the predetermined recording format is HEIF (High Efficiency Image File Format).   The imaging apparatus according to claim 2, wherein the predetermined recording format is HEIF (High Efficiency Image File Format), and the plurality of divided image data are recorded in an overlay format.   The imaging apparatus according to claim 10, wherein the recording control unit designates a reproduction area so that only the reproduction area is reproduced in the overlay format.   The recording control unit records position information of each of the plurality of divided image data so that the plurality of divided image data are arranged in a positional relationship corresponding to the coding area in the overlay format. The imaging apparatus according to claim 10, wherein the imaging apparatus is controlled so as to perform.   13. The imaging apparatus according to claim 12, wherein the recording control unit records position information of each of the plurality of divided image data so that the plurality of divided image data are located at positions that do not overlap each other.   The image pickup apparatus has a predetermined restriction when setting an encoding area to be encoded by the encoding means in the image data obtained by imaging with the image sensor. Item 2. The imaging device according to Item 1.   In the image pickup device, a position where encoding start / end is possible is determined in advance among image data obtained by photographing with the image sensor, and from other than the position where encoding start / end is possible, The imaging apparatus according to claim 14, wherein encoding cannot be started / finished. Dividing image data in an encoding area larger than a reproduction area to be reproduced into a plurality of divided image data, encoding the plurality of divided image data respectively by the encoding means, and the plurality of encoded divisions Reading means for reading the image file from a recording medium in which the image data is recorded as a single image file in a predetermined recording format;
A combination means for combining each of the plurality of divided image data included in the read image file;
Display control means for combining the plurality of divided image data decoded by the decoding means into one image data, and controlling the display area to display the image data in the reproduction area among the combined image data And a display control device. Dividing image data in an encoding area larger than a reproduction area to be reproduced into a plurality of divided image data, encoding the plurality of divided image data respectively by the encoding means, and the plurality of encoded divisions A reading step of reading the image file from a recording medium in which the image data is recorded as a single image file in a predetermined recording format;
A compounding step of compounding each of the plurality of divided image data included in the read image file;
A display control step of combining the plurality of divided image data decoded by the decoding unit into one image data, and controlling the display unit to display the image data in the reproduction area among the combined image data And a control method for the display control device.

Images