JP2024015185A - Imaging device and its control method and program - Google Patents

Abstract

Even when recording RAW image data in an imaging apparatus, developed image data corresponding to a selected dynamic range is included in a RAW image file and recorded. [Solution] An imaging means, an encoding means that performs encoding processing on image data, and a first dynamic range image data generated based on the RAW image data when recording a RAW image file. The first encoded image data that has been encoded in the first encoding format is recorded as a display image of a RAW image file, or the second dynamic range image is generated based on the RAW image data. and a control means for controlling whether second encoded image data obtained by performing encoding processing in a second encoding format on the data is recorded as a display image of the RAW image file. [Selection diagram] Figure 1

Description

The present invention relates to an imaging device, its control method, and a program.

2. Description of the Related Art Imaging devices such as recent digital cameras are capable of capturing HDR images and recording the HDR images on recording media. Here, HDR stands for High Dynamic Range, and is a technology that generates an image with a wider dynamic range than SDR (Standard Dynamic Range). Furthermore, a RAW image refers to a raw image before development.

When recording HDR video content, it is recorded together with identification information indicating whether the content is HDR video (for example, Patent Document 1).

JP 2018-7194 Publication

A normal RAW image file can include data obtained by JPEG compressing a developed image as a display image. However, since JPEG images do not support HDR image quality, display images cannot be saved in HDR image quality. Therefore, even if a RAW image shot in HDR is displayed on an HDR display, it is necessary to develop the RAW image once in order to check the HDR image quality.

The present invention has been made in view of this problem, and even when recording RAW image data in an imaging device that supports multiple dynamic ranges such as SDR and HDR, the present invention does not apply to the selected dynamic range. The present invention attempts to provide a technology that allows images in a selected dynamic range to be confirmed even when playing back the RAW image file by recording the corresponding developed image data in the RAW image file.

In order to solve this problem, for example, the imaging device of the present invention has the following configuration. That is,
an imaging means;
an encoding means that performs encoding processing on image data;
When recording a RAW image file, first encoded image data obtained by performing encoding processing in a first encoding format on image data of a first dynamic range generated based on RAW image data is recorded as a RAW image file. The second encoded image data is recorded as a display image of an image file, or the second encoded image data is obtained by performing encoding processing of the second encoding format on the second dynamic range image data generated based on the RAW image data. , a control means for controlling whether to record the RAW image file as a display image;
has.

According to the present invention, even when recording RAW image data in an imaging device that supports multiple types of dynamic ranges such as SDR and HDR, developed image data corresponding to the selected dynamic range can be used. By including and recording the RAW image file in the RAW image file, it becomes possible to confirm the image in the selected dynamic range, for example, when displaying a file list or the like in a simple manner.

FIG. 1 is an external view of an imaging/display device in an embodiment. FIG. 1 is a block diagram showing the configuration of an imaging/display device in an embodiment. Connection configuration diagram with external equipment. 5 is a flowchart of LV photography mode processing in the embodiment. Quick review flowchart. FIG. 3 is a sequence diagram of HDMI connection processing in the embodiment. FIG. 3 is a sequence diagram of HDMI connection processing in the embodiment. FIG. 3 is a sequence diagram of HDMI connection processing in the embodiment. 5 is a flowchart of HDR shooting menu processing in the embodiment. 5 is a flowchart of HDR shooting menu processing in the embodiment. 5 is a flowchart of HDR photographing processing in the embodiment. FIG. 2 is a configuration diagram of a RAW file in an embodiment. The figure which shows the example of area|region ImageData in a RAW file. 5 is a flowchart of playback mode processing in the embodiment. 5 is a flowchart of playback mode processing in the embodiment. 5 is a flowchart of playback mode processing in the embodiment. 5 is a flowchart of playback mode processing in the embodiment. 5 is a flowchart of playback mode processing in the embodiment. 5 is a flowchart of playback mode processing in the embodiment. 5 is a flowchart of playback mode processing in the embodiment. 5 is a flowchart of playback mode processing in the embodiment. 5 is a flowchart of HDMI playback processing in the embodiment. 5 is a flowchart of HDMI playback processing in the embodiment. FIG. 7 is a flowchart of playback menu processing and a diagram showing the flow of data in the embodiment. Flowchart of development processing in the embodiment. A diagram showing a CxCy plane. Flowchart of gradation correction parameter generation processing. FIG. 7 is a diagram showing gradation correction amounts. A diagram showing an example of appearance in SDR and HDR.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

FIGS. 1A and 1B show external views of a digital camera 100 as an example of a device to which this embodiment is applied. 1(a) is a front perspective view of the digital camera 100, and FIG. 1(b) is a rear perspective view of the digital camera 100. In FIGS. 1A and 1B, the display section 28 is a display section provided on the back of the camera that displays images and various information. The outside viewfinder display section 43 is a display section provided on the top surface of the camera, and displays various setting values of the camera, including shutter speed and aperture. The shutter button 61 is an operation unit for instructing photography. The mode changeover switch 60 is an operation unit for switching between various modes. The terminal cover 40 is a cover that protects a connector (not shown) such as a connection cable that connects a connection cable with an external device and the digital camera 100. The main electronic dial 71 is a rotary operation member included in the operation unit 70, and by rotating the main electronic dial 71, the user can change settings such as shutter speed and aperture. The power switch 72 is an operation member that turns the power of the digital camera 100 on and off. The sub-electronic dial 73 is a rotary operation member included in the operation section 70, and is used for moving the selection frame, forwarding images, and the like. The cross key 74 is included in the operation unit 70 and is a cross key (four-directional key) that can be pressed in the upper, lower, left, and right portions, respectively. Operations can be performed according to the pressed part of the cross key 74. The SET button 75 is included in the operation unit 70, is a push button, and is mainly used for determining selection items. The LV button 76 is included in the operation unit 70 and is a button for switching live view (hereinafter referred to as LV) between ON and OFF in the still image shooting mode. In the video shooting mode, it is used to instruct the start and stop of video shooting (recording). The magnification button 77 is included in the operation unit 70 and is an operation button for turning the magnification mode on and off in live view display in the shooting mode, and changing the magnification ratio in the magnification mode. In the playback mode, it functions as an enlargement button for enlarging the playback image and increasing the enlargement rate. The reduction button 78 is included in the operation unit 70 and is a button for reducing the enlargement rate of the enlarged reproduced image and reducing the displayed image. The playback button 79 is included in the operation unit 70 and is an operation button for switching between shooting mode and playback mode. By pressing the playback button 79 during the shooting mode, the mode shifts to the playback mode, and the latest image among the images recorded on the recording medium 200 can be displayed on the display unit 28 or the external device 300. The quick return mirror 12 is moved up and down by an actuator (not shown) in response to instructions from the system control unit 50. The communication terminal 10 is a communication terminal through which the digital camera 100 communicates with the lens side (which is detachable). The eyepiece finder 16 is a peering type finder for checking the focus and composition of the optical image of the object obtained through the lens unit 150 by observing the focusing screen 13. The lid 202 is a lid of a slot in which the recording medium 200 is stored. The grip section 90 is a holding section shaped so that it can be easily grasped with the right hand when the user holds the digital camera 100.

FIG. 2 is a block diagram showing a configuration example of the digital camera 100 according to this embodiment.

In FIG. 2, a lens unit 150 is a lens unit equipped with an exchangeable photographic lens.

Although the lens 103 is normally composed of a plurality of lenses, only one lens is shown here for simplicity. The communication terminal 6 is a communication terminal for the lens unit 150 to communicate with the digital camera 100 side, and the communication terminal 10 is a communication terminal for the digital camera 100 to communicate with the lens unit 150 side. The lens unit 150 communicates with the system control section 50 via the communication terminals 6 and 10, controls the aperture 1 via the aperture drive circuit 2 by the internal lens system control circuit 4, and controls the aperture 1 via the AF drive circuit 3. The focus is adjusted by shifting the position of the lens 103.

The AE sensor 17 measures the brightness of the subject through the lens unit 150. The focus detection section 11 outputs defocus amount information to the system control section 50. Based on this, the system control section 50 controls the lens unit 150 and performs phase difference AF.

The quick return mirror 12 (hereinafter referred to as mirror 12) is raised and lowered by an actuator (not shown) in response to instructions from the system control unit 50 during exposure, live view photography, and video photography. The mirror 12 is a mirror for switching the light flux incident from the lens 103 to the viewfinder 16 side and the imaging unit 22 side. The mirror 12 is normally arranged so as to reflect the light beam to the finder 16, but when shooting or displaying live view, the mirror 12 is arranged upward to guide the light beam to the imaging section 22. Take shelter from the bouncing beam of light (mirror up). Further, the mirror 12 is a half mirror so that a portion of the light can pass through the center thereof, and a portion of the light beam is transmitted so as to be incident on the focus detection section 11 for performing focus detection.

By observing the focusing screen 13 through the pentaprism 14 and the finder 16, the user (photographer) can confirm the focus and composition of the optical image of the subject obtained through the lens unit 150.

The shutter 101 is a focal plane shutter that can freely control the exposure time of the imaging unit 22 under the control of the system control unit 50.

The imaging unit 22 is an imaging device composed of a CCD, CMOS device, etc. that converts an optical image into an electrical signal. On the imaging surface of the imaging unit 22, filters for each color component of R, G, and B are two-dimensionally and periodically arranged. When focusing on the adjacent 2×2 filters, G component filters are placed in two diagonally adjacent filters, and G and B component filters are placed in the remaining two. These 2×2 filters are arranged on the imaging surface of the imaging section 22. Such an array is generally called a Bayer array. Therefore, the image represented by the signal (analog signal) output from the imaging unit 22 also becomes a Bayer array pixel signal. The A/D converter 23 converts a one-pixel analog signal output from the imaging section 22 into, for example, a 10-bit digital signal. Note that the image data at this stage is undeveloped image data, which is Bayer array image data with one component per pixel and 10 bits per component, as described above. Therefore, the image data at this stage is called RAW image data. Note that the Bayer array image data after defective pixels have been compensated for may be used as RAW image data. In the embodiment, it is assumed that the A/D converter 23 converts the analog signal into 10-bit digital data, but the number of bits is not particularly limited as long as it exceeds 8 bits. The larger the number of bits, the higher the gradation expression becomes possible.

The image processing section 24 performs resizing processing such as predetermined pixel interpolation and reduction, and color conversion processing on the data from the A/D converter 23 or the data from the memory control section 15. Further, the image processing section 24 performs predetermined calculation processing using the captured image data, and the system control section 50 performs exposure control and distance measurement control based on the obtained calculation results. As a result, TTL (through-the-lens) type AF (autofocus) processing, AE (automatic exposure) processing, and EF (flash pre-emission) processing are performed. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data, and also performs TTL-based AWB (auto white balance) processing based on the obtained arithmetic results. Furthermore, the image processing section 24 also performs encoding/decoding processing of image data under the control of the system control section 50. This encoding includes JPEG and HEVC. JPEG is for encoding image data of 8 bits per color component, and HEVC is for encoding image data of more than 8 bits per color component.

Output data from the A/D converter 23 is directly written into the memory 32 via the image processing section 24 and the memory control section 15 or via the memory control section 15. The memory 32 stores image data obtained by the imaging section 22 and converted into digital data by the A/D converter 23, and image data to be displayed on the display section 28 or external device 300. The memory 32 has a storage capacity sufficient to store a predetermined number of still images, a predetermined period of moving images, and audio.

Furthermore, the memory 32 also serves as an image display memory (video memory). The D/A converter 19 converts the display image data stored in the memory 32 into an analog signal and supplies it to the display unit 28 . In this way, the display image data written in the memory 32 is displayed on the display unit 28 via the D/A converter 19. The display unit 28 displays on a display such as an LCD according to the analog signal from the D/A converter 19. A digital signal that has been A/D converted by the A/D converter 23 and stored in the memory 32 is converted into an analog signal by the D/A converter 19, and is sequentially transferred to the display section 28 for display, so that it can be used as an electronic viewfinder. function, and allows through-image display (live view display).

A frame (AF frame) indicating the distance measurement point where autofocus is currently being performed, an icon indicating the camera setting status, etc. are displayed on the finder LCD display section 41 via the finder display section drive circuit 42. be done. Various setting values of the camera, including the shutter speed and aperture, are displayed on the outside-finder liquid crystal display section 43 via the outside-finder display section drive circuit 44.

The digital output I/F 90 supplies the image display data stored in the memory 32 as a digital signal to the external device 300. For example, video data is output in a stream format according to a communication protocol compliant with the HDMI (registered trademark) (High-Definition Multimedia Interface) standard. In this way, the display image data written in the memory 32 is displayed on the external device 300.

The nonvolatile memory 56 is an electrically erasable/recordable memory, such as an EEPROM. The nonvolatile memory 56 stores constants, programs, etc. for the operation of the system control unit 50. The program here refers to a program for executing various flowcharts described later in this embodiment.

The system control unit 50 is a control unit that includes at least one processor, and controls the entire digital camera 100. By executing the program recorded in the nonvolatile memory 56 described above, each process of this embodiment, which will be described later, is realized. Reference numeral 52 is a system memory, and RAM is used. In the system memory 52, constants and variables for the operation of the system control unit 50, programs read from the non-volatile memory 56, etc. are loaded. The system control unit 50 also performs display control by controlling the memory 32, the D/A converter 19, the digital output I/F 90, the display unit 28, and the like.

The system timer 53 is a timer that measures the time used for various controls and the time of a built-in clock.

The mode changeover switch 60, the first shutter switch 62, the second shutter switch 64, and the operation section 70 function as operation means for inputting various operation instructions to the system control section 50.

The mode changeover switch 60 switches the operation mode of the system control unit 50 to one of a still image recording mode, a moving image shooting mode, a playback mode, and the like. Modes included in the still image recording mode include automatic shooting mode, automatic scene discrimination mode, manual mode, aperture priority mode (Av mode), and shutter speed priority mode (Tv mode). Additionally, there are various scene modes, program AE mode, custom mode, etc., which are shooting settings for each shooting scene. The mode changeover switch 60 allows direct switching to any of these modes. Alternatively, after once switching to the shooting mode list screen using the mode changeover switch 60, one of the displayed plurality of modes may be selected and switched using another operating member. Similarly, the video shooting mode may include a plurality of modes.

The shutter button 61 operated by the user includes a first shutter switch 62 and a second shutter switch 63. The first shutter switch 62 is turned on when the user presses the shutter button 61 halfway (instruction to prepare for photographing) while operating the shutter button 61, and generates the first shutter switch signal SW1. When the first shutter switch signal SW1 is input, the system control unit 50 performs operations such as AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and EF (flash pre-flash) processing. Start. The second shutter switch 64 is turned ON when the operation of the shutter button 61 is completed, that is, when the shutter button 61 is fully pressed (photographing instruction), and generates a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 starts a series of photographic processing operations from reading out signals from the imaging unit 22 to writing image data to the recording medium 200.

Each operating member of the operating unit 70 is assigned an appropriate function for each scene by selecting and operating various function icons displayed on the display unit 28 or the external device 300, and acts as various function buttons. Examples of the function buttons include an end button, a return button, an image forwarding button, a jump button, a narrowing down button, and an attribute change button. For example, when the menu button 70e is pressed, various configurable menu screens are displayed on the display unit 28 or the external device 300. The user can intuitively perform various settings using the menu screen displayed on the display unit 28 or the external device 300, four-way buttons (up, down, left, right), and the SET button.

Note that the display unit 28 in the embodiment has an image notation function of SDR image quality, that is, can display each color component of R, G, and B in 8 bits (256 gradations). Further, when the external device 300 is connected to the digital camera 100, the external device 300 is set as the device to output captured images and live images instead of the display unit 28. Further, when the user operates the operation unit 70 to explicitly select either the display unit 28 or the external device 300, the selected one becomes the output target device.

The operation unit 70 is various operation members that serve as an input unit that accepts operations from the user. The operation section 70 includes at least the following operation sections. They are a shutter button 61, a main electronic dial 71, a power switch 72, a sub electronic dial 73, a cross key 74, a SET button 75, an LV button 76, an enlargement button 77, a reduction button 78, and a playback button 79. The cross key 74 is a direction button that can be pressed on each of the upper, lower, right, and left parts of the cross key 74. Although the present embodiment is described as an integrated operation unit, the upper button, lower button, right button, and left button may each be independent buttons. Hereinafter, the upper or lower portion will be referred to as the up/down key, and the left or right portion will be referred to as the left/right key. The operation unit 70 also includes the following operation units.

The AF-ON button 70b is a push button switch included in the operation unit 70, and can be pressed to instruct execution of AF. The direction in which the AF-PN button 70b is pressed is parallel to the direction (optical axis) of subject light entering the imaging unit 22 from the lens 103.

The quick setting button 70c (hereinafter referred to as Q button 70c) is a push button switch included in the operation unit 70, and when pressed, a quick setting menu, which is a list of setting items that can be set in each operation mode, is displayed. . For example, when pressed during live view shooting standby, a list of setting items such as electronic front-curtain shutter, monitor brightness, LV screen WB, 2-point magnification, and silent shooting are superimposed on the LV in a single column. will be displayed. By selecting any option from the displayed quick setting menu using the up and down keys and pressing the set button, the user can change the settings for the selected setting item or shift to the operation mode.

The active frame switching button 70d is a push button switch included in the operation unit 70, and when pressed in the two-point enlargement process described later, the active enlargement position (frame) can be switched between the two enlarged locations. I can do it. Further, different functions are assigned depending on the operation mode, and when pressed in playback mode, a protect attribute can be added to the displayed image.

The menu button 70e is a push button switch included in the operation unit 70, and a menu screen on which various settings can be made is displayed on the display unit 28 or the external device 300.

The function buttons 70f are three pushbutton switches included in the operation unit 70, and functions are assigned to each of them. Each of the function buttons 70f is arranged at a position where it can be operated by the finger (middle finger, ring finger, or little finger) of the right hand holding the grip section 90, and the direction in which it is pressed is the direction of the subject light that enters the imaging section 22 from the lens 103. (optical axis).

The info button 70g is a push button switch included in the operation unit 70, and is used to switch various information displays.

The power supply control unit 80 includes a battery detection circuit, a DC-DC converter, a switch circuit for switching the block to be energized, and the like, and detects whether or not a battery is attached, the type of battery, and the remaining battery level. Further, the power supply control section 80 controls the DC-DC converter based on the detection result and the instruction from the system control section 50, and supplies the necessary voltage to each section including the recording medium 200 for a necessary period.

The power supply unit 30 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like. The recording medium I/F 18 is an interface with a recording medium 200 such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card for recording photographed images, and is composed of a semiconductor memory, a magnetic disk, or the like.

The communication unit 54 is connected via a wireless or wired cable and transmits and receives video signals and audio signals. The communication unit 54 can also be connected to a wireless LAN (Local Area Network) and the Internet. The communication unit 54 can transmit images captured by the imaging unit 22 (including through images) and images recorded on the recording medium 200, and can also receive image data and other various information from external devices. can.

The attitude detection unit 55 detects the attitude of the digital camera 100 with respect to the direction of gravity. Based on the orientation detected by the orientation detection unit 55, it is determined whether the image taken by the imaging unit 22 is an image taken with the digital camera 100 held horizontally or an image taken with the digital camera 100 held vertically. Identifiable. The system control unit 50 can add orientation information corresponding to the orientation detected by the orientation detection unit 55 to the image file of the image captured by the imaging unit 22, or can rotate and record the image. As the posture detection section 55, an acceleration sensor, a gyro sensor, or the like can be used.

Note that one of the operation units 70 includes a touch panel 70a that can detect contact with the display unit 28. The touch panel 70a and the display section 28 can be configured integrally. For example, the touch panel 70a is configured such that its light transmittance does not interfere with the display on the display section 28, and is attached to the upper layer of the display surface of the display section 28. Then, the input coordinates on the touch panel 70a and the display coordinates on the display unit 28 are associated with each other. Thereby, it is possible to configure a GUI (graphical user interface) as if the user could directly operate the screen displayed on the display unit 28. The system control unit 50 can detect the following operations or states on the touch panel 70a.
- A finger or a pen that has not touched the touch panel 70a newly touches the touch panel 70a. That is, the start of a touch (hereinafter referred to as Touch-Down).
- The touch panel 70a is in a state of being touched with a finger or a pen (hereinafter referred to as Touch-On).
- Moving while touching the touch panel 70a with a finger or pen (hereinafter referred to as a touch-move).
- The finger or pen that was touching the touch panel 70a is released. That is, the end of the touch (hereinafter referred to as Touch-Up).
- A state in which nothing is touched on the touch panel 70a (hereinafter referred to as Touch-Off).

When a touchdown is detected, a touch-on is also detected at the same time. After touchdown, touch-ons typically continue to be detected unless a touch-up is detected. A touch move is also detected in a state where a touch-on is detected. Even if a touch-on is detected, if the touch position does not move, a touch move will not be detected. After all touching fingers and pens are detected to have touched up, touch-off occurs.

These operations/states and the coordinates of the position where the finger or pen is touching on the touch panel 70a are notified to the system control unit 50 via the internal bus. The system control unit 50 determines what kind of operation (touch operation) has been performed on the touch panel 70a based on the notified information. Regarding touch moves, the direction of movement of a finger or pen on the touch panel 70a can also be determined for each vertical component and horizontal component on the touch panel 70a based on changes in position coordinates. If it is detected that a touch move has been made over a predetermined distance, it is determined that a slide operation has been performed. An operation in which you keep your finger on the touch panel, quickly move it a certain distance, and then release it is called a flick. In other words, a flick is an operation in which a finger is quickly traced on the touch panel 70a as if flicking. If a touch move over a predetermined distance and at a predetermined speed is detected, and a touch-up is detected as is, it can be determined that a flick has been performed (it can be determined that a flick has occurred following a slide operation). Further, a touch operation in which multiple points (for example, two points) are touched at the same time and the touch positions are brought closer to each other is called a pinch-in, and a touch operation in which the touch positions are moved away from each other is called a pinch-out. Pinch out and pinch in are collectively referred to as a pinch operation (or simply pinch). The touch panel 70a may be any one of various types of touch panels, such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, an optical sensor type, etc. good. Depending on the method, there are methods in which a touch is detected when there is contact with the touch panel, and methods where a touch is detected when a finger or pen approaches the touch panel, but either method may be used.

Furthermore, the present invention is applicable not only to the imaging device itself but also to a control device that communicates with the imaging device (including a network camera) via wired or wireless communication and remotely controls the imaging device. Examples of devices that remotely control the imaging device include devices such as smartphones, tablet PCs, and desktop PCs. The imaging device can be controlled remotely by notifying the imaging device of commands to perform various operations and settings from the control device, based on operations performed on the control device and processing performed on the control device. be. Furthermore, a live view image taken by the imaging device may be received via wired or wireless communication and displayed on the control device side.

In addition, although the case where it applied to a digital camera was explained above as an example, the present invention is not limited to this. For example, the present invention is applicable to any device equipped with a display section, such as a PDA, a mobile phone terminal, a portable image viewer, a printer device equipped with a display, a digital photo frame, a music player, a game console, an electronic book reader, etc. It is.

FIG. 3 is a diagram showing an example of the connection between the digital camera 100 and the external device 300. When the digital camera 100 and the external device 300 are connected with the connection cable 302, the display unit 28 of the digital camera 100 turns off, and the display of the digital camera 100 is switched and displayed on the display 301 of the external device 300.

FIG. 4A is a flowchart showing LV shooting mode processing in the digital camera 100. This processing is realized by loading a program recorded in the nonvolatile memory 56 into the system memory 52 and executing it by the system control unit 50.

First, the HDR photography mode and SDR photography mode in this embodiment will be explained. The digital camera 100 of this embodiment can set the HDR shooting mode or the SDR shooting mode through a menu operation or the like by the user. These modes are for the user to set their intention to ultimately obtain HDR image quality image data or SDR image quality image data, and the following processing will depend on which mode is set. Various controls are performed accordingly. Hereinafter, performing photography in the HDR photography mode and the SDR photography mode may be referred to as "HDR photography" and "SDR photography". However, as will be described later, it is also possible to set recording only in RAW format, so an HDR image is not necessarily recorded when shooting in HDR shooting mode.

In S401, the system control unit 50 determines whether the user's setting on the operation unit 70 is the HDR shooting mode. If the system control unit 50 determines that the HDR shooting mode is set, the process proceeds to S402, and if it determines that the SDR shooting mode is set, the process proceeds to S422.

In S402, the system control unit 50 determines whether the external device 300 is connected to the digital camera 100. If the system control unit 50 determines that the external device 300 is connected, the process proceeds to S403, and if it determines that the external device 300 is not connected, the process proceeds to S404.

In S403, the system control unit 50 performs connection processing between the digital camera 100 and the external device 300. Then, the system control unit 50 advances the process to S404. Details of this connection process will be described later using FIG. 5. Note that if the external device supports HDR connection, HDR connection will be made, and if it does not support HDR connection, SDR connection will be made.

In S404, the system control unit 50 uses the image processing unit 24 to perform HDR image quality development on the live RAW image data obtained by capturing the image with the imaging unit 22 and converting it into a digital signal with the A/D converter 23. Perform processing. Hereinafter, an image obtained by HDR image quality development processing will be referred to as an HDR image.

Note that HDR image data in the embodiment is data in which one pixel is composed of three components (Luv, YCbCr, etc.), and each component is represented by 10 bits (1024 gradations) in the embodiment. A gamma curve for HDR images (for example, PQ and HLG of ITU-R recommendation BT.2100) is applied to the HDR image data.

In S405, the system control unit 50 determines whether the device (display unit 28 or external device 300) that displays the LV image is HDR compatible, and if it is determined that it is not HDR compatible, the process proceeds to S406. If it is determined that they are compatible, the process advances to S409.

In S406, the system control unit 50 confirms the HDR assist display setting. If the system control unit 50 determines that assist 1 is set, the process proceeds to S407, and if assist 2 is set, the process proceeds to S408. Assist 1 is a setting for checking the high brightness area of the HDR image, and a process of assigning many gradations (code values) to the high brightness area of the HDR image is performed. Assist 2 is a setting for checking the intermediate brightness range of the HDR image, and a process of assigning many gradations to the intermediate brightness area of the HDR image is performed.

In S407, the system control unit 50 performs HDR → SDR conversion processing on the HDR image data obtained in the development processing in S404 according to the settings of Assist 1, and also performs HDR → SDR conversion processing on the HDR image data obtained in the development processing in S404, and ) is displayed, and the process proceeds to S410.

In S408, the system control unit 50 performs HDR → SDR conversion processing on the HDR image data obtained in the development process in S404 according to the settings of Assist 2, and also converts the HDR image data obtained in the development process in S404 to the output target device (display unit 28 or external device 300). The SDR image quality LV image data obtained by performing resizing processing to a size suitable for the image data is displayed, and the process proceeds to S410.

Here, the image data of SDR image quality (SDR image data) in S407 and S408 refers to image data of 8 bits per component. A gamma curve for SDR images (for example, a gamma curve of the sRGB standard) is applied to image data of SDR image quality. Note that the gamma curve of the sRGB standard is generally a curve with a straight line in dark areas and a power of 2.4 in bright areas, but a curve with a power of 2.2 may be used for simplicity.

In S409, the system control unit 50 resizes the HDR image data obtained through the development process in S404 to a size suitable for the output target device (display unit 28 or external device 300), and adjusts the HDR image quality after the resize. The image (hereinafter referred to as HDL_LV image) is displayed live, and the process proceeds to S410.

In S410, the system control unit 50 determines whether the menu display button 70e has been pressed. If it is determined that it has been pressed, the process proceeds to S411; if it is determined that it has not been pressed, the process proceeds to S412. In S411, the system control unit 50 performs shooting menu processing, and advances the process to S412. Details of this shooting menu processing will be described later using FIG. 6.

In S412, the system control unit 50 determines whether the info display button 70g has been pressed. If it is determined that it has been pressed, the process proceeds to S413; if it is determined that it has not been pressed, the process proceeds to S414. In S413, the system control unit 50 switches the display of photographing information, and proceeds to S414. The shooting information includes a histogram, a highlight warning table, and the like.

In S414, the system control unit 50 determines whether the shutter button 61 is pressed halfway based on whether or not the signal SW1 is received, and if it is determined that the shutter button 61 is not pressed halfway, the process proceeds to S420. If it is determined that the button is pressed halfway, the process advances to S415.

In S415, the system control unit 50 performs the AF/AE processing described in FIG. 2, and proceeds to S416. In S416, the system control unit 50 determines whether the shutter button 61 is fully pressed based on whether the signal SW2 is received, and if it is determined that the shutter button 61 is not fully pressed, the process proceeds to S417, If it is determined that the button is fully pressed, the process advances to S418. In S417, the system control unit 50 determines whether the shutter button 61 is held in a half-pressed state, returns the process to S415 if the half-pressed state is held, and returns the process to S415 if the half-pressed state is not held. If determined, the process advances to S420. In S418, the system control unit 50 performs HDR imaging processing and records an image data file in accordance with the preset recording format on the recording medium. FIG. 8A shows the data structure of the recorded file. Then, the system control unit 50 advances the process to S419. Note that details of this HDR photographing process will be described later using FIG. 7. Then, in S419, the system control unit 50 performs quick review display processing, and advances the process to S420. Details of this quick review display processing will be described later using FIG. 4B.

In S420, the system control unit 50 determines whether the LV button 76 has been pressed, and if it is determined that it has been pressed, the process proceeds to S421, and if it is determined that it has not been pressed, the process proceeds to S422.

In S421, the system control unit 50 compresses the image data developed to HDR image quality in S404 (image data of 1 pixel with 3 components and 10 bits per component) using HEVC (H.265), and converts it into an HDR video file. The information is recorded and the process proceeds to S438.

In S422, the system control unit 50 determines whether the external device 300 is connected to the digital camera 100. If the system control unit 50 determines that the external device 300 is connected, the process proceeds to S423, and if it determines that the external device 300 is not connected, the process proceeds to S424. In S423, the system control unit 50 performs a connection process between the digital camera 100 and the external device 300, and advances the process to S424. Details of this connection process will be described later using FIG. 5. Note that since the camera is in SDR shooting mode, it will be connected to external devices via SDR.

In S424, the system control unit 50 converts the image captured by the imaging unit 22 and converted into a digital signal by the A/D converter 23 to an SDR image quality (1 pixel, 3 components, 8 bits per component) by the image processing unit 24. (256 gradations)), and the process proceeds to S425. Note that hereinafter, an image after development processing with SDR image quality will be referred to as an SDR image.

In S425, the system control unit 50 resizes the SDR image obtained in the development process in S424 to a size suitable for the resolution of the output destination device (display unit 28 or external device 300), and converts the SDR image into a live image with SDR image quality. An image (SDR_LV image) is generated, and the generated SDR_LV image is displayed.

In S426, the system control unit 50 determines whether the menu display button 70e has been pressed. If it is determined that it has been pressed, the process proceeds to S427; if it is determined that it has not been pressed, the process proceeds to S428. In S427, the system control unit 50 performs shooting menu processing, and advances the process to S428. Details of the photographing menu process in S427 will be described later using FIG.

In S428, the system control unit 50 determines whether the info display button 70g has been pressed. If it is determined that it has been pressed, the process proceeds to S429; if it is determined that it has not been pressed, the process proceeds to S430. In S429, the system control unit 50 switches the display of photographing information, and proceeds to S430. The shooting information includes a histogram, a highlight warning table, and the like.

In S430, the system control unit 50 determines whether or not the shutter button 61 is pressed halfway, and if it is determined that the shutter button 61 is not pressed halfway, the process proceeds to S436, and if it is determined that it is pressed halfway, the process is continued. Proceed to S431.

In S431, the system control unit 50 performs the AF/AE processing described in FIG. 2, and proceeds to S432. In S432, the system control unit 50 determines whether the shutter button 61 is fully pressed based on whether the signal SW2 is received. If the system control unit 50 determines that the button is not fully pressed, the process proceeds to S433, and if it determines that the button is fully pressed, the process proceeds to S434. In S433, the system control unit 50 determines whether the shutter button 61 is held in a half-pressed state based on whether the signal SW1 is received. If the system control unit 50 determines that the half-pressed state is maintained, the process returns to S431, and if it determines that the half-pressed state is not maintained, the system control unit 50 advances the process to S436.

In S434, the system control unit 50 performs SDR imaging processing and proceeds to S435. In this SDR shooting process, the system control unit 50 develops the RAW image data obtained by SDR shooting in SDR image quality, encodes the SDR image quality into JPEG to generate JPEG image data, and converts the RAW image data obtained by SDR shooting into a JPEG file in JPEG format. Record on a recording medium. If the recording setting is to record only the SDR image as a JPEG file, only the JPEG file is recorded. If recording of JPEG files and RAW image files is set as recording settings, the JPEG file is recorded and the encoded data of the RAW image data obtained by SDR shooting and the JPEG image data are shown in Figure 8A. It is recorded on a recording medium as a RAW image file in the RAW image file format shown below. In the RAW image file, ImageData 809 in the data structure of FIG. 8A has the format shown in FIG. 8B(a). That is, images of each size for display are stored in one file by integrating encoded data obtained by JPEG encoding with 8-bit precision. Then, in S435, the system control unit 50 performs quick review display processing, and proceeds to S436. Details of this quick review display process will be described later using FIG. 4B.

In S436, the system control unit 50 determines whether the LV button 76 has been pressed. If it is determined that it has been pressed, the process proceeds to S437; if it is determined that it has not been pressed, the process proceeds to S438. In S437, the system control unit 50 compresses the SDR image obtained by the SDR image quality development process in S425 in H264, records it as an SDR video file, and advances the process to S438.

In S438, the system control unit 50 determines whether the playback button 79 has been pressed. If the system control unit 50 determines that the playback button 79 has been pressed, the process proceeds to S439, and if it determines that it has not been pressed, the process proceeds to S440. In S439, the system control unit 50 performs playback mode processing, and proceeds to S440. Details of this reproduction mode processing will be described later using FIGS. 9 and 10.

In S440, the system control unit 50 determines whether or not there is an instruction to end the LV mode. If it is determined that there is no instruction to end the LV mode, the process returns to S401, and if it is determined that there is an instruction to end the LV mode, the process returns to S401. This process ends.

FIG. 4B is a flowchart showing quick review display processing by the system control unit 50. This process is realized by loading a program recorded in the nonvolatile memory 56 into the system memory 52 and executing it by the system control unit 50.

In S451, the system control unit 50 determines whether the quick review display is set, and if it is determined that the quick review display is set, the process proceeds to S452, and the quick review display is set to not be displayed. If it is determined that there is, this process ends.

In S452, the system control unit 50 determines whether the shooting is in the HDR shooting mode, and if it is determined that the shooting is in the HDR shooting mode, the process proceeds to S453, and if it is determined that the shooting is in the SDR shooting mode. If determined, the process advances to S460.

In S453, the system control unit 50 determines whether the device (display unit 28 or external device 300) displaying the quick review is HDR compatible, and if it is determined that it is not HDR compatible, the process proceeds to S454. If it is determined that this is the case, the process advances to S457.

In S454, the system control unit 50 determines whether the shooting was performed using RAW still image shooting, and if it is determined that the shooting was performed using RAW still image shooting, the process proceeds to S455, and if it is HIEF still image shooting, the process proceeds to S455. Proceed to S456.

In S455, the system control unit 50 converts the display HDR image 828 in the HDR RAW image from HDR to SDR using the same process as in S406 to S408, and outputs it to the output target device (display unit 28 or external device 300). The image is resized to a suitable size, displayed in SDR image quality, and the process proceeds to S463.

In S456, the system control unit 50 converts the display HDR image in the HEIF image from HDR to SDR using the same processing as in S406 to S408, and converts the display HDR image in the HEIF image to a size suitable for the output target device (display unit 28 or external device 300). The image is resized and displayed in SDR image quality, and the process advances to S463.

In S457, the system control unit 50 determines whether the image was shot using RAW still image shooting, and if it is determined that the image was shot using RAW still image shooting, the process proceeds to S458, and it is determined that the image was shot using HIEF still image shooting. If so, the process advances to S459. In S458, the system control unit 50 resizes the display HDR image 828 in the HDR RAW image to a size suitable for the output target device (display unit 28 or external device 300), displays it in HDR image quality, The process advances to S463. In S459, the system control unit 50 resizes the display HDR image in the HEIF image to a size suitable for the output target device (display unit 28 or external device 300), displays it in HDR image quality, and returns the process to S463. Proceed to.

In S460, the external device 300 determines whether the shooting was performed using RAW still image shooting, and if it is determined that the shooting was performed using RAW still image shooting, the process proceeds to S461, and it is determined that the shooting was performed using HIEF still image shooting. If so, the process advances to S462. In S461, the system control unit 50 resizes the display SDR image 823 in the SDR RAW image to a size suitable for the output target device (display unit 28 or external device 300), displays it in SDR image quality, and processes it. Proceed to S463. In S462, the system control unit 50 resizes the display SDR image in the JPEG image to a size suitable for the output target device (display unit 28 or external device 300), displays it in SDR image quality, and then executes the processing. Proceed to S463.

In S463, the system control unit 50 determines whether the shutter button 61 has been pressed. If it is determined that the shutter button 61 has not been pressed, the process proceeds to S464. If it is determined that the shutter button 61 has been pressed, the process ends.

In S464, the system control unit 50 determines whether the time set in the quick review display time has elapsed, and if it is determined that the time has not elapsed, the process returns to S463, and if it is determined that the time has elapsed, ends this process.

FIG. 5A is a sequence diagram showing a control procedure for the digital camera 100 and the external device 300 when the digital camera 100 and the external device 300 are connected. Here, the explanation will be given assuming that the digital camera 100 and the external device 300 are connected via HDMI.

In S501, the system control unit 50 controls the digital output I/F 90 and instructs to start transmitting the +5V signal. As a result, the digital output I/F 90 starts transmitting the +5V signal. The transmitted +5V signal is transmitted to the external device 300 through a +5V signal line (not shown) of the connection cable 302. The external device 300 receives the +5V signal from the connection cable 302, and proceeds to S502.

In S502, the external device 300 determines that the digital camera 100 has confirmed the connection of the external device 300, and advances the process to S503.

In S503, the external device 300 starts transmitting the HPD signal from the HPD signal line (not shown) of the connection cable 302. The digital output I/F 90 of the digital camera 100 receives the transmitted HPD signal via the connection cable 302. When the digital output I/F 90 receives the HPD signal, it notifies the system control unit 50 of HPD reception.

In S504, the system control unit 50 detects a connection response from the external device 300 based on the notification from the HPD, and advances the process to S505.

In S505, the system control unit 50 controls the digital output I/F 90 to transmit an EDID request signal from the connection cable 302. The transmitted EDID request signal is transmitted to the external device 300 through an EDID signal line (not shown) of the connection cable 302. The external device 300 receives this EDID request signal and advances the process to S506.

In S506, the external device 300 transmits the EDID from the EDID signal line (not shown) of the connection cable 302. The digital output I/F 90 of the digital camera 100 receives this EDID via the connection cable 302. Then, upon receiving the EDID, the digital output I/F 90 notifies the system control unit 50 of the EDID reception.

In S507, the system control unit 50 receives the notification of EDID reception and instructs the digital output I/F 90 to copy the EDID received in S506 to the memory 32. After the copy is completed, the system control unit 50 analyzes the EDID expanded in the memory 32, performs processing to determine the ability of the video signal that the external device 300 can accept, and advances the process to S508.

In S508, the system control unit 50 determines to output an HDR signal to the external device 300 if the main unit setting is HDR enabled and the capability of the video signal that can be received by the external device 300 determined in S507 is compatible with HDR signals, In other cases, it is determined that the SDR signal is to be output, and the process advances to S509.

In S509, the system control unit 50 instructs the digital output I/F 90 to start transmitting the HDR or SDR video signal determined in S508. The digital output I/F 90 that has received the video signal transmission start instruction starts transmitting the video signal through the connection cable 302, and proceeds to S510.

In S510, the digital camera 100 outputs a video signal to the TMDS signal line (not shown) of the connection cable 302. The external device 300 receives the video signal via the TMDS signal line (not shown) of the connection cable 302, and proceeds to S511.

In S511, the external device 300 analyzes the video signal received in S508, switches the drive of the display 301 to a setting that can display the video signal, and advances the process to S512. In S512, the external device 300 displays the video signal received in S508 on the display 301 of the external device 300.

FIG. 5B is a sequence diagram showing a process of switching the video output of the digital camera 100 and the external device 300 from an SDR image to an HDR image.

In this sequence, it is assumed that the connection between the digital camera 100 and the external device 300 has been completed in the sequence explained in FIG. 5A.

In S521, the system control unit 50 instructs the digital output I/F 90 to transmit the SDR video signal, and the process proceeds to S522.

In S522, the digital camera 100 outputs the SDR video signal to the TMDS signal line (not shown) of the connection cable 302. The external device 300 receives the SDR video signal via the TMDS signal line (not shown) of the connection cable 302, and advances the process to S523.

In S523, the external device 300 displays the SDR video received in S522 on the display 301 of the external device 300.

While the digital camera 100 is outputting the SDR signal, the SDR image is displayed on the display 301 of the external device 300 by repeating S521 to S523.

When the digital camera 100 switches video output to the external device 300 from an SDR image to an HDR image, the processes from S524 onwards are executed.

In S524, the system control unit 50 instructs the digital output I/F 90 to stop the SDR video signal, and advances the process to S525.

In S525, the system control unit 50 stops transmitting the video signal to the TMDS signal line (not shown) of the connection cable 302. The external device 300 stops receiving the SDR video signal via the TMDS signal line (not shown) of the connection cable 302, and advances the process to S526.

In S526, the external device 300 stops displaying the video on the display 301 of the external device 300 because reception of the video from the digital camera 100 has stopped.

In S527, the system control unit 50 instructs the digital output I/F 90 to transmit the HDR video signal, and advances the process to S528.

In S528, the system control unit 50 outputs the HDR video signal to the TMDS signal line (not shown) of the connection cable 302. The external device 300 receives the HDR video signal via the TMDS signal line (not shown) of the connection cable 302, and advances the process to S529.

In S529, the external device 300 analyzes the video signal received in S528, switches the drive of the display 301 to a setting that can display the HDR video signal, and proceeds to S530.

In S530, the external device 300 displays the HDR video signal received in S528 on the display 301 of the external device 300.

At this time, the processing time from S529 to S530 varies depending on the performance of the external device 300, and it takes about 1 to 5 seconds until the video is displayed.

FIG. 5C is a sequence diagram showing a process of switching the video output of the digital camera 100 and the external device 300 from an HDR image to an SDR image.

In this sequence, it is assumed that the connection between the digital camera 100 and the external device 300 has been completed in the sequence explained in FIG. 5A.

In S541, the system control unit 50 instructs the digital output I/F 90 to transmit an HDR video signal, and advances the process to S542. In S542, the system control unit 50 outputs the HDR video signal to the TMDS signal line (not shown) of the connection cable 302. Further, the external device 300 receives the HDR video signal via the TMDS signal line (not shown) of the connection cable 302, and advances the process to S523.

In S543, the external device 300 displays the HDR video received in S542 on the display 301 of the external device 300.

While the digital camera 100 is outputting the HDR signal, an HDR image is displayed on the display 301 of the external device 300 by repeating S541 to S543.

When the digital camera 100 switches the video output to the external device 300 from an HDR image to an SDR image, processing from S544 onwards is executed.

In S544, the system control unit 50 instructs the digital output I/F 90 to stop the HDR video signal, and advances the process to S545. In S545, the system control unit 50 stops transmitting the video signal to the TMDS signal line (not shown) of the connection cable 302. The external device 300 stops receiving the HDR video signal via the TMDS signal line (not shown) of the connection cable 302, and advances the process to S546.

In S546, the external device 300 stops displaying the video on the display 301 of the external device 300 because reception of the video from the digital camera 100 has stopped.

In S547, the system control unit 50 instructs the digital output I/F 90 to transmit the SDR video signal, and advances the process to S548.

In S548, the system control unit 50 outputs the SDR video signal to the TMDS signal line (not shown) of the connection cable 302. The external device 300 receives the SDR video signal via the TMDS signal line (not shown) of the connection cable 302, and advances the process to S549.

In S549, the external device 300 analyzes the video signal received in S548, switches the drive of the display 301 to a setting that can display the SDR video signal, and advances the process to S530. In S550, the external device 300 displays the SDR video signal received in S528 on the display 301 of the external device 300.

Note that the processing time from S549 to S550 varies depending on the performance of the external device 300, and it takes about 1 to 5 seconds until the video is displayed.

6A and 6B are flowcharts showing details of the shooting menu processing in S411 and S427 of FIG. 4A. This processing is realized by loading a program recorded in the nonvolatile memory 56 into the system memory 52 and executing it by the system control unit 50.

In S601, the system control unit 50 determines whether to perform HDR photography based on whether the HDR photography mode is enabled by the user. When the system control unit 50 determines that HDR photography is not to be performed, the process proceeds to S602, and a menu for normal SDR photography is displayed. Further, if the system control unit 50 determines that HDR shooting is to be performed, the process proceeds to S602, and a menu for HDR shooting is displayed. In step S603, functions that are not used in conjunction with HDR shooting are grayed out or disabled and displayed on the menu.

In S604, the system control unit 50 determines whether the user has selected a setting item for whether or not to perform HDR photography. If it is determined that the item has been selected, the system control unit 50 advances the process to S605; otherwise, the process advances to S611. In S605, the system control unit 50 determines whether the user has enabled the setting of whether or not to perform HDR photography. If it is determined that the switching has been enabled, the system control unit 50 advances the process to S606; otherwise, the process advances to S607. In S606, the system control unit 50 changes the setting of whether or not to perform HDR photography to valid, and stores the setting value in the system memory 52.

In S607 when the setting for whether or not to perform HDR photography is valid, the system control unit 50 determines whether the HDR assist display setting has been changed by the user. If it is determined that the change has been made, the process advances to S608; otherwise, the process advances to S609. Note that it is desirable that the HDR assist display setting cannot be changed when the setting for whether or not to perform HDR photography is disabled.

In S<b>608 , the system control unit 50 changes whether or not to set the HDR assist display during shooting, and stores the setting value in the system memory 52 . There may be two or more variations of the HDR assist display setting "Yes".

Note that when the HDR shooting settings and HDR assist display settings are changed on the menu screen in this manner, the results of changing the display settings may be reflected in the display at the timing of transition to the live view screen. When those settings are changed by using a specific button on the operation unit 70 on the live view screen instead of the menu screen, the change results are displayed at the timing of the change (the timing when the button is pressed). It may be reflected in

In S609, the system control unit 50 determines whether the user has given an instruction to end the HDR setting menu display process. If it is determined that a termination instruction has been given, the system control unit 50 advances the process to S610.

In S610, the system control unit 50 determines whether the still image recording quality setting item has been selected by the user. If the system control unit 50 determines that the selection has been made, the process proceeds to S611; otherwise, the process proceeds to S651.

In S611, the system control unit 50 determines whether or not the user has input an instruction for HDR photography. If the system control unit 50 determines that an HDR photography instruction has been input, the process proceeds to S612, and if it determines that no such instruction has been input, the process proceeds to S614.

In S612, the system control unit 50 displays a screen for HDR shooting, and in S613 accepts a user's selection of the recording quality for HDR shooting. As the recording quality setting for HDR photography, two image simultaneous outputs of RAW, HDR still image file, and RAW+HDR still image file are prepared as file formats. Regarding the image size, there are various sizes such as Large, which is close to the number of pixels at the time of sensor readout, Middle, which is slightly smaller, and Small, which is even smaller. Furthermore, compression rates for reducing file size capacity range from high image quality (low compression rate) to standard (high compression rate), low image quality (high compression rate), and so on.

In S614, the system control unit 50 displays a screen for SDR imaging, and in S615 accepts the user's selection of the recording quality for SDR imaging. The same options as for HDR photography are provided for the recording image quality settings for SDR photography.

In S651, the system control unit 50 determines whether the user has selected a setting item for video recording quality. If the system control unit 50 determines that the video recording quality setting item has been selected, the process proceeds to S652; otherwise, the process proceeds to S657.

In S652, the system control unit 50 determines the user's setting as to whether or not to perform HDR photography. If the setting indicates that the setting is to be performed, the system control unit 50 advances the process to S653; otherwise, the process advances to S655.

In S653, the system control unit 50 displays a screen for HDR shooting, and in S654 accepts a user selection of the recording quality for HDR shooting. As the recording quality setting for HDR shooting, simultaneous output of three moving images is prepared as file format: RAW moving image, RAW moving image + proxy moving image, HDR moving image file, and RAW + proxy moving image + HDR moving image file. Also, regarding image sizes, there are 8K, 4K, FullHD, HD, VGA, etc. Furthermore, compression rates for reducing file size range from high image quality (low compression rate) such as ALL-I to standard to low image quality (high compression rate) such as IPB. In addition, there are frame rate selections and broadcasting format selections such as NTSC/PAL.

In S655, the system control unit 50 displays a screen for SDR imaging as in S653, and in S656 accepts the user's selection of the recording quality for SDR imaging. The same options as for HDR photography are provided for the recording quality settings for SDR photography.

In S657, the system control unit 50 determines whether the HDR output setting item has been selected by the user. If the system control unit 50 determines that the HDR output setting item has been selected by the user, the process proceeds to S658; otherwise, the process proceeds to S660. In S658, the system control unit 50 determines whether the HDR output setting has been enabled by the user. If it is determined that the HDR output setting has been enabled, the process proceeds to S659; otherwise, the process proceeds to S660. In S659, the system control unit 50 changes the HDR output setting to valid and stores the setting value in the system memory 52.

In S660, the system control unit 50 determines whether the user has selected a setting item for view assist during playback. If the system control unit 50 determines that the view assist setting item during playback has been selected, the process proceeds to S661; otherwise, the process proceeds to S663. In S661, the system control unit 50 determines whether the view assist setting during playback has been switched to enabled by the user, and if it is determined that the setting has been switched to enabled, the process proceeds to S662; otherwise, the process proceeds to S663. Proceed to. In S662, the system control unit 50 changes the view assist setting during playback to valid, and stores the setting value in the system memory 52.

In S663, the system control unit 50 determines whether the setting item for SDR conversion at the time of transfer has been selected by the user, and if it is determined that it has been selected, the process proceeds to S664; otherwise, the process proceeds to S665. Proceed to. In S664, the system control unit 50 determines whether the SDR conversion setting at the time of transfer has been switched to valid by the user, and if it is determined that the setting has been switched to valid, the system control unit 50 changes the SDR conversion setting at the time of transfer to S664. The setting is changed to valid, and the process advances to S665.

In S665, the system control unit 50 determines whether the user has selected any other setting item related to HDR shooting, and if it is determined that it has been selected, the process proceeds to S666; otherwise, the process proceeds to S667. Proceed to. In S666, the system control unit 50 changes other processes to valid, and advances the process to S667.

In S667, the system control unit 50 determines whether the user has given an instruction to exit the menu. If it is determined that the menu cannot be exited, the process returns to S660, and if it is determined that the menu can be exited, the process ends.

FIG. 7 is a flowchart showing details of the HDR imaging process by the system control unit 50. This is a flow in which RAW data written in the memory 32 is subjected to HDR development by the image processing unit 24.

Imaging devices such as digital cameras and digital video cameras are equipped with a white balance function that corrects the color tone of a photographed image depending on the light source at the time of photographing. The white balance function is a function that corrects differences in color tone that change depending on the light source (natural light sources such as sunny or cloudy skies, artificial light sources such as fluorescent lights and incandescent lights), and makes white appear the same regardless of the light source. . In S701 to S703, white balance coefficients necessary for white balance processing are calculated. Furthermore, in this embodiment, in order to prevent blown-out gradations in high-brightness areas such as the sky as much as possible, the image is photographed at a lower exposure than the exposure at which the brightness of a person or the like is appropriate.

First, in S701, the system control unit 50 acquires RAW image data via the memory control unit 15.

In S702, the system control unit 50 performs a white search frame determination process to determine which pixels appear to be white in the RAW image data acquired to calculate the white balance coefficient.

In S703, the system control unit 50 calculates a white balance coefficient based on the white search frame determination result.

Details of the processing in S702 and S703 will be explained using the flowchart of FIG. 12.

In RAW image data, as described above, one pixel has only one component signal of R, G, or B. In order to perform a white search, the system control unit 50 performs debayer processing S1201 to generate signals of all R, G, and B channels for each pixel because it is necessary to convert them into color signals. There are several debayer methods, but for example, a signal can be generated by linear interpolation using a low-pass filter. Since RAW image data is generally affected by noise, optical black has a value rather than 0. Therefore, the system control unit 50 performs a process of subtracting the OB portion from the debayered signal (S1202). Then, the system control unit 50 calculates color signals Cx and Cy from the acquired RGB signals using the following equations (S1203).

ここで、Ｇ₁，Ｇ₂は、ベイヤ配列の２×２画素における２つのＧ成分値である。また、Ｃ_xは色温度を表し、Ｃ_yは緑方向補正量を表している。また、Ｙ_iは輝度値である。

Here, G ₁ and G ₂ are two G component values at 2×2 pixels in the Bayer array. Further, C _x represents the color temperature, and C _y represents the green direction correction amount. Moreover, Y _i is a luminance value.

FIG. 13 illustrates the C _x -C _y plane. As shown in the figure, white is previously photographed using an imaging device from a high color temperature (for example, during the day) to a low color temperature (for example, at sunset), and the color evaluation values Cx and Cy are plotted on the coordinates. By doing so, it is possible to define a white axis 1200 that serves as a reference for detecting white. Since there is some variation in white color in an actual light source, the system control unit 50 provides a certain width on both sides of the white axis 1200 (S1204). This frame with a width on the white axis is called a white search frame 1201.

In S1205, the system control unit 50 plots each pixel after debayering on the C _x -C _y coordinate system, and determines whether it is within the white search frame. In S1206, the system control unit 50 performs brightness exclusion processing to limit pixels to be integrated in the luminance direction among the pixels within the white search frame. This is done to prevent the accuracy of calculating the white balance coefficient from deteriorating, since colors that are too dark are susceptible to noise. Similarly, if a color is too bright, the sensor saturation of one of the channels will cause the R/G or B/G ratio to become unbalanced and the color will move away from the correct color, reducing the accuracy of calculating the white balance coefficient. Do it to prevent. At this time, the brightness of the pixel targeted for brightness exclusion processing is made different between SDR and HDR. In other words, pixels used for calculating a white balance coefficient, which will be described later, are different between SDR and HDR. This is because HDR has higher reproducibility in high brightness regions than SDR. In this embodiment, while SDR targets brightness up to +1EV, HDR targets brightness up to +2EV, making it possible to calculate a white balance coefficient more optimized for HDR. It becomes possible.

In S1207, the system control unit 50 calculates integral values SumR, SumG, and SumB of the respective color evaluation values from C _x and C _y that are within the white search frame and have undergone the brightness exclusion process. Then, in S1208, the system control unit 50 calculates white balance coefficients WBCo _R , WBCo _G , and WBCo _B from the calculated integral values using the following equations.

上式における右辺の“１０２４”は、１色成分が１０ビット精度となっているためである。

"1024" on the right side of the above equation is because one color component has 10-bit precision.

Note that the white balance coefficient may be calculated as a value for the shooting mode (SDR shooting or HDR shooting) set by the user, or may be calculated for both SDR and HDR.

Returning to the explanation of FIG. 7. In S704 to S706, the system control unit 50 calculates a tone correction table necessary for tone correction processing. The details of the gradation correction will be explained using the flowchart of FIG. 14.

In S1221, the system control unit 50 performs WB processing using the WB coefficients generated in the processing of S701 to S703 in FIG. In S1222, the system control unit 50 performs histogram detection. Specifically, the white balance gain value obtained in S1221 is applied to the entire image data, and a histogram is created as luminance information for the pixel values subjected to gamma correction processing. The gamma correction process may be performed using a known look-up table, but it is preferable to use the same gamma characteristics as those used in development. However, in order to save processing time and memory capacity, a simplified gamma characteristic, such as a gamma characteristic approximated by a polygonal line, may be used. Note that the edge portion of the image is generally not important in many cases, and is affected by a decrease in the amount of peripheral light depending on the imaging lens, so the histogram may be created excluding pixels in the peripheral portion.

In S1223, the system control unit 50 performs face detection preprocessing. This is a process that performs reduction processing, gamma processing, etc. on image data to make it easier to detect faces. In S1224, the system control unit 50 performs face detection processing on the preprocessed image data using a known method. Through this face detection processing, the position and size of an area that is considered to be a face (face area), and the reliability of detection are obtained.

In S1225, the system control unit 50 calculates a gradation correction amount (gradation correction amount (A)) to compensate for the exposure correction amount (decrease amount) as a first gradation correction amount. At this time, while ensuring appropriate exposure for dark areas of the image, high-brightness pixels that are higher than a predetermined brightness level are not corrected (at least, the exposure correction amount is not completely compensated for), and the input/output characteristics are not corrected. Calculate the key correction amount. Thereby, it is possible to further suppress overexposure in bright areas after gradation correction. This gradation correction amount can be prepared in advance as a plurality of correction tables corresponding to the exposure correction amount.
In S1226, the system control unit 50 determines that a face has been detected if there is a face area whose reliability is higher than a preset evaluation threshold among the face areas detected by the face detection process in S1224. If the system control unit 50 determines that a face has been detected, the process proceeds to S1227, and if it determines that a face has not been detected, the process proceeds to S1231.

In S1227, the system control unit 50 calculates a part of the detected face area as a face brightness acquisition area. The face brightness acquisition area is an area for acquiring the brightness of bright parts of the face, and there are no particular restrictions on the number or position thereof. In S1228, the system control unit 50 calculates an average value for each type of R pixel, G pixel, and B pixel included in each face brightness acquisition area. Further, the system control unit 50 applies a white balance gain value to each average value of the RGB pixels in the same manner as in histogram detection, performs gamma correction, and then converts it into a brightness value Y using the following formula.
Y=0.299×R+0.587×G+0.114×B
Note that the white balance gain value applied in histogram detection and face detection is preferably the gain value used in WB processing for the same image data. Ideally, it is preferable to use the same luminance gamma as that for development, but in order to save processing time and memory capacity, a simplified gamma characteristic, such as a gamma characteristic approximated by a polygonal line, may be used.

In S1229, the system control unit 50 converts the brightness values obtained for each of the face brightness acquisition regions in S1228 into values assuming proper exposure. This is a process for correcting the fact that since the image data is captured with an exposure lower than the proper exposure, the brightness of the face is detected to be lower than when the image is captured with the proper exposure. The conversion of the brightness value may be performed to compensate for the exposure correction amount (decrease amount) determined by exposure control, or may be performed using the gradation correction amount calculated in S1225.

In S1230, the system control unit 50 calculates a representative value of the brightness of the detected face. The representative value may be, for example, finding the maximum value from among the brightness values of each face brightness acquisition area of the detected face area. This is the process to be performed when it is determined that there is no such thing. In S1231, the system control unit 50 detects a histogram feature amount. The histogram feature amount may be, for example, a level (SD) to which a pixel with a cumulative frequency of 1% from the dark side belongs in the histogram, a level (HL) to which a pixel with a cumulative frequency of 1% from the bright side belongs in the histogram, or the like. In subsequent S1232, the system control unit 50 converts the histogram feature amount calculated in S1231 to a value assuming imaging with proper exposure. This is a process for correcting the fact that the histogram feature amount is detected to be lower than when the image data is captured with an appropriate exposure because the image data is captured with an exposure lower than the proper exposure. The conversion of the brightness value may be performed to compensate for the exposure correction amount (decrease amount) determined by exposure control, or may be performed using the gradation correction amount calculated in S1225.

In S1233, the system control unit 50 calculates the target correction amount. The system control unit 50 determines the target brightness level for the representative brightness value of the face or the histogram feature amount. Then, the system control unit 50 uses these target brightness levels and the minimum and maximum brightness values in the image data to create a lookup table (input/output characteristics) is created as the gradation correction amount (B). The gradation correction amount (B) is the second gradation correction amount.

Here, in HDR, the target gradation correction amount may be different from that in SDR. For example, FIG. 16(a) shows an example of the appearance in SDR, and FIG. 16(b) shows an example of the appearance in HDR. Although the luminance value of the subject (person) is the same, in SDR the brightness value of the background is at most 100 cd/m ² , whereas in HDR it reaches a value exceeding 100 cd/m ² . As a result, even if the brightness value of the subject is the same, the HDR image may feel darker. This is called brightness contrast, and is a phenomenon caused by human visual characteristics. For example, although the brightness of the subject is the same in FIGS. 16(a) and 16(b), the difference between the brightness of the subject and the background is larger in FIG. 16(b) than in FIG. 16(a). In such a case, the user perceives the subject to be darker in FIG. 16(b) than in FIG. 16(a). In other words, since HDR can express high-brightness areas such as the sky brighter, there is a high possibility that the subject will appear darker than with SDR. Therefore, in this embodiment, in the case of SDR, the gradation characteristics shown in FIG. 15(a) are used, whereas in the case of HDR, the gradation characteristics shown in FIG. By applying a gradation correction amount that increases the gradation, it is possible to obtain a visually pleasing result. Note that although the gradation correction in this embodiment is an example of correction for compensating for insufficient exposure, similar gradation correction can also be performed in brightness correction for image creation.

The target brightness level for the representative brightness value of the face and the histogram feature of the image data can be set to a fixed value that is empirically considered preferable, but it varies depending on the representative brightness value and the value of the histogram feature. A target brightness level may also be set. In this case, a lookup table that defines the relationship between the target brightness level and the input level may be prepared for each parameter (representative brightness value or histogram feature amount) for setting the target brightness level.

The correction characteristics for realizing the conversion to the target luminance level determined in this way are determined by a method such as spline interpolation, and if necessary, a look-up table (or relational expression) is used to apply the gradation correction amount (B). ).

In S1234, the system control unit 50 combines the tone correction amount (A) calculated in S1225 and the tone correction amount (B) calculated in S1233. For example, the system control unit 50 first applies the gradation correction amount (A) to each input luminance level, and then applies the gradation correction amount (B) to the corrected luminance level, resulting in a luminance value. , and create a lookup table of output brightness levels for each input brightness level.

In S1235, the system control unit 50 performs processing (limiter processing) for limiting the upper limit value of the synthetic correction amount (synthetic gradation correction amount) obtained in S1234. By combining the gradation correction amount (A) and the gradation correction amount (B), the correction amount increases and the amount of noise becomes more noticeable in the corrected image, so a limit is placed on the overall correction amount. In the limiter processing, the maximum correction amount allowed for each luminance value is prepared as a table, and among the values in the lookup table created in S1234, the output level exceeding the maximum correction amount is set to the output level corresponding to the maximum correction amount. This can be achieved by replacing. Note that the gradation correction amount may be calculated as a value for the shooting mode (SDR shooting or HDR shooting) set by the user, or may be calculated for both SDR and HDR.

Returning to the explanation of FIG. 7. In S707, the system control unit 50 performs development using the calculated white balance coefficient, gradation correction parameter, and various HDR parameters. As other development parameters, a color matrix, camera OETF curve data, color adjustment parameters, noise reduction parameters, sharpness parameters, etc. are used to generate an HDR developed image. As a camera OETF (gamma curve), for example, ITU-R recommendation BT. Although it is assumed that the reverse characteristics of EOTF (Electro-Optical Transfer Function) of PQ (Perceptual Quantization) of 2100 are used, it is also possible to combine the seasoning on the camera side as OOTF (Opto-Optical Transfer Function). Or the same ITU-R recommendation BT. OETF of HLG (Hybrid Log-Gamma) of 2100 may be used.

In S708, the system control unit 50 resizes the image developed in S707, generates an MPF (Multi Pixel Format) image for simple display such as a two-screen comparison image, and compresses and encodes it using HEVC.

In S709, the system control unit 50 further resizes the MPF image generated in S708 to generate and compress a thumbnail image used for index display and the like.

In S710, the system control unit 50 compresses the HDR image developed in S707 as the main image. There are various compression methods that can be considered, but for example, 10-bit YUV422 data can be compressed using H. 265 (ISO/IEC 23008-2 HEVC).

In S711, the system control unit 50 determines the recording quality setting made by the user. If the system control unit 50 determines that the settings are such that only RAW images are recorded, the process proceeds to S712, and if the system control unit 50 determines that the settings are such that only HDR images are recorded, the process proceeds to S713. If it is determined that the settings are such that an image is recorded, the process advances to S714.

In S712, the system control unit 50 compresses the RAW image, adds a header, and records it on the recording medium 200 via the recording medium I/F 18 as a RAW image file having a structure as shown in FIG. 8A. There are several possible compression methods, such as lossless compression, which is reversible without deterioration, and lossy compression, which is irreversible but reduces the file size. In addition, the header records the white balance in-white search frame determination result obtained in S1205, the histogram obtained in S704, and the face detection result obtained in S705 as detection metadata. The white search frame inside determination result detected here is the determination result before performing the brightness/darkness exclusion process in S1206. Therefore, the same determination result will be recorded in both HDR photography and SDR photography. In addition, if the user has set the HDR shooting mode, HDR development parameters such as the white balance coefficient obtained in FIG. 12 and the gradation correction amount obtained in FIG. The encoded and generated MPF image for display is also recorded as metadata as shown in FIG. 8B(b). As described above, the content of these data differs depending on whether HDR photography or SDR photography is performed. Note that in the case of SDR photography, the development parameters when using the above-described white search frame determination and gradation characteristics for SDR are recorded. Note that even during HDR shooting, the processing in S702 to S706 may be performed for SDR, and development parameters for SDR may also be generated and both may be recorded. Note that generating development parameters for both HDR and SDR requires a large processing load, so it may not be performed during continuous shooting, but may be performed when the processing load is relatively leeway, such as during single shooting. .

In addition, when there is sufficient processing load such as when shooting a single shot, separate from the HDR display image, create an SDR image quality main image, MPF image, and thumbnail image using SDR development parameters, and create the HDR image. The display image and the SDR display image may be recorded in the same file (FIG. 8B(c)).

Furthermore, when displaying thumbnails, since the images are small, it is sufficient to know what kind of image they are, so it is possible to create and save only the thumbnail image created in S709 as an SDR developed image (Figure 8B ( d)). With this configuration, H. This makes it possible to display only thumbnail images even on display devices or PCs that do not support H.265 decoding.

In S713, the system control unit 50 compresses and encodes the developed HDR image in HEVC format, adds static metadata or dynamic metadata, and records it as a HEIF file in HIEF (High Efficiency Image File Format) format. Recording is performed on the recording medium 200 via the medium I/F 18. Static metadata includes the x, y coordinates of the three primary colors and white point of the display in accordance with CEA-861.3, the maximum brightness value and minimum brightness value of the mastering display, the maximum content light level, For example, the frame average brightness level (Maximum Frame-average Light Level). Furthermore, the dynamic metadata includes dynamic tone mapping metadata for color volume conversion defined in SMPTE ST 2094, and the like. Note that in order to represent HDR characteristics in a PQ signal, it is preferable to have a depth of 10 bits or more, but since the conventional JPEG format is 8 bits, it is necessary to adopt a new container for still image HDR. Here, a HEIF container is used, which is an image file format developed by MPEG (Moving Picture Experts Group) and defined in MPEG-H Part 12 (ISO/IEC 23008-12). HEIF has the feature that not only the main image but also thumbnails, multiple temporally related images, and metadata such as EXIF and XMP can be stored in one file. Therefore, 10-bit image sequences encoded with HEVC can also be stored, making it easy to reuse.

In S714 and S715, the processes of S712 and S713 are sequentially performed, and both the RAW image and the HDR image are recorded.

Next, FIG. 8A shows the file structure of still image RAW image data recorded on the recording medium 200 in the recording process described above. The file format exemplified below is the ISO base media file format defined by ISO/IEC14496-12. Therefore, this file format has a tree structure with each node called a box. Furthermore, each box can have multiple boxes as child elements.

The image data 801 has at the beginning a box ftyp 802 for describing the file type, a box moov 803 containing all metadata, a box mdat 808 for the media data of the track, and an other box 807. The aforementioned box moov 803 has as child elements a box uuid 804 that stores MetaData 805 and a trak box 806 that stores information that refers to ImageData. MetaData 805 describes image metadata, and includes, for example, the date and time of image creation, shooting conditions, information on whether the image was shot with HDR or SDR, the aforementioned detection metadata, and other shooting information. The box mdat 808 described above has ImageData 809, which is captured still image data, as a child element.

Note that the image data recorded in ImageData 809 is different between a RAW image shot with SDR and a RAW image shot with HDR.

FIG. 8B(a) is ImageData 809 recorded in a RAW image captured by SDR. ImageData 809 in this case includes a THM image 821 that has been developed with SDR image quality and compressed with JPEG, an MPF image 822, a main image 823, a RAW image 824, and RAW development parameters 825. Each SDR image quality is an image in which one color component is 8 bits (256 gradations). Note that the RAW development parameters 825 in FIG. 8B(a) include at least development parameters for SDR development.

FIG. 8B(b) shows ImageData 809 that is recorded in a RAW image that has only the HDR image as a display image during HDR shooting. ImageData 809 in this case includes a THM image 826 that has been developed with HDR image quality and compressed with HEVC, an MPF image 827, a main image 828, a RAW image 824, and RAW development parameters 825. Each HDR image quality is an image in which one color component has 10 bits (1024 gradations). The RAW development parameters 825 in FIGS. 8B(b), (c), and (d) include at least development parameters for HDR development.

FIG. 8B(c) is ImageData 809 that is recorded in a RAW image that has both an HDR image and an SDR image as display images during HDR shooting. In this case, ImageData 809 includes a THM image 821 developed with SDR image quality and compressed with JPEG, an MPF image 822, a main image 823, a THM image 826 developed with HDR image quality and compressed with HEVC, an MPF image 827, a main image 828, a RAW image 824, and a RAW image 824. It has development parameters 825.

In FIG. 8B(d), only the THM image is an SDR image when photographed in HDR, and the MPF and main image are ImageData 809 recorded in a RAW image held as a display image of the HDR image. ImageData 809 in this case includes a THM image 821 developed with SDR image quality and compressed with JPEG, an MPF image 827 developed with HDR image quality and compressed with HEVC, an original image 828, a RAW image 82), and RAW development parameters 825.

The file format shown in this example is one of the embodiments, and other boxes may be included as necessary. Alternatively, the display image may be stored in a box within the moov 803 or in another box 807.

By having the above file format, development parameters for SDR images are recorded in RAW image files captured as SDR, and development parameters for HDR images are recorded in RAW image files captured as HDR. That will happen. By doing this, even when developing a RAW image later, it is possible to perform the development using development parameters that reflect the settings at the time of photography. For example, a device that performs RAW development (which may be the digital camera 100 or another device such as a PC) refers to the MetaData 805 of the RAW image and determines whether the image was shot as HDR or SDR. If it is determined that the image was shot as HDR, the RAW image is developed as an HDR image using the development parameters for HDR images included in the file. Further, if it is determined that the image was shot as an SDR image, the RAW image is developed as an SDR image using the development parameters for the SDR image included in the file. In order to perform such processing, the digital camera 100 in this embodiment records development parameters for SDR images in RAW image files shot as SDR, and records development parameters for SDR images in RAW image files shot as HDR. Development parameters for HDR images are recorded in the file. Note that the device that performs RAW development may be configured to record still HDR images developed using the HEIF container described above.

Furthermore, since we decided to record the same judgment result as detection metadata for HDR shooting and SDR shooting, even if it is a RAW image file shot in HDR shooting mode, it will be developed into an SDR image using the recorded detection data. It also becomes possible. Therefore, even a device that supports only SDR images can appropriately display a RAW image file shot in HDR shooting mode.

FIG. 9A is a flowchart showing details of playback mode processing performed by the system control unit 50 using the display unit 28. This processing is realized by loading a program recorded in the nonvolatile memory 56 into the system memory 52 and executing it by the system control unit 50.

In S901, the system control unit 50 determines whether index playback or normal playback is performed. In S901, if the system control unit 50 determines that index playback is to be performed, the process proceeds to S902. In S902, the system control unit 50 determines the number of images to be reproduced.

In S903, the system control unit 50 determines the image to be reproduced. Then, in S904, the system control unit 50 performs drawing processing for the image to be reproduced.

In S905, the system control unit 50 determines whether drawing of all images to be displayed has been completed. If it is determined that drawing has not been completed, the system control unit 50 returns to S903 and continues the drawing process. Further, when the system control unit 50 determines that the drawing process is completed, the system control unit 50 advances the process to S906. In S906, the system control unit 50 performs the image output process of S906 to the display unit 28, and ends the display process. After that, the system control unit 50 performs an operation reception process.

FIG. 9B is a flowchart showing details of the drawing process of the playback mode process using the display unit 28 by the system control unit 50.

In S911, the system control unit 50 acquires information about the image to be reproduced. In S912, the system control unit 50 determines the image to be reproduced. In S913, the system control unit 50 reads the image to be reproduced from the recording medium 200. In S914, the system control unit 50 performs decompression processing on the image to be reproduced. In S915, the system control unit 50 collects data for each pixel from the image data subjected to the decompression process in S914. This image data is, for example, brightness information, and is used for histogram processing, highlight warning processing, and the like.

In S916, the system control unit 50 determines whether the image to be reproduced is an HDR image or an SDR image. If the system control unit 50 determines that the image to be reproduced is an HDR image, the process proceeds to S917, and if it determines that the image to be reproduced is an SDR image, the process proceeds to S920. In S917, the system control unit 50 checks the HDR assist display setting during playback, and if the assist setting is 1, the process proceeds to S918, and if the assist setting is 2, the process proceeds to S919. In S918, the system control unit 50 performs HDR→SDR conversion processing on the image decompressed in S914 according to the assist 1 setting. Further, in S919, the system control unit 50 performs HDR→SDR conversion processing on the image decompressed in S914 according to the assist 2 settings.

In S920, the system control unit 50 performs scaling processing on the image expanded in S914 or the image subjected to SDR conversion processing in S918 and S910 to a size suitable for the display unit 28. Then, in S921, the system control unit 50 determines the arrangement of the generated image, and ends the drawing process.

9C to 9H are flowcharts showing details of the read image selection process by the system control unit 50.

In S926, the system control unit 50 determines the information of the acquired image and determines whether it is reproducible. If it is determined that it is reproducible, the process proceeds to S927; if it is determined that replay is not possible, the process proceeds to S927. Proceed to S936.

In S927, the system control unit 50 determines whether the image to be reproduced is a still image. If the system control unit 50 determines that the image to be reproduced is a still image, the process proceeds to S928; otherwise, the process proceeds to S935.

In S928, the system control unit 50 determines whether the image to be reproduced is a RAW image. If the system control unit 50 determines that the image to be reproduced is a RAW image, the process proceeds to S929; otherwise, the process proceeds to S930.

In S929, the system control unit 50 determines whether the RAW image is a RAW image shot with HDR or a RAW image shot with SDR. The system control unit 50 makes this determination using the metadata in the RAW file, which was explained with reference to FIG. If the system control unit 50 determines that the image is a RAW image shot in HDR, the process proceeds to S931, and if it determines that the image is an SDR image, the process proceeds to S932.

In S930, the system control unit 50 determines whether the still image determined not to be a RAW image is an image shot using HDR or an image shot using SDR. In this embodiment, images shot in HDR are recorded as HEIF, and images shot in SDR are recorded as JPEG, so whether it is an HDR image or an SDR image is determined by HEIF or JPEG. It may be used to determine whether the image is an HDR image or an SDR image.

In S931, the system control unit 50 selects image data to be used for playback from RAW images shot in HDR. In S932, the system control unit 50 selects image data to be used for reproduction from the RAW images shot with SDR. In S933, the system control unit 50 selects image data to be used for reproduction from the HDR developed still image. In S934, the system control unit 50 selects image data to be used for reproduction from the SDR developed still image. In S935, the system control unit 50 selects image data to be displayed from among the video files. In S936, the system control unit 50 performs processing to hide the reproduced image. In this case, information indicating that the image cannot be reproduced is displayed to notify the user that the image cannot be reproduced.

FIG. 9D is a flowchart in which the system control unit 50 selects image data to be used for reproduction from RAW images shot in HDR.

In S941, the system control unit 50 determines whether index playback or normal playback is performed. If the system control unit 50 determines that index playback is being performed, the process proceeds to S942, and if it is determined that normal playback is being performed, the process proceeds to S943.

In S942, the system control unit 50 determines the image data to be used based on the number of images to be reproduced in index reproduction. In this embodiment, the threshold value is 36 images, but this number is just one example, and may be set as appropriate by the user, or may be determined depending on the size of the display unit 28. If the system control unit 50 determines that the number of images to be displayed is 36 or more, the process proceeds to S945, and if it determines that the number of images to be displayed is less than 36, the process proceeds to S944.

In S943, the system control unit 50 determines "HDR main image for display (HEVC)" (828) as image data to be used for reproduction. In S944, the system control unit 50 determines "HDR MPF image for display (HEVC)" (827) as image data to be used for reproduction. In S945, the system control unit 50 determines "HDR THM image for display (HEVC)" (826) as image data to be used for reproduction.

FIG. 9E is a flowchart showing a process of selecting image data to be used for reproduction from the RAW image when the RAW image shot in HDR has an SDR display image.

In S951, the system control unit 50 determines whether index playback or normal playback is performed. If the system control unit 50 determines that index playback is being performed, the process proceeds to S952, and if it is determined that normal playback is being performed, the process proceeds to S953.

In S952, the system control unit 50 determines the image data to be used based on the number of images to be reproduced in index reproduction. Here, 36 images is used as the threshold for determination. If the system control unit 50 determines that the number of reproduced images is 36 or more, the process proceeds to S955, and if it determines that the number is less than 36, the process proceeds to S954.

In S953, S954, and S955, the system control unit 50 checks whether the RAW image to be reproduced includes an SDR image. The determination at this time uses the metadata in the RAW file, which was explained with reference to FIG.

In S956, the system control unit 50 determines "HDR main image for display (HEVC)" (828) as image data to be used for reproduction. In S957, the system control unit 50 determines "SDR main image for display (JPEG)" (823) as image data to be used for reproduction. In S958, the system control unit 50 determines the "HDR MPF image for display (HEVC)" (827) as the image data to be used for reproduction. In S959, the system control unit 50 determines "SDR MPF image for display (JPEG)" (822) as image data to be used for reproduction. In S960, the system control unit 50 determines "HDR THM image for display (HEVC)" (826) as image data to be used for reproduction. In S961, the system control unit 50 determines "SDR THM image for display (JPEG)" (821) as image data to be used for reproduction.

FIG. 9F is a flowchart in which the system control unit 50 selects image data to be used for reproduction from the HDR developed still image.

In S971, the system control unit 50 determines whether index playback or normal playback is performed. If the system control unit 50 determines that index playback is to be performed, the process proceeds to S972, and if it is determined that normal playback is to be performed, the process proceeds to S973.

In S972, the system control unit 50 determines the image data to be used based on the number of images to be reproduced in index reproduction. In the embodiment, the threshold value for the number of sheets is set to 36 sheets. If the system control unit 50 determines that the number of reproduced images is 36 or more, the process proceeds to S975, and if it determines that the number is less than 36, the process proceeds to S974.

In S973, the system control unit 50 determines "HDR main image (HEVC)" (not shown) as image data to be used for reproduction. In S974, the system control unit 50 determines "HDR MPF image (HEVC)" (not shown) as image data to be used for reproduction. In S975, the system control unit 50 determines "HDR THM image (HEVC)" (not shown) as image data to be used for reproduction.

FIG. 9G is a flowchart for selecting image data to be used for reproduction from RAW images shot with SDR.

In S981, the system control unit 50 determines whether index playback or normal playback is performed. If the system control unit 50 determines that index playback is to be performed, the process proceeds to S982, and if it is determined that normal playback is to be performed, the process proceeds to S983.

In S982, the system control unit 50 determines the image data to be used based on the number of images to be reproduced in index reproduction. In the embodiment, the threshold value for the number of sheets is set to 36 sheets. If the system control unit 50 determines that the number of images to be displayed is 36 or more, the process proceeds to S985, and if it determines that the number of images to be displayed is less than 36, the process proceeds to S984.

In S983, the system control unit 50 determines "SDR main image for display (JPEG)" (823) as image data to be used for reproduction. In S984, the system control unit 50 determines "SDR MPF image for display (JPEG)" (822) as image data to be used for reproduction. In S985, the system control unit 50 determines "SDR THM image for display (JPEG)" (821) as image data to be used for reproduction.

FIG. 9H is a flowchart for selecting image data to be used for reproduction from the SDR developed still image.

In S991, the system control unit 50 determines whether index playback or normal playback is performed. If the system control unit 50 determines that index playback is to be performed, the process proceeds to S992, and if it is determined that normal playback is to be performed, the process proceeds to S993.

In S992, the system control unit 50 determines the image data to be used based on the number of images to be reproduced in index reproduction. In the embodiment, the threshold value for the number of sheets is set to 36 sheets. If the system control unit 50 determines that the number of images to be reproduced is 36 or more, the process proceeds to S995, and if it determines that the number of images to be reproduced is less than 36, the process proceeds to S994.

In S993, the system control unit 50 determines "SDR main image (JPEG)" (not shown) as the image data to be used for reproduction. In S994, the system control unit 50 determines "SDR MPF image (JPEG)" (not shown) as image data to be used for reproduction. In S995, the system control unit 50 determines "SDR THM image (JPEG)" (not shown) as image data to be used for reproduction.

FIG. 10A is a flowchart showing details of playback mode processing using external device 300. This processing is realized by loading a program recorded in the nonvolatile memory 56 into the system memory 52 and executing it by the system control unit 50.

In S1001, the system control unit 50 determines whether the external device 300 is connected to the digital camera 100. If the system control unit 50 determines that the external device 300 is connected, the process proceeds to S1002, and if it determines that the external device 300 is not connected, the process proceeds to S1005.

In S1002, the system control unit 50 determines whether the HDR setting during playback is valid. As playback settings, "HDR playback", "HDR not playback", and "shooting mode linkage" can be selected. If "HDR playback" is set, regardless of whether the image to be played back is an HDR image or an SDR image, if the external device 300 supports HDR, the mode is HDR output; ” is the mode for SDR output. "Shooting mode linked" is a mode in which the output during playback is linked to the shooting mode. In other words, in the case of HDR shooting mode where "HDR shooting" is set to "Yes", HDR output is also performed during playback, and in the case of SDR shooting mode where "HDR shooting" is set to "No", during playback. This is also a mode for SDR output. Note that the default setting is "linked to shooting mode", and even if the user changes the shooting mode, the playback setting remains "linked to shooting mode". Only when the user changes the playback setting from "shooting mode linkage" to "HDR playback" or "HDR playback", the linkage is broken. Further, file formats such as "HEIF (playback)" and "JPEG (playback)" may be selected instead of "HDR playback" and "HDR playback not". Note that similarly, file formats such as "HEIF (photography)" and "JPEG (photography)" may be selected instead of "HDR photographing" and "HDR photographing not".

In S1002, the system control unit 50 advances the process to S1003 in the case of "HDR reproduction", and advances the process to S1005 in the case of "Do not perform HDR reproduction". Further, in the case of "shooting mode linked", if "HDR shooting" set in S606 is "Yes", the system control unit 50 advances the process to S1003, and if "No", the system control unit 50 advances the process to S1005. Proceed.

In S1003, the system control unit 50 determines whether the external device 300 is a display compatible with HDR. If the system control unit 50 determines that the display supports HDR, the process proceeds to S1004, and if it determines that the display does not support HDR, the process proceeds to S1005.

In S1004, system control unit 50 outputs an HDR signal to external device 300. In S1005, system control unit 50 outputs an SDR signal to external device 300.

S1006 to S1011 are the same as S901 to S906 in FIG. 9A, so the explanation here will be omitted.

FIG. 10B is a flowchart showing details of the drawing process (S1009) when an HDR signal is output to the external device 300.

S1021 to S1025, S1028, and S1029 are the same as S911 to S915, S920, and S921 described with reference to FIG. 9B, so the description thereof will be omitted here.

In S1026, the system control unit 50 determines whether the image to be reproduced is an HDR image or an SDR image. When the system control unit 50 determines that the image to be reproduced is an HDR image, the process proceeds to S1028, and when it determines that the image to be reproduced is an SDR image, the process proceeds to S1027.

In S1027, the system control unit 50 performs SDR→HDR conversion processing. S1028 and S1029 after this are the same as S920 and S921 in FIG. 9B. Note that the details of the drawing process (S1009) when the SDR signal is output to the external device 300 are the same as those in FIG. 9B, so the description will be omitted.

FIG. 11(a) is a flowchart showing details of the reproduction menu process. This processing is realized by loading a program recorded in the nonvolatile memory 56 into the system memory 52 and executing it by the system control unit 50.

In S1101, the system control unit 50 determines whether the user has set RAW development in a RAW development setting item (not shown). If the system control unit 50 determines that RAW development is not to be performed, the process proceeds to S1103, and if it is determined that RAW development is to be performed, the process proceeds to S1102.

In S1103, the system control unit 50 determines whether HDR→SDR conversion is set in the SDR conversion setting item (not shown) of the HDR file. If the system control unit 50 determines not to perform HDR→SDR conversion, the process proceeds to S1105, and if it determines to perform HDR→SDR conversion, the process proceeds to S1104.

In S1105, the system control unit 50 determines whether file transfer is set in a file transfer setting item (not shown). If the system control unit 50 determines not to transfer the file, the process proceeds to S1107, and if it determines to transfer the file, the process proceeds to S1106.

In S1107, the system control unit 50 determines whether to exit the menu. If the system control unit 50 determines that the menu cannot be exited, the process returns to S1101, and if it determines that the menu can be exited, the system control unit 50 ends this reproduction menu processing.

In S1106, the system control unit 50 performs a transfer process on the image file specified by the user. When transferring an HDR image file, if the receiving destination can only display SDR, the HDR→SDR conversion shown in S1104 may be performed within the camera and then transferred as an SDR image file.

In S1102, the system control unit 50 performs RAW development on the RAW image file specified by the user. The details of this RAW development process will be explained below with reference to the block diagram of FIG. 11(b). Note that each processing unit shown in FIG. 11(b) is assumed to be included in the image processing unit 24, but may be realized by a program executed by the system control unit 50.

The system control unit 50 reads out the photographed RAW image 1101 recorded on the recording medium 200 and causes the image processing unit 24 to perform RAW development processing. Since a RAW image is a collection of pixels in a Bayer array, one pixel has only intensity data of a single color component. Note that RAW images include RAW (SDR) when SDR photography is performed and RAW (HDR) when HDR photography is performed. Further, during development, there may be cases in which RAW (SDR) is directly SDR developed or HDR developed. On the other hand, there are cases where RAW (HDR) is subjected to HDR development and SDR development. The white balance unit 1102 performs processing to make white white. In the case of HDR development of RAW (HDR), white balance processing is performed using a white balance coefficient for HDR which is an HDR development meta recorded in a file. Conversely, when performing SDR development, a white balance coefficient for SDR is generated from the white search frame determination result, which is a detection meta stored in a file, and white balance processing is performed. Of course, if both white balance coefficients for HDR and SDR are recorded in RAW, the necessary one may be used as appropriate.

The color interpolation unit 1103 generates a color image in which each pixel has three components (for example, R, G, and B color information) for all pixels by performing noise reduction and interpolating the color mosaic image. The generated color image passes through a matrix conversion unit 1104 and a gamma conversion unit 1105 to generate a basic color image. Thereafter, the color brightness adjustment unit 1106 performs processing on the color image to improve the appearance of the image. For example, image correction such as evening scene detection and saturation enhancement is performed depending on the scene. Tone correction is performed in the same way, but when RAW (HDR) is subjected to HDR development, tone correction is performed using the HDR tone correction amount, which is the HDR development meta stored in the file. Conversely, in the case of SDR development, the gradation correction amount for SDR is calculated using the face detection result and histogram, which are detection meta recorded in the file, and gradation correction is performed. Of course, if both gradation correction amounts for HDR and SDR are recorded in RAW, the necessary one may be used as appropriate.

A compression unit 1107 compresses the high-resolution image of the image that has been subjected to desired color adjustment using a method such as JPEG or HEVC, and a recording unit 1108 generates a developed image to be recorded on a recording medium such as a flash memory. . Note that since a plurality of images can be stored in the HEIF container described above, it is possible to store not only the HDR developed image but also the SDR developed image.

In S1104, the system control unit 50 performs SDR conversion on the HDR image file specified by the user. The HDR image has an OETF of PQ and a color gamut of BT. Since the image is generated in a color space such as 2020, it is necessary to perform tone mapping and gamut mapping processing in a color space such as SDR γ2.2 or sRGB. As a specific method, a known technique may be used, but for example, if tone mapping is performed to match the appropriate exposure with SDR, it is possible to obtain a result in which the brightness is matched compared to SDR.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the invention. For example, in the above embodiment, HEVC (High Efficiency Video Coding) is adopted to encode image data in which one color component exceeds 8 bits, but it is not possible to encode an image in which one color component exceeds 8 bits. As long as it is possible, the type does not matter. Further, although the above embodiment has been described as being applied to a digital camera, the present invention may be applied to a computer having an imaging function (such as a smartphone or a notebook PC with a camera), and is not limited to the above embodiment.

The invention is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the following claims are hereby appended to disclose the scope of the invention.

DESCRIPTION OF SYMBOLS 100... Digital camera, 150... Lens unit, 300... External device, 50... System control part, 70... Operation part, 200... Recording medium

Claims (1)

an imaging means;
an encoding means that performs encoding processing on image data;
When recording a RAW image file, first encoded image data obtained by performing encoding processing in a first encoding format on image data of a first dynamic range generated based on RAW image data is recorded as a RAW image file. The second encoded image data is recorded as a display image of an image file, or the second encoded image data is obtained by performing encoding processing of the second encoding format on the second dynamic range image data generated based on the RAW image data. , a control means for controlling whether to record the RAW image file as a display image;
An imaging device comprising:

Images