JP2020141189A - Imaging apparatus and control method of the same - Google Patents

Abstract

To provide a technique for obtaining moving image data to which suitable dynamic metadata is added in a photographing step (imaging apparatus) not in post-production (image data editing step).SOLUTION: An imaging apparatus 100 includes: setting means for setting a photographing mode; generation means for generating metadata indicating features of a photographed moving image; and adding means for adding the metadata generated by the generation means to moving image data representing the moving image. The generation means generates metadata indicating the features of the moving image over a period according to the photographing mode set by the setting means.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image pickup apparatus and a control method thereof.

In recent years, a technique for sequentially changing the display method of a moving image (for example, in each frame or each scene) by using metadata (dynamic metadata) indicating the maximum brightness of each frame of the moving image, each scene, etc. has become widespread. I'm starting. Specifically, a technology that sequentially changes the HDR (high dynamic range) display according to the maximum brightness of a moving image frame, scene, etc. by sequentially changing the tone curve (tone map) using dynamic metadata. Is beginning to spread. Patent Document 1 discloses a technique of generating dynamic metadata of each scene of a moving image and adding it to the moving image data in a post-production image data editing process.

International Publication No. 2015/017314

However, in the technique disclosed in Patent Document 1, if there is no post-production (image data editing process), dynamic metadata cannot be obtained, and the display method such as HDR display using a tone curve is sequentially changed. Is difficult.

An object of the present invention is to provide a technique capable of obtaining moving image data to which suitable dynamic metadata is added in a shooting process (imaging apparatus) instead of post-production (image data editing process).

The first aspect of the present invention is
Setting means to set the shooting mode and
A generation means for generating metadata indicating the features of the captured moving image, and
An additional means for adding the metadata generated by the generation means to the moving image data representing the moving image, and
Have,
The generation means is an imaging device characterized in that it generates the metadata indicating the feature amount of the moving image in a period corresponding to the shooting mode set by the setting means.

The second aspect of the present invention is
Setting steps to set the shooting mode and
A generation step to generate metadata showing the features of the captured video,
An additional step of adding the metadata generated in the generation step to the moving image data representing the moving image,
Have,
The generation step is a control method of an image pickup apparatus, characterized in that the metadata indicating the feature amount of the moving image in a period corresponding to the shooting mode set in the setting step is generated.

A third aspect of the present invention is a program for making a computer function as each means of the above-mentioned imaging device.

According to the present invention, it is possible to obtain moving image data to which suitable dynamic metadata is added in a shooting step (imaging apparatus) instead of post-production (image data editing step).

It is a block diagram which shows the structural example of the image pickup apparatus which concerns on Example 1. FIG. It is a flowchart which shows an example of the photographing process which concerns on Example 1. FIG. It is a schematic diagram which shows an example of the time-lapse photography mode which concerns on Example 1. FIG. It is a schematic diagram which shows an example of the dynamic metadata which concerns on Example 1. FIG. It is a schematic diagram which shows an example of the slow shooting mode which concerns on Example 1. FIG. It is a schematic diagram which shows an example of the dynamic metadata which concerns on Example 1. FIG. It is a schematic diagram which shows an example of the tone curve which concerns on Example 1. FIG. It is a schematic diagram which shows an example of the special effect priority mode which concerns on Example 2. FIG. It is a schematic diagram which shows an example of the dynamic metadata which concerns on Example 2. FIG. It is a schematic diagram which shows an example of the time priority mode which concerns on Example 2. FIG. It is a schematic diagram which shows an example of the dynamic metadata which concerns on Example 2. FIG.

<Example 1>
Hereinafter, Example 1 of the present invention will be described. FIG. 1 is a block diagram showing a configuration example of the image pickup apparatus 100 according to the present embodiment. The image pickup device 100 includes an image pickup optical system 101, an image sensor 102, an image pickup control unit 103, a feature amount acquisition unit 104, a metadata generation unit 105, a metadata addition unit 106, a storage unit 107, a reproduction unit 108, and a shooting mode setting unit 109. , The output IF 110, the CPU 111, the RAM 112, the ROM 113, and the operation unit 114. In this embodiment, "imaging" is a process of generating image data of an image (still image or moving image) representing a subject, and "shooting" is an imaging process and stores the image data generated by the imaging. This is a process of recording in 107.

The image pickup optical system 101 forms (forms) an optical image representing the subject on the image pickup element 102. The imaging optical system 101 includes, for example, a lens group (zoom lens, focus lens, etc.), an aperture adjusting device, a shutter device, and the like.

The image sensor 102 captures a still image (subject image) representing the subject to generate still image data representing the still image, or images a moving image (subject image) representing the subject to represent the moving image. Generate data. Specifically, the image sensor 102 performs a photoelectric conversion process for converting the imaged optical image (light incident from the subject via the image pickup optical system 101) into an analog electric signal. Then, the image pickup element 102 further performs an AD conversion process (analog-digital conversion process) for converting the analog electric signal obtained by the photoelectric conversion process into a digital electric signal (image data of one frame of a moving image; frame image data). .. After that, the image sensor 102 outputs the frame image data obtained by the AD conversion process to the feature amount acquisition unit 104 and the metadata addition unit 106. By repeating these processes, the image sensor 102 sequentially outputs a plurality of frame image data corresponding to each of the plurality of frames of the moving image.

The image pickup control unit 103 controls the image pickup conditions of the image pickup apparatus 100. In this embodiment, the image pickup control unit 103 controls the exposure (exposure condition) of the image pickup device 100 according to the user operation on the image pickup device 100 (operation unit 114), the state of the image pickup device 100, and the like. For example, the image pickup control unit 103 controls the exposure by controlling the aperture, the charge accumulation time of the image pickup device 102, and the like. Specifically, the image pickup control unit 103 controls the state of the image pickup optical system 101, the processing of the image pickup element 102, and the like according to the gain value, the shutter speed, the aperture value, and the like specified by the user. Control the exposure. Further, the image pickup control unit 103 controls the focus of the image pickup device 100 according to the user operation on the image pickup device 100, the state of the image pickup device 100, and the like. For example, the image pickup control unit 103 controls the focus by controlling the drive amount, drive direction, and the like of the focus lens. Specifically, when the AF (autofocus) shooting mode is set, the image pickup control unit 103 sets the position of the focus lens to a predetermined position and shapes the contrast of the frame image (one frame image of the moving image). Calculate (distribution). Then, the image pickup control unit 103 performs AF control with the position having the highest contrast among the plurality of positions in the frame image as the position (focus position) for focusing the luminous flux with the image pickup element 102.

The feature amount acquisition unit 104 acquires the feature amount (frame feature amount) of the frame image data output from the image sensor 102 from the frame image data. In this embodiment, the feature amount acquisition unit 104 acquires the maximum brightness (frame maximum brightness) of the image of one frame represented by the frame image data, and notifies the metadata generation unit 105 of the acquired maximum frame brightness. In this embodiment, the feature amount acquisition unit 104 sequentially notifies a plurality of frame maximum luminance corresponding to each of a plurality of frames of the captured moving image.

The metadata generation unit 105 generates dynamic metadata indicating the feature amount of the captured moving image (the moving image to be captured). In this embodiment, the metadata generation unit 105 sets one or more scene maximum brightness (maximum brightness of one scene) of the captured moving image based on the plurality of frame maximum brightness sequentially notified from the feature amount acquisition unit 104. Generate dynamic metadata that shows the scene in. Alternatively, the metadata generation unit 105 generates dynamic metadata indicating the frame maximum brightness of each frame of the captured moving image based on the plurality of frame maximum brightness sequentially notified from the feature amount acquisition unit 104. The metadata generation unit 105 outputs the generated dynamic metadata to the metadata addition unit 106.

In this embodiment, an example in which the feature amount indicated by the dynamic metadata is the maximum brightness such as the frame maximum brightness and the scene maximum brightness will be described, but the feature amount is not limited to the maximum brightness. For example, the feature quantity indicated by the dynamic metadata may be another representative luminance (average luminance, minimum luminance, mode luminance, intermediate luminance, etc.) such as one frame, one scene, and the like. The feature quantity indicated by the dynamic metadata may be a luminance histogram of one frame, one scene, and the like. The feature quantity indicated by the dynamic metadata may be a feature quantity related to a color such as one frame, one scene, and the like. The feature quantity indicated by the dynamic metadata may be a feature quantity relating to both brightness and color such as one frame, one scene, and the like.

The metadata addition unit 106 adds dynamic metadata output from the metadata generation unit 105 to the moving image data representing the captured moving image (moving image to be captured). The moving image data representing the captured moving image (moving image to be captured) is moving image data including a plurality of frame image data sequentially output from the image sensor 102. For example, the metadata addition unit 106 applies encoding processing based on MPEG-4 AVC or HEVC (High Efficiency Video Coding) to the moving image data. Then, the metadata addition unit 106 adds the dynamic metadata to the SEI (Supplemental Enhancement Information) of MPEG-4 AVC or HEVC. The metadata addition unit 106 records the moving image data after the dynamic metadata is added in the storage unit 107.

The storage unit 107 is a random access recording medium such as a CF (compact flash) card, and stores moving image data (moving image data after dynamic metadata is added) output from the metadata addition unit 106. The storage unit 107 is removable from the image pickup device 100, and can be attached to a device (PC (personal computer) or the like) other than the image pickup device 100. The storage unit 107 may be a built-in recording medium that is not removable from the image pickup device 100.

The reproduction unit 108 reproduces the moving image data stored in the storage unit 107. In this embodiment, "reproduction" is a process of reading moving image data from the storage unit 107 and outputting it to the outside of the image pickup apparatus 100 via the output IF 110.

The shooting mode setting unit 109 sets the shooting mode. In this embodiment, the photographing mode setting unit 109 is connected to another functional block of the image pickup apparatus 100 via an internal bus (not shown). The shooting mode setting unit 109 changes the set values, processing, and the like of other functional blocks of the image pickup apparatus 100 according to the set shooting mode. In this embodiment, the shooting mode setting unit 109 changes the processing of the metadata generation unit 105 so that dynamic metadata indicating the feature amount of the moving image in the period corresponding to the set shooting mode is generated. This allows suitable dynamic metadata to be generated. Details will be described later. The shooting mode setting unit 109 sets, for example, a time-lapse shooting mode, a continuous shooting mode, a slow shooting mode, or a moving image simultaneous shooting mode.

The time-lapse shooting mode is a shooting mode in which the playback time of the moving image is shorter than the shooting time of the moving image. For example, the time-lapse shooting mode is a shooting mode for intermittent shooting, and is a shooting mode for shooting a time-lapse movie that is played back faster than real time. In the time-lapse shooting mode, a movie shot with frames thinned out from a movie shot at a normal (standard) frame rate is shot. In a moving image shot in the time-lapse shooting mode, it is possible to visually recognize the temporal change of the subject in a short time.

The image pickup apparatus 100 can also capture and reproduce a still image. For example, the image pickup apparatus 100 can take a still image and record the still image data representing the still image in the storage unit 107 by adding metadata. The continuous shooting mode is a shooting mode in which a plurality of still images are sequentially shot. The playback unit 108 can sequentially reproduce a plurality of still images shot in the continuous shooting mode (continuous playback).

The slow shooting mode is a shooting mode in which the playback time of the moving image is longer than the shooting time of the moving image. For example, the slow shooting mode is a shooting mode in which shooting is performed at a frame rate higher than the normal frame rate, and is a shooting mode for shooting a slow moving image that is reproduced slower than real time.

The moving image simultaneous shooting mode is a shooting mode in which moving image data representing a moving image captured in a predetermined period is recorded in conjunction with shooting of a still image. For example, in the moving image simultaneous shooting mode, the moving image is captured so that the moving image for a predetermined time up to the current timing is stored in the RAM 112, and the moving image stored in the RAM 112 is stored in the still image at the timing of shooting the still image. This is a shooting mode for recording in the storage unit 107. By using the moving image simultaneous shooting mode, it is possible to check not only the still image but also the moving image for a predetermined time until the timing when the still image is taken. It should be noted that the moving image from the timing at which the still image was taken may be confirmed, or the moving image during the period straddling the timing at which the still image was taken may be confirmed.

The output IF 110 outputs the moving image data (moving image data after the dynamic metadata is added) stored in the storage unit 107 to an external device (not shown). For example, the moving image data is output in a stream format according to a communication protocol conforming to the HDMI (registered trademark) (High-Definition Multimedia Interface) standard. The method of transmitting moving image data and dynamic metadata is not particularly limited. For example, the parameters specified in SMPTE (Society of Motion Picture & Television Engineers) ST 2094 may be transmitted as dynamic metadata. Specifically, Scene-MaxCLL (Maximum Content Light Level) defined by HDR10 + may be transmitted as dynamic metadata.

The CPU 111 is connected to another functional block of the image pickup apparatus 100 via an internal bus (not shown). The CPU 111 controls the processing of the image pickup apparatus 100. The RAM 112 is connected to another functional block of the image pickup apparatus 100 via an internal bus (not shown). The RAM 112 is used as a work area of the CPU 111, a temporary storage area for temporarily storing various data, and the like. The ROM 113 is connected to another functional block of the image pickup apparatus 100 via an internal bus (not shown). In the ROM 113, the firmware related to the processing of the CPU 111, the information related to the processing of the CPU 111, and the like are recorded in advance.

The operation unit 114 is connected to the CPU 111 via an internal bus (not shown). The operation unit 114 is various operation members as an input unit that accepts user operations. The operation unit 114 includes a shooting start button for starting shooting, a changeover switch for switching between automatic control / manual control of focus operation, a focus ring for performing focus adjustment operation, and the like. Further, the operation unit 114 has a touch panel and a liquid crystal panel (not shown), and the displayed function icons act as various function buttons. The function buttons include a shooting start button, a white balance setting button, an ISO sensitivity setting button, and the like. The function buttons also include a shooting mode selection button for selecting and setting a shooting mode, a playback start button for starting playback, and the like. The function button also includes a setting button for specifying and setting the shooting frame rate (shooting frame rate). The function button also includes a setting button for specifying and setting the playback frame rate (playback frame rate).

FIG. 2 is a flowchart showing an example of the photographing process of the image pickup apparatus 100. When the CPU 111 detects a shooting start operation with respect to the operation unit 114 while the special shooting mode is set, the shooting process of FIG. 2 starts. The photographing process of FIG. 2 is realized by the CPU 111 controlling each functional block of the image pickup apparatus 100. The shooting start operation is, for example, a user operation of pressing the shooting start button of the operation unit 114 in the non-shooting state of the imaging device 100. The special shooting mode is a general term for a time-lapse shooting mode, a continuous shooting mode, a slow shooting mode, a moving image simultaneous shooting mode, and the like. In the following, it is assumed that the change of the shooting mode is not permitted during the shooting process.

First, a case of shooting in the time-lapse shooting mode will be described. FIG. 3 is a schematic diagram showing an example of the relationship between the shooting frame rate and the playback frame rate in the time-lapse shooting mode. The shooting frame rate and the playback frame rate are set according to the user's operation on the operation unit 114. In this embodiment, it is assumed that the shooting frame rate is 0.05 fps, the reproduction frame rate is 60 fps, and the shooting time is 20 hours. As shown in FIG. 3, the reproduction time is defined as 1 minute (60 seconds = 72000 seconds × 0.05 fps ÷ 60 fps) from these conditions.

Before performing the shooting start operation, the user sets the operation unit 114 for setting the time-lapse shooting mode, setting operation for setting the shooting frame rate, and setting the playback frame rate. Perform the setting operation. Then, the settings are made according to these setting operations. For example, when the user performs a setting operation for specifying 0.05 fps as the shooting frame rate for the operation unit 114, the CPU 111 sets 0.05 fps as the image pickup frame rate representing the image pickup cycle in which the image sensor 102 performs image pickup. To do. Further, the CPU 111 also sets 0.05 fps as a recording frame rate indicating a writing cycle in which the metadata addition unit 106 writes frame image data to the storage unit 107. The imaging frame rate may be different from the recording frame rate. For example, a normal frame rate such as 60 fps may be set as the imaging frame rate, and 0.05 fps may be set as the recording frame rate. In that case, the captured moving image is recorded in the storage unit 107 by thinning out the frames (the shooting frame rate is 0.05 fps).

In the photographing process of FIG. 2, first, the feature amount acquisition unit 104 starts acquiring the frame maximum luminance of the frame image data output from the image sensor 102 (step S201).

Next, the CPU 111 determines whether or not the shooting mode set in the image pickup apparatus 100 is the time-lapse shooting mode (step S202). If the shooting mode is the time-lapse shooting mode (step S202: Yes), the process proceeds to step S203, and if the shooting mode is not the time-lapse shooting mode (step S202: No), the process proceeds to step S206. .. Here, since it is assumed that the time-lapse shooting mode is set (step S202: Yes), the process proceeds to step S203.

In step S203, the metadata generation unit 105 generates metadata indicating the frame maximum brightness of the current frame image data as the metadata of the frame image data. Next, the metadata addition unit 106 uses the metadata output from the metadata generation unit 105 (metadata generated in step S203) as the frame image data (current frame image data) output from the image pickup element 102. (Step S204). Then, the metadata addition unit 106 records the frame image data after the metadata is added in the storage unit 107. The processes of steps S203 and S204 are repeated during imaging. Therefore, in the time-lapse shooting mode, dynamic metadata indicating the feature amount (frame maximum brightness) of each frame of the captured moving image (moving image to be captured) is generated, and the captured moving image (captured) is generated. It is added to the moving image data representing the moving image). As described above, the metadata addition unit 106 appropriately performs the frame image data coding process.

Then, the CPU 111 determines whether or not the shooting end operation for the operation unit 114 has been performed (step S205). If the shooting end operation has not been performed (step S205: No), the process is returned to step S203. The processes of steps S203 to S205 are repeated at the shooting frame rate until the shooting end operation is performed. That is, the processes of steps S203 to S205 are performed for each frame of the captured moving image (moving image to be captured). When the shooting end operation is performed (step S205: Yes), the shooting process of FIG. 2 ends. Here, it is said that the processes of steps S203 to S205 are repeated from the start of shooting (start of the shooting process of FIG. 2) until 20 hours have elapsed, and after 20 hours have passed, the shooting end operation is performed and the shooting process is completed. To do.

FIG. 4 is a schematic diagram showing an example of frame image data, frame maximum brightness, and dynamic metadata of a moving image shot in the time-lapse shooting mode. Since the shooting time is 20 hours and the shooting frame rate is 0.05 fps, 3600 frames (= 0.05 fps × 72000 seconds) of frame images are shot (frame image data no. 1 to no. 3600). The solid line 401 shows the time change of the maximum frame brightness. As shown in FIG. 4, when a moving image is shot in the time-lapse shooting mode, the frame image data no. 1-no. Metadata indicating the corresponding maximum frame brightness is added to each of the 3600s. For example, frame image data no. 1-no. Metadata indicating 100 nits is added to 3. Then, the frame image data no. Metadata indicating 3000 nits is added to M, and the frame image data no. Metadata indicating 1000 nits is added to 3600. That is, dynamic metadata indicating the feature amount (frame maximum brightness) of each frame of the captured moving image (captured moving image) is generated and added to the moving image data representing the captured moving image (captured moving image). To.

Although an example of setting the shooting frame rate and the playback frame rate has been described, other parameters related to the frame rate (shooting frame rate, playback frame rate, etc.) may be set. For example, the shooting time (seconds) and the number of frames to be shot may be set. In that case, the shooting frame rate can be obtained by dividing the number of frames to be shot by the shooting time.

When the continuous shooting mode is set, specifically, when continuous shooting (processing of sequentially shooting a plurality of still images) and then continuous playback (processing of sequentially playing back a plurality of still images) is performed, a time lapse is performed. The same shooting process as when the shooting mode is set may be applied. Specifically, the metadata generation unit 105 may perform a process of generating metadata indicating the feature amount of the captured still image for each of the plurality of still images to be sequentially photographed. Then, the metadata addition unit 106 adds the metadata generated by the metadata generation unit 105 to the still image data representing the still image corresponding to the metadata, respectively, for each of the plurality of still images to be sequentially captured. You may go about.

In this case, for example, the still image data is recorded in the EXIF (Exif image file format) format. Then, the feature amount (maximum brightness) of the still image is described in the metadata defined by EXIF. These are the processes at the time of shooting. After that, when the reproduction unit 108 sequentially reads out a plurality of still image data (a plurality of still images) from the storage unit 107, the reproduction unit 108 extracts a feature amount (maximum brightness) from the metadata defined by EXIF and still image data. At the same time, the feature amount is output to the output IF110. The output IF 110 outputs the still image data output from the reproduction unit 108 to an external device in a stream format conforming to the HDMI standard. That is, the still image data is output as a part of the moving image data (frame image data). At this time, the metadata (feature amount; maximum brightness) output from the reproduction unit 108 is also SMPTE.
It is output to an external device as a part of the dynamic metadata specified in ST 2094 (maximum frame brightness). These are the processes at the time of reproduction.

According to the above processing, even in the case of continuous reproduction after continuous shooting, it is possible to generate dynamic metadata indicating the feature amount (frame maximum brightness) of each frame of the moving image and add it to the moving image data.

Next, a case of shooting in the slow shooting mode will be described. FIG. 5 is a schematic diagram showing an example of the relationship between the shooting frame rate and the playback frame rate in the slow shooting mode. Similar to the case of shooting in the time-lapse shooting mode, the shooting frame rate and the playback frame rate are set according to the user's operation on the operation unit 114. In this embodiment, it is assumed that the shooting frame rate is 240 fps, the reproduction frame rate is 60 fps, and the shooting time is 1 second. As shown in FIG. 5, the reproduction time is determined to be 4 seconds (= 1 second × 240 fps ÷ 60 fps) from these conditions.

In the shooting process of FIG. 2, first, the processes of steps S201 and S202 are performed in the same manner as in the case of shooting in the time-lapse shooting mode. That is, in step S201, the feature amount acquisition unit 104 starts acquiring the frame maximum luminance of the frame image data output from the image sensor 102. Then, in step S202, the CPU 111 determines whether or not the shooting mode set in the image pickup apparatus 100 is the time-lapse shooting mode. Here, since it is assumed that the time-lapse shooting mode is not set and the slow shooting mode is set (step S202: No), the process proceeds to step S206.

In step S206, the CPU 111 determines whether or not the maximum frame brightness (information) of the current frame image data is higher than the maximum frame brightness (information) stored in the RAM 112. Then, when the CPU 111 determines that the value is high, the CPU 111 updates the maximum frame brightness stored in the RAM 112 with the maximum frame brightness of the current frame image data. When the current frame image data corresponds to the first frame of the captured moving image (the moving image to be captured), the CPU 111 records the frame maximum brightness of the current frame image data in the RAM 112.

Then, the CPU 111 determines whether or not the shooting end operation for the operation unit 114 has been performed (step S207). If the shooting end operation has not been performed (step S207: No), the process is returned to step S206. The processes of steps S206 and S207 are repeated at the shooting frame rate until the shooting end operation is performed. That is, the processes of steps S206 and S207 are performed for each frame of the captured moving image (moving image to be captured). When the shooting end operation is performed (step S207: Yes), the process proceeds to step S208. Therefore, the entire feature amount (maximum brightness) of the captured moving image (shooting moving image) is stored in the RAM 112, that is, the maximum scene brightness with the entire period of the moving image as one scene is stored in the RAM 112. Then, the process proceeds to step S208. Here, the processes of steps S206 and S207 are repeated until 1 second elapses from the start of shooting (start of the shooting process of FIG. 2), and after 1 second elapses, the shooting end operation is performed and the process proceeds to step S208. Suppose it was done.

In step S208, the metadata generation unit 105 dynamically meta-generates the metadata indicating the maximum frame brightness (the maximum brightness of the entire moving image; the maximum brightness of the scene with the entire period of the moving image as one scene) stored in the RAM 112. Generate as data.

Then, the metadata addition unit 106 adds the dynamic metadata (dynamic metadata generated in step S208) output from the metadata generation unit 105 to the moving image data output from the image sensor 102 (step). S209). Then, the metadata addition unit 106 records the moving image data after the dynamic metadata is added in the storage unit 107. After that, the shooting process ends. As described above, the metadata addition unit 106 appropriately performs the coding process of the moving image data.

FIG. 6 is a schematic diagram showing an example of frame image data, frame maximum brightness, and dynamic metadata of a moving image shot in the slow shooting mode. Since the shooting time is 1 second and the shooting frame rate is 240 fps, 240 frames (= 240 fps × 1 second) of frame images are shot (frame image data no. 1 to no. 240). The solid line 601 indicates the time change of the maximum frame brightness. As shown in FIG. 6, when a moving image is shot in the slow shooting mode, all the unique dynamic metadata indicating the maximum brightness of the plurality of frame maximum brightness corresponding to the plurality of frame image data is all. It is added to the moving image data so as to correspond to the frame image data. Specifically, the frame image data no. The dynamic metadata showing the maximum frame brightness of N of 3000 nits is the frame image data no. 1-No. It is added to the moving image data as dynamic metadata corresponding to all 240. That is, dynamic metadata indicating the overall feature amount (maximum brightness) of the captured moving image (captured moving image) is generated and added to the moving image data representing the captured moving image (captured moving image).

When the moving image simultaneous shooting mode is set, the same shooting process as when the slow shooting mode is set may be applied. However, in the case of the moving image simultaneous shooting mode, in step S206, the CPU 111 performs the moving image data for a predetermined time up to the current timing (
The current frame image data is recorded in the RAM 112 so that the plurality of frame image data) are stored in the RAM 112. Further, the CPU 111 records the frame maximum luminance corresponding to the current frame image data in the RAM 112 so that the plurality of frame maximum luminance corresponding to each of the plurality of frame image data is stored in the RAM 112. In step S207, the CPU 111 determines whether or not the still image has been photographed. Shooting of a still image is performed, for example, in response to pressing the still image shooting button. Then, in step S208, the metadata generation unit 105 generates metadata indicating the maximum brightness among the plurality of frame maximum brightness stored in the RAM 112 as dynamic metadata.

FIG. 7 is a schematic diagram showing an example of a tone curve (tone map) when the upper limit of the display brightness (brightness of the display surface) of the display device according to the present embodiment is 1000 nits. The tone curve represents the relationship between the brightness (data brightness) represented by the image data input to the display device and the display brightness of the display device. The display device displays the image according to the tone curve. In this embodiment, the frame image data output from the image pickup apparatus 100 (output IF110) is input to the display apparatus. The display device changes the tone curve according to the dynamic metadata output from the image pickup device 100.

When the tone curve 701 is used, the data brightness of 1000 nits is displayed at the display brightness of 1000 nits, but the data brightness higher than 1000 nits is also displayed at the display brightness of 1000 nits. That is, although the data luminance of 1000 nits or less can be faithfully displayed, the data luminance higher than 1000 nits cannot be displayed with high gradation, and the gradation crushing (overexposure) occurs in the area of the data luminance higher than 1000 nits. When the tone curve 702 is used, there is a decrease in brightness such that the data brightness of 1000 nits is displayed at the display brightness of 800 nits, but the data brightness higher than 1000 nits can be displayed with high gradation.

As described above, when the time-lapse shooting mode is set, dynamic metadata indicating the frame maximum brightness of each frame of the moving image is generated. Therefore, in the display device, the tone curve is changed for each frame. For example, the frame image data no. When displaying M (maximum frame brightness 3000 nits), the tone curve 702 is used. Then, the frame image data no. When displaying 3600 (maximum frame brightness 1000 nits), the tone curve 701 is used. By changing the tone curve to be used from the tone curve 702 to the tone curve 701, it is possible to suppress the above-mentioned decrease in brightness (particularly, change from data brightness of 100 nits or more, decrease from data brightness in the vicinity of 1000 nits, etc.). it can. Further, by changing the tone curve to be used from the tone curve 701 to the tone curve 702, it becomes possible to display data brightness higher than 1000 bits with high gradation.

On the other hand, when the slow shooting mode is set, dynamic metadata indicating the maximum brightness of the scene is generated with the entire period of the moving image as one scene. Therefore, the tone curve is not changed on the display device. In the example of FIG. 6, the tone curve 702 is always used in the display device because the dynamic metadata showing the maximum scene brightness of 3000 nits is generated. As a result, it is possible to suppress the change in brightness caused by the change in the tone curve. In the moving image shot in the slow shooting mode, the difference between the images is relatively small between the frames. By suppressing the change in brightness caused by the change in the tone curve, it is possible to suppress a sudden change in the appearance of the image between frames in which the difference between the images is relatively small.

As described above, according to the present embodiment, in the imaging device, dynamic metadata is generated so as to indicate the feature amount of the moving image in the period corresponding to the set shooting mode, and the generated dynamic metadata is generated. Data is added to the video data. As a result, it is possible to obtain moving image data to which suitable dynamic metadata is added in a shooting process (imaging apparatus) instead of post-production (image data editing process). For example, by using the dynamic metadata added in this embodiment, it is possible to display a time-lapse moving image while suppressing a decrease in brightness in a high-luminance area having a brightness close to the maximum frame brightness. It is also possible to suppress a sudden change in brightness and display a slow moving image.

The shooting mode is not limited to the time-lapse shooting mode, the continuous shooting mode, the slow shooting mode, and the moving image simultaneous shooting mode. For example, a shooting mode in which the shooting frame rate can be changed during shooting may be set. When such a shooting mode is set, the metadata generation unit 105 may generate dynamic metadata in which the period corresponding to the feature amount is switched at the timing when the shooting frame rate is changed. That is, the metadata generation unit 105 may generate dynamic metadata indicating the maximum brightness of each scene, with the timing at which the shooting frame rate is changed as the timing at which the scene is switched.

Further, for a period in which the shooting frame rate is lower than a predetermined frame rate, the metadata generation unit 105 dynamically metadata so that the feature amount (frame maximum brightness) of each frame of the moving image is shown instead of the scene maximum brightness. May be generated. The predetermined frame rate is a normal frame rate such as 60 fps.

Further, the metadata generation unit 105 may generate dynamic metadata so that the period corresponding to the feature amount is switched at the timing when the moving image scene is switched in the period when the shooting frame rate is higher than the predetermined frame rate. That is, the metadata generation unit 105 may generate dynamic metadata so that the maximum brightness of each scene is shown for a period in which the shooting frame rate is higher than a predetermined frame rate. The timing at which the scene is switched may include a timing different from the timing at which the shooting frame rate is changed. For example, the timing at which the maximum frame brightness changes by a threshold value or more between frames may be further used as the timing at which the scene is switched.

Alternatively, the metadata generation unit 105 may generate dynamic metadata so that the period corresponding to the feature amount is switched at the predetermined frame rate during the period when the shooting frame rate is higher than the predetermined frame rate. That is, the metadata generation unit 105 may generate dynamic metadata indicating the maximum brightness of each scene as the timing at which the shooting frame rate is changed and the timing at which the predetermined frame rate is switched.

<Example 2>
Hereinafter, Example 2 of the present invention will be described. In the first embodiment, an example in which the reproduction time (reproduction frame rate) is predetermined has been described. In the second embodiment, an example in which the reproduction time can be changed at the time of reproduction will be described. In the following, points different from Example 1 (configuration, processing, etc.) will be described in detail, and description of the same points as in Example 1 will be omitted.

The shooting mode setting unit 109 can set a variable frame rate shooting mode (shooting mode in which the shooting frame rate can be changed). When the variable frame rate mode is set, switching from normal shooting to special shooting, switching from special shooting to other special shooting, and special shooting to normal shooting according to the user's operation on the operation unit 114. Switching to, etc. is performed. The special shooting is the same shooting as the shooting in the special shooting mode, for example, time-lapse shooting which is shooting in the time-lapse shooting mode, slow shooting which is shooting in the slow shooting mode, and the like.

In addition to the processing described in the first embodiment, the reproduction unit 108 can also perform a scene determination process for determining a moving image scene, a regeneration process for regenerating (updating) dynamic metadata, and the like. In the scene determination process, for example, the timing at which the maximum frame brightness changes by a threshold value or more between frames is detected as the timing at which the scene is switched. Further, the reproduction unit 108 can set the reproduction mode according to the user operation on the operation unit 114. In this embodiment, the reproduction unit 108 sets a special effect priority mode or a time priority mode when reproducing a moving image shot in the variable frame rate shooting mode. The special effect priority mode and the time priority mode will be described later.

The dynamic metadata regeneration process according to the reproduction mode of the image pickup apparatus 100 will be described. Here, it is assumed that the moving image data shot in the variable frame rate shooting mode is reproduced, and dynamic metadata indicating the maximum frame brightness of each frame is added to the moving image data to be reproduced.

First, the case where the special effect priority mode is set will be described. The special effect priority mode is a playback mode in which playback is performed with priority given to the effect of special shooting. When the special effect priority mode is set, the moving image shot in the variable frame rate shooting mode is played back at a playback time different from the shooting time of the moving image. For example, during the slow shooting period, the moving image is played back with a playback time longer than the shooting time (slow playback), and during the time-lapse shooting period, the moving image is played back with a playback time shorter than the shooting time (time-lapse playback).

FIG. 8 is a schematic diagram showing an example of the relationship between the shooting frame rate, the reproduction frame rate, and the reproduction state when the special effect priority mode is set.

In FIG. 8, the moving image shot in the variable frame rate shooting mode includes a normal shooting moving image 801, a slow shooting moving image 802, a time-lapse shooting moving image 803, and a normal shooting moving image 804. The normal shooting moving images 801 and 804 are moving images obtained by normal shooting, and the shooting frame rate of the normal shooting moving images 801 and 804 is 60 fps. The slow shooting moving image 802 is a moving image obtained by slow shooting, and the shooting frame rate of the slow shooting moving image 802 is 240 fps. The time-lapse shooting moving image 803 is a moving image obtained by time-lapse shooting, and the shooting frame rate of the time-lapse shooting moving image 803 is 30 fps.

When the special effect priority mode is set, the normal shooting moving image 801 is normally played back, the slow shooting moving image 802 is played back slowly, the time-lapse shooting moving image 803 is played back in time-lapse, and the normal shooting moving image 804 is normally played back. In the example of FIG. 8, the reproduced moving image (reproduced moving image) includes a normal reproduction moving image 811, a slow reproduction moving image 812, a time-lapse reproduction moving image 813, and a normal reproduction moving image 814. The normal playback video 811 corresponds to the normal shooting video 801, the slow playback video 812 corresponds to the slow shooting video 802, the time-lapse playback video 813 corresponds to the time-lapse shooting video 803, and the normal playback video 814 corresponds to the normal shooting video 804. To do. The playback frame rate is always 60 fps.

FIG. 9 is a schematic diagram showing an example of dynamic metadata regenerated when the special effect priority mode is set. Here, the difference between the maximum frame brightness La and the maximum frame brightness Lb, the difference between the maximum frame brightness Ld and the maximum frame brightness Le, and the difference between the maximum frame brightness Lg and the maximum frame brightness Lh are for detecting the change of the scene. It is equal to or higher than the above threshold. The case where the special effect priority mode is set is a case where the moving image shot in the variable frame rate shooting mode is played back at a playback time different from the shooting time of the moving image.

When the special effect priority mode is set, the reproduction unit 108 regenerates the dynamic metadata so as to satisfy the following conditions. Here, the timing at which the moving image scene is switched is not the timing at which the shooting frame rate is changed, but the timing at which, for example, the maximum frame brightness changes by the threshold value or more. The timing at which the shooting frame rate is changed is the timing at which the playback state is switched.
-During a period in which the shooting frame rate is a predetermined frame rate (for example, a normal frame rate such as 60 fps), a period corresponding to the feature amount at the timing when the moving image scene is switched and the timing when the shooting frame rate is changed. Is switched.
-In a period when the shooting frame rate is not a predetermined frame rate, the period corresponding to the feature amount is switched at the timing when the shooting frame rate is changed.

In this embodiment, for the normal playback period, the dynamic meta is shown so that the maximum scene brightness is shown as the timing at which the scene is switched between the timing when the shooting frame rate is changed and the timing when the maximum frame brightness changes by the threshold value or more. The data is regenerated. Specifically, in the period of the normal reproduction moving image 811, the period from the time 0 when the reproduction is started to the time t1 when the maximum frame brightness changes from the brightness La to the brightness Lb is detected as the period of one scene. Then, as a part of the dynamically generated dynamic metadata, specifically, the metadata corresponding to the detected scene (the period from time 0 to time t1), the metadata indicating the scene maximum brightness La of the scene is shown. Is regenerated. Similarly, the period from the time t1 to the time t2 when the playback state changes is detected as one scene, and the metadata indicating the scene maximum brightness Lc of the scene is regenerated as the metadata corresponding to the detected scene. Will be done. As for the period of the normal reproduction moving image 814, the period from the time t6 when the reproduction state changes to the time t7 when the reproduction ends is detected as the period of one scene. Then, as the metadata corresponding to the detected scene (the period from the time t6 to the time t7), the metadata indicating the scene maximum brightness Li of the scene is regenerated.

Then, for the special playback (slow playback, time-lapse playback, etc.) period, dynamic metadata is generated so that the overall feature amount of the moving image in the period is shown. That is, for the special reproduction period, the dynamic metadata is regenerated so that the maximum brightness of the scene with the period as one scene is shown.

Specifically, as for the period of the slow reproduction moving image 812, the period from the time t2 when the slow reproduction is started to the time t4 when the reproduction state changes is detected as one scene. Then, as a part of the dynamically generated dynamic metadata, specifically, the metadata corresponding to the detected scene (the period from the time t2 to the time t4), the metadata indicating the scene maximum brightness Lf of the scene. Is regenerated. In the case of normal playback, the metadata is switched at time t3 when the maximum frame brightness changes from brightness Ld to brightness Le, but in the case of slow playback, the metadata is not switched at time t3.

Similarly, with respect to the period of the time-lapse reproduction moving image 813, the period from the time t4 when the time-lapse reproduction is started to the time t6 when the reproduction state changes is detected as one scene. Then, as the metadata corresponding to the detected scene (the period from the time t4 to the time t6), the metadata indicating the scene maximum brightness Lg of the scene is regenerated. In the case of normal reproduction, the metadata is switched at time t5 when the maximum frame brightness changes from brightness Lg to brightness Lh, but in the case of time-lapse reproduction, the metadata is not switched at time t5.

By using the dynamic metadata regenerated by the above processing, the tone curve is not changed during the special reproduction, so that the (rapid) brightness change caused by the change of the tone curve can be suppressed.

Next, a case where the time priority mode is set will be described. The time priority mode is a playback mode in which playback is performed with priority given to the shooting time. When the time priority mode is set, the moving image shot in the variable frame rate shooting mode is played back with a playback time equal to the shooting time of the moving image.

FIG. 10 is a schematic diagram showing an example of the relationship between the shooting frame rate, the reproduction frame rate, and the reproduction state when the time priority mode is set. Similar to FIG. 8, the moving image shot in the variable frame rate shooting mode includes a normal shooting moving image 801, a slow shooting moving image 802, a time-lapse shooting moving image 803, and a normal shooting moving image 804.

When the time priority mode is set, the normal shooting moving image 801 is normally played back, the slow shooting moving image 802 is thinned out, the time-lapse shooting moving image 803 is duplicated, and the normal shooting moving image 804 is normally played back. In thinned-out playback, frames are thinned out so that the playback time matches the shooting time. In duplicate playback, the same frame is played multiple times so that the playback time matches the shooting time. In the example of FIG. 10, the reproduced moving image includes a normal reproduction moving image 911, a thinned-out reproduction moving image 912, a duplicate reproduction moving image 913, and a normal playing image 914. The normal playback video 911 corresponds to the normal shooting video 801, the thinned playback video 912 corresponds to the slow shooting video 802, the duplicate playback video 913 corresponds to the time-lapse shooting video 803, and the normal playback video 914 corresponds to the normal shooting video 804. To do. The playback frame rate is always 60 fps.

FIG. 11 is a schematic diagram showing an example of dynamic metadata regenerated when the time priority mode is set. Similar to FIG. 9, the difference between the maximum frame brightness La and the maximum frame brightness Lb, the difference between the maximum frame brightness Ld and the maximum frame brightness Le, and the difference between the maximum frame brightness Lg and the maximum frame brightness Lh detect a scene change. It is equal to or higher than the above threshold value. The time priority mode is set when the moving image shot in the variable frame rate shooting mode is played back at a playback time equal to the shooting time of the moving image.

When the time priority mode is set, the playback unit 108 regenerates dynamic metadata in which the period corresponding to the feature amount is switched at the timing when the moving image scene is switched, regardless of the shooting frame rate. Here, the timing at which the moving image scene is switched is not the timing at which the shooting frame rate is changed, but the timing at which, for example, the maximum frame brightness changes by the threshold value or more. In this embodiment, when the time priority mode is set, the timing at which the shooting frame rate is changed is not treated as the timing at which the scene is switched, and the timing at which the maximum frame brightness changes by the threshold value or more is the timing at which the scene is switched. Treated as. That is, the reproduced moving images 911 to 914 are treated as one reproduced moving image, and the dynamic metadata indicating the maximum brightness of the scene is regenerated.

Specifically, the period from the time 0 when the reproduction is started to the time t1 when the maximum frame brightness changes from the brightness La to the brightness Lb is detected as the period of one scene. Then, as a part of the dynamically generated dynamic metadata, specifically, the metadata corresponding to the detected scene (the period from time 0 to time t1), the metadata indicating the scene maximum brightness La of the scene is shown. Is regenerated. Similarly, the period from the time t1 to the time t3 when the frame maximum brightness changes from the brightness Ld to the brightness Le is detected as one scene, and the scene maximum brightness Ld of the scene is used as the metadata corresponding to the detected scene. The metadata indicating is regenerated. The period from time t3 to time t5 when the frame maximum brightness changes from brightness Lg to brightness Lh is detected as one scene, and as metadata corresponding to the detected scene, meta indicating the scene maximum brightness Lg of the scene. The data is regenerated. Then, the period from the time t5 to the time t7 at which the reproduction ends is detected as one scene, and the metadata indicating the scene maximum brightness Lh of the scene is regenerated as the metadata corresponding to the detected scene. .. As described above, the metadata is switched at the time t1, t3, t5 when the maximum frame brightness changes by the threshold value or more, but the metadata is not switched at the time t2, t4, t6 when the reproduction state changes.

By using the dynamic metadata regenerated by the above processing, the tone curve is not changed during the reproduction of the moving image scene, so that the change in brightness due to the change in the tone curve can be suppressed.

As described above, according to the present embodiment, the dynamic metadata is regenerated by a method according to the playback method of the moving image shot in the variable frame rate shooting mode. As a result, it is possible to realize display with suitable brightness by using dynamic metadata suitable for the reproduction method.

It should be noted that each functional block of Examples 1 and 2 (FIG. 1) may or may not be individual hardware. The functions of two or more functional blocks may be realized by common hardware. Each of the plurality of functions of one functional block may be realized by individual hardware. Two or more functions of one functional block may be realized by common hardware. Also, each functional block may or may not be implemented by hardware. For example, the device may have a processor and a memory in which the control program is stored. Then, the function of at least a part of the functional blocks of the device may be realized by the processor reading the control program from the memory and executing it.

It should be noted that Examples 1 and 2 (including the above-mentioned modified examples) are merely examples, and configurations obtained by appropriately modifying or changing the configurations of Examples 1 and 2 within the scope of the gist of the present invention. Is also included in the present invention. A configuration obtained by appropriately combining the configurations of Examples 1 and 2 is also included in the present invention.

<Other Examples>
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100: Imaging device 105: Metadata generation unit 106: Metadata addition unit 109: Shooting mode setting unit

Claims (14)

Setting means to set the shooting mode and
A generation means for generating metadata indicating the features of the captured moving image, and
An additional means for adding the metadata generated by the generation means to the moving image data representing the moving image, and
Have,
The generation means is an image pickup apparatus that generates the metadata indicating the feature amount of the moving image in a period corresponding to the shooting mode set by the setting means. The imaging device according to claim 1, wherein the feature amount has a maximum brightness. When the shooting mode set by the setting means is a shooting mode in which the playback time of the moving image is shorter than the shooting time of the moving image, the generating means is a meta indicating the feature amount of each frame of the moving image. The imaging apparatus according to claim 1 or 2, wherein data is generated. When the shooting mode set by the setting means is a shooting mode for sequentially shooting a plurality of still images,
The generation means performs a process of generating metadata indicating the feature amount of the captured still image for each of the plurality of still images to be sequentially captured.
The addition means performs a process of adding the metadata generated by the generation means to the still image data representing the still image corresponding to the metadata for each of the plurality of still images to be sequentially photographed. The image pickup apparatus according to any one of claims 1 to 3. When the shooting mode set by the setting means is a shooting mode in which the playback time of the moving image is longer than the shooting time of the moving image, the generating means is metadata indicating the overall feature amount of the moving image. The imaging device according to any one of claims 1 to 4, wherein the image pickup apparatus is produced. When the shooting mode set by the setting means is a shooting mode for recording moving image data representing a moving image captured in a predetermined period in conjunction with shooting of a still image, the generating means is used for the predetermined period. The imaging apparatus according to any one of claims 1 to 5, wherein metadata indicating the overall feature amount of the moving image captured in the image is generated. When the shooting mode set by the setting means is a shooting mode in which the frame rate of shooting can be changed, the generation means corresponds to the feature amount at the timing when the frame rate of shooting is changed. The imaging apparatus according to any one of claims 1 to 6, wherein the metadata for which the period is switched is generated. When the shooting mode set by the setting means is the shooting mode in which the frame rate of the shooting can be changed, the generation means said that the frame rate of the shooting is lower than the predetermined frame rate for a period of time. The imaging device according to claim 7, wherein the metadata indicating the feature amount of each frame of the moving image is generated. When the shooting mode set by the setting means is the shooting mode in which the frame rate of the shooting can be changed, the generation means said that the frame rate of the shooting is higher than the predetermined frame rate. The imaging device according to claim 7 or 8, wherein the metadata corresponding to the feature amount is switched at the timing when the moving image scene is switched. When the shooting mode set by the setting means is the shooting mode in which the frame rate of the shooting can be changed, the generation means said that the frame rate of the shooting is higher than the predetermined frame rate. The imaging apparatus according to claim 7 or 8, wherein the metadata corresponding to the feature amount is switched at a predetermined frame rate. When a moving image shot in a shooting mode in which the shooting frame rate can be changed is played back at a playback time equal to the shooting time of the moving image, the generation means uses the feature amount at the timing when the scene of the moving image is switched. The imaging apparatus according to any one of claims 1 to 10, wherein the corresponding metadata for which the period is switched is regenerated. When a moving image shot in a shooting mode in which the shooting frame rate can be changed is played back at a playback time different from the shooting time of the moving image, the generation means
In a period in which the frame rate of the shooting is a predetermined frame rate, the period corresponding to the feature amount is switched between the timing when the scene of the moving image is switched and the timing when the frame rate of the shooting is changed.
Claim 1 is characterized in that, in a period in which the frame rate of shooting is not the predetermined frame rate, metadata corresponding to the feature amount is regenerated at the timing when the frame rate of shooting is changed. The imaging apparatus according to any one of 1 to 11. Setting steps to set the shooting mode and
A generation step to generate metadata showing the features of the captured video,
An additional step of adding the metadata generated in the generation step to the moving image data representing the moving image,
Have,
The generation step is a control method for an image pickup apparatus, which comprises generating the metadata indicating the feature amount of the moving image in a period corresponding to the shooting mode set in the setting step. A program for causing a computer to function as each means of the imaging device according to any one of claims 1 to 12.

Images