JP2020077964A - Imaging apparatus and control method thereof - Google Patents

Abstract

To provide a technology that can acquire dynamic metadata (feature amount of each scene) without going through post production.SOLUTION: An imaging apparatus of the present invention includes image capturing means that captures a moving image, determination means that determines a plurality of scenes of a moving image on the basis of a temporal change of a parameter corresponding to a frame of the moving image, acquisition means that acquires the respective feature quantities of the plurality of scenes, and generating means that generates information in which the feature quantity acquired by the acquisition means is associated with each of the plurality of scenes.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image pickup device and a control method thereof.

  2. Description of the Related Art In recent years, a technique for sequentially changing a display method of a moving image (for each frame or each scene) using metadata (dynamic metadata) including maximum brightness of each frame of each moving image or each scene has become popular. Specifically, a technique for sequentially changing the HDR (high dynamic range) display according to the upper limit display brightness by sequentially changing the tone map using the dynamic metadata is becoming widespread. Patent Document 1 discloses a method of generating metadata (a part of dynamic metadata) of each scene of a moving image and adding the metadata to the moving image data in a post-production image data editing process.

International Publication No. 2015/017314

  However, in the technique disclosed in Patent Document 1, dynamic metadata cannot be obtained without post-production (image data editing process), and moving image display methods (such as HDR display using tone mapping) are sequentially performed. Difficult to change.

  An object of the present invention is to provide a technique capable of acquiring dynamic metadata and the like (feature amount of each scene) without going through post production.

  According to a first aspect of the present invention, an image capturing unit that captures a moving image, a determining unit that determines a plurality of scenes of the moving image based on a temporal change of a parameter corresponding to a frame of the moving image, and each of the plurality of scenes. The image pickup apparatus is characterized by comprising: an acquisition unit configured to acquire the feature amount of 1. and a generation unit configured to generate information in which the feature amount acquired by the acquisition unit is associated with each of the plurality of scenes.

  A second aspect of the present invention includes an imaging step of capturing a moving image, a determining step of determining a plurality of scenes of the moving image based on a temporal change of a parameter corresponding to a frame of the moving image, and each of the plurality of scenes. And a generating step of generating information in which the characteristic amount acquired in the acquiring step is associated with each of the plurality of scenes. is there.

  A third aspect of the present invention is a program for causing a computer to function as each unit of the above-described imaging device.

  According to the present invention, dynamic metadata and the like (feature amount of each scene) can be generated without going through post production.

1 is a block diagram showing a configuration example of an image pickup apparatus according to a first embodiment. FIG. 6 is a diagram showing an example of a temporal change of the maximum frame luminance value according to the first embodiment. FIG. 3 is a diagram showing an example of a temporal change of various parameters during shooting according to the first embodiment. Flowchart showing an example of a photographing process according to the first embodiment FIG. 8 is a diagram showing an example of temporal changes in frame maximum brightness value and aperture value according to the second embodiment. FIG. 8 is a diagram showing an example of temporal changes in frame maximum brightness value and aperture value according to the second embodiment. Block diagram showing a configuration example of an imaging apparatus according to a third embodiment FIG. 8 is a diagram showing an example of a frame image according to the third embodiment.

<Example 1>
Hereinafter, Example 1 of the present invention will be described. FIG. 1 is a block diagram showing a configuration example of the image pickup apparatus 100 according to the present embodiment. The image pickup apparatus 100 includes an image pickup optical system 101, an image pickup element 102, an image pickup control unit 103, a feature amount acquisition unit 104, a scene determination unit 105, a metadata generation unit 106, a metadata addition unit 107, an output unit 108, a storage unit 109, and It has an output IF 110, a CPU 111, a RAM 112, a ROM 113, and an operation unit 114.

  The imaging optical system 101 forms (forms) an optical image representing a subject on the imaging element 102. The imaging optical system 101 has, for example, a lens group such as a zoom lens and a focus lens, an aperture adjustment device, and a shutter device.

  The image sensor 102 captures a subject image (moving image representing the subject). Specifically, the image sensor 102 performs a photoelectric conversion process for converting the formed optical image (light incident from the subject through the imaging optical system 101) into an analog electric signal. Then, the image sensor 102 further performs AD conversion processing (analog-digital conversion processing) for converting an analog electric signal obtained by the photoelectric conversion processing into a digital electric signal (image data of one frame of a moving image; frame image data). .. After that, the image pickup device 102 outputs the frame image data obtained by the AD conversion processing to the feature amount acquisition unit 104 and the scene determination unit 105. The image sensor 102 repeats these processes to sequentially output a plurality of frame image data corresponding to a plurality of frames of a moving image.

  The imaging control unit 103 controls the imaging conditions of the imaging device 100. In the present embodiment, the imaging control unit 103 controls the exposure (exposure condition) of the imaging device 100 according to a user operation on the imaging device 100, the state of the imaging device 100, and the like. For example, the imaging control unit 103 controls the exposure by controlling the aperture and the charge storage time of the image sensor. Specifically, the image pickup control unit 103 controls the state of the image pickup optical system 101, the processing of the image pickup element 102, and the like according to a gain value, a shutter speed, an aperture value, and the like designated by the user, thereby performing exposure. To control. Furthermore, the image capturing control unit 103 controls the focus of the image capturing apparatus 100 according to a user operation on the image capturing apparatus 100, the state of the image capturing apparatus 100, and the like. For example, the imaging control unit 103 controls focus by controlling the drive amount and drive direction of the focus lens. Specifically, when the AF (autofocus) shooting mode is set, the imaging control unit 103 sets the position of the focus lens to a predetermined position and sets the shape of the contrast of the frame image (the image of one frame of the moving image). Calculate (distribution). Then, the imaging control unit 103 performs AF control with the position having the highest contrast among the plurality of positions in the frame image as the position (focus position) at which the light flux is focused by the image sensor 102.

The feature amount acquisition unit 104 acquires the feature amount (frame feature amount) of the frame image data output from the image sensor 102. Specifically, the feature amount acquisition unit 104 includes, as the frame feature amount, a feature amount for determining a plurality of scenes of a moving image and a feature amount for obtaining the feature amount of each scene (scene feature amount). get. It can be said that the frame feature amount is also a “parameter corresponding to the frame”. In the present embodiment, the feature amount acquisition unit 104 acquires the maximum brightness value of the frame image data (frame maximum brightness value) from the frame image data, and outputs the frame maximum brightness value to the scene determination unit 105. The frame maximum brightness value is used as both a feature amount for determining a plurality of scenes and a feature amount for acquiring the scene feature amount.

  The scene determination unit 105 performs a scene determination process that determines a plurality of scenes of a moving image. It can be said that the scene determination processing is “scene division processing for dividing the entire period of the moving image into a plurality of periods”. The frame feature amount often changes (significantly) when the scene is switched. Therefore, in the present embodiment, the scene determination unit 105 determines a plurality of scenes based on the temporal change in the frame feature amount (frame maximum luminance value) output from the feature amount acquisition unit 104. It can be said that the temporal change of the frame feature amount is “change of the frame feature amount between temporally consecutive frames”. The scene determination unit 105 outputs the result of the scene determination processing to the metadata generation unit 106. Furthermore, the scene determination unit 105 outputs the frame feature amount (frame maximum brightness value) output from the feature amount acquisition unit 104 to the metadata generation unit 106, and adds the frame image data output from the image sensor 102 to the metadata. It is output to the unit 107. The scene determination unit 105 may perform various image processing on the frame image data output from the image sensor 102 and output the frame image data after the image processing to the metadata addition unit 107. As the image processing, for example, correction processing for reducing distortion and noise caused by the image pickup optical system 101 and the image pickup device 102 may be performed, and white balance adjustment, color conversion processing, gamma correction, and the like may be performed. ..

  The metadata generation unit 106 generates information (dynamic metadata) to be added to moving image data (moving image data), and outputs the dynamic metadata to the metadata addition unit 107. Specifically, the metadata generation unit 106, based on the information output from the scene determination unit 105 (the result of the scene determination processing, and the frame feature amount (frame maximum brightness value) of each frame of the moving image). The scene feature amount of each scene is acquired (determined). Then, the metadata generation unit 106 generates, as dynamic metadata, information in which the acquired scene feature amount is associated with each scene.

  The metadata addition unit 107 generates moving image data composed of a plurality of frame image data sequentially output from the scene determination unit 105 (imaging device 102), and uses the dynamic metadata output from the metadata generation unit 106 as moving image data. Added to. Then, the metadata adding unit 107 outputs the moving image data to which the dynamic metadata has been added to the output unit 108. For example, the moving image data is data in a file format such as MPEG-4 AVC or HEVC (High Efficiency Video Coding), and the metadata adding unit 107 performs an encoding process to obtain the moving image data in the file format. Then, the metadata adding unit 107 adds the dynamic metadata as SEI (Supplemental Enhancement Information) defined by MPEG-4 AVC or HEVC.

  The output unit 108 outputs the moving image data (moving image data after the dynamic metadata is added) output from the metadata adding unit 107 to the storage unit 109. The output unit 108 may individually output the moving image data and the dynamic metadata without the metadata adding unit 107 adding the dynamic metadata to the moving image data. In that case, the metadata adding unit 107 and the output unit 108 may or may not output the moving image data and the dynamic metadata in association with each other.

  The storage unit 109 is a random access recording medium such as a CF (Compact Flash) card, and stores the moving image data output from the output unit 108 (the moving image data after the dynamic metadata is added). The storage unit 109 is removable from the imaging device 100, and can be attached to a device (personal computer or the like) other than the imaging device 100. The storage unit 109 may be a built-in recording medium that is not removable from the imaging device 100.

  The output IF 110 outputs the moving image data stored in the storage unit 109 (the moving image data after the dynamic metadata is added) to an external device (not shown). For example, the moving image data is output in a stream format according to a communication protocol based on the HDMI (registered trademark) (High-Definition Multimedia Interface) standard. The method of transmitting the moving image data and the dynamic metadata is not particularly limited. For example, a parameter defined by SMPTE (Society of Motion Picture & Television Engineers) ST 2094 may be transmitted as dynamic metadata. Specifically, Scene-Max CLL (Maximum Content Light Level) defined by HDR10 + may be transmitted as dynamic metadata.

  The CPU 111 is connected to other blocks included in the imaging device 100 via an internal bus (not shown). The CPU 111 controls the processing of the imaging device 100. The RAM 112 is connected to other blocks included in the imaging device 100 via an internal bus (not shown). The RAM 112 is used as a work area for the CPU 111 and a temporary storage area for temporarily storing various data. The ROM 113 is connected to other blocks included in the imaging device 100 via an internal bus (not shown). Firmware related to the processing of the CPU 111, information related to the processing of the CPU 111, and the like are recorded in the ROM 113 in advance.

  The operation unit 114 is connected to the CPU 111 via an internal bus (not shown). The operation unit 114 is various operation members as an input unit that receives a user operation. The operation unit 114 includes a shooting start button for starting shooting, a switch for switching between automatic control and manual control of focus operation, a focus ring for performing focus adjustment operation, and the like. The operation unit 114 has a touch panel and a liquid crystal panel (not shown), and causes the displayed function icons to act as various function buttons. The function buttons include a shooting start button, a moving image shooting mode selection button, a white balance setting button, an ISO sensitivity setting button, and the like. The moving image shooting mode includes a manual exposure shooting mode, an auto exposure shooting mode, an MF (manual focus) shooting mode, an AF (auto focus) shooting mode, a time lapse shooting mode, and a custom mode.

Next, an example of the image capturing process of the image capturing apparatus 100 will be described. FIG. 2 shows an example of a temporal change in the maximum frame luminance value acquired by the feature amount acquisition unit 104, and shows an example in the case where the scene determination processing of the scene determination unit 105 is not performed. In FIG. 2, the number of the shooting start frame (the frame obtained at the start of shooting; the first frame of the moving image) is “0”, and the shooting end frame (the frame obtained at the end of shooting; the last frame of the moving image). The example in the case of obtaining the moving image data A with the number “)” is “N” is shown. In this case, the entire period of the moving image represented by the moving image data A is treated as a period of one scene A0, and both the maximum brightness value of the moving image data A and the maximum brightness value of the scene A0 are set to the brightness value AL _MAX. Becomes Hereinafter, an example of a case where the moving image data A of FIG. 2 is obtained will be described as an example of a case where the scene determination unit 105 performs the scene determination process.

3A to 3D show an example of temporal changes of various parameters (frame maximum brightness value, determined scene period, scene feature amount, etc.) during shooting for obtaining the moving image data A. In the present embodiment, it is assumed that the maximum value of the maximum frame brightness acquired for one scene, that is, the maximum brightness value of the moving image data of the scene (scene maximum brightness value) is acquired as the scene feature amount. In FIGS. 3A to 3D, “Fr _NOW ” is the frame number (frame number) of the frame image currently being photographed. “AL _{MAX_NOW} ” is the frame maximum luminance value of the frame Fr _NOW (frame of frame number Fr _NOW ), that is, the maximum luminance value of the frame image currently being photographed. “An” in “AnL _MAX ” is a scene number (scene number), and “AnL _MAX ” is a scene maximum luminance value of the scene An (scene of scene number An). The scene A1 is a scene starting from frame 0 (shooting start frame). The frame number is counted by the scene determination unit 105. For example, the scene determination unit 105 includes a counter that increments each time frame image data is acquired, and uses the value of the counter as a frame number. The scene number is counted by the metadata generation unit 106. Details will be described later.

  FIG. 4 is a flowchart showing an example of the photographing process of the image pickup apparatus 100. The shooting process of FIG. 4 includes a scene determination process and a dynamic metadata addition process (a process of adding dynamic metadata to moving image data). The image capturing process of FIG. 4 is started by the CPU 111 detecting that the operation unit 114 has received an image capturing start operation by the user, and is realized by the CPU 111 controlling each block of the image capturing apparatus 100. The shooting start operation is, for example, a user operation of pressing the shooting start button of the operation unit 114 in the non-shooting state of the imaging apparatus 100.

In step S401, the feature amount acquisition unit 104 starts acquisition of the frame maximum luminance value AL _{MAX_NOW} of the frame image data output from the image sensor 102. The feature amount acquisition unit 104 outputs the frame maximum luminance value AL _{MAX_NOW} to the scene determination unit 105. In addition, when the frame maximum brightness value AL _{MAX_NOW} is higher than the scene maximum brightness value AnL _{MAX of the} scene An including the frame Fr _NOW , the feature amount acquisition unit 104 sets the frame maximum brightness value AL _{MAX_NOW} to the scene maximum brightness value AnL. _It is recorded in the RAM 112 as _MAX . This process can also be said to be a “process for updating the scene maximum brightness value AnL _MAX recorded in the RAM 112 with the frame maximum brightness value AL _{MAX — NOW} ”. FIG. 3A shows a state in which the frame maximum brightness value AL _{MAX_NOW} higher than the scene maximum brightness value A1L _MAX (the scene maximum brightness value of the scene A1) is detected and the scene maximum brightness value A1L _MAX is updated. When the frame Fr _NOW is frame 0, the feature amount acquisition unit 104 _{records the} frame maximum brightness value AL _{MAX_NOW} in the RAM 112 as the scene maximum brightness value A1L _MAX .

In step S402, the scene determination unit 105 determines whether to switch the scene with the frame Fr _NOW . This determination can also be said to be “determination as to whether or not to divide the moving image period by the frame Fr _NOW ” and “determination as to whether to determine (determine) a scene”. If it is determined that the scenes will not be switched (step S402: No), the process proceeds to step S406. If it is determined to switch the scene (step S402: Yes), the process proceeds to step S403.

In step S403, the scene determining unit 105, a period until the frame _{Fr the NOW} of the previous frame _{Fr the NOW} -1, determined as the period of the scene (ok).

In the present embodiment, the processing of steps S402 and S403 is performed such that the scene does not switch at the time position where the frame maximum brightness value does not change, but the scene switches at the time position where the frame maximum brightness value changes. For example, the scene is not switched at a time position where the frame maximum brightness value changes by a change amount smaller than the threshold value, and the scene is switched at a time position where the frame maximum brightness value changes by a change amount larger than the threshold value, in step S402, The process of S403 is performed. Specifically, at step S402, the scene determining unit 105 compares the frame maximum luminance value _{AL MAX_NOW} frame _{Fr the NOW,} and a frame the maximum brightness value _{AL MAX_NOW-1} frame _{Fr the NOW} -1. Then, when the difference (absolute value) between the frame maximum luminance value AL _{MAX_NOW} and the frame maximum luminance value AL _{MAX_NOW-1} is less than the threshold value, the scene does not switch, and when the difference is equal to or more than the threshold value, the scene switches. The processing of steps S402 and S403 is performed. The threshold value is not particularly limited. The threshold may be a predetermined fixed value or a value that can be changed by the user.

FIG. 3B shows a state in which the scene is determined (determined) at the timing of the frame Fr _NOW = M + 1. In the example of FIG. 3B, the scene determination unit 105 determines (determines) the period from frame 0 to frame M as the period of scene A1 when acquiring the frame image data of frame number M + 1. Then, the scene determination unit 105 generates the metadata of the scene start frame number 0 (the first frame number of the scene) 0 of the scene A1 and the scene end frame number (the last frame number of the scene) M of the scene A1. Output to the unit 106. Further, the scene determination unit 105 outputs a scene determination signal indicating that the scene has been determined to the feature amount acquisition unit 104.

When the feature amount acquisition unit 104 acquires the scene determination signal, the feature amount acquisition unit 104 changes the area of the RAM 112 that records the scene maximum luminance value AnL _MAX . As a result, the maximum scene brightness value of each scene is individually recorded in the RAM 112. In the example of FIG. 3B, in the state where the scene maximum luminance value A1L _MAX is recorded in a predetermined area of the RAM 112, the recording area of the scene maximum luminance value A2L _MAX is selected, and the frame maximum luminance value AL _{MAX_NOW} is set to the scene maximum luminance. It is recorded in the RAM 112 as the value A2L _MAX . Thereafter, the frame maximum luminance value _{AL MAX_NOW} is higher than the scene maximum luminance value _{A2L MAX,} recorded scene maximum luminance value _{A2L MAX} in RAM112 is updated in the frame the maximum luminance value _{AL MAX_NOW.}

  In step S404, the metadata generation unit 106 determines the metadata (dynamic metadata) of one scene from the scene start frame number and the scene end frame number acquired from the scene determination unit 105 and the scene maximum brightness value acquired from the RAM 112. Part of the data). The metadata of one scene includes a scene start frame number, a scene end frame number, a scene number, a scene maximum brightness value, and the like. The scene determination unit 105 includes, for example, a counter that increments each time the scene start frame number and the scene end frame number are acquired, and uses the value of the counter as the scene number. The counter may be incremented by using the update of the scene end frame number as a trigger.

In the example of FIG. 3B, the metadata generation unit 106 determines the scene number A1 according to the acquisition of the scene start frame number 0 and the scene end frame number M, and acquires the scene maximum luminance value A1L _MAX . Then, the metadata generation unit 106 generates data in which the scene start frame number 0, the scene end frame number M, the scene number A1, and the scene maximum brightness value A1L _MAX are associated with each other as the metadata of the scene A1. After that, the metadata generation unit 106 outputs the metadata of the scene A1 to the metadata addition unit 107. The metadata generation unit 106 switches the area of the RAM 112 from which the maximum brightness value of the scene is read every time the scene start frame number and the scene end frame number are acquired. As a result, the metadata generation unit 106 can read the maximum scene brightness value corresponding to the scene determined from the scene start frame number and the scene end frame number from the RAM 112.

In step S405, the metadata adding unit 107 sets the metadata (metadata of one scene) acquired from the metadata generation unit 106 to moving image data (a plurality of frame image data sequentially output from the scene determination unit 105 ( It is added to the video data of one scene). In the example of FIG. 3B, the metadata of the scene A1 is added to the moving image data of the scene A1. The moving image data to which the metadata has been added is recorded in the storage unit 109. Here, it is assumed that metadata is added as SEI of HEVC. Then, the HEVC file generated by the HEVC encoding process is recorded in the storage unit 109 as the moving image data after the metadata is added. Although details will be described later, the processes of steps S402 to S406 may be repeated. In step S405 after the second time, the current moving image (one scene of moving image) is combined with the recorded moving image (the moving image recorded in the previous step S405) (update of the recorded moving image). ).

In step S406, the CPU 111 determines whether or not the operation unit 114 has received a shooting end operation by the user. The shooting end operation is, for example, a user operation of pressing the shooting end button of the operation unit 114 in the shooting state of the image pickup apparatus 100. When it is determined that the shooting end operation has not been performed (step S406: No), the process is returned to step S402, and the processes of steps S402 to S406 are repeated. FIG. 3C shows a state after the scene A1 is determined in the frame M + 1. In FIG. 3C, “scene A2” is a scene next to the scene A1. The feature amount acquisition unit 104 records the scene maximum luminance value A2L _MAX of the scene A2 in the RAM 112 as in the case of shooting the scene A1 (as described above). If it is determined that the shooting end operation has been performed (step S406: Yes), the process proceeds to step S407. When the shooting end operation is performed, the scene determination unit 105 determines the scene start frame number of the final scene (the last scene of the video) and the scene end frame number of the final scene (the shooting end frame number; the last frame of the video). No. of) and are output to the metadata generation unit 106.

  In step S407, the metadata generation unit 106 determines the metadata (dynamic metadata) of the final scene from the scene start frame number and the scene end frame number acquired from the scene determination unit 105, and the maximum scene brightness value acquired from the RAM 112. Part of the data).

FIG. 3D shows a state in which the shooting of the image of the shooting end frame N is finished and the acquisition of the moving image data A is finished. In the example of FIG. 3D, the metadata generation unit 106 determines the scene number A2 according to the acquisition of the scene start frame number M + 1 and the scene end frame number N, and acquires the scene maximum luminance value A2L _MAX . Then, the metadata generation unit 106 generates, as the metadata of the scene A2, data in which the scene start frame number M + 1, the scene end frame number N, the scene number A2, and the scene maximum brightness value A2L _MAX are associated with each other. After that, the metadata generation unit 106 outputs the metadata of the scene A2 to the metadata addition unit 107.

  In step S408, the metadata adding unit 107 sets the metadata (final scene metadata) acquired from the metadata generating unit 106 to moving image data (a plurality of frame image data sequentially output from the scene determining unit 105 ( The video data of the final scene). In the example of FIG. 3D, the metadata of the scene A2 is added to the moving image data of the scene A2. The moving image data to which the metadata has been added is recorded in the storage unit 109. When the process of step S405 is performed, the current moving image (the moving image of the final scene) is combined with the recorded moving image (the moving image recorded in step S405) (recorded already). Video update).

  As described above, according to the present embodiment, the imaging device determines a plurality of scenes based on the temporal change of the frame maximum luminance value, and acquires the scene maximum luminance value of each of the plurality of scenes. Then, information (dynamic metadata) in which the maximum scene brightness value is associated with each of a plurality of scenes is generated. That is, the dynamic metadata and the like can be acquired without going through the post production (image data editing process). As a result, the moving image display method can be sequentially changed based on the dynamic metadata. For example, in HDR (high dynamic range) display using tone mapping, the tone map can be sequentially changed based on the dynamic metadata.

Although an example of repeating the process of recording the moving image data of the scene in the storage unit 109 has been described with reference to FIG. 4, the present invention is not limited to this. For example, the metadata adding unit 107 adds the dynamic metadata to the moving image data representing the entire moving image so that the metadata is added to the part of each scene when the moving image capturing is completed, and the dynamic metadata is added. The moving image data after being processed may be recorded in the storage unit 109. In this case, the metadata adding unit 107 temporarily records the acquired moving image data (a plurality of frame image data) and metadata in the RAM 112.

  Although an example in which a scene is specified by a frame number has been described using FIG. 4, the present invention is not limited to this. For example, the scene may be specified by the shooting time or the shooting time. Specifically, the scene shooting start time may be used instead of the scene start frame number, and the scene shooting end time may be used instead of the scene end frame number. The scene shooting start time is the time from the predetermined timing (such as the timing when the shooting of the moving image starts) to the start of the scene shooting. The scene shooting end time is the time from the predetermined timing to the end of the scene shooting. It's time.

Although the example of detecting the scene change from the change in the maximum frame luminance value between consecutive frames has been described with reference to FIG. 4, the method for determining the scene is not limited to this, and the parameter for determining the scene is It is not limited to the frame maximum brightness value. The parameter for determining the scene may be different from the frame feature amount for acquiring the scene feature amount. For example, the parameter for determining the scene includes a frame maximum brightness value of a corresponding frame that is a frame corresponding to the parameter and a frame maximum brightness value of one or more frames that are temporally consecutive with respect to the corresponding frame. It may be an average value. Specifically, the scene determining unit 105 records the frame maximum luminance value of the frame _{Fr the NOW,} and a frame the maximum brightness value of one or more temporally successive frames prior to the frame _{Fr the NOW} the RAM112 You may. Then, the scene determination unit 105 may determine a plurality of scenes so that the scenes are switched when the average value of the frame maximum brightness values changes by a change amount larger than the threshold value. The frame maximum luminance value of the frame that is temporally subsequent to the frame Fr _NOW may be used. A frame the maximum luminance value of the temporally successive frames prior to the frame Fr _{the NOW,} may be used one of the frame a maximum luminance value of the consecutive frames temporally after the frame Fr _{the NOW,} Both may be used. There may be a plurality of parameters (a plurality of types of parameters) for determining the scene.

  The frame feature amount is not limited to the frame maximum brightness value, and the scene feature amount is not limited to the scene maximum brightness value. For example, the frame feature amount may be another representative value (average value, minimum value, mode value, intermediate value, etc.) of luminance values of frame image data or a histogram. The scene feature amount may be another representative value (average value, minimum value, mode value, intermediate value, etc.) of the frame feature amount acquired for the scene. The representative value (maximum value, average value, minimum value, mode value, intermediate value, etc.) of the brightness value of the scene moving image data from the scene moving image data (scene moving image data) without determining the scene characteristic amount from the frame characteristic amount. ) May be acquired as the scene feature amount. The frame feature amount of one frame and the scene feature amount of one scene may include a plurality of values (a plurality of types of values).

<Example 2>
Example 2 of the present invention will be described below. In the following, points (configuration, processing, etc.) different from the first embodiment will be described in detail, and description of the same points as the first embodiment will be omitted. In the first embodiment, the example in which the frame maximum luminance value is used as the parameter for determining the scene has been described. The imaging parameter is often changed when the scene is switched. Therefore, in the present embodiment, an example will be described in which an imaging parameter used when capturing a frame image is used as a parameter for determining a scene. Specifically, an example in which an aperture value is used as a parameter for determining a scene in the manual exposure shooting mode will be described.

The imaging device according to the present embodiment has the same configuration as the imaging device 100 of FIG. 1 (first embodiment). However, in this embodiment, the scene determination processing by the scene determination unit 105 is different from that in the first embodiment. Further, in the present embodiment, the feature amount acquisition unit 104 sets the frame maximum brightness value to the scene determination unit 10.
Do not output to 5. Instead, the imaging control unit 103 outputs the aperture value to the scene determination unit 105. The imaging control unit 103 may also output the gain value, shutter speed, and the like to the scene determination unit 105.

  An example of the processing flow relating to the scene determination processing will be described. First, the operation unit 114 receives a mode change operation (user operation) for changing to the manual exposure shooting mode. The CPU 111 controls each block of the image pickup apparatus 100 according to the mode changing operation and sets the manual exposure shooting mode. Next, the operation unit 114 receives a shooting start operation. The CPU 111 controls each block of the image pickup apparatus 100 in response to a shooting start operation to start shooting. Next, the operation unit 114 receives an aperture change operation (user operation) for changing the aperture value. The imaging control unit 103 changes the state of the imaging optical system 101 according to the changed aperture value, and outputs the changed aperture value to the scene determination unit 105. Next, the scene determination unit 105 determines whether to determine (determine) a scene according to the change in the acquired aperture value. In other words, the scene determination unit 105 determines whether to switch the scene at the current timing according to the acquired aperture value change.

FIG. 5 shows an example of temporal changes in the maximum frame brightness value and the aperture value, and shows an example in which the scene determination processing of the scene determination unit 105 is not performed. FIG. 5 shows an example in which moving image data B in which the shooting start frame number is “0” and the shooting end frame number is “N” is obtained. In FIG. 5, the aperture value is changed from F2.2 to F2.0 when the image of the frame M + 1 is captured. However, since the scene determination process is not performed, the entire period of the moving image represented by the moving image data B is treated as one period of the scene B0, and the maximum luminance value of the moving image data B and the maximum luminance value of the scene B0 are combined. Both result in a brightness value BL _MAX . In the following, as an example of performing the scene determination processing of the scene determination unit 105, an example in which the aperture value is changed and the moving image data B of FIG. 5 is obtained as in FIG. 5 will be described.

FIG. 6 shows an example of a temporal change in the maximum frame brightness value and the aperture value, and shows an example in the case of performing the scene determination processing of the scene determination unit 105. In this embodiment, the scene determination unit 105 determines a plurality of scenes so that the scenes are switched at the time position where the aperture value changes. Therefore, as shown in FIG. 6, the period from frame 0 to frame M is determined as the period of scene B1, and the period from frame M + 1 to frame N is determined as the period of scene B2. Then, by the same processing as that in the first embodiment, the brightness value B1L _MAX is acquired as the scene maximum brightness value of the scene B1, the brightness value B2L _MAX is acquired as the scene maximum brightness value of the scene B2, and the scene maximum brightness value B1L _MAX , Dynamic metadata including B2L _MAX is generated.

  As described above, according to the present embodiment, the imaging apparatus determines a plurality of scenes based on the temporal change of the imaging parameter (aperture value), and the dynamic metadata is generated as in the first embodiment. .. That is, dynamic metadata and the like can be acquired without going through post production. As a result, the moving image display method can be sequentially changed based on the dynamic metadata.

  Although the example in which the manual exposure shooting mode is set has been described, the automatic exposure shooting mode in which the aperture value is automatically changed may be set. In the automatic exposure setting mode, the imaging control unit 103 refers to the frame image data acquired from the image sensor 102 and automatically changes the aperture value (not by a user operation). Even when the automatic exposure setting mode is set, as in the case where the manual exposure setting mode is set, it is possible to determine a plurality of scenes based on the temporal change of the aperture value.

The example in which the scene is switched at the time position where the aperture value changes (even slightly) has been described, but the scene may be switched at the time position where the aperture value changes by a change amount larger than the threshold value. For example,
The control may be performed such that the scene is not switched even if the aperture value changes by less than 1 step (such as 1/3 step), and the scene is switched when the aperture value changes by 1 step or more.

  The imaging parameter for determining the scene is not limited to the aperture value. For example, a plurality of scenes may be determined based on time changes such as ISO sensitivity, shutter speed, focus position, focal length, white balance, and exposure value. The exposure value can be calculated from the ISO sensitivity, the shutter speed, and the aperture value. As the imaging parameter for determining the scene, one type of imaging parameter may be used, or a plurality of types of imaging parameters may be used.

  When there are a plurality of parameters for determining the scene, the scene determination unit 105 may determine the plurality of scenes by using a parameter according to the set shooting mode among the plurality of parameters. .. For example, in the case of a shooting mode in which the ISO sensitivity is automatically set, a plurality of scenes may be determined based on the temporal change of the ISO sensitivity instead of the temporal change of parameters other than the ISO sensitivity. In the case of the shooting mode in which the white balance is automatically set, a plurality of scenes may be determined based on the time change of the white balance instead of the time change of the parameters other than the white balance. Similarly, the aperture value may be used in the aperture priority mode, and the shutter speed may be used in the shutter speed priority mode. In the case of the manual mode in which all the imaging parameters are manually set, a plurality of scenes may be determined in consideration of all the imaging parameters. At least one of the plurality of parameters may be designated by the user as a parameter for determining the scene.

  The control may be performed such that the scene determination process is not performed (a plurality of scenes are not determined) when a specific shooting mode is set. For example, when the focus is changed in the AF shooting mode, it is highly likely that the same scene image is shot. Therefore, the scene is not switched, and the intended image is made when the focus is changed in the MF shooting mode. You may perform control which switches a scene. The execution / non-execution of the scene determination process may be designated by the user regardless of the shooting mode or the like.

  The parameters for determining the scene, the method for switching execution / non-execution of the scene determination processing, the shooting mode for performing (or not performing) the scene determination processing, and the like are not particularly limited.

<Example 3>
Example 3 of the present invention will be described below. In the following, points (configuration, processing, etc.) different from the first embodiment will be described in detail, and description of the same points as the first embodiment will be omitted. In the present embodiment, an example will be described in which a scene is determined and dynamic metadata is generated in consideration of a focus area (area within the depth of field) of a frame image. Although the shooting mode is not particularly limited, an example of the MF shooting mode will be described in this embodiment.

  FIG. 7 is a block diagram showing a configuration example of the image pickup apparatus 700 according to the present embodiment. 7, the same blocks as those in FIG. 1 (Embodiment 1) are denoted by the same reference numerals as those in FIG. The image pickup apparatus 700 has a feature amount acquisition unit 704 instead of the feature amount acquisition unit 104 of the first embodiment, and has a scene determination unit 705 instead of the scene determination unit 105 of the first embodiment. Furthermore, the imaging device 700 has a depth of field calculation unit 715.

The feature amount acquisition unit 704 has the same function as the feature amount acquisition unit 104 of the first embodiment. However, the feature amount acquisition unit 704 sets the maximum brightness value of the image data corresponding to the focused area (in-depth-of-field area) of the frame image as the frame maximum brightness value, not the maximum brightness value of the entire frame image data. get. The focus area (area within the depth of field) is notified from the depth of field calculation unit 715. Similar to the first embodiment, the frame maximum brightness value (maximum brightness value of image data corresponding to the focus area) is a feature amount for determining a plurality of scenes and a feature amount for acquiring the scene feature amount. Used as both. The maximum brightness value of the image data corresponding to the in-focus area is acquired as the feature amount for determining the plurality of scenes, and the maximum brightness value of the entire frame image data is acquired as the feature amount for acquiring the scene feature amount. May be obtained. The reverse is also acceptable.

  The scene determination unit 705 has the same function as the scene determination unit 105 of the first embodiment. However, when the frame maximum luminance value changes by a change amount larger than the threshold value, the scene determination unit 705 considers the temporal change of the focus area (area within the depth of field) and determines whether to switch the scene. To judge. The focus area (area within the depth of field) is notified from the depth of field calculation unit 715.

  The depth of field calculation unit 715 acquires the aperture value, the focus value (focus position), and the zoom value (focal length) from the imaging control unit 103. In other words, the imaging control unit 103 outputs the aperture value, the focus value, and the zoom value to the depth of field calculation unit 715. The depth of field calculation unit 715 calculates the depth of field from the aperture value, the focus value, and the zoom value. Then, the depth-of-field calculation unit 715 notifies the feature amount acquisition unit 704 and the scene determination unit 705 of the in-depth-of-field region of the frame image as the focus region.

  An example of the processing flow relating to the scene determination processing will be described. First, the operation unit 114 receives a mode change operation (user operation) for changing to the MF shooting mode. The CPU 111 controls each block of the image pickup apparatus 100 according to the mode change operation and sets the MF shooting mode. Next, the image pickup control unit 103 controls the state of the image pickup optical system 101, the process of the image pickup element 102, and the like according to the aperture value, shutter speed, focus value, zoom value, and the like. Further, the imaging control unit 103 outputs the aperture value, the focus value, and the zoom value to the depth of field calculation unit 715.

Then, the depth-of-field calculation unit 715 calculates the depth-of-field from the aperture value, the focus value, and the zoom value acquired from the imaging control unit 103. For example, the depth of field is calculated using the following Expression 1.

  Next, the depth of field calculation unit 715 determines, based on the frame image data acquired from the image sensor 102 and the calculated depth of field, that the depth of field of the frame image represented by the frame image data is within the depth of field. Detect the area. Then, the depth-of-field calculation unit 715 notifies the in-depth-of-field area (focus area) to the feature amount acquisition unit 704 and the scene determination unit 705. The method of detecting the focused area is not particularly limited. For example, an edge area having a predetermined spatial frequency band may be detected (edge detection), and an image area in which the density of the detected edge area is larger than a predetermined threshold may be determined as the focus area.

  Next, the operation unit 114 receives a shooting start operation. The CPU 111 controls each block of the image pickup apparatus 100 in response to a shooting start operation to start shooting. Then, the feature amount acquisition unit 704 acquires the maximum luminance value of the image data corresponding to the area within the depth of field notified from the depth of field calculation unit 715 as the frame maximum luminance value, and the frame maximum luminance value is calculated. Output to the scene determination unit 105. Next, the scene determination unit 705 determines a plurality of scenes using the frame maximum brightness value acquired from the feature amount acquisition unit 104 and the depth-of-field internal region notified from the depth-of-field calculation unit 715. To do.

  8A to 8D show examples of frame images. In FIGS. 8A to 8D, a focus value for focusing on the subject 800 is set. In FIGS. 8A and 8D, a deep depth of field is set, and the area 801 (entire frame image) is set as the focus area. In FIGS. 8B and 8C, a shallow depth of field is set, and an area 802 (a part of the frame image) narrower than the area 801 is set as the focus area, and the areas 801 to 801 are used. An area 803 excluding 802 is a non-focus area.

  In this embodiment, a plurality of scenes are determined based on the temporal change in the maximum frame brightness value of the focused area. Therefore, as shown in FIG. 8C, even if the high-luminance region 804 occurs in the non-focused region 803 and the frame maximum luminance value increases by an increase amount larger than the threshold value, the scene is determined (determined). Not done (scenes cannot be switched). Accordingly, in the display based on the dynamic metadata, it is possible to prevent the appearance of the subject (the subject that is likely to be noticed) in the in-focus region from changing due to the change in the brightness in the out-of-focus region.

  In the present embodiment, a state with a shallow depth of field (a state with a narrow focus area; FIG. 8C) is changed to a state with a deep depth of field (a state with a wide focus area; FIG. 8D). If it changes, the scene is determined (determined) according to a large change in the maximum frame brightness (the scene is switched). As a result, it is possible to display the subject 800 affected by the high brightness area 804. On the other hand, the state where the depth of field is deep (the state where the focus area is wide; FIG. 8D) is changed to the state where the depth of field is shallow (the state where the focus area is narrow; FIG. 8C). In this case, the scene is not determined (confirmed) regardless of the temporal change of the maximum frame brightness (the scene is not switched). This enables display without changing the appearance of the subject 800.

  As described above, according to the present exemplary embodiment, more suitable dynamic metadata can be acquired and a plurality of scenes can be more preferably determined by considering the focus area.

  Each block of the first to third embodiments (FIGS. 1 and 7) may or may not be individual hardware. The functions of two or more blocks may be implemented by common hardware. Each of the plurality of functions of one block may be realized by individual hardware. Two or more functions of one block may be implemented by common hardware. Also, each block may or may not be realized by hardware. For example, the device may have a processor and a memory in which a control program is stored. Then, the functions of at least some blocks of the device may be realized by the processor reading the control program from the memory and executing the control program.

  It should be noted that Embodiments 1 to 3 (including the above-described modifications) are merely examples, and configurations obtained by appropriately modifying or changing the configurations of Embodiments 1 to 3 within the scope of the gist of the present invention. Are also included in the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the first to third embodiments.

<Other Examples>
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100, 700: Imaging device 102: Imaging device 104, 704: Feature amount acquisition unit 105, 705: Scene determination unit 106: Metadata generation unit

Claims (17)

An image capturing means for capturing a moving image,
Deciding means for deciding a plurality of scenes of the moving image based on a temporal change of a parameter corresponding to a frame of the moving image,
An acquisition means for acquiring the respective feature quantities of the plurality of scenes,
Generating means for generating information associating the feature quantity acquired by the acquiring means with each of the plurality of scenes;
An image pickup apparatus comprising: The imaging apparatus according to claim 1, further comprising an output unit that outputs the moving image data and the information. The image pickup apparatus according to claim 2, wherein the output unit outputs the moving image data and the information in association with each other. The image pickup apparatus according to claim 1, wherein the feature amount includes a maximum brightness value of moving image data of a scene corresponding to the feature amount. The image pickup apparatus according to claim 1, wherein the parameter includes a maximum brightness value of image data of the frame. The determination unit determines the plurality of scenes such that the scene does not switch at a time position where the parameter does not change, and the scene switches at a time position where the parameter changes. The imaging device according to any one of items. The determination means is configured such that the scene does not switch at a time position where the parameter changes by a change amount smaller than a threshold, and the scene switches at a time position where the parameter changes by a change amount larger than the threshold. The image pickup apparatus according to claim 6, wherein the scene is determined. The image pickup apparatus according to claim 1, wherein the parameter includes an image pickup parameter when the image of the frame is picked up. The image pickup apparatus according to claim 8, wherein the image pickup parameter includes at least one of ISO sensitivity, shutter speed, aperture value, focus position, focal length, white balance, and exposure value. The acquisition means is
The maximum brightness value of the image data corresponding to the focus area of the image of the frame is obtained for each of the plurality of frames of the moving image,
The imaging device according to claim 1, wherein the maximum value of two or more maximum brightness values acquired for the scene of the moving image is included in the feature amount of the scene. The determining unit determines the plurality of scenes so that the scenes are not switched regardless of a temporal change of the parameter at a time position where a focused area of an image of the frame is narrowed. The imaging device according to any one of 10. There are multiple parameters corresponding to the frame,
The said determination means determines the said some scene using the parameter according to the photography mode set among the said several parameters, The any one of the Claims 1-11 characterized by the above-mentioned. Imaging device. 13. The image pickup apparatus according to claim 1, wherein the determination unit does not determine the plurality of scenes when a specific shooting mode is set. The parameter includes an average value of the maximum brightness value of the image data of the corresponding frame that is the frame corresponding to the parameter and the maximum brightness value of one or more frames that are temporally consecutive with respect to the corresponding frame. The imaging device according to any one of claims 1 to 13, characterized in that. The image pickup apparatus according to claim 1, wherein the feature amount is dynamic metadata defined by SMPTE ST 2094. An image capturing step for capturing a moving image,
A determining step of determining a plurality of scenes of the moving image based on a temporal change of a parameter corresponding to a frame of the moving image;
An acquisition step of acquiring respective feature quantities of the plurality of scenes,
A generation step of generating information in which the feature amount acquired in the acquisition step is associated with each of the plurality of scenes;
A method for controlling an imaging device, comprising:   A program for causing a computer to function as each unit of the imaging device according to claim 1.

Images