JP6604822B2 - Movie processing device - Google Patents

Description

  The present invention relates to a moving image processing apparatus that combines moving images having different gamma characteristics.

  In a conventional image recording / playback system, the recording side (imaging side) performs gamma correction that compresses the high-brightness part, and the playback side (display side) performs gamma correction that restores the high-brightness component to the overall brightness. A linear expression is realized. Such processing is mainly to compensate for nonlinear input / output characteristics and narrow dynamic range of the devices involved.

  In addition to the conventional gamma correction shooting mode (normal shooting mode) that compresses high-intensity parts, the shooting mode (high-brightness priority mode) that does not compress high-intensity parts is added in addition to the conventional high-brightness part compression. An imaging device that can be selected has been proposed. In order to enable the display device to determine in which shooting mode the image is taken, the captured image metadata is transmitted including a flag for identifying the normal shooting mode and the high luminance priority mode. In addition, the metadata includes information such as an input dynamic range, a magnification with respect to a display luminance reference value, a peak luminance value, gamma shape information of the imaging unit, and a base gamma of the imaging unit.

  Patent Document 1 describes a technique in which illumination data switching time information is included in the metadata of moving image data, and the display device switches the illuminance at a time according to the illuminance switching time information.

JP 2004-112290 A

  When playing back an image shot in the high brightness priority mode, it is possible to prevent the display image from becoming dark by making the brightness of the medium and low brightness portions equal to the brightness in the normal shooting mode. That is, when displaying an image in the high brightness priority mode, the brightness of the display device is relatively higher than that in the normal shooting mode.

  When continuously playing movies in normal shooting mode and movies shot in high-luminance priority mode, the gamma correction to be processed on the display section differs, so the average brightness of the displayed image changes and Give a sense of incongruity. A similar problem may occur when a plurality of moving images having different gamma characteristics are combined and displayed together.

  In view of such problems, an object of the present invention is to provide a moving image processing apparatus that appropriately processes metadata when a plurality of moving images having different gamma characteristics are combined for continuous display.

In order to solve the above problems, a moving image processing apparatus according to the present invention provides:
Reading means for reading a plurality of moving image data from a recording medium, moving image data shot in the first shooting mode with a gamma characteristic for suppressing a high luminance portion based on metadata of the plurality of moving image data, and a high luminance portion A determination unit that determines whether moving image data shot in the second shooting mode with a relatively small gamma characteristic is mixed in a plurality of moving image data; a combining unit that combines the plurality of moving image data; Depending on whether or not the moving image data shot in the first shooting mode and the moving image data shot in the second shooting mode are mixed in the plurality of moving image data to be combined, the metadata of the moving image data before combining And control means for controlling whether or not to take over the video data after being combined.

  According to the present invention, when a plurality of moving images are combined, the transfer of the metadata of the moving images before combining to the moving image data after the combining is controlled. Thus, the metadata can be handled in a consistent manner depending on whether the gamma characteristics of the moving images to be combined are the same or different.

It is a schematic block diagram of the Example of this invention. It is a control flowchart at the time of imaging of a present Example. It is a control flowchart at the time of the moving image composition of a present Example. It is a schematic diagram explaining a gamma correction. It is an example of the monitor screen at the time of imaging. It is a control flowchart at the time of a display of a present Example.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram of a schematic configuration of an imaging apparatus in which an embodiment of a moving image processing apparatus according to the present invention is implemented. The imaging apparatus 100 illustrated in FIG. 1 has a gamma characteristic (for example, a linear characteristic with respect to luminance) that prioritizes high luminance in addition to a normal shooting mode with a gamma characteristic that suppresses a high luminance portion as usual. Has high brightness priority mode. The user can select which shooting mode to use, the high-luminance priority shooting mode or the normal shooting mode. The high brightness priority mode is an example of a shooting mode having a gamma characteristic in which suppression of a high-brightness portion is relatively reduced as compared with the gamma characteristic of the normal shooting mode.

  The imaging optical system of the imaging apparatus 100 includes a lens 101 and a diaphragm 102, and a subject optical image that passes through the lens 101 and the diaphragm 102 is formed on an imaging surface of an image sensor (imaging device) 105. The image sensor 105 converts the subject optical image on the imaging surface into an electrical signal (image signal), and supplies the generated image signal to the signal processing unit 112.

  The signal processing unit 112 includes a white balance correction processing unit 107, an edge enhancement processing unit 108, a gamma correction processing unit 109, and a luminance / color information detection unit. Processing other than white balance correction processing, edge enhancement processing, gamma correction processing, and luminance / color information detection processing in the signal processing unit 112 is omitted. The signal processing unit 112 converts the image data processed by the gamma correction processing unit 109 into moving image data in a predetermined video format, and then supplies the video data to the camera microcomputer 111 and the recording / playback unit 113.

  The white balance correction processing unit 107 performs white balance correction processing on the image signal from the image sensor 105. The edge enhancement processing unit 108 performs edge enhancement processing on the image signal processed by the white balance correction processing unit 107. The gamma correction processing unit 109 performs gamma correction processing on the image signal processed by the edge enhancement processing unit 108. The luminance / color information detection unit 110 divides the image of the image signal output from the gamma correction processing unit 109 in the horizontal and vertical directions to set a plurality of luminance / color information detection frames, and within each luminance / color information detection frame Are integrated to detect luminance information and color information of each detection frame. The luminance / color information detection unit 110 supplies the detected luminance information and color information to the camera microcomputer (camera control unit) 111.

  The camera microcomputer 111 calculates each correction value used for white balance correction processing, edge enhancement processing, and gamma correction processing using subject information such as luminance information and color information from the luminance / color information detection unit 110. The camera microcomputer 111 supplies the calculated correction values to the processing units 107, 108, and 109, and causes white balance correction processing, edge enhancement processing, and gamma correction processing using these correction values.

  The camera microcomputer 111 controls charge accumulation and reading of the image sensor 105, exposure time (shutter speed), and the like via the image sensor control unit 106. The camera microcomputer 111 controls the focus and zoom of the lens 101 via the lens control unit 103. The camera microcomputer 111 controls the aperture 102 via the aperture controller 104 according to the luminance information and color information detected by the luminance / color information detector 110. Furthermore, the camera microcomputer 111 can correct camera shake at the time of photographing by the user by driving the lens 101 through the lens control unit 103 or by controlling the signal processing unit 112.

  The recording / reproducing unit 113 compresses and encodes the moving image data and its metadata from the signal processing unit 112 and records the obtained compressed data on the recording medium 114. The recording / reproducing unit 113 also restores the moving image data and its metadata by reading the compressed data from the recording medium 114 and decompressing it. The recording / reproducing unit 113 supplies the moving image data being recorded and its metadata or the moving image data reproduced from the recording medium 114 and its metadata to the camera microcomputer 111 for monitoring. The recording medium 114 is made of, for example, a magnetic tape, an optical disk, a nonvolatile semiconductor memory, a hard disk, or the like. The moving image compression / decompression function may be realized by software on the camera microcomputer 111.

  The camera microcomputer 111 includes an analyzing unit 120, a combining unit 121, and a determining unit 122 that function when moving image data having different shooting modes are continuously input. The analysis unit 120 analyzes metadata attached to the moving image data. The combining unit 121 combines moving image data having different shooting modes. The determining unit 122 determines metadata to be taken over by the moving image data combined by the combining unit 121. Details of the functions of these means 120, 121, and 122 will be described later.

  The user can use the operation unit 119 to instruct the camera microcomputer 111 about the operation mode and various operations. For example, in the shooting mode, the user can instruct start and end of recording, and in the playback mode, the user can specify a moving image to be played back and instruct to start / stop playback.

  The display 130 includes a display microcomputer 131 that controls the entire display 130, a signal processing unit 132 that performs image processing on input moving image data, a metadata analysis unit 138 that analyzes metadata, and a display element 139 that displays an image. To do. In the signal processing unit 132, an image quality mode corresponding to the image quality (for example, resolution) of the moving image input by the image quality mode setting unit 133 is set, and the peak luminance setting unit 134 sets the peak luminance. The color correction unit 135 corrects the color data to a color space corresponding to the display element 139. The gamma correction processing unit 136 corrects the gamma characteristic of the input image with the gamma correction characteristic specified by the display microcomputer 131. The high luminance adaptive processing unit 137 performs processing adapted to the high luminance priority mode in accordance with an instruction from the display microcomputer 131 in the high luminance priority mode. The display microcomputer 131 controls the signal processing unit 132, particularly the gamma correction processing unit 136, according to the metadata attached to the input moving image data. The display element 139 displays the moving image data image-processed by the signal processing unit 132 as an image.

  FIG. 2 shows a flowchart of gradation correction processing according to the shooting mode. The camera microcomputer 111 determines the gamma correction characteristic of the gamma correction processing unit 109 according to the shooting mode applied to the moving image shooting according to the flowchart shown in FIG. 2, and generates metadata to be recorded together with the moving image data. For example, when the user sets the shooting mode as the operation mode, the camera microcomputer 111 starts the gradation correction process shown in FIG.

  First, the camera microcomputer 111 determines an input dynamic range (S201). Here, the input dynamic range is a predetermined value set in advance according to the shooting mode, or a value calculated based on the luminance information and color information detected by the luminance / color information detection unit 110. Next, the camera microcomputer 111 performs exposure control based on the luminance information and color information detected by the luminance / color information detection unit 110 (S202).

  The camera microcomputer 111 determines whether the shooting mode is the high brightness priority mode or the normal shooting mode (S203). When the high luminance priority mode is designated (S203), the camera microcomputer 111 determines a gamma correction curve for high luminance priority as a gamma correction characteristic used for gamma correction processing (S204). Then, the camera microcomputer 111 generates metadata corresponding to the high brightness priority mode, and sets the metadata in the recording / playback unit 113 for recording (S205). The metadata generated here includes a shooting mode flag indicating a high luminance priority mode, an input dynamic range, a magnification with respect to a preset reference value of display luminance, a peak luminance value, gamma shape information of the imaging unit, and a base of the imaging unit. It consists of gamma.

  When the normal shooting mode is designated as the shooting mode (S203), the camera microcomputer 111 determines a gamma correction curve for normal shooting as a gamma correction characteristic used for gamma correction processing (S206). Then, the camera microcomputer 111 generates metadata corresponding to the normal shooting mode, and sets it in the recording / playback unit 113 for recording (S207). Here, the generated metadata includes a shooting mode flag indicating the normal shooting mode, an input dynamic range, a magnification with respect to a preset reference value of display luminance, a peak luminance value, gamma shape information of the imaging unit, a base gamma of the imaging unit, and the like. is there. The only difference between the high brightness priority mode and the normal shooting mode as an element to be included in the metadata is a shooting mode flag for identifying the high brightness priority mode or the normal shooting mode.

  After steps S205 and S207, the camera microcomputer 111 sets the gamma correction curves determined in S204 and S206, respectively, in the gamma correction processing unit 109 (S208), and ends the flow shown in FIG. The gamma correction processing unit 109 performs gamma correction processing on the image data from the image sensor 105 using the gamma correction curve set in S208.

  The gradation correction process shown in FIG. 2 is executed again when the shooting mode is switched from the high brightness priority mode to the normal shooting mode or vice versa, or when the operation mode is switched to a mode other than the shooting mode.

  FIG. 3 is an operation flowchart relating to handling of metadata when a moving image in the high brightness priority mode and a moving image in the normal shooting mode are continuously combined. The camera microcomputer 111 executes the processing shown in FIG. 3 when, for example, a moving image in the normal shooting mode (or high luminance priority mode) is reproduced and displayed following a moving image in the high luminance priority mode (or normal shooting mode). .

  The camera microcomputer 111 waits for selection by the user of moving images to be combined (S301). The user can instruct the camera microcomputer 111 of two moving images to be combined using the operation unit 119. Three or more moving images may be specified at the same time. However, in order to facilitate understanding, here, a case where two moving images are specified will be described as an example. The operation of selecting the selected moving image again with the operation unit 119 is regarded as canceling the selection by the camera microcomputer 111. Such inversion processing is well known.

  The analysis unit 120 of the camera microcomputer 111 determines whether the metadata of the two moving images selected in S301 are the same. In the example described here, the camera microcomputer 111 determines whether the shooting mode flags are the same. If the metadata is the same (S302), the camera microcomputer 111 sets “0” to the mixed flag indicating the difference in metadata. If the metadata is different (S302), the camera microcomputer 111 sets “1” in the mixed flag and displays a warning on the display 130 that the metadata cannot be taken over. If metadata is given to one moving image, but metadata is given to the other moving image, the camera microcomputer 111 determines that the metadata is different.

  After S303 and S304, the camera microcomputer 111 waits for an instruction to execute the combination by the user (S305). If the user does not instruct the camera microcomputer 111 to execute the combination using the operation unit 119 (S305), the camera microcomputer 111 returns to S301 and repeats S301 to S304.

  When there is an instruction for combining by the user (S305), the combining means 121 of the camera microcomputer 111 combines the selected moving image data. Then, the determination unit 122 refers to the mixed flag and determines whether or not the metadata should be taken over to the combined moving image data (S306). Specifically, the determination unit 122 checks whether the mixed flag is “0” or “1” (S306). When the mixed flag is “1” (S306), the camera microcomputer 111 cannot transfer the metadata to the combined moving image data (S307). In other words, the combined moving image data is handled as having been shot in the normal shooting mode. On the other hand, when the mixed flag is “0” (S306), the camera microcomputer 111 takes over the metadata in the combined moving image data (S308). That is, the combined moving image data is handled as being captured in the same shooting mode as that of the combined moving image data.

  “Combination” in the flow shown in FIG. 3 is not limited to the case where data of two moving images are combined into one file, but includes the case where two or more moving images are designated to be continuously played back. This is because a similar problem can occur in the display even when a plurality of moving images are continuously reproduced.

With reference to FIG. 4, the gamma correction characteristic for the high luminance priority mode will be described. 4 (a) is a gamma characteristic 402 which the input dynamic range corresponds to _{x 1,} showing a gamma characteristic 404 which the input dynamic range corresponding to _{x 2.} The gamma characteristic 402 is ITU-R BT. This is a gamma characteristic standardized by 709 or the like. The gamma characteristic 404 is obtained by extending the input dynamic range to a practical range. The gamma characteristic 404 with respect to the gamma characteristic 402 is a characteristic that weakens the degree of compression of the contrast, maintains the relationship of the output code with respect to the input code, and fixes the allocation of bits from low luminance to high luminance.

  FIG. 4B shows that the multiplication result of the gamma characteristic 406 at the time of imaging and the gamma characteristic 408 of the display device becomes the gamma characteristic 410 on the monitor screen observed by the user. Ideally, the gamma characteristic 406 or the gamma characteristic 408 is set so that the gamma characteristic 410 is linear with respect to the subject luminance.

The gamma characteristic 406 of the imaging means corresponding to the reference input dynamic range x ₁ is
y = f (x)
Then, the gamma characteristic of the imaging means when the input dynamic range is t times is
y = f (x / t)
And it is sufficient. The input dynamic range may be set to an appropriate value for each product, or may be set to an appropriate value for each shooting mode within the same product. Furthermore, it may be set adaptively for each shooting scene within the same shooting mode.

  With reference to FIG. 5 and FIG. 6, an example of a screen at the time of shooting in the high luminance priority mode and a gamma correction control operation of the display 130 being monitored will be described.

  FIG. 5 shows an example of proper exposure display in the high luminance priority mode. A shooting condition 502 is displayed at the upper left of the screen of a viewfinder (for example, the display 130) that displays a subject image to be shot, and an exposure state 504 is displayed at the lower left. In the illustrated example, the shooting condition 502 includes an aperture value (IRIS), a shutter speed (SHT), and a gain (GAIN). The exposure state 504 illustrates the current exposure value as a triangle on a bar indicating from overexposure to underexposure. When the user changes the aperture value, shutter speed, or gain value using the operation unit 119, the position of the instruction value in the exposure state 504 is also updated.

  The gamma characteristic in the high-luminance priority mode of this embodiment is a relatively dark characteristic as compared with the gamma characteristic in the normal photographing mode, and the appropriate exposure value is also a dark value. Therefore, the user can confirm whether or not the appropriate exposure value is obtained in the high luminance priority mode by looking at the indication value of the exposure state 504.

  FIG. 6 is a flowchart of gradation correction processing by the display microcomputer 131 when an image being shot in the high brightness shooting mode is monitored on the display 130. The metadata analysis unit 138 analyzes the metadata input together with the moving image data, and supplies the analysis result to the display microcomputer 131.

  The display microcomputer 131 uses the metadata shooting mode flag (identification information indicating the normal shooting mode or the high luminance priority mode) from the metadata analysis unit 138 to determine whether or not the input moving image data has been shot in the high luminance shooting mode. Is determined (S601). Of course, the user can manually specify to which display mode the gamma correction corresponding to which shooting mode should be applied to the display microcomputer 131 via the camera microcomputer 111 by the operation unit 119 or the like.

  In the case of the high brightness priority mode (S601), the display microcomputer 131 sets the gamma characteristic of the display unit closest to the inverse characteristic of the gamma characteristic of the imaging unit according to the gamma shape information and the base gamma of the imaging unit included in the metadata. A correction curve is determined (S602). The display microcomputer 131 may employ a gamma correction curve determined in advance for high-luminance priority video without using information included in the metadata.

  On the other hand, in the normal shooting mode (S601), the display microcomputer 131 determines a gamma correction curve predetermined for normal video (S603).

  The display microcomputer 131 sets the gamma correction curve determined in steps S602 and S603 in the gamma correction processing unit 136, and the gamma correction processing unit 136 performs gamma correction processing on the input moving image with the set gamma correction curve (S604).

  The display microcomputer 131 repeats the display gradation correction process shown in FIG. 6 until the display 130 is turned off and the control is completed.

  As described above, in the moving image processing apparatus, it is possible to avoid contrast compression due to gamma correction processing during imaging regardless of the input dynamic range. It is possible to make the gradation characteristic of the system total, which combines the gamma characteristic at the time of imaging and the gamma characteristic at the time of display, linear with respect to the luminance. More natural gradation, color reproducibility and sharpness can be realized in the entire area from the dark part to the highlight part. By switching the gamma characteristic at the time of display so that it is closest to the reverse characteristic of the gamma characteristic at the time of imaging, the linearity accuracy of the gradation characteristic in the system can be improved, and the gradation, color reproducibility, and sharpness can be improved. More improved.

Claims (9)

Reading means for reading a plurality of moving image data from a recording medium;
Based on the plurality of moving image data of metadata, a moving image data photographed by the first photographing mode gamma characteristic for suppressing the high-luminance portion, suppression is relatively small gamma characteristic second of the high-intensity part Determination means for determining whether moving image data shot in a shooting mode is mixed in the plurality of moving image data ;
A combining means for combining the plurality of moving image data;
The moving image before combining depending on whether the moving image data shot in the first shooting mode and the moving image data shot in the second shooting mode are mixed in the plurality of moving image data to be combined moving picture processing device, characterized in that it comprises a control means for controlling whether or not to give inherits metadata data to a video data after binding. The control means includes
If it is determined that the moving image data shot in the first shooting mode and the moving image data shot in the second shooting mode are not mixed in the plurality of moving image data, the moving image data before combining In order to be handled as moving image data shot in either the first shooting mode or the second shooting mode, by taking over the metadata of
If it is determined that the moving image data shot in the first shooting mode and the moving image data shot in the second shooting mode are mixed in the plurality of moving image data, the moving image data before combining The video data after the combination is handled as the video data shot in the first shooting mode without transferring the metadata to the video data after combining.
The image processing apparatus according to claim 1. The metadata, moving image processing apparatus according to claim 1 or 2, characterized in that the information indicating the photographing mode at the time of imaging of the moving picture data. The metadata, information indicating the shooting mode at the time of imaging of the moving image data, the input dynamic range, according to claim 1 or characterized in that it has at least one of the base gamma during gamma shape information and shooting at the time of imaging 2. The moving image processing apparatus according to 2.   5. The moving image processing apparatus according to claim 1, further comprising: a display unit configured to display when the metadata assigned to each of the plurality of moving image data is different. When the predetermined metadata of the plurality of moving image data is different , the determination unit includes a mixture of moving image data shot in the first shooting mode and moving image data shot in the second shooting mode. in which the moving image processing device according to claim 1, wherein in any one of 5 that determines. Wherein the plurality of moving image data to be combined in said coupling means, if it contains the video data imaged by the first imaging mode, the video data imaged by the second imaging mode, the moving image data of after the coupling 7. The moving image processing apparatus according to claim 1, further comprising a gamma correction processing unit that performs gamma correction with a gamma correction curve corresponding to the first shooting mode. Display control means for controlling the moving image to be displayed on the display means based on the moving image data;
Wherein the display control unit, before Symbol For video data photographed by the first photographing mode, and controls to display to correct the video based on predetermined gamma correction information, the second photographing 5. In the case of moving image data shot in a mode , control is performed so that the moving image is corrected and displayed based on information related to gamma identification included in metadata of the moving image data. The moving image processing apparatus described. A reading step of reading a plurality of moving image data from a recording medium;
Based on the metadata of the plurality of moving image data, the moving image data shot in the first shooting mode of the gamma characteristic for suppressing the high luminance part and the second of the gamma characteristic for which the suppression of the high luminance part is relatively small. A determination step of determining whether moving image data shot in the shooting mode is mixed in the plurality of moving image data;
A combining step of combining the plurality of moving image data;
The moving image before combining depending on whether the moving image data shot in the first shooting mode and the moving image data shot in the second shooting mode are mixed in the plurality of moving image data to be combined A control process for controlling whether or not the metadata of the data is inherited and added to the combined video data;
A method for controlling a moving image processing apparatus comprising:

Images