JP2019096984A - Level adaptive video switching processing method and processing apparatus for HDR video - Google Patents

Abstract

To improve the visibility and the video quality as the entire HDR video content by HDR video editing/switching processing in consideration of the light and dark adaptation of human eyes.SOLUTION: A switcher that outputs a video selected from a plurality of input videos includes a median value calculation unit that calculates a median value of all luminance values of each video frame for the plurality of input videos, a difference calculation unit that calculates a difference between a median value of a currently selected and output video and a median value of another video from among the input videos, and a gain offset adjustment unit that adjusts, for a predetermined period, the output level of the luminance value of the frame after switching to a new output video when a difference between the median value of the currently selected and output video and the median value of a video newly selected as a new output from among the input videos is larger than a predetermined threshold value.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a level adaptive video switching processing method and processing apparatus for HDR video.

  Automatic adjustment of the brightness of the television screen has already been put to practical use according to the degree of brightness of the room in which the television is disposed. Such a television includes an illuminance sensor, and adjusts the brightness of the television screen automatically and appropriately according to the output of the illuminance sensor. In addition, this technology is also applied and implemented in in-vehicle monitors such as navigation.

  In the dark area, the screen is also dark, and in the bright area, the screen is also adjusted bright, which takes into consideration the eye characteristics of the human body. For example, Patent Document 1 below describes that a device using a dimmer control element or a light receiving sensor is known as a means for adjusting the luminance of the LCD.

  In addition, as a response to the sudden luminance change in a continuous series of images, a flash phenomenon in which the display screen becomes bright for a while is visually recognized because the luminance of the current frame is larger than the luminance of the previous frame and the luminance of the next frame. As a technique for preventing this flash phenomenon, there is a method of storing a frame in a frame memory and performing brightness adjustment before writing display data in order to prevent a sudden change in brightness of a display image. This is disclosed in Patent Document 2 below.

  Further, in an imaging scene where a new image is captured and acquired, it is described in Patent Document 3 that the brightness adjustment is performed by appropriately changing the exposure of the imaging device according to the degree of brightness of the subject illumination.

  In addition, various methods such as cut, mix, dissolve, wipe, and DVE have been realized as switching processing when editing a video. However, there is no known device that takes into account the visibility of the HDR video. As one close to this, there is one that automatically changes the brightness of the display according to the viewing environment light in the television receiver, and for example, such a technology is disclosed in Patent Document 4 below.

  Also, an auto iris function is known which automatically adjusts the aperture of the camera. However, these relate to a television receiver for viewing and listening to video and a camera for capturing video, and such a technology has not been applied to video production and editing at all and is not used at all.

Japanese Patent Application Laid-Open No. 11-340644 JP, 2016-033607, A JP, 2015-035757, A JP, 2005-295262, A

  The above-described conventionally known luminance adjustment is an imaging device that newly acquires an image, or is a response to a luminance change in a series of continuous video software, or according to the surrounding environment in which a monitor is disposed. The switcher switches the video scene to be output as needed based on the instruction of the operator etc. at any time while constantly capturing in real time a plurality of videos acquired by the plurality of imaging devices in real time. Etc. are not compatible with such devices.

  In an image processing apparatus such as a switcher, a plurality of images are always input in real time, and an image suitable for an announcement or situation at that time is appropriately determined and instantly instructed by an operator etc. and provided to the viewer. A video scene is selected and output. For example, a switching scene for switching between a real-time image capturing the current state of the site in a news image and an image of a progress announcer of a broadcasting station is a typical example.

  In an image processing apparatus for capturing a plurality of videos in real time with a typical example of such a switcher, there may be a situation where the brightness of the provided screen changes rapidly with the switching of the output video scene. . For example, in the case of switching the video scene provided to the viewer from the dark site video showing the current situation at night of the accident site to the announcer in the bright studio, so-called "return the microphone to the studio", etc. Is the case.

  The image processing apparatus itself such as a switcher is always inputted with a plurality of images such as an image showing a scene and an image of a studio, so the luminance of each image in real time can be grasped and detected, but it is outputted. The switching timing of when the video is switched can not be known unless a switching instruction based on the judgment of the operator or the like is actually input. Therefore, the image processing apparatus such as the switcher constantly monitors the brightness of a plurality of input videos so that it can cope with the output video in real time whenever it is switched. It is preferable to be able to perform brightness adjustment to avoid drastic changes.

  Further, in the case of the HDR video with the dynamic range expanded, the video becomes considerably brighter than the conventional SDR (standard dynamic range) video. For this reason, when the video is switched by simple cut processing, or when switching from a bright video to a dark video, conversely, when switching from a dark video to a bright video, the human changes against the brightness change of the video after switching. Visibility is markedly reduced due to the inability to cope with the light and dark adaptation of the eye. In the extreme case, the video disappears for a certain period of time. This is due to the fact that the light and dark adaptation of the human eye takes a certain time while the image switching process is performed instantaneously. As described above, in the conventional video image switching processing, processing taking into consideration the dynamic range of such video and adaptation to light and dark of human eyes is not performed.

  In view of the above-mentioned problems, in the present invention, the visibility after the switching process of the HDR video is improved by the simple video processing. Specifically, the median value of all luminance values for each video frame is calculated, and the level of the luminance value of the frame after video switching is set for a certain period, preferably human eyes, according to the median difference value between videos. The visibility shall be improved by adjusting for a certain period corresponding to the light-dark adaptation required time of. Although it is possible to uniformly adjust the gain regardless of the luminance level of the pixel as the adjustment content of the luminance level, it is preferable to use the gain adjustment to which an offset is added.

  In the present invention, after switching from a bright video to a dark video and vice versa, from a dark video to a bright video, by the video switching process by level adaptation in consideration of the brightness for each HDR video and the brightness difference between the HDR videos. The visibility of the image can be improved. By the HDR video editing / switching process in consideration of the light and dark adaptation of the human eye, it is possible to improve the visibility and the video quality as the entire HDR video content.

It is a conceptual diagram explaining the level application switching processing structure of this embodiment. It is a figure explaining the histogram which shows the relation between the luminosity value and appearance frequency value in a HDR picture. FIG. 8 is a diagram for explaining an example in which a cumulative histogram is calculated from a luminance value histogram and a pixel level exceeding half of the total number of pixels corresponds to a median value. FIG. 8 is a diagram for explaining an example of gain offset control when switching from bright video to dark video and when switching from dark video to bright video. FIG. 8 is a diagram for explaining an example of gain offset control when switching from bright video to dark video and when switching from dark video to bright video.

  When switching a video with an expanded dynamic range (hereinafter referred to as “HDR video” (High-dynamic-range rendering) as appropriate), the visibility may significantly decrease due to the balance with the time required for light and dark adaptation of the human eye. It occurs. That is, when switching from a bright video to a dark video instantaneously, and vice versa, when switching from a dark video to a bright video instantaneously, the light and dark adaptation of the human eye does not catch up in a certain period after switching, and in extreme cases Sometimes disappears. In the HDR video, there is a great concern that the above-mentioned undesirable situation is caused due to its characteristics.

  Therefore, in the present embodiment, a device for improving the visibility when switching to a video with different brightness is performed by video processing in a device that handles switching of a plurality of videos such as a switcher. Specifically, the median value of all luminance values is calculated for each frame, and the level of the luminance value of the frame after image switching is operated for a certain period according to the median difference value between the images, thereby controlling the brightness of the eye. According to the adaptation time, the rapid change in luminance is mitigated to improve the visibility.

  The processing method first calculates the median value of all luminance values in a video frame for each HDR video. In the heavy distribution at the bottom including a small number of high luminance values (wide luminance distribution at the bottom) such as HDR video, the average value hardly makes sense, and the average is on the high luminance side due to the influence of a small number of high luminance values I will be heavily biased. Therefore, in the present invention, a median value is used instead of the average value as an index indicating the brightness of the entire frame.

  Next, at the time of video switching, the brightness of the video after switching is adjusted to the brightness of the video before switching for a certain period according to the difference value of the median value of the luminance value for each frame of both videos switched. Level control. In this case, the level control of the brightness of the video after the video switching is such that the gain offset is controlled by a defined time constant so as to return to the original brightness after a certain frame has passed.

  Furthermore, a luminance value histogram for calculating the median value of all luminance values in the video frame from the luminance value histogram, and median value calculation means, calculating the difference value of the median values of all luminance values in each video frame at the time of video switching. Luminance value median difference calculation means for switching, video image switching means, gain offset means for video after switching according to a predetermined time constant according to a difference value of median values of all luminance values in each video frame, and A gain offset control means or the like for controlling them can be provided.

  The level adaptive video switching process according to the present invention can be performed in real time by hardware, and can also be used for video editing by software. However, in the case of real-time processing by hardware, it takes one frame of time to calculate the median value of the luminance value in the entire video frame, so there is a delay of one frame as it is. However, if the median value of all luminance values of the frame one frame before the switching signal is used or if the all luminance value median value of the current frame is predicted by the Kalman filter, autoregression, etc. using the results up to the previous frame There is no frame delay, and application in live relay is also possible.

  FIG. 1 is a conceptual diagram for explaining the level application switching processing configuration of the present embodiment. As shown in FIG. 1, it is assumed that A input and B input are switched as video input. Then, for each input, the median value of all luminance values for each input frame is calculated (median value calculation processing). Then, the difference value of the median value of all the luminance values of each frame is calculated. Here, it is assumed that the A input and the B input have their time axes aligned with each other.

  Then, using the video switching signal as a trigger, calculation of the gain and offset of the video after video switching is started from the difference value of the median values of all luminance values of each input (gain and offset calculation). The gain and the offset are calculated, for example, as follows according to the number of elapsed frames after the image switching.

  In the above [Equation 1], “a” is a constant that determines the brightness in the first frame after the video switching, and “τ” is the time during which the level controlled video after the video switching returns to the original brightness It is a time constant to decide. In addition, “x” is the number of elapsed frames for which the video switching time is 0.

  A constant "a" that determines the brightness of the first frame after image switching, and a time constant "r" that determines the time when the level-controlled video returns to the original brightness, according to the magnitude of the median difference value for each frame If it is set and the absolute value of the median difference value is below the appropriate threshold, level control may not be performed.

  And for the video after switching the video,

  And the output after final switching. Here, “Sel” is a luminance value of each pixel of the video selected by the selector among the A input video and the B input video. That is, the luminance value of the pixel to be selected by the selector and to be output is multiplied by the gain, the offset value is further added, the adjustment is made, and then the output is performed.

  FIG. 2 is a diagram for explaining a histogram showing the relationship between the luminance value and the appearance frequency value in the HDR image. As shown in FIG. 2, the HDR image has a small number of extremely high luminance pixels, resulting in a so-called heavy-end (wide-end) luminance distribution. For this reason, even if the average value of the luminance values in the image is calculated, it becomes a value pulled to the high luminance value side under the influence of a small number of super-high luminance pixel values. It is appropriate. From this point of view, in the present embodiment, it is assumed that the median value (the luminance value at the center of the number of pixels when arranged in order from the pixel with high luminance) is used without using the average value.

  FIG. 3 is a view for explaining an example in which a cumulative histogram is calculated from a luminance value histogram and a pixel level exceeding half of the total number of pixels corresponds to a median value. FIG. 3 (a) is a diagram of a luminance value histogram showing the relationship between the luminance value of each pixel and the number of pixels having the luminance value for a female photo sample image, and FIG. 3 (b) is the one having a smaller luminance value. FIG. 6 is a diagram showing a cumulative histogram (the cumulative addition of luminance value histograms) in which the number of pixels is accumulated in order from. In the example shown in FIG. 3B, the luminance value "120" when the cumulative number of pixels is half the total number of pixels is the median value.

  4 and 5 show examples of gain offset control in the case of switching from a bright video to a dark video and in the case of switching from a dark video to a bright video. However, “Offset = 0” is set for switching from dark video to bright video. FIG. 4A is a view for explaining an example of gain offset adjustment in the case of switching from a bright image to a dark image, and FIG. 4B is a diagram for explaining gain offset adjustment in the case of switching from a dark image to a bright image It is a figure explaining an example.

  As shown in FIG. 4 (a), when switching from a bright image to a dark image suddenly, it takes a certain time for the human eye to adapt to the darkness. Thus, the output luminance value is offset with respect to the input luminance value to adjust the gain. That is, since the one with a large input signal level (input luminance value) is a relatively bright pixel, the gain adjustment is small, and the one with a small input signal level (input luminance value) is a relatively dark pixel. Adjust to make it easy to view as a large gain adjustment. Then, as time passes so that a human can adapt to darkness, as shown in a graph 2000 of FIG. 4A, it is gradually returned to a natural state in which gain offset adjustment is not performed.

  On the other hand, as shown in FIG. 4 (b), when switching from a dark image to a bright image suddenly, it takes a certain time for the human eye to adapt to the brightness. As shown in, the output luminance value is offset with respect to the input luminance value to adjust the gain. That is, since those with large input signal levels (input luminance values) are extremely bright pixels, the output value is lowered to obtain large gain adjustment, and those with small input signal levels (input luminance values) are relatively dark pixels, so they are small. Adjust to make it easy to read as gain adjustment. Then, as time passes so that the human can adapt to the brightness, as shown in the graph 4000 of FIG. 4B, it is gradually returned to the natural state where the gain offset adjustment is not performed.

  FIG. 5 shows the gain and offset when switching from a bright image to a dark image in the case of “a = 0.3, τ = 6.0” in the equation described as [Equation 1] above. It is a figure explaining the time change of adjustment amount. In the example shown in FIG. 5, after about 30 frames (about 1 second), the gain offset adjustment amount becomes zero, and the output state of the original video level is restored. The adjustment time and the change with time of the adjustment amount can be appropriately set in accordance with the characteristics of the image environment and the human eye.

  Further, in the above-described example, the video switching scene between the two inputs of the A input and the B input has been described. However, although switching processing between three or more plural inputs is possible, it is possible to apply. Furthermore, the present invention can be applied to the case where output switching is performed between videos having a large difference between light and dark even in the case of HDR video.

The present invention is applicable to an apparatus for performing image switching such as various video processing devices, video switchers, routers, etc., and is particularly suitable for an HDR video production processing apparatus etc. in which the dynamic range is expanded.

Claims (17)

Calculating a median value of all luminance values for each video frame for a plurality of input videos;
Calculating the difference of the median value between the input images;
Adjusting the level of the luminance value of the frame after switching to the output video selected as the output video among the input video when the difference of the median value between the input video is larger than a predetermined threshold; A method of outputting an image selected from a plurality of input images, comprising: The method according to claim 1, wherein the video selected from the plurality of input videos is output,
The method according to the present invention, wherein the predetermined period is a certain period corresponding to the light-dark adaptation required time of human eyes. In the method of outputting a video selected from a plurality of input videos according to claim 1 or 2,
When the difference between the median values between the input images is larger than a predetermined threshold value,
The median value of all luminance values for each video frame for the currently output video,
The method is characterized in that the difference between the video after the switching and the median value of all luminance values of each video frame is larger than the predetermined threshold. A method of outputting a video selected from a plurality of input videos according to any one of claims 1 to 3.
The step of adjusting the level of the luminance value of the frame after switching to the output image for a predetermined period is:
A method characterized in that it is gain adjustment with an offset according to the luminance level. A method of outputting a video selected from a plurality of input videos according to any one of claims 1 to 4.
The step of adjusting the level of the luminance value of the frame after switching to the output image for a predetermined period is:
When switching from a video having a large median value to a video having a small median value, adjustment is made to increase the level of the luminance value of the frame after switching to the output video,
And adjusting the level of the luminance value of the frame after the switching to the output video when the video is switched from a video having a small median value to a video having a large median value. In the image processing method of the switcher,
Calculating a median value of all luminance values for each video frame for a plurality of input videos;
Calculating the difference of the median value between the input images;
Adjusting the level of the luminance value of the frame after switching to the output video selected as the output video among the input video when the difference of the median value between the input video is larger than a predetermined threshold; A video processing method for a switcher, comprising: In the switcher video processing method according to claim 6,
The video processing method of the switcher according to claim 1, wherein the predetermined period is a certain period corresponding to the light-dark adaptation required time of a human eye. In the switcher video processing method according to claim 6 or 7,
When the difference between the median values between the input images is larger than a predetermined threshold value,
The median value of all luminance values for each video frame for the currently output video,
A video processing method of a switcher, wherein a difference between the video after the switching and the median value of all luminance values for each video frame is larger than the predetermined threshold. A video processing method of a switcher according to any one of claims 6 to 8.
The step of adjusting the level of the luminance value of the frame after switching to the output image for a predetermined period is:
A video signal processing method of a switcher characterized by gain adjustment with an offset according to a luminance level. In the switcher image processing method according to any one of claims 6 to 9,
The step of adjusting the level of the luminance value of the frame after switching to the output image for a predetermined period is:
When switching from a video having a large median value to a video having a small median value, adjustment is made to increase the level of the luminance value of the frame after switching to the output video,
The video processing method of a switcher, comprising performing adjustment to reduce the level of the luminance value of the frame after switching to the output video when switching from a video with a small median value to a video with a large median value. In a switcher that outputs a video selected from a plurality of input video,
A median value calculator configured to calculate a median value of full luminance values for each video frame for the plurality of input videos;
A difference calculating unit that calculates a difference between a median value of a currently selected and output video and a median value of another video of the input video;
When the difference between the median value of the video currently selected and output and the median value of the video newly selected as a new output of the input video is larger than a predetermined threshold value, the new output A switcher, comprising: a gain offset adjustment unit configured to adjust an output level of a luminance value of a frame after switching to an image for a predetermined period. In the switcher according to claim 11,
The switcher characterized in that the predetermined period is a certain period corresponding to the light-dark adaptation required time of human eyes. In the switcher according to claim 11 or 12,
When the difference between the median values is larger than a predetermined threshold value,
The median value of all luminance values for each video frame for the currently output video,
A switcher characterized in that the difference between the video after the switching and the median value of all luminance values of each video frame is larger than the predetermined threshold. The switcher according to any one of claims 11 to 13.
The gain offset adjustment unit
A switcher characterized by performing gain adjustment with an offset according to the luminance level of an output video signal. The switcher according to any one of claims 11 to 14.
The gain offset adjustment unit
When switching from a video having a large median value to a video having a small median value, adjustment is made to increase the level of the luminance value of the frame after switching to the output video,
When switching from a video having a small median value to a video having a large median value, the switcher is adjusted to lower the level of the luminance value of the frame after switching to the output video. The switcher according to any one of claims 11 to 15.
The gain offset adjustment unit
Determine the gain and offset respectively based on
Here, “a” is a constant that determines the brightness in the first frame after image switching, and “τ” is a time constant that determines the time when the level-controlled image after image switching returns to the original brightness Yes, let "x" be the number of elapsed frames with 0 at the time of video switching,
A switcher characterized by In the switcher according to claim 16,
The gain offset adjustment unit
Adjust the output after switching based on,
Here, “Sel” is a luminance value of each pixel of the video selected by the selector among a plurality of input video,
A switcher characterized by