JP4246711B2 - Moving picture coding method, moving picture coding apparatus, moving picture coding program, and computer-readable recording medium - Google Patents

Description

  The present invention relates to a high-efficiency image signal encoding method, and in particular, a moving-image code for performing pre-filtering that enables high-speed and high-efficiency encoding when the luminance value of an image changes suddenly due to flashing or flashing of a light. It relates to the conversion method.

H. In the case of an encoding method based on inter-frame prediction represented by H.264 (see Non-Patent Document 1), encoding efficiency deteriorates in a scene including a luminance change such as flash. This is because a luminance change that cannot be predicted only by motion compensation occurs between the reference signal and the predicted signal. Therefore, a method called luminance compensation (see Non-Patent Document 2) has been studied in order to cope with a scene with such luminance change. This is an inter-frame prediction that corrects a luminance change, and is used in combination with motion compensation that corrects a spatial position. A similar method is described in H.264. H.264 is adopted as Weighted Prediction.
Joint Video Team. Draft ITU-T Recommendation and Final draft international standard of joint video specification.ITU-T Rec.H.264 and ISO / IEC 14496-10 AVC, 2003. Hitoshi Uekura, Hiroshi Watanabe, Naoki Kobayashi, Susumu Ichinose, Hiroshi Yasuda. Global motion / brightness change compensation video coding with reduced computational complexity. IEICE Transactions, Vol. J82-B, No. 9, pp 1676-1688, Spt. 1999.

  In the case of encoding using the above-described luminance compensation, the following two problems occur. The first problem is an increase in code amount by adding a luminance compensation parameter. Since a luminance compensation parameter is required in addition to the motion vector required for normal motion compensation, there is a possibility that the amount of code allocated to the prediction residual is insufficient at low rates, resulting in image quality degradation.

  The second problem is an increase in the amount of calculation by calculating the brightness compensation parameter. H. In coding based on inter-frame prediction such as H.264, most of the coding processing is devoted to motion compensation. Furthermore, if luminance compensation is performed, the calculation load for inter-frame prediction increases. For this reason, when it is necessary to perform encoding processing in real time in an environment of limited computer resources, an increase in calculation load becomes a serious problem.

The present invention has been made in view of such circumstances, and an object of the present invention is to establish a high-speed and high-efficiency encoding method particularly for a frame accompanied by a rapid luminance change due to blinking of a light or the like .

Visual sensitivity decreases in frames with rapid brightness changes. For this reason, only the brightness level is important from the viewpoint of subjective image quality, and it is not always necessary to faithfully reproduce the waveform. Therefore, in the present invention, pre-filter processing for replacing a simple image in which the luminance change is emphasized is applied by applying the concept of the visual masking effect in the encoding of the frame accompanying the luminance change due to blinking of the light or the like.

The basic idea is as follows .
• Frames whose brightness values suddenly decrease due to flashing lights are replaced with black images.
-It should be noted that a frame whose luminance value is suddenly increased by flash or the like may be replaced with a white monochrome image.

  In the case of a monochromatic image signal, the amount of generated code can be reduced as compared with the case of encoding the original signal. This can solve the problem of increased code amount. Also, when replacing with a white / black monochrome image on the encoding side, the encoded data can be created before encoding. The process at the time of encoding only needs to read encoded data created in advance. For this reason, the processing time required for encoding can be reduced to a negligible level compared to other frames. This can solve the problem of increasing the amount of calculation.

That is, the main features of the present invention are as follows. When encoding a frame that includes low- brightness pixels due to a change in luminance in moving image encoding, the frame that includes low-brightness pixels is appropriately detected, and the frame to be encoded is a black monochrome image. It performs processing that converts the signal.

In particular, when detecting a frame in which a luminance change has occurred in the above moving image encoding, the luminance change amount between the encoding target frame and the previous frame is larger than a predetermined threshold, and the average luminance in the encoding target frame is A luminance change detection condition is determined such that the value is smaller than a predetermined threshold and the luminance change amount in the encoding target frame is smaller than the predetermined threshold.

  Further, when the image signal after the filter processing is encoded, an encoded bit stream that is an encoding result of the signal is created and stored in advance, and the stored bit stream is read when the signal is encoded.

  The bit stream output from the pre-filter that performs the filtering process is multiplexed with the bit stream output from the encoder.

In the present invention, pre-filter processing for replacing with a black single color image is performed in encoding of a frame accompanied by a sudden low luminance change due to blinking of a light or the like. Frame including a blinking lights, in order to faithfully reproduce the its signal waveform requires a code amount corresponding. However, in the case of a black monochromatic image signal, the amount of generated code can be suppressed to a very small level, and the image quality of other frames can be improved with the code amount saved here. Even if a low-brightness frame in a video with flashing lights is changed to a black color image, visual abruptness cannot follow such a sudden change in luminance, so that the observer does not know the original image and the image quality deteriorates. It is not perceived as. In addition, it is possible to expect a masking effect that image quality deterioration of the frames before and after the black monochrome frame is difficult to be detected.

  In addition, by replacing the image with blinking light with a black image, the noise component superimposed due to insufficient light quantity can be removed. As a result, it is possible to avoid the code amount being assigned to the noise component. With this saved code amount, the image quality of other frames can be improved.

The replacement with a black monochrome image on the encoding side means that the encoded data can be created before encoding. The process at the time of encoding only needs to read encoded data created in advance. For this reason, the processing time required for encoding can be reduced to a negligible level compared to other frames.

In the following embodiment, an example will be described in which, in addition to the encoding process for a black screen due to the blinking of light, the encoding process for converting the flash emission frame to a white screen is performed. For the pre-filter process described above, it is necessary to detect a frame in which a luminance change has occurred. For the detection, the following tendency regarding the inter-frame difference signal and the average luminance value / dispersion within the frame is used.

In the case of flash emission frames, the following trends are shown.
1. The luminance change from the previous frame is large, and the difference between frames increases.
2. The flash increases the luminance value in the frame.

On the other hand, in the case of a black screen due to the flickering of lights, the following tendency is shown.
1. The luminance change from the previous frame is large, and the difference between frames increases.
2. The luminance value in the frame becomes smaller due to the black screen.
3. The change in brightness in the frame is reduced due to the black screen.

Therefore, a case where the following inequalities (1) and (2) are satisfied is replaced with a white monochromatic image as a flashed frame. On the other hand, a case where the inequalities (1), (3), and (4) are satisfied is replaced with a black monochrome image as a black screen frame. In the following, it is assumed that the luminance value at the coordinates (x, y) in the frame at time t is f (x, y, t) and the frame size is X × Y. Each threshold value T _d , T _f , T _b , T _v is assumed to be given from the outside.

なお，Ｍ（ｔ）は，次式で表されるフレーム内の平均輝度値である。

M (t) is an average luminance value in the frame represented by the following equation.

各不等式で行う判定の意味は，次の通りである。
不等式（１）：前フレームとのフレーム間差分≧Ｔ_d
（前フレームから大きく変化している）
不等式（２）：フレーム内の平均輝度≧Ｔ_f
（フラッシュの影響で輝度値が大きくなっている）
不等式（３）：フレーム内の平均輝度≦Ｔ_b
（黒画面のため，輝度値が小さくなっている）
不等式（４）：フレーム内の分散≦Ｔ_v
（黒画面化の影響でフレーム内の細部がつぶれている）
なお，輝度変化が複数のフレームにわたる場合にも，各閾値Ｔ_d ，Ｔ_f ，Ｔ_b ，Ｔ_v を適切に設定することで，複数フレームをプレフィルタ処理の対象とすることが可能である。

The meaning of the determination performed in each inequality is as follows.
Inequality (1): Interframe difference from previous frame ≧ T _d
(It has changed greatly from the previous frame)
Inequality (2): Average luminance within the frame ≧ T _f
(The brightness value has increased due to the effect of the flash.)
Inequality (3): Average luminance within the frame ≦ T _b
(The brightness value is small because of the black screen)
Inequality (4): variance within frame ≦ T _v
(Details in the frame are crushed due to the black screen)
Even when the luminance change extends over a plurality of frames, it is possible to set a plurality of frames as a target for prefiltering by appropriately setting the threshold values T _d , T _f , T _b , and T _v .

  The frame after such filtering does not cause visual image quality degradation due to the visual masking effect, but from the viewpoint of the signal waveform, it has a low correlation with the previous and subsequent frames. High prediction efficiency cannot be expected. For this reason, it is necessary to change the frame after this filter processing to a picture type that is not referred to by other frames. For example, in the case of MPEG-2, if the frame filtered to a white monochrome image is compatible with flash and is a frame of picture type P Picture or I Picture, the picture type is set to B Picture (in MPEG-2, B Picture is another (Not referenced from the frame). The same applies to black monochrome images.

  Hereinafter, specific embodiments of the pre-filter process according to the present invention will be described with reference to the drawings. FIG. 1 is a process flowchart according to the embodiment of the present invention.

  First, in step S11, the image signal of the encoding target frame and the image signal of the immediately preceding frame are input, and determination processing for determining whether or not the amount of change in luminance value from the immediately preceding frame to the encoding target frame exceeds a certain threshold value. And output the true / false value that is the judgment result. If the output is a true value, the process proceeds to the subsequent prefiltering process (step S12). If the output is a false value, the input encoding target frame is not filtered and sent as an encoding target signal to the encoder, and the process proceeds to step S17. As a specific determination process, a process according to the above-described formula (1) is performed in step S11.

  In step S12, the image signal of the encoding target frame is input, a determination process is performed as to whether or not the average luminance value of the signal is below a certain threshold value, and a true / false value as a determination result is output. If the output is a true value, the process proceeds to the subsequent prefiltering process (step S13). If the output is a false value, the process proceeds to the prefiltering process in step S15. As a specific determination process, a process according to the above-described equation (3) is performed in step S12.

  In step S13, the image signal of the encoding target frame is input, a determination process is performed as to whether or not the amount of change in the luminance value in the frame is below a certain threshold value, and a true / false value as a determination result is output. If the output is a true value, the process proceeds to the subsequent prefiltering process (step S14). If the output is a false value, the input encoding target frame is not filtered and sent as an encoding target signal to the encoder, and the process proceeds to step S17. As a specific determination process, a process according to the above-described equation (4) is performed in step S13.

  In step S14, an encoded bit stream for a black monochrome image stored in advance is read, and the stream is output as the encoding result of the encoding target frame. The stream is obtained by previously encoding a black monochrome image with an encoder.

  In step S15, the image signal of the encoding target frame is input, a determination process is performed as to whether or not the average luminance value of the signal exceeds a certain threshold value, and a true / false value as a determination result is output. If the output is a true value, the process proceeds to the subsequent prefiltering process (step S16). If the output is a false value, the input encoding target frame is not filtered and sent as an encoding target signal to the encoder, and the process proceeds to step S17. As a specific determination process, a process according to the above-described equation (2) is performed in step S15.

  In step S16, an encoded bit stream for a white monochrome image stored in advance is read, and the stream is output as the encoding result of the encoding target frame. The stream is obtained by previously encoding a white monochrome image with an encoder.

  In step S17, the image signal of the encoding target frame is input, and the encoded bit stream is output according to the encoding process by the encoder. The encoding method is assumed to be given from the outside.

  In step S18, the bit stream output from step S14, step S16, or step S17 is input, a process for multiplexing the bit stream for each frame is performed by the multiplexer, and the multiplexed bit stream is output.

  FIG. 2 shows a configuration diagram of an encoding apparatus incorporating the prefilter according to the embodiment of the present invention. The pre-filter processing unit 1 receives the image signal of the encoding target frame and the image signal of the immediately preceding frame, performs the nonlinear filter processing described with reference to FIG. 1, and performs the processing on the image signal of the encoding target frame or the signal after filtering. Output the encoded bitstream.

  The encoder 2 receives the image signal of the encoding target frame and outputs an encoded bit stream according to the encoding process. The encoding method is assumed to be given from the outside. As the encoding method, various kinds of moving image encoding methods that have been conventionally used can be used.

  The multiplexer (MUX) 3 receives the bit stream output from the pre-filter processing unit 1 and the encoder 2, performs a process of multiplexing the bit stream for all frames, and outputs the multiplexed bit stream To do.

  FIG. 3 shows a configuration diagram of the prefilter processing unit 1 in FIG. Here, the solid line represents the flow of the image signal, and the broken line represents the flow of the control signal.

  The inter-frame change amount determiner 11 receives the image signal of the encoding target frame and the image signal of the immediately preceding frame as input, and performs a process of calculating the amount of change in the luminance value from the immediately preceding frame to the encoding target frame. Judgment is made based on the quantity, and two control signals are output. If it is determined that the amount of change is large, a control signal is sent to the switch SW2, and the switch SW2 is turned on. Further, a false value is sent to the logical sum circuit (OR) 14. If it is determined that the amount of change is small, a true value is sent to the OR circuit 14. In this case, the subsequent determination processing is stopped, the switch SW1 is turned on, and the image signal of the encoding target frame is output as it is.

The intra-frame average value determination unit 12 receives the image signal of the encoding target frame, performs a process of calculating an average value of luminance values of the signal,
(i) The average value exceeds the threshold T _f ,
(ii) The average value is below the threshold T _b ,
(iii) Other than (i) and (ii) above,
As a result of the determination, 1 is output if (i) is satisfied, 2 is output if (ii) is satisfied, and 3 is satisfied if (iii) is satisfied. Is output.

  When the output is 1, the control signal is sent to the switch SW4 and the switch SW5, the switch SW4 is turned on, and the switch SW5 is turned on to the white monochrome image encoded bitstream storage unit 15 side. Further, a false value is sent to the logical sum circuit 14. When the output is 2, the control signal is sent to the switch SW3, and the switch SW3 is turned on. Further, a false value is sent to the logical sum circuit 14. When the output is 3, a true value is sent to the OR circuit 14. In this case, the subsequent determination processing is stopped, the switch SW1 is turned on, and the image signal of the encoding target frame is output as it is.

  An intra-frame change amount determination unit 13 receives an image signal of an encoding target frame, performs a process of calculating a luminance change amount in the frame of the signal, makes a determination based on the change amount, and performs two controls. Output a signal. When it is determined that the amount of change is small, a control signal is sent to the switches SW4 and SW5, the switch SW4 is turned on, and the switch SW5 is turned on to the black monochrome image encoded bitstream storage unit 16 side. Further, a false value is sent to the logical sum circuit 14. On the other hand, if it is determined that the amount of change is large, a true value is sent to the OR circuit 14. In this case, the switch SW1 is turned on and the image signal of the encoding target frame is output as it is.

  FIG. 4 shows a configuration diagram of the three determiners in FIG. The inter-frame change amount determination unit 11 includes the following devices.

  The inter-frame change amount calculation unit 111 receives the image signal of the encoding target frame and the image signal of the immediately preceding frame, performs a process of calculating the amount of change in the luminance value from the immediately preceding frame to the encoding target frame, and the calculation result Is written into the inter-frame change amount storage unit 112.

The inter-frame change amount determination unit 113 reads the inter-frame change amount and the threshold value T _d used in the determination process from the inter-frame change amount storage unit 112 and the threshold value storage unit 114, respectively, and the luminance value from the immediately preceding frame to the encoding target frame It is determined whether or not the amount of change exceeds a certain threshold value _Td, and a control signal as a determination result is output.

  The intra-frame average value determiner 12 includes the following devices. The intra-frame average value calculation unit 121 receives the image signal of the encoding target frame, performs a process of calculating the average value of the luminance value of the signal, and writes the calculation result to the intra-frame average value storage unit 122.

The intra-frame average value determination unit 123 reads the intra-frame average value and the threshold values (T _b , T _f ) used in the determination process from the intra-frame average value storage unit 122 and the threshold value storage unit 124, respectively.
(i) The average value of the signal exceeds the threshold value T _f ,
(ii) The average value of the signal is below the threshold value T _b ,
(iii) Other than (i) and (ii) above,
When (i) is satisfied, 1 is output as the control signal of the determination result, 2 is output when (ii) is satisfied, and (iii) is satisfied Is output 3.

  The intra-frame change amount determination unit 13 includes the following devices. The intra-frame variation calculation unit 131 receives the image signal of the encoding target frame, performs a process of calculating the luminance variation in the frame of the signal, and writes the calculation result to the intra-frame variation storage unit 132.

Frame variation determining section 133, from each frame change amount storage unit 132 and the threshold storage unit 134, reads the threshold T _v is used in the amount of change and determination processing frames, frame variation of the signal is fixed threshold T _v performs process of determining whether or not below, and outputs the truth value of the determination result as a control signal.

  The above pre-filtering process can be realized by a computer and a software program, and the program can be provided by being recorded on a computer-readable recording medium or provided through a network.

It is a processing flowchart concerning an embodiment of the invention. It is a block diagram of the encoding apparatus incorporating the pre filter which concerns on embodiment of this invention. It is a block diagram of a pre filter process part. It is a block diagram of an inter-frame variation determination unit, an intra-frame average value determination unit, and an intra-frame variation determination unit.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Prefilter processing part 2 Encoder 3 Multiplexer (MUX)
11 Inter-frame change amount determiner 12 In-frame average value determiner 13 In-frame change amount determiner 14 OR circuit (OR)
DESCRIPTION OF SYMBOLS 15 White monochrome image coding bit stream memory | storage part 16 Black monochrome image coding bit stream memory | storage part SW1-SW5 switch 111 Inter-frame variation calculation part 112 Inter-frame variation memory | storage part 113 Inter-frame variation determination part 121 Intra-frame average value Calculation unit 122 Intra-frame average value storage unit 123 In-frame average value determination unit 131 In-frame change amount calculation unit 132 In-frame change amount storage unit 133 In-frame change amount determination unit 114, 124, 134 Threshold storage unit

Claims (5)

A moving image encoding method for encoding an image signal by performing pre-filtering on a frame with a luminance change,
For the encoding target frame, the amount of change in luminance between the encoding target frame and the previous frame is greater than a predetermined threshold and is used to detect whether or not the frame includes low luminance pixels . Determining a luminance change detection condition that an average luminance value in the target frame is smaller than a predetermined threshold and a luminance change amount in the encoding target frame is smaller than a predetermined threshold ;
And when the encoding target frame is determined in the detection condition is determined to have reached including low luminance pixel, and Luz step to convert the coding target frame into a black monochrome image signal,
And a step of encoding the converted image signal. In the moving image encoding method according to claim 1 ,
In the step of encoding the converted image signal,
A moving picture coding method comprising: reading a coded bit stream that is created in advance as a coding result for the signal and stored in a storage unit; A moving image encoding device that encodes an image signal by performing pre-filtering on a frame with a luminance change,
For the encoding target frame, the amount of change in luminance between the encoding target frame and the previous frame is greater than a predetermined threshold and is used to detect whether or not the frame includes low luminance pixels . Means for determining a detection condition for a luminance change in which an average luminance value in the target frame is smaller than a predetermined threshold and a luminance change amount in the encoding target frame is smaller than a predetermined threshold ;
Storage means for storing an encoded bitstream indicating a black monochrome image created in advance as an encoding result for an encoding target frame determined to have included low-luminance pixels;
Means for reading out an encoded bit stream stored in the storage means and making it an encoding result when it is determined by the determination of the detection condition that the encoding target frame includes low-luminance pixels; A moving picture encoding apparatus comprising: A moving picture coding program for causing a computer to execute the moving picture coding method according to claim 1 . A computer-readable recording medium on which a moving image encoding program for causing a computer to execute the moving image encoding method according to claim 1 or 2 is recorded.

Images