JP2009017107A - Video encoder and video encoding method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain high-efficiency encoding at zooming. <P>SOLUTION: The video encoder adaptively selects the size of a unit area from among the sizes of the plurality of kinds of unit areas, calculates the movement information for each selected unit area, and encodes video data through the use of the calculated movement information. While imaging equipment is performing zooming operation, it selects a unit area smaller than that when zooming operation is not performed. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a moving image encoding apparatus, a moving image encoding method, and a program, and more particularly to a technique suitable for use in improving the encoding efficiency of a moving image encoding apparatus mounted on a digital camera or a digital video camera. .

  In order to effectively use digital moving image data having an enormous amount of data, it is necessary to efficiently store and transmit it, and compression encoding of moving image data is indispensable. As a method for compressing moving image data, first, block division is performed on image data to be compressed, as used in MPEG-2 (Motion Picture Experts Group Phase 2), which is an international standard for moving image coding. . Then, motion information in the block is detected. Inter-frame predictive coding that effectively reduces the amount of code using the correlation between temporally different frames, such as performing orthogonal transform, quantization, and entropy coding after motion compensation using motion information The method is generally used.

  FIG. 9 shows a conceptual diagram of inter-frame prediction. For example, the current frame is divided into 16 × 16 blocks (hereinafter referred to as macroblocks), and a motion vector is searched for each macroblock using the macroblock as a processing target area. Specifically, using the reconstructed video data as a reference frame, a region at a position corresponding to the processing target region of the current frame in the reference frame is used as a reference, and a surrounding pixel group including the region is set as a motion search range. Then, the relative position of the region estimated to have the smallest code amount due to the prediction error of the orthogonal transform coefficient is obtained. Note that the code amount due to the prediction error of the orthogonal transform coefficient is estimated using block matching calculation.

  This relative position can be obtained by comparing the processing target macroblock of the current frame with a macroblock in a reference frame within the motion search range (hereinafter referred to as a reference macroblock) while shifting. In block matching operations, the sum of absolute differences (SAD) of pixels at positions corresponding to each other in a macroblock is generally used, and it is estimated that the smaller the SAD value, the higher the correlation and the smaller the code amount. The

  In recent years, H.264 has been newly established. In the H.264 moving image standard encoding method, in order to further improve the encoding efficiency, a method of adaptively changing the size of a unit area (hereinafter simply referred to as a block size) for motion compensation (hereinafter referred to as a variable block size). ) Is stipulated.

In FIG. 2 shows a conceptual diagram of variable block size motion compensation in the H.264 encoding scheme.
In MPEG-2, only a macro block having a block size of 16 × 16 is defined. On the other hand, H.H. In H.264, block sizes of various sizes and shapes are defined as shown in FIG. 10, and the degree of freedom of motion compensation is increased. If the block size is reduced, the code amount of the orthogonal transform coefficient can be easily reduced, but the number of motion vectors increases, so that the code amount necessary for encoding the motion vector increases.

  On the other hand, when the block size is increased, the number of motion vectors decreases, and even if the amount of motion vector codes can be reduced, it is difficult to reduce the amount of orthogonal transform coefficient codes. In a moving image sequence in which the target object is moving horizontally, if the motion vector search is sufficiently performed, the code amount of the orthogonal transform coefficient can be sufficiently reduced even with a large block size. In this case, the code amount of the motion vector is also reduced, and as a result, high coding efficiency can be realized.

  However, when the moving image sequence includes complex movements that are not simple movements in the horizontal and vertical directions, the code amount of the orthogonal transform coefficient cannot be sufficiently reduced with a large block size. Examples of the complicated movement include a sequence including a rotation operation of a windmill or the like and a sequence in which the target object is deformed. In order to encode such a complex sequence, the code amount of the orthogonal transform coefficient cannot be sufficiently reduced by any motion vector search with a large block size, resulting in a decrease in encoding efficiency. There was a problem.

  In addition, the above-described real-time moving image data encoding apparatus is often mounted on an imaging device such as a digital camera or a digital video camera, and a zoom operation (zoom-in and zoom-out) is characteristic of such an imaging device. ) In a moving image sequence including such a zoom operation, there is a high possibility that the size of the target object and the spatial frequency of the background will change from frame to frame. As a result, there is a problem that the code amount of the orthogonal transform coefficient is increased and the encoding efficiency is lowered.

  In such a difficult video sequence, the code amount of the orthogonal transform coefficient occupies 80% or more of the entire code amount, so the block size is reduced (however, it is acceptable to increase the code amount of the motion vector) and orthogonal transform is performed. It is necessary to reduce the code amount of the coefficient. As described above, in order to realize high encoding efficiency, H.264 is used according to the content of the moving image sequence. It is necessary to effectively select a variable block size defined by H.264 or the like.

  In non-real-time coding, all defined block sizes are exhaustively searched. That is, a motion vector search is performed for each block size, and a block size that obtains the best result is adopted. It is possible to determine the block size by comparing the average value of the prediction error when encoding with a large block size and the prediction error when encoding with a small block size (for example, Patent Document 1).

  However, exhaustive search for all block sizes is extremely computationally intensive and difficult to implement in real-time encoding as required for video cameras, etc. It will increase. Therefore, in real-time encoding, it is necessary to select a block size to some extent in advance according to the characteristics of the moving image sequence. As a conventional method for determining a block size in advance, there is a method using a correlation with surrounding pixel data (see, for example, Patent Document 2).

Japanese Patent No. 2716703 JP 2004-320437 A

  However, the conventional method described above does not take into account changes in the entire scene such as the zoom operation of the camera. For this reason, for example, there is a problem that encoding is performed with a large block size despite zooming, resulting in deterioration of encoding efficiency.

  The present invention has been made in view of the above-described problems, and an object thereof is to realize high encoding efficiency during zooming.

  The moving picture encoding apparatus of the present invention adaptively selects a unit area size from a plurality of types of unit area sizes, calculates motion information for each of the selected unit areas, and calculates the calculated motion information. Is a moving image encoding device that encodes moving image data using an image pickup device, and when a zoom operation of an imaging device is performed, a smaller unit region is selected than when the zoom operation is not performed It is characterized by.

  The moving image encoding method of the present invention adaptively selects a unit area size from a plurality of types of unit area sizes, calculates motion information for each of the selected unit regions, and calculates the calculated motion information. A method of selecting a smaller unit area when a zoom operation of an imaging device is performed than when a zoom operation is not performed It is characterized by having.

  A program according to the present invention causes a computer to execute each step of the moving picture encoding method described above.

  ADVANTAGE OF THE INVENTION According to this invention, while reducing the code amount of an orthogonal transformation coefficient, encoding efficiency can be improved.

(First embodiment)
FIG. 1 is a block diagram illustrating a configuration example of a moving image encoding apparatus according to the present embodiment.
As illustrated in FIG. 1, the moving image encoding apparatus includes a subtractor 101, an orthogonal transformer 102, a quantizer 103, a scan processor 104, an entropy encoder 105, and an inverse quantizer 106. In addition, an inverse orthogonal transformer 107, an adder 108, a frame memory 109, a block matching calculator 110, a reference region generator 111, a block size controller 112, a motion vector controller 113, a memory address generator 114, and an activity extractor 115.

  In FIG. 1, a subtracter 101 subtracts a reference region output from the reference region generator 111 from a processing target region in interframe prediction encoding in which prediction is performed from temporally different frames. The prediction error of the subtraction result is output to the orthogonal transformer 102. The orthogonal transformer 102 performs orthogonal transformation on the input data from the subtractor 101 in units of blocks (this block does not necessarily match the block for motion compensation), and outputs the orthogonal transformation coefficient to the quantizer 103. To do.

  The quantizer 103 quantizes the orthogonal transform coefficient using the quantization table value corresponding to the position in the block and the quantization scale value of the block, and outputs the quantized coefficient to the scan processor 104 and the inverse quantizer 106. The scan processor 104 performs scan processing such as zigzag scanning according to the encoding mode. The entropy encoder 105 performs entropy encoding on the output from the scan processor 104 and outputs the result as a code.

  Here, in the moving picture coding system shown in FIG. 1, in order to perform motion vector detection and motion compensation, a local decoding process is performed using an inverse quantizer 106 and an inverse orthogonal transformer 107. The inverse quantizer 106 performs inverse quantization of the quantized orthogonal transform coefficient of the block using the quantization scale value of the block, and outputs the inverse quantized coefficient to the inverse orthogonal transformer 107. The inverse orthogonal transformer 107 performs inverse orthogonal transform on the inversely quantized orthogonal transform coefficient in units of the block, and outputs the decoded prediction error to the adder 108.

  The adder 108 adds the prediction value output from the reference region generator 111 and the decoded prediction error from the inverse orthogonal transformer 107. The decoded reconstructed image data is stored in the frame memory 109.

  The block size controller 112 and the motion vector controller 113 calculate a block size and a candidate motion vector, respectively, in order to perform processing of the next processing target area. Then, the data is output to the block matching calculator 110 and the memory address generator 114, respectively. Here, the block size controller 112 uses the activity information of the processing target area obtained from the activity extractor 115 and the zoom control signal input from the outside of the moving image encoding device. Then, a smaller block size is selected when the zoom operation is being performed.

  The memory address generator 114 generates a memory address for reading a reference area corresponding to the block size and the motion vector, and outputs the memory address to the frame memory 109. Then, the reference area read from the frame memory 109 is output to the block matching calculator 110, and block matching calculation such as SAD is performed.

  The block matching calculator 110 outputs the motion vector finally determined from the calculated block matching result to the reference region generator 111. Then, the reference area generator 111 outputs a reference area for performing motion compensation to the subtractor 101.

FIG. 3 is a block diagram illustrating a configuration example of a digital video camera system in which the moving image encoding apparatus illustrated in FIG. 1 is mounted.
As illustrated in FIG. 3, the digital video camera system includes a lens 301, an image sensor 302, a signal processing circuit 303, a storage device 304, a moving image encoding device 305, a recording medium 306, and a lens control unit 307.

  An image input from the object to be imaged is input to the image sensor 302 through the lens 301 and converted into an image signal. The converted video signal is subjected to color space conversion, noise removal, and the like in the signal processing circuit 303. The video signal output from the signal processing circuit 303 is temporarily stored in the storage device 304 in order to adjust the processing speed. The video signal read from the storage device 304 is input to the moving image encoding device 305 and output as encoded data. The output encoded data is recorded on a recording medium 306 such as a flash memory.

  The lens control unit 307 outputs a lens control signal to the lens 301 in order to perform a zoom operation based on user operation information. By inputting this lens control signal to the moving image encoding device 305 simultaneously with the lens 301, it becomes possible to adaptively determine the block size according to the presence or absence of the zoom operation. As a result, it is possible to reduce the situation where a large block size is selected despite the zoom operation being performed, and to improve the encoding efficiency.

FIG. 4 is a flowchart illustrating an example of a processing procedure performed by the moving image encoding device 305 according to the present embodiment.
First, in step S401, the block size controller 112 of the video encoding device 305 determines whether or not a zoom operation is being performed from the input zoom control signal when performing inter-frame predictive encoding.

  If the result of this determination is that a zoom operation is being performed, only unit areas (block sizes) of 8 × 8 pixels and 4 × 4 pixels, which are small block sizes, are selected in step S402. In step S403, a motion vector search is performed for each selected block size. For this motion vector search, various algorithms such as a full search method, a step search method, and a tracking method have been proposed, but any motion vector search algorithm may be used in this embodiment.

  On the other hand, if the result of determination in step S401 is that zooming has not been performed, in step S405, the activity of the processing target area extracted by the activity extractor 115 shown in FIG. 1 is determined.

  As a result of this determination, when the activity of the processing target area is low, the code amount of the orthogonal transform coefficient originally possessed by the processing target area is small. That is, the code amount of the orthogonal transform coefficient can be reduced more, but the code amount of the motion vector increases, so that motion compensation with a small block size is not necessary. Accordingly, the process proceeds to step S405, and motion compensation is performed with a large block size of 16 × 16 pixels. Then, the process proceeds to step S403.

  On the other hand, if the activity of the processing target area is high as a result of the determination in step S404, the code amount of the orthogonal transform coefficient originally possessed by the area is large. Therefore, the code amount of the orthogonal transform coefficient is obtained by motion compensation using a small block size. Need to be reduced. Accordingly, the process proceeds to step S406, and motion compensation is performed with block sizes of 8 × 8 pixels and 4 × 4 pixels. Then, the process proceeds to step S403.

(Second Embodiment)
FIG. 5 is a flowchart illustrating an example of a processing procedure performed by the video encoding apparatus according to the present embodiment. In the present embodiment, the configuration of the moving image encoding device and the configuration of the digital video camera system are the same as those in the first embodiment, and thus description thereof is omitted. Also, the same number is assigned to the same process as each step shown in FIG. Therefore, description of the same processing is omitted.

  First, in step S401, the block size controller 112 of the moving image encoding device 305 performs zooming based on the input zoom control signal and the position in the frame of the current processing target region when performing inter-frame predictive encoding. Determine whether an action is taking place. If the result of this determination is that zooming has not been performed, processing proceeds to step S404.

  On the other hand, if the result of determination in step S401 is that a zoom operation is being performed, it is determined in step S510 whether or not the processing target area is close to the zoom center. If the result of this determination is that it is close to the center of the zoom, processing proceeds to step S404. On the other hand, if the result of determination in step S510 is that the zoom is far from the center of the zoom, processing proceeds to step S402, and 8 × 8 pixels and 4 × 4 pixels, which are small block sizes, are allocated. This is because when the zoom operation is being performed, the further away from the zoom center, the faster the movement, and as a result, the code amount of the orthogonal transform coefficient tends to increase.

  In this embodiment, it is possible to realize high speed and improvement of encoding efficiency during zooming by allocating motion compensation with a small block size that requires a large amount of processing, particularly to a peripheral portion with fast motion. Become.

(Third embodiment)
FIG. 6 is a flowchart illustrating an example of a processing procedure performed by the moving image encoding apparatus according to the present embodiment. In the present embodiment, the configuration of the moving image encoding device and the configuration of the digital video camera system are the same as those in the first embodiment, and thus description thereof is omitted. Further, the same processes as those in each step shown in FIG. Therefore, description of the same processing is omitted.

  If the result of determination in step S510 is that the processing target area is far from the zoom center, in step S611, the relative position of the processing target area from the zoom center is determined. Thereby, the optimal block size is determined according to the relative position. This is because unique image processing is performed during the zoom operation. When the zoom operation is performed, a filtering process using a radial low-pass filter is performed on the image input to the moving image encoding device 305 illustrated in FIG. 3 due to the influence of the signal processing circuit 303.

FIG. 7 is a diagram showing a direction in which the spatial correlation becomes strong when the zoom center coincides with the center of the frame.
When the above-described radial filtering process is performed, the frequency in the radial direction becomes low (correlation becomes strong) as shown in FIG. For example, when the processing target region is located directly below the zoom center, the correlation in the vertical direction becomes strong.

  As a result of the determination in step S611, if the horizontal position of the zoom center is close to the horizontal position of the processing target area in this way, in step S612, the vertical and horizontal blocks of 16 × 8 pixels and 8 × 4 pixels are used. Perform motion compensation by size. In this way, by performing motion compensation using a block that is long in the vertical direction, it is possible to perform coding with a small number of motion vectors while sufficiently reducing the code amount of the orthogonal transform coefficient using the correlation in the vertical direction. It becomes possible.

  On the other hand, if the result of determination in step S611 is that the vertical position of the zoom center is close to the vertical position of the processing target area, in step S613 vertical and horizontal are 8 × 16 pixels and 4 × 8 pixels. Perform motion compensation with block size. On the other hand, if it is determined in step S611 that none of the above applies, the process proceeds to step S402. As described above, the block size as shown in FIG. 8 is determined for each region of the peripheral portion in the frame in accordance with the relative position from the zoom center.

  FIG. 8 shows the block size selected in each region in the frame, and the block size is determined for the region close to the center (white portion) in the same manner as the center portion in the second embodiment.

  By limiting the block size to be used by utilizing the radial filtering characteristic specific to the zoom operation as in the present embodiment, it is possible to realize both high speed and improvement in encoding efficiency during zooming. . Therefore, the user can enjoy the advantage that a larger screen image can be taken for a longer time due to the effects of higher speed and improved encoding efficiency.

  FIG. 2 is a diagram showing a comparison of encoding results for each of a moving image sequence in which the entire screen moves in the horizontal direction without zooming and two types of moving image sequences including zoom-out. Specifically, when encoding is performed only with a large block size of 16 × 16 pixels, four types of block sizes of 16 × 16 pixels, 16 × 8 pixels, 8 × 16 pixels, and 8 × 8 pixels are used. It is the graph which compared the encoding result in the case of performing encoding using.

  The bar shown in FIG. 2 indicates the average generated code amount of the P frame when encoding is always performed with the same quantization value. On the other hand, the broken line shows the ratio of how much the code amount can be reduced by adding three types of block sizes. Here, all of the target moving image sequences are VGA size (640 × 480 pixels) and 30 frames per second.

  As shown in FIG. 2, in the moving image sequence not including the zoom, the reduction was about 3%, but in the moving image sequence including the zoom out, the reduction was about 10%. It can be seen that the code amount can be greatly reduced in the moving image sequence including zoom-out. This is because when zoomed out, even if there is a part with a low spatial frequency in the screen, it is reduced and the spatial frequency becomes high. This is because the amount increases.

  There is also a new area that appears when zooming out. This is because even in such a region, even if an increase in the code amount of the motion vector is taken into account, it is possible to reduce the code amount of the orthogonal transform coefficient and improve the encoding efficiency by making the block size small. .

(Other embodiments according to the present invention)
Each means constituting the moving picture coding apparatus and each step of the moving picture coding method in the embodiment of the present invention described above can be realized by operating a program stored in a RAM or ROM of a computer. This program and a computer-readable recording medium recording the program are included in the present invention.

  Further, the present invention can be implemented as, for example, a system, apparatus, method, program, or recording medium. Specifically, the present invention may be applied to a system including a plurality of devices. The present invention may be applied to an apparatus composed of a single device.

  The present invention includes a case where a software program for realizing the functions of the above-described embodiments (in the embodiment, a program corresponding to the flowcharts shown in FIGS. 4 to 6) is directly or remotely supplied to the system or apparatus. . This includes the case where the system or the computer of the apparatus is also achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Examples of the recording medium for supplying the program include a flexible disk, a hard disk, an optical disk, and a magneto-optical disk. Further, there are MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R) and the like.

  As another program supply method, there is a method of connecting to a homepage on the Internet using a browser of a client computer. The computer program itself of the present invention or a compressed file including an automatic installation function can be downloaded from the homepage by downloading it to a recording medium such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  As another method, the program of the present invention is encrypted, stored in a recording medium such as a CD-ROM, distributed to users, and encrypted from a homepage via the Internet to users who have cleared predetermined conditions. Download the key information to be solved. It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. Furthermore, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can be realized by the processing.

  As another method, the program read from the recording medium is first written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Then, based on the instructions of the program, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are also realized by the processing.

It is a block diagram which shows the structural example of the moving image encoder in the 1st Embodiment of this invention. It is a figure which shows the comparison of an encoding result about each of the moving image sequence which the whole screen moves in a horizontal direction not including zoom, and two types of moving image sequences including zoom out. FIG. 2 is a block diagram illustrating a configuration example of a digital video camera system including the moving image encoding apparatus illustrated in FIG. 1 in the first embodiment of the present invention. It is a flowchart which shows an example of the process sequence by the moving image encoder of the 1st Embodiment of this invention. It is a flowchart which shows an example of the process sequence by the moving image encoder of the 2nd Embodiment of this invention. It is a flowchart which shows an example of the process sequence by the moving image encoder of the 3rd Embodiment of this invention. It is a figure which shows the direction where a spatial correlation becomes strong when the center of zoom corresponds with the center of a flame | frame. It is a figure which shows the block size selected in each area | region in a flame | frame. It is a conceptual diagram which shows the detail of inter-frame prediction. H. 2 is a conceptual diagram illustrating variable block size motion compensation in the H.264 encoding scheme. FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Subtractor 102 Orthogonal transformer 103 Quantizer 104 Scan processor 105 Entropy encoder 106 Inverse quantizer 107 Inverse orthogonal transformer 108 Adder 109 Frame memory 110 Block matching calculator 111 Reference area generator 112 Block size control 113 Motion vector controller 114 Memory address generator 115 Activity extractor

Claims (5)

The size of the unit region is adaptively selected from the sizes of the plurality of types of unit regions, motion information is calculated for each of the selected unit regions, and moving image data is encoded using the calculated motion information. A video encoding device,
A moving picture coding apparatus, wherein when a zoom operation of an imaging device is performed, a smaller unit area is selected than when a zoom operation is not performed.   The moving image encoding according to claim 1, wherein when the zoom operation is performed, a smaller unit region is selected in a region far from the zoom center than in a region near the zoom center. apparatus.   The moving image encoding apparatus according to claim 1, wherein when the zoom operation is being performed, selection is made according to a relative position of each region with respect to the zoom center. The size of the unit region is adaptively selected from the sizes of the plurality of types of unit regions, motion information is calculated for each of the selected unit regions, and moving image data is encoded using the calculated motion information. A video encoding method comprising:
A moving image encoding method comprising a step of selecting a unit region smaller when the zoom operation of the imaging device is performed than when the zoom operation is not performed.   A program for causing a computer to execute each step of the moving picture encoding method according to claim 4.

Images