JP5171675B2 - Image processing apparatus and imaging apparatus equipped with the same - Google Patents

Description

  The present invention relates to an image processing apparatus for encoding a moving image and an imaging apparatus using the same.

  In recent years, digital movie cameras capable of shooting moving images have become widespread. Digital movie cameras have improved image quality year by year, and those compatible with full HD (High Definiton) image quality have also been put into practical use. Accordingly, H.H. has high image compression efficiency. Digital movie cameras that compress and code moving images according to the H.264 / AVC standard have also been put into practical use.

  In compression encoding and decoding of a moving image, moving image frames are stored in a frame memory, and motion compensation is performed with reference to the frame memory. Therefore, data transfer from the frame memory is frequently performed. In particular, in order to generate a higher-quality moving image, motion search may be performed in units of decimal pixels, and the amount of data handled by motion compensation increases. For this reason, memory bandwidth tends to be a bottleneck for processing.

  Patent Document 1 discloses a digital image decoding apparatus that can improve the bandwidth use efficiency of a frame memory.

JP 11-298903 A

  When performing compression encoding of a moving image, in order to detect motion of the target macroblock of the encoding target image frame, a pixel area within a predetermined search range of the reference image is read from the frame memory, and the target macro is read within the pixel area. Search for a macroblock that matches the block. At this time, if the motion of the target macroblock is predicted in advance and the search range is set so as to include the position where the target macroblock is predicted to exist in the reference image, motion detection can be performed efficiently. However, when the predicted motion is large, a large search range must be set, the number of times of reading from the frame memory increases, the amount of data transfer increases, and the transfer bandwidth of the frame memory is reduced. For this reason, access to the frame memory becomes a bottleneck, which causes a problem that the processing speed of compression encoding decreases.

  The present invention has been made in view of such circumstances, and provides an image processing apparatus capable of reducing the access to the frame memory and increasing the processing efficiency of the encoding, and an imaging apparatus equipped with the image processing apparatus. Objective.

  One embodiment of the present invention is an image processing apparatus. The image processing apparatus includes: a first detection unit that detects a motion vector for each unit region of a target picture; and a global motion vector in the target picture or a region defined in the target picture from the motion vector. A second detection unit to detect, and when the size of the global motion vector is larger than a predetermined threshold, in a reference picture to be referred to when detecting the motion vector, a target unit region for detecting the motion vector The first search range is set around the same position as the position, and the first search range is set around the destination position of the target unit region predicted from the global motion vector. An image processing apparatus comprising: a control unit that controls the detection unit.

  In addition, when the magnitude of the motion vector is smaller than a predetermined threshold, the control unit sets a third search range around the same position as the position of the target unit region in the reference picture. It is preferable to control the first detection unit.

  Moreover, it is preferable that the said control part prescribes | regulates the said threshold value based on a magnitude | size to the said 3rd search range.

  According to the present invention, it is possible to improve the processing efficiency of motion detection of moving images.

1 is a conceptual diagram illustrating a configuration of an imaging device 1 according to an embodiment of the present invention. Example of search range set when global motion vector is smaller than threshold Example of search range set when global motion vector is greater than threshold Flow chart for determining a search range by a global motion vector

  The outline will be described before the present invention is specifically described. An image processing apparatus according to an embodiment of the present invention is described in H.264. Compressive encoding conforming to the H.264 / AVC standard is performed.

  H. In the H.264 / AVC standard, an image frame to be subjected to intra-frame predictive coding is an I (Intra) frame, an image frame to be subjected to forward inter-frame predictive coding with a past frame as a reference image, a P (Predictive) frame, An image frame for which inter-frame predictive coding is performed regardless of before and after and regardless of the number of frames that can be used as a reference image is referred to as a B (Bi-predictive) frame.

  Here, in this specification, a frame and a picture are used in the same meaning, and an I frame, a P frame, and a B frame are also referred to as an I picture, a P picture, and a B picture, respectively. In the present specification, a frame is taken as an example, but the unit of encoding may be a field.

  In the case of inter-screen predictive coding, a predetermined search range is set in a reference image that is referred to in order to detect the motion of the target macroblock in the encoding target image frame, and the macroblock that matches the target macroblock within the search range Explore. The search range is generally set around the same position as the position of the target macroblock in the reference image. If the motion of the target macroblock can be predicted to some extent, there is a high probability of matching around the predicted movement destination position of the target macroblock in the reference image. Setting the range enables efficient motion detection.

  At this time, if the target macroblock is predicted to move violently, it is necessary to increase the search range. However, if a large search range is set, access to the frame memory becomes a bottleneck, and compression encoding processing is performed. The speed is reduced.

  Therefore, the image processing apparatus according to the embodiment of the present invention detects a motion vector for each target macroblock, calculates a global motion vector that represents the motion of the entire global region including the target macroblock, and the global motion vector is When larger than the predetermined threshold, a search range (hereinafter referred to as a first search range as appropriate) is set around the same position as the target macroblock in the reference image, and separately from the first search range, In the reference image, a search range (hereinafter, appropriately referred to as a second search range) is set around the position of the movement destination of the target macroblock predicted from the global motion vector.

  Thus, even without setting a large search range, the search range can include the periphery of the same position as the position of the target macroblock in the reference image and the vicinity of the predicted movement destination position of the target macroblock. A motion vector can be detected at high speed and with high accuracy.

  FIG. 1 is a conceptual diagram illustrating a configuration of an imaging apparatus 1 according to an embodiment of the present invention. The imaging device 1 includes an imaging unit 10 and an image processing device 20.

  The imaging unit 10 acquires a moving image and supplies it to the image processing device 20. The imaging unit 10 is a solid-state imaging device such as a CCD (Charge Coupled Devices) sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or a signal processing unit that changes an analog three primary color signal from the solid-state imaging device to a digital luminance signal and a color difference signal. including.

  The image processing apparatus 20 uses a moving image acquired from the imaging unit 10 as, for example, H.264. An encoded stream is generated by compression encoding according to the H.264 / AVC standard, and output to a recording unit (not shown) (for example, HDD (Hard Disk Drive), optical disk, flash memory, etc.).

  The image processing apparatus 20 includes a difference unit 201, an orthogonal transformation unit 202, a quantization unit 203, a variable length coding unit 204, an output buffer 205, an inverse quantization unit 206, an inverse orthogonal transformation unit 207, an addition unit 208, a local motion vector. A detection unit 209, a motion compensation unit 210, a frame memory 211, a global motion vector calculation unit 212, and a control unit 213 are included.

  If the image frame input from the imaging unit 201 is an I frame, the difference unit 201 outputs it to the orthogonal transformation unit 202 as it is. If the frame is a P frame or a B frame, the difference from the predicted image input from the motion compensation unit 210 is calculated and output to the orthogonal transform unit 202.

  The orthogonal transform unit 202 performs discrete cosine transform (DCT) on the difference image input from the difference unit 201 and outputs DCT coefficients to the quantization unit 203.

  The quantization unit 203 quantizes the DCT coefficient input from the orthogonal transform unit 202 with reference to a predetermined quantization table, and outputs it to the variable length coding unit 204. The quantization table defines a quantization scale on which each DCT coefficient is to be divided. In the quantization table, the quantization scale corresponding to the low frequency component of the DCT coefficient is set small, and the quantization scale corresponding to the high frequency component is set large. Thereby, omission is enlarged as the high frequency component. The quantization unit 203 also outputs the quantized DCT coefficient to the inverse quantization unit 206.

  The variable length encoding unit 204 entropy encodes the quantized DCT coefficient input from the quantization unit 203, the motion vector detected by the motion vector detection unit 209, and other parameters, and outputs the result to the output buffer 205.

  The output buffer 205 temporarily stores the encoded stream input from the variable length encoding unit 204, and outputs the encoded stream to a recording unit (not shown) at a predetermined timing or sends it to a network.

  The inverse quantization unit 206 performs inverse quantization on the quantized DCT coefficient input from the quantization unit 203 and outputs the quantized DCT coefficient to the inverse orthogonal transform unit 207.

  The inverse orthogonal transform unit 207 restores the image frame by performing inverse discrete cosine transform on the inverse quantized DCT coefficient input from the inverse quantization unit 206.

  If the image frame input from the inverse orthogonal transform unit 207 is an I frame, the adding unit 208 stores the image frame in the frame memory 211 as it is. If it is a P frame or a B frame, it is a difference image, so the difference image input from the inverse orthogonal transform unit 207 and the prediction image input from the motion compensation unit 210 are added to re-create the original image frame. Configured and stored in the frame memory 211.

  The local motion vector detection unit 209 detects a motion vector in units of macroblocks (hereinafter, referred to as a local motion vector as appropriate) by block matching. For this reason, the local motion vector detection unit 209 searches for a macroblock of a reference image having the smallest error with respect to the target macroblock (hereinafter, appropriately referred to as a prediction macroblock). At this time, the local motion vector detection unit 209 stores pixel data within a predetermined search range of the reference image held in the frame memory 211 in a cache memory (not shown) while referring to the image data stored in the cache memory. Perform block matching.

  As an example of block matching, the local motion vector detection unit 209 calculates the sum of absolute differences of pixel data between a target macroblock and a macroblock that is a candidate for a predicted macroblock, and the value is the minimum. This macroblock is a predicted macroblock. A vector indicating the motion from the position of the target macroblock to the position of the detected predicted macroblock is the local motion vector of the target macroblock. The local motion vector detection unit 209 outputs the detected local motion vector to the variable length coding unit 204, the motion compensation unit 210, and the global motion vector calculation unit 212.

  The motion compensation unit 210 performs motion compensation for each macroblock of the P frame or the B frame using the local motion vector input from the local motion vector detection unit 209, generates a prediction image, and the difference unit 201 and the addition unit 208 Output to.

  The global motion vector calculation unit 212 sets the entire screen of the encoding target image frame or the specific region of the encoding target image frame as a global region, and indicates a global motion in the global region (hereinafter referred to as a global motion vector as appropriate). Ask for. The global motion vector is a vector that represents the motion of the entire global region.

  The specific area may be a predetermined area such as the vicinity of the center of the encoding target image frame, or may be an arbitrary area of the encoding target image frame set by the user. When an object such as a person is shown in the encoding target image frame, it may be a region where the object is shown, or a region occupied by macroblocks that share some motion.

  The global motion vector calculation unit 212 calculates a global motion vector for the global region based on the local motion vector input from the local motion vector detection unit 209. For example, the global motion vector calculation unit 212 obtains an average of local motion vectors in the global region and sets it as the global motion vector. In addition, a difference between the global motion vector calculated last time and a local motion vector for each macroblock in the global region may be obtained, and the previously calculated global motion vector may be corrected according to the difference value. The global motion vector calculation unit 212 outputs the calculated global motion vector to the control unit 213.

  The control unit 213 controls the entire image processing apparatus. Further, the control unit 213 sets a search range for detecting a local motion vector in the reference image based on the global motion vector input from the global motion vector calculation unit 212.

  Since the global motion vector is a motion vector representing the motion of the entire global region, when the size of the global motion vector is small, it is predicted that the motion of the macroblock included in the global region is not so intense. If the movement of the macroblock is not so intense, the probability that matching can be achieved within a narrow range centering on the target macroblock is also increased. Therefore, when the magnitude of the global motion vector is smaller than a predetermined threshold, the control unit 213 searches for a search range around the same position as the target macroblock in the reference image (hereinafter, referred to as a third search range as appropriate). Set.

  FIG. 2 shows an example of the search range set in the reference image when the global motion vector is smaller than a predetermined threshold value. In the reference image, a macroblock at the same position as the position of the target macroblock of the encoding target image frame is indicated by a hatched macroblock 30 in the drawing. The third search range 32 is indicated by a dashed-dotted square area surrounding the macroblock 30. Here, as an example, the third search range 32 is a square of 6 vertical macroblocks × 6 horizontal macroblocks.

  When the size of the global motion vector is large, the macroblock included in the global region is predicted to be largely moving. When the correlation between the motion vector of the target macroblock and the global motion vector is strong, there is a probability that matching can be achieved in a range centered on a position separated from the position of the target macroblock in the reference image by the global motion vector. Get higher.

  At this time, if a large search range is set around the target macroblock according to the size of the global motion vector, the probability of matching within the search range can be increased. However, local motion vector detection from the frame memory 211 is possible. The transfer load when transferring image data to the unit 209 increases.

  Therefore, the control unit 213 sets a search range (first search range) around the same position as the position of the target macroblock in the reference image, and from the global motion vector in the reference image separately from the first search range. A search range (second search range) is set around the predicted position of the target macroblock.

  FIG. 3 shows an example of the search range set in the reference image when the global motion vector is larger than a predetermined threshold value. In the reference image, a macroblock at the same position as the position of the target macroblock of the encoding target image frame is indicated by a macroblock 34 with a hatched pattern in the drawing. In the reference image, the macro block at the position of the movement destination of the target macro block predicted from the global motion vector is indicated by a macro block 38 with a lattice pattern.

  In FIG. 3, the size of the global motion vector is equivalent to 10 macroblocks, and the direction is the left horizontal direction. The macroblock 38 is positioned at the position where the global motion vector is added to the coordinate position of the macroblock 34 as a vector. doing. That is, the position of the movement destination of the target macroblock in the reference image is predicted by adding the global motion vector as a vector starting from the same position as the position of the target macroblock in the reference image.

  In FIG. 3, the first search range 36 is indicated by a dashed-dotted rectangular region surrounding the macroblock 34, and the second search range 40 is indicated by a dashed-dotted rectangular region surrounding the macroblock 38.

  In FIG. 3, the first and second search ranges are rectangles of 3 vertical macroblocks × 6 horizontal macroblocks. That is, in FIG. 3, the third search range in FIG. 2 is divided into two vertically, and one is set as the first search range and the other is set as the second search range. For this reason, the transfer load when transferring image data within the first and second search ranges is equal to the transfer load when transferring image data within the third search range. Therefore, when the size of the global motion vector is large, an increase in the amount of image data transfer can be suppressed even if the search range includes the vicinity of the position of the movement destination of the target macroblock predicted from the global motion vector.

  In addition, since the second search range is set around the position of the movement destination of the target macroblock predicted from the global motion vector in the reference image, when the correlation with the global motion vector is strong, the second search range The probability of matching is high, and the search efficiency of local motion vectors is improved.

  Furthermore, even if the prediction accuracy is poor and mismatching occurs in the second search range, the first search range is set around the same position as the target macroblock position in the reference image. Even if there is a possibility that matching can be performed within one search range and the correlation with the global motion vector is weak, it is possible to prevent deterioration in the detection accuracy of the local motion vector.

  In the above description, the third search range is divided into upper and lower parts, and one is set as the first search range and the other is set as the second search range. However, the size of the first search range and the second search range are used. The third search range may be divided so that the size of the range is different. In addition, a value obtained by adding the sizes of the first search range and the second search range may be smaller than the size of the third search range.

  FIG. 4 is a flowchart showing a procedure of a search range setting method based on the magnitude of the global motion vector. The global motion vector calculation unit 212 receives the detected local motion vector LMn from the local motion vector detection unit 209 for the image frame Fn to be encoded at the time Tn. Based on the received local motion vector LMn, the global motion vector calculation unit 212 sets the search range for the reference image of the image frame Fn + 1 to be encoded at the time Tn + 1, in order to set the global motion vector GMn + 1. Is calculated and supplied to the controller 213 (S10).

  When the image frame Fn + 1 becomes the encoding target image frame at the time point Tn + 1, the control unit 213 compares the global motion vector GMn + 1 with the threshold value of the global motion vector (S12). Here, the threshold value of the global motion vector can be defined in advance by a simulation or experiment. When the threshold value of the global motion vector is defined based on the size of the third search range, the movement destination position of the target macroblock predicted from the global motion vector is out of the third search range. Moreover, adaptive setting of the search range, which will be described later, can be executed, and an increase in the amount of image data transfer can be suppressed.

  When the magnitude of the global motion vector GMn + 1 is smaller than the threshold value (N in S12), the control unit 213 sets the third search range in the reference image (S14). When the magnitude of the global motion vector GMn + 1 is larger than the threshold value (N in S12), the first search range and the second search range are set in the reference image (S16).

  The local motion vector detection unit 209 transfers the image data within the search range set by the control unit 213 to the cache memory, and detects the local motion vector LMn + 1 for the image frame Fn + 1. The local motion vector detection unit 209 supplies the detected local motion vector LMn + 1 to the global motion vector calculation unit 212. Based on the received local motion vector LMn + 1, the global motion vector calculation unit 212 calculates a global motion vector GMn + 2 to be referred to at time Tn + 2. Hereinafter, the search range is adaptively set while repeating the processing described so far.

  According to the embodiment of the present invention, when the global motion vector is calculated based on the local motion vector and the magnitude of the global motion vector is larger than the threshold, the first search range and the second search are included in the reference image. Since the range is set, it is possible to suppress an increase in the transfer amount of image data referred to when detecting a motion vector. Further, since the second search range is set, the motion vector can be detected efficiently. Furthermore, since the first search range is set, the motion vector can be detected with high accuracy. Therefore, encoding efficiency can be improved.

  As mentioned above, although the form for implementing this invention has been demonstrated, this invention is not limited to the structure of this embodiment, It exists in the application range of this invention prescribed | regulated by the claim. Various modifications are possible as long as the functions of the configuration of the above-described embodiment can be achieved.

  For example, in the embodiment of the present invention, the image processing apparatus 20 is the H.264 standard. The compression encoding is performed according to the H.264 / AVC standard, but the compression encoding may be performed according to a standard such as MPEG-2 or MPEG-4.

201 Difference unit, 202 Orthogonal transformation unit, 203 Quantization unit, 204 Variable length coding unit, 205 Output buffer, 206 Inverse quantization unit, 207 Inverse orthogonal transformation unit, 208 Adder, 209 Local motion vector detection unit, 210 Motion Compensator, 211 frame buffer, 212 global motion vector calculator, 213 controller.

Claims (4)

A first detection unit for detecting a motion vector for each unit area of the target picture; and a second detection for detecting a global motion vector in the target picture or an area defined in the target picture from the motion vector And a reference picture that is referred to when detecting the motion vector, when the magnitude of the global motion vector is larger than a predetermined threshold, the periphery of the same position as the position of the target unit region for detecting the motion vector And setting the first search range, and controlling the first detection unit to set the second search range around the position of the movement destination of the target unit region predicted from the global motion vector , If the size of the global motion vector is smaller than a predetermined threshold value, a third image is located around the same position as the target unit area in the reference picture. And a control unit for controlling the first detector to set the search range, wherein the control unit, the size and value of the sum of the size of the second search range of said first search range An image processing apparatus that is set to be equal to or smaller than the size of the third search range . A first detection unit for detecting a motion vector for each unit area of the target picture; and a second detection for detecting a global motion vector in the target picture or an area defined in the target picture from the motion vector And a reference picture that is referred to when detecting the motion vector, when the magnitude of the global motion vector is larger than a predetermined threshold, the periphery of the same position as the position of the target unit region for detecting the motion vector And setting the first search range, and controlling the first detection unit to set the second search range around the position of the movement destination of the target unit region predicted from the global motion vector, If the size of the global motion vector is smaller than a predetermined threshold value, a third image is located around the same position as the target unit area in the reference picture. And a control unit for controlling the first detector to set the search range, wherein the control unit, characterized in that defining the threshold value based on the magnitude of the third search range An image processing apparatus. Wherein the control unit, an image processing apparatus according to claim 1, characterized in that defining the threshold value based on the magnitude of the third search range. An imaging unit for acquiring a moving image;
The image processing apparatus according to any one of claims 1 to 3, which processes a moving image acquired by the imaging unit;
An imaging apparatus comprising:

Images