JP4998134B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method for detecting a motion vector between two different screens. In this specification, the screen means an image that is composed of image data for one frame or one field and is displayed on the display as one image.

  A block matching method for obtaining a motion vector between two screens from image information itself is an old technology. Development has been centered on pan / tilt detection of TV cameras, subject tracking, moving picture experts group (MPEG) video coding, and sensorless camera shake correction by superimposing images since the 1990s. Applications are being promoted in various ways, such as noise removal during low-light shooting.

  In block matching, a motion vector between two screens between a reference screen that is a target screen and an original screen (referred to as a target screen) that is a source of motion of the reference screen is a block of a rectangular area having a predetermined size. Is calculated by calculating the correlation between the reference screen and the original screen. When the original screen is temporally before the reference screen (for example, in the case of motion detection in MPEG), and when the reference screen is temporally prior to the original screen (for example, described later) Both in the case of noise reduction by superimposing image frames to be performed).

  As described above, in this specification, the screen means an image composed of image data of one frame or one field, but for the convenience of the following description in this specification, the screen is composed of one frame. As an example, the screen is referred to as a frame. Therefore, the reference screen is referred to as a reference frame, and the original screen is referred to as an original frame.

  33 to 38 are diagrams for explaining the outline of conventional block matching. In the block matching method described here, for example, as shown in FIG. 33A, the original frame (target frame) 100 is divided into a plurality of parts in each of the horizontal direction and the vertical direction, and a plurality of horizontal frames are respectively obtained. A rectangular area 102 (referred to as a block) having a predetermined size composed of pixels and a plurality of lines in the vertical direction is divided. Each of the plurality of blocks 102 in the target frame is referred to as a target block.

  In block matching, a block having a high correlation with the target block 102 is searched from the reference frame 101. As a result of this search, the block 103 (see FIG. 33B) detected in the reference frame 101 as having the highest correlation is referred to as a motion compensation block. Further, the amount of positional deviation between the target block 102 and the motion compensation block 103 is referred to as a motion vector (see reference numeral 104 in FIG. 33B).

  A motion vector 104 corresponding to a position shift (including a position shift amount and a position shift direction) between the target block 102 and the motion compensation block 103 is the same as the position of each target block 102 of the target frame 100 in the reference frame 101. Assuming that the projection image block 109 of the target block 102 is assumed as the position, a positional shift between the position of the projection image block 109 of the target block (for example, the center position) and the position of the motion compensation block 103 (for example, the center position). It also has a positional deviation amount and a positional deviation direction component.

  An outline of the block matching process will be described. As shown by a dotted line in FIG. 34, a projected image block 109 of the target block is assumed at the same position as the position of each target block 102 of the target frame 100 in the reference frame 101, and the center of the projected image block 109 of this target block is assumed. The coordinates are set as a motion detection origin 105. Then, assuming that the motion vector 104 exists within a certain range from the motion detection origin 105, a predetermined range centered on the motion detection origin 105 is set as the search range 106 (see the one-dot chain line in FIG. 34). .

  Next, a block (referred to as a reference block) 108 having the same size as the target block 102 is set on the reference screen. The position of the reference block 108 is moved within the search range 106 in units of one pixel or a plurality of pixels, for example, in the horizontal direction and the vertical direction. Accordingly, a plurality of reference blocks 108 are set in the search range 106.

  Here, the reference block 108 is moved within the search range 106. In this example, since the motion detection origin 105 is the center position of the target block, the center position of the reference block 108 is moved within the search range 106. This means that the pixels constituting the reference block 108 may protrude beyond the search range 106.

  Then, for each reference block 108 to be set in the search range, a vector (referred to as a reference vector) 107 (refer to FIG. (B)) representing a positional deviation amount and a positional deviation direction between each reference block 108 and the target block 102. ) And the correlation between the image content of the reference block 108 at the position indicated by each reference vector 107 and the image content of the target block 102 is evaluated.

  As shown in FIG. 35, the reference vector 107 is a vector (Vx, Vy) where the horizontal displacement (X direction) of the reference block 108 is Vx and the vertical displacement (Y direction) is Vy. When the position coordinates (for example, center position coordinates) of the reference block 108 and the position coordinates (for example, center position coordinates) of the target block 102 are the same, the reference vector 107 is represented as a vector (0, 0). .

  For example, when the reference block 108 is at a position shifted by one pixel in the X direction from the position of the target block 102, the reference vector 107 is a vector (1, 0). As shown in FIG. 36, when the reference block 108 is located at a position shifted by 3 pixels in the X direction and 2 pixels in the Y direction from the position of the target block 102, the reference vector 107 is a vector (3, 2). )

  That is, as shown in the example of FIG. 36, when the positions of the target block 102 and the reference block 108 are set to the center positions of the respective blocks, the reference vector 108 has the reference vector 108 of each corresponding reference block 108. This means a displacement between the center position and the center position of the target block 102 (a vector including a displacement amount and a displacement direction).

  The reference block 108 moves within the search range 106. In this case, the center position of the reference block 108 moves within the search range 106. As described above, since the reference block 108 includes a plurality of pixels in the horizontal direction and the vertical direction, the maximum range in which the reference block 108 that is subject to block matching processing with the target block 102 is as shown in FIG. Furthermore, the matching processing range 110 is wider than the search range 106.

  Then, the position of the reference block 108 detected as having the strongest correlation with the image content of the target block 102 is detected as the position of the target block 102 of the target frame 100 in the reference frame 101 (position after moving), The detected reference block is set as the motion compensation block 103 described above. Then, the displacement amount between the detected position of the motion compensation block 103 and the position of the target block 102 is detected as a motion vector 104 as an amount including a direction component (see FIG. 33B). ).

  Here, the correlation value indicating the strength of correlation between the target block 102 and the reference block 108 moving in the search range 106 is basically calculated using the pixel values corresponding to the target block 102 and the reference block 108. However, as the calculation method, a method using a root mean square and various other methods have been proposed.

  Among them, as a correlation value generally used when calculating a motion vector, for example, the absolute value of the difference between the luminance value of each pixel in the target block 102 and the luminance value of each corresponding pixel in the reference block 108 is used. The sum of the values of all the pixels in the block (the sum of the absolute values of the differences is referred to as the sum of absolute values of the difference. Hereinafter, the sum of the absolute values of the differences is referred to as SAD (Sum of Absolute Difference)). Used (see FIG. 37).

  When the SAD value is used as the correlation value, the smaller the SAD value, the stronger the correlation. Therefore, among the reference blocks 108 that move in the search range 106, the reference block 108 at the position where the SAD value is minimum becomes the strongest correlation reference block with the strongest correlation, and this strongest correlation reference block is detected as the motion compensation block 103. The amount of displacement of the detected motion compensation block 103 relative to the position of the target block 102 is detected as a motion vector.

  As described above, in block matching, the positional shift of each of the plurality of reference blocks 108 set in the search range 106 with respect to the position of the target block 102 is expressed by the reference vector 107 that includes a direction component. . The reference vector 107 of each reference block 108 has a value corresponding to the position of the reference block 108 on the reference frame 101. As described above, in block matching, the reference vector of the reference block 108 in which the SAD value that is the correlation value has the minimum value is detected as the motion vector 104.

  Therefore, in block matching, generally, as shown in FIG. 38, SAD values between each of a plurality of reference blocks 108 set in the search range 106 and the target block 102 (hereinafter referred to as reference blocks for the sake of simplicity). The reference vector 107 corresponding to the position of each reference block 108 (hereinafter referred to as the reference vector 107 corresponding to the position of the reference block 106 is referred to as the reference vector 107 of the reference block 108 for the sake of simplicity). By detecting the reference block 108 having the smallest SAD value from the SAD values for all the reference blocks 108 stored in the memory. The vector 104 is detected.

  Corresponding to each of the reference vectors 107 corresponding to the positions of the plurality of reference blocks 108 set in the search range 106, a correlation value (in this example, SAD value) for each reference block 108 is stored. This is called a correlation value table. In this example, since the SAD value which is the sum of absolute differences is used as the correlation value, this correlation value table will be referred to as a difference absolute value sum table (hereinafter referred to as SAD table).

  This is shown in the SAD table TBL of FIG. 38. In this SAD table TBL, the correlation value (SAD value in this example) for each reference block 108 is referred to as a correlation value table element. In the example of FIG. 38, the SAD value indicated by reference numeral 111 is the SAD value when the reference vector is the vector (0, 0). In the example of FIG. 38, since the minimum value of the SAD value indicated by reference numeral 112 is “7” when the reference vector is the vector (3, 2), the motion vector 104 to be obtained is the vector (3, 2). )

  In the above description, the positions of the target block 102 and the reference block 108 mean any specific position of those blocks, for example, the center position, and the reference vector 107 is the target block in the reference frame 101. The amount of deviation (including direction) between the position of the projection image block 109 of 102 and the position of the reference block 108 is shown.

  Since the reference vector 107 corresponding to each reference block 108 is a position shift of each reference block 108 from the position of the projection image block 109 corresponding to the target block 102 on the reference frame 101, the reference block When the position 108 is specified, the value of the reference vector is also specified corresponding to the position. Therefore, when the address of the correlation value table element of the reference block in the memory of the SAD table TBL is specified, the corresponding reference vector is specified.

  Note that the SAD value may be calculated for two or more target blocks at the same time. As the number of target blocks to be processed simultaneously increases, the processing speed increases. However, since the hardware scale for calculating the SAD value increases, there is a trade-off between speeding up the processing and increasing the circuit scale.

  By the way, in the block matching method described above, when the resolution of an image to be processed increases, the number of pixels detected as motion between two images (between two screens) increases as the resolution increases. In order to follow the above, it is necessary to widen the search range of the motion vector (increase the number of pixels included in the search range).

  However, as described above, when the search range is wide, the number of times of reading pixels from the frame memory per block to be processed increases, so that the processing time becomes long.

  In addition, when the resolution of the image to be processed is small or the frame rate is fast, the movement between pixels becomes small, so it is necessary to detect a minute movement of less than one pixel. It was necessary to detect a motion vector of less than one pixel by obtaining a motion vector. However, in such a case, there is a problem in that the circuit scale increases due to oversampling and the processing time becomes long.

  As described above, the block matching method tends to increase the processing time and the circuit scale in response to the demand for a wide search range and a minute motion of less than one pixel, and it has been demanded how to reduce this. It was.

  In recent years, the development of high-definition moving images (hereinafter referred to as HD moving images) has progressed, and demands for image resolution and image quality have increased. Along with this, there has been an increasing demand for realizing both techniques of detecting a small motion of a wide search range and less than one pixel in the block matching method.

  In order to solve these problems, conventionally, a method for improving the efficiency of processing in detecting a minute motion of less than one pixel (see, for example, Patent Document 1 (Japanese Patent Laid-Open No. 7-95585)) or a frame by thinning out a reference image Many methods have been proposed, such as a method for reducing memory and calculation amount (see, for example, Patent Document 2 (Japanese Patent Laid-Open No. 2006-160829)).

  Among them, the most practical method is to use the SAD table minimum SAD value position and a plurality of SAD values in the vicinity thereof, and to improve the accuracy of the SAD table, that is, the pixel pitch accuracy in the target frame and the reference frame. This is a technique for performing an interpolation process so as to calculate the minimum value of the SAD value with a small and high accuracy (see, for example, Patent Document 3 (Japanese Patent Laid-Open No. 05-91492)). Impact is small.

The above-mentioned patent documents are as follows.
JP-A-7-95585 JP 2006-160829 A Japanese Patent Laid-Open No. 05-91492

  However, in the above-described conventional technique, the problem of interpolation accuracy, the capacity of the SAD table, and the like for the block matching circuit that is actually an image with high resolution such as HD moving images and that requires high accuracy in a wide search range. Etc., was not practical in terms of.

  In the block matching process, the memory capacity required for executing the process, such as the capacity of the SAD table, can be reduced, sufficient interpolation accuracy can be secured, and the processing speed can be increased. An object of the present invention is to provide an image processing apparatus and method capable of realizing reduction and the like to a practically satisfactory level.

In order to solve the above-described problem, an image processing apparatus according to the present invention is configured such that a reference block having the same size as a target block having a predetermined size composed of a plurality of pixels set in a target screen is the target screen. A plurality of search ranges set in different reference screens are set, and the reference block having the strongest correlation with the target block among the plurality of reference blocks is based on the positional deviation on the screen with respect to the target block. In an image processing apparatus for detecting a motion vector for a target block,
A correlation value between each of the reference blocks and the target block is calculated using pixel values of the plurality of pixels in the reference block and pixel values of a plurality of pixels at corresponding positions in the target block. A strongest correlation reference block detecting means for detecting a position of the strongest correlation reference block having the strongest correlation with the target block;
Obtaining means for obtaining the strongest correlation value for the strongest correlation reference block and the correlation values for a plurality of neighboring reference blocks in the vicinity of the position of the strongest correlation reference block;
Interpolation processing is performed using the strongest correlation value of the strongest correlation reference block acquired by the acquisition unit and the plurality of correlation values of the plurality of neighboring reference blocks, and is smaller than the pixel pitch of the target screen. with high accuracy, the interpolation correlation with the target block to detect the position of the strongest high definition strongest correlation reference block, calculates the pre kidou-out vector as positional deviation from the target block of the high-definition strongest correlation reference block Processing means, wherein the strongest correlation reference block detection means includes a holding unit that holds information about the strongest correlation value and the position of the strongest correlation reference block, and the newly calculated reference block of the reference block The new correlation value for the target block is compared with the strongest correlation value of the holding unit, and the new phase When it is determined that the value is stronger in correlation, the information on the strongest correlation value and the position of the holding unit is updated to the information on the new correlation value and the position, and the acquisition means includes at least the for neighboring reference blocks several double of the vicinity of the strongest correlation reference block held in the holding unit, by re-calculating the correlation value, and obtains a correlation value for said neighboring reference blocks .

In the image processing method of the present invention, a reference block having the same size as a target block having a predetermined size composed of a plurality of pixels set in the target screen is set in a search range set in a reference screen different from the target screen. A plurality of reference blocks are set, and a motion vector for the target block is detected based on a positional deviation on the screen of the reference block having the strongest correlation with the target block among the plurality of reference blocks. In the image processing method, between each of the reference blocks and the target block, using pixel values of a plurality of pixels in the reference block and pixel values of a plurality of pixels at corresponding positions in the target block. The correlation value of the target block is the most A strongest correlation reference block detecting step for detecting a position of the strongest correlation reference block, a strongest correlation value for the strongest correlation reference block, and the correlation values for a plurality of neighboring reference blocks in the vicinity of the position of the strongest correlation reference block And interpolating using the strongest correlation value of the strongest correlation reference block and the plurality of correlation values of the plurality of neighboring reference blocks acquired in the acquisition step, The position of the high-definition strongest correlation reference block having the highest correlation with the target block is detected with high accuracy smaller than the pixel pitch of the screen, and the movement is detected as the positional deviation of the high-definition strongest correlation reference block from the target block. An interpolation processing step for calculating a vector is executed by the image processing apparatus, and The image processing apparatus includes a holding unit that holds information about the strongest correlation value and the position of the strongest correlation reference block, and in the strongest correlation reference block detection step, the target block of the newly calculated reference block When the new correlation value is compared with the strongest correlation value of the holding unit and it is determined that the new correlation value has a stronger correlation, information on the strongest correlation value of the holding unit and its position Is updated to information on the new correlation value and its position, and in the obtaining step, the correlation value is determined for at least a plurality of neighboring reference blocks in the vicinity of the strongest correlation reference block held in the holding unit. A correlation value for the neighborhood reference block is obtained by recalculation.

  According to the present invention, it is possible to perform motion vector detection with higher accuracy than the pixel accuracy in the target block and the reference block with a hardware scale smaller than the conventional one. Further, according to the present invention, it is possible to realize speeding up of processing, reduction of bus bandwidth, and the like to a practically satisfactory level.

  Embodiments of an image processing apparatus and method according to the present invention will be described below with reference to the drawings.

[Embodiment of Image Processing Device According to the Invention]
An embodiment of an image processing apparatus using the image processing method according to the present invention will be described with reference to the drawings, taking the case of an imaging apparatus as an example. In the imaging apparatus according to the embodiment described below, as shown in FIG. 2, a plurality of images, for example, P1, P2, and P3, which are continuously captured, are aligned using motion detection and motion compensation. Thereafter, by superimposing, an image Pmix with reduced noise can be obtained. That is, since noise in each of a plurality of images is random, the noise is reduced with respect to the images by superimposing images having the same content.

  In the following description, a system that reduces noise by superimposing a plurality of images using motion detection and motion compensation is referred to as an NR (Noise Reduction) system, and an image in which noise is reduced by the NR system is referred to as an NR image. I will do it.

  In this specification, a screen (image) to be subjected to noise reduction is defined as a target screen (target frame), and a screen to be superimposed is defined as a reference screen (reference frame). Images taken consecutively are misaligned due to the camera shake of the photographer, etc., and alignment is important to superimpose both images. Here, it is necessary to consider that there is a movement of the subject in the screen as well as a blur of the entire screen such as a camera shake.

  For this reason, in order to enhance the noise reduction effect for the subject as well, as shown in FIG. 3, it is necessary to align each of the plurality of target blocks 102 generated by dividing the target frame 100. Become.

  Therefore, in this embodiment, a motion vector (hereinafter referred to as a block motion vector) 121 is detected for each of the plurality of target blocks 102, and the corresponding block motion vector 104B is detected for each target block 102. To align the images and superimpose the images.

  In the image pickup apparatus of this embodiment, at the time of still image shooting, as shown in FIG. 4, a plurality of images are shot at high speed, the first shot image is used as the target frame 100, and the second and subsequent frames. A predetermined number of photographed images are used as a reference frame 101 and are superimposed, and the superimposed image is recorded as a still image photographed image. That is, when the photographer depresses the shutter button of the imaging device, the predetermined number of images are photographed at a high speed, and a plurality of images (frames) photographed later in time for the first image (frame). One sheet of image (frame) is superimposed.

  Further, at the time of moving image shooting, as shown in FIG. 5, the current frame image output from the image sensor is set as the target frame 100 image, and the previous frame image is set as the reference frame 101 image. Therefore, in order to reduce the noise of the current frame image, the image of the previous frame of the current frame is superimposed on the current frame.

  In the description of the method for superimposing the images in FIGS. 4 and 5 described above, when the frame rate of the moving image to be recorded is 60 fps (frame / second), the frame rate of 60 fps is obtained from the image sensor. This is a case where two image frames are superimposed, and as a result, a captured image signal is obtained at a frame rate of 60 fps with reduced noise.

  However, when the captured image is output from the image sensor at a higher frame rate of, for example, 240 fps, even when capturing a moving image, one image is superimposed on each other. It is also possible to obtain a captured image signal having a frame rate of 60 fps by generating a moving image frame. Of course, a 240 fps picked-up image can be obtained by superimposing two image frames in the same manner as in this example, and as a result, a picked-up image signal with a frame rate of 240 fps can be obtained.

  Next, in this embodiment, in order to be able to superimpose images with higher accuracy, the accuracy of the pixel pitch of the processing target image (referred to as pixel accuracy), that is, the original screen (target The block motion vector is detected with a high accuracy with a pitch smaller than the pixel pitch of the frame (this high accuracy is referred to as sub-pixel accuracy). In order to calculate the motion vector with the sub-pixel accuracy, as will be described later, in the imaging apparatus according to the present embodiment, an interpolation process is performed using the motion vector obtained with the pixel accuracy and a reference vector in the vicinity thereof. Like that.

  FIG. 1 shows a block diagram of an example of an imaging apparatus as an embodiment of the image processing apparatus of the present invention.

  As shown in FIG. 1, in the imaging apparatus of this embodiment, a CPU (Central Processing Unit) 1 is connected to a system bus 2, and an imaging signal processing system 10 and a user operation input unit 3 are connected to the system bus 2. The image memory unit 4 and the recording / reproducing device unit 5 are connected to each other. In this specification, the CPU 1 includes a ROM (Read Only Memory) and a work area RAM (Random Access Memory) that store programs for performing various software processes, although not shown. .

  Upon receiving an imaging recording start operation through the user operation input unit 3, the imaging apparatus in FIG. 1 performs captured image data recording processing as described later. Further, upon receiving a reproduction start operation of the captured / recorded image through the user operation input unit 3, the imaging apparatus of FIG. 1 performs a reproduction process of the captured image data recorded on the recording medium of the recording / reproducing apparatus unit 5.

  As shown in FIG. 1, incident light from a subject through a camera optical system (not shown) provided with an imaging lens 10 </ b> L is irradiated on an imaging element 11 and imaged. In this example, the imaging device 11 is configured by a CCD (Charge Coupled Device) imager. Note that the image sensor 12 may be a CMOS (Complementary Metal Oxide Semiconductor) imager.

  In the imaging apparatus of this example, when an imaging recording start operation is performed, a video input through the lens 10L is converted into a captured image signal by the imaging element 11, and a signal synchronized with the timing signal from the timing signal generation unit 12 As a result, an analog imaging signal, which is a RAW signal (raw signal) in a Bayer array composed of three primary colors of red (R), green (G), and blue (B), is output. The output analog imaging signal is supplied to the preprocessing unit 13, subjected to preprocessing such as defect correction and γ correction, and supplied to the data conversion unit 14.

  The data converter 14 converts the analog imaging signal, which is a RAW signal input thereto, into a digital imaging signal (YC data) composed of a luminance signal component Y and a color difference signal component Cb / Cr, and the digital image The imaging signal is supplied to the resolution conversion unit 15. The resolution conversion unit 15 converts the digital imaging signal to the resolution designated through the user operation input unit 3 and supplies the converted signal to the image memory unit 4 via the system bus.

  When the photographing instruction through the user operation input unit 3 is a still image photographing instruction by pressing the shutter button, the digital imaging signal whose resolution has been converted by the resolution converting unit 15 includes the above-described plurality of frames in the image memory unit 4. Written. Then, after the images for a plurality of frames are written in the image memory unit 4, the image data of the target frame and the image data of the reference frame are read by the motion detection / motion compensation unit 16, and after passing through the image superposition unit 17, The codec unit 18 performs codec conversion, and the data is recorded on a recording medium such as a DVD (Digital Versatile Disc) or a hard disk of the recording / reproducing apparatus unit 5 through the system bus 2.

  At the time of taking a still image, before the shutter button is pressed, the image data from the image superimposing unit 17 is converted into an NTSC standard color video signal by an NTSC (National Television System Committee) encoder 19, for example. The image is supplied to a monitor display 6 composed of an LCD (Liquid Crystal Display; liquid crystal display), and a monitor image at the time of still image shooting is displayed on the display screen.

  When the shooting instruction through the user operation input unit 3 is a moving image shooting instruction by pressing the moving image recording button, the resolution-converted image data is written in the image memory unit 4 and the motion detection / movement is performed in real time. It is sent to the compensation unit 16, passes through the image superposition unit 17, is output to the display screen of the monitor display 6 through the NTSC encoder unit 19, is codec converted by the codec unit 18, and is supplied to the recording / reproducing device unit 5 through the system bus 2. And recorded on a recording medium such as a DVD or a hard disk.

  The captured image data recorded on the recording medium of the recording / reproducing apparatus unit 5 is read in response to a reproduction start operation through the user operation input unit 3, supplied to the codec unit 18, and reproduced and decoded. The reproduced and decoded image data is supplied to the monitor display 6 through the NTSC encoder 19, and the reproduced image is displayed on the display screen. Although not shown in FIG. 1, the output video signal from the NTSC encoder 19 can be derived outside through a video output terminal.

  The above-mentioned camera shake motion vector detection unit 16 can be configured by hardware, or can be configured by using a DSP (Digital Signal Processor). Furthermore, software processing can be performed by the CPU 1.

[Description of Motion Detection / Compensation Unit 16]
In this embodiment, the motion detection / compensation unit 16 basically performs motion vector detection by performing block matching processing using the SAD values described with reference to FIGS. To. However, in this embodiment, the hardware scale can be reduced without using the SAD table TBL that stores all of the SAD values for a plurality of reference blocks that move the search range with respect to the target block. I try to figure it out. This will be described in detail later.

  FIG. 6 shows a block diagram of a configuration example of the motion detection / motion compensation unit 16. In this example, the motion detection / compensation unit 16 includes a target block buffer unit 161 that stores pixel data of the target block 102, a reference block buffer unit 162 that stores pixel data of the reference block 108, and the target block 102. A matching processing unit 163 that calculates an SAD value for a pixel corresponding to the block 108, a motion vector calculation unit 164 that calculates a motion vector from the SAD value information output from the matching processing unit 163, and each block is controlled. A control unit 165.

  At the time of still image capturing, the first captured frame after pressing the shutter button stored in the image memory 4 is written in the target block buffer unit 161 as the target frame 102, and the image is stored in the reference block buffer unit 162. An imaging frame after the first imaging frame stored in the memory 4 is sequentially written as a reference frame 108.

  Further, at the time of moving image shooting, the imaging frame from the resolution conversion unit 15 is written as the target frame 102 in the target block buffer unit 161, and the reference block buffer unit 162 is one frame before the target frame. The imaging frame stored in the image memory 4 is written as the reference frame 108.

  In the matching processing unit 163, the block matching processing described with reference to FIGS. 33 to 38 is performed on the target block stored in the target block buffer unit 161 and the reference block stored in the reference block buffer unit 162. .

  Here, in order to detect the strength of the correlation between the target block and the reference block in block matching, the SAD value is calculated using the luminance information of the image data in this embodiment, and the minimum SAD value is obtained. And the reference block exhibiting the minimum SAD value is detected as the strongest correlation reference block.

  Needless to say, the SAD value may be calculated by using not only the luminance information but also the color difference signal and the information of the three primary color signals R, G and B. In calculating the SAD value, all the pixels in the block are usually used. However, in order to reduce the amount of calculation, only the pixel value of the pixel with a limited jump position is used by thinning out or the like. It may be.

  The motion vector calculation unit 164 detects the motion vector of the reference block with respect to the target block from the matching processing result of the matching processing unit 163. In this embodiment, a high-precision motion vector with sub-pixel accuracy is detected.

  A high-precision motion vector with subpixel accuracy will be described. In the block matching method described above, since block matching is performed in units of pixels, the motion vector is calculated only with pixel accuracy. As shown in FIG. 38, the point where the matching process is performed, that is, the position of the reference block exists with pixel accuracy, and in order to calculate a more accurate motion vector, matching processing in sub-pixel units is required. become.

In order to calculate a motion vector with N times the pixel accuracy (pixel pitch is 1 / N), if the matching process is performed in units of N times the pixel, the SAD table becomes about N ² times larger, Memory is required. In addition, for the block matching process, an image that is N times the upper sample must be generated, and the scale of hardware increases dramatically.

  Therefore, it is considered that a motion vector with sub-pixel accuracy is calculated from the SAD table that has been subjected to the matching process in units of pixels by interpolating the SAD table using a quadratic curve. In this case, instead of quadratic curve approximation interpolation, linear interpolation or higher-order approximation curve interpolation of cubic or higher may be used. However, in this example, a quadratic curve is used in consideration of the balance between accuracy and hardware implementation. Approximate interpolation is used.

  In this quadratic curve approximate interpolation, as shown in FIG. 7, the SAD value minimum value Smin (see 113 in FIG. 7) indicated by the pixel precision motion vector 104 and the vicinity of the position of the minimum value Smin. A plurality of SAD values at positions (referred to as neighboring SAD values), in this example, four neighboring SAD values Sx1, Sx2 and Sy1, Sy2 (114 in FIG. 7) adjacent to the position of the minimum value Smin in the X and Y directions. 115, 116, 117).

  As shown in FIG. 8, a quadratic approximate curve 118 is fitted using a minimum value Smin of the SAD value and neighboring SAD values Sx1 and Sx2 of two neighboring points in the X direction (horizontal direction), and this quadratic curve 118 is applied. Is the X coordinate Vx of the motion vector (high-precision motion vector) that is the minimum value SXmin of the SAD value with sub-pixel accuracy. The equation of quadratic curve approximate interpolation at this time is shown in the following equation (1).

SXmin = 1/2 × (Sx2-Sx1) / (Sx2-2Smin + Sx1) (1)
The X coordinate Vx which the minimum value SXmin of the sub-pixel accuracy SAD value obtained by the calculation formula (1) takes on the SAD table is the X coordinate Vx which is the minimum value of the sub-pixel accuracy SAD value.

  The division of the calculation formula (1) can be realized by a plurality of subtractions. If the subpixel accuracy to be obtained is, for example, an accuracy of 1/4 pixel pitch of the original pixel pitch, it can be obtained by subtracting only twice, so both the circuit scale and the computation time are small, and the quadratic curve Performance that is almost the same as cubic curve interpolation, which is considerably more complicated than interpolation, can be realized.

  Similarly, using a minimum value Smin of the SAD value and neighboring SAD values Sy1 and Sy2 of two neighboring points in the Y direction (vertical direction), a quadratic approximate curve is fitted, and a minimum value SYmin of the quadratic curve is obtained. The Y coordinate taken is the Y coordinate Vy that is the minimum value of the SAD value with subpixel accuracy. The equation of quadratic curve approximate interpolation at this time is shown in the following equation (2).

SYmin = 1/2 × (Sy2−Sy1) / (Sy2−2Smin + Sy1) (2)
As described above, by performing approximation of the quadratic curve twice in the X direction and the Y direction, a highly accurate motion vector (Vx, Vy) with subpixel accuracy is obtained.

  In the above description, the minimum SAD value and two SAD values in the vicinity in the X direction (horizontal direction) and the Y direction (vertical direction) are used. However, the SAD value in the vicinity in each direction is two points or more. There may be. Further, instead of the quadratic curve in the X direction and the Y direction, for example, an approximate curve may be fitted in an oblique direction. Furthermore, an approximate curve may be applied by adding an oblique direction to the X direction and the Y direction.

  FIG. 9 shows that a vector detection result with sub-pixel accuracy can be obtained from the value of the SAD table with pixel accuracy by using the above means and procedures. The horizontal axis in FIG. 9 is the interpolation magnification and represents how many times the resolution is increased in the one-dimensional direction. Since the SAD table is two-dimensional, the table area is reduced at a ratio of the square, whereas the error due to interpolation increases only to a linear level, so that the usefulness of the above-described interpolation method can be seen.

[Noise reduction processing for captured images]
<When shooting still images>
In the imaging apparatus of this embodiment, a flowchart of noise reduction processing by image superposition at the time of still image shooting is shown in FIG. Each step of the flowchart of FIG. 10 is executed under the control of the CPU 1 and the control unit 165 of the motion detection / compensation unit 16 controlled by the CPU 1.

  First, when the shutter button is pressed, in the imaging apparatus of this example, a plurality of images are taken at high speed under the control of the CPU 1. In this example, captured image data of M sheets (M frame; M is an integer of 2 or more) to be superimposed at the time of still image shooting is captured and pasted to the image memory unit 4 (step S1).

  Next, the reference frame is Nth in time among the M image frames stored in the image memory unit 4 (N is an integer of 2 or more, and the maximum value is M). The unit 165 sets the initial value of the value N, which is the order, to N = 2 (step S2). Next, the control unit 165 sets the first image frame as a target image (target frame) and sets the Nth image as a reference image (reference frame) (step S3).

  Next, the control unit 165 sets a target block in the target frame (step S4), and the motion detection / motion compensation unit 16 reads the target block from the image memory unit 4 into the target block buffer unit 161 (step S5). The pixel data in the matching processing range is read into the reference block buffer unit 162 (step S6).

  Next, the control unit 165 reads the reference block within the search range from the reference block buffer unit 162, and the matching processing unit 163 performs matching processing as described later. In this example, as will be described later, among the SAD values calculated by the matching processing unit 163, the SAD value described later holds the minimum value of the SAD value and the SAD value in the vicinity thereof, and approximates a quadratic curve. In order to perform the interpolation processing, it is sent to the motion vector calculation unit 164. After this is repeated for all reference vectors in the search range, the quadratic curve approximate interpolation processing unit 1646 performs the above-described interpolation processing and outputs a highly accurate motion vector (step S7).

  Next, the control unit 165 reads out the motion compensated image block from the reference block buffer unit 162 according to the high-precision motion vector ((Step S8), and synchronizes with the target block to the subsequent image superimposing unit 17. Send (step S9).

  Next, under the control of the CPU 1, the image superimposing unit 17 superimposes the target block and the motion compensation image block, and pastes the image data of the superimposed block on the image memory unit 4. That is, the image superimposing unit 17 writes the image data of the superimposed block in the image memory unit 4 (step S10).

  Next, the control unit 165 determines whether or not block matching has been completed for all target blocks in the target frame (step S11), and block matching processing has not been completed for all target blocks. When it is determined, the process returns to step S4, the next target block is set, and the processes from step S4 to step S11 are repeated.

  Further, when the control unit 165 determines in step S11 that the block matching for all target blocks in the target frame has ended, whether or not the processing for all reference frames to be superimposed has ended, that is, It is determined whether or not M = N (step S12).

  If it is determined in step S12 that M = N is not satisfied, N = N + 1 is set (step S13), the process returns to step S3, and the processes after step S3 are repeated. In other words, when M is 3 or more, the above processing is repeated with the image that has been superimposed in all target blocks as the next target image and the third image as the reference frame. This is repeated until the Mth overlap is completed. If it is determined in step S12 that M = N, this processing routine is terminated.

  The above-described still image noise reduction processing method is a method of pasting M pieces of image data on the image memory unit 4, but the image may be superimposed each time one image is taken. In that case, since only one image frame may be stored in the image memory unit 4, the shooting interval is longer than the noise reduction processing method of the processing routine of FIG. 10, but the memory cost is minimized. Is possible.

<When shooting movies>
Next, in the imaging apparatus of this embodiment, a flowchart of noise reduction processing by image superposition at the time of moving image shooting is shown in FIG. Each step of the flowchart of FIG. 11 is also executed under the control of the CPU 1 and the control unit 165 of the motion detection / compensation unit 16 controlled by the CPU 1. When the moving image recording button is operated by the user, the CPU 1 instructs the process of FIG. 11 to start from the start.

  In this embodiment, the motion detection / compensation unit 16 has a configuration suitable for performing matching processing in units of target blocks. Therefore, the resolution conversion unit 15 holds the frame image and sends the image data to the motion detection / compensation unit 16 in units of target blocks according to the control of the CPU 1 (step S21).

  The target block image data sent to the motion detection / motion compensation unit 16 is stored in the target block buffer unit 161. Next, the control unit 165 sets a reference vector corresponding to the target block (step S22), and reads image data in the matching processing range from the image memory unit 4 into the reference block buffer unit 162 (step S23).

  Next, the matching processing unit 163 and the motion vector calculation unit 164 perform motion detection processing by block matching (step S24). That is, the matching processing unit 163 calculates an SAD value between the pixel value of the target block and the pixel value of the reference block, and sends the calculated SAD value to the motion vector calculation unit 164. The matching processing unit 163 repeats this process for all reference blocks in the search range. After the calculation of the SAD values for all the reference blocks in the search range is finished, the motion vector calculation unit 164 specifies the minimum SAD value, specifies the SAD value in the vicinity thereof, and uses those SAD values. Then, the above-described quadratic curve approximate interpolation process is performed, and a high-precision motion vector with sub-pixel accuracy is output.

  Next, the control unit 165 reads out the image data of the motion compensated image block from the reference block buffer unit 162 according to the high-precision motion vector calculated in step S24 (step S25), and synchronizes with the target block. To the image superimposing unit 17 (step S26).

  The image superimposing unit 17 superimposes the target block and the motion compensation image block. Then, the image superimposing unit 17 outputs the image data of the superimposed NR image to the monitor display 6 through the NTSC encoder 19 so as to monitor the moving image recording and the recording / reproducing device unit through the codec unit 18. 6 to be recorded on the recording medium (step S27).

  The image superimposed by the image superimposing unit 17 is also stored in the image memory unit 4 so that it becomes a reference frame in the next frame (target frame) (step S28).

  Then, the CPU 1 determines whether or not the moving image recording stop operation has been performed by the user (step S29). When it is determined that the moving image recording stop operation has not been performed by the user, the CPU 1 returns to step S21, and after this step S21. Instruct to repeat the process. If it is determined in step S29 that the moving image recording stop operation has been performed by the user, the CPU 1 ends this processing routine.

In the above-described processing routine for noise reduction processing of a moving image, an image frame one frame before is used as a reference frame. However, an image of a frame earlier than one frame may be used as a reference frame. In addition, the image before the first frame and the image before the second frame may be stored in the image memory unit 4, and the image frame to be used as the reference frame may be selected from the contents of the two pieces of image information. . (Please give an example of what kind of image content you want to identify and what kind of identification process to do to select which image)
By using the means, procedure, and system configuration as described above, it is possible to perform still image noise reduction processing and moving image noise reduction processing with one common block matching processing hardware.

[About the motion vector calculation unit 164]
Next, a configuration example and operation of the motion vector calculation unit 164 will be described. First, before describing the configuration example and the operation of the motion vector calculation unit 164 according to the embodiment of the present invention, a comparative example using an SAD table will be described as in the prior art.

<Comparative example>
FIG. 12 shows a configuration diagram of a comparative example of the motion vector calculation unit 164. The motion vector calculation unit 164 of this comparative example includes an SAD value writing unit 1641, an SAD table TBL, an SAD value comparison unit 1642, an SAD value holding unit 1643, an X direction (horizontal direction) neighborhood value extraction unit 1644, A Y-direction (vertical direction) neighborhood value extraction unit 1645 and a quadratic curve approximation interpolation processing unit 1646 are included.

  In the motion vector calculation unit 164 of the comparative example, the SAD value writing unit 1641 receives the position information (reference vector) of the reference block and the SAD value of the reference block sent from the matching processing unit 163 as the SAD table TBL. To store. When all the matching processes in the search range are completed in the matching processing unit 163, the SAD value comparison unit 1642 performs a comparison process for all the SAD values in the SAD table TBL, searches for the minimum SAD value, and detects the minimum detected value. The SAD value and the position information (reference vector) of the reference block exhibiting the SAD value are held in the SAD value holding unit 1643.

  Next, four SAD values in the vicinity of the detected minimum SAD value and reference block position information (reference vector) representing each SAD value are read from the SAD table TBL and held in the SAD value holding unit 1643.

  Next, the X direction (horizontal direction) neighborhood value extraction unit 1644 refers to the detected minimum SAD value and the X direction (horizontal direction) neighborhood value held in the SAD value holding unit 1643, respectively. This is read together with the block position information (reference block) and sent to the quadratic curve approximation interpolation processing unit 1646. The quadratic curve approximate interpolation processing unit 1646 performs the interpolation process using the quadratic curve in the X direction (horizontal direction) as described above.

  Next, the Y direction (vertical direction) neighborhood value extraction unit 1645 refers to the detected minimum SAD value and the neighborhood value in the Y direction (vertical direction) held in the SAD value holding unit 1643, respectively. This is read together with the block position information (reference block) and sent to the quadratic curve approximation interpolation processing unit 1646. The quadratic curve approximate interpolation processing unit 1646 executes the interpolation process using the quadratic curve in the Y direction (vertical direction) as described above.

  In this way, the quadratic curve approximate interpolation processing unit 1646 performs interpolation processing by the quadratic curve twice in the X direction and the Y direction, and calculates a high-precision motion vector with subpixel accuracy.

  An example of the flow of block matching processing in this comparative example is shown in the flowchart of FIG. Each step of the flowchart of this comparative example is performed by the matching processing unit 163 and the motion vector calculation unit 164.

  First, the matching processing unit 163 sets the reference vector (Vx, Vy), sets the reference block position for calculating the SAD value (step S31), and sets the reference block pixel data as the reference block buffer unit 162. (Step S32), the pixel data of the target block in the target block buffer unit 161 is read, and the sum of absolute values of differences between the pixel data of the target block and the reference block, that is, the SAD value is obtained (step S33). The obtained SAD value is sent to the motion vector calculation unit 164. In the motion vector calculation unit 164, the SAD value writing unit 1641 stores the received SAD value at the position of the corresponding reference block, that is, the position corresponding to the reference vector (step S34).

  Next, the matching processing unit 163 determines whether or not the matching process has been completed at the positions (reference vectors) of all reference blocks in the search range (step S35), and the unprocessed reference block is still in the search range. When it is determined that there is, the process returns to step S31, and the processes after step S31 described above are repeated.

  If the matching processing unit 163 determines in step S35 that the matching processing has been completed at the positions (reference vectors) of all the reference blocks in the search range, the matching processing unit 163 notifies the motion vector calculation unit 164 accordingly.

  In response to this, in the motion vector calculation unit 164, the SAD value comparison unit 1642 compares each SAD value in the SAD table TBL, detects the SAD value that is the smallest in the SAD table TBL, and minimizes the detected SAD value. The value Smin is held in the minimum value storage unit of the SAD value holding unit 1643 together with the reference vector (information of the reference block position). Then, the SAD value holding unit 1643 acquires neighboring SAD values Sx1, Sx2, Sy1, and Sy2 near the detected position of the minimum SAD value Smin from the SAD table TBL and holds them in the respective storage units. (Step S36).

  Next, the X direction (horizontal direction) neighborhood value extraction unit 1644 and the Y direction (vertical direction) neighborhood value extraction unit 1645 hold the detected minimum SAD value Smin and its neighborhood SAD held in the SAD value holding unit 1643. The values Sx1, Sx2, Sy1, Sy2 and their position information are read out and sent to the quadratic curve approximation interpolation processing unit 1646. Receiving this, the quadratic curve approximation interpolation processing unit 1646 performs interpolation by the quadratic curve twice in the X direction and the Y direction, and calculates a high-precision motion vector with subpixel accuracy as described above ( Step S37). Thus, the block matching process of the comparative example for one reference frame is completed.

  In the comparative example described above, the SAD table TBL is generated, and the minimum SAD value for quadratic curve approximation interpolation and four SAD values in the vicinity thereof are extracted from the generated SAD table TBL. A large memory is required for the SAD table TBL.

  In this embodiment, by not generating the SAD table TBL, the circuit scale can be further reduced and the processing time can be reduced.

<First Example of Motion Vector Calculation Unit 164 of Embodiment>
As described above, the block matching process uses the position indicated by the reference vector as the position of the reference block, calculates the SAD value of each pixel of each reference block and each pixel of the target block, and performs the calculation process within the search range. This is performed for reference blocks at positions indicated by all reference vectors.

  Here, when searching for a motion compensation block by changing the position of the reference block within the search range, the search is performed in order from the end of the screen (frame), and from the center of the screen (frame) to the outside. In this embodiment, the search direction is set as indicated by an arrow 120 in FIG. 14A, and the search is started in the horizontal direction from the upper left end of the search range. After the minute search is completed, a search method is adopted in which a procedure is repeated in which a line one line below in the vertical direction is searched in the horizontal direction from the left end.

  That is, as shown in FIG. 14B, in the search range 106, the reference block 108 is sequentially set and searched in the horizontal direction from the upper left of the search range 106, and the SAD value for each reference block 108 is calculated. I do. Then, as shown in FIG. 14C, the corresponding SAD table is also buried in the horizontal direction from the upper left. At this time, the pixel data range actually used for the matching process becomes the matching processing range 110 corresponding to the size of the reference block 108 as described above.

  As shown in FIG. 7, in order to perform the sub-pixel accuracy quadratic curve approximate interpolation processing of this example, the minimum value Smin of the SAD value and SAD values Sx1, Sx2, Sy1, and Sy2 in the vicinity thereof are obtained. It ’s fine.

  When the SAD value for each reference block is calculated, the calculated SAD value is compared with the minimum value of the SAD value up to that point, and the calculated SAD value is greater than the minimum value of the SAD value up to that point. If it is smaller, the calculated SAD value is held as the minimum value, and the SAD value and the reference vector at that time are held, so that the minimum value of the SAD value and the position information of the reference block that takes the minimum value (reference vector Information) can be obtained without generating the SAD table.

  Then, if the minimum value of the detected SAD value is held and the SAD value of the reference block near the reference block position that becomes the minimum SAD value is held as the neighboring SAD value, the neighboring SAD value is also obtained. The SAD table can be held without being generated.

  At this time, in this example, as shown in FIG. 15 (A), the search method as shown in FIG. 14 (A) described above is adopted, so that one line in the horizontal direction in the conventional SAD table TBL is used. If a memory having a capacity for storing the SAD value (hereinafter referred to as a line memory) is provided, when the SAD value of the reference block is newly calculated, the SAD table TBL is hatched in FIG. As shown, the SAD values of a plurality of reference blocks for one line of the SAD table TBL calculated before the newly calculated SAD value 121 are stored as the storage data 122 in the line memory. Will be stored.

  Therefore, when the newly calculated SAD value of the reference block is detected as the minimum SAD value, the reference block at a position on one line of the position of the reference block exhibiting the minimum SAD value 121 on the SAD table TBL. SAD value 123 (neighboring SAD value (Sy1)) and the SAD value 124 (neighboring SAD value (Sx1)) of the reference block on the left side of the position of the reference block exhibiting the minimum SAD value 121 are the line memory. Can be obtained from.

  Then, on the SAD table TBL, the neighboring SAD value (Sx2) (see 125 in FIG. 15C) that is the SAD value of the reference block at the position to the right of the reference block position of the minimum SAD value is The SAD value calculated at the reference block position may be held. Similarly, in the SAD table TBL, the neighborhood SAD value (Sy2) which is the SAD value of the reference block at a position one line below the position of the reference block of the newly calculated minimum SAD value (126 in FIG. 15C). (Reference) may also hold the SAD value calculated at the reference block position later.

  Considering the above, the first example of the motion vector calculation unit 164 has a hardware configuration as shown in FIG.

  That is, in the first example of the motion vector calculation unit 164, the SAD table TBL of the comparative example of FIG. 12 is not provided, the SAD value writing unit 1641, the SAD value comparison unit 1642, the SAD value holding unit 1643, X direction (horizontal direction) neighborhood value extraction unit 1644, Y direction (vertical direction) neighborhood value extraction unit 1645, quadratic curve approximation interpolation processing unit 1646, and memory (referred to as line memory) 1647 for one line of the SAD table TBL. And comprising.

  The X direction (horizontal direction) neighborhood value extraction unit 1644, the Y direction (vertical direction) neighborhood value extraction unit 1645, and the quadratic curve approximation interpolation processing unit 1646 perform the same operations as in the comparative example described above. The SAD value writing unit 1641, the SAD value comparison unit 1642, the SAD value holding unit 1643, and the line memory 1647 perform different operations from the above-described comparative example.

  Similar to the comparative example of FIG. 12, the SAD value holding unit 1643 includes a holding unit (memory) for the minimum SAD value Smin and the neighboring SAD values Sx1, Sx2, Sy1, and Sy2. In this first example, the SAD value holding unit 1643 supplies the minimum SAD value Smin from the minimum SAD value holding unit to the SAD value comparison unit 1642, and among the stored nearby SAD values, The position information (reference vector) of the reference block of the neighboring SAD value Sx2 on the right side of the minimum SAD value Smin and the position information (reference vector) of the reference block of the neighboring SAD value Sy2 on the lower side of the minimum SAD value Smin The data is supplied to the writing unit 1641.

  In this first example, the SAD value comparison unit 1642 receives the position information (reference vector) of the reference block from the matching processing unit 163 and the SAD value Sin of the reference block, and the SAD value holding unit 1643 The minimum SAD value Smin from the minimum SAD value holding unit is received.

  Then, the SAD value comparison unit 1642 compares both the SAD value Sin calculated at that time from the matching processing unit 163 and the minimum SAD value Smin from the minimum SAD value holding unit of the SAD value holding unit 1643, When the SAD value Sin calculated at the time from the matching processing unit 163 is smaller, the SAD value is detected as the minimum SAD value at the time, and when the SAD value Sin is smaller, At the time, it is detected that the minimum SAD value Smin from the minimum SAD value holding unit of the SAD value holding unit 1643 is still the minimum value. The SAD value comparison unit 1642 supplies the detection result information DET to the SAD value writing unit 1641 and the SAD value holding unit 1643.

  The SAD value writing unit 1641 includes a buffer memory for one pixel for temporarily storing the SAD value Sin calculated from the matching processing unit 163 and its position information (reference vector). In this first example, The reference block position information (reference vector) from the matching processing unit 163 and the SAD value Sin of the reference block are written in the line memory 1647. In this case, the line memory 1647 performs the same operation as that of the shift register, and when there is no empty space, when the new position information and SAD value are stored, the oldest position information and SAD value in the line memory 1647 are discarded. Is done.

  In addition, the SAD value writing unit 1641 performs the following processing in the first example before writing the calculated SAD value Sin and its position information into the line memory 1647.

  That is, the SAD value writing unit 1641, when the comparison detection result information DET from the SAD value comparison unit 1642 indicates that the SAD value Sin is the minimum value, the position information of the reference block from the matching processing unit 163. (Reference vector) and the SAD value Sin of the reference block are sent to the SAD value holding unit 1643.

  The SAD value holding unit 1643 detects that the SAD value Sin is the minimum value based on the comparison detection result information DET from the SAD value comparison unit 1642, and the position information of the reference block transmitted from the SAD value writing unit 1641. (Reference vector) and the SAD value Sin of the reference block are stored in the minimum SAD value holding unit.

  In the first example, the SAD value writing unit 1641 uses the position information (reference vector) of the reference block from the matching processing unit 163 as the position of the neighboring SAD value Sx2 or Sy2 received from the SAD value holding unit 1643. Even when the information matches, the reference block position information (reference vector) from the matching processing unit 163 and the SAD value Sin of the reference block are sent to the SAD value holding unit 1643.

  The SAD value holding unit 1643 recognizes which neighboring SAD value is information from the received reference block position information (reference vector), and stores it in the corresponding neighboring SAD value holding unit.

  When the above processing is completed for all the reference blocks in the search range, the SAD value holding unit 1643 holds the minimum SAD value and its position information, and the four neighboring SAD values and its position information, as described above. .

  Therefore, in the same manner as in the comparative example, the X direction (horizontal direction) neighborhood value extraction unit 1644 and the Y direction (vertical direction) neighborhood value extraction unit 1645 are stored in the SAD value holding unit 1643 and the detected minimum SAD. The value Smin and its neighboring SAD values Sx1, Sx2, Sy1, Sy2 and their position information are read out and sent to the quadratic curve approximation interpolation processing unit 1646. Receiving this, the quadratic curve approximation interpolation processing unit 1646 performs interpolation by the quadratic curve twice in the X direction and the Y direction, and calculates a high-precision motion vector with sub-pixel accuracy as described above.

  As described above, in the first example, by using a line memory for one line of the SAD table TBL instead of the SAD table TBL, it is possible to detect a motion vector with sub-pixel accuracy.

  17 and 18 are flowcharts of the block matching technique in the motion vector calculation unit 164 and the matching processing unit 163 of the first example shown in FIG.

  First, an initial value of the minimum SAD value Smin of the SAD value holding unit 1643 of the motion vector calculation unit 164 is set (step S41 in FIG. 17). As the initial value of the minimum SAD value Smin, for example, the maximum value of the pixel difference is set.

  Next, the matching processing unit 163 sets reference vectors (Vx, Vy), sets reference block positions for calculating SAD values (step S42), and sets the reference block pixel data as reference block buffer units. At the same time as reading from 162 (step S43), the pixel data of the target block of the target block buffer unit 161 is read to obtain the sum of absolute values of differences between the pixel data of the target block and the reference block, that is, the SAD value (step S44). ), And sends the obtained SAD value to the motion vector calculation unit 164.

  In the motion vector calculation unit 164, the SAD value comparison unit 1642 is calculated by comparing the SAD value Sin calculated by the matching processing unit 163 with the minimum SAD value Smin held in the SAD value holding unit 1643. It is determined whether or not the SAD value Sin is smaller than the minimum SAD value Smin held so far (step S45).

  When it is determined in step S45 that the calculated SAD value Sin is smaller than the minimum SAD value Smin, the process proceeds to step S46, and information on the minimum SAD value Smin held in the SAD value holding unit 1643 and the minimum SAD value Smin are obtained. The SAD value and the position information (reference vector) of the reference block at the position one pixel above and one pixel to the left of the reference block position to be presented are updated.

  That is, the SAD value comparison unit 1642 sends the comparison result information DET to the SAD value writing unit 1641 that the calculated SAD value Sin is smaller than the minimum SAD value Smin. Then, the SAD value writing unit 1641 sends the calculated SAD value Sin and its position information (reference vector) to the SAD value holding unit 1643 as information of a new minimum SAD value Smin, and from FIG. As can be seen, the oldest SAD value and its position information (reference vector) in the line memory 1647 and the newest SAD value and its position information (reference vector) are referenced to the position one pixel above the position of the minimum SAD value. The information is sent to the SAD value holding unit 1643 as information on the SAD value Sy1 of the block and information on the SAD value Sx1 of the reference block located one pixel to the left. The SAD value holding unit 1643 receives the new minimum SAD value Smin information, the SAD value Sy1 information of the reference block at the position one pixel above, and the SAD value Sx1 information of the reference block at the position one pixel left, respectively. Update the corresponding retention information.

  After step S46, the SAD value writing unit 1641 writes the calculated SAD value Sin in the line memory 1647 (step S47).

  If it is determined in step S45 that the calculated SAD value Sin is greater than the minimum SAD value Smin, the process advances to step S47 without performing the update process of the retained information in step S46, and the SAD value writing unit 1641 In 1647, the calculated SAD value Sin is written.

  Next, the SAD value writing unit 1641 indicates the position of the reference block one pixel below the position of the reference block where the position indicated by the position information (reference vector) for the calculated SAD value Sin is held as the minimum SAD value Smin. It is determined whether or not it is a position (step S51), and when it is determined that it is the position of a reference block one pixel below, the calculated SAD value Sin and its position information (reference vector) are sent to the SAD value holding unit 1643. . The SAD value holding unit 1643 updates the holding information of the neighboring SAD value Sy2 for the reference block at the position one pixel below by using the received SAD value and its position information (step S52).

  When it is determined in step S51 that the position indicated by the position information (reference vector) for the calculated SAD value Sin is not the position of the reference block one pixel below the position of the reference block held as the minimum SAD value Smin In the SAD value writing unit 1641, the position indicated by the position information (reference vector) about the calculated SAD value Sin is the position of the reference block one pixel below the position of the reference block held as the minimum SAD value Smin. It is determined whether or not there is (step S53).

  In step S53, it is determined that the position indicated by the position information (reference vector) for the calculated SAD value Sin is the position of the reference block one pixel to the right of the position of the reference block held as the minimum SAD value Smin. In this case, the SAD value writing unit 1641 sends the calculated SAD value Sin and its position information (reference vector) to the SAD value holding unit 1643. The SAD value holding unit 1643 updates the holding information of the neighboring SAD value Sx2 for the reference block at the position one pixel to the right based on the received SAD value and its position information (step S54).

  When it is determined in step S53 that the position indicated by the position information (reference vector) for the calculated SAD value Sin is not the position of the reference block one pixel to the right of the position of the reference block held as the minimum SAD value Smin The SAD value writing unit 1641 writes the calculated SAD value Sin and its position information (reference vector) only in the line memory 1647 (step S47 described above) and does not send it to the SAD value holding unit 1643.

  Then, the matching processing unit 163 determines whether or not the matching process has been completed at the positions (reference vectors) of all the reference blocks in the search range (step S55), and an unprocessed reference block is still in the search range. If it is determined that there is, the process returns to step S42 in FIG. 17 to repeat the processes after step S42 described above.

  If the matching processing unit 163 determines in step S55 that the matching processing has been completed at the positions (reference vectors) of all the reference blocks in the search range, the matching processing unit 163 notifies the motion vector calculation unit 164 accordingly.

  In response to this, the motion vector calculation unit 164 detects that the X direction (horizontal direction) neighborhood value extraction unit 1644 and the Y direction (vertical direction) neighborhood value extraction unit 1645 are held in the SAD value holding unit 1643. The minimum SAD value Smin and its neighboring SAD values Sx1, Sx2, Sy1, Sy2 and their position information are read out and sent to the quadratic curve approximation interpolation processing unit 1646. Receiving this, the quadratic curve approximation interpolation processing unit 1646 performs interpolation by the quadratic curve twice in the X direction and the Y direction, and calculates a high-precision motion vector with subpixel accuracy as described above ( Step S56). This completes the block matching process of the first example for one reference frame.

  As described above, in the first example, only one line of the SAD table is provided without holding the SAD table, and the SAD value writing unit 1641 only holds the SAD value for one pixel. Thus, motion vector detection with sub-pixel accuracy by interpolation processing can be performed.

  The method of the first example has the effect that the processing time is not changed and the hardware scale can be reduced because the number of times of block matching is the same as the method of holding the SAD table.

  In the description of the first example described above, the SAD value comparison unit 1642 uses the calculated SAD value Sin from the matching processing unit 163 and the minimum SAD value Smin held in the SAD value holding unit 163. Although the comparison is made, the SAD value comparison unit 1642 includes a holding unit for the minimum SAD value, and the held minimum SAD value is compared with the calculated SAD value Sin. When the calculated Sin is smaller The held minimum SAD value is updated, and the position information is sent to the SAD value holding unit 1643 through the SAD value writing unit 1641 so as to be held in the minimum SAD value holding unit of the SAD value holding unit 1643. .

<Second Example of Motion Vector Calculation Unit 164 of Embodiment>
The second example of the motion vector calculation unit 164 is an example in which the line memory 1647 in the case of the first example is also omitted to further reduce the hardware scale.

  In the second example, the minimum value Smin of the SAD value of the reference block in the search range and its position information (reference vector) are detected and held in the same manner as in the first example described above. However, the acquisition and holding of the neighboring SAD value and its position information are not performed simultaneously with the detection of the SAD value Smin, as in the first example, and the position information of the detected minimum SAD value Smin is matched. Returning to the unit 163, the matching processing unit 163 again calculates the SAD values for the reference blocks at the four point positions in the vicinity of the minimum SAD value Smin, and supplies them to the motion vector calculation unit 164.

  The motion vector calculation unit 164 receives the SAD values of the four neighboring positions calculated by the block matching process again from the matching processing unit 163 and its position information (reference vector), and each of the SAD value holding units 1643 Store in the holding part.

  A hardware configuration example of the first example of the motion vector calculation unit 164 is shown in FIG. That is, in the second example of the motion vector calculation unit 164, the SAD table TBL and the line memory 1647 are not provided, but as shown in FIG. 19, the SAD value writing unit 1641, the SAD value comparison unit 1642, and the SAD A value holding unit 1643, an X direction (horizontal direction) neighborhood value extraction unit 1644, a Y direction (vertical direction) neighborhood value extraction unit 1645, and a quadratic curve approximation interpolation processing unit 1646 are provided.

  Also in the second example, the X direction (horizontal direction) neighborhood value extraction unit 1644, the Y direction (vertical direction) neighborhood value extraction unit 1645, and the quadratic curve approximation interpolation processing unit 1646 are the same as those in the first example described above. The SAD value writing unit 1641, the SAD value comparison unit 1642, the SAD value holding unit 1643, and the line memory 1647 perform different operations from those in the first example.

  Similar to the first example, the SAD value holding unit 1643 includes a holding unit (memory) for the minimum SAD value Smin and the neighboring SAD values Sx1, Sx2, Sy1, and Sy2. Also in this second example, the SAD value holding unit 1643 supplies the minimum SAD value Smin from the minimum SAD value holding unit to the SAD value comparison unit 1642. However, in the second example, unlike the first example, the SAD value holding unit 1643 does not supply the position information of the neighboring SAD value to the SAD value writing unit 1641.

  In this second example, the SAD value comparison unit 1642 also calculates the SAD value Sin calculated at that time from the matching processing unit 163 and the minimum SAD value Smin from the minimum SAD value storage unit of the SAD value storage unit 1643. When the SAD value Sin calculated at the time from the matching processing unit 163 is smaller, the SAD value is detected as the minimum SAD value at the time, and the SAD value Sin When the value is smaller, it is detected that the minimum SAD value Smin from the minimum SAD value holding unit of the SAD value holding unit 1643 is still the minimum value at that time. The SAD value comparison unit 1642 supplies the detection result information DET to the SAD value writing unit 1641 and the SAD value holding unit 1643.

  Similar to the above example, the SAD value writing unit 1641 includes a buffer memory for one pixel for temporarily storing the SAD value Sin calculated from the matching processing unit 163 and its position information (reference vector). In this second example, when the SAD value writing unit 1641 indicates that the comparison detection result information DET from the SAD value comparison unit 1642 indicates that the SAD value Sin is the minimum value, the matching processing unit The reference block position information (reference vector) from 163 and the SAD value Sin of the reference block are sent to the SAD value holding unit 1643.

  The SAD value holding unit 1643 knows that the SAD value Sin is the minimum value based on the comparison detection result information DET from the SAD value comparison unit 1642, and the position information of the reference block sent from the SAD value writing unit 1641 ( Reference vector) and the SAD value Sin of the reference block are stored in the minimum SAD value holding unit.

  The above processing is performed on the SAD values calculated by the matching processing unit 163 for all reference blocks in the search range. In the second example, when the calculation of the SAD values for all the reference blocks within the search range is completed, the SAD value holding unit 1643 stores the position information (reference (reference) of the held minimum SAD value Smin. Vector) Vmin is supplied to matching processing section 163 to request recalculation of SAD values for four reference blocks in the vicinity of the position information.

  When the matching processing unit 163 receives from the SAD value holding unit 1643 a request for recalculation of the SAD value for the neighborhood reference block including the position information (reference vector) Vmin of the minimum SAD value Smin, the position information of the minimum SAD value Smin From the reference vector Vmin, the positions of the four neighboring reference blocks are detected, and the SAD value is calculated for the reference block at the detected position. Then, the calculated SAD value is sequentially supplied to the SAD value writing unit 1641 together with the position information (reference vector).

  In this case, since the matching processing unit 163 performs block matching processing in the order of the search direction, the neighboring SAD values are calculated in the order of SAD values Sy1, Sx1, Sx2, and Sy2. The SAD value writing unit 1641 sequentially supplies the received recalculated SAD value and its position information (reference vector) to the SAD value holding unit 1643.

  The SAD value holding unit 1643 sequentially writes and holds the recalculated SAD value and its position information (reference vector) in the corresponding storage unit.

  When the recalculation of the SAD value for the neighborhood reference block is thus completed, the X direction (horizontal direction) neighborhood value extraction unit 1644 and the Y direction (vertical direction) neighborhood value extraction unit 1645 perform the SAD value in the same manner as in the above example. The detected minimum SAD value Smin and its neighboring SAD values Sx1, Sx2, Sy1, Sy2 and their position information held in the holding unit 1643 are read out and sent to the quadratic curve approximation interpolation processing unit 1646. Receiving this, the quadratic curve approximation interpolation processing unit 1646 performs interpolation by the quadratic curve twice in the X direction and the Y direction, and calculates a high-precision motion vector with sub-pixel accuracy as described above.

  As described above, in the second example, the motion vector can be detected with sub-pixel accuracy without using the SAD table TBL or the line memory.

  FIG. 20 is a flowchart of the block matching technique in the motion vector calculation unit 164 and the matching processing unit 163 of the second example shown in FIG.

  First, an initial value of the minimum SAD value Smin of the SAD value holding unit 1643 of the motion vector calculation unit 164 is set (step S61). As the initial value of the minimum SAD value Smin, for example, the maximum value of the pixel difference is set.

  Next, the matching processing unit 163 sets reference vectors (Vx, Vy), sets reference block positions for calculating SAD values (step S62), and sets reference block pixel data as reference block buffer units. In addition to reading from 162 (step S63), the pixel data of the target block in the target block buffer unit 161 is read, and the sum of absolute values of differences between the pixel data of the target block and the reference block, that is, the SAD value is obtained. The SAD value is sent to the motion vector calculation unit 164 (step S64).

  In the motion vector calculation unit 164, the SAD value comparison unit 1642 is calculated by comparing the SAD value Sin calculated by the matching processing unit 163 with the minimum SAD value Smin held in the SAD value holding unit 1643. It is determined whether or not the SAD value Sin is smaller than the minimum SAD value Smin held so far (step S65).

  When it is determined in step S65 that the calculated SAD value Sin is smaller than the minimum SAD value Smin, the process proceeds to step S66, and the minimum SAD value Smin held in the SAD value holding unit 1643 and its position information are updated. .

  That is, the SAD value comparison unit 1642 sends the comparison result information DET to the SAD value writing unit 1641 that the calculated SAD value Sin is smaller than the minimum SAD value Smin. Then, the SAD value writing unit 1641 sends the calculated SAD value Sin and its position information (reference vector) to the SAD value holding unit 1643 as new minimum SAD value Smin information. The SAD value holding unit 1643 updates the minimum SAD value Smin to be held and its position information to the received new SAD value Sin and its position information.

  After step S66, the process proceeds to step S67. If it is determined in step S65 that the calculated SAD value Sin is greater than the minimum SAD value Smin, the process advances to step S67 without performing the holding information update process in step S66.

  In step S67, the matching processing unit 163 determines whether or not the matching process has been completed at the positions (reference vectors) of all reference blocks in the search range. If there is still an unprocessed reference block in the search range. When it is determined, the process returns to step S62, and the processes after step S62 are repeated.

  If the matching processing unit 163 determines in step S67 that the matching processing has been completed at the positions (reference vectors) of all reference blocks in the search range, the position information of the minimum SAD value Smin from the SAD value holding unit 1643. , The SAD value is recalculated for the reference block at the position of the four neighboring points, and the recalculated neighboring SAD value is supplied to the SAD value holding unit 1643 through the SAD value writing unit 1641 so as to be retained ( Step S68).

  Next, in the motion vector calculation unit 164, the X direction (horizontal direction) neighborhood value extraction unit 1644 and the Y direction (vertical direction) neighborhood value extraction unit 1645 are stored in the SAD value holding unit 1643 and the detected minimum The SAD value Smin and its neighboring SAD values Sx1, Sx2, Sy1, Sy2 and their position information are read out and sent to the quadratic curve approximation interpolation processing unit 1646. Receiving this, the quadratic curve approximation interpolation processing unit 1646 performs interpolation by the quadratic curve twice in the X direction and the Y direction, and calculates a high-precision motion vector with subpixel accuracy as described above ( Step S69). This completes the block matching process of the second example for one reference frame.

  In this second example, the processing time is increased by recalculating the SAD value as compared with the first example described above, but since no line memory is required, the circuit scale is reduced to the first. It can be reduced more than the example. In addition, since the SAD value is recalculated only for the neighborhood SAD value, in the above example, since the number of times is at most four, the increase in the processing time is small.

  In the description of the second example described above, the minimum SAD value is held in the SAD value holding unit 1643 while being detected. However, in the SAD value comparison unit 1642, the position information of the reference block that exhibits the minimum SAD value. (Reference vector) is detected and held, and when the first block matching is completed, the position information of the minimum SAD value is supplied from the SAD value comparison unit 1642 to the matching processing unit 163. Good.

  In this case, in the recalculation of the SAD value in the matching processing unit 163, the minimum SAD value is recalculated in addition to the SAD values of the four neighboring points. In this case, the number of recalculations of the SAD value is 5 and increases by 1. However, in the first block matching, only the SAD value comparison unit 1642 needs to operate, and the SAD value writing unit 1641 and the SAD value holding Since the unit 1643 only needs to operate so as to hold the recalculated SAD value, there is an advantage that the processing operation is simplified.

  Further, the processing in the motion detection and motion compensation unit 16 can be executed in parallel and in parallel for a plurality of target blocks set in the target frame. In that case, it is necessary to provide as many hardware systems as the motion detection and motion compensation unit 16 described above in accordance with the number of target blocks to be processed in parallel and in parallel.

  In the case of a method for generating SAD tables as in the comparative example, it is necessary to generate as many SAD tables as the number of target blocks, and a very large memory is required. However, in the first example, one line of the SAD table is sufficient for one target block, and the memory capacity can be greatly reduced. Furthermore, in the case of the second example, since no line memory is required, the memory capacity can be greatly reduced.

[Second Embodiment]
In the motion vector detection process of the first embodiment described above, the reference block is moved in units of pixels, the SAD value is calculated, and the motion with sub-pixel accuracy is calculated based on the calculated SAD value. This is an effective technique for reducing the matching processing time and circuit scale.

  However, in the first embodiment described above, since the reference block is moved in the search range in units of pixels, the matching processing time increases in proportion to the search range to be searched, and the first example When the line memory is used as described above, there is a problem that the capacity of the line memory increases.

  Therefore, in the second embodiment, a reduced image is created for the target image (target frame), block matching is performed on the created reduced image, and the original motion is detected based on the motion detection result in the reduced image. Perform block matching on the target image. Here, the reduced image is referred to as a reduced surface, and the original image that has not been reduced is referred to as a base surface. Therefore, in the second embodiment, after performing block matching on the reduced surface, block matching on the base surface is performed using the matching result.

  FIG. 21 and FIG. 22 show images of image reduction of the target frame (image) and the reference frame (image). That is, in the second embodiment, for example, as shown in FIG. 21, the basal plane target frame 130 is reduced to 1 / n (n is a positive number) in the horizontal direction and the vertical direction. The reduction plane target frame 132 is assumed. Accordingly, the basal plane target block 131 generated by dividing the basal plane target frame 130 into a plurality is a reduced plane in which each of the horizontal direction and the vertical direction is reduced to 1 / n × 1 / n in the reduced plane target frame. The target block 133 is obtained.

  Then, the reference frame is reduced in accordance with the image reduction magnification n of the target frame. That is, as shown in FIG. 22, the base surface reference frame 134 is reduced to 1 / n in each of the horizontal direction and the vertical direction to form a reduced surface reference frame 135. The motion vector 104 for the motion compensation block 103 detected on the base plane reference frame 134 is detected as a reduced plane motion vector 136 reduced to 1 / n × 1 / n in the reduced plane reference frame 135. .

  In the above example, the image reduction magnification of the target frame and the reference frame is the same. However, in order to reduce the calculation amount, different image reduction magnifications are used for the target frame (image) and the reference frame (image). Matching may be performed by matching the number of pixels in both frames by processing such as interpolation.

  Further, although the reduction ratios in the horizontal direction and the vertical direction are the same, the reduction ratios may be different between the horizontal direction and the vertical direction. For example, when the horizontal direction is reduced to 1 / n and the vertical direction is reduced to 1 / m, the reduced screen has a size of 1 / n × 1 / m of the original screen.

  FIG. 23 shows the relationship between the reduced plane reference vector and the base plane reference vector. Assuming that the motion detection origin 105 and the search range 106 are determined in the basal plane reference frame 134 as shown in FIG. 23A, the reduced plane reference frame 135 is reduced to 1 / n × 1 / n. Then, as shown in FIG. 23B, the search range is a reduced plane search range 37 reduced to 1 / n × 1 / n.

  In the second embodiment, within the reduction plane search range 137, the reduction plane reference vector 138 representing the amount of positional deviation from the motion detection origin 105 in the reduction plane reference frame 135 is set, and each reduction plane is displayed. The correlation between the reduced surface reference block 139 at the position indicated by the reference vector 138 and the reduced surface target block 131 is evaluated.

  In this case, since block matching is performed on the reduced image, the number of reduced surface reference block positions (reduced surface reference vectors) from which SAD values should be calculated in the reduced surface reference frame can be reduced, and the number of times the SAD value is calculated. The processing can be speeded up as much as it decreases.

  A reduction plane motion vector 136 in the reduction plane reference frame 135 shown in FIG. 22 is calculated by evaluating the correlation by block matching between the reduction plane reference block 139 and the reduction plane target block 131. The accuracy of the reduced plane motion vector 136 is as low as n times one pixel because the image is reduced to 1 / n × 1 / n. Therefore, even if the calculated reduced plane motion vector 136 is multiplied by n, the motion vector 104 with 1 pixel accuracy cannot be obtained in the base plane reference frame 134.

  However, in the base plane reference frame 134, it is clear that the base plane motion vector 141 with 1 pixel accuracy exists in the vicinity of the motion vector obtained by multiplying the reduced plane motion vector 136 by n.

  Therefore, in the second embodiment, as shown in FIG. 23C, the base plane motion vector is centered on the position indicated by the motion vector obtained by multiplying the reduced plane motion vector 136 by n in the base plane reference frame 134. The basal plane search range 140 is set to a narrow range in which 141 may be present.

  Then, as shown in FIG. 23C, a basal plane reference vector 141 in the basal plane reference frame 134 is set to indicate the position within the basal plane search range, and the position indicated by each basal plane reference vector 141 is set. A basal plane reference block 139 is set to perform block matching in the basal plane reference frame 134.

  Since the search range 140 set here may be a narrow range centered on the position indicated by the motion vector obtained by multiplying the calculated reduced plane motion vector 136 by n, the base plane for which the SAD value should be calculated in the base plane reference frame 134. The number of reference block positions (base plane reference vectors) can be reduced, and the processing can be speeded up as the number of SAD value calculations is reduced.

  Thus, if the pixel-basis motion vector 136 can be detected in the base-surface reference frame 134, the base-surface motion vector 136 is also obtained in the second embodiment in the same manner as in the first embodiment described above. In this example, a quadratic curve approximation interpolation process is also performed in this example using the SAD value of the reference block pointed to, that is, the minimum SAD value and the neighboring SAD value in the vicinity thereof, to calculate a motion vector with subpixel accuracy. To.

  In the second embodiment, the hardware configuration can be the same as that shown in FIG. However, in the second embodiment, in the block matching process in the motion detection / compensation unit 16, as described above, a reduced image motion vector is detected by performing a matching process once on a reduced image, In the narrow search range based on the reduced plane motion vector, the configuration in which the motion vector is calculated on the base plane of the original size and the interpolation process is performed is different from that of the first embodiment.

  In the block matching process, the first example or the second example that does not use the above-described SAD table TBL is used as in the first embodiment.

  FIG. 24 and FIG. 25 show a flowchart of an operation example of the block matching processing in the motion detection / motion compensation unit 16 in the second embodiment. This operation example is a case where the second example described above is used as the motion vector calculation method. It goes without saying that the first example described above can also be used as the motion vector calculation method.

  First, the motion detection / compensation unit 16 reads a reduced image of the target block, that is, a reduced surface target block, from the target block buffer unit 161 (step S71 in FIG. 24). Next, the initial value of the reduced surface minimum SAD value is set as the initial value of the minimum SAD value Smin of the SAD value holding unit 1643 of the motion vector calculation unit 164 (step S72). As an initial value of the reduced surface minimum SAD value Smin, for example, a maximum value of pixel difference is set.

  Next, the matching processing unit 163 sets a reduction plane search range, sets a reduction plane reference vector (Vx / n, Vy / n) in the set reduction search range, and calculates a SAD value. A reference block position is set (step S73). Then, the pixel data of the set reduction plane reference block is read from the reference block buffer unit 162 (step S74), and the sum of absolute values of differences between the pixel data of the reduction plane target block and the reduction plane reference block, that is, the reduction plane The SAD value is obtained, and the obtained reduced surface SAD value is sent to the motion vector calculation unit 164 (step S75).

  In the motion vector calculation unit 164, the SAD value comparison unit 1642 compares the reduced surface SAD value Sin calculated by the matching processing unit 163 with the reduced surface minimum SAD value Smin held in the SAD value holding unit 1643. Then, it is determined whether or not the calculated reduced surface SAD value Sin is smaller than the reduced surface minimum SAD value Smin held so far (step S76).

  If it is determined in step S76 that the calculated reduction surface SAD value Sin is smaller than the reduction surface minimum SAD value Smin, the process proceeds to step S77, and the reduction surface minimum SAD value Smin held in the SAD value holding unit 1643 and its position. Information is updated.

  That is, the SAD value comparison unit 1642 sends the comparison result information DET to the SAD value writing unit 1641 that the calculated reduction surface SAD value Sin is smaller than the reduction surface minimum SAD value Smin. Then, the SAD value writing unit 1641 sends the calculated reduced surface SAD value Sin and its position information (reference vector) to the SAD value holding unit 1643 as information on a new reduced surface minimum SAD value Smin. The SAD value holding unit 1643 updates the reduced reduction surface minimum SAD value Smin and its position information to be held to the received new reduction surface SAD value Sin and its position information.

  After step S77, the process proceeds to step S78. If it is determined in step S76 that the calculated reduced surface SAD value Sin is larger than the reduced surface minimum SAD value Smin, the process advances to step S78 without performing the holding information update process in step S77.

  In step S78, the matching processing unit 163 determines whether or not the matching process has been completed at the positions of all reduced surface reference blocks (reduced surface reference vectors) in the reduced surface search range. When it is determined that there is an unprocessed reduced surface reference block, the process returns to step S73, and the processes after step S73 are repeated.

  If the matching processing unit 163 determines in step S78 that the matching process has been completed at the positions of all reduced surface reference blocks (reduced surface reference vectors) in the reduced surface search range, the reduction processing from the SAD value holding unit 1643 is performed. The position information (reduced surface motion vector) of the surface minimum SAD value Smin is received, and the base surface search range is set as a relatively narrow range centered on the position coordinate indicated by the vector obtained by multiplying the received reduced surface motion vector by n times. The target frame is set (step S79), and the pixel data of the basal plane target block is read from the target block buffer unit 161 (step S80).

  Next, the initial value of the base surface minimum SAD value is set as the initial value of the minimum SAD value Smin of the SAD value holding unit 1643 of the motion vector calculation unit 164 (step S81 in FIG. 25). As the initial value of the basal plane minimum SAD value Smin, for example, the maximum value of the pixel difference is set.

  Next, the matching processing unit 163 sets the basal plane reference vector (Vx, Vy) in the basal plane reduction search range set in step S79, and sets the basal plane reference block position for calculating the SAD value (step). S82). Then, the set pixel data of the base plane reference block is read from the reference block buffer unit 162 (step S83), and the sum of absolute values of differences between the pixel data of the base plane target block and the base plane reference block, that is, the base plane The SAD value is obtained, and the obtained basal plane SAD value is sent to the motion vector calculation unit 164 (step S84).

  In the motion vector calculation unit 164, the SAD value comparison unit 1642 compares the basal plane SAD value Sin calculated by the matching processing unit 163 with the basal plane minimum SAD value Smin held in the SAD value holding unit 1643. Then, it is determined whether or not the calculated basal plane SAD value Sin is smaller than the basal plane minimum SAD value Smin held so far (step S85).

  When it is determined in this step S85 that the calculated basal plane SAD value Sin is smaller than the basal plane minimum SAD value Smin, the process proceeds to step S86, and the basal plane minimum SAD value Smin held in the SAD value holding unit 1643 and its position. Information is updated.

  That is, the SAD value comparison unit 1642 sends the comparison result information DET to the SAD value writing unit 1641 that the calculated basal plane SAD value Sin is smaller than the basal plane minimum SAD value Smin. Then, the SAD value writing unit 1641 sends the calculated base surface SAD value Sin and its position information (reference vector) to the SAD value holding unit 1643 as information of a new base surface minimum SAD value Smin. The SAD value holding unit 1643 updates the base surface minimum SAD value Smin to be held and its position information to the received new base surface SAD value Sin and its position information.

  After step S86, the process proceeds to step S87. If it is determined in step S85 that the calculated basal plane SAD value Sin is greater than the basal plane minimum SAD value Smin, the process proceeds to step S87 without performing the holding information update process in step S86.

  In step S87, the matching processing unit 163 determines whether or not the matching process has been completed at the positions of all base plane reference blocks (base plane reference vectors) in the base plane search range. When it is determined that there is an unprocessed base plane reference block, the process returns to step S82, and the processes after step S82 are repeated.

  When the matching processing unit 163 determines in step S87 that the matching processing has been completed at all base plane reference block positions (base plane reference vectors) in the base plane search range, the basis from the SAD value holding unit 1643 is determined. The position information (base plane motion vector) of the surface minimum SAD value Smin is received, the base plane SAD value is recalculated for the base plane reference block at the positions of the four neighboring points, and the recalculated vicinity base plane SAD value is SAD. The value is supplied to the SAD value holding unit 1643 through the value writing unit 1641 and held (step S88).

  Next, in the motion vector calculation unit 164, the X direction (horizontal direction) neighborhood value extraction unit 1644 and the Y direction (vertical direction) neighborhood value extraction unit 1645 are stored in the SAD value holding unit 1643 and the detected minimum The basal plane SAD value Smin and its neighboring basal plane SAD values Sx1, Sx2, Sy1, Sy2 and their position information are read out and sent to the quadratic curve approximation interpolation processing unit 1646. Receiving this, the quadratic curve approximation interpolation processing unit 1646 performs interpolation by the quadratic curve twice in the X direction and the Y direction, and calculates a high-precision basal plane motion vector with subpixel accuracy as described above. (Step S69). This completes the block matching process of this example for one reference frame.

  FIGS. 24 and 25 described above are cases where the second example described above is used as the motion vector calculation unit 164 of the motion detection / compensation unit 16, but it goes without saying that the first example can also be used. .

  Next, the effect of the image processing method according to the second embodiment will be described with a specific example.

  FIG. 26 shows specific examples of reference blocks, search ranges, and matching processing ranges on the base plane and reduced plane used in the description of the above-described embodiment. In the example of FIG. 26, the reduction ratio n in the horizontal direction and the vertical direction is set to n = 4.

  As a comparative example, as shown in FIGS. 26A and 26B, for example, the reference block 108 in the first embodiment that does not use a reduced image is horizontal (horizontal) × vertical (vertical) = 32 × 32. The search range 106 is horizontal (horizontal) × vertical (vertical) = 144 × 64 pixels, and the matching processing range 110 is horizontal (horizontal) × vertical (vertical) = 176 × 96 pixels.

  Then, in the second embodiment described above, the reduced surface reduced to 1 / n = 1/4 in the horizontal direction and the vertical direction, as shown in FIGS. 26C and 26D, The reference block 139 is horizontal (horizontal) × vertical (vertical) = 8 × 8 pixels, the reduction plane search range 137 is horizontal (horizontal) × vertical (vertical) = 36 × 16 pixels, and the reduction plane matching processing range 143 is Horizontal (horizontal) × vertical (vertical) = 44 × 24 pixels.

  When block matching is performed with a reduction plane reduced to ¼ in the horizontal direction and the vertical direction, the reduction plane motion vector is a motion vector with a precision of 4 pixels. An error occurs with respect to the precision motion vector. That is, if the pixels on the base surface are as shown in FIG. 27, the matching processing point 148 on the base surface covers all pixels on the base surface, but is reduced to ¼. This is because the matching processing points when performing block matching on the surface are in units of 4 pixels indicated by black circles in FIG.

  However, it can be predicted that at least a 1-pixel precision motion vector will exist within a 4-pixel range around the matching processing point on the reduced plane indicated by the reduced plane motion vector.

  Therefore, in the second embodiment, in the basal plane matching that determines the basal plane search range based on the calculated reduced plane motion vector, the pixel position indicated by the reference vector indicated by 4 times the reduced plane motion vector is centered. A search range for 4 pixels (base plane search range 140) is determined, base plane block matching is performed, and a motion vector is calculated again.

  Therefore, as shown in FIGS. 26E and 26F, the basal plane reference block 142 has horizontal (horizontal) × vertical (vertical) = 32 × 32 pixels, and the basal plane search range 140 has horizontal (horizontal) × The vertical (vertical) = 4 × 4 pixels and the basal plane matching processing range 144 is horizontal (horizontal) × vertical (vertical) = 40 × 40 pixels.

  In FIG. 28, in the case of performing two-layer matching of reduced plane matching and basal plane matching as in the second embodiment, in the case of assuming that the SAD table is used, Indicates the size of the SAD table. The example in FIG. 28 corresponds to the specific example shown in FIG.

  The SAD table TBL in the case of the 144 × 64 search range (see FIG. 26B) before reduction is 145 × 65 points as shown in FIG.

  On the other hand, the reduced surface SAD table in the case of a 36 × 16 reduced surface search range (see FIG. 26D) is 37 points × 17 points as shown in FIG.

  In addition, in the case of the 4 × 4 basal plane search range (see FIG. 26F), the basal plane SAD table is 5 points × 5 points.

  Therefore, the number of matching processes when layer matching is not performed is 145 × 65 = 9425 times, whereas the number of matching processes when layer matching is applied is 37 × 17 + 5 × 5 = 654 times, which greatly increases the processing time. It can be seen that it can be shortened.

  The line memory in the case of the first example of the motion vector detection method described above may be one line in the reduced-surface SAD table, so that it can store 37 SAD values and their position information. In the case of TBL, a very small memory is sufficient for the memory storing 9425 SAD values and their position information.

  Further, in the case of the second example of the motion vector detection method described above, even the reduced surface SAD table for storing the 37 SAD values and the position information thereof becomes unnecessary, so that the circuit scale can be further reduced. .

  As described above, according to the second embodiment, after performing the hierarchical matching process, the interpolation process is performed on the basal plane, so that motion vector detection with sub-pixel accuracy can be performed in a wide search range. It becomes possible to do.

[Third Embodiment]
A moving image NR system requires real-time performance, that is, speed as well as accuracy. Further, in a system in which various processing units are connected to the system bus 2 as shown in FIG. 1 and perform various processes in parallel, the bus bandwidth is also regarded as important. Here, the bus bandwidth is a data rate that can be transferred while avoiding congestion on the system bus.

  Speed and bus bandwidth and motion detection accuracy are in a trade-off relationship. While seeking motion detection accuracy, trying to increase processing speed or reduce bus bandwidth may also be a cost-effective problem. It must be.

  In the second embodiment described above, an example of performing motion vector detection of sub-pixels or less in a wide search range has been shown. However, in the third embodiment, the accuracy of motion detection is lowered and the processing speed is increased. A method for reducing the bus bandwidth will be described.

  In the second embodiment described above, after performing block matching with pixel accuracy on the base surface, motion vector detection with sub-pixel accuracy is performed by interpolation processing. On the other hand, in the third embodiment, after performing block matching on the reduction plane, the SAD value of the reduction plane reference block near the reduction plane reference block position indicated by the calculated reduction plane motion vector and Interpolation processing is performed using the position information, and motion vector detection with pixel accuracy is performed.

  As shown in FIG. 27, when block matching is performed on a reduced surface that is reduced to ¼ both vertically and horizontally, the reduced surface motion vector is a motion vector with 4-pixel accuracy. Therefore, as shown in FIG. 29, when the minimum SAD value 149 in the reduction plane is obtained, interpolation processing is performed using the SAD values 150, 151, 152, and 153 in the vicinity thereof to detect a motion vector with pixel accuracy. When trying to do so, the interpolation magnification needs to be four times.

  For example, as shown in FIG. 30, when the approximate interpolation process using the quadratic curve 120 is performed as the interpolation process, four times the interpolation magnification is divided (subtracted) in the above-described equations (1) and (2). ) Is equivalent to performing 2 bits. In FIG. 30, a minimum value (SAD value) 154 on the quadratic curve 120 indicates a minimum SAD value with pixel accuracy obtained by interpolation processing on the reduction plane.

  In the above description, in order to calculate a motion vector with pixel accuracy, the reduction magnification and the interpolation magnification are the same, but the two may not be the same. If the interpolation magnification is larger than the reduction magnification, a motion vector with higher accuracy than the pixel accuracy can be calculated.

  FIG. 31 is a flowchart showing an example of block matching and motion vector detection processing in the motion detection / compensation unit 16 in the case of the third embodiment.

  First, the motion detection / compensation unit 16 reads a reduced image of the target block, that is, a reduced surface target block from the target block buffer unit 161 (step S91 in FIG. 31). Next, the initial value of the reduced surface minimum SAD value is set as the initial value of the minimum SAD value Smin of the SAD value holding unit 1643 of the motion vector calculation unit 164 (step S92). As an initial value of the reduced surface minimum SAD value Smin, for example, a maximum value of pixel difference is set.

  Next, the matching processing unit 163 sets a reduction plane search range, sets a reduction plane reference vector (Vx / n, Vy / n) in the set reduction search range, and calculates a SAD value. A reference block position is set (step S93). Then, the pixel data of the set reduction plane reference block is read from the reference block buffer unit 162 (step S94), and the sum of absolute values of differences between the pixel data of the reduction plane target block and the reduction plane reference block, that is, the reduction plane The SAD value is obtained, and the obtained reduced surface SAD value is sent to the motion vector calculation unit 164 (step S95).

  In the motion vector calculation unit 164, the SAD value comparison unit 1642 compares the reduced surface SAD value Sin calculated by the matching processing unit 163 with the reduced surface minimum SAD value Smin held in the SAD value holding unit 1643. Then, it is determined whether or not the calculated reduced surface SAD value Sin is smaller than the reduced surface minimum SAD value Smin held so far (step S96).

  When it is determined in this step S96 that the calculated reduced surface SAD value Sin is smaller than the reduced surface minimum SAD value Smin, the process proceeds to step S97, and the reduced surface minimum SAD value Smin held in the SAD value holding unit 1643 and its position. Information is updated.

  That is, the SAD value comparison unit 1642 sends the comparison result information DET to the SAD value writing unit 1641 that the calculated reduction surface SAD value Sin is smaller than the reduction surface minimum SAD value Smin. Then, the SAD value writing unit 1641 sends the calculated reduced surface SAD value Sin and its position information (reference vector) to the SAD value holding unit 1643 as information on a new reduced surface minimum SAD value Smin. The SAD value holding unit 1643 updates the reduced reduction surface minimum SAD value Smin and its position information to be held to the received new reduction surface SAD value Sin and its position information.

  After step S97, the process proceeds to step S98. If it is determined in step S96 that the calculated reduced surface SAD value Sin is larger than the reduced surface minimum SAD value Smin, the process advances to step S98 without performing the holding information update process in step S97.

  In step S98, the matching processing unit 163 determines whether or not the matching process has been completed at the positions of all reduced surface reference blocks (reduced surface reference vectors) in the reduced surface search range. When it is determined that there is an unprocessed reduced-surface reference block, the process returns to step S93, and the processes after step S93 are repeated.

  If the matching processing unit 163 determines in step S98 that the matching process has been completed at the positions of all the reduced surface reference blocks (reduced surface reference vectors) in the reduced surface search range, the reduction processing from the SAD value holding unit 1643 is performed. The position information (reduced surface motion vector) of the surface minimum SAD value Smin is received, the reduced surface SAD value is recalculated for the reduced surface reference block at the positions of the four neighboring points, and the recalculated neighboring reduced surface SAD value is calculated as SAD. The value is supplied to the SAD value holding unit 1643 through the value writing unit 1641 and held (step S99).

  Next, in the motion vector calculation unit 164, the X direction (horizontal direction) neighborhood value extraction unit 1644 and the Y direction (vertical direction) neighborhood value extraction unit 1645 are stored in the SAD value holding unit 1643 and the detected minimum The reduced surface SAD value Smin and its adjacent reduced surface SAD values Sx1, Sx2, Sy1, Sy2 and their position information are read out and sent to the quadratic curve approximation interpolation processing unit 1646. Receiving this, the quadratic curve approximation interpolation processing unit 1646 performs the interpolation by the quadratic curve twice in the X direction and the Y direction on the reduction plane, and calculates the pixel precision motion vector as described above ( Step S100). This completes the block matching process of this example for one reference frame.

  The processing speed and the bus bandwidth are compared in the second embodiment and the third embodiment. According to the image processing method of the third embodiment, there are the following three advantages in processing speed and bus bandwidth.

  First, in the third embodiment, the SAD value is calculated only on the reduction plane, and the SAD value is not calculated on the base plane. Will be faster.

  That is, as shown in FIG. 28, as in the second embodiment, the number of SAD value calculations when applying the motion vector detection method using the hierarchical matching and the interpolation processing on the base plane is 37 × 17 + 5. In contrast to x5 = 654 times, the number of SAD value calculations when the interpolation process on the reduction plane is applied as in the third embodiment is 629 times, which can be reduced by 25 times.

  Secondly, in the third embodiment, since block matching is not performed on the basal plane, the basal plane target block and the basal plane matching processing range need not be read from the image memory. Therefore, not only the above 25 times of SAD value calculation, but also the reading time of the base block target block and the transfer time for all the pixels in the base plane matching processing range are shortened. At the same time, the bus bandwidth for 25 base planes is also reduced.

  Third, the basal plane block matching and the reduced plane block matching have the same time for calculating the SAD value, but the number of pixels transferred from the target block buffer unit 161 and the reference block buffer unit 162 to the matching processing unit 163 is the same. Different.

  For example, in the comparison of the above example, as shown in FIG. 26, the reduction plane reference block 139 is 8 × 8 pixels, the base plane reference block 142 is 32 × 32 pixels, and the number of transfer pixels is 16 times. Therefore, the presence / absence of basal plane block matching has a great influence on the processing time.

  In addition, the processing speed can be increased by calculating SAD values in parallel for a plurality of blocks by reduction plane block matching. When the parallel processing is performed, the matching processing range to be taken into the reference block buffer unit 162 becomes large. However, as shown in FIG. 32A, if the adjacent reduced surface reference blocks 171 and 172 are processed in parallel, most of the matching processing is performed. Ranges can be shared. For this reason, the bus bandwidth can also be reduced.

  FIGS. 32B and 32C process block matching of two reduced plane reference blocks 171 and 172 in parallel in the reduced plane reference block and reduced plane search range of the same size as in FIG. It is an example of a reduction plane search range and a matching processing range at the time.

  As shown in FIG. 26D, the reduction plane search range 137 without parallel processing is 36 × 16 pixels, and the reduction plane matching processing range 143 is 44 × 24 pixels, whereas FIG. As shown in (B) and (C), the reduction plane search range 173 of the reduction plane reference block 171 is 40 × 16 pixels, the reduction plane search range 174 of the reduction plane reference block 172 is 40 × 16 pixels, and The reduction plane matching processing range 175 corresponding to both the reduction plane reference blocks 171 and 172 is 52 × 24 pixels.

  As described above, in the basal plane block matching performed in the second embodiment, since the basal plane matching processing range is determined by the value of the reduced plane motion vector, the matching processing range cannot be shared during parallel processing. Therefore, in the case of the second embodiment, when the speed is increased by parallelizing the calculation of the SAD value, the processing time and the bus band required for the basal plane matching process become dominant.

  On the other hand, according to the third embodiment, as described above, in the hierarchical matching process, after the reduced surface block matching is performed, the base surface block matching is not performed, and the interpolation on the reduced surface is performed. By performing the processing, it is possible to realize a high-speed processing and a reduction in bus bandwidth.

[Other Embodiments and Modifications]
In the first embodiment, in the first example of the motion vector calculation unit, the search direction in the search range is set to the horizontal line direction, and the search is performed by moving the reference block in order from the upper left of the search range, for example. In addition, a memory for one line of the SAD table is provided, but the search direction in the search range is set to the vertical direction, for example, the search is started in the vertical direction from the upper left end of the search range, and one column in the vertical direction is set. After the search for minutes is completed, the procedure is repeated in which the position of the reference block is moved to the vertical column on the right by one pixel, for example, one pixel in the horizontal direction, and the search is performed in the vertical direction from the upper end of the column. You may make it employ | adopt a search method. As described above, when the search is performed by moving the reference block in the vertical direction sequentially from the upper left end of the search range, a memory for one column in the vertical direction of the SAD table may be provided.

  Here, whether the search direction is taken in the horizontal direction or the search direction in the vertical direction is determined by taking into account the circuit scales of the matching processing unit and the motion vector calculation unit and adopting the one with a smaller circuit scale. preferable.

  As described above, the reference block may be moved not for each pixel and for each line, but for each of a plurality of pixels and for each of a plurality of lines. Therefore, the memory for one line in the horizontal direction in the former case only needs to correspond to the movement position of the reference block in the horizontal direction, and the memory for one column in the vertical direction in the latter case moves the reference block in the vertical direction. Just the position. That is, when the reference block is moved for each pixel or for each line, the memory for one line needs a memory having the capacity corresponding to the number of pixels for one line, and for one column in the vertical direction. As the memory, a memory having a capacity corresponding to the number of lines is required. However, when the reference block is moved for each of a plurality of pixels and for each of a plurality of lines, the memory for one line and the memory for one column are compared with the case where the reference block is moved for each pixel and for each line. Less capacity.

  Further, the interpolation processing method is not limited to the above-described quadratic curve approximation interpolation processing, and as described above, interpolation processing using a cubic curve or a higher order curve may be performed.

  Moreover, although the above-mentioned embodiment is a case where the image processing apparatus by this invention is applied to an imaging device, this invention is not necessarily restricted to an imaging device, When detecting the motion between image frames, It is applicable to all cases where block matching is used.

  The above-described embodiment is a case where the present invention is applied when detecting a motion vector in units of blocks for noise reduction processing by image superposition. It goes without saying that it can also be used for detection. The motion vector due to camera shake can be obtained, for example, as an average value of a plurality of block motion vectors.

1 is a block diagram illustrating a configuration example of an imaging apparatus that is an embodiment of an image processing apparatus according to the present invention. It is a figure for demonstrating the noise reduction process of the captured image in the imaging device of FIG. 1 of embodiment. It is a figure used for description of the block matching process in the imaging device of FIG. 1 of embodiment. It is a figure for demonstrating the noise reduction process of the captured image in the imaging device of FIG. 1 of embodiment. It is a figure for demonstrating the noise reduction process of the captured image in the imaging device of FIG. 1 of embodiment. It is a block diagram which shows the structural example of the motion detection and the motion compensation part in the imaging device of FIG. 1 of embodiment. It is a figure used in order to demonstrate the operation | movement in the motion detection and the motion compensation part in the imaging device of FIG. 1 of embodiment. It is a figure used in order to demonstrate the operation | movement in the motion detection and the motion compensation part in the imaging device of FIG. 1 of embodiment. It is a figure for demonstrating the effect by the process in the motion detection and the motion compensation part in the imaging device of FIG. 1 of embodiment. It is a figure which shows the flowchart for demonstrating 1st Embodiment of the image processing method by this invention. It is a figure which shows the flowchart for demonstrating 1st Embodiment of the image processing method by this invention. It is a block diagram which shows the structural example compared with the structure of the motion vector calculation part of the motion detection / motion compensation part in the imaging device of FIG. 1 of embodiment. It is a figure which shows the flowchart for demonstrating the processing operation of the comparative example of FIG. It is a figure used in order to demonstrate the example of the motion vector calculation part of the motion detection / motion compensation part in the imaging device of FIG. 1 of embodiment. It is a figure used in order to demonstrate the 1st example of the motion vector calculation part of the motion detection / motion compensation part in the imaging device of Drawing 1 of an embodiment. It is a block diagram which shows the structural example of the 1st example of the motion vector calculation part of the motion detection / motion compensation part in the imaging device of FIG. 1 of embodiment. It is a figure which shows a part of flowchart for demonstrating the processing operation in the 1st example of the motion vector calculation part of the motion detection / motion compensation part in the imaging device of FIG. 1 of embodiment. It is a figure which shows a part of flowchart for demonstrating the processing operation in the 1st example of the motion vector calculation part of the motion detection / motion compensation part in the imaging device of FIG. 1 of embodiment. It is a block diagram which shows the structural example of the 1st example of the motion vector calculation part of the motion detection / motion compensation part in the imaging device of FIG. 1 of embodiment. It is a figure which shows the flowchart for demonstrating the processing operation in the 1st example of the motion vector calculation part of the motion detection / motion compensation part in the imaging device of FIG. 1 of embodiment. It is a figure used in order to explain the image processing method of a 2nd embodiment of this invention. It is a figure used in order to explain the image processing method of a 2nd embodiment of this invention. It is a figure used in order to explain the image processing method of a 2nd embodiment of this invention. It is a figure which shows a part of flowchart for demonstrating the image processing method of 2nd Embodiment of this invention. It is a figure which shows a part of flowchart for demonstrating the image processing method of 2nd Embodiment of this invention. It is a figure used in order to demonstrate the specific example in the image processing method of the 2nd Embodiment of this invention. It is a figure used in order to demonstrate the specific example in the image processing method of the 2nd Embodiment of this invention. It is a figure used in order to demonstrate the effect of the image processing method of the 2nd Embodiment of this invention. It is a figure used for description of the image processing method of the 3rd Embodiment of this invention. It is a figure used for description of the image processing method of the 3rd Embodiment of this invention. It is a figure which shows the flowchart for demonstrating the image processing method of 3rd Embodiment of this invention. It is a figure used for description of the image processing method of the 3rd Embodiment of this invention. It is a figure used in order to explain block matching processing. It is a figure used in order to explain block matching processing. It is a figure used in order to explain block matching processing. It is a figure used in order to explain block matching processing. It is a figure used in order to explain block matching processing. It is a figure used in order to explain block matching processing.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 16 ... Motion detection / motion compensation part, 17 ... Image superimposition part, 100 ... Target image (target frame), 101 ... Reference image (reference frame), 102 ... Target block, 104 ... Motion vector 106 ... Search range, 107 ... Reference vector 108... Reference block 163 Matching processing unit 164 Motion vector calculation unit 1643 SAD value holding unit 1646 Quadratic curve approximation interpolation processing unit

Claims (4)

A plurality of reference blocks having the same size as a target block of a predetermined size composed of a plurality of pixels set in the target screen are set in a search range set in a reference screen different from the target screen, and the plurality In the image processing apparatus for detecting a motion vector for the target block based on a positional deviation of the reference block having the strongest correlation with the target block on the screen with respect to the target block.
A correlation value between each of the reference blocks and the target block is calculated using pixel values of the plurality of pixels in the reference block and pixel values of a plurality of pixels at corresponding positions in the target block. A strongest correlation reference block detecting means for detecting a position of the strongest correlation reference block having the strongest correlation with the target block;
Obtaining means for obtaining the strongest correlation value for the strongest correlation reference block and the correlation values for a plurality of neighboring reference blocks in the vicinity of the position of the strongest correlation reference block;
Interpolation processing is performed using the strongest correlation value of the strongest correlation reference block acquired by the acquisition unit and the plurality of correlation values of the plurality of neighboring reference blocks, and is smaller than the pixel pitch of the target screen. with high accuracy, the interpolation correlation with the target block to detect the position of the strongest high definition strongest correlation reference block, calculates the pre kidou-out vector as positional deviation from the target block of the high-definition strongest correlation reference block Processing means;
With
The strongest correlation reference block detection means includes:
A holding unit for holding information on the strongest correlation value and the position of the strongest correlation reference block;
The newly calculated new correlation value of the reference block with respect to the target block is compared with the strongest correlation value of the holding unit, and when the new correlation value is determined to have a stronger correlation, Update the information on the strongest correlation value and its position of the holding unit to the information on the new correlation value and its position,
It said acquisition means, at least, about double several adjacent reference blocks in the vicinity of the strongest correlation reference block held in the holding portion, by re-calculating the correlation value, the correlation value for the neighborhood reference block An image processing apparatus characterized by acquiring The image processing apparatus according to claim 1 .
Dividing the target screen into a plurality of pieces, setting a plurality of the target blocks,
Detecting the motion vector for each of the plurality of target blocks;
An image processing apparatus, wherein low-noise image information is obtained by superimposing a plurality of images in units of the divided screens using the detected motion vectors for the plurality of target blocks. The image processing apparatus according to claim 1 .
The sum of absolute values of differences between pixel values of a plurality of pixels in the reference block and pixel values of a plurality of pixels at corresponding positions in the target block is calculated as the correlation value. Processing equipment. A plurality of reference blocks having the same size as a target block of a predetermined size composed of a plurality of pixels set in the target screen are set in a search range set in a reference screen different from the target screen, and the plurality In the image processing method for detecting a motion vector for the target block based on a positional shift of the reference block having the strongest correlation with the target block on the screen relative to the target block.
A correlation value between each of the reference blocks and the target block is calculated using pixel values of the plurality of pixels in the reference block and pixel values of a plurality of pixels at corresponding positions in the target block. A strongest correlation reference block detecting step for detecting a position of the strongest correlation reference block having the strongest correlation with the target block;
Obtaining the strongest correlation value for the strongest correlation reference block and the correlation values for a plurality of neighboring reference blocks in the vicinity of the position of the strongest correlation reference block;
Interpolation is performed using the strongest correlation value of the strongest correlation reference block acquired in the acquisition step and the plurality of correlation values of the plurality of neighboring reference blocks, and is smaller than the pixel pitch of the target screen. An interpolation process step of detecting a position of a high-definition strongest correlation reference block having the strongest correlation with the target block with high accuracy and calculating the motion vector as a positional deviation of the high-definition strongest correlation reference block from the target block When,
Is executed by the image processing apparatus,
The image processing apparatus includes:
A holding unit for holding information on the strongest correlation value and the position of the strongest correlation reference block;
In the strongest correlation reference block detection step,
The newly calculated new correlation value of the reference block with respect to the target block is compared with the strongest correlation value of the holding unit, and when the new correlation value is determined to have a stronger correlation, Update the information on the strongest correlation value and its position of the holding unit to the information on the new correlation value and its position,
In the acquisition step,
At least, the plurality of near-neighbor reference blocks in the vicinity of the strongest correlation reference block held in the holding portion, by re-calculating the correlation values, to obtain a correlation value for said neighboring reference blocks A featured image processing method.

Images