JP2009116763A - Image processing apparatus, and memory access method for image data - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing method capable of reducing a bus band when a plurality of image data processing parts is connected to an image memory via a system. <P>SOLUTION: A memory controller writes an image data of results processed in the first image processing part, into the first memory area of an image memory part, by the first block unit with a combination of the number of vertical-directional lines of a screen and the number of horizontal-directional pixels thereof serving as the first combination, and writes the image data, into the second memory area of the image memory part, by the second block unit with a combination of the number of vertical-directional lines of the screen and the number of horizontal-directional pixels thereof serving as the second combination different from the first combination. The controller reads out the image data written into the first memory area of the image memory part, to be supplied to the second image processing part, and reads out the image data written into the second memory area of the image memory part, to be supplied to the third image processing part. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  According to the present invention, an image memory unit and various image processing units are connected via a system bus, and the various image processing units execute respective image data processing while accessing the image memory. More particularly, the present invention relates to a method for accessing image data to an image memory unit.

  In this specification, the screen means an image that is composed of image data for one frame or one field and is displayed on the display as one image.

  A block matching method for obtaining a motion vector between two screens from image information itself is an old technology. Development has been centered on pan / tilt detection of TV cameras, subject tracking, moving picture experts group (MPEG) video coding, and sensorless camera shake correction by superimposing images since the 1990s. Applications such as noise reduction (Noise Reduction: hereinafter referred to as NR) at the time of low-illuminance photography are being promoted.

In block matching, a motion vector between two screens between a reference screen that is a target screen and an original screen (referred to as a target screen) that is a source of motion of the reference screen is a block of a rectangular area having a predetermined size. Is a method of calculating by calculating the correlation between the reference screen and the original screen When the original screen is temporally before the reference screen (for example, in the case of motion detection in MPEG) In some cases, the reference screen is temporally the screen before the original screen (for example, in the case of noise reduction by superimposing image frames described later).

  In this specification, the screen means an image that is composed of one frame or one field of image data and is displayed on the display as one frame. In the following description of this specification, For convenience, the screen may be referred to as a frame, assuming that the screen is composed of one frame. For example, the reference screen may be referred to as a reference frame, and the original screen may be referred to as an original frame.

  FIGS. 64 to 69 are diagrams for explaining the outline of the conventional block matching. In the block matching method described here, for example, as shown in FIG. 64 (A), an original frame (target frame) 100 has a predetermined size composed of a plurality of pixels in the horizontal direction and a plurality of lines in the vertical direction, respectively. Divide into rectangular areas (called blocks). Each of the plurality of blocks 102 in the target frame is referred to as a target block.

  In block matching, a block having a high correlation with the target block 102 is searched from the reference frame 101. As a result of this search, the block 103 (see FIG. 64B) detected in the reference frame 101 as having the highest correlation is referred to as a motion compensation block. In addition, the amount of positional deviation between the target block 102 and the motion compensation block 103 is referred to as a motion vector (see reference numeral 104 in FIG. 64B).

  A motion vector 104 corresponding to a position shift (including a position shift amount and a position shift direction) between the target block 102 and the motion compensation block 103 is the same as the position of each target block 102 of the target frame 100 in the reference frame 101. Assuming that the projection image block 109 of the target block 102 is assumed as the position, a positional shift between the position of the projection image block 109 of the target block (for example, the center position) and the position of the motion compensation block 103 (for example, the center position). It also has a positional deviation amount and a positional deviation direction component.

  An outline of the block matching process will be described. As shown by a dotted line in FIG. 65, a projected image block 109 of the target block is assumed at the same position as the position of each target block 102 of the target frame 100 in the reference frame 101, and the center of the projected image block 109 of this target block is assumed. The coordinates are set as a motion detection origin 105. Then, assuming that the motion vector 104 exists within a certain range from the motion detection origin 105, a predetermined range centered on the motion detection origin 105 is set as the search range 106 (see the one-dot chain line in FIG. 65). .

  Next, a block (referred to as a reference block) 108 having the same size as the target block 102 is set on the reference screen. The position of the reference block 108 is moved within the search range 106 in units of one pixel or a plurality of pixels, for example, in the horizontal direction and the vertical direction. Accordingly, a plurality of reference blocks 108 are set in the search range 106.

  Here, the reference block 108 is moved within the search range 106. In this example, since the motion detection origin 105 is the center position of the target block, the center position of the reference block 108 is moved within the search range 106. This means that the pixels constituting the reference block 108 may protrude beyond the search range 106.

  Then, for each reference block 108 to be set in the search range, a vector (referred to as a reference vector) 107 (refer to FIG. 65) representing the positional deviation amount and the positional deviation direction between each reference block 108 and the target block 102. Then, the correlation between the image content of the reference block 108 at the position indicated by each reference vector 107 and the image content of the target block 102 is evaluated.

  As shown in FIG. 66, the reference vector 107 is a vector (Vx, Vy) where the horizontal direction (X direction) displacement amount Vx of the reference block 108 and the vertical direction (Y direction) displacement amount is Vy. When the position coordinates (for example, center position coordinates) of the reference block 108 and the position coordinates (for example, center position coordinates) of the target block 102 are the same, the reference vector 107 is represented as a vector (0, 0). .

  For example, when the reference block 108 is at a position shifted by one pixel in the X direction from the position of the target block 102, the reference vector 107 is a vector (1, 0). As shown in FIG. 67, when the reference block 108 is at a position shifted by 3 pixels in the X direction and 2 pixels in the Y direction from the position of the target block 102, the reference vector 107 is a vector (3, 2 )

  That is, as shown in the example of FIG. 67, when the positions of the target block 102 and the reference block 108 are the center positions of the respective blocks, each reference vector 108 has a corresponding reference block 108. This means a displacement between the center position and the center position of the target block 102 (a vector including a displacement amount and a displacement direction).

  The reference block 108 moves within the search range 106. In this case, the center position of the reference block 108 moves within the search range 106. As described above, since the reference block 108 includes a plurality of pixels in the horizontal direction and the vertical direction, the maximum range in which the reference block 108 that is subject to block matching processing with the target block 102 is as shown in FIG. Furthermore, the matching processing range 110 is wider than the search range 106.

  Then, the position of the reference block 108 detected as having the strongest correlation with the image content of the target block 102 is detected as the position of the target block 102 of the target frame 100 in the reference frame 101 (position after moving), The detected reference block is set as the motion compensation block 103 described above. Then, the displacement amount between the detected position of the motion compensation block 103 and the position of the target block 102 is detected as a motion vector 104 as an amount including a direction component (see FIG. 64B). ).

  Here, the correlation value indicating the strength of correlation between the target block 102 and the reference block 108 moving in the search range 106 is basically calculated using the pixel values corresponding to the target block 102 and the reference block 108. However, as the calculation method, a method using a root mean square and various other methods have been proposed.

  Among them, as a correlation value generally used when calculating a motion vector, for example, the absolute value of the difference between the luminance value of each pixel in the target block 102 and the luminance value of each corresponding pixel in the reference block 106 is used. The sum of the values of all the pixels in the block (the sum of the absolute values of the differences is referred to as the sum of absolute values of the difference. Hereinafter, the sum of the absolute values of the differences is referred to as SAD (Sum of Absolute Difference)). Used (see FIG. 68).

  When the SAD value is used as the correlation value, the smaller the SAD value, the stronger the correlation. Therefore, among the reference blocks 108 that move in the search range 106, the reference block 108 at the position where the SAD value is minimum becomes the strongest correlation reference block with the strongest correlation, and this strongest correlation reference block is detected as the motion compensation block 103. The amount of displacement of the detected motion compensation block 103 relative to the position of the target block 102 is detected as a motion vector.

  As described above, in block matching, the amount of positional deviation of each of the plurality of reference blocks 108 set in the search range 106 with respect to the position of the target block 102 is expressed by the reference vector 107 as an amount including a direction component. The The reference vector 107 of each reference block 108 has a value corresponding to the position of the reference block 108 on the reference frame 102. As described above, in block matching, the reference vector of the reference block 108 in which the SAD value that is the correlation value has the minimum value is detected as the motion vector 104.

  Therefore, in block matching, generally, as shown in FIG. 69, SAD values between each of the plurality of reference blocks 108 set in the search range 106 and the target block 102 (hereinafter referred to as reference block for the sake of simplicity). The reference vector 107 corresponding to the position of each reference block 108 (hereinafter referred to as the reference vector 107 corresponding to the position of the reference block 106 is referred to as the reference vector 107 of the reference block 108 for the sake of simplicity). By detecting the reference block 108 having the smallest SAD value from the SAD values for all the reference blocks 108 stored in the memory. The vector 104 is detected.

  Corresponding to each of the reference vectors 107 corresponding to the positions of the plurality of reference blocks 108 set in the search range 106, a correlation value (in this example, SAD value) for each reference block 108 is stored. This is called a correlation value table. In this example, since the SAD value which is the sum of absolute differences is used as the correlation value, this correlation value table will be referred to as a difference absolute value sum table (hereinafter referred to as SAD table).

  This is shown in the SAD table TBL of FIG. 69. In this SAD table TBL, the correlation value (SAD value in this example) for each reference block 108 is referred to as a correlation value table element. In the example of FIG. 69, the SAD value indicated by reference numeral 111 is the SAD value when the reference vector is the vector (0, 0). In the example of FIG. 69, since the minimum value of the SAD value is “7” when the reference vector is the vector (3, 2), the motion vector 104 to be obtained is (3, 2).

  In the above description, the positions of the target block 102 and the reference block 108 mean arbitrary specific positions of these blocks, for example, the center position, and the reference vector 107 is the target block in the reference frame 102. The amount of deviation (including direction) between the position of the projection image block 109 of 102 and the position of the reference block 108 is shown.

  Since the reference vector 107 corresponding to each reference block 108 is a position shift of each reference block 108 from the position of the projection image block 109 corresponding to the target block 102 on the reference frame 101, the reference block When the position 108 is specified, the value of the reference vector is also specified corresponding to the position. Therefore, when the address of the correlation value table element of the reference block in the memory of the SAD table 110 is specified, the corresponding reference vector is specified.

  Note that the SAD value may be calculated for two or more target blocks at the same time. As the number of target blocks to be processed simultaneously increases, the processing speed increases. However, since the hardware scale for calculating the SAD value increases, there is a trade-off between speeding up the processing and increasing the circuit scale.

  As described above, the block matching method divides the target frame into a plurality of target blocks, sets a search range in the reference frame for each target block, and refers to the target block of the target frame from the image memory. A reference block in the frame search range is read, the SAD value of each pixel is calculated, a SAD value table corresponding to the width of the search range is formed, and the coordinate value of the minimum value is used as a motion vector for the target block.

  The image memory and the motion vector detection unit are connected through a system bus, and writing and reading to and from the image memory are controlled by the memory controller.

  As described above, in the block matching method, the SAD value for the pixel unit data from the image memory is calculated, so that the image is proportional to the number of pixels of the image itself and the width of the search range. Access to the memory increases and the bus bandwidth needs to be increased. Here, the bus bandwidth is an amount including a data rate, a bus width (number of bits), and the like at which data can be transferred while avoiding congestion on a bus for transferring data. When the bus band is increased, there is a problem that the scale of the system is increased and the cost is increased.

  To solve this problem, conventionally, if the image is in a stationary state with no motion, a method that does not perform a matching process (eg, Patent Document 1 (Japanese Patent Laid-Open No. 2005-210503)) or a reference frame image is thinned to obtain an information amount. A method of dropping the image (Patent Document 2 (Japanese Patent Laid-Open No. 2006-287583) and the like) has been proposed.

  Further, as a method of reducing the bus bandwidth without reducing the block matching accuracy, a method of holding only the shared part of the reference block of the reference frame and updating only the added part (Japanese Patent Laid-Open No. 2006-340139). No.)) is effective.

The above-mentioned reference patent documents are as follows.
JP 2005-210503 A JP 2006-287583 A JP 2006-340139 A

  However, the methods of Patent Literature 1 and Patent Literature 2 have a problem that the accuracy of block matching is lowered and can be applied only to a limited application in which high accuracy is not required.

  Further, according to the method of Patent Document 3, the bus bandwidth can be reduced without degrading the block matching accuracy, but the order of the blocks for which the matching process is performed becomes important, and the adjacent blocks are not continuously matched. When the order of matching processing cannot be specified, there is a problem that the bus bandwidth cannot be reduced.

  In addition, when a plurality of image data processing units other than the block matching processing unit are connected to the image memory via the system bus, the memory access method of the image data with respect to the image memory is determined by the block matching processing. For this reason, even if an efficient method is adopted, the memory access method of the image data for the image memory is not suitable for other image data processing units, and the bus bandwidth is likely to increase. There is a problem that there is.

  In view of the above points, the present invention reduces the bus bandwidth while avoiding the above-mentioned problems when a plurality of image data processing units are connected to the image memory via the system bus. An object of the present invention is to provide an image processing apparatus and a memory access method for image data.

In order to solve the above problems, the present invention provides:
An image memory unit capable of recording image data for a plurality of screens;
A first image processing unit that performs first processing on input image data and stores image data of the processing result in the image memory unit;
A second image processing unit that reads out the image data of the processing result from the image memory unit and performs a second process;
A third image processing unit that reads out the image data of the processing result from the image memory unit and performs a third process;
A bus to which the image memory unit, the first image processing unit, the second image processing unit, and the third image processing unit are connected;
A memory controller that controls writing and reading of image data via the bus to the image memory unit;
With
The image data as a result of the first processing in the first image processing unit has the first combination of the number of lines in the vertical direction and the number of pixels in the horizontal direction on the screen by the memory controller. The second block is written in the first memory area of the image memory unit in the first block unit, and the combination of the number of lines in the vertical direction and the number of pixels in the horizontal direction of the screen is different from the first combination. And is written in the second memory area of the image memory unit in a second block unit that is a combination of
The second image processing unit is read and supplied with the image data written in the first memory area of the image memory unit by the memory controller,
Provided to the third image processing unit is an image processing device in which the memory controller reads and supplies the image data written in the second memory area of the image memory unit. To do.

  In the present invention having the above-described configuration, the image data obtained as a result of processing in the first image processing unit is improved in bus efficiency and can reduce the bus bandwidth in the image processing in the second image processing unit. The data can be written to, read from, and transferred to the second image processing unit in units of one block.

  At the same time, the image data obtained as a result of processing in the first image processing unit is imaged in units of the second block in which the bus efficiency is improved and the bus bandwidth can be reduced in the image processing in the third image processing unit. The data can be written to and read from the memory unit and transferred to the third image processing unit.

  According to the present invention, even when a plurality of image processing units are connected to the image memory via the system bus, the image memory unit has an optimum image data format for each of the plurality of image processing units. Can be accessed, and the bus bandwidth can be reduced.

  Hereinafter, an embodiment of an image processing apparatus according to the present invention will be described with reference to the drawings, taking the case of an imaging apparatus as an example. In addition, as image processing, a motion vector between two screens is detected by block matching, a motion compensated image is generated using the detected motion vector, and the generated motion compensated image and a noise reduction target image are superimposed to reduce noise. The case of performing is described as an example.

[Imaging Device as an Embodiment of an Image Processing Device According to the Invention]
In the imaging apparatus according to the embodiment described below, as shown in FIG. 2, a plurality of images, for example, P1, P2, and P3, which are continuously captured, are aligned using motion detection and motion compensation. Thereafter, by superimposing, an image Pmix with reduced noise can be obtained. That is, since noise in each of a plurality of images is random, the noise is reduced with respect to the images by superimposing images having the same content.

  In the following description, the noise reduction by superimposing a plurality of images using motion detection and motion compensation is referred to as NR (Noise Reduction), and the image whose noise is reduced by NR is referred to as an NR image. To do.

  In this specification, a screen (image) to be subjected to noise reduction is defined as a target screen (target frame), and a screen to be superimposed is defined as a reference screen (reference frame). Images taken consecutively are misaligned due to the camera shake of the photographer, etc., and alignment is important to superimpose both images. Here, it is necessary to consider that there is a movement of the subject in the screen as well as a blur of the entire screen such as a camera shake.

  For this reason, in order to enhance the noise reduction effect for the subject as well, as shown in FIG. 3, it is necessary to align each of the plurality of target blocks 102 generated by dividing the target frame 100. Become.

  Accordingly, in this embodiment, a motion vector (hereinafter referred to as a block motion vector) 104B is detected for each of the plurality of target blocks 102, and the corresponding block motion vector 104B is detected for each target block 102. To align the images and superimpose the images.

  In the image pickup apparatus of this embodiment, at the time of still image shooting, as shown in FIG. 4, a plurality of images are shot at high speed, the first shot image is used as the target frame 100, and the second and subsequent frames. A predetermined number of photographed images are used as a reference frame 101 and are superimposed, and the superimposed image is recorded as a still image photographed image. That is, when the photographer depresses the shutter button of the imaging device, the predetermined number of images are photographed at a high speed, and a plurality of images (frames) photographed later in time for the first image (frame). One sheet of image (frame) is superimposed.

  At the time of moving image shooting, as shown in FIG. 5, the current frame image output from the image sensor is used as the target frame 100 image, and the previous image of the previous frame is used as the reference frame 101 image. Therefore, in order to reduce the noise of the current frame image, the image of the previous frame of the current frame is superimposed on the current frame.

  In the description of the method for superimposing the images in FIGS. 4 and 5 described above, when the frame rate of the moving image to be recorded is 60 fps (frame / second), the frame rate of 60 fps is obtained from the image sensor. This is a case where two image frames are superimposed, and as a result, a captured image signal is obtained at a frame rate of 60 fps with reduced noise.

  However, when the captured image is output from the image sensor at a higher frame rate of, for example, 240 fps, even when capturing a moving image, one image is superimposed on each other. It is also possible to obtain a captured image signal having a frame rate of 60 fps by generating a moving image frame. Needless to say, 240 fps captured moving image can be obtained by superimposing two image frames in the same manner as in this example, and as a result, a noise-captured captured image signal having a frame rate of 240 fps can be obtained.

  In this embodiment, the accuracy of the pixel pitch of the image to be processed (referred to as pixel (pixel) accuracy) is higher, that is, the original screen so that the images can be superimposed with higher accuracy. The block motion vector is detected with high accuracy with a pitch smaller than the pixel pitch of (target frame) (this high accuracy will be referred to as sub-pixel accuracy). In order to calculate the motion vector with the sub-pixel accuracy, as will be described later, in this embodiment, the interpolation processing is performed using the motion vector obtained with the pixel accuracy and the reference vector in the vicinity thereof. .

  FIG. 1 shows a block diagram of an example of an imaging apparatus as an embodiment of the image processing apparatus of the present invention.

  As shown in FIG. 1, in the imaging apparatus of this embodiment, a CPU (Central Processing Unit) 1 is connected to a system bus 2, and an imaging signal processing system 10 and a user operation input unit 3 are connected to the system bus 2. The image memory unit 4 and the recording / reproducing device unit 5 are connected to each other. In this specification, the CPU 1 includes a ROM (Read Only Memory) and a work area RAM (Random Access Memory) that store programs for performing various software processes, although not shown. .

  The image memory unit 4 is connected to the system bus 2 via a memory controller 8 that controls writing to and reading from the image memory unit 4.

  In response to an imaging recording start operation through the user operation input unit 3, the imaging signal processing system of the imaging apparatus in FIG. 1 performs recording processing of captured image data as described later. In response to an operation for starting the reproduction of the captured image through the user operation input unit 3, the imaging signal processing system 10 of the imaging apparatus in FIG. 1 reproduces the captured image data recorded on the recording medium of the recording / reproduction device unit 5. Perform processing. Note that each unit described later of the imaging signal processing system 10 receives a control command from the CPU 3 through the control register unit 7 and executes each process while receiving control from the CPU 3.

  As shown in FIG. 1, in the imaging signal processing system 10, incident light from a subject through a camera optical system (not shown) provided with an imaging lens 10 </ b> L is irradiated to the imaging element 11 and imaged. In this example, the imaging device 11 is configured by a CCD (Charge Coupled Device) imager. Note that the image sensor 12 may be a CMOS (Complementary Metal Oxide Semiconductor) imager.

  In the imaging apparatus of this example, when an imaging recording start operation is performed, a video input through the lens 10L is converted into a captured image signal by the imaging element 11, and a signal synchronized with the timing signal from the timing signal generation unit 12 As a result, an analog imaging signal, which is a RAW signal (raw signal) in a Bayer array composed of three primary colors of red (R), green (G), and blue (B), is output. The output analog imaging signal is supplied to the preprocessing unit 13, subjected to preprocessing such as defect correction and γ correction, and supplied to the data conversion unit 14.

  The data converter 14 converts the analog imaging signal, which is a RAW signal input thereto, into a digital imaging signal (YC data) composed of a luminance signal component Y and a color difference signal component Cb / Cr, and the digital image The imaging signal is supplied to the image correction / resolution conversion unit 15. The image correction / resolution conversion unit 15 converts the digital imaging signal to the resolution designated through the user operation input unit 3 and supplies the converted signal to the image memory unit 4 via the system bus.

  When the shooting instruction through the user operation input unit 3 is a still image shooting instruction by pressing the shutter button, the digital imaging signal whose resolution has been converted by the image correction / resolution conversion unit 15 includes the above-described plurality of frames in the image memory. Part 4 is written. Then, after images for a plurality of frames are written in the image memory unit 4, the image data of the target frame and the image data of the reference frame are read by the motion detection / motion compensation unit 16, and in this embodiment as will be described later. A block matching process is performed to detect a motion vector, and based on the detected motion vector, an image superimposing process as described later is performed by the image superimposing unit 17, and as a result of the superimposition, noise is reduced. The image data of the NR image is stored in the image memory unit 4.

  Then, the image data of the superposed NR image stored in the image memory unit 4 is codec converted by the still image codec unit 18, and for example, a DVD (Digital Versatile Disc) or the like of the recording / reproducing device unit 5 through the system bus 2. It is recorded on a recording medium such as a hard disk. In this embodiment, the still image codec unit 18 performs image compression encoding processing on a still image of JPEG (Joint Photographic Experts Group) system.

  At the time of still image shooting, before the shutter button is pressed, the image data from the image correction / resolution conversion unit 15 is supplied to the NTSC (National Television System Committee) encoder 20 through the image memory unit 4. This NTSC encoder 20 converts it into an NTSC standard color video signal and supplies it to a monitor display 6 comprising, for example, an LCD (Liquid Crystal Display), and a monitor image at the time of still image shooting is displayed on the display screen. The

  When the shooting instruction through the user operation input unit 3 is a moving image shooting instruction by pressing the moving image recording button, the resolution-converted image data is written in the image memory unit 4 and the motion detection / movement is performed in real time. A block matching process in this embodiment, which will be described later, is sent to the compensation unit 16 to detect a motion vector. Based on the detected motion vector, an image superimposing unit 17 performs image processing as described later. An overlay process is performed, and the image data of the NR image with the noise reduced as a result of the overlay is stored in the image memory unit 4.

  The image data of the superimposed NR image stored in the image memory unit 4 is codec-converted by the moving image codec unit 19 while being output to the display screen of the monitor display 6 through the NTSC encoder unit 20, and the system bus 2 Is supplied to the recording / reproducing apparatus unit 5 and recorded on a recording medium such as a DVD or a hard disk. In this embodiment, the moving image codec unit 18 performs image compression coding processing on moving images of the MPEG (Moving Picture Experts Group) method.

  The captured image data recorded on the recording medium of the recording / reproducing apparatus unit 5 is read in response to a reproduction start operation through the user operation input unit 3, supplied to the moving image codec unit 19, and reproduced and decoded. The reproduced and decoded image data is supplied to the monitor display 6 through the NTSC encoder 20, and the reproduced image is displayed on the display screen. Although not shown in FIG. 1, the output video signal from the NTSC encoder 20 can be derived outside through a video output terminal.

  The motion detection / compensation unit 16 described above can be configured by hardware, or can be configured by using a DSP (Digital Signal Processor). Furthermore, software processing can be performed by the CPU 1.

  Similarly, the image superimposing unit 17 can also be configured by hardware or can be configured by using a DSP. Furthermore, software processing can be performed by the CPU 1. The same applies to the still image codec unit 18 and the moving image codec unit 19.

[Description of Motion Detection / Compensation Unit 16]
In this embodiment, the motion detection / compensation unit 16 basically performs motion vector detection by performing block matching processing using the SAD values described with reference to FIGS. To. However, in this embodiment, the motion detection / motion compensation unit 16 is configured by hardware as described later, and performs block matching processing hierarchized by common hardware. Further, as will be described later, the noise reduction processing for the still image and the noise reduction processing for the moving image can be realized by common hardware.

<Outline of Hierarchical Block Matching Processing of Embodiment>
A motion vector detection process in a general conventional block matching is performed by moving a reference block in pixel (pixel) units (one pixel unit or a plurality of pixel units), and calculating a SAD value for the reference block at each moving position. A SAD value indicating a minimum value is detected from the calculated SAD values, and a motion vector is detected based on a reference block position exhibiting the minimum SAD value.

  However, in such a conventional motion vector detection process, the reference block is moved in units of pixels within the search range, so that the number of matching processes for calculating the SAD value increases in proportion to the search range to be searched. As a result, the matching processing time increases and the capacity of the SAD table also increases.

  Therefore, in this embodiment, a reduced image is created for the target image (target frame) and the reference image (reference frame), block matching is performed on the created reduced image, and based on the motion detection result in the reduced image. Perform block matching on the original target image. Here, the reduced image is referred to as a reduced surface, and the original image that has not been reduced is referred to as a base surface. Therefore, in the second embodiment, after performing block matching on the reduced surface, block matching on the base surface is performed using the matching result.

  FIG. 6 and FIG. 7 show images of image reduction of the target frame (image) and the reference frame (image). That is, in this embodiment, for example, as shown in FIG. 6, the basal plane target frame 130 is reduced to 1 / n (n is a positive number) in the horizontal direction and the vertical direction to reduce the reduction surface. The target frame 132 is assumed. Accordingly, the basal plane target block 131 generated by dividing the basal plane target frame 130 into a plurality is a reduced plane in which each of the horizontal direction and the vertical direction is reduced to 1 / n × 1 / n in the reduced plane target frame. The target block 133 is obtained.

  Then, the reference frame is reduced in accordance with the image reduction magnification 1 / n of the target frame. That is, as shown in FIG. 7, the basal plane reference frame 134 is reduced to 1 / n in the horizontal direction and the vertical direction to form a reduced plane reference frame 135. The motion vector 104 for the motion compensation block 103 detected on the base plane reference frame 134 is detected as a reduced plane motion vector 136 reduced to 1 / n × 1 / n in the reduced plane reference frame 135. .

  In the above example, the image reduction magnification of the target frame and the reference frame is the same. However, in order to reduce the calculation amount, different image reduction magnifications are used for the target frame (image) and the reference frame (image). Matching may be performed by matching the number of pixels in both frames by processing such as interpolation.

  Further, although the reduction ratios in the horizontal direction and the vertical direction are the same, the reduction ratios may be different between the horizontal direction and the vertical direction. For example, when the horizontal direction is reduced to 1 / n and the vertical direction is reduced to 1 / m (m is a positive number, n ≠ m), the reduced screen is 1 / n × 1 of the original screen. / M.

  FIG. 8 shows the relationship between the reduced plane reference vector and the base plane reference vector. If the motion detection origin 105 and the search range 106 are determined as shown in FIG. 8A in the basal plane reference frame 134, the reduced plane reference frame 135 on which the image is reduced to 1 / n × 1 / n. Then, as shown in FIG. 8B, the search range is a reduced plane search range 137 reduced to 1 / n × 1 / n.

  In this embodiment, within the reduction plane search range 137, reduction plane reference vectors 138 representing the amount of positional deviation from the motion detection origin 105 at the reduction plane reference frame 135 are set, and the respective reduction plane reference vectors 138 are set. The correlation between the reduction plane reference block 139 at the position indicated by and the reduction plane target block 131 (not shown in FIG. 8) is evaluated.

  In this case, since block matching is performed on the reduced image, the number of reduced surface reference block positions (reduced surface reference vectors) for which SAD values should be calculated in the reduced surface reference frame 135 can be reduced, and the number of times of calculating the SAD value. The processing can be speeded up by the amount of (the number of matching processings), and the SAD table can be made small.

  As shown in FIG. 9, by correlation evaluation by block matching between a plurality of reduced surface reference blocks 139 and a reduced surface target block 131 set in a reduced surface matching processing range 143 determined according to the reduced surface search range 137. The reduced plane motion vector 136 in the reduced plane reference frame 135 is calculated. The accuracy of the reduced plane motion vector 136 is as low as n times one pixel because the image is reduced to 1 / n × 1 / n. Therefore, even if the calculated reduced plane motion vector 136 is multiplied by n, the motion vector 104 with 1 pixel accuracy cannot be obtained in the base plane reference frame 134.

  However, in the base plane reference frame 134, it is clear that the base plane motion vector 104 with 1 pixel accuracy exists in the vicinity of the motion vector obtained by multiplying the reduced plane motion vector 136 by n.

  Therefore, in this embodiment, as shown in FIGS. 8C and 9, in the base plane reference frame 134, the position indicated by the motion vector (base plane reference vector 141) obtained by multiplying the reduced plane motion vector 136 by n times points. A basal plane search range 140 is set in a narrow range where the basal plane motion vector 104 is considered to exist as a center, and a basal plane matching processing range 144 is set according to the set basal plane search range 140.

  Then, as shown in FIG. 8C, a base surface reference vector 141 in the base surface reference frame 134 is set to indicate the position in the base surface search range 140, and the position indicated by each base surface reference vector 141 is set. The base plane reference block 142 is set to block matching in the base plane reference frame 134.

  As shown in FIG. 9, the base plane search range 140 and the base plane matching processing range 144 set here are obtained by multiplying the reduction plane search range 137 and the reduction plane matching processing range 143 by n times the reciprocal of the reduction ratio. It may be a very narrow range compared to the search range 137 ′ and the matching processing range 143 ′.

  Therefore, when block matching processing is performed only on the base plane without performing hierarchical matching, a plurality of reference blocks are set in the search range 137 ′ and matching processing range 143 ′ on the base plane, Although it is necessary to perform an operation for obtaining a correlation value with the target block, in the hierarchical matching process, the matching process may be performed only in a very narrow range as shown in FIG.

  For this reason, the number of base plane reference blocks set in the base plane search range 140 and base plane matching processing range 144, which are the narrow ranges, is very small, and the number of matching processes (number of correlation value calculations) and the SAD to be held The value can be made very small, the processing can be speeded up, and the SAD table can be reduced in size.

  In this way, if the pixel-basis motion vector 136 is detected in the base-surface reference frame 134, in this embodiment, the SAD value of the reference block pointed to by the base-surface motion vector 136, that is, the minimum SAD value, In this example, a quadratic curve approximate interpolation process is also performed using the neighboring SAD values in the neighborhood to calculate a base-plane high-precision motion vector with sub-pixel accuracy.

  The base plane high-precision motion vector with subpixel accuracy will be described. In the block matching method described above, since block matching is performed in units of pixels, the motion vector is calculated only with pixel accuracy. As shown in FIG. 69, the point at which the matching process is performed, that is, the position of the reference block exists with pixel accuracy, and in order to calculate a more accurate motion vector, matching processing in units of subpixels is necessary. become.

In order to calculate a motion vector with N times the pixel accuracy (pixel pitch is 1 / N), if the matching process is performed in units of N times the pixel, the SAD table becomes about N ² times larger, Memory is required. In addition, for the block matching process, an image that is N times the upper sample must be generated, and the scale of hardware increases dramatically.

  Therefore, it is considered that a motion vector with sub-pixel accuracy is calculated from the SAD table that has been subjected to the matching process in units of pixels by interpolating the SAD table using a quadratic curve. In this case, instead of quadratic curve approximation interpolation, linear interpolation or higher-order approximation curve interpolation of cubic or higher may be used. However, in this example, a quadratic curve is used in consideration of the balance between accuracy and hardware implementation. Approximate interpolation is used.

  In this quadratic curve approximate interpolation, as shown in FIG. 10, the basal plane minimum value Smin (see reference numeral 113 in FIG. 10) of the SAD value indicated by the basal plane motion vector 104 with pixel accuracy and the minimum A plurality of SAD values near the position of the value Smin (referred to as a basal plane vicinity SAD value), in this example, four basal plane vicinity SAD values adjacent to the position of the minimum basal plane value Smin in the X and Y directions Sx1, Sx2 and Sy1, Sy2 (see reference numerals 114, 115, 116, 117 in FIG. 10) are used.

  As shown in FIG. 11, a quadratic approximate curve 118 is fitted using the base plane minimum value Smin and the SAD values Sx1 and Sx2 near the two base plane vicinity in the X direction (horizontal direction). The coordinate at which 118 is minimum is the X coordinate Vx of the motion vector (high-precision motion vector) that becomes the base-plane high-precision minimum value SXmin with sub-pixel accuracy. The equation of quadratic curve approximate interpolation at this time is shown in the following equation (1).

SXmin = 1/2 × (Sx2-Sx1) / (Sx2-2Smin + Sx1) (1)
The X coordinate taken on the SAD table by the basal plane high-precision minimum value SXmin with sub-pixel accuracy obtained by the calculation formula (1) is the X-coordinate Vx that is the minimum value of the SAD value with sub-pixel accuracy.

  The division of the calculation formula (1) can be realized by a plurality of subtractions. If the subpixel accuracy to be obtained is, for example, an accuracy of 1/4 of the pixel pitch on the original base surface, it can be obtained by subtraction only twice, so both the circuit scale and the computation time are small. Performance that is almost the same as cubic curve interpolation, which is considerably more complicated than recent curve interpolation, can be realized.

  Similarly, using a basal plane minimum value Smin and two basal plane vicinity SAD values Sy1 and Sy2 in the vicinity of the Y direction (vertical direction), a quadratic approximate curve is fitted, and a minimum value SYmin of the quadratic curve is applied. The Y coordinate taking Y becomes the Y coordinate Vy that is the minimum value of the base surface high accuracy with sub-pixel accuracy. The equation of quadratic curve approximate interpolation at this time is shown in the following equation (2).

SYmin = 1/2 × (Sy2−Sy1) / (Sy2−2Smin + Sy1) (2)
As described above, the approximation of the quadratic curve is performed twice in the X direction and the Y direction, whereby the motion vector (Vx, Vy) of the base pixel high accuracy with subpixel accuracy is obtained.

  In the above description, the minimum SAD value and two SAD values in the vicinity in the X direction (horizontal direction) and the Y direction (vertical direction) are used. However, the SAD value in the vicinity in each direction is two points or more. There may be. Further, instead of the quadratic curve in the X direction and the Y direction, for example, an approximate curve may be fitted in an oblique direction. Furthermore, an approximate curve may be applied by adding an oblique direction to the X direction and the Y direction.

  FIG. 12 shows that a vector detection result with sub-pixel accuracy can be obtained from the value of the SAD table with pixel accuracy by using the above means and procedures. The horizontal axis in FIG. 12 represents the interpolation magnification, and represents how many times the resolution is increased in the one-dimensional direction. Since the SAD table is two-dimensional, the table area is reduced at a ratio of the square, whereas the error due to interpolation increases only to a linear level, so that the usefulness of the above-described interpolation method can be seen.

<Configuration Example of Motion Detection / Motion Compensation Unit 16>
FIG. 13 shows a block diagram of a configuration example of the motion detection / motion compensation unit 16. In this example, the motion detection / compensation unit 16 includes a target block buffer unit 161 that stores pixel data of the target block 102, a reference block buffer unit 162 that stores pixel data of the reference block 108, and the target block 102. The matching processing unit 163 that calculates the SAD value for the corresponding pixel in the block 108, the motion vector calculation unit 164 that calculates the motion vector from the SAD value information output from the matching processing unit 163, and the respective blocks are controlled. A control unit 165.

  The motion detection / compensation unit 16 and the image memory unit 4 are connected through the system bus 2. That is, in this example, the bus interface unit 21 and the bus interface unit 22 are connected between the system bus 2 and the target block buffer unit 161 and the reference block buffer unit 162, respectively. Further, a memory controller 8 is connected between the image memory unit 4 and the system bus 2 for controlling memory access for writing to and reading from the image memory unit 4. Here, the AXI interconnect is used as the protocol in the system bus 2 in this example.

  At the time of still image capturing, the target block buffer unit 161 includes a reduced plane target block or a basal plane target block from the image frame of the reduced plane target image Prt or the basal plane target image Pbt stored in the image memory unit 4. The image data is read from the image memory unit 4 and written according to the read control by the memory controller 8.

  As for the reduced surface target image Prt or the base surface target image Pbt, in the first image, the image of the first imaging frame after the shutter button is pressed is read from the image memory unit 4, and the target block buffer unit 161 is used as the target frame 102. Is written to. When the images are superimposed based on the block matching with the reference image, the NR image after the image is superimposed is written in the image memory unit 4, and the target frame 102 of the target block buffer unit 161 is added to the NR image. It will be rewritten.

  In the reference block buffer unit 162, image data of the reduced surface matching processing range or the base surface matching processing range from the image frame of the reduced surface reference image Prr or the base surface reference image Pbr stored in the image memory unit 4 is stored in the image. It is read from the memory unit 4 and written. In the reduced plane reference image Prr or the base plane reference image Pbr, the imaging frame after the first imaging frame is written in the image memory unit 4 as the reference frame 108.

  In this case, when image superimposition processing is performed while capturing a plurality of continuously captured images (hereinafter referred to as addition during shooting), the base plane reference image and the reduced plane reference image are used. The image frames after the first image frame are sequentially taken into the image memory unit 4 one by one. Therefore, the image memory unit 4 only needs to hold one base surface reference image and one reduced surface reference image.

  However, after capturing a plurality of continuously captured images into the image memory unit 4, the motion detection / motion compensation unit 16 and the image superimposing unit 17 perform motion vector detection and perform image superposition. In the case of doing so (hereinafter referred to as “addition after shooting”), as the base plane reference image and the reduced plane reference image, all of the plurality of imaging frames after the first imaging frame are stored in the image memory. It is necessary to store and hold in the unit 4.

  As an imaging device, both addition during shooting and addition after shooting can be used, but in this embodiment, the still image NR processing requires a clean image with reduced noise even if processing time is somewhat long. In consideration of this, the post-shooting addition process is employed. Detailed operation description of the still image NR processing in this embodiment will be described later.

  On the other hand, at the time of moving image shooting, the imaging frame from the image correction / resolution conversion unit 15 is input to the motion detection / motion compensation unit 16 as the target frame 102. The target block extracted from the target frame from the image correction / resolution conversion unit 15 is written in the target block buffer unit 161. In the reference block buffer unit 162, the imaging frame stored in the image memory unit 4 immediately before the target frame is set as a reference frame 108, and this reference frame (base reference image Pbr or reduced size) is used. The basal plane matching process range or the reduced plane matching process range from the plane reference image Prr) is written.

  At the time of this moving image shooting, the image memory unit 4 stores the previous captured image frame to be subjected to block matching with the target frame from the image correction / resolution conversion unit 15 as a base plane reference image Pbr and a reduced plane reference. Since the image information stored in the image memory unit 4 may be stored in one image (one frame), the base plane reference image Pbr and the reduced plane reference image Prr may be used as the base plane reference image Prr. Image data is not compressed.

  The matching processing unit 163 performs matching processing on the reduced surface and matching processing on the base surface for the target block stored in the target block buffer unit 161 and the reference block stored in the reference block buffer unit 162.

  Here, what is stored in the target block buffer unit 161 is image data of the reduced plane target block, and what is stored in the reference block buffer 162 is image data of the reduced plane matching processing range extracted from the reduced plane reference screen. In the case of, the matching processing unit 163 performs a reduced surface matching process. Also, what is stored in the target block buffer unit 161 is image data of the base plane target block, and what is stored in the reference block buffer 162 is image data of the base plane matching processing range extracted from the base plane reference screen. In some cases, the matching processing unit 163 performs the basal plane matching process.

  In order to detect the strength of the correlation between the target block and the reference block in the block matching by the matching processing unit 163, the SAD value is calculated using the luminance information of the image data in this embodiment as well. The minimum SAD value is detected, and the reference block exhibiting the minimum SAD value is detected as the strongest correlation reference block.

  Needless to say, the SAD value may be calculated by using not only the luminance information but also the color difference signal and the information of the three primary color signals R, G and B. In calculating the SAD value, all the pixels in the block are usually used. However, in order to reduce the amount of calculation, only the pixel value of the pixel with a limited jump position is used by thinning out or the like. It may be.

  The motion vector calculation unit 164 detects the motion vector of the reference block with respect to the target block from the matching processing result of the matching processing unit 163. In this embodiment, the motion vector calculation unit 164 detects and holds the minimum value of the SAD value and also holds the SAD values of a plurality of reference vectors in the vicinity of the reference vector exhibiting the minimum SAD value. A function of detecting a high-precision motion vector with sub-pixel accuracy by performing a quadratic approximate interpolation process is also provided.

  The control unit 165 controls the processing operation of the hierarchical block matching process in the motion detection / compensation unit 16 while being controlled by the CPU 1.

<Configuration Example of Target Block Buffer 161>
A block diagram of a configuration example of the target block buffer 161 is shown in FIG. As shown in FIG. 14, the target block buffer 161 includes a base plane buffer unit 1611, a reduction plane buffer unit 1612, a reduction processing unit 1613, and selectors 1614, 1615 and 1616. The selectors 1614, 1615, and 1616 are selected and controlled by a selection control signal from the control unit 165, although not shown in FIG. 14.

  The basal plane buffer unit 1611 is for temporarily storing the basal plane target block. The basal plane buffer unit 1611 sends the basal plane target block to the image superimposing unit 17 and supplies it to the selector 1616.

  The reduction plane buffer unit 1612 is for temporarily storing reduction plane target blocks. The reduction plane buffer unit 1612 supplies the reduction plane target block to the selector 1616.

  As described above, since the target block is sent from the image correction / resolution conversion unit 15 at the time of moving image shooting, the reduction processing unit 1613 is provided for generating a reduction plane target block by the reduction processing unit 1613. It has been. The reduced surface target block from the reduction processing unit 1613 is supplied to the selector 1615.

  The selector 1614 selects a target block (basal plane target block) from the image correction / resolution conversion unit 15 when shooting a moving image, and a base block target block or a reduced plane target block from the image memory unit 4 when shooting a still image. Is selected and output in accordance with a selection control signal from, and the output is supplied to a base plane buffer unit 1611, a reduction processing unit 1613, and a selector 1615.

  The selector 1615 selects and outputs the reduction plane target block from the reduction processing unit 15 during moving image shooting, and the reduction plane target block from the image memory unit 4 during still image shooting using a selection control signal from the control unit 165. The output is supplied to the reduction plane buffer unit 1612.

  In accordance with a selection control signal from the control unit 1615, the selector 1616 selects the reduced surface target block from the reduced surface buffer unit 1612 at the time of block matching on the reduced surface, and the base surface buffer unit 1611 at the time of block matching at the base surface. The base plane target blocks from are respectively selected and output, and the output reduced plane target block or base plane target block is sent to the matching processing unit 163.

<Configuration Example of Reference Block Buffer 162>
A block diagram of a configuration example of the reference block buffer 162 is shown in FIG. As shown in FIG. 15, the reference block buffer 162 includes a base surface buffer unit 1621, a reduced surface buffer unit 1622, and a selector 1623. Although not shown in FIG. 15, the selector 1623 is selected and controlled by a selection control signal from the control unit 165.

  The basal plane buffer unit 1621 temporarily stores the basal plane reference block from the image memory unit 4, supplies the basal plane reference block to the selector 1623 and sends it to the image overlay unit 17 as a motion compensation block.

  The reduction plane buffer unit 1622 is for temporarily storing the reduction plane reference block from the image memory unit 4. The reduction plane buffer unit 1622 supplies the reduction plane reference block to the selector 1623.

  In accordance with a selection control signal from the control unit 1615, the selector 1623 selects a reduced surface reference block from the reduced surface buffer unit 1612 when performing block matching on the reduced surface, and a base surface buffer unit 1611 when performing block matching on the base surface. The base plane reference blocks from are selected and output, and the output reduced plane reference block or base plane reference block is sent to the matching processing unit 163.

<Configuration Example of Image Superimposing Unit 17>
A block diagram of a configuration example of the image superimposing unit 17 is shown in FIG. As shown in FIG. 16, the image superposition unit 17 includes an addition rate calculation unit 171, an addition unit 172, a basal plane output buffer unit 173, a reduction plane generation unit 174, and a reduction plane output buffer unit 175. It is prepared for.

  The image superposition unit 17 and the image memory unit 4 are connected through the system bus 2. That is, in this example, the bus interface unit 23 and the bus interface unit 24 are connected between the system bus 2 and the base plane output buffer unit 173 and the reduced plane output buffer unit 162, respectively.

  The addition rate calculation unit 171 receives the target block and the motion compensation block from the motion detection / motion compensation unit 16 and adopts the addition rate of both the simple addition method or the average addition method. The determined addition rate is supplied to the adding unit 172 together with the target block and the motion compensation block.

  The basal plane NR image as a result of the addition by the adding unit 172 is written into the image memory unit 4 through the basal plane output buffer unit 173 and the bus interface 23. Further, the base surface NR image as a result of addition by the adding unit 172 is converted into a reduced surface NR image by the reduced surface generation unit 174, and the reduced surface NR image from the reduced surface generation unit 174 is converted into a reduced surface output buffer unit. 175 and the bus interface 24 to be written in the image memory unit 4.

[Overview of noise reduction processing for captured images]
<When shooting still images>
FIG. 17 and FIG. 18 are flowcharts of noise reduction processing by image superposition at the time of still image shooting in the image pickup apparatus having the above-described configuration. Each step in the flowcharts of FIGS. 17 and 18 is executed under the control of the CPU 1 and the control unit 165 of the motion detection / compensation unit 16 controlled by the CPU 1, and is also executed by the image superposition unit 17. Is.

  First, when the shutter button is pressed, in the imaging apparatus of this example, high-speed shooting of a plurality of images is performed at high speed under the control of the CPU 1. In this example, M (M frames; M is an integer of 2 or more) captured image data to be superimposed at the time of still image shooting is captured at high speed and pasted in the image memory unit 4 (step S1).

  Next, the reference frame is Nth in time among the M image frames stored in the image memory unit 4 (N is an integer of 2 or more, and the maximum value is M). The unit 165 sets the initial value of the value N, which is the order, to N = 2 (step S2). Next, the control unit 165 sets the first image frame as a target image (target frame), and sets the N = 2nd image as a reference image (reference frame) (step S3).

  Next, the control unit 165 sets a target block in the target frame (step S4), and the motion detection / motion compensation unit 16 reads the target block from the image memory unit 4 into the target block buffer unit 161 (step S5). Further, the pixel data in the matching processing range is read from the image memory unit 4 to the reference block buffer unit 162 (step S6).

  Next, the control unit 165 reads the reference block within the search range from the reference block buffer unit 162, and the matching processing unit 163 performs the hierarchical matching process in this embodiment. In this example, as will be described later, of the SAD values calculated by the matching processing unit 163, a SAD value described later holds a minimum value of the SAD value and a SAD value in the vicinity thereof, and approximates quadratic curve approximation. In order to perform processing, it is sent to the motion vector calculation unit 164. After the above processing is repeated for all the reference vectors in the search range, the quadratic curve approximation interpolation processing unit performs the above-described interpolation processing and outputs a highly accurate basal plane motion vector (step S7).

  Next, the control unit 165 reads out a motion compensation block that compensates for the motion corresponding to the detected motion vector from the reference block buffer unit 162 according to the high-accuracy basal plane motion vector detected as described above (step). S8) In synchronization with the target block, the image is sent to the subsequent image superimposing unit 17 (step S9).

  Next, under the control of the CPU 1, the image superimposing unit 17 superimposes the target block and the motion compensation block, and pastes the NR image data of the superimposed block on the image memory unit 4. That is, the image superimposing unit 17 writes the NR image data of the superimposed block in the image memory unit 4 (step S10).

  Next, the control unit 165 determines whether or not block matching has been completed for all target blocks in the target frame (step S11), and block matching processing has not been completed for all target blocks. When it is determined, the process returns to step S4, the next target block in the target frame is set, and the processes from step S4 to step S11 are repeated.

  Further, when the control unit 165 determines in step S11 that the block matching for all the target blocks in the target frame has been completed, it is determined whether or not the processing for all the reference frames to be overlapped is completed, that is, It is determined whether or not M = N (step S12).

  If it is determined in step S12 that M = N is not satisfied, N = N + 1 is set (step S13). Next, the NR image generated by the superposition in step S10 is set as a target image (target frame). = N + 1-th image is set as a reference image (reference frame) (step S14). Then, it returns to step S4 and repeats the process after this step S4. In other words, when M is 3 or more, the above processing is repeated with the image that has been superimposed in all target blocks as the next target image and the third and subsequent images as reference frames. This is repeated until the Mth overlap is completed. If it is determined in step S12 that M = N, this processing routine is terminated.

  The image data of the NR image as a result of superimposing the M photographed images is compression-encoded by the still image codec unit 18, supplied to the recording / reproducing device unit 5, and recorded on the recording medium. Is.

  The above-described still image noise reduction processing method is a method of pasting M pieces of image data on the image memory unit 4, but the image may be superimposed each time one image is taken. In that case, since only one image frame is stored in the image memory unit 4, the shooting interval is longer than that in the noise reduction processing method of the processing routines of FIGS. 17 and 18, but the memory cost is minimized. It is possible to

<When shooting movies>
Next, FIG. 19 shows a flowchart of the noise reduction processing by image superposition at the time of moving image shooting in the image pickup apparatus of this embodiment. Each step in the flowchart of FIG. 19 is also executed under the control of the CPU 1 and the control unit 165 of the motion detection / compensation unit 16 controlled by the CPU 1. When the moving image recording button is operated by the user, the CPU 1 instructs the process of FIG. 19 to start from the start.

  In this embodiment, the motion detection / compensation unit 16 has a configuration suitable for performing matching processing in units of target blocks. Therefore, the image correction / resolution conversion unit 15 holds the frame image according to the control of the CPU 1 and sends the image data to the motion detection / motion compensation unit 16 in units of target blocks (step S21).

  The target block image data sent to the motion detection / motion compensation unit 16 is stored in the target block buffer unit 161. Next, the control unit 165 sets a reference vector corresponding to the target block (step S22), and reads image data in the matching processing range from the image memory unit 4 into the reference block buffer unit 162 (step S23).

  Next, the matching processing unit 163 and the motion vector calculation unit 164 perform motion detection processing by hierarchical block matching in this embodiment (step S24). That is, the matching processing unit 163 first calculates the SAD value between the pixel value of the reduction plane target block and the pixel value of the reduction plane reference block on the reduction plane, and sends the calculated SAD value to the motion vector calculation section 164. send. The matching processing unit 163 repeats this process for all reduced surface reference blocks in the search range. After the calculation of the SAD values for all the reduced surface reference blocks in the search range is completed, the motion vector calculation unit 164 identifies the minimum SAD value and detects the reduced surface motion vector.

  The control unit 165 multiplies the reduced plane motion vector detected by the motion vector calculation unit 164 by a reciprocal of the reduction ratio to convert it into a motion vector on the basal plane, and the converted vector indicates a position indicated on the basal plane. The center area is set as a search range on the basal plane. Then, the control unit 165 controls the matching processing unit 163 to perform block matching processing on the basal plane in the search range. The matching processing unit 163 calculates an SAD value between the pixel value of the base surface target block and the pixel value of the base surface reference block, and sends the calculated SAD value to the motion vector calculation unit 164.

  After the calculation of the SAD values for all the reduced surface reference blocks in the search range is finished, the motion vector calculation unit 164 identifies the minimum SAD value, detects the reduced surface motion vector, and also detects the SAD in the vicinity thereof. A value is specified, and the quadratic curve approximation interpolation process described above is performed using those SAD values, and a high-precision motion vector with sub-pixel accuracy is output.

  Next, the control unit 165 reads out the image data of the motion compensation block from the reference block buffer unit 162 in accordance with the high-precision motion vector calculated in step S24 (step S25), and synchronizes with the target block. The image is sent to the image superimposing unit 17 (step S26).

  The image superimposing unit 17 superimposes the target block and the motion compensation block, and writes the image data of the NR image as a result of the superimposition to the image memory unit 4 (step S27). Then, the image superimposing unit 17 records the image data of the NR image as a reference frame for the next target frame, as a moving image recording monitor to be output to the monitor display 6 through the NTSC encoder 20, and through the moving image codec unit 19. The data is stored in the image memory unit 4 for recording to be recorded in the playback device unit 6 (step S28).

  Then, the CPU 1 determines whether or not the moving image recording stop operation has been performed by the user (step S29). When it is determined that the moving image recording stop operation has not been performed by the user, the CPU 1 returns to step S21, and after this step S21. Instruct to repeat the process. If it is determined in step S29 that the moving image recording stop operation has been performed by the user, the CPU 1 ends this processing routine.

  In the above-described processing routine for noise reduction processing of a moving image, an image frame one frame before is used as a reference frame. However, an image of a frame earlier than one frame may be used as a reference frame. In addition, the image before the first frame and the image before the second frame may be stored in the image memory unit 4, and the image frame to be used as the reference frame may be selected from the contents of the two pieces of image information. .

  By using the means, procedure, and system configuration as described above, it is possible to perform still image noise reduction processing and moving image noise reduction processing with one common block matching processing hardware.

[About the motion vector calculation unit 164]
Next, a configuration example and operation of the motion vector calculation unit 164 will be described.

  In an example of a conventionally known motion vector calculation unit, an SAD table TBL that stores all SAD values calculated in a search range is generated, a minimum SAD value is detected from the SAD table TBL, and the minimum SAD value is calculated. A reference vector corresponding to the position of the reference block to be presented is calculated as a motion vector. When quadratic curve approximate interpolation is performed, a plurality of SAD values in the vicinity of the minimum SAD value, in this example, four SAD values are extracted from the generated SAD table TBL, and these SAD values are used. Interpolate. Therefore, this method requires a large-scale memory corresponding to the SAD table TBL.

  In the example described below, by not generating the SAD table TBL that stores all the SAD values calculated in the search range, the circuit scale can be further reduced and the processing time can also be reduced. It is.

<First Example of Motion Vector Calculation Unit 164 of Embodiment>
As described above, the block matching process uses the position indicated by the reference vector as the position of the reference block, calculates the SAD value of each pixel of each reference block and each pixel of the target block, and performs the calculation process within the search range. This is performed for reference blocks at positions indicated by all reference vectors.

  Here, when searching for a motion compensation block by changing the position of the reference block within the search range, the search is performed in order from the end of the screen (frame), and from the center of the screen (frame) to the outside. In this embodiment, the search direction is set as indicated by an arrow 120 in FIG. 20A, and the search is started in the horizontal direction from the upper left end of the search range. After the minute search is completed, a search method is adopted in which a procedure is repeated in which a line one line below in the vertical direction is searched in the horizontal direction from the left end.

  That is, as shown in FIG. 20B, in the search range 106, the reference block 108 is sequentially set and searched in the horizontal direction from the upper left of the search range 106, and the SAD value for each reference block 108 is calculated. I do. Then, as shown in FIG. 20C, the corresponding SAD table is also buried in the horizontal direction from the upper left. At this time, the range of pixel data actually used for the matching process becomes the matching process range 110 corresponding to the size of the reference block 108 as described above.

  As shown in FIGS. 10 and 11, in order to perform the sub-pixel accuracy quadratic curve approximate interpolation processing of this example, the minimum value Smin of the SAD value and SAD values Sx1, Sx2, Sy1, Sy2 in the vicinity thereof are used. Find what you want.

  When the SAD value for each reference block is calculated, the calculated SAD value is compared with the minimum value of the SAD value up to that point, and the calculated SAD value is greater than the minimum value of the SAD value up to that point. If it is smaller, the calculated SAD value is held as the minimum value, and the SAD value and the reference vector at that time are held, so that the minimum value of the SAD value and the position information of the reference block that takes the minimum value (reference vector Information) can be obtained without generating the SAD table.

  Then, if the minimum value of the detected SAD value is held and the SAD value of the reference block near the reference block position that becomes the minimum SAD value is held as the neighboring SAD value, the neighboring SAD value is also obtained. The SAD table can be held without being generated.

  At this time, in this example, as shown in FIG. 21 (A), the search method as shown in FIG. 20 (A) described above is adopted, so that one line in the horizontal direction in the conventional SAD table TBL is used. If a memory having a capacity for storing the SAD value (hereinafter referred to as a line memory) is provided, when the SAD value of the reference block is newly calculated, the SAD table TBL is hatched in FIG. As shown, the SAD values of a plurality of reference blocks for one line of the SAD table TBL calculated before the newly calculated SAD value 121 are stored as the storage data 122 in the line memory. Will be stored.

  Therefore, when the newly calculated SAD value of the reference block is detected as the minimum SAD value, the reference block at a position on one line of the position of the reference block exhibiting the minimum SAD value 121 on the SAD table TBL. SAD value 123 (neighboring SAD value (Sy1)) and the SAD value 124 (neighboring SAD value (Sx1)) of the reference block on the left side of the position of the reference block exhibiting the minimum SAD value 121 are the line memory. Can be obtained from.

  Then, on the SAD table TBL, the neighborhood SAD value (Sx2) (see reference numeral 125 in FIG. 21C) that is the SAD value of the reference block at the position to the right of the reference block position of the minimum SAD value is The SAD value calculated at the reference block position may be held. Similarly, on the SAD table TBL, the neighborhood SAD value (Sy2) that is the SAD value of the reference block at a position one line below the position of the reference block of the newly calculated minimum SAD value (sign of FIG. 21C) 126), the SAD value calculated at the reference block position may be held later.

  Considering the above, the first example of the motion vector calculation unit 164 has a hardware configuration as shown in FIG.

  In other words, the first example of the motion vector calculation unit 164 does not include the SAD table TBL that holds all the calculated SAD values, but the SAD value writing unit 1641, the SAD value comparison unit 1642, and the SAD value holding. Unit 1643, X direction (horizontal direction) neighborhood value extraction unit 1644, Y direction (vertical direction) neighborhood value extraction unit 1645, quadratic curve approximation interpolation processing unit 1646, and memory for one line of SAD table TBL (line 1647).

  The SAD value holding unit 1643 includes a holding unit (memory) for the minimum SAD value Smin and the neighboring SAD values Sx1, Sx2, Sy1, and Sy2. In this first example, the SAD value holding unit 1643 supplies the minimum SAD value Smin from the minimum SAD value holding unit to the SAD value comparison unit 1642, and among the stored nearby SAD values, The position information (reference vector) of the reference block of the neighboring SAD value Sx2 on the right side of the minimum SAD value Smin and the position information (reference vector) of the reference block of the neighboring SAD value Sy2 on the lower side of the minimum SAD value Smin The data is supplied to the writing unit 1641.

  In this first example, the SAD value comparison unit 1642 receives the position information (reference vector) of the reference block from the matching processing unit 163 and the SAD value Sin of the reference block, and the SAD value holding unit 1643 The minimum SAD value Smin from the minimum SAD value holding unit is received.

  Then, the SAD value comparison unit 1642 compares both the SAD value Sin calculated at that time from the matching processing unit 163 and the minimum SAD value Smin from the minimum SAD value holding unit of the SAD value holding unit 1643, When the SAD value Sin calculated at the time from the matching processing unit 163 is smaller, the SAD value is detected as the minimum SAD value at the time, and when the SAD value Sin is smaller, At the time, it is detected that the minimum SAD value Smin from the minimum SAD value holding unit of the SAD value holding unit 1643 is still the minimum value. The SAD value comparison unit 1642 supplies the detection result information DET to the SAD value writing unit 1641 and the SAD value holding unit 1643.

  The SAD value writing unit 1641 includes a buffer memory for one pixel for temporarily storing the SAD value Sin calculated from the matching processing unit 163 and its position information (reference vector). In this first example, The reference block position information (reference vector) from the matching processing unit 163 and the SAD value Sin of the reference block are written in the line memory 1647. In this case, the line memory 1647 performs the same operation as that of the shift register, and when there is no empty space, when the new position information and SAD value are stored, the oldest position information and SAD value in the line memory 1647 are discarded. Is done.

  In addition, the SAD value writing unit 1641 performs the following processing in the first example before writing the calculated SAD value Sin and its position information into the line memory 1647.

  That is, the SAD value writing unit 1641, when the comparison detection result information DET from the SAD value comparison unit 1642 indicates that the SAD value Sin is the minimum value, the position information of the reference block from the matching processing unit 163. (Reference vector) and the SAD value Sin of the reference block are sent to the SAD value holding unit 1643.

  The SAD value holding unit 1643 detects that the SAD value Sin is the minimum value based on the comparison detection result information DET from the SAD value comparison unit 1642, and the position information of the reference block transmitted from the SAD value writing unit 1641. (Reference vector) and the SAD value Sin of the reference block are stored in the minimum SAD value holding unit.

  In the first example, the SAD value writing unit 1641 uses the position information (reference vector) of the reference block from the matching processing unit 163 as the position of the neighboring SAD value Sx2 or Sy2 received from the SAD value holding unit 1643. Even when the information matches, the reference block position information (reference vector) from the matching processing unit 163 and the SAD value Sin of the reference block are sent to the SAD value holding unit 1643.

  The SAD value holding unit 1643 recognizes which neighboring SAD value is information from the received reference block position information (reference vector), and stores it in the corresponding neighboring SAD value holding unit.

  When the above processing is completed for all the reference blocks in the search range, the SAD value holding unit 1643 holds the minimum SAD value and its position information, and the four neighboring SAD values and its position information, as described above. .

  Therefore, the X direction (horizontal direction) neighborhood value extraction unit 1644 and the Y direction (vertical direction) neighborhood value extraction unit 1645 store the detected minimum SAD value Smin and its neighboring SAD values held in the SAD value holding unit 1643. Sx1, Sx2, Sy1, Sy2 and their position information are read out and sent to the quadratic curve approximation interpolation processing unit 1646. Receiving this, the quadratic curve approximation interpolation processing unit 1646 performs interpolation by the quadratic curve twice in the X direction and the Y direction, and calculates a high-precision motion vector with sub-pixel accuracy as described above.

  As described above, in the first example, by using a line memory for one line of the SAD table TBL instead of the SAD table TBL, it is possible to detect a motion vector with sub-pixel accuracy.

  An example of the flow at the time of block matching processing on the reduced surface in the first example is shown in the flowchart of FIG. 23, and an example of the flow at the time of block matching processing on the basal plane is shown in the flowcharts of FIGS. Each step of the flowchart of the first example is performed in the matching processing unit 163 and the motion vector calculation unit 164 under the control of the control unit 165.

  First, the flowchart of FIG. 23 which is an example of the flow at the time of the block matching process in a reduction surface is demonstrated.

  First, an initial value of the minimum SAD value Smin of the SAD value holding unit 1643 of the motion vector calculation unit 164 is set (step S31). As the initial value of the minimum SAD value Smin, for example, the maximum value of the pixel difference is set.

  Next, the matching processing unit 163 sets a reduced plane reference vector (Vx, Vy), sets a reduced plane reference block position for calculating the SAD value (step S32), and sets the pixels of the set reduced plane reference block. The data is read from the reference block buffer unit 162 (step S33), and the pixel data of the reduction plane target block is read from the target block buffer unit 161, and the absolute difference between each pixel data of the reduction plane target block and the reduction plane reference block is read. The sum of the values, that is, the SAD value is obtained, and the obtained SAD value is sent to the motion vector calculation unit 164 (step S34).

  In the motion vector calculation unit 164, the SAD value writing unit 1641 writes the SAD value into the line memory 1647 (step S35).

  In the motion vector calculation unit 164, the SAD value comparison unit 1642 compares the SAD value Sin calculated by the matching processing unit 163 with the minimum SAD value Smin held in the SAD value holding unit 1643 to calculate It is determined whether or not the SAD value Sin is smaller than the minimum SAD value Smin held so far (step S36).

  If it is determined in step S36 that the calculated SAD value Sin is smaller than the minimum SAD value Smin, the process proceeds to step S37, and information on the minimum SAD value Smin held in the SAD value holding unit 1643 and its position information (reduction plane) The reference vector is updated.

  When it is determined in step S36 that the calculated SAD value Sin is larger than the minimum SAD value Smin, the process proceeds to step S38 without performing the update processing of the retained information in step S37, and the matching processing unit 163 determines whether the search range is within the search range. It is determined whether or not the matching process has been completed at the positions of all reduced surface reference blocks (reduced surface reference vectors). If it is determined that there is still an unprocessed reference block in the search range, the process returns to step S32. The processes after step S32 described above are repeated.

  If the matching processing unit 163 determines in step S38 that the matching processing has been completed at the positions of all the reduced surface reference blocks (reduced surface reference vectors) in the search range, the matching processing unit 163 notifies the motion vector calculating unit 164 to that effect. .

  In response to this, the motion vector calculation unit 164 outputs the position information (reduced plane reference vector) of the minimum SAD value Smin held in the SAD value holding unit 1643 to the control unit 165 (step S39).

  This is the end of the block matching process on the reduced surface in this example.

  Next, the flowcharts of FIGS. 24 and 25 which are examples of the flow at the time of block matching processing on the base surface will be described.

  First, an initial value of the minimum SAD value Smin of the SAD value holding unit 1643 of the motion vector calculation unit 164 is set (step S41). As the initial value of the minimum SAD value Smin, for example, the maximum value of the pixel difference is set.

  Next, the matching processing unit 163 sets the reference vector (Vx, Vy) on the base plane, sets the base plane reference block position for calculating the SAD value (step S42), and sets the pixels of the set base plane reference block Data is read from the reference block buffer unit 162 (step S43).

  Then, the pixel data of the base plane target block is read from the target block buffer unit 161, and the sum of absolute values of differences between the pixel data of the base plane target block and the base plane reference block, that is, the SAD value is obtained. The value is sent to the motion vector calculation unit 164 (step S44).

  In the motion vector calculation unit 164, the SAD value writing unit 1641 writes the SAD value into the line memory 1647 (step S45). In the motion vector calculation unit 164, the SAD value comparison unit 1642 compares the SAD value Sin calculated by the matching processing unit 163 with the minimum SAD value Smin held in the SAD value holding unit 1643 to calculate It is determined whether or not the SAD value Sin is smaller than the minimum SAD value Smin held so far (step S46).

  When it is determined in step S46 that the calculated SAD value Sin is smaller than the minimum SAD value Smin, the process proceeds to step S47, and information on the minimum SAD value Smin held in the SAD value holding unit 1643 and the minimum SAD value Smin are obtained. The SAD value and the position information (base plane reference vector) of the reference block at a position one pixel above and one pixel to the left of the reference block position to be presented are updated.

  That is, the SAD value comparison unit 1642 sends the comparison result information DET to the SAD value writing unit 1641 that the calculated SAD value Sin is smaller than the minimum SAD value Smin. Then, the SAD value writing unit 1641 sends the calculated SAD value Sin and its position information (base plane reference vector) to the SAD value holding unit 1643 as information on a new minimum SAD value Smin, and FIG. ), The oldest SAD value and its position information (base plane reference vector) of the line memory 1647 and the newest SAD value and its position information (base plane reference vector) are set to 1 of the position of the minimum SAD value. The information is sent to the SAD value holding unit 1643 as information on the SAD value Sy1 of the base plane reference block at the position on the pixel and information on the SAD value Sx1 of the base plane reference block at the position one pixel to the left. The SAD value holding unit 1643 receives the received information on the new minimum SAD value Smin, the information on the SAD value Sy1 of the base plane reference block at the position one pixel above, and the SAD value Sx1 of the base plane reference block at the position one pixel left. The corresponding holding information is updated with the information.

  Then, after step 47, the process proceeds to step S51 in FIG. If it is determined in step S46 that the calculated SAD value Sin is greater than the minimum SAD value Smin, the process advances to step S51 without performing the holding information update process in step S47.

  In step S51, the SAD value writing unit 1641 indicates that the position indicated by the position information (reference vector) for the calculated SAD value Sin is one pixel below the position of the base surface reference block held as the minimum SAD value Smin. It is determined whether or not the position is the position of the base plane reference block, and when it is determined that the position is the position of the base plane reference block one pixel below, the calculated SAD value Sin and its position information (reference vector) are stored in the SAD value holding unit. Send to 1643. The SAD value holding unit 1643 updates the holding information of the neighboring SAD value Sy2 for the basal plane reference block at the position one pixel below by using the received SAD value and its position information (step S52).

  In step S51, the position pointed to by the position information (base plane reference vector) for the calculated SAD value Sin is that of the base plane reference block one pixel below the position of the base plane reference block held as the minimum SAD value Smin. When it is determined that the position is not a position, the SAD value writing unit 1641 indicates the position of the base plane reference block in which the position indicated by the position information (base plane reference vector) regarding the calculated SAD value Sin is held as the minimum SAD value Smin. It is determined whether or not it is the position of the base plane reference block 1 pixel below (step S53).

  In this step S53, the position pointed to by the position information (base plane reference vector) for the calculated SAD value Sin is the base plane reference block one pixel to the right of the position of the base plane reference block held as the minimum SAD value Smin. When it is determined that the position is, the SAD value writing unit 1641 sends the calculated SAD value Sin and its position information (base plane reference vector) to the SAD value holding unit 1643. The SAD value holding unit 1643 updates the holding information of the neighboring SAD value Sx2 with respect to the basal plane reference block at the position one pixel to the right based on the received SAD value and its position information (step S54).

  In step S53, the position pointed to by the position information (base plane reference vector) for the calculated SAD value Sin is the base plane reference block one pixel to the right of the base plane reference block position held as the minimum SAD value Smin. When it is determined that the position is not the position, the SAD value writing unit 1641 writes the calculated SAD value Sin and its position information (base plane reference vector) only in the line memory 1647 (step S45 described above), and the SAD value holding unit 1643. Do not send to.

  Then, the matching processing unit 163 determines whether or not the matching process has been completed at the positions (reference vectors) of all the base plane reference blocks in the search range (step S55). When it is determined that there is a surface reference block, the process returns to step S42 in FIG. 28, and the processes after step S42 described above are repeated.

  If the matching processing unit 163 determines in step S45 that the matching processing has been completed at the positions of all the base plane reference blocks in the search range (base plane reference vectors), the matching processing unit 163 notifies the motion vector calculation unit 164 accordingly. .

  In response to this, the motion vector calculation unit 164 detects that the X direction (horizontal direction) neighborhood value extraction unit 1644 and the Y direction (vertical direction) neighborhood value extraction unit 1645 are held in the SAD value holding unit 1643. The minimum SAD value Smin and its neighboring SAD values Sx1, Sx2, Sy1, Sy2 and their position information are read out and sent to the quadratic curve approximation interpolation processing unit 1646. Receiving this, the quadratic curve approximation interpolation processing unit 1646 performs interpolation by the quadratic curve twice in the X direction and the Y direction, and calculates a high-precision motion vector with subpixel accuracy as described above ( Step S56). The block matching process of the second example for one base plane reference frame is thus completed.

  As described above, in the first example, one line of the SAD table is provided without holding the SAD table holding all the calculated SAD values, and one pixel is provided in the SAD value writing unit 1641. It is possible to perform sub-pixel precision motion vector detection by interpolation processing only by holding the SAD value.

  The method of the first example has the effect that the processing time does not change and the hardware scale can be reduced because the number of times of block matching is the same as the method of holding all of the conventional SAD table TBL.

  In the description of the first example described above, the SAD value comparison unit 1642 uses the calculated SAD value Sin from the matching processing unit 163 and the minimum SAD value Smin held in the SAD value holding unit 163. Although the comparison is made, the SAD value comparison unit 1642 includes a holding unit for the minimum SAD value, and the held minimum SAD value is compared with the calculated SAD value Sin. When the calculated Sin is smaller The held minimum SAD value is updated, and the position information is sent to the SAD value holding unit 1643 through the SAD value writing unit 1641 so as to be held in the minimum SAD value holding unit of the SAD value holding unit 1643. .

<Second Example of Motion Vector Calculation Unit 164 of Embodiment>
The second example of the motion vector calculation unit 164 is an example in which the line memory 1647 in the case of the first example is also omitted to further reduce the hardware scale.

  In the second example, the minimum value Smin of the SAD value of the reference block in the search range and its position information (reference vector) are detected and held in the same manner as in the first example described above. However, the acquisition and holding of the neighboring SAD value and its position information are not performed simultaneously with the detection of the SAD value Smin, as in the first example, and the position information of the detected minimum SAD value Smin is matched. Returning to the unit 163, the matching processing unit 163 again calculates the SAD values for the reference blocks at the four point positions in the vicinity of the minimum SAD value Smin, and supplies them to the motion vector calculation unit 164.

  The motion vector calculation unit 164 receives the SAD values of the four neighboring positions calculated by the block matching process again from the matching processing unit 163 and its position information (reference vector), and each of the SAD value holding units 1643 Store in the holding part.

  A hardware configuration example of the second example of the motion vector calculation unit 164 is shown in FIG. That is, in the third example of the motion vector calculation unit 164, the line memory 1647 of the first example is not provided, but as shown in FIG. 26, an SAD value writing unit 1641, an SAD value comparison unit 1642, An SAD value holding unit 1643, an X direction (horizontal direction) neighborhood value extraction unit 1644, a Y direction (vertical direction) neighborhood value extraction unit 1645, and a quadratic curve approximation interpolation processing unit 1646 are provided.

  Also in the second example, the X direction (horizontal direction) neighborhood value extraction unit 1644, the Y direction (vertical direction) neighborhood value extraction unit 1645, and the quadratic curve approximation interpolation processing unit 1646 are the same as those in the first example described above. The SAD value writing unit 1641, the SAD value comparison unit 1642, the SAD value holding unit 1643, and the line memory 1647 perform different operations from those in the first example.

  Similar to the first example, the SAD value holding unit 1643 includes a holding unit (memory) for the minimum SAD value Smin and the neighboring SAD values Sx1, Sx2, Sy1, and Sy2. Also in this second example, the SAD value holding unit 1643 supplies the minimum SAD value Smin from the minimum SAD value holding unit to the SAD value comparison unit 1642. However, in the second example, unlike the first example, the SAD value holding unit 1643 does not supply the position information of the neighboring SAD value to the SAD value writing unit 1641.

  In this second example, the SAD value comparison unit 1642 also calculates the SAD value Sin calculated at that time from the matching processing unit 163 and the minimum SAD value Smin from the minimum SAD value storage unit of the SAD value storage unit 1643. When the SAD value Sin calculated at the time from the matching processing unit 163 is smaller, the SAD value is detected as the minimum SAD value at the time, and the SAD value Sin When the value is smaller, it is detected that the minimum SAD value Smin from the minimum SAD value holding unit of the SAD value holding unit 1643 is still the minimum value at that time. The SAD value comparison unit 1642 supplies the detection result information DET to the SAD value writing unit 1641 and the SAD value holding unit 1643.

  Similar to the above example, the SAD value writing unit 1641 includes a buffer memory for one pixel for temporarily storing the SAD value Sin calculated from the matching processing unit 163 and its position information (reference vector). In this second example, when the SAD value writing unit 1641 indicates that the comparison detection result information DET from the SAD value comparison unit 1642 indicates that the SAD value Sin is the minimum value, the matching processing unit The reference block position information (reference vector) from 163 and the SAD value Sin of the reference block are sent to the SAD value holding unit 1643.

  The SAD value holding unit 1643 knows that the SAD value Sin is the minimum value based on the comparison detection result information DET from the SAD value comparison unit 1642, and the position information of the reference block sent from the SAD value writing unit 1641 ( Reference vector) and the SAD value Sin of the reference block are stored in the minimum SAD value holding unit.

  The above processing is performed on the SAD values calculated by the matching processing unit 163 for all reference blocks in the search range. In the second example, when the calculation of the SAD values for all the reference blocks within the search range is completed, the SAD value holding unit 1643 stores the position information (reference (reference) of the held minimum SAD value Smin. Vector) Vmin is supplied to matching processing section 163 to request recalculation of SAD values for four reference blocks in the vicinity of the position information.

  When the matching processing unit 163 receives from the SAD value holding unit 1643 a request for recalculation of the SAD value for the neighborhood reference block including the position information (reference vector) Vmin of the minimum SAD value Smin, the position information of the minimum SAD value Smin From the reference vector Vmin, the positions of the four neighboring reference blocks are detected, and the SAD value is calculated for the reference block at the detected position. Then, the calculated SAD value is sequentially supplied to the SAD value writing unit 1641 together with the position information (reference vector).

  In this case, since the matching processing unit 163 performs block matching processing in the order of the search direction, the neighboring SAD values are calculated in the order of SAD values Sy1, Sx1, Sx2, and Sy2. The SAD value writing unit 1641 sequentially supplies the received recalculated SAD value and its position information (reference vector) to the SAD value holding unit 1643.

  The SAD value holding unit 1643 sequentially writes and holds the recalculated SAD value and its position information (reference vector) in the corresponding storage unit.

  Thus, when the recalculation of the SAD value for the neighborhood reference block is completed, the X direction (horizontal direction) neighborhood value extraction unit 1644 and the Y direction (vertical direction) neighborhood value extraction unit 1645 are the same as in the first example described above. Reads out the detected minimum SAD value Smin, its neighboring SAD values Sx1, Sx2, Sy1, Sy2 and their position information held in the SAD value holding unit 1643, and sends them to the quadratic curve approximation interpolation processing unit 1646. . Receiving this, the quadratic curve approximation interpolation processing unit 1646 performs interpolation by the quadratic curve twice in the X direction and the Y direction, and calculates a high-precision motion vector with sub-pixel accuracy as described above.

  As described above, in the second example, the motion vector can be detected with sub-pixel accuracy without using the SAD table TBL or the line memory.

  The example of the flow at the time of block matching processing on the reduction surface in the second example is the same as the case of the first example shown in FIG. 23 except that writing to the line memory 1647 is not performed. Here, the description is omitted. An example of the flow at the time of block matching processing on the base surface in the second example will be described with reference to the flowchart of FIG.

  First, an initial value of the minimum SAD value Smin of the SAD value holding unit 1643 of the motion vector calculation unit 164 is set (step S61). As the initial value of the minimum SAD value Smin, for example, the maximum value of the pixel difference is set.

  Next, the matching processing unit 163 sets a reference vector (Vx, Vy) at the base plane, sets a base plane reference block position for calculating the SAD value (step S62), and sets the base plane reference block of the set base plane reference block. The pixel data is read from the reference block buffer unit 162 (step S63), the pixel data of the base plane target block is read from the target block buffer unit 161, and the difference between the pixel data of the base plane target block and the base plane reference block is calculated. The sum of absolute values, that is, the SAD value is obtained, and the obtained SAD value is sent to the motion vector calculating unit 164 (step S64).

  In the motion vector calculation unit 164, the SAD value comparison unit 1642 is calculated by comparing the SAD value Sin calculated by the matching processing unit 163 with the minimum SAD value Smin held in the SAD value holding unit 1643. It is determined whether or not the SAD value Sin is smaller than the minimum SAD value Smin held so far (step S65).

  When it is determined in step S65 that the calculated SAD value Sin is smaller than the minimum SAD value Smin, the process proceeds to step S66, and the minimum SAD value Smin held in the SAD value holding unit 1643 and its position information are updated. .

  That is, the SAD value comparison unit 1642 sends the comparison result information DET to the SAD value writing unit 1641 that the calculated SAD value Sin is smaller than the minimum SAD value Smin. Then, the SAD value writing unit 1641 sends the calculated SAD value Sin and its position information (base plane reference vector) to the SAD value holding unit 1643 as new minimum SAD value Smin information. The SAD value holding unit 1643 updates the minimum SAD value Smin to be held and its position information to the received new SAD value Sin and its position information.

  After step S66, the process proceeds to step S67. If it is determined in step S65 that the calculated SAD value Sin is greater than the minimum SAD value Smin, the process advances to step S67 without performing the holding information update process in step S66.

  In step S67, the matching processing unit 163 determines whether or not the matching process has been completed at the positions of all the base plane reference blocks (base plane reference vectors) in the search range. When it is determined that there is a surface reference block, the process returns to step S62, and the processes after step S62 are repeated.

  If the matching processing unit 163 determines in step S67 that the matching process has been completed at the positions (reference vectors) of all the base plane reference blocks in the search range, the minimum SAD value Smin from the SAD value holding unit 1643 is determined. The position information is received, the SAD value is recalculated for the basal plane reference block at the positions of the four neighboring points, and the recalculated neighboring SAD value is supplied to the SAD value holding unit 1643 through the SAD value writing unit 1641 and held. (Step S68).

  Next, in the motion vector calculation unit 164, the X direction (horizontal direction) neighborhood value extraction unit 1644 and the Y direction (vertical direction) neighborhood value extraction unit 1645 are stored in the SAD value holding unit 1643 and the detected minimum The SAD value Smin and its neighboring SAD values Sx1, Sx2, Sy1, Sy2 and their position information are read out and sent to the quadratic curve approximation interpolation processing unit 1646. Receiving this, the quadratic curve approximation interpolation processing unit 1646 performs interpolation by the quadratic curve twice in the X direction and the Y direction, and calculates a high-precision motion vector with subpixel accuracy as described above ( Step S69). This completes the block matching process on the base surface of the second example for one reference frame.

  In this second example, the processing time is increased by recalculating the SAD value as compared with the first example described above, but since no line memory is required, the circuit scale is reduced to the first. It can be reduced more than the example. In addition, since the SAD value is recalculated only for the neighborhood SAD value, in the above example, since the number of times is at most four, the increase in the processing time is small.

  In the description of the second example described above, the minimum SAD value is held in the SAD value holding unit 1643 while being detected. However, in the SAD value comparison unit 1642, the position information of the reference block that exhibits the minimum SAD value. (Reference vector) is detected and held, and when the first block matching is completed, the position information of the minimum SAD value is supplied from the SAD value comparison unit 1642 to the matching processing unit 163. Good.

  In this case, in the recalculation of the SAD value in the matching processing unit 163, the minimum SAD value is recalculated in addition to the SAD values of the four neighboring points. In this case, the number of recalculations of the SAD value is 5 and increases by 1. However, in the first block matching, only the SAD value comparison unit 1642 needs to operate, and the SAD value writing unit 1641 and the SAD value holding Since the unit 1643 only needs to operate so as to hold the recalculated SAD value, there is an advantage that the processing operation is simplified.

  Further, the processing in the motion detection and motion compensation unit 16 can be executed in parallel and in parallel for a plurality of target blocks set in the target frame. In that case, it is necessary to provide as many hardware systems as the motion detection and motion compensation unit 16 described above in accordance with the number of target blocks to be processed in parallel and in parallel.

  In the case of the method for generating the SAD table TBL as in the conventional example, it is necessary to generate as many SAD tables as the number of target blocks, and a very large memory is required. However, in the first example, one line of the SAD table is sufficient for one target block, and the memory capacity can be greatly reduced. Furthermore, in the case of the second example, since no line memory is required, the memory capacity can be greatly reduced.

[Example of hierarchical block matching process flow]
Next, FIG. 28 and FIG. 29 show flowcharts of an operation example of the hierarchical block matching processing in the motion detection / compensation unit 16 in this embodiment.

  Note that the processing flow shown in FIG. 28 and FIG. 29 partially overlaps with the description of the processing example of the matching processing unit 163 and the motion calculation unit 164 described above. The operation will be described for easier understanding.

  First, the motion detection / compensation unit 16 reads a reduced image of the target block, that is, a reduced surface target block from the target block buffer unit 161 (step S71 in FIG. 28). Next, the initial value of the reduced surface minimum SAD value is set as the initial value of the minimum SAD value Smin of the SAD value holding unit 1643 of the motion vector calculation unit 164 (step S72). As an initial value of the reduced surface minimum SAD value Smin, for example, a maximum value of pixel difference is set.

  Next, the matching processing unit 163 sets a reduction plane search range, sets a reduction plane reference vector (Vx / n, Vy / n: 1 / n is a reduction ratio) in the set reduction search range, The reduced surface reference block position for calculating the SAD value is set (step S73). Then, the pixel data of the set reduction plane reference block is read from the reference block buffer unit 162 (step S74), and the sum of absolute values of differences between the pixel data of the reduction plane target block and the reduction plane reference block, that is, the reduction plane The SAD value is obtained, and the obtained reduced surface SAD value is sent to the motion vector calculation unit 164 (step S75).

  In the motion vector calculation unit 164, the SAD value comparison unit 1642 compares the reduced surface SAD value Sin calculated by the matching processing unit 163 with the reduced surface minimum SAD value Smin held in the SAD value holding unit 1643. Then, it is determined whether or not the calculated reduced surface SAD value Sin is smaller than the reduced surface minimum SAD value Smin held so far (step S76).

  If it is determined in step S76 that the calculated reduction surface SAD value Sin is smaller than the reduction surface minimum SAD value Smin, the process proceeds to step S77, and the reduction surface minimum SAD value Smin held in the SAD value holding unit 1643 and its position. Information is updated.

  That is, the SAD value comparison unit 1642 sends the comparison result information DET to the SAD value writing unit 1641 that the calculated reduction surface SAD value Sin is smaller than the reduction surface minimum SAD value Smin. Then, the SAD value writing unit 1641 sends the calculated reduced surface SAD value Sin and its position information (reduced surface reference vector) to the SAD value holding unit 1643 as information on a new reduced surface minimum SAD value Smin. The SAD value holding unit 1643 updates the reduced reduction surface minimum SAD value Smin and its position information to be held to the received new reduction surface SAD value Sin and its position information.

  After step S77, the process proceeds to step S78. If it is determined in step S76 that the calculated reduced surface SAD value Sin is larger than the reduced surface minimum SAD value Smin, the process advances to step S78 without performing the holding information update process in step S77.

  In step S78, the matching processing unit 163 determines whether or not the matching process has been completed at the positions of all reduced surface reference blocks (reduced surface reference vectors) in the reduced surface search range. When it is determined that there is an unprocessed reduced surface reference block, the process returns to step S73, and the processes after step S73 are repeated.

  If the matching processing unit 163 determines in step S78 that the matching process has been completed at the positions of all reduced surface reference blocks (reduced surface reference vectors) in the reduced surface search range, the reduction processing from the SAD value holding unit 1643 is performed. The position information (reduced surface motion vector) of the surface minimum SAD value Smin is received, and the received reduced surface motion vector is the reciprocal of the reduction magnification, that is, the vector obtained by multiplying by n is centered on the position coordinates indicated in the base plane target frame. The base plane target block is set at the position and the base plane search range is set as the base plane target frame as a relatively narrow range centered on the position coordinate indicated by the n-fold vector (step S79). From the target block buffer 161, the pixel data of the base surface target block is displayed. Read the data (step S80).

  Next, the initial value of the base surface minimum SAD value is set as the initial value of the minimum SAD value Smin of the SAD value holding unit 1643 of the motion vector calculation unit 164 (step S81 in FIG. 29). As the initial value of the basal plane minimum SAD value Smin, for example, the maximum value of the pixel difference is set.

  Next, the matching processing unit 163 sets the basal plane reference vector (Vx, Vy) in the basal plane reduction search range set in step S79, and sets the basal plane reference block position for calculating the SAD value (step). S82). Then, the set pixel data of the base plane reference block is read from the reference block buffer unit 162 (step S83), and the sum of absolute values of differences between the pixel data of the base plane target block and the base plane reference block, that is, the base plane The SAD value is obtained, and the obtained basal plane SAD value is sent to the motion vector calculation unit 164 (step S84).

  In the motion vector calculation unit 164, the SAD value comparison unit 1642 compares the basal plane SAD value Sin calculated by the matching processing unit 163 with the basal plane minimum SAD value Smin held in the SAD value holding unit 1643. Then, it is determined whether or not the calculated basal plane SAD value Sin is smaller than the basal plane minimum SAD value Smin held so far (step S85).

  When it is determined in this step S85 that the calculated basal plane SAD value Sin is smaller than the basal plane minimum SAD value Smin, the process proceeds to step S86, and the basal plane minimum SAD value Smin held in the SAD value holding unit 1643 and its position. Information is updated.

  That is, the SAD value comparison unit 1642 sends the comparison result information DET to the SAD value writing unit 1641 that the calculated basal plane SAD value Sin is smaller than the basal plane minimum SAD value Smin. Then, the SAD value writing unit 1641 sends the calculated base surface SAD value Sin and its position information (reference vector) to the SAD value holding unit 1643 as information of a new base surface minimum SAD value Smin. The SAD value holding unit 1643 updates the base surface minimum SAD value Smin to be held and its position information to the received new base surface SAD value Sin and its position information.

  After step S86, the process proceeds to step S87. If it is determined in step S85 that the calculated basal plane SAD value Sin is greater than the basal plane minimum SAD value Smin, the process proceeds to step S87 without performing the holding information update process in step S86.

  In step S87, the matching processing unit 163 determines whether or not the matching process has been completed at the positions of all base plane reference blocks (base plane reference vectors) in the base plane search range. When it is determined that there is an unprocessed base plane reference block, the process returns to step S82, and the processes after step S82 are repeated.

  When the matching processing unit 163 determines in step S87 that the matching processing has been completed at all base plane reference block positions (base plane reference vectors) in the base plane search range, the basis from the SAD value holding unit 1643 is determined. The position information (base plane motion vector) of the surface minimum SAD value Smin is received, the base plane SAD value is recalculated for the base plane reference block at the positions of the four neighboring points, and the recalculated vicinity base plane SAD value is SAD. The value is supplied to the SAD value holding unit 1643 through the value writing unit 1641 and held (step S88).

  Next, in the motion vector calculation unit 164, the X direction (horizontal direction) neighborhood value extraction unit 1644 and the Y direction (vertical direction) neighborhood value extraction unit 1645 are stored in the SAD value holding unit 1643 and the detected minimum The basal plane SAD value Smin and its neighboring basal plane SAD values Sx1, Sx2, Sy1, Sy2 and their position information are read out and sent to the quadratic curve approximation interpolation processing unit 1646. Receiving this, the quadratic curve approximation interpolation processing unit 1646 performs interpolation by the quadratic curve twice in the X direction and the Y direction, and calculates a high-precision basal plane motion vector with subpixel accuracy as described above. (Step S89). This completes the block matching process of this example for one reference frame.

  Next, the effect of the image processing method using the hierarchical block matching method according to this embodiment will be described with a specific example.

  FIG. 30 shows specific examples of the reference block, the search range, and the matching processing range on the base surface and the reduction surface used in the description of the above-described embodiment. In the example of FIG. 30, the reduction ratio 1 / n in the horizontal direction and the vertical direction is set to n = 4.

  As a comparative example, as shown in FIGS. 30A and 30B, for example, a reference block 108 that does not use a reduced image has horizontal (horizontal) × vertical (vertical) = 32 × 32 pixels and a search range. 106 is horizontal (horizontal) × vertical (vertical) = 144 × 64 pixels, and the matching processing range 110 is horizontal (horizontal) × vertical (vertical) = 176 × 96 pixels.

  Then, in the above-described embodiment, in the reduced surface reduced to 1 / n = 1/4 in the horizontal direction and the vertical direction, as shown in FIGS. 30C and 30D, the reduced surface reference block 139 is used. Is horizontal (horizontal) × vertical (vertical) = 8 × 8 pixels, the reduction plane search range 137 is horizontal (horizontal) × vertical (vertical) = 36 × 16 pixels, and the reduction plane matching processing range 143 is horizontal (horizontal). ) × vertical (vertical) = 44 × 24 pixels.

  When block matching is performed with a reduction plane reduced to ¼ in the horizontal direction and the vertical direction, the reduction plane motion vector is a motion vector with a precision of 4 pixels. An error occurs with respect to the precision motion vector. In other words, if the pixels on the base surface are as shown in FIG. 35, the matching processing point 148 on the base surface covers all pixels on the base surface, but is reduced to 1/4. This is because the matching processing points when performing block matching on the surface are the matching processing points 147 in units of 4 pixels indicated by black circles in FIG.

  However, it can be predicted that at least a 1-pixel precision motion vector will exist within a 4-pixel range around the matching processing point on the reduced plane indicated by the reduced plane motion vector.

  Therefore, in this embodiment, in basal plane matching that determines the basal plane search range based on the calculated reduced plane motion vector, the pixel position indicated by the reference vector obtained by multiplying the reduced plane motion vector by 4 that is the reciprocal of the reduction magnification is indicated. A base plane target block is set at the center, a search range for 4 pixels (base plane search range 140) is determined, base plane block matching is performed, and a motion vector is calculated again.

  Therefore, as shown in FIGS. 30E and 30F, the basal plane reference block 142 is horizontal (horizontal) × vertical (vertical) = 32 × 32 pixels, and the basal plane search range 140 is horizontal (horizontal) ×. The vertical (vertical) = 4 × 4 pixels and the basal plane matching processing range 144 is horizontal (horizontal) × vertical (vertical) = 40 × 40 pixels.

  In FIG. 32, in the case of performing two-layer matching of reduced plane matching and basal plane matching as in this embodiment, it is assumed that the SAD table is used, and the SAD table of the basal plane and reduced plane is used. Indicates size. The example of FIG. 32 corresponds to the specific example shown in FIG.

  The SAD table TBL in the case of the 144 × 64 search range (see FIG. 30B) before the reduction is 145 × 65 points as shown in FIG.

  On the other hand, the reduced surface SAD table in the case of the 36 × 16 reduced surface search range (see FIG. 30D) is 37 points × 17 points as shown in FIG.

  In the case of a 4 × 4 base plane search range (see FIG. 30F), the base plane SAD table is 5 points × 5 points.

  Therefore, the number of matching processes when layer matching is not performed is 145 × 65 = 9425 times, whereas the number of matching processes when layer matching is applied is 37 × 17 + 5 × 5 = 654 times, which greatly increases the processing time. It can be seen that it can be shortened.

  The line memory in the case of the first example of the motion vector detection method described above may be one line in the reduced-surface SAD table, so that it can store 37 SAD values and their position information. In the case of TBL, a very small memory is sufficient for the memory storing 9425 SAD values and their position information.

  Further, in the case of the second example of the configuration example of the motion vector calculation unit 164 described above, even the reduced surface SAD table for storing the 37 SAD values and the position information thereof becomes unnecessary, so that the circuit scale can be further reduced. can do.

  As described above, according to the above-described embodiment, after performing the hierarchical matching process, the interpolation process is performed on the basal plane, thereby performing motion vector detection with sub-pixel accuracy in a wide search range. Is possible.

  FIG. 33 shows an example of target block sizes and matching processing ranges on the reduction plane and the base plane in the hierarchical block matching method of this embodiment. In the example of FIG. 33, the size of the reduction plane target block 133 is a fixed value of horizontal × vertical = 8 × 8 pixels as shown in FIG. For this reason, the reduced surface matching processing range 143 is horizontal × vertical = 44 × 24 pixels as shown in FIG.

  The size of the base plane target block 131, the size of the base plane matching processing range 144, and the number of base plane reference vectors are as shown in FIG. FIG. 33C shows the size of the base plane target block 131 and the base plane matching processing range 144 in each of the examples when the reduction ratio of the reduction plane with respect to the base plane is 1/2, 1/4, and 1/8. The size and the number of base plane reference vectors are shown.

[Reduced memory capacity and efficient memory access during still image NR processing]
As described above, this embodiment supports still image NR processing and moving image NR processing. In moving image NR processing, real-time performance, that is, speed is required rather than accuracy, whereas still image NR processing requires a clean image with no noise even if processing time is somewhat long.

  As described above, when the image overlay method is largely classified from the viewpoint of the system architect, the during-shoot addition method, which is a method for performing image overlay in real time during high-speed continuous shooting during still image shooting, After high-speed continuous shooting, there are two types, a post-shooting addition method, which is a method of superimposing images after all reference images are gathered. In order to perform the NR process with sufficient processing time, the post-shooting addition method is preferable rather than the during-shooting addition method. In this embodiment, the post-shooting addition method is used.

  However, the post-shooting addition method has to store the reference image in the image memory, and there is a problem that a larger capacity is required as the image memory as the number of superimposed images increases.

  In this embodiment, in view of this problem, in using the post-shooting addition method at the time of still image NR processing, the capacity of the image memory unit 4 is reduced as much as possible and the memory access can be performed efficiently. is doing. Hereinafter, this point will be described.

<Data format for memory access during still image NR processing>
First, a data format at the time of memory access to the image memory unit 4 in this embodiment will be described.

  As shown in the block diagrams of FIGS. 13 and 16, the motion detection / motion compensation unit 16 and the image superposition unit 17 are connected to the system bus 2 via bus interface units 21, 22, 23, and 24, respectively. Has been.

  The system bus 2 in this embodiment has a bus width of, for example, 64 bits, and the burst length by the memory controller 8 (the number of times burst transfer can be continuously performed in units of predetermined plural pixel data) is, for example, 16 bursts at maximum ing. In other words, the maximum number q of continuous burst transfers that can be performed is q = 16 in this example.

  Read requests from the target block buffer unit 161 and the reference block buffer unit 162 of the motion detection / compensation unit 16 are supplied to the memory controller 8 through the bus interface units 21 and 22, and the memory controller 8 uses the image memory unit. 4 predetermined memories are accessed by generating a start address, a burst length, and other bus protocols.

  In the image data used in the imaging apparatus of this embodiment, one pixel is composed of a luminance signal Y and a chroma signal C (Cr, Cb). The format is Y: Cr: Cb = 4: 2: 2, and the luminance signal Y and chroma signal C (Cr, Cb) are both unsigned 8 bits (Cr = 4 bits, Cb = 4 bits). Therefore, YC4 pixels are arranged in a memory access width of 64 bits. The data processing unit of the system bus 2 (the amount of data that can be transferred by one burst transfer) is 64 bits, and the YC pixel data for one pixel is 16 bits. That is, the number of pixels p that can be transferred in one burst transfer is p = 4 pixels in this example.

  When each processing unit of the imaging device is connected to the global system bus 2 as shown in FIG. 1, the bus bandwidth and the processing unit of the memory controller 8 shared by each processing unit are closely related. There is. Here, the bus bandwidth is an amount including a data rate, a bus width (number of bits), and the like at which data can be transferred while avoiding congestion on a bus for transferring data.

  By the way, as described above, in this embodiment, the maximum burst length that can be processed by the memory controller 8 is 16 bursts. For example, even when image data of the same data size is written in the memory of the image memory unit 4, if a large number of 16 burst transfers can be used, the number of bus arbitrations is reduced, and the memory access processing by the memory controller 8 is efficient. The bus bandwidth can be reduced.

  Therefore, in this embodiment, it is efficient if the data (64 bits) of 4 pixels of YC pixel data can be bus-accessed by 16 bursts. When this is converted into the number of pixels, the data is 4 × 16 = 64 pixels.

  Therefore, in this embodiment, as shown in FIG. 34 (A), the horizontal direction of the image for one screen is divided in units of 1 burst transfer (64 pixels), and the image for one screen is divided vertically. Suppose that it is composed of a plurality of block image units (hereinafter referred to as strips). Then, as one of the memory access formats of the image data to the image memory unit 4, a memory access method for writing image data to the image memory unit 4 and reading it from the image memory unit 4 (hereinafter referred to as “the memory access method”). Adopt a strip access format). During still image NR processing, image data memory access to the image memory unit 4 is basically performed using this strip access format.

  Here, a format in which 64 pixels in the horizontal direction are bus-accessed by 16 bursts is referred to as a 64 × 1 format. This 64 × 1 format is a memory access method in which burst transfer for every four pixels is continuously repeated 16 times, which is the maximum number of continuous bursts, as shown in FIG. The memory controller 8 includes an address generation unit (not shown) for executing 64 × 1 format bus access. That is, as shown in FIG. 36 (B), when the start addresses A1, A2, A3... A16 of burst transfer for every four pixels are determined, the bus access in the 64 × 1 format can be automatically performed. The memory controller 8 includes an address generation unit (not shown) for executing 64 × 1 format memory access.

  The strip access format is a system in which the 64 × 1 format is continuously performed in the vertical direction of the screen. When the 64 × 1 format is repeated in the horizontal direction and all accesses for one line are completed, the same access is made for the next line, thereby enabling access to image data in the raster scan format.

  The memory controller 8 of this embodiment includes an address generation unit (not shown) for executing such a strip access format.

  If the horizontal image data is not divisible by 64 pixels, a dummy area 151 is provided at the right end in the horizontal direction as shown with a shadow line in FIG. Or white pixel data is added as dummy data so that the number of pixels in the horizontal direction is a multiple of 64.

  The conventional raster scan method is suitable for reading data line by line because the addresses are continuous in the horizontal direction when accessing the image memory, whereas the strip access format is 1 burst. Since the address is incremented in the vertical direction in units of transfer (64 pixels), it is suitable for reading block-like data whose horizontal direction is within 64 pixels.

  For example, when reading a strip block of 64 pixels × 64 lines, the memory controller 8 accesses the four pixels of YC pixel data (64 bits) in 16 bursts for the image memory unit 4. Then, since the data of 4 × 16 = 64 pixels is obtained in the 16 bursts, after setting the address of the first line composed of 64 pixels in the horizontal direction, the remaining 63 is simply incremented in the vertical direction. It is possible to set the address of pixel data for a line.

  In this embodiment, in order to allow the motion detection / motion compensation unit 16 to access the image memory unit 4 in the above-described strip access format, the image correction / resolution conversion unit 15 in the preceding stage also accesses the strip access format. It corresponds to. In this embodiment, in still image NR processing at the time of still image shooting, a plurality of captured images obtained by high-speed continuous shooting are compressed and stored in the image memory unit 4, as necessary. Therefore, the still image codec unit 19 is also compatible with the strip access format so that the motion detection / compensation unit 16 can perform the block matching process by decoding and reading the data.

  FIG. 35A is an example of division in strip units in an example in which an image for one screen includes horizontal × vertical = 640 × 480 pixels. In this example, the image for one screen is divided into ten strips T0 to T9 as a result of being divided in units of 64 pixels in the horizontal direction.

  In the case of the reduction ratio ¼ shown in FIG. 33 described above, in one strip, as shown in FIG. 35 (B), the basal plane target block of 32 × 32 pixels is shown as B0 to B29. Thus, there are 30 in total, 2 in the horizontal direction and 15 in the vertical direction.

  At the time of still image NR processing, when reading the basal plane target block from the image memory unit 4, in order to take advantage of the strip access format, in this embodiment, 64 pixels × 1 line is accessed to increase bus efficiency. I have to.

  For example, as shown in FIG. 36, when the reduction ratio of the reduction surface with respect to the base surface is 1/2, in this embodiment, the size of the base surface target block is horizontal × vertical = 16 pixels × 16 lines. The Therefore, when four basal plane target blocks are arranged in the horizontal direction, the number of pixels in the horizontal direction is 64 pixels. Therefore, in this embodiment, when the reduction ratio is 1/2, as shown on the right side of FIG. 36, the access unit to the image memory unit 4 is four of the target blocks, and the above-described strip of 64 × 1 format is used. Access by access type. That is, the four target blocks can be transferred by repeating access in units of 64 pixels in the horizontal direction (4 pixels × 16 bursts) 16 times while changing the address in the vertical direction.

  Similarly, as shown in FIG. 36, when the reduction ratio of the reduction surface with respect to the base surface is 1/4, in this embodiment, the size of the base surface target block is horizontal × vertical = 32 pixels × 32 lines. Therefore, by accessing the image memory two by two, the base plane target blocks can be accessed in the above-described 64 × 1 format strip access format. When the reduction ratio of the reduction plane with respect to the base plane is 1/8, in this embodiment, the size of the base plane target block is horizontal × vertical = 64 pixels × 64 lines. By accessing the image memory one by one, it is possible to access in the strip access format of the 64 × 1 format described above.

  The strip access format is a very useful technique not only in terms of bus bandwidth but also in the internal processing of the above circuit. First, it is possible to reduce the size of the internal line memory of the vertical filter processing unit for performing vertical filter processing and the like.

  That is, when accessing pixel data in the raster scan format, for vertical filter processing, all the pixels for one line must be stored in the internal line memory of the vertical filter processing unit. Since the accumulation of the pixel data of the next line is not started, the internal line memory requires a plurality of line memories having a capacity for one line. In the case of the strip access format, in the above example, in the unit of 64 pixels Thus, since the pixel data of the next line in the vertical direction arrives, the internal line memory of the vertical filter processing unit only needs to have a plurality of line memories having a capacity of 64 pixels, and the size of the line memory Can be reduced.

  For circuits that perform processing in block format, such as resolution conversion processing, it is more efficient to convert from strip format to arbitrary block format than to convert from raster scan format to arbitrary block There is.

<Compression and decompression processing of captured image in still image NR processing during still image shooting>
As described above, in this embodiment, post-shooting addition processing is employed as the addition processing in the still image NR processing during still image shooting. For this reason, in the still image NR processing at the time of still image shooting, it is necessary to store all the reference images in the image memory unit 4. That is, in this embodiment, when the shutter button is pressed, the imaging apparatus captures two or more captured images by high-speed continuous shooting and stores them in the image memory unit 4.

  In the addition method during shooting, the image memory unit 4 only needs to store reference images for a maximum of two frames, whereas in the addition method after shooting, a larger number of reference images are stored in the image memory unit 4. Need to hold. Therefore, in this embodiment, in order to reduce the capacity of the image memory 4, the captured image captured by high-speed continuous shooting is compressed and encoded by the still image codec unit 18, stored in the image memory unit 4, and captured. When an image is used as a reference image, the compressed and encoded image data is decompressed and used.

  The outline of the flow of the still image NR processing at the time of still image shooting has been described in the flowcharts shown in FIGS. 17 and 18, but the image data is processed at some steps in the flowcharts shown in FIGS. The compression encoding process or the decompression decoding process is performed.

  In step S1 of FIG. 17, a plurality of captured images captured by high-speed continuous shooting based on the shutter button operation are written into the image memory unit 4. At this time, the captured image data of the captured images is a still image. The codec unit 18 performs compression encoding. FIG. 37 shows the flow of image data at this time by dotted lines.

  That is, the captured image from the image sensor 11 is subjected to preprocessing such as sensor correction in the preprocessing unit 13 and then temporarily stored in the image memory unit 4 in the RAW signal format (data format before performing camera signal processing). Stored.

  Thereafter, the photographed image is read from the image memory unit 4 in a strip access format, converted from the RAW signal to YC pixel data by the data conversion unit 14, and subjected to image correction, resolution conversion, and the like by the image correction / resolution conversion unit 15. After being transmitted to the image memory unit 4 in a state of being directly compressed and encoded by the still image codec unit 18 without using the image memory unit 4, in this example by the JPEG (Joint Photographic Experts Group) method. Written. Therefore, here, the unit for compressing the image is not a single image unit but a strip unit. During high-speed continuous shooting, the above procedure is repeated for a plurality of sheets, and the image memory unit 4 stores and holds compressed data of the plurality of captured image data.

  When all of the captured images captured by high-speed continuous shooting are compressed and stored in the image memory unit 4, the target image setting process in the process of step S3 in FIG. 17 is performed. FIG. 38 shows the flow of image data at this time.

  That is, compressed image data of a captured image that is first captured after the shutter button is pressed is read from the image memory unit 4 in a strip access format, supplied to the still image codec unit 19, and decompressed and decoded. The decompressed and decoded image data is written in the image memory 4 as a base surface target image Pbt in a strip access format.

  Then, the decompressed and decoded base surface target image Pbt image data written in the image memory unit 4 is read out in a strip access format and supplied to the image superposition unit 17 through the motion detection / motion compensation unit 16. The image is reduced by the reduction plane generation unit 174 of the image superposition unit 17 shown in FIG. Then, the reduced target image from the reduced surface generation unit 174 is written in the image memory unit 4 in strip units as the reduced surface target image Prt.

  After the process for the target image is completed as described above, the setting process for the reference image in the process of step S3 in FIG. 17 is performed. The flow of the image data at this time is shown in FIG.

  That is, among the plurality of compression-encoded captured images stored in the image memory 4, a second captured image serving as a block matching reference image is read from the image memory unit 4 in a strip access format. Then, it is supplied to the still image codec unit 18 and decompressed and decoded. The decompressed and decoded image data is written in the image memory 4 as a base reference image Pbr in a strip access format.

  The decompressed and decoded base plane reference image Pbr image data written in the image memory unit 4 is read out in a strip access format and supplied to the image superposition unit 17 through the motion detection / motion compensation unit 16. The image is reduced by the reduction plane generation unit 174 of the image superposition unit 17 shown in FIG. Then, the reduced reference image from the reduced surface generation unit 174 is written in the image memory unit 4 in a strip access format as a reduced surface reference image Prr.

  Next, motion detection / motion compensation is performed on the decompression-decoded target image and the decompressed and decoded reference image. This corresponds to the processing in steps S4 to S9 in FIG. Further, it corresponds to the processing shown in the flowcharts of FIGS. FIG. 40 shows the flow of image data at this time.

  First, reduction plane block matching processing is performed. That is, the image data of the set reduction plane target block is read out from the reduction plane target image Prt of the image memory unit 4 in the strip access format, and is stored in the target block buffer unit 161 of the motion detection / motion compensation unit 16. The reduced surface matching processing range corresponding to the search range for the reduced surface target block is read from the reduced surface reference image Prr in the strip access format and stored in the reference block buffer unit 162.

  Then, the pixel data of the reduced surface target block and the reduced surface reference block are read out from the respective buffer units 161 and 162, the reduced surface block matching process is performed in the matching processing unit 163, and the set reduced surface target block is set. A reduction plane motion vector is detected, and a reduction plane motion vector for the set reduction plane target block is detected.

  Next, based on the reduced plane motion vector, as described above, the base plane target block is set, the base plane search range is set, and the base set from the base plane target image Pbt of the image memory unit 4 is set. The plane target block is read in the strip access format and stored in the target block buffer 161, and the base plane matching processing range corresponding to the set base plane search range is read from the base plane reference image Pbr of the image memory unit 4 in the strip access format. Read and store in the reference block buffer unit 162.

  Then, pixel data of the basal plane target block and the basal plane reference block are read from the respective buffer units 161 and 162, the basal plane block matching process is performed by the matching processing unit 163, and the motion vector calculation unit 164 sets the pixel data. The base plane motion vector with sub-pixel accuracy is detected for the base plane target block.

  Then, the motion detection / compensation unit 16 reads out the motion compensation block from the base plane reference image of the reference block buffer unit 162 based on the detected base plane motion vector, and the image overlay unit 17 together with the base plane target block. To supply. The image superimposing unit 17 superimposes the images in block units, and pastes the resulting base NR image Pbnr in block units on the image memory unit 4 in a strip access format. The image superimposing unit 17 also pastes the reduced surface NR image Prnr generated by the reduced surface generation unit 174 to the image memory unit 4.

  The above operation is repeated for all target blocks of the target image, and when block matching for all target blocks is completed, the third image is used as a reference image, and the base NR image Pbnr and reduced surface NR image Prnr is used as a target image (step S14 in FIG. 18), and the same block matching process as described above is repeated (step S4 in FIG. 17 to step S14 in FIG. 18). When all the captured images have been processed, the finally obtained base NR image Pbnr is compressed and encoded by the still image codec unit 18 and recorded on the recording medium of the recording / reproducing device unit 5. To do.

  As described above, in this embodiment, the NR image obtained by superimposing the target image and the reference image becomes a target image for the next superposition. In addition, the basal plane target block becomes unnecessary data after the block matching process is completed.

  Due to the above characteristics, in this embodiment, the image superimposing unit 17 overwrites the NR-completed block obtained by superimposing the basal plane target block and the motion compensation block on the address where the original basal plane target block was located. Yes.

  For example, as shown in FIGS. 41A and 41B, when block matching processing is performed using the block B4 of the strip T5 as a target block, the strip T0 to the strip T4 and the block B0 to the block B0. B3 is all NR completed blocks.

  As described above, the image correction / resolution conversion unit 15 performs memory access in the strip access format, and the compression encoding process for the still image is also accessed in the strip access format. Image data can be transferred directly to the still image codec unit 18 without going through the image memory unit 4, and high-speed continuous shot images can be efficiently compression-encoded and stored in the image memory unit 4.

<Reduction of image memory capacity>
Next, reduction of the image memory capacity at the time of decompression decoding of the reference image will be described.

  At the time of still image NR processing at the time of still image shooting in this embodiment, the target image is all decompressed and decoded for the image captured by compression encoding in the image memory unit 4, but as shown in the image data flow of FIG. In addition, with respect to the reference image, only the image data in strip units used for the matching process is decompressed and decoded, thereby reducing the image memory capacity. That is, the capacity of the image memory that stores the decompressed and decoded reference images (base plane and reduced plane) in the image memory unit 4 is reduced by decompressing and decoding the image data in strip units.

  This will be described with an example. For example, the basal plane target image is divided into strips T0 to T9 having 64 pixels in the horizontal direction as shown in FIG. 43 (A), and the basal plane reference image is as shown in FIG. 43 (B). Similarly, suppose that the horizontal direction is divided into strips R0 to R9 having 64 pixels.

  Now, assuming that block matching is performed for the target block in the strip T4, among the strips R0 to R9 of the basal plane reference image, the strip that may be accessed as the matching processing range is centered on the strip R4. It can be limited to several.

  For example, in the case where the reduction magnification is 1/4, in order to reduce and obtain a 44 pixel matching processing range in the horizontal direction (see FIG. 33B), in the basal plane reference image, the horizontal direction 176 pixel image data is required. Therefore, as can be seen from FIG. 43 (C), as shown in FIG. 43 (B), the basal plane reference image has two strips R1, two on each of the left and right sides of the strip R4. R2 and strips R3 and R4 are necessary, and only five strips R2 to R6 need to be decompressed and decoded.

  Next, a case will be considered in which the matching process for the target block included in the strip T4 is completed and the matching process for the target block included in the strip T5 is entered as shown in FIG. At this time, in order to obtain the matching processing range, as shown in FIG. 44B, two strips R3 to R7 are required on the left and right sides of the strip R5. That is, since the strips R3 to R6 are already stored in the image memory unit 4 at the time of matching processing of the target block included in the strip T4, it is only necessary to newly decompress and decode the strip R7. At the same time, the strip R2 becomes unnecessary.

  Therefore, in this embodiment, when the strip R7 is decompressed and decoded, the striped and decrypted strip R7 is overwritten on the reference image memory of the image memory unit 4 at the address where the strip R2 is located. As a result, the reference image data decompressed and decoded on the reference image memory of the image memory unit 4 is always only for five strips, and the reference image memory of the image memory unit 4, that is, the image memory unit It is possible to effectively save four work areas.

  When the reference image data is decoded by the above method, the arrangement of addresses on the reference image memory in the image memory unit 4 changes as shown in FIGS. That is, as for the target block in the strip T4, when the reference image in the reference image memory is centered on the strip R4 as shown in FIG. 45 (A), the image data is the image sequentially from the strip R2. Since it is stored in the reference image memory of the memory unit, when referring to the pixel in the left direction from the standard coordinate Sc4 of the strip R4, a desired value is subtracted from the address of the standard coordinate Sc4 and the right direction from the standard coordinate Sc4. When referring to this pixel, a desired value is added to the address of the standard coordinate Sc4.

  On the other hand, as for the target block in the strip T5, when the reference image in the reference image memory is centered on the strip R5 as shown in FIG. 45B, the pixels within 128 pixels are referred to in the right direction. In this case, the desired value is added to the address of the reference coordinate Sc5 of the strip R5, and when referring to the pixel of 129 pixels or more in the right direction, the desired value is added to the address of the reference coordinate Sc5, and then the total strip It is necessary to subtract the address value for the width.

  The reference coordinates Sc0 to Sc9 of the strips R0 to R9 are examples of memory address pointers when accessing the reference image memory. The memory address pointer is not limited to the address position as in the example of FIG.

  As described above, the unit of compression encoding for still images is also in the strip format, so that only the range necessary for block matching processing can be decoded, and the work area of the image memory unit 4 can be saved. . Furthermore, the size of the decoded reference image on the work area is always kept constant by decoding and overwriting the strip of the reference image to be used next on the address of the unnecessary reference image strip on the image memory. can do.

  In the example described with reference to FIGS. 43 and 44, the horizontal size SA of the matching processing range set in the reference image is equal to or larger than the horizontal size SB of the strip as a compression unit (SA> SB). This is the case.

  If the horizontal size of the matching processing range is an integral multiple of the horizontal size of the strip, and the matching processing range matches the strip separation position, useless data access is eliminated and efficiency is improved. Go up further.

  If the horizontal size of the matching processing range set in the reference image is smaller than the horizontal size of the strip that is the compression unit, the matching processing range is the size included in one strip. If it is included in one strip, the one strip may be decrypted, and if the matching processing range spans two strips, the two strips may be decrypted.

  Even in such a case, when the position of the target block changes and the position of the matching processing range changes, and the matching processing range spans two strips, the matching processing range is newly straddled. It is the same as in the above example that only the strips that have become necessary should be decrypted.

[Efficient memory access during video NR processing]
As described above, the still image NR processing requires a clean image with no noise even if it takes some processing time, whereas the moving image NR processing requires real-time performance, that is, speed rather than accuracy. .

  The flow of the moving image NR processing operation is as shown in FIG. 19, and the flow of image data during the moving image NR processing is as shown in FIGS. The flow of image data during the moving image NR process will be described with reference to FIGS.

  In the case of moving image NR processing, YC image data from the data conversion unit 14 is written in the image memory unit 4 in a strip access format after being corrected in the horizontal direction by the image correction / resolution conversion unit 15. Then, the image correction / resolution conversion unit 15 reads the horizontal correction image from the image memory unit 4 in the strip access format, performs the correction processing in the vertical direction, and then performs the strip access format to the motion detection / motion compensation unit 16. The target block is sent in real time (step S21 in FIG. 19).

  The motion detection / motion compensation unit 16 stores the target block sent from the image correction / resolution conversion unit 15 in the target block buffer unit 161, and also performs block matching according to the target block as shown in FIG. The processing range is read from the reference image stored in the image memory unit 4 (the first image is a non-superimposed image, but the second and subsequent images are NR images) and stored in the reference block buffer unit 162 (FIG. 19). Step S23).

  Then, the motion detection / compensation unit 16 reads the target block from the target block buffer unit 161 and the reference block from the block matching processing range, performs the above-described hierarchical block matching process, and performs motion vector detection and A motion compensation block is generated, and the motion compensation block and the target block are supplied to the image superimposing unit 17 (steps S24 to S26 in FIG. 19).

  The image superimposing unit 17 superimposes the received target block and the motion compensation block (step S27 in FIG. 19). Then, as shown in FIG. 46, the image superimposing unit 17 pastes the NR-completed image on which the image is superimposed on the image memory unit 4 (Step S28 in FIG. 19).

  As shown in FIG. 47, the NR image pasted on the image memory unit 4 is read into the video codec unit 19 at the next frame (the NR image one frame before), compressed, and then recorded. It is stored in the recording medium of the playback device unit 5 and used as a reference image in the next frame. Further, the NR image (an NR image one frame before) pasted on the image memory unit 4 is simultaneously read from the NTSC encoder 20 and is also output to the monitor display 6.

  In the moving image NR processing, the above processing is repeated for each captured image.

  Here, the reference image is one frame, but as shown in FIG. 47, a plurality of reference images may be held and an image with higher correlation may be used.

  Further, in the moving image NR process, real-time characteristics are important. Therefore, unlike the still image NR process, the reduced surface target image is not created and stored in the image memory unit 4, and the target block shown in FIG. The reduction processing unit 1613 in the buffer unit 161 generates a reduction plane target block and stores it in the reduction plane buffer 1612 as it is.

<Data format for memory access during video NR processing>
At the time of still image NR processing, the data access format for the memory access to the image memory unit 4 employs a strip access format in which the 64 × 1 format is continuously performed in the vertical direction.

  However, in the strip access format, when a block having a horizontal number of pixels smaller than the horizontal width of the strip (the number of pixels in the horizontal direction) is read, the burst length becomes less than 16 bursts. The most efficient data transfer (16 bursts) cannot be used, and the efficiency deteriorates.

  Also, when reading a block with a horizontal pixel count larger than the horizontal width of the strip, it must be divided into two transfers on the left and right of the border between two strips adjacent in the horizontal direction, Since the addresses are far apart on the left and right of the boundary between the two strips, there is a problem that the address processing is rather complicated.

  Furthermore, the basal plane matching process range in the hierarchical matching process is determined based on the reduced plane motion vector detected by the reduced plane matching, and is not a fixed area. May be. For this reason, when reading the basal plane matching processing range from the reference image (NR image) stored in the image memory unit 4, as described above, the boundary between two adjacent strips across the basal plane matching processing range. However, there is a problem that the address processing becomes complicated because the addresses become far apart on the left and right of the boundary between the two strips.

  Then, the size of the pixel block contained in each of the two divided strips is smaller than the horizontal width of the strip, and as described above, the access efficiency is deteriorated in the strip access format, and it looks like a movie. In addition, there is a problem that it is not suitable for applications requiring high-speed real-time processing.

  In order to deal with the above-described problems, in this embodiment, a memory access method for pixel data in units of blocks in a rectangular area composed of a plurality of lines and a plurality of pixels is prepared for accessing a reference image at the time of moving image NR processing. I have to.

  In this embodiment, as one of the reference image accesses at the time of moving image NR processing, an image (screen) as shown in FIG. 48A is considered in consideration of the above-described 64 pixels that can be burst transferred. Is divided into blocks of horizontal × vertical = 8 pixels × 8 lines, which is a unit of maximum burst transfer (64 pixels), and a block access format for writing / reading to / from the image memory unit 4 is used. Hereinafter, the block access format in units of blocks composed of 8 lines × 8 pixels will be referred to as an 8 × 8 format.

  In the 8 × 8 format, as shown in FIG. 48B, four horizontal pixels are used as one burst transfer unit, and eight horizontal pixels are transferred in two burst transfers (two bursts). When the burst transfer of 8 pixels in the horizontal direction is completed, the 8 pixels of the next line are similarly transferred in 2 bursts. Then, 64 pixels are transferred at a time in 16 bursts, which is the maximum burst length.

  Assuming that AD1 is the initial address of the block access format in the 8 × 8 format, as shown in FIG. 48B, addresses AD2 to AD16 in units of four pixels are automatically determined, and in 16 consecutive bursts, Memory access in 8 × 8 format is possible. The memory controller 8 includes an address generation unit for memory access in the 8 × 8 format.

  When the start address AD1 in the 8 × 8 format is designated on the frame memory of the image memory unit 4, the memory controller 8 uses the address generation unit to generate addresses AD1 to AD1 for memory access in the 8 × 8 format. AD16 is calculated and 8 × 8 format memory access is performed.

  As shown in FIG. 48B, the 8 × 8 format basically provides access in units of 64 pixels (16 bursts), and is the most efficient as the imaging apparatus of this embodiment. However, as will be described later, 16 bursts is the longest burst length, and may be shorter than that. In that case, the 8 × 8 format is not practically realized. However, for example, in the case of 8 pixels × 4 lines, image data can be transferred in half 8 bursts.

  If the horizontal and vertical image data cannot be divided by 8 pixels, dummy areas 152 are provided at the right end in the horizontal direction and the lower end in the vertical direction as shown by the shadow lines in FIG. For example, black or white pixel data is added as dummy data to the dummy area 152 so that the image size in the horizontal and vertical directions is a multiple of eight.

  The fact that the memory access can be accessed with 8 × 8 pixels by the 8 × 8 format memory access means that the reduction plane matching processing range and the base plane matching processing range are all 8 × 8 pixel blocks from the image memory unit 4. Can be read in units. Therefore, in the imaging apparatus of this embodiment, it is possible to perform bus access only by the most efficient data transfer (16 bursts), and the bus efficiency is maximized.

  The 8 × 8 format can be applied to access in units of a plurality of pixels as long as it is a multiple of 8 × 8 pixels, such as 16 × 16 pixels, 16 × 8 pixels, or the like.

  The most efficient data transfer unit (unit of maximum burst length) of the bus is 64 pixels in the above example, but the most efficient data transfer unit is the number of pixels p that can be transferred in one burst, and It becomes p × q pixels determined by the maximum burst length (maximum number of continuous bursts) q. Then, the block access format to be written in the image memory unit 4 may be determined based on the number of pixels (p × q). The bus transfer efficiency is best when the reduced plane matching processing range and the base plane matching processing range have a size close to a multiple of the horizontal and vertical directions of the block format.

  The reduction plane matching processing range and the base plane matching processing range when the reduction ratio is 1/2 shown in FIG. 33 will be compared with the case of the strip access format of the 64 × 1 format.

  As shown in FIG. 33B, the reduction plane matching processing range 143 is 44 pixels × 24 lines in this example. When the reduced surface matching processing range 143 is accessed in the 64 × 1 format, as shown in the upper side of FIG. 49, it is necessary to access the transfer by repeating 4 pixels × 11 bursts 24 times in the vertical direction.

  On the other hand, when the same reduced plane matching processing range 143 is accessed in the 8 × 8 format, as shown in FIG. 50, the reduced plane reference vector (0, 0) is centered on the reduced plane reference block. A reduction plane matching processing range 143 of pixels × 24 lines is determined. The reduced plane reference block of the reduced plane reference vector (0, 0) is a reduced plane reference block having the same coordinates as the reduced plane target block. In this case, as shown in FIG. 33A, the reduction plane target block is always set only in units of 8 pixels and 8 lines.

  Since the vertical direction of the reduction plane matching processing range 143 is 24 lines, as shown in FIG. 50, if the vertical search range is a multiple of the block size of 8 × 8, the vertical direction is 8 × 8 format. Can be accessed without waste. On the other hand, since the horizontal direction of the reduction plane matching processing range 143 is 44 pixels, it is not a multiple of 8. For this reason, in the example shown in FIG. Dummy image data 153 made of black or white is inserted on both sides in the horizontal direction by 6 pixels.

  From FIG. 50, although the dummy image data 153 includes 6 pixels on the left and right sides, the 64 × 1 format requires 24 times of transfer, whereas the 8 × 8 format reads in 21 times of transfer. You can see that Furthermore, in the 8 × 8 format, all transfers are the most efficient data transfer (16 bursts) in the imaging apparatus of this embodiment, and the bus efficiency is also improved.

  In addition, as shown in FIG. 33C, the base plane matching processing range 144 when the reduction ratio is 1/2 is 20 pixels × 20 lines. However, when the base plane matching processing range 144 is accessed. As shown in the lower side of FIG. 49, in the case of the 64 × 1 format, it is necessary to access for transferring 4 pixels × 5 bursts 20 times in the vertical direction.

  On the other hand, in the case of the 8 × 8 format, the basal plane matching processing range 144 is a matching processing range of 20 pixels × 20 lines around the basal plane reference block 142 of the basal plane reference vector (0, 0). Is determined. Since the basal plane matching processing range 144 is determined by the reduced plane motion vector calculated by the reduced plane matching, various combinations can be considered in units of one pixel.

  For example, the most efficient case is when the base pixel matching processing range 144 of 20 pixels × 20 lines is allocated to 9 blocks as shown in FIG. 51 for an 8 × 8 block. In this case, as shown in FIG. 51, in the basal plane matching processing range 144 of 20 pixels × 20 lines, there are regions less than 8 pixels and 8 lines at the horizontal end and the vertical end. Then, by inserting dummy pixel data 153 there and accessing the 8 × 4 format 4 pixel × 16 burst transfer 9 times, the image data of the base pixel matching processing range 144 of the 20 pixel × 20 line is obtained. Can be accessed.

  However, as shown in FIG. 52, the dummy image data 153 is not inserted into the vertical end portion of the basal plane matching processing range 144 of 20 pixels × 20 lines. Can be set to match the data to be transferred (for 4 lines), and 4 pixels × 8 bursts can be used as an access for 3 repeated transfers. In this case, in the example shown in FIG. 52, a total of 9 transfers, a transfer of repeating 4 pixels × 16 burst of 8 × 8 format 6 times and a transfer of repeating 4 pixels × 8 burst 3 times, The image data in the base plane matching processing range 144 of 20 pixels × 20 lines can be accessed.

  On the other hand, the most inefficient example is a case where the basal plane matching processing range 144 of 20 pixels × 20 lines covers 16 blocks of 8 × 8 blocks as shown in FIG. In this case, as shown in FIG. 53, in the basal plane matching processing range 144 of 20 pixels × 20 lines, there are regions less than 8 pixels and 8 lines at the horizontal end and the vertical end. Then, by inserting dummy pixel data 153 there and accessing the transfer of the 4 × 16 burst of 8 × 8 format 16 times repeatedly, the image data of the basal plane matching processing range 144 of the 20 pixels × 20 lines is obtained. Can be accessed.

  However, as shown in FIG. 54, the dummy image data 153 is not inserted into the vertical end portion of the basal plane matching processing range 144 of 20 pixels × 20 lines. As the number of times matched to the data to be transferred (each for two lines), a transfer of 4 pixels × 4 bursts can be made a total of 8 transfers. In this case, in the example shown in FIG. 54, a total of 16 transfers of 8 × 8 format 4 pixel × 16 burst transfer 8 times and 4 pixel × 4 burst repeat 8 times, The image data in the basal plane matching processing range 144 can be accessed.

  Therefore, the access to the image memory unit 4 of the image data of the base plane matching processing range 144 of 20 pixels × 20 lines needs to be transferred 20 times in the 64 × 1 format, but in the 8 × 8 format. 9 transfers at the shortest and 16 transfers at the longest. Further, in the 8 × 8 format, more than half of the transfer is the most efficient data transfer (16 bursts) in the imaging apparatus system of this example, and the bus efficiency is also improved.

  Further, in this embodiment, in order to further improve the bus bandwidth, as another one for accessing the reference image at the time of moving image NR processing, as shown in FIG. The block access format for writing / reading to / from the image memory unit can be selected by dividing into blocks of 4 pixels × 4 lines, which are 1/4 units of (64 pixels). This block access format in units of blocks consisting of 4 lines × 4 pixels is hereinafter referred to as a 4 × 4 format.

  In this 4 × 4 format, as shown in FIG. 55 (B), four horizontal pixels are used as a unit of one burst transfer, and four horizontal pixels are transferred in one burst transfer (one burst). When the burst transfer of 4 pixels in the horizontal direction is completed, the 4 pixels of the next line can be similarly transferred in 1 burst, and image data of 4 × 4 format can be transferred in 4 bursts. In the case of the 4 × 4 format, as shown in FIG. 55 (B), a block composed of 4 pixels × 4 lines = 16 pixels is continuously accessed four times in the horizontal direction (for 4 blocks) to obtain 64 pixels. Are transferred once.

  In other words, since 4 × 4 format image data can be transferred in 4 bursts, it is possible to continuously access 4 blocks in a horizontal direction of 4 pixels × 4 lines with a maximum burst length of 16 bursts. The 64 pixels for the block of four 4 pixels × 4 lines can be transferred at one time.

  When 64 pixels (16 bursts, which is the maximum burst length) are transferred at a time using this 4 × 4 format, the start address of the block of 4 pixels × 4 lines is set to the initial address AD1, as shown in FIG. Then, the addresses AD2 to AD16 in units of 4 pixels are determined as shown in the figure, and the memory access in units of 64 pixels in the 4 × 4 format can be performed with continuous 16 bursts.

  As shown in FIG. 55B, the 4 × 4 format is most efficient when accessed in units of 64 pixels (16 bursts). However, it may be necessary to access in units of 4 blocks or less in the horizontal direction. For example, in access in units of 2 blocks (32 pixels) in the horizontal direction, addresses AD1 to AD8 in units of 4 pixels are determined in FIG. 55B, and access in units of 32 pixels can be performed in 8 consecutive bursts. A block of 4 pixels × 4 lines, 1 block (16 pixels) unit, 3 blocks (24 pixels) unit, etc. can be accessed in the same manner. The memory controller 8 includes an address generation unit for memory access in the 4 × 4 format.

  When the start address AD1 of 4 × 4 format and the number of blocks of 4 pixels × 4 lines in the horizontal direction are specified on the frame memory of the image memory unit 4, the memory controller 8 uses the address generation unit, The address AD2 and subsequent addresses for memory access in 4 × 4 format can be calculated, and memory access in 4 × 4 format can be executed.

  As in the case of the 8 × 8 format described above, when the image data is not divisible by 4 pixels, a dummy area 154 is provided at the right end in the horizontal direction and the lower end in the vertical direction as shown in FIG. Thus, the horizontal and vertical sizes are set to be a multiple of four.

  With this 4 × 4 format, the above-described bus access in the base plane matching processing range (20 pixels × 20 lines) 144 is further improved over the 8 × 8 format.

  For the block unit of 4 pixels × 4 lines, the most efficient is that the base plane matching processing range 144 of 20 pixels × 20 lines is exactly 5 blocks in the horizontal direction × vertical direction as shown in FIG. In this case, 5 blocks = 25 blocks are allocated.

  In order to access these 25 blocks, it can be divided into 5 blocks of 4 blocks in the horizontal direction and 5 lines of 1 block in the horizontal direction. A total of 10 transfers, 5 transfers of 4 pixels × 16 bursts, 5 transfers of 4 pixels × 4 bursts per block, may be used.

  On the other hand, the worst example is that the base plane matching processing range 144 of 20 pixels × 20 lines is 6 blocks in the horizontal direction × 6 blocks in the vertical direction of the block of 4 pixels × 4 lines as shown in FIG. = 36 blocks. In this case, as shown in FIG. 57, in the basal plane matching processing range 144 of 20 pixels × 20 lines, there are areas less than 4 pixels and 4 lines at the left and right ends in the horizontal direction and the upper and lower ends in the vertical direction. Therefore, dummy pixel data 154 is inserted therein.

  In order to access these 36 blocks in the 4 × 4 format, it can be divided into 6 lines of 4 blocks in the horizontal direction and 6 lines of 2 blocks in the horizontal direction, and the maximum burst length of FIG. 4 blocks × 16 bursts in units of 4 blocks in 6 transfers, 6 transfers in 4 blocks × 4 bursts in units of 2 blocks, and a total of 12 transfers may be sufficient.

  Therefore, the number of transfer accesses is further smaller than the 16 transfers in the 8 × 8 format shown in FIG. 54, and the ratio of using 16 bursts, which is the most efficient data transfer in the imaging apparatus of this embodiment, is higher. Increasing bus efficiency.

  As described above, during the moving image NR process, when the motion detection / compensation unit 16 reads the matching processing range from the image memory unit 4, it is not the 64 × 1 format and the strip access format, but the block format, in this example. The block access format based on 8 × 8 format or 4 × 4 format is more suitable for memory access.

  Therefore, the moving image NR image as a result of superposition may be written in the image memory unit 4 in the block access format block format as described above.

  However, the image memory unit 4 is accessed through the system bus 2 and the memory controller 8 in addition to the motion detection / motion compensation unit 16, the image superposition unit 17, and the moving image codec unit, as well as other processing units. However, the memory access according to the block format is not necessarily suitable for all of the other processing units.

  For example, in the imaging apparatus of this embodiment, at the time of moving image NR processing, an image after the NR processing is converted into display image data by the NTSC encoder 20 and an image monitor display is performed on the monitor display 6. Since the NTSC encoder 20 basically processes image data in a raster scan format, the image data is received from the image memory unit 4 in a block access format such as an 8 × 8 format or a 4 × 4 format. When accessed, there is a problem that transfer efficiency deteriorates.

  Therefore, in this embodiment, in step S28 of FIG. 19 described above, the NR image of the moving image NR processing result from the image superimposing unit 17 is an 8 × 8 format, a 4 × 4 format, or the like as shown in FIG. Is written in the first memory area 41NR for the NR image in the image memory unit 4 and is written in the second memory area 42NR for the NR image in the image memory unit 4 in the 64 × 1 format. As shown in FIG. 58, in the first memory area 41NR, the reduced surface image is written together with the NR image.

  The motion detection / motion compensation unit 16 obtains the target block from the image correction / resolution conversion unit 15 in a strip format, and the motion detection / motion compensation unit 16 and the image overlay unit 17 have a strip format, 64 Processing is performed for each 64 pixels in the × 1 format. For this reason, when writing the NR image resulting from the image overlay from the image overlay unit 17 into the image memory unit 4 in the 64 × 1 format, the address is skipped in the raster scan format and the strip access format is used. It is done by doing.

  In this embodiment, the NR image written in the block format at the time before one frame in the first memory area 41NR for the NR image of the image memory unit 4 is set as the reference image, and the memory controller 8 The matching processing range is read from the reference image in a block format such as 8 × 8 format or 4 × 4 format, and transferred to the motion detection / compensation unit 16.

  In this embodiment, the NR image written in the 64 × 1 format in the second memory area 42NR for the NR image of the image memory unit 4 in the 64 × 1 format at the time one frame before is in the 64 × 1 format, and The data is read out in the raster scan format and transferred to the moving image codec unit 19. In this example, the moving image is compressed and encoded by the MPEG method and recorded in the recording / reproducing device unit 5. The NR image written in the second memory area 42NR in the 64 × 1 format one frame before is read out in the 64 × 1 format and in the raster scan format and supplied to the NTSC encoder 20. Then, it is converted into an NTSC signal, supplied to the monitor display 6, and displayed as a recorded monitor image.

  As described above, in this embodiment, the NR image is written twice in the image memory unit 4 in both the first memory area 41NR and the second memory area 42NR. Although the capacity of the image memory unit 4 and the power consumption of the image memory unit 4 and the memory controller 8 increase, the efficiency of bus access of the image data from the image memory unit 4 to the motion detection / motion compensation unit 16 is good, and the bus bandwidth is reduced. There is a merit that it can be greatly reduced.

  As described above, in the imaging apparatus of this embodiment, whether to write in the image memory unit 4 only in the strip access format in the 64 × 1 format or in the block access format based on the strip access format and the block format. , Either can be selected. If you want to reduce power consumption or reduce memory capacity, choose to use only the strip access method, and if you want to reduce the bus bandwidth, use two types of strip access format and block access format. It is desirable to choose.

[Second Embodiment of Imaging Device]
As shown in FIG. 1, the imaging apparatus according to the above-described embodiment records an NR image as a result of image superimposition as it is. On the other hand, as shown in FIG. 59, a post-processing / effect unit 30 is provided in the subsequent stage of the image superimposing unit 17, and the post-processing / effect unit 30 performs image synthesis, graphic synthesis, sepia processing, and the like. It is also possible to configure an imaging device that can perform recording after performing post-processing or effect processing for.

  In this case, the processing operation of the post-processing / effect unit 30 is turned on / off by a user operation input through the user operation input unit 3. That is, when the user instructs post-processing or effect processing for an image such as screen synthesis, graphic synthesis, or sepia processing through the user operation input unit 3, the post-processing / effect unit 30 is turned on by the CPU 1. When post-processing or effect processing is not designated, the post-processing / effect unit 30 is turned off by the CPU 1.

  When the post-processing / effect unit 30 is turned off, in the imaging apparatus of the second embodiment, as in the imaging apparatus of the above-described embodiment shown in FIG. As shown in FIG. 60, the NR image of the moving image NR processing result from is written in the first memory area 41NR for the NR image of the image memory unit 4 in a block format such as 8 × 8 format or 4 × 4 format. At the same time, the NR image of the image memory unit 4 is written into the second memory area 42NR in the 64 × 1 format. As shown in FIG. 60, in the first memory area 41NR, the reduced plane image is written together with the NR image.

  Further, when the post-processing / effect unit 30 is turned on, in the imaging apparatus of the second embodiment, the NR image of the moving image NR processing result from the image superimposing unit 17 is as shown in FIG. The data is written only in the first memory area 41NR of the NR image of the image memory unit 4 in a block format such as 8 × 8 format or 4 × 4 format.

  When the post-processing / effect unit 30 is turned on, the post-processing / effect unit 30 performs post-processing or effect processing on images such as screen synthesis, graphic synthesis, and sepia processing. Are written in the second memory area 42NR for the NR image of the image memory unit 4 in the 64 × 1 format. The NR image that has been subjected to post-processing or effect processing cannot be used as a reference image in the next frame.

  In the image pickup apparatus according to the second embodiment, post-processing and the like written in the 64 × 1 format in the second memory area 42NR for the NR image in the image memory unit 4 at the time one frame before is performed. The NR image is read out in the 64 × 1 format and in the raster scan format, transferred to the moving image codec unit 19, subjected to moving image compression encoding, and recorded in the recording / reproducing device unit 5. Further, the post-processed NR image written in the second memory area 42NR in the 64 × 1 format one frame before is read out in the 64 × 1 format and in the raster scan format. , Supplied to the NTSC encoder 20 to be converted into an NTSC signal, supplied to the monitor display 6, and displayed as a recorded monitor image.

  As described above, in the imaging apparatus according to the second embodiment, when the post-processing / effect unit 30 is turned on, the post-processing used as a reference image in the next frame is not performed. It is necessary to write the image and the NR image subjected to post-processing used for recording and recording monitor display in the image memory unit 4. The NR image that has not been post-processed or the like is written in the image memory unit 4 in a block format for use as a reference image. The NR image that has been post-processed or the like is used for recording and recording monitoring. For purposes such as display, data is written in a 64 × 1 format that is easy to read in raster scan format.

[Changes in memory access between base plane and reduced plane during video NR processing]
In the imaging devices of the first and second embodiments described above, a hierarchized block matching method including a reduced surface matching process and a base surface matching process is employed as a block matching method in the motion detection / compensation unit 16. Yes.

  In the basal plane matching process at the time of the moving image NR process, the basal plane target image and the basal plane reference image written in the image memory unit 4 are not only the motion detection / motion compensation unit 16 as shown in FIG. Since other processing units connected to the system bus 2 may access, restrictions on the memory access format occur as described above. However, since a reduced surface image in reduced surface matching is accessed only by the motion detection / motion compensation unit 16, there is no restriction on the format of memory access.

  Therefore, the bus bandwidth can be further reduced by changing the memory access format for the image data in the image memory unit 4 between the base plane matching process and the reduction plane matching process.

  As an example of enabling reduction of the bus bandwidth, here, image data is reduced during the reduction plane matching process. That is, as described above, in the embodiment according to the present invention, only the luminance information Y is used in the calculation of the SAD value in the block matching method.

  However, not only the luminance information Y but also the chroma signal information C (Cr, Cb) is necessary to record the image data by performing the NR process. For this reason, in the above description, when the bus width of the system bus 2 is 64 bits and the maximum burst length is 16 bursts, the data amount at the time of maximum efficiency transfer, that is, 64 bits × 16 bursts = 128 bytes is Y : Cr: Cb = 4: 2: 2 format luminance signal Y (8 bits) and chroma signal C (8 bits) correspond to 4 pixels.

  Incidentally, since the reduced surface image is not a recording target, it is not necessary to use the chroma signal C. Therefore, only the luminance information Y (8 bits) is extracted from the 8 × 8 format YC image data as shown in FIG. 62A, and the 8 × 8 format Y as shown in FIG. The image data can be used for reduced surface matching processing, but the amount of YC image data in that case is half that of the above case.

  Therefore, the amount of data at the time of transfer with the maximum efficiency in the above-described embodiment, that is, 64 bits × 16 bursts = 128 bytes is 2 in a block of 8 lines × 8 pixels = 64 pixels as shown in FIG. It is for 128 pixels, which is a block. Therefore, at the time of the reduction plane matching process, in the memory access to the image memory unit 4, the 128 pixels can be accessed by one transfer.

  That is, at the time of reduced surface matching processing, only Y information is extracted from YC information in the 8 × 8 format, so that the maximum efficiency is 8 in one access (128 bytes) compared to the above example. Data of pixel × 8 lines × 2 blocks can be transferred, and the bus bandwidth can be reduced to ½ without changing the matching accuracy.

  In the reduced plane matching process, the Y image data in FIG. 62B is thinned out in a staggered pattern and packed vertically in the image memory as shown in FIGS. 63A and 63B. By writing to the unit 4, the bus bandwidth can be further reduced.

  At this time, as shown in FIG. 63 (C), data of 4 pixels × 8 lines × 4 blocks at maximum can be transferred with one access (128 bytes), and the matching accuracy is lowered, but the bus bandwidth is 1 / It is possible to reduce to 4.

[Effect of the embodiment]
As described above, the memory access data format for the image memory unit 4 is determined in units of blocks having the most efficient data transfer size in the imaging apparatus of the embodiment, and the image is selected according to the still image NR process and the moving image NR process. By selecting a data format for memory access to the memory unit 4, the bus bandwidth can be reduced and the bus access efficiency can be improved.

  In addition, the memory access data format for the image memory unit 4 is changed during the base plane matching process and during the reduction plane matching process. During the reduction plane matching process, other processing units other than the motion detection / motion compensation unit 16 are used. By making use of the fact that there is no access to the image memory unit 4 and making the data format suitable for motion detection, the bus bandwidth can be further reduced.

  In addition, when post-processing and effect processing are performed after image superimposition, it is necessary to paste image data in the image memory unit 4 in a plurality of formats, but a reference image for matching processing is displayed on the image memory unit 4. By performing write and read access in block format, the bus bandwidth can be reduced.

  Further, according to the above-described embodiment, the image correction / resolution conversion unit 15 performs memory access in a format in which an image is divided into strips, and the unit of compression encoding in the still image codec unit 18 is also a strip format. By doing so, data can be directly transferred from the image correction / resolution conversion unit 15 to the still image codec unit 18 without using a memory, and high-speed continuous shot images can be efficiently compression-encoded and stored in the image memory unit 4. Can be stored.

  Further, when a compression-encoded image stored in the image memory unit 4 is used as a reference image, only the range necessary for block matching processing is decoded by decompression decoding in units of strips obtained by dividing the image. The work area of the image memory unit 4 can be saved. Furthermore, the size of the decoded reference image on the work area is always kept constant by decoding and overwriting the reference image strip to be used next at the address of the unnecessary reference image strip on the image memory unit 4. can do.

  Further, as described above, according to this embodiment, a desired NR image is obtained by changing the selection method of the target image and the reference image, the order of overlay, and the calculation method of overlay according to the setting information. It is possible.

[Other Embodiments and Modifications]
In the above-described embodiment, the JPEG method is used as the compression encoding method in the still image codec unit 18, and the MPEG method is used as the compression encoding method in the moving image codec unit 19, but this is an example. Therefore, it goes without saying that the present invention can be applied to any image compression coding system.

  In the above-described embodiment, the compression encoding processing unit for compressing and encoding a recorded image at the time of still image shooting performs compression encoding of the captured image at the time of block matching. Different compression methods may be used for the compression method and the compression method for the captured image at the time of block matching.

  In the above-described embodiment, a plurality of memory access data formats for the image memory unit 4 are used in the moving image NR processing. However, the moving image NR processing may be used for still image NR processing. In particular, when the still image NR process is performed by the addition method during shooting, the processing speed is also important. Therefore, it is better to adopt the method for reducing the bus bandwidth as described above.

  In addition, the image is divided into a plurality of pixels in the horizontal direction and memory access is performed as strip data. This is because the read / write direction of image data from the image memory is based on the horizontal line direction. Because it was. When the image data is read from and written to the image memory in the vertical direction, the same effect as described above can be obtained by accessing the data as band-shaped data obtained by dividing the image into a plurality of lines in the vertical direction. It goes without saying that can be obtained. In that case, for example, the 64 × 1 format is a format of 64 lines × 1 pixel in the vertical direction.

  In the first embodiment described above, in the first example of the motion vector calculation unit, the search direction in the search range is set to the horizontal line direction, and the search is performed by moving the reference block in order from the upper left of the search range, for example. In addition, the memory for one line of the SAD table is provided, but the search direction in the search range is taken in the vertical direction, for example, the search is started in the vertical direction from the upper left end of the search range, and the vertical direction After the search for one column is completed, the position of the reference block is moved in the horizontal direction by one, for example, one pixel to the right in the vertical column, and the search is performed in the vertical direction from the upper end of the column. You may make it employ | adopt the search method which repeats. As described above, when the search is performed by moving the reference block in the vertical direction sequentially from the upper left end of the search range, a memory for one column in the vertical direction of the SAD table may be provided.

  Here, whether the search direction is taken in the horizontal direction or the search direction in the vertical direction is determined by taking into account the circuit scales of the matching processing unit and the motion vector calculation unit and adopting the one with a smaller circuit scale. preferable.

  As described above, the reference block may be moved not for each pixel and for each line, but for each of a plurality of pixels and for each of a plurality of lines. Therefore, the memory for one line in the horizontal direction in the former case only needs to correspond to the movement position of the reference block in the horizontal direction, and the memory for one column in the vertical direction in the latter case moves the reference block in the vertical direction. Just the position. That is, when the reference block is moved for each pixel or for each line, the memory for one line needs a memory having the capacity corresponding to the number of pixels for one line, and for one column in the vertical direction. As the memory, a memory having a capacity corresponding to the number of lines is required. However, when the reference block is moved for each of a plurality of pixels and for each of a plurality of lines, the memory for one line and the memory for one column are compared with the case where the reference block is moved for each pixel and for each line. Less capacity.

  Further, the interpolation processing method is not limited to the above-described quadratic curve approximation interpolation processing, and as described above, interpolation processing using a cubic curve or a higher order curve may be performed.

  Moreover, although the above-mentioned embodiment is a case where the image processing apparatus by this invention is applied to an imaging device, this invention is not necessarily restricted to an imaging device, It can apply to various image processing apparatuses.

  The above embodiment is a case where the present invention is applied when noise reduction processing is performed by superimposing images using a block matching method. However, the present invention is not limited to this. Can be applied to all image processing apparatuses that are accessed by a plurality of processing units.

1 is a block diagram illustrating a configuration example of an imaging apparatus that is an embodiment of an image processing apparatus according to the present invention. It is a figure for demonstrating the noise reduction process of the captured image in the imaging device of FIG. 1 of embodiment. It is a figure used for description of the block matching process in the imaging device of FIG. 1 of embodiment. It is a figure for demonstrating the noise reduction process of the captured image in the imaging device of FIG. 1 of embodiment. It is a figure for demonstrating the noise reduction process of the captured image in the imaging device of FIG. 1 of embodiment. It is a figure used in order to explain the image processing method of an embodiment of this invention. It is a figure used in order to explain the image processing method of an embodiment of this invention. It is a figure used in order to explain the image processing method of an embodiment of this invention. It is a figure used in order to demonstrate the operation | movement in the motion detection and the motion compensation part in the imaging device of FIG. 1 of embodiment. It is a figure used in order to demonstrate the operation | movement in the motion detection and the motion compensation part in the imaging device of FIG. 1 of embodiment. It is a figure used in order to demonstrate the operation | movement in the motion detection and the motion compensation part in the imaging device of FIG. 1 of embodiment. It is a figure for demonstrating the effect by the process in the motion detection and the motion compensation part in the imaging device of FIG. 1 of embodiment. It is a block diagram which shows the structural example of the motion detection and the motion compensation part in the imaging device of FIG. 1 of embodiment. FIG. 14 is a block diagram of a detailed configuration example of a part of the motion detection / compensation unit of FIG. 13. FIG. 14 is a block diagram of a detailed configuration example of a part of the motion detection / compensation unit of FIG. 13. It is a block diagram which shows the structural example of the image superimposition part in the imaging device of FIG. 1 of embodiment. It is a figure which shows a part of flowchart for demonstrating embodiment of the image processing method by this invention. It is a figure which shows a part of flowchart for demonstrating embodiment of the image processing method by this invention. It is a figure which shows a part of flowchart for demonstrating embodiment of the image processing method by this invention. It is a figure used in order to demonstrate the 1st example of the motion vector calculation part of the motion detection / motion compensation part in the imaging device of an embodiment. It is a figure used in order to demonstrate the 1st example of the motion vector calculation part of the motion detection / motion compensation part in the imaging device of an embodiment. It is a block diagram which shows the structure of the 1st example of the motion vector calculation part of the motion detection / motion compensation part in the imaging device of embodiment. It is a figure which shows the flowchart for demonstrating the processing operation example of the motion vector calculation part of FIG. It is a figure which shows the flowchart for demonstrating the processing operation example of the motion vector calculation part of FIG. It is a figure which shows the flowchart for demonstrating the processing operation example of the motion vector calculation part of FIG. It is a block diagram which shows the structure of the 2nd example of the motion vector calculation part of the motion detection and the motion compensation part in the imaging device of embodiment. FIG. 27 is a diagram illustrating a flowchart for explaining a processing operation example of a motion vector calculation unit in FIG. 26. FIG. 27 is a diagram illustrating a flowchart for explaining a processing operation example of a motion vector calculation unit in FIG. 26. FIG. 27 is a diagram illustrating a flowchart for explaining a processing operation example of a motion vector calculation unit in FIG. 26. It is a figure used in order to demonstrate the specific example in the image processing method of embodiment of this invention. It is a figure used in order to demonstrate the specific example in the image processing method of embodiment of this invention. It is a figure used in order to demonstrate the specific example in the image processing method of embodiment of this invention. It is a figure used in order to demonstrate the specific example in the image processing method of embodiment of this invention. It is a figure for demonstrating the strip access format in the image processing method of embodiment of this invention. It is a figure for demonstrating the memory access system of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the example of the processing unit of the image in the image processing method of embodiment of this invention. It is a figure for demonstrating the flow of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the flow of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the flow of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the flow of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the flow of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the flow of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the flow of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the image processing method of embodiment of this invention. It is a figure for demonstrating the flow of the image data in the image processing method of embodiment of this invention. It is a block diagram which shows the structural example of the imaging device which is other embodiment of the image processing apparatus by this invention. It is a figure for demonstrating the flow of the image data in the image processing method of other embodiment of this invention. It is a figure for demonstrating the flow of the image data in the image processing method of other embodiment of this invention. It is a figure for demonstrating the structural example of the image data at the time of the memory access in the image processing method of embodiment of this invention. It is a figure for demonstrating the structural example of the image data at the time of the memory access in the image processing method of embodiment of this invention. It is a figure used in order to explain block matching processing. It is a figure used in order to explain block matching processing. It is a figure used in order to explain block matching processing. It is a figure used in order to explain block matching processing. It is a figure used in order to explain block matching processing. It is a figure used in order to explain block matching processing.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 4 ... Image memory part, 8 ... Memory controller, 16 ... Motion detection / motion compensation part, 17 ... Image superposition part, 18 ... Still image codec part, 100 ... Target image (target frame), 101 ... Reference image (reference frame) , 102 ... Target block, 104 ... Motion vector 106 ... Search range, 107 ... Reference vector, 108 ... Reference block, 163 ... Matching processing unit, 164 ... Motion vector calculation unit

Claims (16)

An image memory unit capable of recording image data for a plurality of screens;
A first image processing unit that performs first processing on input image data and stores image data of the processing result in the image memory unit;
A second image processing unit that reads out the image data of the processing result from the image memory unit and performs a second process;
A third image processing unit that reads out the image data of the processing result from the image memory unit and performs a third process;
A bus to which the image memory unit, the first image processing unit, the second image processing unit, and the third image processing unit are connected;
A memory controller that controls writing and reading of image data via the bus to the image memory unit;
With
The image data as a result of the first processing in the first image processing unit has the first combination of the number of lines in the vertical direction and the number of pixels in the horizontal direction on the screen by the memory controller. The second block is written in the first memory area of the image memory unit in the first block unit, and the combination of the number of lines in the vertical direction and the number of pixels in the horizontal direction of the screen is different from the first combination. And is written in the second memory area of the image memory unit in a second block unit that is a combination of
Image data written in the first memory area of the image memory unit by the memory controller is read and supplied in units of the first block to the second image processing unit,
Image data written in the second memory area of the image memory unit by the memory controller is read and supplied to the third image processing unit in units of the second block. A featured image processing apparatus. The image processing apparatus according to claim 1.
The memory controller can burst transfer image data in units of a predetermined number of pixels via the bus, and can continuously burst transfer a plurality of times.
The image data of the first block unit and the second block unit is continuously burst transferred by the memory controller a plurality of times in units of the predetermined number of the plurality of pixels when writing to or reading from the image memory unit. An image processing apparatus. The image processing apparatus according to claim 2,
The first block unit includes a plurality of vertical lines and a plurality of horizontal pixels,
The image processing apparatus according to claim 2, wherein the second block unit includes only a plurality of pixels in a writing direction of image data in the image memory unit in a vertical direction or a horizontal direction. The image processing apparatus according to claim 2,
When writing the image data at the edge of the screen to the image memory unit, when the image data has a smaller number of pixels than the first block unit or the second block unit, An image processing apparatus, wherein data is added to form a plurality of pixels in the first block unit or the second block unit. The image processing apparatus according to claim 3.
The predetermined number of the plurality of pixels is p (p is an integer of 2 or more) pixels, and when the maximum continuation possible number of the burst transfer in the memory controller is q times, the second block unit The number of pixels in the writing direction is p × q,
The access to the image memory of the image data in the second block unit is to perform the burst transfer of the p pixels q times for the p × q pixels in the readout direction. A featured image processing apparatus. The image processing apparatus according to claim 2,
Access to the image memory of the image data in the first block unit is performed in the next row or column after burst transfer of pixels in the writing direction to the image memory is completed in the rectangular image region in the first block unit. An image processing apparatus characterized by continuously performing burst transfer of pixels in the writing direction. The image processing apparatus according to claim 1.
The first image processing unit includes a post-processing unit that performs processing reflected on a display image with respect to image data as a result of the first processing, and the second memory area of the image memory unit includes: The image processing apparatus, wherein image data obtained as a result of processing in the post-processing unit is written in units of the second block. A motion vector between two screens is detected, a motion compensated image in which motion compensation is performed on one of the two screens is generated based on the detected motion vector, and the other of the two screens is superimposed on the motion compensated image. An image processing apparatus for obtaining a noise-reduced image,
An image memory unit for storing at least one of the two screens;
A motion compensation image obtained by detecting a motion vector between one of the two screens read from the image memory unit and the other of the two screens and performing motion compensation on one of the two screens based on the detected motion vector. Motion detection / motion compensation means to be generated;
Image superimposing means for superimposing the motion compensated image from the motion detection / compensation means and one or the other of the two screens;
Post-processing means for performing predetermined processing on the image data of the superposition result in the image superposition means;
A bus to which the image memory unit, the motion detection / motion compensation unit, the image superimposing unit, and the post-processing unit are connected;
A memory controller that controls writing of image data from the image superimposing unit to the image memory unit via the bus, and controls reading of image data from the image memory unit;
With
The memory controller is
The image data of the image as a result of the superimposition from the image superimposing means is a first block unit in which the combination of the number of lines in the vertical direction and the number of pixels in the horizontal direction is the first combination, and A second block in which the combination of the number of lines in the vertical direction and the number of pixels in the horizontal direction on the screen is a second combination different from the first combination, while writing to the first memory area of the image memory unit Writing in a second memory area of the image memory unit in units, and
The image data written in the first memory area of the image memory unit is read out in units of the first block, and the motion detection and detection via the bus is performed as one of the two screens for performing the motion detection. Supply to motion compensation means,
The image processing apparatus, wherein the image data written in the second memory area of the image memory unit is read in units of the second block and supplied to the post-processing means via the bus. The image processing apparatus according to claim 8.
The memory controller is capable of continuous burst transfer a plurality of times in units of a predetermined number of pixels via the bus,
The image data of the first block unit and the second block unit is continuously burst-transferred a plurality of times in units of the predetermined number of the plurality of pixels by the memory controller when writing to or reading from the image memory unit. An image processing apparatus. The imaging device according to claim 9,
The first block unit includes a plurality of vertical lines and a plurality of horizontal pixels,
The image processing apparatus according to claim 2, wherein the second block unit includes only a plurality of pixels in a writing direction of image data in the image memory unit in a vertical direction or a horizontal direction. The image processing apparatus according to claim 9.
When writing the image data at the edge of the screen to the image memory unit, when the image data has a smaller number of pixels than the first block unit or the second block unit, An image processing apparatus, wherein data is added to form a plurality of pixels in the first block unit or the second block unit. The imaging device according to claim 10.
The predetermined number of the plurality of pixels is p (p is an integer of 2 or more) pixels, and when the maximum continuation possible number of the burst transfer in the memory controller is q times, the second block unit The number of pixels in the writing direction is p × q,
The access of the image data of the second block unit to the image memory is such that the burst transfer of the p pixels is performed q times for p × q pixels in the readout direction, and
Access to the image memory of the image data of the second block unit is made in a strip form that is continuously performed in a direction that is not a writing direction of the image data to the image memory unit in the vertical direction or the horizontal direction. A featured image processing apparatus. The image processing apparatus according to claim 12.
The other of the two screens supplied to the motion detection / motion compensation means is image data in the strip format, and the motion detection / motion compensation means includes the other image data of the two screens and the motion. Image data of the compensation image is supplied to the image superimposing means in the strip format,
The image data resulting from the superposition of the image superimposing means is written into the first memory area of the image memory unit by the memory controller in units of the first blocks and in units of the second blocks. The image processing device is written in the second memory area of the image memory unit in the strip format. The image processing apparatus according to claim 13.
Image data is read and supplied to the post-processing means from the second memory area of the image memory unit in the second block unit and in a raster scan format by the memory controller. An image processing apparatus. An image memory unit capable of recording image data for a plurality of screens;
A first image processing unit that performs first processing on input image data and stores image data of the processing result in the image memory unit;
A second image processing unit that reads out the image data of the processing result from the image memory unit and performs a second process;
A third image processing unit that reads out the image data of the processing result from the image memory unit and performs a third process;
A bus to which the image memory unit, the first image processing unit, the second image processing unit, and the third image processing unit are connected;
A memory controller that controls writing and reading of image data via the bus to the image memory unit;
A memory access method for image data in an image processing apparatus comprising:
The memory controller is
The first block unit in which the combination of the number of lines in the vertical direction and the number of pixels in the horizontal direction is the first combination of the image data as a result of the first processing in the first image processing unit Thus, the first combination is written to the first memory area of the image memory unit, and the combination of the number of lines in the vertical direction and the number of pixels in the horizontal direction is a second combination different from the first combination. An image data writing step for writing to the second memory area of the image memory unit in units of two blocks;
Reading the image data written in the first memory area in the image data writing step and supplying the image data to the second image processing unit via the bus;
Reading the image data written in the second memory area in the image data writing step and supplying the image data to the third image processing unit via the bus;
A memory access method for image data. A motion vector between two screens is detected, a motion compensated image in which motion compensation is performed on one of the two screens is generated based on the detected motion vector, and the other of the two screens is superimposed on the motion compensated image. An image processing apparatus for obtaining a noise-reduced image,
An image memory unit for storing at least one of the two screens;
A motion compensation image obtained by detecting a motion vector between one of the two screens read from the image memory unit and the other of the two screens and performing motion compensation on one of the two screens based on the detected motion vector. Motion detection / motion compensation means to be generated;
Image superimposing means for superimposing the motion compensated image from the motion detection / compensation means and one or the other of the two screens;
Post-processing means for performing predetermined processing on the image data of the superposition result in the image superposition means;
A bus to which the image memory unit, the motion detection / motion compensation unit, the image superimposing unit, and the post-processing unit are connected;
A memory controller that controls writing of image data from the image superimposing unit to the image memory unit via the bus, and controls reading of image data from the image memory unit;
A memory access method for image data in an image processing apparatus comprising:
The memory controller is
The image data of the image as a result of the superimposition from the image superimposing means is a first block unit in which the combination of the number of lines in the vertical direction and the number of pixels in the horizontal direction is the first combination, and A second block in which the combination of the number of lines in the vertical direction and the number of pixels in the horizontal direction on the screen is a second combination different from the first combination, while writing to the first memory area of the image memory unit An image writing step for writing to the second memory area of the image memory unit in units;
The image data written in the first memory area of the image memory unit in the image data writing step is read and one of the two screens for performing the motion detection is used as the motion detection / motion compensation via the bus. Supplying the means;
Reading the image data written in the second memory area of the image memory unit in the image data writing step, and supplying the image data to the post-processing means via the bus. Memory access method.

Images