JP2009130489A - Image device, imaging/recording method, picked-up image recording and reproducing device, and picked-up image recording and reproducing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a picked-up image recording and reproducing device capable of reproducing a recorded still image by taking the movement of a moved object into consideration at the time of reproduction of the recorded still image, even if the moved object is photographed and recorded as a still image. <P>SOLUTION: The picked-up image recording and reproducing device includes a movement feature detection means for detecting movement features of an object, according to photographing data of the object obtained from an image sensor for a recording instruction operation by a user. The device associates image data of a still image of the object at the time of the recording instruction operation with information on movement feature of the object detected by the movement feature detection means, and records them on a record medium, based on the recording instruction operation by the user. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an imaging apparatus such as a digital still camera and an imaging recording method thereof. The present invention also relates to a captured image recording / reproducing apparatus and a captured image recording / reproducing method for recording and reproducing a captured still image.

  In recent years, digital still cameras have improved the display performance of captured images as well as the shooting performance with the increase in the capacity of image memory and the increase in the number of pixels of the display unit. For example, thumbnails of captured still images are displayed. In addition, there has been proposed one having a function of executing a slide show in which a plurality of captured still images are sequentially displayed on a display at predetermined time intervals.

  For example, in Patent Document 1 (Japanese Patent Laid-Open No. 2005-182196), when displaying each still image in a slide show, the still image is displayed according to, for example, the number of face detections and the face position. A technique for performing a display effect process for zooming from the entire display to the face position or panning on a line passing through the face is described.

  Further, in Patent Document 2 (Japanese Patent Laid-Open No. 2006-260029), shooting date information is added to a captured still image and recorded, and in a slide show, an image of a still image having a shooting date and time adjacent to each other is displayed. At the time of switching, it is described that a contrivance is made such as controlling to gradually switch from the previous image display to the subsequent image display.

The above-mentioned reference patent documents are as follows.
JP 2005-182196 A JP 2006-260029 A

  The conventional slide show, including the case of Patent Document 1, is merely for switching and displaying still images at the time of switching, which is interesting for a user who views images.

  In this regard, in the image processing apparatus described in Patent Document 2, in the slide show, the display at the time of switching the still image is devised according to the shooting time information, so that the user can enjoy a slide show with high entertainment properties. There is a merit that you can.

  By the way, the captured still image includes not only a stationary subject but also a moving subject recorded as a still image at the time when the shutter button is pressed. For this reason, in a slide show, when a special effect display at the time of switching still images is performed by a predetermined method, the display of the moving subject is ignored, which may cause a sense of discomfort. For example, when a subject that has moved in the right direction is displayed so as to be displayed sequentially from the left direction to the right direction of the screen, the direction of movement may be reversed and a sense of discomfort may occur.

  In view of the above points, the present invention makes it possible to perform reproduction in consideration of the movement of a moving subject when reproducing the recorded still image even when the subject is photographed and recorded as a still image. Objective.

In order to solve the above problems, the present invention provides:
An image sensor;
Motion feature detection means for detecting a motion feature of the subject from imaging data of the subject from the imaging device before a user's recording instruction operation;
Based on the recording instruction operation of the user, the image data of the still image of the subject at the time of the recording instruction operation and the information on the motion feature of the subject detected by the motion feature detection unit are associated with each other on a recording medium. Recording means for recording;
An imaging device is provided.

  In the present invention having the above-described configuration, the movement feature of the subject is detected from the imaging data of the subject before the user's recording instruction operation (for example, shutter button operation). When the user performs a recording instruction operation, the detected motion feature information is recorded on a recording medium in association with image data of a still image of the subject at the time of the user's recording instruction operation.

  The recorded motion feature is read and extracted together with the still image data from the recording medium, and the motion feature is recognized. Therefore, for example, when a slide show is performed, it is possible to display a special effect at the time of switching still images according to the recognized motion characteristics, and to expect a slide show that does not feel strange as a motion.

  According to the present invention, since the motion characteristics of the recorded still image immediately before recording are recorded in association with the image data of the still image, the information of the motion characteristics recorded in association with the still image is reproduced. The special effect processing used can be performed.

  Hereinafter, embodiments of an imaging apparatus according to the present invention will be described with reference to the drawings. In the imaging device according to the embodiment described below, it is possible to capture and record a still image and capture and record a moving image.

  In the imaging apparatus according to this embodiment, the still image and the moving image to be recorded are noise-reduced images obtained by superimposing images while performing motion compensation. That is, in the imaging apparatus of this embodiment, a motion vector between two screens is detected by block matching, a motion compensated image is generated using the detected motion vector, and the generated motion compensated image and a noise reduction target image are detected. Noise reduction is performed by superimposing, and image data of the noise-reduced image is recorded.

[Description of block matching overview]
In block matching, a motion vector between two screens between a reference screen that is a target screen and an original screen (referred to as a target screen) that is a source of motion of the reference screen is a block of a rectangular area having a predetermined size. Is calculated by calculating the correlation between the reference screen and the original screen.

  When the original screen is temporally before the reference screen (for example, in the case of motion detection in MPEG), and when the reference screen is temporally prior to the original screen (for example, described later) Both in the case of noise reduction by superimposing image frames to be performed).

  In this specification, the screen means an image that is composed of image data of one frame or one field and is displayed on the display as one sheet. For convenience of the following description in this specification, the screen is composed of one frame, and the screen may be referred to as a frame. For example, the reference screen may be referred to as a reference frame, and the original screen may be referred to as an original frame.

  2-7 is a figure for demonstrating the outline | summary of a block matching. In the block matching method described here, for example, as shown in FIG. 2 (A), an original frame (target frame) 100 having a predetermined size composed of a plurality of pixels in the horizontal direction and a plurality of lines in the vertical direction, respectively. Divide into rectangular areas (called blocks). Each of the plurality of blocks 102 in the target frame is referred to as a target block.

  In block matching, a block having a high correlation with the target block 102 is searched from the reference frame 101. As a result of this search, the block 103 (see FIG. 2B) detected in the reference frame 101 as having the highest correlation is referred to as a motion compensation block. Further, the amount of displacement between the target block 102 and the motion compensation block 103 is referred to as a motion vector (see reference numeral 104 in FIG. 2B).

  A motion vector 104 corresponding to a position shift (including a position shift amount and a position shift direction) between the target block 102 and the motion compensation block 103 is the same as the position of each target block 102 of the target frame 100 in the reference frame 101. Assuming that the projection image block 109 of the target block 102 is assumed as the position, a positional shift between the position of the projection image block 109 of the target block (for example, the center position) and the position of the motion compensation block 103 (for example, the center position). It also has a positional deviation amount and a positional deviation direction component.

  An outline of the block matching processing operation will be described. As shown by a dotted line in FIG. 3, a projected image block 109 of the target block is assumed at the same position as the position of each target block 102 of the target frame 100 in the reference frame 101, and the center of the projected image block 109 of the target block is assumed. The coordinates are set as a motion detection origin 105. Then, assuming that the motion vector 104 exists within a certain range from the motion detection origin 105, a predetermined range centered on the motion detection origin 105 is set as the search range 106 (see the one-dot chain line in FIG. 3). .

  Next, a block (referred to as a reference block) 108 having the same size as the target block 102 is set on the reference screen. The position of the reference block 108 is moved within the search range 106 in units of one pixel or a plurality of pixels, for example, in the horizontal direction and the vertical direction. Accordingly, a plurality of reference blocks 108 are set in the search range 106.

  Here, the reference block 108 is moved within the search range 106. In this example, since the motion detection origin 105 is the center position of the target block, the center position of the reference block 108 is moved within the search range 106. This means that the pixels constituting the reference block 108 may protrude beyond the search range 106. Therefore, the range in which pixel data is read as a reference block is wider than the search range 106. The range in which pixel data is read as the reference block is referred to as a matching processing range.

  Then, for each reference block 108 to be set in the search range, a vector (referred to as a reference vector) 107 (refer to FIG. 3) representing the amount of displacement and the direction of displacement between each reference block 108 and the target block 102. Then, the correlation between the image content of the reference block 108 at the position indicated by each reference vector 107 and the image content of the target block 102 is evaluated.

  As shown in FIG. 4, the reference vector 107 is a vector (Vx, Vy) where the horizontal displacement (X direction) of the reference block 108 is Vx and the vertical displacement (Y direction) is Vy. When the position coordinates (for example, center position coordinates) of the reference block 108 and the position coordinates (for example, center position coordinates) of the target block 102 are the same, the reference vector 107 is represented as a vector (0, 0). .

  For example, when the reference block 108 is at a position shifted by one pixel in the X direction from the position of the target block 102, the reference vector 107 is a vector (1, 0). As shown in FIG. 5, when the reference block 108 is at a position shifted by 3 pixels in the X direction and 2 pixels in the Y direction from the position of the target block 102, the reference vector 107 is a vector (3, 2 )

  That is, as shown in the example of FIG. 5, when the positions of the target block 102 and the reference block 108 are the center positions of the respective blocks, each reference vector 108 has a corresponding reference block 108. This means a displacement between the center position and the center position of the target block 102 (a vector including a displacement amount and a displacement direction).

  The reference block 108 moves within the search range 106. In this case, the center position of the reference block 108 moves within the search range 106. As described above, since the reference block 108 includes a plurality of pixels in the horizontal direction and the vertical direction, the maximum range in which the reference block 108 that is subject to block matching processing with the target block 102 is as shown in FIG. Furthermore, the matching processing range 110 is wider than the search range 106.

  Then, the position of the reference block 108 detected as having the strongest correlation with the image content of the target block 102 is detected as the position of the target block 102 of the target frame 100 in the reference frame 101 (position after moving), The detected reference block is set as the motion compensation block 103 described above. Then, the displacement amount between the detected position of the motion compensation block 103 and the position of the target block 102 is detected as a motion vector 104 as an amount including a direction component (see FIG. 2B). ).

  Here, the correlation value indicating the strength of correlation between the target block 102 and the reference block 108 moving in the search range 106 is basically calculated using the pixel values corresponding to the target block 102 and the reference block 108. However, as the calculation method, a method using a root mean square and various other methods have been proposed.

  Among them, as a correlation value generally used when calculating a motion vector, for example, the absolute value of the difference between the luminance value of each pixel in the target block 102 and the luminance value of each corresponding pixel in the reference block 106 is used. The sum of the values of all the pixels in the block (the sum of the absolute values of the differences is referred to as the sum of absolute values of the difference. Hereinafter, the sum of the absolute values of the differences is referred to as SAD (Sum of Absolute Difference)). Used (see FIG. 6).

  When the SAD value is used as the correlation value, the smaller the SAD value, the stronger the correlation. Therefore, among the reference blocks 108 that move in the search range 106, the reference block 108 at the position where the SAD value is minimum becomes the strongest correlation reference block with the strongest correlation, and this strongest correlation reference block is detected as the motion compensation block 103. The amount of displacement of the detected motion compensation block 103 relative to the position of the target block 102 is detected as a motion vector.

  As described above, in block matching, the amount of positional deviation of each of the plurality of reference blocks 108 set in the search range 106 with respect to the position of the target block 102 is expressed by the reference vector 107 as an amount including a direction component. The The reference vector 107 of each reference block 108 has a value corresponding to the position of the reference block 108 on the reference frame 102. As described above, in block matching, the reference vector of the reference block 108 in which the SAD value that is the correlation value has the minimum value is detected as the motion vector 104.

  Therefore, in block matching, generally, as shown in FIG. 7, the SAD value between each of the plurality of reference blocks 108 set in the search range 106 and the target block 102 (hereinafter referred to as reference block for simplicity of explanation). The reference vector 107 corresponding to the position of each reference block 108 (hereinafter referred to as the reference vector 107 corresponding to the position of the reference block 106 is referred to as the reference vector 107 of the reference block 108 for the sake of simplicity). By detecting the reference block 108 having the smallest SAD value from the SAD values for all the reference blocks 108 stored in the memory. The vector 104 is detected.

  Corresponding to each of the reference vectors 107 corresponding to the positions of the plurality of reference blocks 108 set in the search range 106, a correlation value (in this example, SAD value) for each reference block 108 is stored. This is called a correlation value table. In this example, since the SAD value which is the sum of absolute differences is used as the correlation value, this correlation value table will be referred to as a difference absolute value sum table (hereinafter referred to as SAD table).

  The SAD table TBL in FIG. 7 shows this, and in this SAD table TBL, the correlation value (SAD value in this example) for each reference block 108 is referred to as a correlation value table element. In the example of FIG. 7, the SAD value indicated by reference numeral 111 is the SAD value when the reference vector is the vector (0, 0). In the example of FIG. 7, since the minimum value of the SAD value is “7” when the reference vector is the vector (3, 2), the motion vector 104 to be obtained is (3, 2).

  In the above description, the positions of the target block 102 and the reference block 108 mean arbitrary specific positions of these blocks, for example, the center position, and the reference vector 107 is the target block in the reference frame 102. The amount of deviation (including direction) between the position of the projection image block 109 of 102 and the position of the reference block 108 is shown.

  Since the reference vector 107 corresponding to each reference block 108 is a position shift of each reference block 108 from the position of the projection image block 109 corresponding to the target block 102 on the reference frame 101, the reference block When the position 108 is specified, the value of the reference vector is also specified corresponding to the position. Therefore, when the address of the correlation value table element of the reference block in the memory of the SAD table 110 is specified, the corresponding reference vector is specified.

  Note that the SAD value may be calculated for two or more target blocks at the same time. As the number of target blocks to be processed simultaneously increases, the processing speed increases. However, since the hardware scale for calculating the SAD value increases, there is a trade-off between speeding up the processing and increasing the circuit scale.

[Regarding Noise Reduced Image as Recorded Image in Imaging Device of Embodiment]
In the imaging apparatus according to the embodiment described below, as shown in FIG. 8, a plurality of images, for example, P1, P2, and P3, which are continuously captured, are aligned using motion detection and motion compensation. Thereafter, by superimposing, an image Pmix with reduced noise can be obtained. That is, since noise in each of a plurality of images is random, the noise is reduced with respect to the images by superimposing images having the same content.

  In the following description, the noise reduction by superimposing a plurality of images using motion detection and motion compensation is referred to as NR (Noise Reduction), and the image whose noise is reduced by NR is referred to as an NR image. To do.

  In this specification, a screen (image) to be subjected to noise reduction is defined as a target screen (target frame), and a screen to be superimposed is defined as a reference screen (reference frame). Images taken consecutively are misaligned due to the camera shake of the photographer, etc., and alignment is important to superimpose both images. Here, it is necessary to consider that there is a movement of the subject in the screen as well as a blur of the entire screen such as a camera shake.

  For this reason, in order to increase the noise reduction effect for the subject as well, as shown in FIG. 9, it is necessary to align each of the plurality of target blocks 102 generated by dividing the target frame 100. Become.

  Accordingly, in this embodiment, a motion vector (hereinafter referred to as a block motion vector) 104B is detected for each of the plurality of target blocks 102, and the corresponding block motion vector 104B is detected for each target block 102. To align the images and superimpose the images.

  In the imaging apparatus of this embodiment, at the time of still image shooting, as shown in FIG. 10, a plurality of images are shot at a higher speed than a normal frame rate (for example, 60 fps (frame / second)). The captured image of the eye is used as the target frame 100, and the second and subsequent images are used as the reference frame 101, and the images are superimposed and recorded as a still image captured image. That is, when the photographer depresses the shutter button of the image pickup apparatus, the predetermined number of images are photographed at a high speed such as 240 fps, and the image (frame) photographed first is later in time. A plurality of captured images (frames) are superimposed.

  At the time of moving image shooting, as shown in FIG. 11, the current frame image output from the image sensor is the image of the target frame 100, and the past image of the previous frame is the image of the reference frame 101. Therefore, in order to reduce the noise of the current frame image, the image of the previous frame of the current frame is superimposed on the current frame.

  In the description of the method for superimposing the images shown in FIGS. 10 and 11 above, when the frame rate of the moving image to be recorded is set to a normal frame rate of 60 fps, the frame rate of 2 from the image sensor is 60 fps. This is a case where the captured image signal is obtained by superimposing the image frames, and as a result, the frame rate of 60 fps with reduced noise.

  However, when the captured image is output from the image sensor at a higher frame rate of, for example, 240 fps, even when capturing a moving image, one image is superimposed on each other. It is also possible to obtain a captured image signal having a frame rate of 60 fps by generating a moving image frame. Needless to say, 240 fps captured moving image can be obtained by superimposing two image frames in the same manner as in this example, and as a result, a noise-captured captured image signal having a frame rate of 240 fps can be obtained.

  In this embodiment, the accuracy of the pixel pitch of the image to be processed (referred to as pixel (pixel) accuracy) is higher, that is, the original screen so that the images can be superimposed with higher accuracy. The block motion vector is detected with high accuracy with a pitch smaller than the pixel pitch of (target frame) (this high accuracy will be referred to as sub-pixel accuracy). In order to calculate the motion vector with the sub-pixel accuracy, as will be described later, in this embodiment, the interpolation processing is performed using the motion vector obtained with the pixel accuracy and the reference vector in the vicinity thereof. .

[Hardware Configuration Example of Imaging Device of Embodiment]
FIG. 1 is a block diagram illustrating a hardware configuration example of an example of an embodiment of an imaging apparatus according to the present invention.

  As shown in FIG. 1, in the imaging apparatus of this embodiment, a CPU (Central Processing Unit) 1 is connected to a system bus 2, and an imaging signal processing system 10 and a user operation input unit 3 are connected to the system bus 2. The image memory unit 4 and the recording / reproducing device unit 5 are connected to each other. In this specification, the CPU 1 includes a ROM (Read Only Memory) and a work area RAM (Random Access Memory) that store programs for performing various software processes, although not shown. .

  The image memory unit 4 is connected to the system bus 2 via a memory controller 8 that controls writing to and reading from the image memory unit 4.

  In response to an imaging recording start operation through the user operation input unit 3, the imaging signal processing system of the imaging apparatus in FIG. 1 performs recording processing of captured image data as described later. In response to an operation for starting the reproduction of the captured image through the user operation input unit 3, the imaging signal processing system 10 of the imaging apparatus in FIG. 1 reproduces the captured image data recorded on the recording medium of the recording / reproduction device unit 5. Perform processing. Note that each unit described later of the imaging signal processing system 10 receives a control command from the CPU 3 through the control register unit 7 and executes each process while receiving control from the CPU 3.

  As shown in FIG. 1, in the imaging signal processing system 10, incident light from a subject through a camera optical system (not shown) provided with an imaging lens 10 </ b> L is irradiated to the imaging element 11 and imaged. In this example, the imaging device 11 is configured by a CCD (Charge Coupled Device) imager. Note that the image sensor 12 may be a CMOS (Complementary Metal Oxide Semiconductor) imager.

  In the imaging apparatus of this example, when an imaging recording start operation is performed, a video input through the lens 10L is converted into a captured image signal by the imaging element 11, and a signal synchronized with the timing signal from the timing signal generation unit 12 As a result, an analog imaging signal, which is a RAW signal (raw signal) in a Bayer array composed of three primary colors of red (R), green (G), and blue (B), is output. The output analog imaging signal is supplied to the preprocessing unit 13, subjected to preprocessing such as defect correction and γ correction, and supplied to the data conversion unit 14.

  The data converter 14 converts the analog imaging signal, which is a RAW signal input thereto, into a digital imaging signal (YC data) composed of a luminance signal component Y and a color difference signal component Cb / Cr, and the digital image The imaging signal is supplied to the image correction / resolution conversion unit 15. The image correction / resolution conversion unit 15 converts the digital imaging signal to the resolution designated through the user operation input unit 3 and supplies the converted signal to the image memory unit 4 via the system bus.

  When the shooting instruction through the user operation input unit 3 is a still image shooting instruction by pressing the shutter button, the digital imaging signal whose resolution has been converted by the image correction / resolution conversion unit 15 includes the above-described plurality of frames in the image memory. Part 4 is written. Then, after images for a plurality of frames are written in the image memory unit 4, the image data of the target frame and the image data of the reference frame are read by the motion detection / motion compensation unit 16, and in this embodiment as will be described later. A block matching process is performed to detect a motion vector, and based on the detected motion vector, an image overlay process as described later is performed by the image overlay unit 17. As a result of the overlay, noise reduction is performed. The image data of the obtained NR image is stored in the image memory unit 4.

  Then, the image data of the superposed NR image stored in the image memory unit 4 is codec converted by the still image codec unit 18, and for example, a DVD (Digital Versatile Disc) or the like of the recording / reproducing device unit 5 through the system bus 2. It is recorded on a recording medium such as a hard disk. In this embodiment, the still image codec unit 18 performs image compression encoding processing on a still image of JPEG (Joint Photographic Experts Group) system.

  At the time of still image shooting, before the shutter button is pressed, the image data from the image correction / resolution conversion unit 15 is supplied to the NTSC (National Television System Committee) encoder 20 through the image memory unit 4. The NTSC encoder 20 converts it into an NTSC standard color video signal. Then, the image signal converted into the NTSC standard color video signal from the NTSC encoder 20 is supplied to a monitor display 6 composed of, for example, an LCD (Liquid Crystal Display), and before the still image shooting at the time of still image shooting. The monitor image is displayed on the monitor screen.

  This monitor image before shooting a still image is the same as the monitor image before starting moving image shooting. Hereinafter, an image output from the image sensor at the time of non-shooting recording is displayed as an EE image or a liquid crystal display. The pre-photographing monitor is called an EE monitor.

  In the still image shooting mode, the image data of the normal frame rate (60 fps) from the image correction / resolution conversion unit 15 is supplied to the motion detection / motion compensation unit 16 in the EE pre-photographing EE monitor. A motion vector for each block is detected, and the motion vector for each detected block of each frame is held in the motion feature detection unit 30.

  Then, when there is an instruction to press the shutter button, which means the time of recording a still image, the still image NR image obtained by the still image NR process as described above is recorded on the recording medium by the recording / reproducing apparatus unit 5. However, at this time, the motion feature detection unit 30 uses the motion vector for each block for each of a plurality of frames, and the motion feature detection unit 30 performs the motion immediately before shooting the recorded still image. A feature and a feature motion vector for representing the direction and magnitude of the motion feature are detected as described later. Here, the feature motion vector is a motion vector for a motion feature for one screen, and represents a motion direction and a motion amount of a motion feature immediately before shooting a recorded still image.

  In this embodiment, the information of the motion feature and the feature motion vector detected by the motion feature detection unit 30 is supplied to the additional information processing unit 31 as motion information of a still image to be recorded. The image is recorded on a recording medium in association with the image data of the still image NR image to be recorded.

  Here, as a method of recording the still image NR image and the motion information in association with each other, the still image NR image is recorded as a file for each sheet and the motion information is recorded in the additional information file associated with the still image NR image file. In this embodiment, tag information defined in an image file format, which is a recording format of a still image NR image, for example, Exif (Exchangeable Image File Format Digital Signature Cameras) may be adopted. As an example, a method of inserting and adding into an image filter is employed.

  Although details of motion information will be described later, in this embodiment, motion information and feature motion vector information are recorded as motion information, but only motion feature information may be recorded. In addition to the feature motion vector, information regarding motion may be further added.

  The captured still image data recorded on the recording medium of the recording / reproducing apparatus unit 5 as described above is read in response to a still image reproduction start operation through the user operation input unit 3 and supplied to the still image codec unit 18. Playback decoding. The reproduced and decoded image data is supplied to the monitor display 6 through the NTSC encoder 20, and the reproduced still image is displayed on the display screen. Although not shown in FIG. 1, the output video signal from the NTSC encoder 20 can be derived outside through a video output terminal.

  Still image reproduction can be performed for each still image, but the image pickup apparatus of this embodiment reproduces a plurality of recorded still images at predetermined time intervals. It has a slideshow playback function that enables sequential switching playback.

  The slide show generation unit 32 is a processing unit for realizing the slide show reproduction function, and is activated by the CPU 1 when a user gives a slide show reproduction start instruction through the user operation input unit 3.

  When a slide show playback start instruction is given, the image data of a plurality of still images read from the recording / playback device unit 5 are sequentially played back and decoded by the still image codec unit 18, and the playback decoded still images are played back. The image data is supplied to the slide show generation unit 32.

  In this embodiment, during the slide show reproduction, the additional information processing unit 31 extracts the motion information as the additional information recorded in association with the still image data and supplies the motion information to the slide show generating unit 32.

  As will be described later, the slide show generation unit 32 uses the motion information to combine the two adjacent still images with the special effect processing at the time of switching the still images and the still images to be displayed as the slide show. Process such as. Then, the slide show generating unit 32 supplies the processed image data to the monitor display 6 through the NTSC encoder 20 so as to display the playback image of the slide show.

  Details of the special effect processing at the time of still image switching based on motion information in the slide show reproduction and a specific example of the synthesis processing of two adjacent still images will be described later.

  Next, a case where the shooting instruction through the user operation input unit 3 is a moving image shooting instruction by pressing a moving image recording button will be described. At the time of moving image recording, the resolution-converted image data is written to the image memory unit 4 and sent to the motion detection / motion compensation unit 16 in real time, and block matching processing in this embodiment as described later is performed. Then, a motion vector is detected, and based on the detected motion vector, an image superimposing process as described later is performed in the image superimposing unit 17, and as a result of the superimposition, the noise-reduced NR image Image data is stored in the image memory unit 4.

  The image data of the superimposed NR image stored in the image memory unit 4 is codec-converted by the moving image codec unit 19 while being output to the display screen of the monitor display 6 through the NTSC encoder unit 20, and the system bus 2 Is supplied to the recording / reproducing apparatus unit 5 and recorded on a recording medium such as a DVD or a hard disk. In this embodiment, the moving image codec unit 18 performs image compression coding processing on moving images of the MPEG (Moving Picture Experts Group) method.

  As described above, in the moving image shooting mode, the EE image similar to that in the above-described still image shooting mode is displayed on the display 6 before the moving image recording button is pressed, and after the moving image recording button is pressed, The moving image NR image to be recorded is displayed on the display 6 as an EE image, and a recorded image monitor is performed.

  The captured moving image data recorded on the recording medium of the recording / reproducing apparatus unit 5 as described above is read in response to a moving image reproduction start operation through the user operation input unit 3, supplied to the moving image codec unit 19, and reproduced. Decoded. The reproduced and decoded image data is supplied to the monitor display 6 through the NTSC encoder 20, and the reproduced image is displayed on the display screen. Although not shown in FIG. 1, the output video signal from the NTSC encoder 20 can be derived outside through a video output terminal.

  The motion detection / compensation unit 16 described above can be configured by hardware, or can be configured by using a DSP (Digital Signal Processor). Furthermore, software processing can be performed by the CPU 1.

  Similarly, the image superimposing unit 17 can also be configured by hardware or can be configured by using a DSP. Furthermore, software processing can be performed by the CPU 1. The same applies to the still image codec unit 18 and the moving image codec unit 19.

[Description of Motion Detection / Compensation Unit 16]
In this embodiment, the motion detection / compensation unit 16 basically performs motion vector detection by performing block matching processing using the SAD values described with reference to FIGS. To. However, in this embodiment, the motion detection / motion compensation unit 16 is configured by hardware as described later, and performs block matching processing hierarchized by common hardware. Further, as will be described later, the noise reduction processing for the still image and the noise reduction processing for the moving image can be realized by common hardware.

<Outline of Hierarchical Block Matching Processing of Embodiment>
A motion vector detection process in a general conventional block matching is performed by moving a reference block in pixel (pixel) units (one pixel unit or a plurality of pixel units), and calculating a SAD value for the reference block at each moving position. A SAD value indicating a minimum value is detected from the calculated SAD values, and a motion vector is detected based on a reference block position exhibiting the minimum SAD value.

  However, in such a conventional motion vector detection process, the reference block is moved in units of pixels within the search range, so that the number of matching processes for calculating the SAD value increases in proportion to the search range to be searched. As a result, the matching processing time increases and the capacity of the SAD table also increases.

  Therefore, in this embodiment, a reduced image is created for the target image (target frame) and the reference image (reference frame), block matching is performed on the created reduced image, and based on the motion detection result in the reduced image. Perform block matching on the original target image. Here, the reduced image is referred to as a reduced surface, and the original image that has not been reduced is referred to as a base surface. Therefore, in the second embodiment, after performing block matching on the reduced surface, block matching on the base surface is performed using the matching result.

  12 and 13 show images of image reduction of the target frame (image) and the reference frame (image). That is, in this embodiment, for example, as shown in FIG. 12, the base surface target frame 130 is reduced to 1 / n (n is a positive number) in the horizontal direction and the vertical direction to reduce the reduction surface. The target frame 132 is assumed. Accordingly, the basal plane target block 131 generated by dividing the basal plane target frame 130 into a plurality is a reduced plane in which each of the horizontal direction and the vertical direction is reduced to 1 / n × 1 / n in the reduced plane target frame. The target block 133 is obtained.

  Then, the reference frame is reduced in accordance with the image reduction magnification 1 / n of the target frame. That is, as shown in FIG. 13, the base plane reference frame 134 is reduced to 1 / n in each of the horizontal direction and the vertical direction to form a reduced plane reference frame 135. The motion vector 104 for the motion compensation block 103 detected on the base plane reference frame 134 is detected as a reduced plane motion vector 136 reduced to 1 / n × 1 / n in the reduced plane reference frame 135. .

  In the above example, the image reduction magnification of the target frame and the reference frame is the same. However, in order to reduce the calculation amount, different image reduction magnifications are used for the target frame (image) and the reference frame (image). Matching may be performed by matching the number of pixels in both frames by processing such as interpolation.

  Further, although the reduction ratios in the horizontal direction and the vertical direction are the same, the reduction ratios may be different between the horizontal direction and the vertical direction. For example, when the horizontal direction is reduced to 1 / n and the vertical direction is reduced to 1 / m (m is a positive number, n ≠ m), the reduced screen is 1 / n × 1 of the original screen. / M.

  FIG. 14 shows the relationship between the reduced plane reference vector and the base plane reference vector. If the motion detection origin 105 and the search range 106 are determined as shown in FIG. 14A in the basal plane reference frame 134, the reduced plane reference frame 135 is reduced to 1 / n × 1 / n. Then, as shown in FIG. 14B, the search range is a reduced plane search range 137 reduced to 1 / n × 1 / n.

  In this embodiment, within the reduction plane search range 137, reduction plane reference vectors 138 representing the amount of positional deviation from the motion detection origin 105 at the reduction plane reference frame 135 are set, and the respective reduction plane reference vectors 138 are set. The correlation between the reduction plane reference block 139 at the position indicated by and the reduction plane target block 131 (not shown in FIG. 14) is evaluated.

  In this case, since block matching is performed on the reduced image, the number of reduced surface reference block positions (reduced surface reference vectors) for which SAD values should be calculated in the reduced surface reference frame 135 can be reduced, and the number of times of calculating the SAD value. The processing can be speeded up by the amount of (the number of matching processings), and the SAD table can be made small.

  As shown in FIG. 15, by correlation evaluation by block matching between a plurality of reduced surface reference blocks 139 and a reduced surface target block 131 set within a reduced surface matching processing range 143 determined according to the reduced surface search range 137. The reduced plane motion vector 136 in the reduced plane reference frame 135 is calculated. The accuracy of the reduced plane motion vector 136 is as low as n times one pixel because the image is reduced to 1 / n × 1 / n. Therefore, even if the calculated reduced plane motion vector 136 is multiplied by n, the motion vector 104 with 1 pixel accuracy cannot be obtained in the base plane reference frame 134.

  However, in the base plane reference frame 134, it is clear that the base plane motion vector 104 with 1 pixel accuracy exists in the vicinity of the motion vector obtained by multiplying the reduced plane motion vector 136 by n.

  Therefore, in this embodiment, as shown in FIGS. 14C and 15, in the base plane reference frame 134, the position indicated by the motion vector (base plane reference vector 141) obtained by multiplying the reduced plane motion vector 136 by n times is indicated. A basal plane search range 140 is set in a narrow range where the basal plane motion vector 104 is considered to exist as a center, and a basal plane matching processing range 144 is set according to the set basal plane search range 140.

  Then, as shown in FIG. 14C, a base surface reference vector 141 in the base surface reference frame 134 is set to indicate the position in the base surface search range 140, and the position indicated by each base surface reference vector 141 is set. The base plane reference block 142 is set to block matching in the base plane reference frame 134.

  As shown in FIG. 15, the base plane search range 140 and base plane matching processing range 144 set here are obtained by multiplying the reduction plane search range 137 and the reduction plane matching processing range 143 by n times the reciprocal of the reduction ratio. It may be a very narrow range compared to the search range 137 ′ and the matching processing range 143 ′.

  Therefore, when block matching processing is performed only on the base plane without performing hierarchical matching, a plurality of reference blocks are set in the search range 137 ′ and matching processing range 143 ′ on the base plane, Although it is necessary to perform an operation for obtaining a correlation value with the target block, in the hierarchical matching process, the matching process may be performed only in a very narrow range as shown in FIG.

  For this reason, the number of base plane reference blocks set in the base plane search range 140 and base plane matching processing range 144, which are the narrow ranges, is very small, and the number of matching processes (number of correlation value calculations) and the SAD to be held The value can be made very small, the processing can be speeded up, and the SAD table can be reduced in size.

  In this way, if the pixel-basis motion vector 136 is detected in the base-surface reference frame 134, in this embodiment, the SAD value of the reference block pointed to by the base-surface motion vector 136, that is, the minimum SAD value, In this example, a quadratic curve approximate interpolation process is also performed using the neighboring SAD values in the neighborhood to calculate a base-plane high-precision motion vector with sub-pixel accuracy.

  The base plane high-precision motion vector with subpixel accuracy will be described. In the block matching method described above, since block matching is performed in units of pixels, the motion vector is calculated only with pixel accuracy. As shown in FIG. 7, the point where the matching process is performed, that is, the position of the reference block exists with pixel accuracy, and in order to calculate a more accurate motion vector, matching processing in sub-pixel units is required. become.

In order to calculate a motion vector with N times the pixel accuracy (pixel pitch is 1 / N), if the matching process is performed in units of N times the pixel, the SAD table becomes about N ² times larger, Memory is required. In addition, for the block matching process, an image that is N times the upper sample must be generated, and the scale of hardware increases dramatically.

  Therefore, it is considered that a motion vector with sub-pixel accuracy is calculated from the SAD table that has been subjected to the matching process in units of pixels by interpolating the SAD table using a quadratic curve. In this case, instead of quadratic curve approximation interpolation, linear interpolation or higher-order approximation curve interpolation of cubic or higher may be used. However, in this example, a quadratic curve is used in consideration of the balance between accuracy and hardware implementation. Approximate interpolation is used.

  In this quadratic curve approximate interpolation, as shown in FIG. 16, the basal plane minimum value Smin (see reference numeral 113 in FIG. 16) of the SAD value of the SAD table indicated by the basal plane motion vector 104 with pixel accuracy and the minimum A plurality of SAD values near the position of the value Smin (referred to as a basal plane vicinity SAD value), in this example, four basal plane vicinity SAD values adjacent to the position of the minimum basal plane value Smin in the X and Y directions Sx1, Sx2 and Sy1, Sy2 (see reference numerals 114, 115, 116, 117 in FIG. 16) are used.

  As shown in FIG. 17, a quadratic approximate curve 118 is fitted using a minimum value Smin of the basal plane and two neighboring SAD values Sx1 and Sx2 in the vicinity of the basal plane in the X direction (horizontal direction). The coordinate at which 118 is minimum is the X coordinate Vx of the motion vector (high-precision motion vector) that becomes the base-plane high-precision minimum value SXmin with sub-pixel accuracy. The equation of quadratic curve approximate interpolation at this time is shown in the following equation (1).

SXmin = 1/2 × (Sx2-Sx1) / (Sx2-2Smin + Sx1) (1)
The X coordinate taken on the SAD table by the basal plane high-precision minimum value SXmin with sub-pixel accuracy obtained by the calculation formula (1) is the X-coordinate Vx that is the minimum value of the SAD value with sub-pixel accuracy.

  The division of the calculation formula (1) can be realized by a plurality of subtractions. If the subpixel accuracy to be obtained is, for example, an accuracy of 1/4 of the pixel pitch on the original base surface, it can be obtained by subtraction only twice, so both the circuit scale and the computation time are small. Performance that is almost the same as cubic curve interpolation, which is considerably more complicated than recent curve interpolation, can be realized.

  Similarly, using a basal plane minimum value Smin and two basal plane vicinity SAD values Sy1 and Sy2 in the vicinity of the Y direction (vertical direction), a quadratic approximate curve is fitted, and a minimum value SYmin of the quadratic curve is applied. The Y coordinate taking Y becomes the Y coordinate Vy that is the minimum value of the base surface high accuracy with sub-pixel accuracy. The equation of quadratic curve approximate interpolation at this time is shown in the following equation (2).

SYmin = 1/2 × (Sy2−Sy1) / (Sy2−2Smin + Sy1) (2)
As described above, the approximation of the quadratic curve is performed twice in the X direction and the Y direction, whereby the motion vector (Vx, Vy) of the base pixel high accuracy with subpixel accuracy is obtained.

  In the above description, the minimum SAD value and two SAD values in the vicinity in the X direction (horizontal direction) and the Y direction (vertical direction) are used. However, the SAD value in the vicinity in each direction is two points or more. There may be. Further, instead of the quadratic curve in the X direction and the Y direction, for example, an approximate curve may be fitted in an oblique direction. Furthermore, an approximate curve may be applied by adding an oblique direction to the X direction and the Y direction.

  FIG. 18 shows that a vector detection result with sub-pixel accuracy can be obtained from the value of the SAD table with pixel accuracy by using the above means and procedures. The horizontal axis in FIG. 18 represents the interpolation magnification, and represents how many times the resolution is increased in the one-dimensional direction. Since the SAD table is two-dimensional, the table area is reduced at a ratio of the square, whereas the error due to interpolation increases only to a linear level, so that the usefulness of the above-described interpolation method can be seen.

<Configuration Example of Motion Detection / Motion Compensation Unit 16>
FIG. 19 shows a block diagram of a configuration example of the motion detection / motion compensation unit 16. In this example, the motion detection / compensation unit 16 includes a target block buffer unit 161 that stores pixel data of the target block 102, a reference block buffer unit 162 that stores pixel data of the reference block 108, and the target block 102. The matching processing unit 163 that calculates the SAD value for the corresponding pixel in the block 108, the motion vector calculation unit 164 that calculates the motion vector from the SAD value information output from the matching processing unit 163, and the respective blocks are controlled. A control unit 165.

  The motion detection / compensation unit 16 and the image memory unit 4 are connected through the system bus 2. That is, in this example, the bus interface unit 21 and the bus interface unit 22 are connected between the system bus 2 and the target block buffer unit 161 and the reference block buffer unit 162, respectively. Further, a memory controller 8 is connected between the image memory unit 4 and the system bus 2 for controlling memory access for writing to and reading from the image memory unit 4. Here, the AXI interconnect is used as the protocol in the system bus 2 in this example.

  At the time of still image capturing, the target block buffer unit 161 includes a reduced plane target block or a basal plane target block from the image frame of the reduced plane target image Prt or the basal plane target image Pbt stored in the image memory unit 4. The image data is read from the image memory unit 4 and written according to the read control by the memory controller 8.

  As for the reduced surface target image Prt or the base surface target image Pbt, in the first image, the image of the first imaging frame after the shutter button is pressed is read from the image memory unit 4, and the target block buffer unit 161 is used as the target frame 102. Is written to. When the images are superimposed based on the block matching with the reference image, the NR image after the image is superimposed is written in the image memory unit 4, and the target frame 102 of the target block buffer unit 161 is added to the NR image. It will be rewritten.

  In the reference block buffer unit 162, image data of the reduced surface matching processing range or the base surface matching processing range from the image frame of the reduced surface reference image Prr or the base surface reference image Pbr stored in the image memory unit 4 is stored in the image. It is read from the memory unit 4 and written. In the reduced plane reference image Prr or the base plane reference image Pbr, the imaging frame after the first imaging frame is written in the image memory unit 4 as the reference frame 108.

  In this case, when image superimposition processing is performed while capturing a plurality of continuously captured images (hereinafter referred to as addition during shooting), the base plane reference image and the reduced plane reference image are used. The image frames after the first image frame are sequentially taken into the image memory unit 4 one by one. Therefore, the image memory unit 4 only needs to hold one base surface reference image and one reduced surface reference image.

  However, after capturing a plurality of continuously captured images into the image memory unit 4, the motion detection / motion compensation unit 16 and the image superimposing unit 17 perform motion vector detection and perform image superposition. In the case of doing so (hereinafter referred to as “addition after shooting”), as the base plane reference image and the reduced plane reference image, all of the plurality of imaging frames after the first imaging frame are stored in the image memory. It is necessary to store and hold in the unit 4.

  As an imaging device, both addition during shooting and addition after shooting can be used, but in this embodiment, the still image NR processing requires a clean image with reduced noise even if processing time is somewhat long. In consideration of this, the post-shooting addition process is employed. Detailed operation description of the still image NR processing in this embodiment will be described later.

  On the other hand, at the time of moving image shooting, the imaging frame from the image correction / resolution conversion unit 15 is input to the motion detection / motion compensation unit 16 as the target frame 102. The target block extracted from the target frame from the image correction / resolution conversion unit 15 is written in the target block buffer unit 161. In the reference block buffer unit 162, the imaging frame stored in the image memory unit 4 immediately before the target frame is set as a reference frame 108, and this reference frame (base reference image Pbr or reduced size) is used. The basal plane matching process range or the reduced plane matching process range from the plane reference image Prr) is written.

  At the time of this moving image shooting, the image memory unit 4 stores the previous captured image frame to be subjected to block matching with the target frame from the image correction / resolution conversion unit 15 as a base plane reference image Pbr and a reduced plane reference. Since the image information stored in the image memory unit 4 may be stored in one image (one frame), the base plane reference image Pbr and the reduced plane reference image Prr may be used as the base plane reference image Prr. Image data is not compressed.

  The matching processing unit 163 performs matching processing on the reduced surface and matching processing on the base surface for the target block stored in the target block buffer unit 161 and the reference block stored in the reference block buffer unit 162.

  Here, what is stored in the target block buffer unit 161 is image data of the reduced plane target block, and what is stored in the reference block buffer 162 is image data of the reduced plane matching processing range extracted from the reduced plane reference screen. In the case of, the matching processing unit 163 performs a reduced surface matching process. Also, what is stored in the target block buffer unit 161 is image data of the base plane target block, and what is stored in the reference block buffer 162 is image data of the base plane matching processing range extracted from the base plane reference screen. In some cases, the matching processing unit 163 performs the basal plane matching process.

  In order to detect the strength of the correlation between the target block and the reference block in the block matching by the matching processing unit 163, the SAD value is calculated using the luminance information of the image data in this embodiment as well. The minimum SAD value is detected, and the reference block exhibiting the minimum SAD value is detected as the strongest correlation reference block.

  Needless to say, the SAD value may be calculated by using not only the luminance information but also the color difference signal and the information of the three primary color signals R, G and B. In calculating the SAD value, all the pixels in the block are usually used. However, in order to reduce the amount of calculation, only the pixel value of the pixel with a limited jump position is used by thinning out or the like. It may be.

  The motion vector calculation unit 164 detects the motion vector of the reference block with respect to the target block from the matching processing result of the matching processing unit 163. In this embodiment, the motion vector calculation unit 164 detects and holds the minimum value of the SAD value and also holds the SAD values of a plurality of reference vectors in the vicinity of the reference vector exhibiting the minimum SAD value. A function of detecting a high-precision motion vector with sub-pixel accuracy by performing a quadratic approximate interpolation process is also provided.

  In this embodiment, the motion vector calculation unit 164 sends the detected motion vector for each block to the motion feature detection unit 30 during EE monitoring before still image shooting in the still image shooting mode. The motion feature detection unit 30 includes a motion vector holding buffer unit (not shown) that holds a motion vector for each block in a plurality of frames, in this example, a plurality of frames of three frames or more.

  This motion vector holding buffer unit is configured in, for example, a ring buffer format, and when there is no free space, the motion vector of the oldest frame is discarded to generate a free space, and the motion vector of the new frame is written in the free space. Like that. The motion vector of the oldest frame may be overwritten with the motion vector of the oldest frame without discarding the motion vector of the oldest frame.

  As will be described later, the motion feature detection unit 30 detects the pre-shooting motion feature of the still image to be recorded using the stored motion vectors of a plurality of frames.

  The control unit 165 controls the processing operation of the hierarchical block matching process in the motion detection / compensation unit 16 while being controlled by the CPU 1.

<Configuration Example of Target Block Buffer 161>
A block diagram of a configuration example of the target block buffer 161 is shown in FIG. As shown in FIG. 20, the target block buffer 161 includes a base plane buffer unit 1611, a reduction plane buffer unit 1612, a reduction processing unit 1613, and selectors 1614, 1615 and 1616. Although not shown in FIG. 20, the selectors 1614, 1615, and 1616 are controlled by selection control signals from the control unit 165, respectively.

  The basal plane buffer unit 1611 is for temporarily storing the basal plane target block. The basal plane buffer unit 1611 sends the basal plane target block to the image superimposing unit 17 and supplies it to the selector 1616.

  The reduction plane buffer unit 1612 is for temporarily storing reduction plane target blocks. The reduction plane buffer unit 1612 supplies the reduction plane target block to the selector 1616.

  As described above, since the target block is sent from the image correction / resolution conversion unit 15 at the time of moving image shooting, the reduction processing unit 1613 is provided for generating a reduction plane target block by the reduction processing unit 1613. It has been. The reduced surface target block from the reduction processing unit 1613 is supplied to the selector 1615.

  The selector 1614 selects a target block (basal plane target block) from the image correction / resolution conversion unit 15 when shooting a moving image, and a base block target block or a reduced plane target block from the image memory unit 4 when shooting a still image. Is selected and output in accordance with a selection control signal from, and the output is supplied to a basal plane buffer unit 1611, a reduction processing unit 1613, and a selector 1615.

  The selector 1615 selects and outputs the reduction plane target block from the reduction processing unit 15 during moving image shooting, and the reduction plane target block from the image memory unit 4 during still image shooting using a selection control signal from the control unit 165. The output is supplied to the reduction plane buffer unit 1612.

  In accordance with a selection control signal from the control unit 1615, the selector 1616 selects the reduced surface target block from the reduced surface buffer unit 1612 at the time of block matching on the reduced surface, and the base surface buffer unit 1611 at the time of block matching at the base surface. The base plane target blocks from are respectively selected and output, and the output reduced plane target block or base plane target block is sent to the matching processing unit 163.

<Configuration Example of Reference Block Buffer 162>
A block diagram of a configuration example of the reference block buffer 162 is shown in FIG. As shown in FIG. 21, the reference block buffer 162 includes a basal plane buffer unit 1621, a reduced plane buffer unit 1622, and a selector 1623. Although not shown in FIG. 21, the selector 1623 is selected and controlled by a selection control signal from the control unit 165.

  The basal plane buffer unit 1621 temporarily stores the basal plane reference block from the image memory unit 4, supplies the basal plane reference block to the selector 1623 and sends it to the image overlay unit 17 as a motion compensation block.

  The reduction plane buffer unit 1622 is for temporarily storing the reduction plane reference block from the image memory unit 4. The reduction plane buffer unit 1622 supplies the reduction plane reference block to the selector 1623.

  In accordance with a selection control signal from the control unit 1615, the selector 1623 selects a reduced surface reference block from the reduced surface buffer unit 1612 when performing block matching on the reduced surface, and a base surface buffer unit 1611 when performing block matching on the base surface. The base plane reference blocks from are selected and output, and the output reduced plane reference block or base plane reference block is sent to the matching processing unit 163.

<Configuration Example of Image Superimposing Unit 17>
A block diagram of a configuration example of the image superimposing unit 17 is shown in FIG. As shown in FIG. 22, the image superimposing unit 17 includes an addition rate calculating unit 171, an adding unit 172, a base plane output buffer unit 173, a reduction plane generation unit 174, and a reduction plane output buffer unit 175. It is prepared for.

  The image superposition unit 17 and the image memory unit 4 are connected through the system bus 2. That is, in this example, the bus interface unit 23 and the bus interface unit 24 are connected between the system bus 2 and the base plane output buffer unit 173 and the reduced plane output buffer unit 162, respectively.

  The addition rate calculation unit 171 receives the target block and the motion compensation block from the motion detection / motion compensation unit 16 and adopts the addition rate of both the simple addition method or the average addition method. The determined addition rate is supplied to the adding unit 172 together with the target block and the motion compensation block.

  The basal plane NR image as a result of the addition by the adding unit 172 is written into the image memory unit 4 through the basal plane output buffer unit 173 and the bus interface 23. Further, the base surface NR image as a result of addition by the adding unit 172 is converted into a reduced surface NR image by the reduced surface generation unit 174, and the reduced surface NR image from the reduced surface generation unit 174 is converted into a reduced surface output buffer unit. 175 and the bus interface 24 to be written in the image memory unit 4.

[Overview of noise reduction processing for captured images]
<When shooting still images>
FIG. 23 and FIG. 24 show flowcharts of noise reduction processing by image superposition at the time of still image shooting in the image pickup apparatus having the above-described configuration. Each step of the flowcharts of FIGS. 23 and 24 is executed under the control of the CPU 1 and the control unit 165 of the motion detection / compensation unit 16 controlled by the CPU 1, and is also executed by the image superposition unit 17. Is.

  First, when the shutter button is pressed, in the imaging apparatus of this example, high-speed shooting of a plurality of images is performed at high speed under the control of the CPU 1. In this example, M (M frames; M is an integer of 2 or more) captured image data to be superimposed at the time of still image shooting is captured at high speed and pasted in the image memory unit 4 (step S1).

  Next, the reference frame is Nth in time among the M image frames stored in the image memory unit 4 (N is an integer of 2 or more, and the maximum value is M). The unit 165 sets the initial value of the value N, which is the order, to N = 2 (step S2). Next, the control unit 165 sets the first image frame as a target image (target frame), and sets the N = 2nd image as a reference image (reference frame) (step S3).

  Next, the control unit 165 sets a target block in the target frame (step S4), and the motion detection / motion compensation unit 16 reads the target block from the image memory unit 4 into the target block buffer unit 161 (step S5). Further, the pixel data in the matching processing range is read from the image memory unit 4 to the reference block buffer unit 162 (step S6).

  Next, the control unit 165 reads the reference block within the search range from the reference block buffer unit 162, and the matching processing unit 163 performs the hierarchical matching process in this embodiment. In this example, as will be described later, of the SAD values calculated by the matching processing unit 163, a SAD value described later holds a minimum value of the SAD value and a SAD value in the vicinity thereof, and approximates quadratic curve approximation. In order to perform processing, it is sent to the motion vector calculation unit 164. After the above processing is repeated for all the reference vectors in the search range, the quadratic curve approximation interpolation processing unit performs the above-described interpolation processing and outputs a highly accurate basal plane motion vector (step S7).

  Next, the control unit 165 reads out a motion compensation block that compensates for the motion corresponding to the detected motion vector from the reference block buffer unit 162 according to the high-accuracy basal plane motion vector detected as described above (step). S8) In synchronization with the target block, the image is sent to the subsequent image superimposing unit 17 (step S9).

  Next, under the control of the CPU 1, the image superimposing unit 17 superimposes the target block and the motion compensation block, and pastes the NR image data of the superimposed block on the image memory unit 4. That is, the image superimposing unit 17 writes the NR image data of the superimposed block in the image memory unit 4 (step S10).

  Next, the control unit 165 determines whether or not block matching has been completed for all target blocks in the target frame (step S11), and block matching processing has not been completed for all target blocks. When it is determined, the process returns to step S4, the next target block in the target frame is set, and the processes from step S4 to step S11 are repeated.

  Further, when the control unit 165 determines in step S11 that the block matching for all the target blocks in the target frame has been completed, it is determined whether or not the processing for all the reference frames to be overlapped is completed, that is, It is determined whether or not M = N (step S12).

  If it is determined in step S12 that M = N is not satisfied, N = N + 1 is set (step S13). Next, the NR image generated by the superposition in step S10 is set as a target image (target frame). = N + 1-th image is set as a reference image (reference frame) (step S14). Then, it returns to step S4 and repeats the process after this step S4. In other words, when M is 3 or more, the above processing is repeated with the image that has been superimposed in all target blocks as the next target image and the third and subsequent images as reference frames. This is repeated until the Mth overlap is completed. If it is determined in step S12 that M = N, this processing routine is terminated.

  The image data of the NR image as a result of superimposing the M photographed images is compression-encoded by the still image codec unit 18, supplied to the recording / reproducing device unit 5, and recorded on the recording medium. Is.

  The above-described still image noise reduction processing method is a method of pasting M pieces of image data on the image memory unit 4, but the image may be superimposed each time one image is taken. In that case, since only one image frame is stored in the image memory unit 4, the shooting interval is longer than the noise reduction processing method of the processing routines of FIGS. 23 and 24, but the memory cost is minimized. It is possible to

<When shooting movies>
Next, FIG. 25 shows a flowchart of noise reduction processing by superimposing images during moving image shooting in the imaging apparatus of this embodiment. Each step of the flowchart of FIG. 25 is also executed under the control of the CPU 1 and the control unit 165 of the motion detection / compensation unit 16 controlled by the CPU 1. When the moving image recording button is operated by the user, the CPU 1 instructs the process of FIG. 25 to start from the start.

  In this embodiment, the motion detection / compensation unit 16 has a configuration suitable for performing matching processing in units of target blocks. Therefore, the image correction / resolution conversion unit 15 holds the frame image according to the control of the CPU 1 and sends the image data to the motion detection / motion compensation unit 16 in units of target blocks (step S21).

  The target block image data sent to the motion detection / motion compensation unit 16 is stored in the target block buffer unit 161. Next, the control unit 165 sets a reference vector corresponding to the target block (step S22), and reads image data in the matching processing range from the image memory unit 4 into the reference block buffer unit 162 (step S23).

  Next, the matching processing unit 163 and the motion vector calculation unit 164 perform motion detection processing by hierarchical block matching in this embodiment (step S24). That is, the matching processing unit 163 first calculates the SAD value between the pixel value of the reduction plane target block and the pixel value of the reduction plane reference block on the reduction plane, and sends the calculated SAD value to the motion vector calculation section 164. send. The matching processing unit 163 repeats this process for all reduced surface reference blocks in the search range. After the calculation of the SAD values for all the reduced surface reference blocks in the search range is completed, the motion vector calculation unit 164 identifies the minimum SAD value and detects the reduced surface motion vector.

  The control unit 165 multiplies the reduced plane motion vector detected by the motion vector calculation unit 164 by a reciprocal of the reduction ratio to convert it into a motion vector on the basal plane, and the converted vector indicates a position indicated on the basal plane. The center area is set as a search range on the basal plane. Then, the control unit 165 controls the matching processing unit 163 to perform block matching processing on the basal plane in the search range. The matching processing unit 163 calculates an SAD value between the pixel value of the base surface target block and the pixel value of the base surface reference block, and sends the calculated SAD value to the motion vector calculation unit 164.

  After the calculation of the SAD values for all the reduced surface reference blocks in the search range is finished, the motion vector calculation unit 164 identifies the minimum SAD value, detects the reduced surface motion vector, and also detects the SAD in the vicinity thereof. A value is specified, and the quadratic curve approximation interpolation process described above is performed using those SAD values, and a high-precision motion vector with sub-pixel accuracy is output.

  Next, the control unit 165 reads out the image data of the motion compensation block from the reference block buffer unit 162 in accordance with the high-precision motion vector calculated in step S24 (step S25), and synchronizes with the target block. The image is sent to the image superimposing unit 17 (step S26).

  The image superimposing unit 17 superimposes the target block and the motion compensation block, and writes the image data of the NR image as a result of the superimposition to the image memory unit 4 (step S27). Then, the image superimposing unit 17 records the image data of the NR image as a reference frame for the next target frame, as a moving image recording monitor to be output to the monitor display 6 through the NTSC encoder 20, and through the moving image codec unit 19. The data is stored in the image memory unit 4 for recording to be recorded in the playback device unit 6 (step S28).

  Then, the CPU 1 determines whether or not the moving image recording stop operation has been performed by the user (step S29). When it is determined that the moving image recording stop operation has not been performed by the user, the CPU 1 returns to step S21, and after this step S21. Instruct to repeat the process. If it is determined in step S29 that the moving image recording stop operation has been performed by the user, the CPU 1 ends this processing routine.

  In the above-described processing routine for noise reduction processing of a moving image, an image frame one frame before is used as a reference frame. However, an image of a frame earlier than one frame may be used as a reference frame. In addition, the image before the first frame and the image before the second frame may be stored in the image memory unit 4, and the image frame to be used as the reference frame may be selected from the contents of the two pieces of image information. .

  By using the means, procedure, and system configuration as described above, it is possible to perform still image noise reduction processing and moving image noise reduction processing with one common block matching processing hardware.

[About the motion vector calculation unit 164]
Next, a configuration example and operation of the motion vector calculation unit 164 of the motion detection / compensation unit 16 will be described.

  In an example of a conventionally known motion vector calculation unit, an SAD table TBL that stores all SAD values calculated in a search range is generated, a minimum SAD value is detected from the SAD table TBL, and the minimum SAD value is calculated. A reference vector corresponding to the position of the reference block to be presented is calculated as a motion vector. When quadratic curve approximate interpolation is performed, a plurality of SAD values in the vicinity of the minimum SAD value, in this example, four SAD values are extracted from the generated SAD table TBL, and these SAD values are used. Interpolate. Therefore, this method requires a large-scale memory corresponding to the SAD table TBL.

  In the example described below, by not generating the SAD table TBL that stores all the SAD values calculated in the search range, the circuit scale can be further reduced and the processing time can also be reduced. It is.

<First Example of Motion Vector Calculation Unit 164 of Embodiment>
As described above, the block matching process uses the position indicated by the reference vector as the position of the reference block, calculates the SAD value of each pixel of each reference block and each pixel of the target block, and performs the calculation process within the search range. This is performed for reference blocks at positions indicated by all reference vectors.

  Here, when searching for a motion compensation block by changing the position of the reference block within the search range, the search is performed in order from the end of the screen (frame), and from the center of the screen (frame) to the outside. In this embodiment, the search direction is set as indicated by an arrow 120 in FIG. 26A, and the search is started in the horizontal direction from the upper left end of the search range. After the minute search is completed, a search method is adopted in which a procedure is repeated in which a line one line below in the vertical direction is searched in the horizontal direction from the left end.

  That is, as shown in FIG. 26B, in the search range 106, the reference block 108 is sequentially set and searched in the horizontal direction from the upper left of the search range 106, and the SAD value for each reference block 108 is calculated. I do. Then, as shown in FIG. 26C, the corresponding SAD table is also buried in the horizontal direction from the upper left. At this time, the range of pixel data actually used for the matching process becomes the matching process range 110 corresponding to the size of the reference block 108 as described above.

  As shown in FIGS. 16 and 17, in order to perform the sub-pixel accuracy quadratic curve approximate interpolation processing of this example, the minimum value Smin of the SAD value and the SAD values Sx1, Sx2, Sy1, Sy2 in the vicinity thereof are used. Find what you want.

  When the SAD value for each reference block is calculated, the calculated SAD value is compared with the minimum value of the SAD value up to that point, and the calculated SAD value is greater than the minimum value of the SAD value up to that point. If it is smaller, the calculated SAD value is held as the minimum value, and the SAD value and the reference vector at that time are held, so that the minimum value of the SAD value and the position information of the reference block that takes the minimum value (reference vector Information) can be obtained without generating the SAD table.

  Then, if the minimum value of the detected SAD value is held and the SAD value of the reference block near the reference block position that becomes the minimum SAD value is held as the neighboring SAD value, the neighboring SAD value is also obtained. The SAD table can be held without being generated.

  At this time, in this example, as shown in FIG. 27 (A), the search method as shown in FIG. 26 (A) described above is adopted, so that one line in the horizontal direction in the conventional SAD table TBL is used. If a memory having a capacity to store the SAD value (hereinafter referred to as a line memory) is provided, when a new SAD value of the reference block is calculated, a hatched line is added to the SAD table TBL in FIG. As shown, the SAD values of a plurality of reference blocks for one line of the SAD table TBL calculated before the newly calculated SAD value 121 are stored as the storage data 122 in the line memory. Will be stored.

  Therefore, when the newly calculated SAD value of the reference block is detected as the minimum SAD value, the reference block at a position on one line of the position of the reference block exhibiting the minimum SAD value 121 on the SAD table TBL. SAD value 123 (neighboring SAD value (Sy1)) and the SAD value 124 (neighboring SAD value (Sx1)) of the reference block on the left side of the position of the reference block exhibiting the minimum SAD value 121 are the line memory. Can be obtained from.

  Then, on the SAD table TBL, the neighboring SAD value (Sx2) (see reference numeral 125 in FIG. 27C), which is the SAD value of the reference block at the position to the right of the reference block position of the minimum SAD value, is later The SAD value calculated at the reference block position may be held. Similarly, in the SAD table TBL, the neighborhood SAD value (Sy2) which is the SAD value of the reference block at a position one line below the position of the reference block of the newly calculated minimum SAD value (sign of FIG. 27C) 126), the SAD value calculated at the reference block position may be held later.

  Considering the above, the first example of the motion vector calculation unit 164 has a hardware configuration as shown in FIG.

  In other words, the first example of the motion vector calculation unit 164 does not include the SAD table TBL that holds all the calculated SAD values, but the SAD value writing unit 1641, the SAD value comparison unit 1642, and the SAD value holding. Unit 1643, X direction (horizontal direction) neighborhood value extraction unit 1644, Y direction (vertical direction) neighborhood value extraction unit 1645, quadratic curve approximation interpolation processing unit 1646, and memory for one line of SAD table TBL (line 1647).

  The SAD value holding unit 1643 includes a holding unit (memory) for the minimum SAD value Smin and the neighboring SAD values Sx1, Sx2, Sy1, and Sy2. In this first example, the SAD value holding unit 1643 supplies the minimum SAD value Smin from the minimum SAD value holding unit to the SAD value comparison unit 1642, and among the stored nearby SAD values, The position information (reference vector) of the reference block of the neighboring SAD value Sx2 on the right side of the minimum SAD value Smin and the position information (reference vector) of the reference block of the neighboring SAD value Sy2 on the lower side of the minimum SAD value Smin The data is supplied to the writing unit 1641.

  In this first example, the SAD value comparison unit 1642 receives the position information (reference vector) of the reference block from the matching processing unit 163 and the SAD value Sin of the reference block, and the SAD value holding unit 1643 The minimum SAD value Smin from the minimum SAD value holding unit is received.

  Then, the SAD value comparison unit 1642 compares both the SAD value Sin calculated at that time from the matching processing unit 163 and the minimum SAD value Smin from the minimum SAD value holding unit of the SAD value holding unit 1643, When the SAD value Sin calculated at the time from the matching processing unit 163 is smaller, the SAD value is detected as the minimum SAD value at the time, and when the SAD value Sin is smaller, At the time, it is detected that the minimum SAD value Smin from the minimum SAD value holding unit of the SAD value holding unit 1643 is still the minimum value. The SAD value comparison unit 1642 supplies the detection result information DET to the SAD value writing unit 1641 and the SAD value holding unit 1643.

  The SAD value writing unit 1641 includes a buffer memory for one pixel for temporarily storing the SAD value Sin calculated from the matching processing unit 163 and its position information (reference vector). In this first example, The reference block position information (reference vector) from the matching processing unit 163 and the SAD value Sin of the reference block are written in the line memory 1647. In this case, the line memory 1647 performs the same operation as that of the shift register, and when there is no empty space, when the new position information and SAD value are stored, the oldest position information and SAD value in the line memory 1647 are discarded. Is done.

  In addition, the SAD value writing unit 1641 performs the following processing in the first example before writing the calculated SAD value Sin and its position information into the line memory 1647.

  That is, the SAD value writing unit 1641, when the comparison detection result information DET from the SAD value comparison unit 1642 indicates that the SAD value Sin is the minimum value, the position information of the reference block from the matching processing unit 163. (Reference vector) and the SAD value Sin of the reference block are sent to the SAD value holding unit 1643.

  The SAD value holding unit 1643 detects that the SAD value Sin is the minimum value based on the comparison detection result information DET from the SAD value comparison unit 1642, and the position information of the reference block transmitted from the SAD value writing unit 1641. (Reference vector) and the SAD value Sin of the reference block are stored in the minimum SAD value holding unit.

  In the first example, the SAD value writing unit 1641 uses the position information (reference vector) of the reference block from the matching processing unit 163 as the position of the neighboring SAD value Sx2 or Sy2 received from the SAD value holding unit 1643. Even when the information matches, the reference block position information (reference vector) from the matching processing unit 163 and the SAD value Sin of the reference block are sent to the SAD value holding unit 1643.

  The SAD value holding unit 1643 recognizes which neighboring SAD value is information from the received position information (reference vector) of the reference block, and stores it in the corresponding neighboring SAD value holding unit.

  When the above processing is completed for all the reference blocks in the search range, the SAD value holding unit 1643 holds the minimum SAD value and its position information, and the four neighboring SAD values and its position information, as described above. .

  Therefore, the X direction (horizontal direction) neighborhood value extraction unit 1644 and the Y direction (vertical direction) neighborhood value extraction unit 1645 store the detected minimum SAD value Smin and its neighboring SAD values held in the SAD value holding unit 1643. Sx1, Sx2, Sy1, Sy2 and their position information are read out and sent to the quadratic curve approximation interpolation processing unit 1646. Receiving this, the quadratic curve approximation interpolation processing unit 1646 performs interpolation by the quadratic curve twice in the X direction and the Y direction, and calculates a high-precision motion vector with sub-pixel accuracy as described above.

  As described above, in the first example, by using a line memory for one line of the SAD table TBL instead of the SAD table TBL, it is possible to detect a motion vector with sub-pixel accuracy.

  An example of the flow at the time of block matching processing on the reduced surface in the first example is shown in the flowchart of FIG. 29, and an example of the flow at the time of block matching processing on the basal plane is shown in the flowcharts of FIGS. Each step of the flowchart of the first example is performed in the matching processing unit 163 and the motion vector calculation unit 164 under the control of the control unit 165.

  First, the flowchart of FIG. 29 which is an example of the flow at the time of the block matching process in a reduction surface is demonstrated.

  First, an initial value of the minimum SAD value Smin of the SAD value holding unit 1643 of the motion vector calculation unit 164 is set (step S31). As the initial value of the minimum SAD value Smin, for example, the maximum value of the pixel difference is set.

  Next, the matching processing unit 163 sets a reduced plane reference vector (Vx, Vy), sets a reduced plane reference block position for calculating the SAD value (step S32), and sets the pixels of the set reduced plane reference block. The data is read from the reference block buffer unit 162 (step S33), and the pixel data of the reduction plane target block is read from the target block buffer unit 161, and the absolute difference between each pixel data of the reduction plane target block and the reduction plane reference block is read. The sum of the values, that is, the SAD value is obtained, and the obtained SAD value is sent to the motion vector calculation unit 164 (step S34).

  In the motion vector calculation unit 164, the SAD value writing unit 1641 writes the SAD value into the line memory 1647 (step S35).

  In the motion vector calculation unit 164, the SAD value comparison unit 1642 compares the SAD value Sin calculated by the matching processing unit 163 with the minimum SAD value Smin held in the SAD value holding unit 1643 to calculate It is determined whether or not the SAD value Sin is smaller than the minimum SAD value Smin held so far (step S36).

  If it is determined in step S36 that the calculated SAD value Sin is smaller than the minimum SAD value Smin, the process proceeds to step S37, and information on the minimum SAD value Smin held in the SAD value holding unit 1643 and its position information (reduction plane) The reference vector is updated.

  When it is determined in step S36 that the calculated SAD value Sin is larger than the minimum SAD value Smin, the process proceeds to step S38 without performing the update processing of the retained information in step S37, and the matching processing unit 163 determines whether the search range is within the search range. It is determined whether or not the matching process has been completed at the positions of all reduced surface reference blocks (reduced surface reference vectors). If it is determined that there is still an unprocessed reference block in the search range, the process returns to step S32. The processes after step S32 described above are repeated.

  If the matching processing unit 163 determines in step S38 that the matching processing has been completed at the positions of all the reduced surface reference blocks (reduced surface reference vectors) in the search range, the matching processing unit 163 notifies the motion vector calculating unit 164 to that effect. .

  In response to this, the motion vector calculation unit 164 outputs the position information (reduced plane reference vector) of the minimum SAD value Smin held in the SAD value holding unit 1643 to the control unit 165 (step S39).

  This is the end of the block matching process on the reduced surface in this example.

  Next, the flowcharts of FIGS. 30 and 31 that are examples of the flow at the time of block matching processing on the base surface will be described.

  First, an initial value of the minimum SAD value Smin of the SAD value holding unit 1643 of the motion vector calculation unit 164 is set (step S41). As the initial value of the minimum SAD value Smin, for example, the maximum value of the pixel difference is set.

  Next, the matching processing unit 163 sets the reference vector (Vx, Vy) on the base plane, sets the base plane reference block position for calculating the SAD value (step S42), and sets the pixels of the set base plane reference block Data is read from the reference block buffer unit 162 (step S43).

  Then, the pixel data of the base plane target block is read from the target block buffer unit 161, and the sum of absolute values of differences between the pixel data of the base plane target block and the base plane reference block, that is, the SAD value is obtained. The value is sent to the motion vector calculation unit 164 (step S44).

  In the motion vector calculation unit 164, the SAD value writing unit 1641 writes the SAD value into the line memory 1647 (step S45). In the motion vector calculation unit 164, the SAD value comparison unit 1642 compares the SAD value Sin calculated by the matching processing unit 163 with the minimum SAD value Smin held in the SAD value holding unit 1643 to calculate It is determined whether or not the SAD value Sin is smaller than the minimum SAD value Smin held so far (step S46).

  When it is determined in step S46 that the calculated SAD value Sin is smaller than the minimum SAD value Smin, the process proceeds to step S47, and information on the minimum SAD value Smin held in the SAD value holding unit 1643 and the minimum SAD value Smin are obtained. The SAD value and the position information (base plane reference vector) of the reference block at a position one pixel above and one pixel to the left of the reference block position to be presented are updated.

  That is, the SAD value comparison unit 1642 sends the comparison result information DET to the SAD value writing unit 1641 that the calculated SAD value Sin is smaller than the minimum SAD value Smin. Then, the SAD value writing unit 1641 sends the calculated SAD value Sin and its position information (base plane reference vector) to the SAD value holding unit 1643 as information on a new minimum SAD value Smin, and FIG. ), The oldest SAD value and its position information (base plane reference vector) of the line memory 1647 and the newest SAD value and its position information (base plane reference vector) are set to 1 of the position of the minimum SAD value. The information is sent to the SAD value holding unit 1643 as information on the SAD value Sy1 of the base plane reference block at the position on the pixel and information on the SAD value Sx1 of the base plane reference block at the position one pixel to the left. The SAD value holding unit 1643 receives the received information on the new minimum SAD value Smin, the information on the SAD value Sy1 of the base plane reference block at the position one pixel above, and the SAD value Sx1 of the base plane reference block at the position one pixel left. The corresponding holding information is updated with the information.

  Then, after step 47, the process proceeds to step S51 in FIG. If it is determined in step S46 that the calculated SAD value Sin is greater than the minimum SAD value Smin, the process advances to step S51 without performing the holding information update process in step S47.

  In step S51, the SAD value writing unit 1641 indicates that the position indicated by the position information (reference vector) for the calculated SAD value Sin is one pixel below the position of the base surface reference block held as the minimum SAD value Smin. It is determined whether or not the position is the position of the base plane reference block, and when it is determined that the position is the position of the base plane reference block one pixel below, the calculated SAD value Sin and its position information (reference vector) are stored in the SAD value holding unit. Send to 1643. The SAD value holding unit 1643 updates the holding information of the neighboring SAD value Sy2 for the basal plane reference block at the position one pixel below by using the received SAD value and its position information (step S52).

  In step S51, the position pointed to by the position information (base plane reference vector) for the calculated SAD value Sin is that of the base plane reference block one pixel below the position of the base plane reference block held as the minimum SAD value Smin. When it is determined that the position is not a position, the SAD value writing unit 1641 indicates the position of the base plane reference block in which the position indicated by the position information (base plane reference vector) regarding the calculated SAD value Sin is held as the minimum SAD value Smin. It is determined whether or not it is the position of the base plane reference block 1 pixel below (step S53).

  In this step S53, the position pointed to by the position information (base plane reference vector) for the calculated SAD value Sin is the base plane reference block one pixel to the right of the position of the base plane reference block held as the minimum SAD value Smin. When it is determined that the position is, the SAD value writing unit 1641 sends the calculated SAD value Sin and its position information (base plane reference vector) to the SAD value holding unit 1643. The SAD value holding unit 1643 updates the holding information of the neighboring SAD value Sx2 with respect to the basal plane reference block at the position one pixel to the right based on the received SAD value and its position information (step S54).

  In step S53, the position pointed to by the position information (base plane reference vector) for the calculated SAD value Sin is the base plane reference block one pixel to the right of the base plane reference block position held as the minimum SAD value Smin. When it is determined that the position is not the position, the SAD value writing unit 1641 writes the calculated SAD value Sin and its position information (base plane reference vector) only in the line memory 1647 (step S45 described above), and the SAD value holding unit 1643. Do not send to.

  Then, the matching processing unit 163 determines whether or not the matching process has been completed at the positions (reference vectors) of all the base plane reference blocks in the search range (step S55). When it is determined that there is a surface reference block, the process returns to step S42 in FIG. 34, and the processes after step S42 described above are repeated.

  If the matching processing unit 163 determines in step S45 that the matching processing has been completed at the positions of all the base plane reference blocks in the search range (base plane reference vectors), the matching processing unit 163 notifies the motion vector calculation unit 164 accordingly. .

  In response to this, the motion vector calculation unit 164 detects that the X direction (horizontal direction) neighborhood value extraction unit 1644 and the Y direction (vertical direction) neighborhood value extraction unit 1645 are held in the SAD value holding unit 1643. The minimum SAD value Smin and its neighboring SAD values Sx1, Sx2, Sy1, Sy2 and their position information are read out and sent to the quadratic curve approximation interpolation processing unit 1646. Receiving this, the quadratic curve approximation interpolation processing unit 1646 performs interpolation by the quadratic curve twice in the X direction and the Y direction, and calculates a high-precision motion vector with subpixel accuracy as described above ( Step S56). The block matching process of the second example for one base plane reference frame is thus completed.

  As described above, in the first example, one line of the SAD table is provided without holding the SAD table holding all the calculated SAD values, and one pixel is provided in the SAD value writing unit 1641. It is possible to perform sub-pixel precision motion vector detection by interpolation processing only by holding the SAD value.

  The method of the first example has the effect that the processing time does not change and the hardware scale can be reduced because the number of times of block matching is the same as the method of holding all of the conventional SAD table TBL.

  In the description of the first example described above, the SAD value comparison unit 1642 uses the calculated SAD value Sin from the matching processing unit 163 and the minimum SAD value Smin held in the SAD value holding unit 163. Although the comparison is made, the SAD value comparison unit 1642 includes a holding unit for the minimum SAD value, and the held minimum SAD value is compared with the calculated SAD value Sin. When the calculated Sin is smaller The held minimum SAD value is updated, and the position information is sent to the SAD value holding unit 1643 through the SAD value writing unit 1641 so as to be held in the minimum SAD value holding unit of the SAD value holding unit 1643. .

<Second Example of Motion Vector Calculation Unit 164 of Embodiment>
The second example of the motion vector calculation unit 164 is an example in which the line memory 1647 in the case of the first example is also omitted to further reduce the hardware scale.

  In the second example, the minimum value Smin of the SAD value of the reference block in the search range and its position information (reference vector) are detected and held in the same manner as in the first example described above. However, the acquisition and holding of the neighboring SAD value and its position information are not performed simultaneously with the detection of the SAD value Smin, as in the first example, and the position information of the detected minimum SAD value Smin is matched. Returning to the unit 163, the matching processing unit 163 again calculates the SAD values for the reference blocks at the four point positions in the vicinity of the minimum SAD value Smin, and supplies them to the motion vector calculation unit 164.

  The motion vector calculation unit 164 receives the SAD values of the four neighboring positions calculated by the block matching process again from the matching processing unit 163 and its position information (reference vector), and each of the SAD value holding units 1643 Store in the holding part.

  A hardware configuration example of the second example of the motion vector calculation unit 164 is shown in FIG. That is, in the third example of the motion vector calculation unit 164, the line memory 1647 of the first example is not provided, but as shown in FIG. 32, an SAD value writing unit 1641, an SAD value comparison unit 1642, An SAD value holding unit 1643, an X direction (horizontal direction) neighborhood value extraction unit 1644, a Y direction (vertical direction) neighborhood value extraction unit 1645, and a quadratic curve approximation interpolation processing unit 1646 are provided.

  Also in the second example, the X direction (horizontal direction) neighborhood value extraction unit 1644, the Y direction (vertical direction) neighborhood value extraction unit 1645, and the quadratic curve approximation interpolation processing unit 1646 are the same as those in the first example described above. The SAD value writing unit 1641, the SAD value comparison unit 1642, the SAD value holding unit 1643, and the line memory 1647 perform different operations from those in the first example.

  Similar to the first example, the SAD value holding unit 1643 includes a holding unit (memory) for the minimum SAD value Smin and the neighboring SAD values Sx1, Sx2, Sy1, and Sy2. Also in this second example, the SAD value holding unit 1643 supplies the minimum SAD value Smin from the minimum SAD value holding unit to the SAD value comparison unit 1642. However, in the second example, unlike the first example, the SAD value holding unit 1643 does not supply the position information of the neighboring SAD value to the SAD value writing unit 1641.

  In this second example, the SAD value comparison unit 1642 also calculates the SAD value Sin calculated at that time from the matching processing unit 163 and the minimum SAD value Smin from the minimum SAD value storage unit of the SAD value storage unit 1643. When the SAD value Sin calculated at the time from the matching processing unit 163 is smaller, the SAD value is detected as the minimum SAD value at the time, and the SAD value Sin When the value is smaller, it is detected that the minimum SAD value Smin from the minimum SAD value holding unit of the SAD value holding unit 1643 is still the minimum value at that time. The SAD value comparison unit 1642 supplies the detection result information DET to the SAD value writing unit 1641 and the SAD value holding unit 1643.

  Similar to the above example, the SAD value writing unit 1641 includes a buffer memory for one pixel for temporarily storing the SAD value Sin calculated from the matching processing unit 163 and its position information (reference vector). In this second example, when the SAD value writing unit 1641 indicates that the comparison detection result information DET from the SAD value comparison unit 1642 indicates that the SAD value Sin is the minimum value, the matching processing unit The reference block position information (reference vector) from 163 and the SAD value Sin of the reference block are sent to the SAD value holding unit 1643.

  The SAD value holding unit 1643 knows that the SAD value Sin is the minimum value based on the comparison detection result information DET from the SAD value comparison unit 1642, and the position information of the reference block sent from the SAD value writing unit 1641 ( Reference vector) and the SAD value Sin of the reference block are stored in the minimum SAD value holding unit.

  The above processing is performed on the SAD values calculated by the matching processing unit 163 for all reference blocks in the search range. In the second example, when the calculation of the SAD values for all the reference blocks within the search range is completed, the SAD value holding unit 1643 stores the position information (reference (reference) of the held minimum SAD value Smin. Vector) Vmin is supplied to matching processing section 163 to request recalculation of SAD values for four reference blocks in the vicinity of the position information.

  When the matching processing unit 163 receives from the SAD value holding unit 1643 a request for recalculation of the SAD value for the neighborhood reference block including the position information (reference vector) Vmin of the minimum SAD value Smin, the position information of the minimum SAD value Smin From the reference vector Vmin, the positions of the four neighboring reference blocks are detected, and the SAD value is calculated for the reference block at the detected position. Then, the calculated SAD value is sequentially supplied to the SAD value writing unit 1641 together with the position information (reference vector).

  In this case, since the matching processing unit 163 performs block matching processing in the order of the search direction, the neighboring SAD values are calculated in the order of SAD values Sy1, Sx1, Sx2, and Sy2. The SAD value writing unit 1641 sequentially supplies the received recalculated SAD value and its position information (reference vector) to the SAD value holding unit 1643.

  The SAD value holding unit 1643 sequentially writes and holds the recalculated SAD value and its position information (reference vector) in the corresponding storage unit.

  Thus, when the recalculation of the SAD value for the neighborhood reference block is completed, the X direction (horizontal direction) neighborhood value extraction unit 1644 and the Y direction (vertical direction) neighborhood value extraction unit 1645 are the same as in the first example described above. Reads out the detected minimum SAD value Smin, its neighboring SAD values Sx1, Sx2, Sy1, Sy2 and their position information held in the SAD value holding unit 1643, and sends them to the quadratic curve approximation interpolation processing unit 1646. . Receiving this, the quadratic curve approximation interpolation processing unit 1646 performs interpolation by the quadratic curve twice in the X direction and the Y direction, and calculates a high-precision motion vector with sub-pixel accuracy as described above.

  As described above, in the second example, the motion vector can be detected with sub-pixel accuracy without using the SAD table TBL or the line memory.

  The example of the flow at the time of block matching processing on the reduction surface in the second example is the same as the case of the first example shown in FIG. 29 except that writing to the line memory 1647 is not performed. Here, the description is omitted. An example of the flow at the time of block matching processing on the base surface in the second example will be described with reference to the flowchart of FIG.

  First, an initial value of the minimum SAD value Smin of the SAD value holding unit 1643 of the motion vector calculation unit 164 is set (step S61). As the initial value of the minimum SAD value Smin, for example, the maximum value of the pixel difference is set.

  Next, the matching processing unit 163 sets a reference vector (Vx, Vy) at the base plane, sets a base plane reference block position for calculating the SAD value (step S62), and sets the base plane reference block of the set base plane reference block. The pixel data is read from the reference block buffer unit 162 (step S63), the pixel data of the base plane target block is read from the target block buffer unit 161, and the difference between the pixel data of the base plane target block and the base plane reference block is calculated. The sum of absolute values, that is, the SAD value is obtained, and the obtained SAD value is sent to the motion vector calculating unit 164 (step S64).

  In the motion vector calculation unit 164, the SAD value comparison unit 1642 is calculated by comparing the SAD value Sin calculated by the matching processing unit 163 with the minimum SAD value Smin held in the SAD value holding unit 1643. It is determined whether or not the SAD value Sin is smaller than the minimum SAD value Smin held so far (step S65).

  When it is determined in step S65 that the calculated SAD value Sin is smaller than the minimum SAD value Smin, the process proceeds to step S66, and the minimum SAD value Smin held in the SAD value holding unit 1643 and its position information are updated. .

  That is, the SAD value comparison unit 1642 sends the comparison result information DET to the SAD value writing unit 1641 that the calculated SAD value Sin is smaller than the minimum SAD value Smin. Then, the SAD value writing unit 1641 sends the calculated SAD value Sin and its position information (base plane reference vector) to the SAD value holding unit 1643 as new minimum SAD value Smin information. The SAD value holding unit 1643 updates the minimum SAD value Smin to be held and its position information to the received new SAD value Sin and its position information.

  After step S66, the process proceeds to step S67. If it is determined in step S65 that the calculated SAD value Sin is greater than the minimum SAD value Smin, the process advances to step S67 without performing the holding information update process in step S66.

  In step S67, the matching processing unit 163 determines whether or not the matching process has been completed at the positions of all the base plane reference blocks (base plane reference vectors) in the search range. When it is determined that there is a surface reference block, the process returns to step S62, and the processes after step S62 are repeated.

  If the matching processing unit 163 determines in step S67 that the matching process has been completed at the positions (reference vectors) of all the base plane reference blocks in the search range, the minimum SAD value Smin from the SAD value holding unit 1643 is determined. The position information is received, the SAD value is recalculated for the basal plane reference block at the positions of the four neighboring points, and the recalculated neighboring SAD value is supplied to the SAD value holding unit 1643 through the SAD value writing unit 1641 and held. (Step S68).

  Next, in the motion vector calculation unit 164, the X direction (horizontal direction) neighborhood value extraction unit 1644 and the Y direction (vertical direction) neighborhood value extraction unit 1645 are stored in the SAD value holding unit 1643 and the detected minimum The SAD value Smin and its neighboring SAD values Sx1, Sx2, Sy1, Sy2 and their position information are read out and sent to the quadratic curve approximation interpolation processing unit 1646. Receiving this, the quadratic curve approximation interpolation processing unit 1646 performs interpolation by the quadratic curve twice in the X direction and the Y direction, and calculates a high-precision motion vector with subpixel accuracy as described above ( Step S69). This completes the block matching process on the base surface of the second example for one reference frame.

  In this second example, the processing time is increased by recalculating the SAD value as compared with the first example described above, but since no line memory is required, the circuit scale is reduced to the first. It can be reduced more than the example. In addition, since the SAD value is recalculated only for the neighborhood SAD value, in the above example, since the number of times is at most four, the increase in the processing time is small.

  In the description of the second example described above, the minimum SAD value is held in the SAD value holding unit 1643 while being detected. However, in the SAD value comparison unit 1642, the position information of the reference block that exhibits the minimum SAD value. (Reference vector) is detected and held, and when the first block matching is completed, the position information of the minimum SAD value is supplied from the SAD value comparison unit 1642 to the matching processing unit 163. Good.

  In this case, in the recalculation of the SAD value in the matching processing unit 163, the minimum SAD value is recalculated in addition to the SAD values of the four neighboring points. In this case, the number of recalculations of the SAD value is 5 and increases by 1. However, in the first block matching, only the SAD value comparison unit 1642 needs to operate, and the SAD value writing unit 1641 and the SAD value holding Since the unit 1643 only needs to operate so as to hold the recalculated SAD value, there is an advantage that the processing operation is simplified.

  Further, the processing in the motion detection and motion compensation unit 16 can be executed in parallel and in parallel for a plurality of target blocks set in the target frame. In that case, it is necessary to provide as many hardware systems as the motion detection and motion compensation unit 16 described above in accordance with the number of target blocks to be processed in parallel and in parallel.

  In the case of the method for generating the SAD table TBL as in the conventional example, it is necessary to generate as many SAD tables as the number of target blocks, and a very large memory is required. However, in the first example, one line of the SAD table is sufficient for one target block, and the memory capacity can be greatly reduced. Furthermore, in the case of the second example, since no line memory is required, the memory capacity can be greatly reduced.

[Example of hierarchical block matching process flow]
Next, FIG. 34 and FIG. 35 show flowcharts of an operation example of the hierarchical block matching process in the motion detection / compensation unit 16 in this embodiment.

  Note that the processing flow shown in FIGS. 34 and 35 partially overlaps with the description of the processing example of the matching processing unit 163 and the motion calculation unit 164 described above. The operation will be described for easier understanding.

  First, the motion detection / compensation unit 16 reads a reduced image of the target block, that is, a reduced surface target block from the target block buffer unit 161 (step S71 in FIG. 34). Next, the initial value of the reduced surface minimum SAD value is set as the initial value of the minimum SAD value Smin of the SAD value holding unit 1643 of the motion vector calculation unit 164 (step S72). As an initial value of the reduced surface minimum SAD value Smin, for example, a maximum value of pixel difference is set.

  Next, the matching processing unit 163 sets a reduction plane search range, sets a reduction plane reference vector (Vx / n, Vy / n: 1 / n is a reduction ratio) in the set reduction search range, The reduced surface reference block position for calculating the SAD value is set (step S73). Then, the pixel data of the set reduction plane reference block is read from the reference block buffer unit 162 (step S74), and the sum of absolute values of differences between the pixel data of the reduction plane target block and the reduction plane reference block, that is, the reduction plane The SAD value is obtained, and the obtained reduced surface SAD value is sent to the motion vector calculation unit 164 (step S75).

  In the motion vector calculation unit 164, the SAD value comparison unit 1642 compares the reduced surface SAD value Sin calculated by the matching processing unit 163 with the reduced surface minimum SAD value Smin held in the SAD value holding unit 1643. Then, it is determined whether or not the calculated reduced surface SAD value Sin is smaller than the reduced surface minimum SAD value Smin held so far (step S76).

  If it is determined in step S76 that the calculated reduction surface SAD value Sin is smaller than the reduction surface minimum SAD value Smin, the process proceeds to step S77, and the reduction surface minimum SAD value Smin held in the SAD value holding unit 1643 and its position. Information is updated.

  That is, the SAD value comparison unit 1642 sends the comparison result information DET to the SAD value writing unit 1641 that the calculated reduction surface SAD value Sin is smaller than the reduction surface minimum SAD value Smin. Then, the SAD value writing unit 1641 sends the calculated reduced surface SAD value Sin and its position information (reduced surface reference vector) to the SAD value holding unit 1643 as information on a new reduced surface minimum SAD value Smin. The SAD value holding unit 1643 updates the reduced reduction surface minimum SAD value Smin and its position information to be held to the received new reduction surface SAD value Sin and its position information.

  After step S77, the process proceeds to step S78. If it is determined in step S76 that the calculated reduced surface SAD value Sin is larger than the reduced surface minimum SAD value Smin, the process advances to step S78 without performing the holding information update process in step S77.

  In step S78, the matching processing unit 163 determines whether or not the matching process has been completed at the positions of all reduced surface reference blocks (reduced surface reference vectors) in the reduced surface search range. When it is determined that there is an unprocessed reduced surface reference block, the process returns to step S73, and the processes after step S73 are repeated.

  If the matching processing unit 163 determines in step S78 that the matching process has been completed at the positions of all reduced surface reference blocks (reduced surface reference vectors) in the reduced surface search range, the reduction processing from the SAD value holding unit 1643 is performed. The position information (reduced surface motion vector) of the surface minimum SAD value Smin is received, and the received reduced surface motion vector is the reciprocal of the reduction magnification, that is, the vector obtained by multiplying by n is centered on the position coordinates indicated in the base plane target frame. The base plane target block is set at the position and the base plane search range is set as the base plane target frame as a relatively narrow range centered on the position coordinate indicated by the n-fold vector (step S79). From the target block buffer 161, the pixel data of the base surface target block is displayed. Read the data (step S80).

  Next, the initial value of the base surface minimum SAD value is set as the initial value of the minimum SAD value Smin of the SAD value holding unit 1643 of the motion vector calculation unit 164 (step S81 in FIG. 35). As the initial value of the basal plane minimum SAD value Smin, for example, the maximum value of the pixel difference is set.

  Next, the matching processing unit 163 sets the basal plane reference vector (Vx, Vy) in the basal plane reduction search range set in step S79, and sets the basal plane reference block position for calculating the SAD value (step). S82). Then, the set pixel data of the base plane reference block is read from the reference block buffer unit 162 (step S83), and the sum of absolute values of differences between the pixel data of the base plane target block and the base plane reference block, that is, the base plane The SAD value is obtained, and the obtained basal plane SAD value is sent to the motion vector calculation unit 164 (step S84).

  In the motion vector calculation unit 164, the SAD value comparison unit 1642 compares the basal plane SAD value Sin calculated by the matching processing unit 163 with the basal plane minimum SAD value Smin held in the SAD value holding unit 1643. Then, it is determined whether or not the calculated basal plane SAD value Sin is smaller than the basal plane minimum SAD value Smin held so far (step S85).

  When it is determined in this step S85 that the calculated basal plane SAD value Sin is smaller than the basal plane minimum SAD value Smin, the process proceeds to step S86, and the basal plane minimum SAD value Smin held in the SAD value holding unit 1643 and its position. Information is updated.

  That is, the SAD value comparison unit 1642 sends the comparison result information DET to the SAD value writing unit 1641 that the calculated basal plane SAD value Sin is smaller than the basal plane minimum SAD value Smin. Then, the SAD value writing unit 1641 sends the calculated base surface SAD value Sin and its position information (reference vector) to the SAD value holding unit 1643 as information of a new base surface minimum SAD value Smin. The SAD value holding unit 1643 updates the base surface minimum SAD value Smin to be held and its position information to the received new base surface SAD value Sin and its position information.

  After step S86, the process proceeds to step S87. If it is determined in step S85 that the calculated basal plane SAD value Sin is greater than the basal plane minimum SAD value Smin, the process proceeds to step S87 without performing the holding information update process in step S86.

  In step S87, the matching processing unit 163 determines whether or not the matching process has been completed at the positions of all base plane reference blocks (base plane reference vectors) in the base plane search range. When it is determined that there is an unprocessed base plane reference block, the process returns to step S82, and the processes after step S82 are repeated.

  When the matching processing unit 163 determines in step S87 that the matching processing has been completed at all base plane reference block positions (base plane reference vectors) in the base plane search range, the basis from the SAD value holding unit 1643 is determined. The position information (base plane motion vector) of the surface minimum SAD value Smin is received, the base plane SAD value is recalculated for the base plane reference block at the positions of the four neighboring points, and the recalculated vicinity base plane SAD value is SAD. The value is supplied to the SAD value holding unit 1643 through the value writing unit 1641 and held (step S88).

  Next, in the motion vector calculation unit 164, the X direction (horizontal direction) neighborhood value extraction unit 1644 and the Y direction (vertical direction) neighborhood value extraction unit 1645 are stored in the SAD value holding unit 1643 and the detected minimum The basal plane SAD value Smin and its neighboring basal plane SAD values Sx1, Sx2, Sy1, Sy2 and their position information are read out and sent to the quadratic curve approximation interpolation processing unit 1646. Receiving this, the quadratic curve approximation interpolation processing unit 1646 performs interpolation by the quadratic curve twice in the X direction and the Y direction, and calculates a high-precision basal plane motion vector with subpixel accuracy as described above. (Step S89). This completes the block matching process of this example for one reference frame.

  Next, the effect of the image processing method using the hierarchical block matching method according to this embodiment will be described with a specific example.

  FIG. 36 shows specific examples of the reference block, the search range, and the matching processing range on the base surface and the reduction surface used in the description of the above-described embodiment. In the example of FIG. 36, the reduction ratio 1 / n in the horizontal direction and the vertical direction is set to n = 4.

  As a comparative example, as shown in FIGS. 36A and 36B, for example, a reference block 108 that does not use a reduced image has horizontal (horizontal) × vertical (vertical) = 32 × 32 pixels and a search range. 106 is horizontal (horizontal) × vertical (vertical) = 144 × 64 pixels, and the matching processing range 110 is horizontal (horizontal) × vertical (vertical) = 176 × 96 pixels.

  Then, in the above-described embodiment, in the reduction plane reduced to 1 / n = 1/4 in the horizontal direction and the vertical direction, as shown in FIGS. 36 (C) and (D), the reduction plane reference block 139 is used. Is horizontal (horizontal) × vertical (vertical) = 8 × 8 pixels, the reduction plane search range 137 is horizontal (horizontal) × vertical (vertical) = 36 × 16 pixels, and the reduction plane matching processing range 143 is horizontal (horizontal). ) × vertical (vertical) = 44 × 24 pixels.

  When block matching is performed with a reduction plane reduced to ¼ in the horizontal direction and the vertical direction, the reduction plane motion vector is a motion vector with a precision of 4 pixels. An error occurs with respect to the precision motion vector. In other words, when the pixels on the base surface are as shown in FIG. 41, the matching processing point 148 on the base surface covers all pixels on the base surface, but is reduced to ¼. This is because the matching processing points when performing block matching on the surface are the matching processing points 147 in units of 4 pixels indicated by black circles in FIG.

  However, it can be predicted that at least a 1-pixel precision motion vector will exist within a 4-pixel range around the matching processing point on the reduced plane indicated by the reduced plane motion vector.

  Therefore, in this embodiment, in basal plane matching that determines the basal plane search range based on the calculated reduced plane motion vector, the pixel position indicated by the reference vector obtained by multiplying the reduced plane motion vector by 4 that is the reciprocal of the reduction magnification is indicated. A base plane target block is set at the center, a search range for 4 pixels (base plane search range 140) is determined, base plane block matching is performed, and a motion vector is calculated again.

  Therefore, as shown in FIGS. 36E and 36F, the basal plane reference block 142 is horizontal (horizontal) × vertical (vertical) = 32 × 32 pixels, and the basal plane search range 140 is horizontal (horizontal) × The vertical (vertical) = 4 × 4 pixels and the basal plane matching processing range 144 is horizontal (horizontal) × vertical (vertical) = 40 × 40 pixels.

  In FIG. 38, in the case of performing two-layer matching of reduced plane matching and basal plane matching as in this embodiment, it is assumed that the SAD table is used, and the SAD table of the basal plane and reduced plane is used. Indicates size. The example of FIG. 38 corresponds to the specific example shown in FIG.

  As shown in FIG. 38A, the SAD table TBL in the case of a 144 × 64 search range (see FIG. 36B) before reduction is 145 × 65 points.

  On the other hand, the reduced surface SAD table in the case of a 36 × 16 reduced surface search range (see FIG. 36D) has 37 points × 17 points as shown in FIG.

  In the case of the 4 × 4 base plane search range (see FIG. 36F), the base plane SAD table is 5 points × 5 points.

  Therefore, the number of matching processes when layer matching is not performed is 145 × 65 = 9425 times, whereas the number of matching processes when layer matching is applied is 37 × 17 + 5 × 5 = 654 times, which greatly increases the processing time. It can be seen that it can be shortened.

  The line memory in the case of the first example of the motion vector detection method described above may be one line in the reduced-surface SAD table, so that it can store 37 SAD values and their position information. In the case of TBL, a very small memory is sufficient for the memory storing 9425 SAD values and their position information.

  Further, in the case of the second example of the configuration example of the motion vector calculation unit 164 described above, even the reduced surface SAD table for storing the 37 SAD values and the position information thereof becomes unnecessary, so that the circuit scale can be further reduced. can do.

  As described above, according to the above-described embodiment, after performing the hierarchical matching process, the interpolation process is performed on the basal plane, thereby performing motion vector detection with sub-pixel accuracy in a wide search range. Is possible.

  FIG. 39 shows an example of the target block size and the matching processing range on the reduction plane and the base plane in the hierarchical block matching method of this embodiment. In the example of FIG. 39, the size of the reduction plane target block 133 is a fixed value of horizontal × vertical = 8 × 8 pixels as shown in FIG. For this reason, the reduction plane matching processing range 143 is horizontal × vertical = 44 × 24 pixels as shown in FIG.

  Then, the size of the base plane target block 131, the size of the base plane matching processing range 144, and the number of base plane reference vectors are as shown in FIG. FIG. 39C shows the size of the basal plane target block 131 and the basal plane matching processing range 144 in each of the examples when the reduction ratio of the reduction plane with respect to the basal plane is 1/2, 1/4, and 1/8. The size and the number of base plane reference vectors are shown.

[Reduced memory capacity and efficient memory access during still image NR processing]
As described above, this embodiment supports still image NR processing and moving image NR processing. In moving image NR processing, real-time performance, that is, speed is required rather than accuracy, whereas still image NR processing requires a clean image with no noise even if processing time is somewhat long.

  As described above, when the image overlay method is largely classified from the viewpoint of the system architect, the during-shoot addition method, which is a method for performing image overlay in real time during high-speed continuous shooting during still image shooting, After high-speed continuous shooting, there are two types, a post-shooting addition method, which is a method of superimposing images after all reference images are gathered. In order to perform the NR process with sufficient processing time, the post-shooting addition method is preferable rather than the during-shooting addition method. In this embodiment, the post-shooting addition method is used.

  However, the post-shooting addition method has to store the reference image in the image memory, and there is a problem that a larger capacity is required as the image memory as the number of superimposed images increases.

  In this embodiment, in view of this problem, in using the post-shooting addition method at the time of still image NR processing, the capacity of the image memory unit 4 is reduced as much as possible and the memory access can be performed efficiently. is doing. Hereinafter, this point will be described.

<Data format for memory access during still image NR processing>
First, a data format at the time of memory access to the image memory unit 4 in this embodiment will be described.

  As shown in the block diagrams of FIGS. 19 and 22, the motion detection / compensation unit 16 and the image superposition unit 17 are connected to the system bus 2 via bus interface units 21, 22, 23, and 24, respectively. Has been.

  The system bus 2 in this embodiment has a bus width of, for example, 64 bits, and the burst length by the memory controller 8 (the number of times burst transfer can be continuously performed in units of predetermined plural pixel data) is, for example, 16 bursts at maximum ing. In other words, the maximum number q of continuous burst transfers that can be performed is q = 16 in this example.

  Read requests from the target block buffer unit 161 and the reference block buffer unit 162 of the motion detection / compensation unit 16 are supplied to the memory controller 8 through the bus interface units 21 and 22, and the memory controller 8 uses the image memory unit. 4 predetermined memories are accessed by generating a start address, a burst length, and other bus protocols.

  In the image data used in the imaging apparatus of this embodiment, one pixel is composed of a luminance signal Y and a chroma signal C (Cr, Cb). The format is Y: Cr: Cb = 4: 2: 2, and the luminance signal Y and chroma signal C (Cr, Cb) are both unsigned 8 bits (Cr = 4 bits, Cb = 4 bits). In this case, YC4 pixels are arranged in a 4-bit memory access width. The data processing unit of the system bus 2 (the amount of data that can be transferred by one burst transfer) is 64 bits, and the YC pixel data for one pixel is 16 bits. That is, the number of pixels p that can be transferred in one burst transfer is p = 4 pixels in this example.

  When each processing unit of the imaging device is connected to the global system bus 2 as shown in FIG. 1, the bus bandwidth and the processing unit of the memory controller 8 shared by each processing unit are closely related. There is. Here, the bus bandwidth is an amount including a data rate, a bus width (number of bits), and the like at which data can be transferred while avoiding congestion on a bus for transferring data.

  By the way, as described above, in this embodiment, the maximum burst length that can be processed by the memory controller 8 is 16 bursts. For example, even when image data of the same data size is written in the memory of the image memory unit 4, if a large number of 16 burst transfers can be used, the number of bus arbitrations is reduced, and the memory access processing by the memory controller 8 is efficient. The bus bandwidth can be reduced.

  Therefore, in this embodiment, it is efficient if the data (64 bits) of 4 pixels of YC pixel data can be bus-accessed by 16 bursts. When this is converted into the number of pixels, the data is 4 × 16 = 64 pixels.

  Therefore, in this embodiment, as shown in FIG. 40A, the horizontal direction of the image for one screen is divided in units of one burst transfer (64 pixels), and the image for one screen is divided vertically. Suppose that it is composed of a plurality of block image units (hereinafter referred to as strips). Then, as one of the memory access formats of the image data to the image memory unit 4, a memory access method for writing image data to the image memory unit 4 and reading it from the image memory unit 4 (hereinafter referred to as “the memory access method”). Adopt a strip access format). During still image NR processing, image data memory access to the image memory unit 4 is basically performed using this strip access format.

  Here, a format in which 64 pixels in the horizontal direction are bus-accessed by 16 bursts is referred to as a 64 × 1 format. The 64 × 1 format is a memory access method in which burst transfer for every four pixels is continuously repeated 16 times, which is the maximum number of continuous bursts, as shown in FIG. The memory controller 8 includes an address generation unit (not shown) for executing 64 × 1 format bus access. That is, as shown in FIG. 42 (B), when the head addresses A1, A2, A3,... A16 of burst transfer for every four pixels are determined, the bus access in the 64 × 1 format can be automatically performed. The memory controller 8 includes an address generation unit (not shown) for executing 64 × 1 format memory access.

  The strip access format is a system in which the 64 × 1 format is continuously performed in the vertical direction of the screen. When the 64 × 1 format is repeated in the horizontal direction and all accesses for one line are completed, the same access is made for the next line, thereby enabling access to image data in the raster scan format.

  The memory controller 8 of this embodiment includes an address generation unit (not shown) for executing such a strip access format.

  If the horizontal image data is not divisible by 64 pixels, a dummy area 151 is provided at the right end in the horizontal direction as shown with a shadow line in FIG. Or white pixel data is added as dummy data so that the number of pixels in the horizontal direction is a multiple of 64.

  The conventional raster scan method is suitable for reading data line by line because the addresses are continuous in the horizontal direction when accessing the image memory, whereas the strip access format is 1 burst. Since the address is incremented in the vertical direction in units of transfer (64 pixels), it is suitable for reading block-like data whose horizontal direction is within 64 pixels.

  For example, when reading a strip block of 64 pixels × 64 lines, the memory controller 8 accesses the four pixels of YC pixel data (64 bits) in 16 bursts for the image memory unit 4. Then, since the data of 4 × 16 = 64 pixels is obtained in the 16 bursts, after setting the address of the first line composed of 64 pixels in the horizontal direction, the remaining 63 is simply incremented in the vertical direction. It is possible to set the address of pixel data for a line.

  In this embodiment, in order to allow the motion detection / motion compensation unit 16 to access the image memory unit 4 in the above-described strip access format, the image correction / resolution conversion unit 15 in the preceding stage also accesses the strip access format. It corresponds to. In this embodiment, in still image NR processing at the time of still image shooting, a plurality of captured images obtained by high-speed continuous shooting are compressed and stored in the image memory unit 4, as necessary. Therefore, the still image codec unit 19 is also compatible with the strip access format so that the motion detection / compensation unit 16 can perform the block matching process by decoding and reading the data.

  FIG. 41A shows an example of strip unit division in an example in which an image for one screen includes horizontal × vertical = 640 × 480 pixels. In this example, the image for one screen is divided into ten strips T0 to T9 as a result of being divided in units of 64 pixels in the horizontal direction.

  In the case of the reduction ratio ¼ shown in FIG. 39 described above, the base surface target block of 32 × 32 pixels is shown as B0 to B29 in one strip as shown in FIG. Thus, there are 30 in total, 2 in the horizontal direction and 15 in the vertical direction.

  At the time of still image NR processing, when reading the basal plane target block from the image memory unit 4, in order to take advantage of the strip access format, in this embodiment, 64 pixels × 1 line is accessed to increase bus efficiency. I have to.

  For example, as shown in FIG. 42, when the reduction ratio of the reduction surface with respect to the base surface is 1/2, in this embodiment, the size of the base surface target block is horizontal × vertical = 16 pixels × 16 lines. The Therefore, when four basal plane target blocks are arranged in the horizontal direction, the number of pixels in the horizontal direction is 64 pixels. Therefore, in this embodiment, when the reduction ratio is 1/2, as shown on the right side of FIG. 42, the access unit for the image memory unit 4 is four of the target blocks, and the above-described strip of 64 × 1 format is used. Access by access type. That is, the four target blocks can be transferred by repeating access in units of 64 pixels in the horizontal direction (4 pixels × 16 bursts) 16 times while changing the address in the vertical direction.

  Similarly, as shown in FIG. 42, when the reduction ratio of the reduction surface relative to the base surface is 1/4, in this embodiment, the size of the base surface target block is horizontal × vertical = 32 pixels × 32 lines. Therefore, by accessing the image memory two by two, the base plane target blocks can be accessed in the above-described 64 × 1 format strip access format. When the reduction ratio of the reduction plane with respect to the base plane is 1/8, in this embodiment, the size of the base plane target block is horizontal × vertical = 64 pixels × 64 lines. By accessing the image memory one by one, it is possible to access in the strip access format of the 64 × 1 format described above.

  The strip access format is a very useful technique not only in terms of bus bandwidth but also in the internal processing of the above circuit. First, it is possible to reduce the size of the internal line memory of the vertical filter processing unit for performing vertical filter processing and the like.

  That is, when accessing pixel data in the raster scan format, for vertical filter processing, all the pixels for one line must be stored in the internal line memory of the vertical filter processing unit. Since the accumulation of the pixel data of the next line is not started, the internal line memory requires a plurality of line memories having a capacity for one line. In the case of the strip access format, in the above example, in the unit of 64 pixels Thus, since the pixel data of the next line in the vertical direction arrives, the internal line memory of the vertical filter processing unit only needs to have a plurality of line memories having a capacity of 64 pixels, and the size of the line memory Can be reduced.

  For circuits that perform processing in block format, such as resolution conversion processing, it is more efficient to convert from strip format to arbitrary block format than to convert from raster scan format to arbitrary block There is.

<Compression and decompression processing of captured image in still image NR processing during still image shooting>
As described above, in this embodiment, post-shooting addition processing is employed as the addition processing in the still image NR processing during still image shooting. For this reason, in the still image NR processing at the time of still image shooting, it is necessary to store all the reference images in the image memory unit 4. That is, in this embodiment, when the shutter button is pressed, the imaging apparatus captures two or more captured images by high-speed continuous shooting and stores them in the image memory unit 4.

  In the addition method during shooting, the image memory unit 4 only needs to store reference images for a maximum of two frames, whereas in the addition method after shooting, a larger number of reference images are stored in the image memory unit 4. Need to hold. Therefore, in this embodiment, in order to reduce the capacity of the image memory 4, the captured image captured by high-speed continuous shooting is compressed and encoded by the still image codec unit 18, stored in the image memory unit 4, and captured. When an image is used as a reference image, the compressed and encoded image data is decompressed and used.

  The outline of the flow of the still image NR processing at the time of still image shooting has been described in the flowcharts shown in FIGS. 23 and 24. However, in some steps in the flowcharts shown in FIGS. The compression encoding process or the decompression decoding process is performed.

  In step S1 of FIG. 23, a plurality of captured images captured by high-speed continuous shooting based on the shutter button operation are written in the image memory unit 4. At this time, the captured image data of the captured images is a still image. The codec unit 18 performs compression encoding. FIG. 43 shows the flow of image data at this time by dotted lines.

  That is, the captured image from the image sensor 11 is subjected to preprocessing such as sensor correction in the preprocessing unit 13 and then temporarily stored in the image memory unit 4 in the RAW signal format (data format before performing camera signal processing). Stored.

  Thereafter, the photographed image is read from the image memory unit 4 in a strip access format, converted from the RAW signal to YC pixel data by the data conversion unit 14, and subjected to image correction, resolution conversion, and the like by the image correction / resolution conversion unit 15. After being transmitted to the image memory unit 4 in a state of being directly compressed and encoded by the still image codec unit 18 without using the image memory unit 4, in this example by the JPEG (Joint Photographic Experts Group) method. Written. Therefore, here, the unit for compressing the image is not a single image unit but a strip unit. During high-speed continuous shooting, the above procedure is repeated for a plurality of sheets, and the image memory unit 4 stores and holds compressed data of the plurality of captured image data.

  When all of the captured images captured by high-speed continuous shooting are compressed and stored in the image memory unit 4, the target image setting process in the process of step S3 in FIG. 23 is performed. FIG. 44 shows the flow of image data at this time.

  That is, compressed image data of a captured image that is first captured after the shutter button is pressed is read from the image memory unit 4 in a strip access format, supplied to the still image codec unit 19, and decompressed and decoded. The decompressed and decoded image data is written in the image memory 4 as a base surface target image Pbt in a strip access format.

  Then, the decompressed and decoded base surface target image Pbt image data written in the image memory unit 4 is read out in a strip access format and supplied to the image superposition unit 17 through the motion detection / motion compensation unit 16. The image is reduced by the reduction plane generation unit 174 of the image superposition unit 17 shown in FIG. Then, the reduced target image from the reduced surface generation unit 174 is written in the image memory unit 4 in strip units as the reduced surface target image Prt.

  After the process for the target image is completed as described above, the setting process for the reference image in the process of step S3 in FIG. 23 is performed. The flow of the image data at this time is shown in FIG.

  That is, among the plurality of compression-encoded captured images stored in the image memory 4, a second captured image serving as a block matching reference image is read from the image memory unit 4 in a strip access format. Then, it is supplied to the still image codec unit 18 and decompressed and decoded. The decompressed and decoded image data is written in the image memory 4 as a base reference image Pbr in a strip access format.

  The decompressed and decoded base plane reference image Pbr image data written in the image memory unit 4 is read out in a strip access format and supplied to the image superposition unit 17 through the motion detection / motion compensation unit 16. The image is reduced by the reduction plane generation unit 174 of the image superposition unit 17 shown in FIG. Then, the reduced reference image from the reduced surface generation unit 174 is written in the image memory unit 4 in a strip access format as a reduced surface reference image Prr.

  Next, motion detection / motion compensation is performed on the decompression-decoded target image and the decompressed and decoded reference image. This corresponds to the processing in steps S4 to S9 in FIG. Further, it corresponds to the processing shown in the flowcharts of FIGS. FIG. 46 shows the flow of image data at this time.

  First, reduction plane block matching processing is performed. That is, the image data of the set reduction plane target block is read out from the reduction plane target image Prt of the image memory unit 4 in the strip access format, and is stored in the target block buffer unit 161 of the motion detection / motion compensation unit 16. The reduced surface matching processing range corresponding to the search range for the reduced surface target block is read from the reduced surface reference image Prr in the strip access format and stored in the reference block buffer unit 162.

  Then, the pixel data of the reduced surface target block and the reduced surface reference block are read out from the respective buffer units 161 and 162, the reduced surface block matching process is performed in the matching processing unit 163, and the set reduced surface target block is set. A reduction plane motion vector is detected, and a reduction plane motion vector for the set reduction plane target block is detected.

  Next, based on the reduced plane motion vector, as described above, the base plane target block is set, the base plane search range is set, and the base set from the base plane target image Pbt of the image memory unit 4 is set. The plane target block is read in the strip access format and stored in the target block buffer 161, and the base plane matching processing range corresponding to the set base plane search range is read from the base plane reference image Pbr of the image memory unit 4 in the strip access format. Read and store in the reference block buffer unit 162.

  Then, pixel data of the basal plane target block and the basal plane reference block are read from the respective buffer units 161 and 162, the basal plane block matching process is performed by the matching processing unit 163, and the motion vector calculation unit 164 sets the pixel data. The base plane motion vector with sub-pixel accuracy is detected for the base plane target block.

  Then, the motion detection / compensation unit 16 reads out the motion compensation block from the base plane reference image of the reference block buffer unit 162 based on the detected base plane motion vector, and the image overlay unit 17 together with the base plane target block. To supply. The image superimposing unit 17 superimposes the images in block units, and pastes the resulting base NR image Pbnr in block units on the image memory unit 4 in a strip access format. The image superimposing unit 17 also pastes the reduced surface NR image Prnr generated by the reduced surface generation unit 174 to the image memory unit 4.

  The above operation is repeated for all target blocks of the target image, and when block matching for all target blocks is completed, the third image is used as a reference image, and the base NR image Pbnr and reduced surface NR image Prnr is used as a target image (step S14 in FIG. 24), and the same block matching process as described above is repeated (step S4 in FIG. 23 to step S14 in FIG. 24). When all the captured images have been processed, the finally obtained base NR image Pbnr is compressed and encoded by the still image codec unit 18 and recorded on the recording medium of the recording / reproducing device unit 5. To do.

  As described above, in this embodiment, the NR image obtained by superimposing the target image and the reference image becomes a target image for the next superposition. In addition, the basal plane target block becomes unnecessary data after the block matching process is completed.

  Due to the above characteristics, in this embodiment, the image superimposing unit 17 overwrites the NR-completed block obtained by superimposing the basal plane target block and the motion compensation block on the address where the original basal plane target block was located. Yes.

  For example, as shown in FIGS. 47A and 47B, when the block matching process is performed using the block B4 of the strip T5 as a target block, the strip T0 to the strip T4 and the block B0 to the block B0. B3 is all NR completed blocks.

  As described above, the image correction / resolution conversion unit 15 performs memory access in the strip access format, and the compression encoding process for the still image is also accessed in the strip access format. Image data can be transferred directly to the still image codec unit 18 without going through the image memory unit 4, and high-speed continuous shot images can be efficiently compression-encoded and stored in the image memory unit 4.

<Reduction of image memory capacity>
Next, reduction of the image memory capacity at the time of decompression decoding of the reference image will be described.

  At the time of still image NR processing at the time of still image shooting in this embodiment, all target images are decompressed and decoded with respect to images captured by compression encoding in the image memory unit 4, but as shown in the image data flow of FIG. In addition, with respect to the reference image, only the image data in strip units used for the matching process is decompressed and decoded, thereby reducing the image memory capacity. That is, the capacity of the image memory that stores the decompressed and decoded reference images (base plane and reduced plane) in the image memory unit 4 is reduced by decompressing and decoding the image data in strip units.

  This will be described with an example. For example, the basal plane target image is divided into strips T0 to T9 having 64 pixels in the horizontal direction as shown in FIG. 49A, and the basal plane reference image is as shown in FIG. 49B. Similarly, suppose that the horizontal direction is divided into strips R0 to R9 having 64 pixels.

  Now, assuming that block matching is performed for the target block in the strip T4, among the strips R0 to R9 of the basal plane reference image, the strip that may be accessed as the matching processing range is centered on the strip R4. It can be limited to several.

  For example, in the case where the reduction magnification is 1/4, in order to reduce and obtain a 44 pixel matching processing range in the horizontal direction (see FIG. 39B), in the basal plane reference image, the horizontal direction 176 pixel image data is required. Therefore, as can be seen from FIG. 49 (C), as shown in FIG. 49 (B), the basal plane reference image has two strips R1, two on each of the left and right sides of the strip R4. R2 and strips R3 and R4 are necessary, and only five strips R2 to R6 need to be decompressed and decoded.

  Next, consider the case where the target block matching process included in the strip T4 is completed and the target block matching process included in the strip T5 is entered as shown in FIG. At this time, in order to obtain the matching processing range, as shown in FIG. 50B, two strips R3 to R7 are required on the left and right sides of the strip R5. That is, since the strips R3 to R6 are already stored in the image memory unit 4 at the time of matching processing of the target block included in the strip T4, it is only necessary to newly decompress and decode the strip R7. At the same time, the strip R2 becomes unnecessary.

  Therefore, in this embodiment, when the strip R7 is decompressed and decoded, the striped and decrypted strip R7 is overwritten on the reference image memory of the image memory unit 4 at the address where the strip R2 is located. As a result, the reference image data decompressed and decoded on the reference image memory of the image memory unit 4 is always only for five strips, and the reference image memory of the image memory unit 4, that is, the image memory unit It is possible to effectively save four work areas.

  When the reference image data is decoded by the above method, as shown in FIGS. 51A and 51B, the arrangement of addresses on the reference image memory in the image memory unit 4 changes. That is, as for the target block in the strip T4, when the reference image in the reference image memory is centered on the strip R4 as shown in FIG. 51 (A), the image data is sequentially imaged from the strip R2. Since it is stored in the reference image memory of the memory unit, when referring to the pixel in the left direction from the standard coordinate Sc4 of the strip R4, a desired value is subtracted from the address of the standard coordinate Sc4 and the right direction from the standard coordinate Sc4. When referring to this pixel, a desired value is added to the address of the standard coordinate Sc4.

  On the other hand, as for the target block in the strip T5, when the reference image in the reference image memory is centered on the strip R5 as shown in FIG. 51B, the pixels within 128 pixels are referred to in the right direction. In this case, the desired value is added to the address of the standard coordinate Sc5 of the strip R5, and when referring to the pixel of 129 pixels or more in the right direction, the desired value is added to the address of the standard coordinate Sc5, and then the total strip It is necessary to subtract the address value for the width.

  The reference coordinates Sc0 to Sc9 of the strips R0 to R9 are examples of memory address pointers when accessing the reference image memory. The memory address pointer is not limited to the address position as in the example of FIG.

  As described above, the unit of compression encoding for still images is also in the strip format, so that only the range necessary for block matching processing can be decoded, and the work area of the image memory unit 4 can be saved. . Furthermore, the size of the decoded reference image on the work area is always kept constant by decoding and overwriting the strip of the reference image to be used next on the address of the unnecessary reference image strip on the image memory. can do.

  In the example described with reference to FIGS. 49 and 50, the horizontal size SA of the matching processing range set in the reference image is equal to or larger than the horizontal size SB of the strip that is the compression unit (SA> SB). This is the case.

  If the horizontal size of the matching processing range is an integral multiple of the horizontal size of the strip, and the matching processing range matches the strip separation position, useless data access is eliminated and efficiency is improved. Go up further.

  If the horizontal size of the matching processing range set in the reference image is smaller than the horizontal size of the strip that is the compression unit, the matching processing range is the size included in one strip. If it is included in one strip, the one strip may be decrypted, and if the matching processing range spans two strips, the two strips may be decrypted.

  Even in such a case, when the position of the target block changes and the position of the matching processing range changes, and the matching processing range spans two strips, the matching processing range is newly straddled. It is the same as in the above example that only the strips that have become necessary should be decrypted.

[Efficient memory access during video NR processing]
As described above, the still image NR processing requires a clean image with no noise even if it takes some processing time, whereas the moving image NR processing requires real-time performance, that is, speed rather than accuracy. .

  The flow of the moving image NR processing operation is as shown in FIG. 25 described above, and the flow of image data during the moving image NR processing is as shown in FIGS. 52 and 53. The flow of image data during the moving image NR process will be described with reference to FIGS. 52 and 53.

  In the case of moving image NR processing, YC image data from the data conversion unit 14 is written in the image memory unit 4 in a strip access format after being corrected in the horizontal direction by the image correction / resolution conversion unit 15. Then, the image correction / resolution conversion unit 15 reads the horizontal correction image from the image memory unit 4 in the strip access format, performs the correction processing in the vertical direction, and then performs the strip access format to the motion detection / motion compensation unit 16. The target block is sent in real time (step S21 in FIG. 25).

  The motion detection / motion compensation unit 16 stores the target block sent from the image correction / resolution conversion unit 15 in the target block buffer unit 161, and also performs block matching according to the target block as shown in FIG. The processing range is read from the reference image stored in the image memory unit 4 (the first image is a non-superimposed image, but the second and subsequent images are NR images) and stored in the reference block buffer unit 162 (FIG. 25). Step S23).

  Then, the motion detection / compensation unit 16 reads the target block from the target block buffer unit 161 and the reference block from the block matching processing range, performs the above-described hierarchical block matching process, and performs motion vector detection and A motion compensation block is generated, and the motion compensation block and the target block are supplied to the image superimposing unit 17 (steps S24 to S26 in FIG. 25).

  The image superimposing unit 17 superimposes the received target block and the motion compensation block (step S27 in FIG. 25). Then, as shown in FIG. 52, the image superimposing unit 17 pastes the NR-completed image on which the image is superimposed on the image memory unit 4 (step S28 in FIG. 25).

  As shown in FIG. 53, the NR image pasted in the image memory unit 4 is read into the video codec unit 19 at the next frame (the NR image one frame before), compressed, and then recorded. It is stored in the recording medium of the playback device unit 5 and used as a reference image in the next frame. Further, the NR image (an NR image one frame before) pasted on the image memory unit 4 is simultaneously read from the NTSC encoder 20 and is also output to the monitor display 6.

  In the moving image NR processing, the above processing is repeated for each captured image.

  Here, the reference image is one frame, but as shown in FIG. 53, a plurality of reference images may be held and an image with higher correlation may be used.

  In the moving image NR process, real-time characteristics are important. Therefore, unlike the still image NR process, the reduced block target image is not created and stored in the image memory unit 4, and the target block shown in FIG. The reduction processing unit 1613 in the buffer unit 161 generates a reduction plane target block and stores it in the reduction plane buffer 1612 as it is.

<Data format for memory access during video NR processing>
At the time of still image NR processing, the data access format for the memory access to the image memory unit 4 employs a strip access format in which the 64 × 1 format is continuously performed in the vertical direction.

  However, in the strip access format, when a block having a horizontal number of pixels smaller than the horizontal width of the strip (the number of pixels in the horizontal direction) is read, the burst length becomes less than 16 bursts. The most efficient data transfer (16 bursts) cannot be used, and the efficiency deteriorates.

  Also, when reading a block with a horizontal pixel count larger than the horizontal width of the strip, it must be divided into two transfers on the left and right of the border between two strips adjacent in the horizontal direction, Since the addresses are far apart on the left and right of the boundary between the two strips, there is a problem that the address processing is rather complicated.

  Furthermore, the basal plane matching process range in the hierarchical matching process is determined based on the reduced plane motion vector detected by the reduced plane matching, and is not a fixed area. May be. For this reason, when reading the basal plane matching processing range from the reference image (NR image) stored in the image memory unit 4, as described above, the boundary between two adjacent strips across the basal plane matching processing range. However, there is a problem that the address processing becomes complicated because the addresses become far apart on the left and right of the boundary between the two strips.

  Then, the size of the pixel block contained in each of the two divided strips is smaller than the horizontal width of the strip, and as described above, the access efficiency is deteriorated in the strip access format, and it looks like a movie. In addition, there is a problem that it is not suitable for applications requiring high-speed real-time processing.

  In order to deal with the above-described problems, in this embodiment, a memory access method for pixel data in units of blocks in a rectangular area composed of a plurality of lines and a plurality of pixels is prepared for accessing a reference image at the time of moving image NR processing. I have to.

  In this embodiment, as one of the reference image accesses at the time of moving image NR processing, an image (screen) is displayed as shown in FIG. 54A in consideration of the above-described 64 pixels that can be burst transferred. Is divided into blocks of horizontal × vertical = 8 pixels × 8 lines, which is a unit of maximum burst transfer (64 pixels), and a block access format for writing / reading to / from the image memory unit 4 is used. Hereinafter, the block access format in units of blocks composed of 8 lines × 8 pixels will be referred to as an 8 × 8 format.

  In this 8 × 8 format, as shown in FIG. 54 (B), four horizontal pixels are used as one burst transfer unit, and eight horizontal pixels are transferred in two burst transfers (two bursts). When the burst transfer of 8 pixels in the horizontal direction is completed, the 8 pixels of the next line are similarly transferred in 2 bursts. Then, 64 pixels are transferred at a time in 16 bursts, which is the maximum burst length.

  Assuming that AD1 is the initial address of the block access format in the 8 × 8 format, as shown in FIG. 54 (B), addresses AD2 to AD16 in units of four pixels are automatically determined, and in 16 consecutive bursts, Memory access in 8 × 8 format is possible. The memory controller 8 includes an address generation unit for memory access in the 8 × 8 format.

  When the start address AD1 in the 8 × 8 format is designated on the frame memory of the image memory unit 4, the memory controller 8 uses the address generation unit to generate addresses AD1 to AD1 for memory access in the 8 × 8 format. AD16 is calculated and 8 × 8 format memory access is performed.

  As shown in FIG. 54B, the 8 × 8 format basically provides access in units of 64 pixels (16 bursts), and is the most efficient as the imaging apparatus of this embodiment. However, as will be described later, 16 bursts is the longest burst length, and may be shorter than that. In that case, the 8 × 8 format is not practically realized. However, for example, in the case of 8 pixels × 4 lines, image data can be transferred in half 8 bursts.

  If the image data in the horizontal direction and the vertical direction cannot be divided by 8 pixels, dummy regions 152 are provided at the right end in the horizontal direction and the lower end in the vertical direction as shown with a shadow line in FIG. For example, black or white pixel data is added as dummy data to the dummy area 152 so that the image size in the horizontal and vertical directions is a multiple of eight.

  The fact that the memory access can be accessed with 8 × 8 pixels by the 8 × 8 format memory access means that the reduction plane matching processing range and the base plane matching processing range are all 8 × 8 pixel blocks from the image memory unit 4. Can be read in units. Therefore, in the imaging apparatus of this embodiment, it is possible to perform bus access only by the most efficient data transfer (16 bursts), and the bus efficiency is maximized.

  The 8 × 8 format can be applied to access in units of a plurality of pixels as long as it is a multiple of 8 × 8 pixels, such as 16 × 16 pixels, 16 × 8 pixels, or the like.

  The most efficient data transfer unit (unit of maximum burst length) of the bus is 64 pixels in the above example, but the most efficient data transfer unit is the number of pixels p that can be transferred in one burst, and It becomes p × q pixels determined by the maximum burst length (maximum number of continuous bursts) q. Then, the block access format to be written in the image memory unit 4 may be determined based on the number of pixels (p × q). The bus transfer efficiency is best when the reduced plane matching processing range and the base plane matching processing range have a size close to a multiple of the horizontal and vertical directions of the block format.

  The reduction plane matching process range and the base plane matching process range when the reduction magnification is 1/2 shown in FIG. 39 will be compared with the case of the strip access format of the 64 × 1 format.

  As shown in FIG. 39B, the reduction plane matching processing range 143 is 44 pixels × 24 lines in this example. When this reduced plane matching processing range 143 is accessed in the 64 × 1 format, as shown in the upper side of FIG. 55, it is necessary to access the transfer by repeating 4 pixels × 11 bursts 24 times in the vertical direction.

  On the other hand, when the same reduced plane matching processing range 143 is accessed in the 8 × 8 format, as shown in FIG. 56, the reduced plane reference vector (0, 0) is centered on the reduced plane reference block. A reduction plane matching processing range 143 of pixels × 24 lines is determined. The reduced plane reference block of the reduced plane reference vector (0, 0) is a reduced plane reference block having the same coordinates as the reduced plane target block. In this case, as shown in FIG. 39A, the reduced surface target block is always set only in units of 8 pixels and 8 lines.

  Since the vertical direction of the reduction plane matching processing range 143 is 24 lines, as shown in FIG. 56, if the vertical search range is a multiple of the block size of 8 × 8, the vertical direction is 8 × 8 format. Can be accessed without waste. On the other hand, since the horizontal direction of the reduction plane matching processing range 143 is 44 pixels, it is not a multiple of 8. For this reason, in the example shown in FIG. Dummy image data 153 made of black or white is inserted on both sides in the horizontal direction by 6 pixels.

  FIG. 56 shows that dummy image data 153 includes six pixels on the left and right sides, but 24 × transfer is required in the 64 × 1 format, whereas 21 × transfer is required in the 8 × 8 format. You can see that Furthermore, in the 8 × 8 format, all transfers are the most efficient data transfer (16 bursts) in the imaging apparatus of this embodiment, and the bus efficiency is also improved.

  Further, as shown in FIG. 39C, the base plane matching processing range 144 when the reduction ratio is 1/2 is 20 pixels × 20 lines. When the base plane matching processing range 144 is accessed, As shown in the lower side of FIG. 55, in the case of the 64 × 1 format, it is necessary to access for transferring 4 pixels × 5 bursts 20 times in the vertical direction.

  On the other hand, in the case of the 8 × 8 format, the basal plane matching processing range 144 is a matching processing range of 20 pixels × 20 lines around the basal plane reference block 142 of the basal plane reference vector (0, 0). Is determined. Since the basal plane matching processing range 144 is determined by the reduced plane motion vector calculated by the reduced plane matching, various combinations can be considered in units of one pixel.

  For example, the most efficient case is when the base pixel matching processing range 144 of 20 pixels × 20 lines is allocated to 9 blocks as shown in FIG. 57 for 8 × 8 blocks. In this case, as shown in FIG. 57, in the basal plane matching processing range 144 of 20 pixels × 20 lines, there are regions that are less than 8 pixels and 8 lines at the end in the horizontal direction and the end in the vertical direction. Then, by inserting dummy pixel data 153 there and accessing the 8 × 4 format 4 pixel × 16 burst transfer 9 times, the image data of the base pixel matching processing range 144 of the 20 pixel × 20 line is obtained. Can be accessed.

  However, as shown in FIG. 58, the dummy image data 153 is not inserted into the vertical end portion of the basal plane matching processing range 144 of 20 pixels × 20 lines, and the number of bursts at the vertical end portion. Can be set to match the data to be transferred (for 4 lines), and 4 pixels × 8 bursts can be used as an access for 3 repeated transfers. In this case, in the example shown in FIG. 58, a total of 9 transfers including a transfer of 4 pixels × 16 burst of 8 × 8 format 6 times and a transfer of repeating 4 pixels × 8 burst 3 times, The image data in the base plane matching processing range 144 of 20 pixels × 20 lines can be accessed.

  On the other hand, the most inefficient example is a case where the base plane matching processing range 144 of 20 pixels × 20 lines covers 16 blocks of 8 × 8 blocks as shown in FIG. In this case, as shown in FIG. 59, in the basal plane matching processing range 144 of 20 pixels × 20 lines, there are regions of less than 8 pixels and 8 lines at the horizontal end and the vertical end. Then, by inserting dummy pixel data 153 there and accessing the transfer of the 4 × 16 burst of 8 × 8 format 16 times repeatedly, the image data of the basal plane matching processing range 144 of the 20 pixels × 20 lines is obtained. Can be accessed.

  However, as shown in FIG. 60, the dummy image data 153 is not inserted into the vertical end portion of the basal plane matching processing range 144 of 20 pixels × 20 lines. As the number of times matched to the data to be transferred (each for two lines), a transfer of 4 pixels × 4 bursts can be made a total of 8 transfers. In this case, in the example shown in FIG. 60, a total of 16 transfers of 8 × 8 format 4 pixel × 16 burst transfer that repeats 8 times and 4 pixel × 4 burst repeat 8 times, The image data in the basal plane matching processing range 144 can be accessed.

  Therefore, the access to the image memory unit 4 of the image data of the base plane matching processing range 144 of 20 pixels × 20 lines needs to be transferred 20 times in the 64 × 1 format, but in the 8 × 8 format. 9 transfers at the shortest and 16 transfers at the longest. Further, in the 8 × 8 format, more than half of the transfer is the most efficient data transfer (16 bursts) in the imaging apparatus system of this example, and the bus efficiency is also improved.

  In this embodiment, in order to further improve the bus bandwidth, as another one for accessing the reference image at the time of moving image NR processing, as shown in FIG. The block access format for writing / reading to / from the image memory unit can be selected by dividing into blocks of 4 pixels × 4 lines, which are 1/4 units of (64 pixels). This block access format in units of blocks consisting of 4 lines × 4 pixels is hereinafter referred to as a 4 × 4 format.

  In this 4 × 4 format, as shown in FIG. 61 (B), four horizontal pixels are used as one burst transfer unit, and four horizontal pixels are transferred in one burst transfer (one burst). When the burst transfer of 4 pixels in the horizontal direction is completed, the 4 pixels of the next line can be similarly transferred in 1 burst, and image data of 4 × 4 format can be transferred in 4 bursts. In the case of the 4 × 4 format, as shown in FIG. 61 (B), a block composed of 4 pixels × 4 lines = 16 pixels is continuously accessed four times (for 4 blocks) in the horizontal direction to obtain 64 pixels. Are transferred once.

  In other words, since 4 × 4 format image data can be transferred in 4 bursts, it is possible to continuously access 4 blocks in a horizontal direction of 4 pixels × 4 lines with a maximum burst length of 16 bursts. The 64 pixels for the block of four 4 pixels × 4 lines can be transferred at one time.

  When 64 pixels (16 bursts, which is the maximum burst length) are transferred at one time using this 4 × 4 format, as shown in FIG. 61B, the start address of the block of 4 pixels × 4 lines is set to AD1 as the initial address. Then, the addresses AD2 to AD16 in units of 4 pixels are determined as shown in the figure, and the memory access in units of 64 pixels in the 4 × 4 format can be performed with continuous 16 bursts.

  As shown in FIG. 61B, the 4 × 4 format is most efficient when accessed in units of 64 pixels (16 bursts). However, it may be necessary to access in units of 4 blocks or less in the horizontal direction. For example, in access in units of 2 blocks (32 pixels) in the horizontal direction, addresses AD1 to AD8 in units of 4 pixels are determined in FIG. 61B, and access in units of 32 pixels can be performed in 8 consecutive bursts. A block of 4 pixels × 4 lines, 1 block (16 pixels) unit, 3 blocks (24 pixels) unit, etc. can be accessed in the same manner. The memory controller 8 includes an address generation unit for memory access in the 4 × 4 format.

  When the start address AD1 of 4 × 4 format and the number of blocks of 4 pixels × 4 lines in the horizontal direction are specified on the frame memory of the image memory unit 4, the memory controller 8 uses the address generation unit, The address AD2 and subsequent addresses for memory access in 4 × 4 format can be calculated, and memory access in 4 × 4 format can be executed.

  As in the case of the 8 × 8 format described above, when the image data is not divisible by 4 pixels, a dummy area 154 is provided at the right end in the horizontal direction and the lower end in the vertical direction as shown in FIG. Thus, the horizontal and vertical sizes are set to be a multiple of four.

  With this 4 × 4 format, the above-described bus access in the base plane matching processing range (20 pixels × 20 lines) 144 is further improved over the 8 × 8 format.

  For the block unit of 4 pixels × 4 lines, the most efficient is that the base plane matching processing range 144 of 20 pixels × 20 lines is exactly 5 blocks × vertical direction in the horizontal direction as shown in FIG. In this case, 5 blocks = 25 blocks are allocated.

  To access these 25 blocks, it can be divided into 5 lines of 4 blocks in the horizontal direction and 5 lines of 1 block in the horizontal direction. A total of 10 transfers, 5 transfers of 4 pixels × 16 bursts, 5 transfers of 4 pixels × 4 bursts per block, may be used.

  On the other hand, the worst example is that the basal plane matching processing range 144 of 20 pixels × 20 lines has 6 blocks in the horizontal direction × 6 blocks in the vertical direction of the block of 4 pixels × 4 lines as shown in FIG. = 36 blocks. In this case, as shown in FIG. 63, in the basal plane matching processing range 144 of 20 pixels × 20 lines, there are areas less than 4 pixels and 4 lines at the left and right ends in the horizontal direction and the upper and lower ends in the vertical direction. Therefore, dummy pixel data 154 is inserted therein.

  In order to access these 36 blocks in the 4 × 4 format, it can be divided into 6 lines of 4 blocks in the horizontal direction and 6 lines of 2 blocks in the horizontal direction, and the maximum burst length of FIG. 4 blocks × 16 bursts in units of 4 blocks in 6 transfers, 6 transfers in 4 blocks × 4 bursts in units of 2 blocks, and a total of 12 transfers may be sufficient.

  Therefore, the number of transfer accesses is further smaller than the 16 transfers in the 8 × 8 format shown in FIG. 60, and the ratio of using 16 bursts, which is the most efficient data transfer in the imaging apparatus of this embodiment, is higher. Increasing bus efficiency.

  As described above, during the moving image NR process, when the motion detection / compensation unit 16 reads the matching processing range from the image memory unit 4, it is not the 64 × 1 format and the strip access format, but the block format, in this example. The block access format based on 8 × 8 format or 4 × 4 format is more suitable for memory access.

  Therefore, the moving image NR image as a result of superposition may be written in the image memory unit 4 in the block access format block format as described above.

  However, the image memory unit 4 is accessed through the system bus 2 and the memory controller 8 in addition to the motion detection / motion compensation unit 16, the image superposition unit 17, and the moving image codec unit, as well as other processing units. However, the memory access according to the block format is not necessarily suitable for all of the other processing units.

  For example, in the imaging apparatus of this embodiment, at the time of moving image NR processing, an image after the NR processing is converted into display image data by the NTSC encoder 20 and an image monitor display is performed on the monitor display 6. Since the NTSC encoder 20 basically processes image data in a raster scan format, the image data is received from the image memory unit 4 in a block access format such as an 8 × 8 format or a 4 × 4 format. When accessed, there is a problem that transfer efficiency deteriorates.

  Therefore, in this embodiment, in step S28 of FIG. 25 described above, the NR image of the moving image NR processing result from the image superimposing unit 17 is an 8 × 8 format, a 4 × 4 format, or the like as shown in FIG. Is written in the first memory area 41NR for the NR image in the image memory unit 4 and is written in the second memory area 42NR for the NR image in the image memory unit 4 in the 64 × 1 format. As shown in FIG. 64, in the first memory area 41NR, an NR image and a reduced surface image are written together.

  The motion detection / motion compensation unit 16 obtains the target block from the image correction / resolution conversion unit 15 in a strip format, and the motion detection / motion compensation unit 16 and the image overlay unit 17 have a strip format, 64 Processing is performed for each 64 pixels in the × 1 format. For this reason, when writing the NR image resulting from the image overlay from the image overlay unit 17 into the image memory unit 4 in the 64 × 1 format, the address is skipped in the raster scan format and the strip access format is used. It is done by doing.

  In this embodiment, the NR image written in the block format at the time before one frame in the first memory area 41NR for the NR image of the image memory unit 4 is set as the reference image, and the memory controller 8 The matching processing range is read from the reference image in a block format such as 8 × 8 format or 4 × 4 format, and transferred to the motion detection / compensation unit 16.

  In this embodiment, the NR image written in the 64 × 1 format in the second memory area 42NR for the NR image of the image memory unit 4 in the 64 × 1 format at the time one frame before is in the 64 × 1 format, and The data is read out in the raster scan format and transferred to the moving image codec unit 19. In this example, the moving image is compressed and encoded by the MPEG method and recorded in the recording / reproducing device unit 5. The NR image written in the second memory area 42NR in the 64 × 1 format one frame before is read out in the 64 × 1 format and in the raster scan format and supplied to the NTSC encoder 20. Then, it is converted into an NTSC signal, supplied to the monitor display 6, and displayed as a recorded monitor image.

  As described above, in this embodiment, the NR image is written twice in the image memory unit 4 in both the first memory area 41NR and the second memory area 42NR. Although the capacity of the image memory unit 4 and the power consumption of the image memory unit 4 and the memory controller 8 increase, the efficiency of bus access of the image data from the image memory unit 4 to the motion detection / motion compensation unit 16 is good, and the bus bandwidth is reduced. There is a merit that it can be greatly reduced.

  As described above, in the imaging apparatus of this embodiment, whether to write in the image memory unit 4 only in the strip access format in the 64 × 1 format or in the block access format based on the strip access format and the block format. , Either can be selected. If you want to reduce power consumption or reduce memory capacity, choose to use only the strip access method, and if you want to reduce the bus bandwidth, use two types of strip access format and block access format. It is desirable to choose.

<Motion information generation and additional processing in still image shooting mode>
As described above, in the still image shooting mode in the imaging apparatus of this embodiment, as shown in FIG. 65, before the shutter button is pressed as a still image recording instruction, the pre-still image shooting EE mode is set.

  As shown in FIG. 65, in the pre-still image shooting EE mode, captured image data G1 to G7 having a normal frame rate of 60 fps are output from the image sensor 11, and an EE image E1 based on the captured image data G1 to G7. To E5 are displayed on the monitor display 6.

  In the pre-still image shooting EE mode, as shown in FIG. 65, the block-by-block motion vector group information MV1 to MV5 for each EE image detected by the motion detection / motion compensation unit 16 is a motion feature. It is held in the buffer unit of the detection unit 30.

  When the shutter button is pressed from this state and a still image recording instruction is issued, the imaging apparatus changes from the EE mode before still image shooting to the still image NR process. That is, a plurality of captured images are captured at a frame rate higher than the normal frame rate, and a still image NR process is performed on the plurality of captured images, and image data of a recorded still image is generated and recorded. It is recorded on the recording medium of the playback device 5.

  Then, the motion feature detection unit 30 detects the motion feature of the recorded still image from the block-by-block motion vectors of the plurality of EE images E3 to E5 in this example immediately before the shutter button is pressed. The

  Here, the reason why the motion vector obtained at the time of EE monitoring before shooting a still image is used to detect the motion feature of the recorded still image is as follows.

  That is, in the still image NR process, a plurality of images are shot at high speed so that the motion (blur) of the image is reduced. For this reason, the motion vector required at the time of still image NR processing is a minute amount, and is not suitable as a motion vector for determining the motion feature of a still image. On the other hand, the moving image NR process at the time of EE monitoring before still image shooting is performed at a normal frame rate.

  Therefore, in this embodiment, at the time of EE monitoring before still image shooting, the motion detection / motion compensation unit 16 performs motion vector detection from the captured image data of 60 fps, which is a normal frame rate, and is obtained at the time of EE monitoring before still image shooting. The motion vectors are stored for the last few frames before the still image recording instruction, and a still image is taken based on the still image recording instruction, and used for analysis of motion information about the still image to be recorded.

  The most recent number of frames before the still image recording instruction used for detecting the motion feature can be arbitrarily changed according to shooting conditions, settings by the user, and the like. Further, in the buffer unit of the motion feature detection unit 30, it is only necessary to hold the number of the latest frames, and the motion vector information of a frame temporally prior to the number of the latest frames may be deleted.

[Motion feature detection processing]
In this embodiment, for the still image shooting mode, various settings for motion feature detection are made in advance by the user through the user operation input unit 3. Here, the various settings include designation of the size of the target block, designation of an area in the screen where motion features are detected, designation of block matching used when detecting a motion vector during EE monitoring before still image shooting, and the like.

  The target block size is designated as a small size when it is desired to detect a fine motion feature, and as a large size when a rough motion feature is acceptable.

  The area within the screen to detect motion features is specified by the user according to the movement of the subject, for example, the entire screen, only the center of the screen, the right half of the screen, the left half of the screen, the upper half of the screen, the lower half of the screen Is set. In addition, by setting an arbitrary area on the EE image, it is possible to designate an area in the screen for detecting a motion feature.

  The block matching used for motion vector detection in the EE monitor before still image shooting only needs to be able to detect the motion characteristics of the subject to be recorded as a still image, and the layered matching process described above may not be performed. Therefore, it can be specified. For example, the user selects and designates one of the three types of block matching methods: only block matching on the reduced surface, only block matching on the base surface, or layered matching. Can do.

  As described above, in the motion feature detection unit 30, motion vectors are stored for a plurality of frames as motion parameters corresponding to the various settings described above. As shown in FIG. 66, the motion vector is calculated for each block BK obtained by dividing the screen into a plurality of parts. Therefore, motion vectors (motion vectors for each block) are obtained for the number of blocks BK for each captured image, here, one EE image. The motion feature detection unit 30 extracts motion features according to the above-described various settings using the motion vectors for each block for a plurality of frames.

  In this embodiment, in the motion feature detection unit 30, the size of the motion vector for each block exceeds a predetermined threshold in each block specified on the screen or within the area specified on the screen. This block is referred to as a hit block HBK (see FIG. 66), and attention is paid to the hit block HBK so that the motion feature of the frame (image) can be detected.

  In this embodiment, for example, the following types of motion features to be detected are detected. The types of motion features to be detected will be described with reference to FIG.

(1) Horizontal movement For example, as shown in FIG. 67 (A), most of the motion vectors of each block BK specified on the screen or in the area specified in the screen are arbitrarily specified horizontal. When concentrated on the direction, the motion feature of the still image is detected as “horizontal movement”. In this case, the moving direction is also detected, and the information on the moving direction is also included in the movement feature information. In this case, the determination of the moving direction may be calculated from the motion vector of the block BK specified on the screen or in the entire area specified in the screen, or the trend of the motion vector of the hit block HBK. You may judge from.

(2) Vertical movement For example, as shown in FIG. 67 (B), most of the motion vectors of each block BK specified on the screen or in the area specified in the screen are arbitrarily specified vertical. When concentrated on the direction, the motion feature of the still image is detected as “vertical movement”. In this case, the moving direction is also detected, and the information on the moving direction is also included in the movement feature information. In this case, the determination of the moving direction may be calculated from the motion vector of the block BK specified on the screen or in the entire area specified in the screen, or the trend of the motion vector of the hit block HBK. You may judge from.

(3) Movement in the depth direction For example, as shown in FIG. 67 (C), most of the motion vectors of each block BK specified on the screen or in the area specified in the screen have an arbitrary specific depth. When concentrated on the direction, the motion feature of the still image is detected as “movement in the depth direction”. In this case, the moving direction is also detected, and the information on the moving direction is also included in the movement feature information. In this case, the determination of the moving direction may be calculated from the motion vector of the block BK specified on the screen or in the entire area specified in the screen, or the trend of the motion vector of the hit block HBK. You may judge from.

(4) Movement concentrates in a certain range For example, as shown in FIG. 67 (D), hit blocks HBK specified on the screen or in an area specified in the screen are concentrated in a certain area range. If it is, the motion feature of the still image is detected as “concentration of motion within a certain range”. In this case, the position of the concentrated motion area is also detected, and the position information of the motion area is also included in the motion feature information.

(5) No motion For example, as shown in FIG. 67 (E), when there is no hit block in a region designated on the screen or designated in the screen, the motion feature of the still image is “ Detects no movement. In this embodiment, the still image determined to be “no motion” is further subjected to character recognition, face recognition, and the like for the recorded still image to further investigate the characteristics of the still image. In this example, when it is determined that the image is a character image, information indicating that, such as a character flag, and when it is determined that the image is a face image, information indicating this, such as a face flag, is displayed. If it is determined that the image is not a face image, information such as other flags indicating that the image is not a character image or a face image is included in the motion feature information.

  Although character recognition means and face recognition means are not shown in FIG. 1, they may be provided connected to the system bus 2 as hardware, and the CPU 1 performs character recognition processing and face recognition as software processing. Processing may be performed.

  The motion feature detection unit 30 not only detects the motion features as described above, but also detects, for example, the maximum motion vector from each motion vector designated on the screen or in the region designated on the screen. The feature motion vector described above is also calculated by a calculation method such as taking a value. The feature motion vector may be calculated only from the hit block HBK. For example, when the feature motion vector is generated from the motion vector of the hit block HBK, an average value of the motion vectors of the hit block HBK is obtained and used as the feature motion vector.

  If the motion feature is “moving in the horizontal direction”, “moving in the vertical direction”, “moving in the depth direction”, the feature motion vector is calculated as the average value of the motion vectors of the hit block HBK, and “ In the case of “concentrated”, the maximum value of the motion vector of the hit block HBK may be calculated as the feature motion vector.

  In this embodiment, the motion feature information and the feature motion vector information are added to the recorded still image data as motion information of the recorded still image and recorded on the recording medium by the recording / reproducing device 5. Is done.

  An example of motion information recorded as additional information is shown in FIG. As described above, the motion information can be recorded as a separate file associated with the still image information file. However, in this embodiment, the motion information is added and recorded as tag information of the image file format.

[Processing flow in still image shooting mode]
FIG. 69 is a flowchart for explaining the flow of processing in the still image shooting mode in the imaging apparatus of this embodiment. The flowchart of FIG. 69 starts from the start when the still image shooting mode is selected and started.

  The CPU 1 monitors the user's pressing of the shutter button (step S91), and when it is determined that the shutter button is not pressed, the CPU 1 is set for the motion detection / motion compensation unit 16 and the image superimposing unit 17. The block matching method generates an EE image at the time of EE monitoring before shooting a still image, and controls the motion detection / compensation unit 16 to perform motion detection (step S92).

  The motion feature detection unit 30 stores and holds the motion vector for each block detected by the motion detection / compensation unit 16 (step S93). Then, the CPU 1 controls to display on the monitor display 6 an EE image at the time of EE monitoring before still image shooting (step S94). And it returns to step S91 and repeats the process of step S92-step S94 until a shutter button is pressed down.

  If it is determined in step S91 that the shutter button has been pressed, the CPU 1 captures a plurality of captured images from the image sensor 11 at a high frame rate, and at the same time with respect to the motion detection / motion compensation unit 16 and the image overlay unit 17. Then, hierarchical matching is performed, an instruction is given to perform still image NR processing, and control is performed to execute still image NR processing (step S95).

  When the still image NR process is completed, the CPU 1 reads the still image NR image from the image memory unit 4 and controls the JPEG encoding process in the still image codec 18 (step S96). Then, the CPU 1 controls the motion feature detection unit 30 so that the motion feature and the feature motion vector of the still image to be recorded are stored from the motion vectors for each block for a plurality of frames before the shutter button is pressed. Calculation is performed (step S97).

  Next, the CPU 1 controls to record the image data of the still image NR image that has been JPEG-encoded by the still image codec 18 on the recording medium of the recording / reproducing apparatus unit 5 and also records the still image NR image to be recorded. Control is performed so that motion information composed of the motion feature information and the feature motion vector information generated in step S97 is added to the image data (step S98). Thus, the still image recording process is completed.

  Next, the flow of motion feature detection processing in step S97 will be described with reference to the flowchart of FIG. The motion feature detection unit 30 receives the control from the CPU 1 and starts the processing of the flowchart shown in FIG. 70 immediately after the shutter button is pressed.

  First, the motion feature detection unit 30 reads out block-by-block motion vectors from the buffer for a plurality of recent frames, for example, three frames before the shutter button is pressed (step S101). Next, each size of the read motion vector for each block is compared with a predetermined threshold value, and a block having a motion vector for each block larger than the threshold value is detected as a hit block (step S102).

  Next, motion analysis is performed using each block BK and / or hit block HBK designated on the screen or in the region designated on the screen, and the types of motion features as described above are detected (steps). S103).

  Next, a feature motion vector is calculated using each block BK and / or hit block HBK designated on the screen or in the area designated on the screen (step S104).

  Then, the motion feature information detected in step S103 and the feature motion vector information are output as motion information (step S105). Thus, the motion information detection processing routine ends.

[Special effects processing during playback using motion features; slideshow]
Next, an example of special effect processing in a slide show will be described as an example of playback processing using motion feature information in motion information in which recording still images are added to a slot. An example to be described first is a case where the present invention is applied to a playback still image switching method as one of special effect processing in a slide show.

  There are various methods for switching the playback still image during the slide show. Among these, especially for those that divide and switch an image in an arbitrary direction, disregarding the motion characteristics of the image may give the viewer a sense of incongruity.

  For example, what is shown in FIG. 71 is an image switching method called blind. When the still image XA is to be switched to the next still image XB and displayed, the still image XA and the still image XB are divided into a plurality of vertical strips or horizontal strip blocks, respectively. In the divided strip unit, the still image XA is gradually switched to the still image XB. The example of FIG. 71 shows a case where the switching is performed by dividing into vertical strip blocks.

  FIG. 72 shows an image switching method called slide-in. In this case, when the still image XA is to be switched to the next still image XB and displayed, the still image XA is shifted and switched for each screen while being gradually hidden by the next still image XB. The example of FIG. 72 is a case where the still image XA is switched to the next still image XB by shifting the still image XB to the left for each screen while gradually hiding it from the right side in the horizontal direction.

  When performing the blind or slide-in switching process, if the motion characteristics of the still image are different from the division direction or the shift direction, there is a risk of unnatural screen switching.

  Therefore, in the slide show in this embodiment, when performing the blind or slide-in switching process, the motion feature information added to the still image data is referred to, and the motion of the entire still image (horizontal, vertical, depth, or In consideration of such combinations), the blind splitting direction and switching direction, the slide-in shift direction, and the like are determined.

  That is, in this embodiment, when there is a slide show playback instruction, the recording / playback apparatus unit 5 sequentially reads out the image data of a plurality of recorded still images and supplies them to the slide show generation unit 32. At this time, however, the additional information processing unit 31 separates the motion information added to the image data of the still image, and the separated motion information is also supplied to the slide show generation unit 32.

  The slide show generation unit 32 recognizes the motion feature information in the received motion information, determines the motion of the entire still image in the horizontal direction, the vertical direction, the depth direction, and the like. In performing the switching process, the blind division direction and switching direction, the slide-in shift direction, and the like are determined, and an image at the time of switching is generated.

  The image data including the image at the time of switching generated by the slide show generation unit 32 is supplied to the monitor display 6 through the NTSC encoder 20 to display a playback still image, and a slide show is played back for the still image.

  The above explanation is based on the case where special effect display processing is performed when switching the playback still image during the slide show using the motion feature, but the motion feature information is also used for the composite display of the still image during the slide show. Can be applied.

  That is, at the time of a slide show, it is possible not only to switch the screen for the still images before and after switching, but also to temporarily display two images at the same time, instead of switching the above-mentioned switching images, thereby producing various effects. .

  For example, when only slideless images are displayed during a slide show playback display, the viewer gets bored. In addition, even when character information such as a signboard or a score table is taken as a still image instead of a memo, it is difficult for viewers to be impressed. Therefore, during slide show playback display, referring to the motion feature information added to the still image data, still images that have moved before shooting can be combined, or there can be movement before shooting. It is conceivable that the still image of the subject and the still image of the subject that does not originally move can be effectively combined to increase the value of the still image and entertain the viewer.

  When combining still images of subjects that have moved before shooting, an example as shown in FIG. 73 can be considered. That is, in the example of FIG. 73, a still image PA1 (FIG. 73A) to which movement feature information indicating that it is moving horizontally in the same direction and a still image PB1 (FIG. 73B) are added. When the image is continuous, before the still image PB2 is reproduced and displayed following the still image PA1, the still image PB1 is displayed on the still image PA1 by moving it in the moving direction (FIG. 73C). )) Is displayed. In this case, as shown in FIG. 73, in the synthesized image PC1, the still image PB1 to be moved and displayed on the still image PA1 is preferably moved as a reduced image or a small image obtained by cutting out a part thereof.

  As described above, the composite image PC1 which is the display example of FIG. 73 obtains an effect that the still image PB1 crosses the still image PA1. In this case, the viewer of the slide show looks as if the composite image PC1 has overtaken the still image PB1 by the still image PA1.

  In the example of FIG. 73, the moving speed of the still image PA1 is larger in consideration of the size of the feature motion vector in the motion information added to the still image PA1 and the still image PB1 (feature motion If the vector size is large), the still image PB2 may be moved from the left to the right with respect to the still image PB1.

  Similarly, even when the still image PB1 and the still image PB2 are moving in opposite directions, a composite image similar to the example of FIG. 73 can be displayed. When the moving direction is the vertical direction, the still image PB1 and the still image PB2 may be combined and displayed so as to cross in the vertical direction (up and down direction of the screen).

  Also, if the still image PB2 after switching is a still image of a subject that has moved in the depth direction, the still image PB2 is gradually increased in the depth direction when judged from the added feature information. The composite image PC1 combined with the still image PA1 can also be displayed. In this way, by using the motion information recorded in association with the recorded still image, it is possible to easily produce a display effect associated with motion in the slide show.

  Next, in a case where a still image of a subject that has moved and a still image of a subject that has not moved originally are combined, an example as shown in FIG. 74 can be considered. That is, when a character information image, which is a subject that does not move, is photographed, an image of a subject that has some other related motion is often photographed. Therefore, the example of FIG. 74 is to combine these two still images and display them simultaneously.

  In the composite image PC2 (FIG. 74C) in the example of FIG. 74, the motion of the recorded still image PA2 (FIG. 74A) whose motion feature information added is “concentration of motion within a certain range”. A still image PB2 (FIG. 74B), which is a character information image, is superimposed on a portion where the image information is not concentrated, and the feature information of the added image is “no motion”. In this embodiment, when the motion feature information is “concentration of motion within a certain range”, information indicating in which region of one still image the motion is concentrated is also added. It is possible to easily superimpose other images that do not move, such as character images, avoiding the subject.

  Next, when combining images that do not move together, an example as shown in FIG. 75 can be considered. In the example of FIG. 75, the feature information of the added image is “no motion” and the feature information of the added image is the same as the still image PA3 (FIG. 75A) that is a face recognition image. It is “no motion”, and other images other than the character recognition image and the face recognition image, for example, a landscape image PB3 (FIG. 75B) are associated with each other and simultaneously displayed as a composite image PC3 (FIG. 75C). Is. In this case, as shown in FIG. 75C, the face recognition image is preferably generated as a small image so as to be superimposed on the landscape image to generate a composite image PC3.

  As described above, when shooting a person or landscape, the subject is often shot in a stationary state. In this embodiment, “no motion” is added to the still images as motion feature information. Therefore, it can be seen from this motion feature information that there is no motion in the recorded still image, and it is possible to easily display a person image and a landscape image simultaneously.

[Flow chart of playback processing during slide show]
FIG. 76 is a flowchart for explaining the flow of playback processing during a slide show using a recorded still image to which motion information is added.

  First, the CPU 1 reads out the first still image data (with motion information) of a plurality of recorded still image information to be subjected to a slide show from the recording medium to the recording / reproducing apparatus 5 (step S111), and additional information. The data is sent to the processing unit 31.

  For example, the additional information processing unit 31 extracts motion information from the read still image data from the tag information of the still image file format, and sends the extracted motion information to the slide show generation unit 32. The slide show generation unit 32 extracts and analyzes the motion feature from the motion information, and recognizes the motion feature (step S112). At this time, the slide show generating unit 32 also detects the feature motion vector, and detects the motion direction and the magnitude of the motion from the feature motion vector.

  The additional information processing unit 31 sends the still image data itself to the still image codec unit 18 to reproduce and decode the recorded still image data (step S113).

  Next, the CPU 1 determines whether or not the reproduced still image is the first slide show (step S114). If it is determined that the still image is the first image, the still image decoded by the still image codec unit 18 is determined. The slide show generator 32 is instructed to output the data to the NTSC encoder 20 as it is, and the first still image is displayed on the display 6 (step S116).

  If it is determined in step S114 that the decoded still image is not the first slide show, the CPU 1 causes the slide show generation unit 32 to perform the switching special effect process using the motion feature and the feature motion vector described above. Control is performed to execute the image composition processing at the time of switching. The slide show generation unit 32 uses the motion feature and feature motion vector of the previous still image (currently being displayed) and the motion feature and feature motion vector of the current still image (still image to be displayed next) as described above. The processing considered to be appropriate is selected from the switching special effect processing and the image composition processing at the time of switching, and the selection processing is executed (step S115).

  Then, the slide show generation unit 32 supplies the image data of the processing result to the display 6 through the NTSC encoder 20 so as to reproduce and display the image (step S116). Then, after the image reproduction of the switching special effect processing and the image composition processing at the time of switching is continued for a predetermined time, only the image data of the current still image is supplied to the display 6 through the NTSC encoder 20 to reproduce the image. It is made to display (step S116).

  Note that the slide show generation unit 32 considers only the motion information about the current still image without considering the motion information about the previous still image in the switching special effect processing and the image composition processing at the time of switching. The switching special effect processing and the image composition processing at the time of switching may be selected.

  After step S116, the process proceeds to step S117, where the slide show end conditions, for example, all the still images to be subjected to the slide show are read, and the end conditions such as an instruction to stop by the user are given. When it is determined that the end condition is satisfied, the slide show processing routine is terminated. When it is determined that the end condition is not yet satisfied, the recording / playback device unit 5 receives the next still image data. (With motion information) is read out (step S118), the process returns to step S112, and the processes after step S112 are repeated.

[Effect of the embodiment]
As described above, according to the imaging device of the above-described embodiment, a screen switching method at the time of a still image slide show by adding motion information including a motion feature and a feature motion vector of an image to a still image, and recording it. New effects can be easily created for image composition methods and the like.

  Further, according to the imaging apparatus of the embodiment, the data format of the memory access to the image memory unit 4 is determined for each block of the most efficient data transfer size, and the image is determined according to the still image NR process and the moving image NR process. By selecting a data format for memory access to the memory unit 4, the bus bandwidth can be reduced and the bus access efficiency can be improved.

  Further, according to the above-described embodiment, the image correction / resolution conversion unit 15 performs memory access in a format in which an image is divided into strips, and the unit of compression encoding in the still image codec unit 18 is also a strip format. By doing so, data can be directly transferred from the image correction / resolution conversion unit 15 to the still image codec unit 18 without using a memory, and high-speed continuous shot images can be efficiently compression-encoded and stored in the image memory unit 4. Can be stored.

  Further, when a compression-encoded image stored in the image memory unit 4 is used as a reference image, only the range necessary for block matching processing is decoded by decompression decoding in units of strips obtained by dividing the image. The work area of the image memory unit 4 can be saved. Furthermore, the size of the decoded reference image on the work area is always kept constant by decoding and overwriting the reference image strip to be used next at the address of the unnecessary reference image strip on the image memory unit 4. can do.

[Other Embodiments and Modifications]
In the above-described embodiment, hierarchized matching consisting of basal plane matching processing and reduced surface matching processing is used. Therefore, block matching processing for detecting motion features and feature motion vectors at the time of EE is classified into hierarchical matching and reduction. It is possible to select from the surface matching and the basal plane matching. Of course, one of the hierarchical matching, the reduced surface matching, and the basal plane matching may be fixedly determined in advance. For example, since it is only necessary to capture the motion feature before recording a still image, only the reduced surface matching process is sufficient.

  In this embodiment, the use example of the motion information to be recorded in association with the recorded still image is a slide show. However, it goes without saying that the use example of the motion information is not limited to the slide show. For example, when a recorded still image is played back alone, the types of motion features described above are displayed using text characters, symbols, graphics, etc., and feature motion vectors are also displayed as symbols, graphics, text characters (horizontal and vertical). It is also possible to display a direction such as a direction or a size such as large or small).

  Further, the JPEG method is used as the compression encoding method in the still image codec unit 18 and the MPEG method is used as the compression encoding method in the moving image codec unit 19, but this is an example. It goes without saying that the invention can be applied to any image compression coding system.

  In the above-described embodiment, the compression encoding processing unit for compressing and encoding a recorded image at the time of still image shooting performs compression encoding of the captured image at the time of block matching. Different compression methods may be used for the compression method and the compression method for the captured image at the time of block matching.

  In the above-described embodiment, a plurality of memory access data formats for the image memory unit 4 are used in the moving image NR processing. However, the moving image NR processing may be used for still image NR processing. In particular, when the still image NR process is performed by the addition method during shooting, the processing speed is also important. Therefore, it is better to adopt the method for reducing the bus bandwidth as described above.

  In addition, the image is divided into a plurality of pixels in the horizontal direction and memory access is performed as strip data. This is because the read / write direction of image data from the image memory is based on the horizontal line direction. Because it was. When the image data is read from and written to the image memory in the vertical direction, the same effect as described above can be obtained by accessing the data as band-shaped data obtained by dividing the image into a plurality of lines in the vertical direction. It goes without saying that can be obtained. In that case, for example, the 64 × 1 format is a format of 64 lines × 1 pixel in the vertical direction.

  In the first embodiment described above, in the first example of the motion vector calculation unit, the search direction in the search range is set to the horizontal line direction, and the search is performed by moving the reference block in order from the upper left of the search range, for example. In addition, the memory for one line of the SAD table is provided, but the search direction in the search range is taken in the vertical direction, for example, the search is started in the vertical direction from the upper left end of the search range, and the vertical direction After the search for one column is completed, the position of the reference block is moved in the horizontal direction by one, for example, one pixel to the right in the vertical column, and the search is performed in the vertical direction from the upper end of the column. You may make it employ | adopt the search method which repeats. As described above, when the search is performed by moving the reference block in the vertical direction sequentially from the upper left end of the search range, a memory for one column in the vertical direction of the SAD table may be provided.

  Here, whether the search direction is taken in the horizontal direction or the search direction in the vertical direction is determined by taking into account the circuit scales of the matching processing unit and the motion vector calculation unit and adopting the one with a smaller circuit scale. preferable.

  As described above, the reference block may be moved not for each pixel and for each line, but for each of a plurality of pixels and for each of a plurality of lines. Therefore, the memory for one line in the horizontal direction in the former case only needs to correspond to the movement position of the reference block in the horizontal direction, and the memory for one column in the vertical direction in the latter case moves the reference block in the vertical direction. Just the position. That is, when the reference block is moved for each pixel or for each line, the memory for one line needs a memory having the capacity corresponding to the number of pixels for one line, and for one column in the vertical direction. As the memory, a memory having a capacity corresponding to the number of lines is required. However, when the reference block is moved for each of a plurality of pixels and for each of a plurality of lines, the memory for one line and the memory for one column are compared with the case where the reference block is moved for each pixel and for each line. Less capacity.

  Further, the interpolation processing method is not limited to the above-described quadratic curve approximation interpolation processing, and as described above, interpolation processing using a cubic curve or a higher order curve may be performed.

  Moreover, although the above-mentioned embodiment is a case where the image processing apparatus by this invention is applied to an imaging device, this invention is not necessarily restricted to an imaging device, It can apply to various image processing apparatuses.

  The above-described embodiment is a case where the present invention is applied to the case where a still image subjected to noise reduction processing by image superposition is recorded using a block matching method. Even when a still image is recorded in one screen unit without using a plurality of overlapping images, motion detection is performed before the shutter button is pressed, and the motion information is associated with the recorded still image data. It can also be applied to recording.

  In the above-described embodiment, the display that displays the playback image of the slide show is a display included in the imaging device. However, the imaging device of the embodiment separately outputs the output image signal of the slide show via the image output terminal. Of course, it is possible to display the image on the screen.

It is a block diagram which shows the structural example of one Embodiment of the imaging device by this invention. It is a figure used in order to explain block matching processing. It is a figure used in order to explain block matching processing. It is a figure used in order to explain block matching processing. It is a figure used in order to explain block matching processing. It is a figure used in order to explain block matching processing. It is a figure used in order to explain block matching processing. It is a figure for demonstrating the noise reduction process of the captured image in the imaging device of FIG. 1 of embodiment. It is a figure used for description of the block matching process in the imaging device of FIG. 1 of embodiment. It is a figure for demonstrating the noise reduction process of the captured image in the imaging device of FIG. 1 of embodiment. It is a figure for demonstrating the noise reduction process of the captured image in the imaging device of FIG. 1 of embodiment. It is a figure used in order to demonstrate the image processing method in embodiment of this invention. It is a figure used in order to demonstrate the image processing method in embodiment of this invention. It is a figure used in order to demonstrate the image processing method in embodiment of this invention. It is a figure used in order to demonstrate the operation | movement in the motion detection and the motion compensation part in the imaging device of FIG. 1 of embodiment. It is a figure used in order to demonstrate the operation | movement in the motion detection and the motion compensation part in the imaging device of FIG. 1 of embodiment. It is a figure used in order to demonstrate the operation | movement in the motion detection and the motion compensation part in the imaging device of FIG. 1 of embodiment. It is a figure for demonstrating the effect by the process in the motion detection and the motion compensation part in the imaging device of FIG. 1 of embodiment. It is a block diagram which shows the structural example of the motion detection and the motion compensation part in the imaging device of FIG. 1 of embodiment. FIG. 20 is a block diagram of a detailed configuration example of a part of the motion detection / compensation unit in FIG. 19. FIG. 20 is a block diagram of a detailed configuration example of a part of the motion detection / compensation unit in FIG. 19. It is a block diagram which shows the structural example of the image superimposition part in the imaging device of FIG. 1 of embodiment. It is a figure which shows a part of flowchart for demonstrating embodiment of this invention. It is a figure which shows a part of flowchart for demonstrating embodiment of this invention. It is a figure which shows a part of flowchart for demonstrating embodiment of this invention. It is a figure used in order to demonstrate the 1st example of the motion vector calculation part of the motion detection / motion compensation part in the imaging device of an embodiment. It is a figure used in order to demonstrate the 1st example of the motion vector calculation part of the motion detection / motion compensation part in the imaging device of an embodiment. It is a block diagram which shows the structure of the 1st example of the motion vector calculation part of the motion detection / motion compensation part in the imaging device of embodiment. It is a figure which shows the flowchart for demonstrating the processing operation example of the motion vector calculation part of FIG. It is a figure which shows the flowchart for demonstrating the processing operation example of the motion vector calculation part of FIG. It is a figure which shows the flowchart for demonstrating the processing operation example of the motion vector calculation part of FIG. It is a block diagram which shows the structure of the 2nd example of the motion vector calculation part of the motion detection and the motion compensation part in the imaging device of embodiment. FIG. 33 is a diagram illustrating a flowchart for explaining a processing operation example of a motion vector calculation unit in FIG. 32. FIG. 33 is a diagram illustrating a flowchart for explaining a processing operation example of a motion vector calculation unit in FIG. 32. FIG. 33 is a diagram illustrating a flowchart for explaining a processing operation example of a motion vector calculation unit in FIG. 32. It is a figure used in order to demonstrate the specific example of the hierarchical block matching process in embodiment of this invention. It is a figure used in order to demonstrate the specific example of the hierarchical block matching process in embodiment of this invention. It is a figure used in order to demonstrate the specific example of the hierarchical block matching process in embodiment of this invention. It is a figure used in order to demonstrate the specific example of the hierarchical block matching process in embodiment of this invention. It is a figure for demonstrating the strip access format used by embodiment of this invention. It is a figure for demonstrating the memory access system of the image data used by embodiment of this invention. It is a figure for demonstrating the example of the process unit of the image used by embodiment of this invention. It is a figure for demonstrating the flow of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the flow of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the flow of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the flow of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the flow of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the flow of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the flow of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the memory access operation | movement of the image data in the imaging device of embodiment of this invention. It is a figure for demonstrating the flow of the image data in the imaging device of embodiment of this invention. It is a figure which shows the timing chart for demonstrating the operation example of the principal part in the imaging device of embodiment of this invention. It is a figure used in order to demonstrate the operation example of the principal part of the imaging device of embodiment of this invention. It is a figure used in order to demonstrate the operation example of the principal part of the imaging device of embodiment of this invention. It is a figure for demonstrating an example of the movement information linked and recorded in the imaging device of embodiment of this invention. It is a figure which shows the flowchart for demonstrating the operation example of the principal part in the imaging device of embodiment of this invention. It is a figure which shows the flowchart for demonstrating the operation example of the principal part in the imaging device of embodiment of this invention. It is a figure for demonstrating the operation example of the principal part in the imaging device of embodiment of this invention. It is a figure for demonstrating the operation example of the principal part in the imaging device of embodiment of this invention. It is a figure for demonstrating the operation example of the principal part in the imaging device of embodiment of this invention. It is a figure for demonstrating the operation example of the principal part in the imaging device of embodiment of this invention. It is a figure for demonstrating the operation example of the principal part in the imaging device of embodiment of this invention. It is a figure which shows the flowchart for demonstrating the operation example of the principal part in the imaging device of embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 4 ... Image memory part, 8 ... Memory controller, 16 ... Motion detection / motion compensation part, 17 ... Image superposition part, 18 ... Still image codec part, 30 ... Motion feature detection part, 31 ... Additional information processing part, 32 ... Slideshow generating unit, 100 ... target image (target frame), 101 ... reference image (reference frame), 102 ... target block, 104 ... motion vector 106 ... search range, 107 ... reference vector, 108 ... reference block, 163 ... matching process , 164... Motion vector calculation unit

Claims (13)

An image sensor;
Motion feature detection means for detecting a motion feature of the subject from imaging data of the subject from the imaging device before a user's recording instruction operation;
Based on the recording instruction operation of the user, the image data of the still image of the subject at the time of the recording instruction operation and the information on the motion feature of the subject detected by the motion feature detection unit are associated with each other on a recording medium. Recording means for recording;
An imaging apparatus comprising: The imaging device according to claim 1,
A display for display;
Before the user's recording instruction operation, monitor display image data for generating the monitor display image data of the subject to be displayed on the display for display from the imaging data of the subject from the imaging device and supplying the monitor display image data to the display for display Generating means;
With
The imaging apparatus according to claim 1, wherein the motion feature detecting means detects a motion feature of the subject from imaging data of the subject from the imaging device when generating monitor display image data of the subject. The imaging device according to claim 1,
By detecting motion vectors between images for images for a plurality of screens from the user's recording instruction operation time point and superimposing the images for the plurality of screens while compensating for motion based on the detected motion vectors. Obtaining a still image with reduced noise and recording the image data of the still image with reduced noise on the recording medium as image data of the still image of the subject at the time of the recording instruction operation,
Motion detection / motion compensation means for detecting a motion vector between one of the two screens and the other of the two screens, and generating a motion compensated image in which one of the two screens is motion-compensated based on the detected motion vector When,
Image superimposing means for superimposing the motion compensated image from the motion detection / compensation means on the other of the two screens;
With
The imaging apparatus characterized in that the motion feature detection means detects a motion feature of the subject from the motion vector detected by the motion detection / compensation means. The imaging device according to claim 3.
A display for display;
Before the user's recording instruction operation, monitor display image data for generating the monitor display image data of the subject to be displayed on the display for display from the imaging data of the subject from the imaging device and supplying the monitor display image data to the display for display Generating means;
With
The motion detection / compensation means detects the motion vector from the imaging data of the subject from the imaging device when generating the monitor display image data of the subject, and the motion feature detection means detects the detected An imaging apparatus, wherein a motion feature of the subject is detected from a motion vector. The imaging device according to claim 3.
The motion detection / compensation means detects the motion vector in blocks obtained by dividing one screen into a plurality of blocks,
The imaging apparatus, wherein the motion feature detection unit detects the motion feature using a plurality of block-unit motion vectors for the one screen detected by the motion detection / compensation unit. The imaging apparatus according to claim 4,
When generating the monitor display image data of the subject in the monitor display image data generation means before the user's recording instruction operation, the imaging data of the subject from the imaging element is set to a standard television frame rate,
A plurality of screen images from the user's recording instruction operation time point are output from the image sensor at a frame rate higher than a frame rate of the standard television system. . An imaging recording method by an imaging device comprising a movement feature detection means and a recording means,
A motion feature detection step in which the motion feature detection means detects the motion feature of the subject from imaging data of the subject from the imaging device before the user's recording instruction operation;
Based on the recording instruction operation of the user, the recording means includes image data of the still image of the subject at the time of the recording instruction operation and information on the movement feature of the subject detected in the motion feature detection step. A recording step of recording on the recording medium in association with each other;
The imaging recording method characterized by having. An image sensor;
Motion feature detection means for detecting a motion feature of the subject from imaging data of the subject from the imaging device before a user's recording instruction operation;
Based on the recording instruction operation of the user, the image data of the still image of the subject at the time of the recording instruction operation and the information on the motion feature of the subject detected by the motion feature detection unit are associated with each other on a recording medium. Recording means for recording;
Reading means for reading out the image data of the still image of the subject and the information on the movement characteristics of the subject from the recording medium;
Recognizing means for extracting information on the movement characteristics of the subject from the data read by the reading means and recognizing the movement characteristics of the subject;
When outputting display image data based on the image data of the still image read by the reading unit, display effect processing is performed on the display image according to the motion characteristics of the subject recognized by the recognition unit. Display effect processing means to
A captured image recording / reproducing apparatus comprising: The captured image recording / reproducing apparatus according to claim 8,
The image data of a plurality of still images recorded on the recording medium is sequentially read out to generate display image data, and each display image data is sequentially transmitted to the display for display at a predetermined time interval. It has a slide show function to output,
When the slide show function is activated, the display effect processing means performs display effect processing at the time of switching the display image of the still image in accordance with the movement feature. The captured image recording / reproducing apparatus according to claim 8,
The image data of a plurality of still images recorded on the recording medium is sequentially read out to generate display image data, and each display image data is sequentially transmitted to the display for display at a predetermined time interval. It has a slide show function to output,
When the slide show function is activated, the display effect processing means performs display effect processing for combining and displaying a plurality of images according to the motion characteristics. apparatus. A captured image recording / reproducing method by a captured image recording / reproducing apparatus comprising a movement feature detecting means, a recording means, a reading means, a recognition means, and a display effect processing means,
A motion feature detection step in which the motion feature detection means detects the motion feature of the subject from imaging data of the subject from the imaging device before the user's recording instruction operation;
Based on the recording instruction operation of the user, the recording means includes image data of the still image of the subject at the time of the recording instruction operation and information on the movement feature of the subject detected in the movement feature detection step. A recording process for recording in association with the recording medium;
A reading step in which the reading unit reads image data of the still image of the subject and information on the movement characteristics of the subject from the recording medium;
A step of recognizing the movement feature of the subject by extracting information on the movement feature of the subject from the data read by the reading unit;
When the display effect processing means outputs display image data based on the image data of the still image read by the reading means, the display effect according to the motion characteristics of the subject recognized by the recognition means A display effect processing step for applying processing to the display image;
A captured image recording / reproducing method comprising: The captured image recording / reproducing method according to claim 11,
The captured image recording / reproducing apparatus sequentially reads out the image data of a plurality of still images recorded on the recording medium to generate display image data, and the display image data at predetermined time intervals, Equipped with a slide show function that sequentially outputs to the display for display,
In the display effect processing step, when the slide show function is activated, display effect processing at the time of switching the display image of the still image is performed in accordance with the motion characteristics. The captured image recording / reproducing method according to claim 11,
The captured image recording / reproducing apparatus sequentially reads out the image data of a plurality of still images recorded on the recording medium to generate display image data, and the display image data at predetermined time intervals, Equipped with a slide show function that sequentially outputs to the display for display,
In the display effect processing step, when the slide show function is activated, display effect processing is performed in which a plurality of images are combined and displayed according to the motion characteristics. Method.

Images