JP2008301161A - Image processing device, digital camera, and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To interpolate a motion vector that cannot be detected due to moving of an object in perspective direction, in a simple process. <P>SOLUTION: When video recording is started, a face extracting circuit starts an extraction process of a face and stops the process when the face is extracted. A motion detector sequentially detects motion vectors. A motion compensator utilizes the motion vector to track the extracted face. If a face expands/contracts as it moves in such direction as imaging distance fluctuates, a macro block which works as a face cannot be recognized between frames, nor its motion vector is detected. The motion detector stops detection of the motion vector for that frame and the motion compensator stops the process for tracking the face. The face detecting circuit resumes the process for extracting the face, and stops the process when the face is extracted. The motion detector resumes detecting of the motion vector, and the motion compensator resumes the process of tracking the face. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, a digital camera, and an image processing method for interpolating motion vectors.

  2. Description of the Related Art Conventionally, some photographing apparatuses such as digital cameras have a camera shake correction function that reduces blurring of a photograph due to camera shake or image shaking. There are several types of camera shake correction functions, one of which is an electronic camera shake correction function (see, for example, Patent Document 1). The electronic camera shake correction function detects a motion vector by comparing captured images between frames, and corrects camera shake by adjusting the image position using the motion vector.

One of the motion vector detection methods is a pattern matching method. The pattern matching method searches for a pixel array corresponding to the same subject between frames and detects the spatial distance and direction of the corresponding pixel as a motion vector.
JP 2006-287814 A

  However, since the subject moving in the perspective direction is enlarged / reduced in the image, the pattern matching method may not be able to identify the pixel arrangement corresponding to the same subject between frames, and the motion vector may not be detected.

  The present invention has been made in view of the above problems, and an image processing apparatus, a digital camera, and an image processing method for interpolating a motion vector that can no longer be detected due to a subject moving in a perspective direction are provided at low cost. It is intended to provide to.

  In order to achieve the above object, an image processing apparatus according to the present invention identifies a subject extraction unit that extracts a specific subject from an image acquired by moving image shooting, and a pixel corresponding to the same subject between frames, and then identifies the subject. A motion detection unit that detects a motion vector; and a tracking unit that follows a subject that moves between frames using the motion vector detected by the motion detection unit. When the corresponding pixel is not identified between the two and the motion vector cannot be detected, the subject extracting unit extracts the specific subject again and interpolates the motion vector.

  When the corresponding pixel between the frames is no longer identified for the subject extracted by the subject extraction unit, the motion detection unit temporarily stops detecting the motion vector for the image of the frame, and the motion vector is extracted from the image of the next frame. It is preferable to restart the detection.

  The subject extracting means is preferably a face extracting means for extracting a person's face.

  In addition, it is preferable to compare images between frames by pattern matching.

  A digital camera of the present invention includes the image processing apparatus.

  Note that it is preferable to have a camera shake correction function using a motion vector of a pixel that has not been extracted as a specific subject.

  The image processing method of the present invention detects a motion vector of a specific subject extracted from an image acquired by moving image shooting, identifies corresponding pixels between frames, and uses the detected motion vector. An image processing method for following an object moving between frames, and for a specific object, if a corresponding pixel is not identified between frames and a motion vector cannot be detected, the specific object is extracted again and the motion vector is extracted. Is interpolated.

  Note that when a pixel corresponding to a specific subject is no longer identified between frames, it is preferable to stop the motion vector detection for the image of the frame and restart the motion vector detection from the image of the next frame.

  It is also preferable to extract a person's face as a specific subject.

  In addition, it is preferable to compare images between frames by pattern matching.

  According to the image processing apparatus, the digital camera, and the image processing method of the present invention, when the motion vector cannot be detected because the corresponding pixels are not identified between frames with respect to the subject extracted by the subject extraction unit, the subject detection unit By extracting a specific subject again, a motion vector that can no longer be detected due to the subject moving in the perspective direction can be interpolated at a low cost.

  In addition, regarding the subject extracted by the subject extraction unit, when the corresponding pixel is no longer identified between frames, the motion detection unit temporarily stops detecting the motion vector for the image of the frame, and the motion vector is extracted from the image of the next frame. Therefore, unnecessary calculation processing can be omitted.

  Hereinafter, the digital camera 21 of the present invention will be described with reference to the drawings. As shown in FIG. 1, a lens barrel 24 incorporating a photographing lens 23 and a strobe light emitting unit 25 are provided on the front surface of the camera body 22 of the digital camera 21. The photographing lens 23 guides light into the camera body 22 and enables photographing. The strobe light emitting unit 25 emits light in synchronization with shooting and adjusts the exposure amount.

  On the upper surface of the camera body 22, a power button 26 for switching the power state of the digital camera 21, a shutter button 27 used for a shutter release operation, and various operation units for a mode switching dial 28 for switching various modes are provided. The modes include a still image shooting mode for recording still images, a moving image shooting mode for recording moving images, a playback mode for reproducing recorded images, and a menu mode for manually changing setting values.

  The shutter button 27 is a two-stage push button, and when a fixed stroke is pressed (half-pressed) in the still image shooting mode, various shooting preparation processes such as automatic focus adjustment (AF) and automatic exposure adjustment (AE). Is executed. When a certain stroke is further pressed (fully pressed) from this half-pressed state, an instruction to execute shooting is issued to the digital camera 21, and the imaging signal for one screen subjected to the shooting preparation process is converted into image data. . In addition, when a certain stroke is pressed in the movie shooting mode, movie shooting starts, and when a certain stroke is pressed again, movie shooting ends.

  The lens barrel 24 is of a retractable type and is housed in a lens barrel housing portion 29 provided on the front surface of the camera body 22 when the digital camera 21 is in a power-off state. 22 protrudes from the front surface and stops at the wide-angle end.

  In addition, as shown in FIG. 2, operation units such as a zoom button 32, a cross key 33, and an enter button 34 are provided on the rear surface of the camera body 22 in addition to a liquid crystal display (LCD) 31. Various settings screens are displayed on the LCD 31 in the menu mode, and through images are displayed in the shooting standby state in the still image shooting mode and the moving image shooting mode. The user performs shooting by pressing the shutter button 27 while viewing the through image displayed on the LCD 31. In addition, the moving image is displayed in real time in the shooting state in the moving image shooting mode.

  As shown in FIG. 3, the digital camera 21 (see FIGS. 1 and 2) is provided with a zoom lens 41, an aperture 42, a focus lens 43, and a CCD image sensor 44. The zoom lens 41 is zoomed by driving the zoom lens motor 45. The aperture 42 changes its aperture by driving the iris motor 46. The focus lens 43 adjusts the focus by driving the focusing lens motor 47. The driving of these motors 45 to 47 is controlled via a motor driver 52 connected to the CPU 51. The above-described photographing lens 23 (see FIG. 1) includes a zoom lens 41, a diaphragm 42, and a focus lens 43.

  The CCD image sensor 44 captures a subject image formed by the photographing lens 23. A timing generator (TG) 53 controlled by the CPU 51 is connected to the CCD image sensor 44, and the shutter speed of the electronic shutter is determined by a timing signal (clock pulse) input from the TG 53.

  An imaging signal obtained from the CCD image sensor 44 is input to a CDS (correlated double sampling circuit) 54 and an AMP (amplifier) 55. The CDS 54 outputs R, G, B image data that accurately corresponds to the amount of charge stored in each cell of the CCD image sensor 44, and the AMP 55 amplifies the image data. The amplified image data is converted into a digital signal by the A / D converter 56.

  The image input controller 57 is connected to the CPU 51 via the data bus 58 and controls the CCD image sensor 44, the CDS 54, the AMP 55, and the A / D converter 56 in accordance with a control command from the CPU 51. The image data output from the A / D converter 56 is output at predetermined intervals and temporarily stored in the SDRAM 59. The SDRAM 59 is a working memory.

The image signal processing circuit 60 reads the image data from the SDRAM 59, performs various image processing such as gradation conversion, white balance (WB) correction, and γ correction, and stores the image data in the SDRAM 59 again. The YC conversion processing circuit 61 reads the image data subjected to various processes by the image signal processing circuit 60 from the SDRAM 59 and converts it into a luminance signal Y and color difference signals C _r and C _b .

  The VRAM 62 is a memory for outputting a through image to the liquid crystal display (LCD) 31 and stores image data that has passed through the image signal processing circuit 60 and the YC conversion processing circuit 61. The VRAM 62 has a memory for two frames so that image data can be written and read in parallel. The image data stored in the VRAM 62 is converted into an analog composite signal by the LCD driver 63 and displayed on the LCD 31 as a through image.

  The compression / decompression processing circuit 64 compresses the image data YC converted by the YC conversion processing circuit 61 in a predetermined compression format (for example, JPEG format for still images and MPEG format for moving images). Apply. The compressed image data is recorded on a memory card 66 that is a recording medium via a media controller 65. In the reproduction mode, the image data is read from the memory card 66 to the compression / decompression processing circuit 64, and after the decompression processing is performed by the compression / decompression processing circuit 64, the recorded image is displayed on the LCD 31. The compression / decompression processing circuit 64 includes a motion detector 67 that detects a motion vector and a motion compensator 68 that performs a motion compensation process using the motion vector.

  The motion detector (motion detection means) 67 detects the motion vector by comparing the image of the Nth frame with the image of the immediately preceding (N−1) frame by pattern matching. The motion vector represents the spatial distance and direction of a macroblock (for example, 8 × 8 pixels) having the same pattern between frames.

For example, regarding the image of the (N−1) th frame shown in FIG. 4A and the image of the Nth frame shown in FIG. 4B, the nth frame from the upper end of the image frame and the mth frame from the left end. If the pixel is _{a n-1, m-1} , a 0,0 ~a 0,7 in the n _{frame, a 1,0 ~a 1,7, a 2,0} ~a 2,7, a 3, _{0 to} a _3,7 , a _{4,0 to} a _4,7 , a _{5,0 to} a _5,7 , a _{6,0 to} a _6,7 , a _{7,0 to} a _7,7 The macroblocks 101 are a _{3,4 to} a _3,11 , a _{4,4 to} a _4,11 , a _{5,4 to} a _5,11 , a _6,4 to, in the (N-1) th frame. _{a 6,11, a 7,4 ~a 7,11,} a 8,4 ~a 8,11, a 9,4 ~a 9,11, composed of pixels of _{a 10, 4 ~a 10, 11} Black blocks 102 have the same pattern as the corresponding as the same subject. That is, the motion vector 103 of the macroblock 101 in the Nth frame represents the distance and direction for four pixels in the left direction and three pixels in the upward direction.

  The motion compensator (following means) 68 includes an image of the (N−1) th frame (see FIG. 4A) and a motion vector 103 of the Nth frame detected by the motion detector 67 (FIG. 4B). (See FIG. 4B), and follows a subject that moves spatially between frames, and predicts an image of the Nth frame (see FIG. 4B) to obtain a difference image between frames.

  Here, a case where a subject moving in a direction that does not cause a change in the shooting distance is shot will be described as an example. It is assumed that the image is divided into 4 × 4 macro blocks BLK0 to BLK15. The cat 104 reflected in the macroblock BLK5 of the (N-1) th frame shown in FIG. 5A is followed by the motion vector 105 (see FIG. 5B) of the macroblock BLK5, and is shown in FIG. An image of the Nth frame is predicted, and a difference image between frames is obtained. Thus, high compression efficiency can be obtained by subtracting data that is common between frames and obtaining the difference image.

  On the other hand, an example will be described in which a subject moving in a direction that causes a change in the shooting distance is shot. It is assumed that the image is divided into 4 × 4 macro blocks BLK0 to BLK15. The human face 106 shown in the macroblock BLK5 in the (N-1) th frame shown in FIG. 6A is enlarged and shown in the macroblocks BLK6 and BLK10 in the Nth frame shown in FIG. 6B. That is, the motion detector 67 cannot recognize a corresponding macroblock as the human face 106 and cannot detect a motion vector in comparison between images by pattern matching. If the macro block corresponding between the frames cannot be recognized due to the enlargement / reduction of the subject, the motion detector 67 aborts the process of detecting the motion vector for the frame. Since the motion vector cannot be detected and the motion vector cannot be detected, the motion compensator 68 cannot predict the image of the Nth frame and cannot obtain a difference image between frames. Note that motion vectors that cannot be detected are interpolated by processing by a face extraction circuit 75 described later in detail.

  The motion vector is used for an electronic camera shake correction function. When the camera shake correction function is ON, the image formation range of the CCD image sensor 44 is moved according to the detected motion vector to prevent image blurring. At this time, the accuracy is improved by using only the motion vector regarding the background without using the motion vector regarding the moving subject. Note that the electronic camera shake correction function is well known, and a detailed description thereof is omitted here.

  In addition to the power button 26, shutter button 27, mode switching dial 28, zoom button 32, cross key 33, and enter button 34 described above, an EEPROM 69 is connected to the CPU 51. In the EEPROM 69, various control programs and setting information are recorded. The CPU 51 reads these pieces of information from the EEPROM 69 to the SDRAM 59 and executes various processes.

  The data bus 58 includes a strobe control circuit 71 that controls the light emission state of the strobe light emitting unit 25, an AF detection circuit 72 that detects whether the focus position of the zoom lens 41 is appropriate, and an electronic shutter of the CCD image sensor 44. An AE detection circuit 73 that detects whether exposure conditions such as shutter speed, shooting sensitivity, and aperture value of the aperture 42 are appropriate for shooting, and an AWB detection circuit 74 that detects whether white balance correction is appropriate for shooting. Is connected.

  Each of the detection circuits 72 to 74 sequentially transmits detection results to the CPU 51 via the data bus 58. The CPU 51 individually controls the operations of the zoom lens 41, the diaphragm 42, the focus lens 43, the CCD image sensor 44, and the like based on the detection results transmitted from the detection circuits 72 to 74.

  The AF detection circuit 72 performs focus control by a contrast detection method. First, a shooting distance to a subject such as a person's face is calculated based on one or a plurality of focus evaluation areas (AF areas) set in advance to evaluate the in-focus state of the image. Determine the focal position. In the still image shooting mode, focus control is performed when the shutter button 27 is half-pressed.

  The AE / AWB detection circuits 73 and 74 calculate appropriate exposure values and white balance correction amounts based on the luminance information of the image data stored in the VRAM 62 at predetermined intervals, and send these pieces of information to the CPU 51. The CPU 51 continuously controls the electronic shutter, the aperture 42, and the image processing based on the obtained information.

  The data bus 58 is connected to a face extraction circuit (face extraction means) 75 that extracts a human face from an image. The face extraction circuit 75 reads out the image data displayed on the LCD 31 as a through image from the VRAM 62 and extracts the person's face by extracting both eyes of the person.

  Specifically, when extracting both eyes of a person, image data is divided into a grid (for example, 16 × 16), and the skin level is determined from the signal levels of the R, G, and B color signals included in each divided region. A divided region including many estimated skin color pixels is extracted. Then, an area having white pixels of the eye part and black pixels estimated to be pupils is selected from the extracted area, and the positions of both eyes of the person on the image are obtained from the areas, and the positions of both eyes are determined. The point is defined as the face position with the representative point. The face extraction circuit 75 extracts an area including a lot of surrounding skin color pixels determined as the face position as a face area.

  As shown in FIGS. 6A and 6B, when a motion vector cannot be detected for the human face 106, the face extraction circuit 75 extracts the human face 106, thereby detecting the motion vector. Can be interpolated. By limiting the shooting target to a person's face, it is possible to follow even if the person's face moves in a direction in which the shooting distance varies.

  Hereinafter, the digital camera 21 (see FIGS. 1 and 2) of the above embodiment will be described with reference to FIGS. First, the power button 26 is operated, and the power of the digital camera 21 is set to ON. When the power is set to ON, a through image is displayed on the LCD 31 when the still image shooting mode or the moving image shooting mode is set, and the user can determine the composition while viewing the through image. Further, when the power is set to ON, the camera shake correction function is automatically set to ON, thereby preventing an image from blurring due to camera shake.

  When the moving image shooting mode is set, when the shutter button 27 is pressed to start moving image shooting, the face extraction circuit 75 is driven to start extracting a person's face. When a person's face is extracted, the processing by the face extraction circuit 75 stops. On the other hand, the motion detector 67 sequentially detects motion vectors. The motion compensator 68 follows the face of the person extracted by the face extraction circuit 75 using the motion vector. By using the motion vector of the macroblock excluding the face of the person to be followed, it is possible to correct camera shake with high accuracy. Further, by following the face of a person, a difference image obtained by subtracting data common between frames can be obtained, and high compression efficiency can be obtained.

  When a person's face is enlarged or reduced by moving in a direction that causes a change in the shooting distance, a macroblock corresponding to the person's face cannot be recognized between frames, and the motion vector cannot be detected. Since no motion vector is detected, it cannot be followed. In this case, the motion detector 67 stops detecting the motion vector for the frame, and the motion compensator 68 stops the process of following the face of the person. On the other hand, the face extraction circuit 75 resumes the process of extracting a person's face.

  When a person's face is extracted, the processing by the face extraction circuit 75 stops. Then, the motion detector 67 resumes the detection of the motion vector, and the motion compensator 68 resumes the process of following the human face. By repeating the above process until the photographing is completed, it is possible to interpolate a motion vector that can no longer be detected due to the movement of the person's face in a direction that causes a variation in the photographing distance. Thereby, even if the face of a person moves in a direction that causes a change in the shooting distance, the movement can be accurately followed.

  In the above-described embodiment, the case of the digital camera 21 has been described as an example. However, it is sufficient if the face extraction circuit 75 is provided and the motion vector can be interpolated by extracting the face when the motion vector cannot be detected. It can be applied to other image processing apparatuses.

  In the above-described embodiment, the case where the face extraction circuit 75 that extracts a person's face is provided has been described as an example. However, the present invention is not limited thereto, and the subject extraction circuit that extracts a specific subject is provided. Just do it.

  Further, in the above-described embodiment, the case where images between frames are compared by pattern matching has been described as an example. However, it is sufficient if a motion vector can be detected, and another method may be used.

  Further, in the above embodiment, the face extraction by the face extraction circuit 75 has been described by taking the case where the face area is extracted by extracting the skin color. However, it is only necessary that the face area can be extracted, for example, pattern recognition using a template. The face area may be extracted by

  In the above embodiment, when the macroblock corresponding to the person's face cannot be recognized, the motion detector 67 has been described as an example in which the detection of the motion vector is aborted for the entire frame. The motion vector detection may be aborted only for the corresponding macroblock.

  In the above-described embodiment, the case of shooting in the moving image shooting mode has been mainly described. However, it is only necessary to acquire temporally continuous frame images by moving image shooting, and a through image in the still image shooting mode is interpreted as moving image shooting. be able to.

  In the above embodiment, the case where a motion vector in units of macroblocks is detected has been described as an example. However, the present invention is not limited to this, and a motion vector in units of pixels may be detected.

  In the above-described embodiment, the example in which the motion vector is interpolated when the face of the person expands or contracts by moving in the direction in which the shooting distance varies is described. However, the present invention is not limited to this. A motion vector may be interpolated when the face of a person is enlarged or reduced by a scaling of 41.

It is an external appearance perspective view of a digital camera. It is a rear view of a digital camera. It is a block diagram which shows the electric constitution of a digital camera. It is a figure for demonstrating the detection method of a motion vector, (A) shows the image of the (N-1) th frame, (B) shows the image of the Nth frame, respectively. It is a figure which shows the example in which a motion vector is detected, (A) shows the image of the (N-1) th frame, (B) shows the image of the Nth frame, respectively. It is a figure which shows the example in which a motion vector is not detected, (A) shows the image of the (N-1) th frame, (B) shows the image of the Nth frame, respectively. It is a timing chart which shows the relationship between the operation state of a motion detector, and the size of a face. It is a flowchart which shows the process sequence of an image processing apparatus.

Explanation of symbols

21 Digital Camera 64 Compression / Expansion Processing Circuit 67 Motion Detector (Motion Detection Unit)
68 Motion compensator (Follow-up means)
75 Face extraction circuit (face extraction means)

Claims (10)

Subject extraction means for extracting a specific subject from an image acquired by video shooting;
Motion detection means for detecting a motion vector after identifying corresponding pixels as the same subject between frames;
Tracking means for following a subject moving between frames using the motion vector detected by the motion detection means;
For the subject extracted by the subject extraction means, if the corresponding pixel is not identified between frames and a motion vector cannot be detected, the subject extraction means re-extracts a specific subject and interpolates the motion vector. A featured image processing apparatus.   When a corresponding pixel is no longer identified between frames for the subject extracted by the subject extraction unit, the motion detection unit temporarily stops the motion vector detection for the image of the frame, and the motion vector is extracted from the image of the next frame. The image processing apparatus according to claim 1, wherein the detection of the image is resumed.   The image processing apparatus according to claim 1, wherein the subject extraction unit is a face extraction unit that extracts a human face.   4. The image processing apparatus according to claim 1, wherein images between frames are compared by pattern matching.   A digital camera comprising the image processing apparatus according to any one of 1 to 4 above.   6. The digital camera according to claim 5, further comprising a camera shake correction function using a motion vector of a pixel that has not been extracted as a specific subject. For a specific subject extracted from an image acquired by moving image shooting, the corresponding pixel is identified between frames and its motion vector is detected. The detected motion vector is used to follow the subject that moves between frames. An image processing method for
An image processing method characterized by reextracting a specific subject and interpolating the motion vector when a motion vector cannot be detected because a corresponding pixel is not identified between frames for the specific subject.   When a corresponding pixel is no longer identified between frames for a specific subject, motion vector detection is temporarily stopped for the image of the frame, and motion vector detection is resumed from the image of the next frame. 8. The image processing method according to 7.   9. The image processing method according to claim 7, wherein a person's face is extracted as a specific subject.   The image processing method according to claim 7, wherein images between frames are compared by pattern matching.

Images