JP2009071679A - Image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To appropriately suppress random noise with a small amount of throughput, and to suppress each of FPN and the motion between field images by camera shake. <P>SOLUTION: An image processor such as a digital movie camera 10, includes an image sensor 14. A plurality of field images following time series outputted from the image sensor are given to each of a motion detection circuit 22 and a first cutting circuit 26. The motion detection circuit 22 detects the motion of feature points among the plurality of field images. The first cutting circuit 26 performs cutting processing to each of the plurality of field images at a position based on the detected result. A 3DDNR circuit 28 performs addition processing for adding a near-field field image to each of the plurality of field images that have passed through the first cutting circuit 26. When performing the addition processing, each of a set of field images to be added mutually is divided into blocks B11-B66, and the divided result is weighted by a coefficient for each block. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus that performs image processing such as camera shake correction and noise removal on a plurality of object scene images that follow a time series captured by an image sensor, for example.

  An example of this type of device is disclosed in Patent Document 1. In this background art, random noise of each object scene image is suppressed by weighting and adding a plurality of object scene images continuous in time series to each other while shifting the position (3DDNR: Three Dimensional Digital Noise Reduction). In the weighted addition, a coefficient to be multiplied by each object scene image is changed according to the magnitude of the motion between the object scene images. The magnitude of the movement between the object scene images is determined in units of the object scene image (frame or field).

Another example is disclosed in Patent Document 2. This background art also performs 3DDNR, but the magnitude of the motion is discriminated on a pixel-by-pixel basis, and the 3DDNR is performed only on the portion of each object scene image where the motion is small.
JP 2006-197455 A [H04N 5/21, 5/232, 101/00] JP 2006-237716 A [H04N 5/21]

  However, in Patent Document 1, when a subject with a small size and a fast movement (such as a hit ball) is included in the object scene, the image of the subject portion is blurred by 3DDNR.

  In this regard, in Patent Document 2, since 3DDNR is applied only to a portion with a small motion, blurring of a portion with a large motion is avoided. However, since the magnitude of the movement is discriminated on a pixel basis, the amount of processing becomes enormous.

  In addition, when a CMOS (Complementary Metal-Oxide Semiconductor) is used as an image sensor, it is highly necessary to suppress FPN (Fixed Pattern Noise) in addition to random noise. Furthermore, it is also important to suppress not only noise but also movement between object scene images (movement component in the time axis direction) due to camera shake.

  Therefore, a main object of the present invention is to provide a novel image processing apparatus.

    Another object of the present invention is to provide an image processing apparatus that can appropriately suppress random noise with a small amount of processing, and can also suppress each movement between object scene images due to FPN and camera shake.

    Another object of the present invention is to provide an image processing apparatus that can reduce the required memory capacity and power consumption.

  The present invention employs the following configuration in order to solve the above problems. Note that reference numerals in parentheses, supplementary explanations, and the like indicate correspondence with embodiments to be described later in order to help understanding of the present invention, and do not limit the present invention.

  An image processing apparatus according to a first invention is an image processing apparatus that performs image processing on a plurality of object scene images according to a time series output from an imaging means that repeatedly captures an optical image of the object scene, Motion detection means for detecting movement of feature points between field images, first cutout means for cutting out each of a plurality of object scene images at positions based on detection results of the motion detection means, and first cutout Adding means for performing an addition process for adding one or a plurality of object scene images temporally close to the object scene image to each of the plurality of object scene images after passing through the means; A means for dividing each set of object scene images to be added to each other into a common partial image, and a weight for weighting the division results of the dividing means by a coefficient for each common partial image and adding together Including addition means.

  In the first invention, the optical image of the object scene is repeatedly captured by the image pickup means (14), and the image processing device (10, 10A) displays the images in a plurality of object scene images according to the time series output from the image pickup means. Apply processing.

  In the image processing apparatus, the motion detection means (22) detects the motion of the feature point among the plurality of object scene images, and the first cutout means (26) detects the motion for each of the plurality of object scene images. Cutout processing is performed at a position based on the detection result of the means. That is, the cutting position of the first cutting means moves according to the movement of the feature point. In this way, the movement between the plurality of object scene images is suppressed by making the cutout position follow the movement of the feature point.

  Each of the plurality of object scene images after passing through the first cutout means is given to the adding means (28, 28A), and one or more object scene images close in time to the object scene image are obtained. Addition processing to add is performed. By adding the near object scene images in this manner, random noise included in each of the plurality of object scene images is suppressed.

  In addition, FPN is also suppressed by providing the adding means after the first cutting means. This is because the movement of the cutout position causes the FPN to have the property of random noise, and such a noise component is also suppressed by the addition process.

  In addition, in the adding means, the dividing means (50a, 50b, 52a, 52b) divides each set of object scene images to be added to each other into common partial images (B11 to B66), and weighted adding means ( 54a, 54b, 56) weights the division results of the dividing means with coefficients for each common partial image and adds them together. By determining a coefficient for weighting for each common partial image, it is possible to avoid blurring in a portion with large motion.

  Here, the common partial image may be a single pixel or a block of m pixels × n pixels (m and n are integers of 1 or more), but by appropriately selecting the size of the common partial image (for example, The processing amount for determining the coefficient can be suppressed.

  According to the first invention, random noise can be appropriately suppressed with a small amount of processing, and each movement between object scene images due to FPN and camera shake can also be suppressed.

  An image processing apparatus according to a second invention is dependent on the first invention, and the detection means assigns a plurality of detection areas to each of the plurality of object scene images, performs motion detection for each detection area, and adds means Further includes coefficient determination means for determining a coefficient for each common partial image, and the coefficient determination means specifies a detection area corresponding to each common partial image from among a plurality of detection areas, and a detection result of the detection means The coefficient is determined based on the movement of the identified detection area.

  In the second invention, a plurality of detection areas (E11 to E33) are assigned to each of the plurality of object scene images, and motion detection is performed for each detection area. In the addition, coefficient determination means (60, S35 to S39) determines a coefficient for each common partial image. Specifically, a detection area corresponding to each common partial image is specified from a plurality of detection areas (86, FIG. 6), and a coefficient is determined based on the movement of the specified detection area among the detected movements. Is determined (FIG. 8).

  According to the second invention, it is possible to further reduce the processing amount by using the detection result of the motion detection means for coefficient determination.

  An image processing apparatus according to a third invention is dependent on the first invention, wherein the adding means calculates a difference between the division results of the dividing means for each common partial image, and a coefficient for the common part Coefficient determination means for determining each image is further included, and the coefficient determination means performs coefficient determination based on the calculation result of the difference calculation means.

  In the third invention, in the addition, the difference between the division results of the dividing means is calculated for each common partial image by the difference calculating means (53). The coefficient is determined by coefficient determination means (55, 60, S34 and S36) based on the calculation result (formula (1)).

  According to the third aspect, since the coefficient is determined based on the difference, random noise can be more appropriately suppressed.

  An image processing apparatus according to a fourth invention is dependent on any one of the first to third inventions, wherein the motion detected by the FPN correction means for applying the FPN correction processing to the plurality of object scene images and the motion detection means is a threshold value. When the determination result of the first determination means is invalid and the FPN correction means is invalidated while the determination result of the first determination means is negative First control means for enabling the FPN correction means is further provided.

  In the fourth invention, the FPN correction means (18a) performs FPN correction processing on the plurality of object scene images, and the first discrimination means (S5) determines whether or not the motion detected by the motion detection means exceeds a threshold value. Is repeatedly determined. The first control means (S7, S9) invalidates the FPN correction means when the determination result of the first determination means is affirmative, and turns off the FPN correction means when the determination result of the first determination means is negative. Activate.

  According to the fourth invention, the power consumption can be reduced because the FPN correction means is invalidated during the period in which the motion exceeds the threshold. Thus, even if the FPN correction means is invalidated, FPN is suppressed through the cutting process by the first cutting means and the adding process by the subsequent adding means.

  An image processing apparatus according to a fifth invention is according to the fourth invention, further comprising a first memory in which FPN information unique to the imaging means is written, and the FPN correction means is based on the FPN information stored in the first memory. The FPN correction process is executed.

  In the fifth invention, FPN information unique to the imaging means is written in the first memory (18a), and the FPN correction means executes FPN correction processing based on the FPN information stored in the first memory.

  According to the fifth aspect, the first memory can be released during the period when the FPN correction is invalidated, and the required memory capacity can be reduced.

  An image processing apparatus according to a sixth invention is according to the fifth invention, wherein the FPN information includes size information indicating a size of the FPN, and the threshold value is a value related to the size information.

  The image processing device according to the seventh invention is dependent on the sixth invention, and the threshold value is larger than a value indicated by the size information.

  According to the sixth or seventh invention, FPN can be appropriately suppressed.

  An image processing apparatus according to an eighth invention is according to the seventh invention, wherein the FPN information includes presence / absence information indicating presence / absence of FPN, and second memory in which mode information indicating validity / invalidity of the camera shake correction mode is written, and Update means for updating the mode information stored in the memory 2 in response to the mode switching operation, and the first determination means indicates that the presence / absence information stored in the first memory indicates FPN and is stored in the second memory. When the mode information indicates that the camera shake correction mode is valid, a determination process is executed.

  In the eighth invention, the FPN information includes presence / absence information indicating the presence / absence of FPN, and mode information indicating the validity / invalidity of the camera shake correction mode is written in the second memory (84). The mode information stored in the second memory is updated by the updating means (24, 82c) when the mode switching operation is performed. The discrimination process of the first discrimination means is executed when the presence / absence information stored in the first memory indicates that FPN is present and the mode information stored in the second memory indicates that the camera shake correction mode is valid.

  The image processing apparatus according to the ninth invention is dependent on the eighth invention, and the first control means enables the first cutout means regardless of the determination result of the first determination means.

  In the ninth invention, when the FPN is present and the camera shake correction mode is valid, the first cutout means is always validated.

  An image processing apparatus according to a tenth invention is according to the eighth or ninth invention, wherein the presence / absence information stored in the first memory indicates that the FPN is present and the mode information stored in the second memory is invalid in the camera shake correction mode. Is further provided with second control means for enabling the FPN correction means and disabling the first cutout means.

  In the tenth invention, the second control means (S11) performs the FPN correction when the presence / absence information stored in the first memory indicates that the FPN is present and the mode information stored in the second memory indicates that the camera shake correction mode is invalid. The means is validated and the first cutting means is invalidated.

  According to the tenth invention, when the FPN is present and the camera shake correction mode is invalid, the FPN correction means is validated and the first cutout means is invalidated, so that power consumption can be reduced while suppressing FPN.

  An image processing apparatus according to an eleventh invention is dependent on the tenth invention, invalidates the FPN correction means when the presence / absence information indicates no FPN and the mode information stored in the second memory indicates that the camera shake correction mode is valid, and Third control means for activating the first cutout means is further provided.

  In the eleventh invention, the third control means (S15) invalidates the FPN correction means when the presence / absence information indicates that there is no FPN and the mode information stored in the second memory indicates that the camera shake correction mode is valid, and the first cutout is performed. Activate the means.

  According to the eleventh aspect, when no FPN and camera shake correction mode are enabled, FPN correction is unnecessary and the FPN correction means is invalidated, so that power consumption can be reduced.

  An image processing apparatus according to a twelfth aspect is according to the eleventh aspect, wherein the presence / absence information indicates no FPN and the mode information stored in the second memory indicates that the camera shake correction mode is invalid. Fourth control means for invalidating each of the means is further provided.

  In the twelfth invention, the fourth control means (S17) is configured to detect the FPN correction means and the first cut-out means when the presence / absence information indicates no FPN and the mode information stored in the second memory indicates the camera shake correction mode invalid. Disable each one.

  According to the twelfth invention, when there is no FPN and the camera shake correction mode is invalid, neither FPN correction nor camera shake correction is required, and each of the FPN correction means and the first cutout means is invalidated, thereby reducing power consumption. it can.

  An image processing apparatus according to a thirteenth invention is according to the twelfth invention, further comprising second cutout means for cutting out each of the plurality of object scene images at a fixed position, the first control means and the first control means Each of the three control means further invalidates the second cutting means in response to the validation process of the first cutting means, and each of the second control means and the fourth control means further invalidates the first cutting means. In response to the second cutout means.

  In the thirteenth aspect, the second cutout means (32) performs the cutout process at the fixed position for each of the plurality of object scene images. The second cutting means is activated when the first cutting means is invalidated by the first control means or the third control means, and when the first cutting means is validated by the second control means or the fourth control means. It is invalidated.

  According to the thirteenth aspect, it is possible to suppress a change in the image size due to invalidation / validation of the first cutout means.

  An image processing apparatus according to a fourteenth invention is dependent on any one of the first to thirteenth inventions, and the imaging means includes a CMOS.

  In general, CMOS is more likely to generate FPN than other imaging elements such as a CCD, and the need for FPN correction is high. For this reason, in the fourteenth aspect, the effect related to the FPN correction becomes significant.

  An image processing apparatus according to a fifteenth invention is dependent on any one of the first to fourteenth inventions, and further includes an imaging means.

  In the fifteenth invention, the imaging means is a component of the image processing apparatus.

  According to the present invention, random noise can be appropriately suppressed with a small amount of processing, and each movement between object scene images due to FPN and camera shake can also be suppressed. In addition, each of the required memory capacity and power consumption can be reduced.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

  Referring to FIG. 1, a digital movie camera 10 of this embodiment includes an optical lens 12, an image sensor 14, a signal preprocessing circuit 16, an FPN correction circuit 18a, a line memory 18b, a DRAM 20, a motion detection circuit 22, a CPU 24, a first A cutting circuit 26, a 3DDNR circuit 28, a signal post-processing circuit 30, a second cutting circuit 32, a display system circuit 34, a recording system circuit 36, a flash memory 38, and a key input device 40 are included.

  The image sensor 14 is preferably composed of a complementary metal-oxide semiconductor (CMOS), but may be composed of another image sensor such as a charge-coupled device (CCD).

  Among these components, each of the FPN correction circuit 18a, the line memory 18b, the first cut-out circuit 26, and the second cut-out circuit 32 depends on the presence or absence of FPN, the ON / OFF state of the camera shake correction mode, and the magnitude of the motion vector. Based on this, it is turned ON / OFF by the CPU 24. This ON / OFF control (see FIG. 11) is based on the FPN correction & cut-out ON / OFF control program (82d: see FIG. 4) stored in the flash memory 38. The 3DDNR circuit 28 is always in an ON state.

  Here, the software configuration of the digital movie camera 10 will be described. A part of the memory map of the flash memory 38 is shown in FIG. Referring to FIG. 4, FPN information area 80, program area 82, mode information area 84 and correspondence information area 86 are formed in flash memory 38.

  In the FPN information area 80, FPN information related to the image sensor 14 is stored. The FPN information includes an FPN flag 80a and size, occurrence position & correction value information 80b. The FPN flag 80a is a flag indicating the presence / absence of an FPN, and the size / occurrence position & correction value information 80b is information describing the FPN size, the occurrence position and a correction value corresponding thereto. Since the size and raw position of the FPN are common among a plurality of lines constituting the frame, it is sufficient that the size, generation position & correction value information 80b is for one line. When there is no FPN, the size, generation position & correction value information 80b may be omitted.

  The program area 82 stores a cutting position control program 82a, a line memory control program 82b, a mode switching control program 82c, an FPN correction & cutting ON / OFF control program 82d, and the like. The cut-out position control program 82a calculates a cut-out position based on the detection result of the motion detection circuit 22 (motion vector data stored in the motion vector areas 70 to 78 in the memory R built in the CPU 24: see FIG. 3). This is a program for notifying the first cutout circuit 26 of the calculation result (cutout position data stored in the cutout position area 79 in the memory R).

  The line memory control program 82b is a program for reading the size, occurrence position & correction value information 80b (see FIG. 4) from the FPN information area 80, and writing it in the line memory 18b. The mode switching control program 82c is a program for receiving a mode switching operation by the key input device 40 and updating the mode information in the mode information area 84 according to the operation result.

  The FPN correction & cut-out ON / OFF control program 82d includes FPN presence / absence (FPN flag 80a), camera shake correction mode ON / OFF state (camera shake correction mode flag 84a: described later), and motion vectors (for detection areas E11 to E33). This is a program for performing ON / OFF control of the FPN correction circuit 18a, the first cut-out circuit 26, and the second cut-out circuit 32 based on the size of the motion vector areas 70 to 78 (see FIG. 3).

  In the mode information area 84, a camera shake correction mode flag 84a is stored. The camera shake correction mode flag 84a is a flag indicating the ON / OFF state of the camera shake correction mode, and is controlled by the mode switching control program 82c.

  In the correspondence information area 86, correspondence between the detection areas (E11 to E33) of the motion detection circuit 22 and blocks (B11 to B66) which are units of weighted addition processing (described later) by the 3DDNR circuit 28 (FIG. 6). Information) is stored.

  The information / program described above is written in the flash memory 38 before shipment. The camera shake correction mode flag 84a is “OFF” at the time of shipment, and is updated in response to a mode switching operation by the key input device 40 thereafter. Further, the CPU 24 can process these programs (82a, 82b,...) In parallel under the control of a multitask OS such as μITRON.

  A flowchart corresponding to the FPN correction & cut-out ON / OFF control program 82d is shown in FIG. Referring to FIG. 11, first in step S1, CPU 24 determines the presence or absence of FPN based on FPN flag 80a (see FIG. 4). If the determination result in step S1 is YES, that is, “FPN exists”, the process proceeds to step S3.

  In step S3, whether or not the camera shake correction mode is ON is determined based on the camera shake correction mode flav 84a. If the determination result in step S3 is YES, that is, “camera shake correction ON state”, the process proceeds to step S5. If the determination result is NO, that is, “shake correction OFF state”, the process proceeds to step S11.

  In step S5, whether or not the magnitude of the motion vector (hereinafter referred to as “k: k−1”) of the kth frame with respect to the (k−1) th frame is greater than the threshold (Th1). Is determined based on the motion vector data stored in the memory R. The threshold value Th1 is a value determined in relation to the FPN size, and is preferably a value slightly larger than the FPN size described in the size, occurrence position & correction value information 80b.

  Referring now to FIG. 3, memory R includes nine motion vector areas 70-78 corresponding to nine detection areas E11-E33, respectively. In the motion vector areas 70 to 78, data indicating the motion vectors detected in the detection areas E11 to E33 are stored, respectively.

  If even one of the nine “k: k−1” stored in the motion vector areas 70 to 78 exceeds the threshold, YES is determined in step S5 and the process proceeds to step S7. On the other hand, if all of the sizes are equal to or smaller than the threshold, NO is determined in step S5, and the process proceeds to step S9.

  In step S7, the FPN correction circuit 18a is turned off to turn on the first cutout circuit 26, while the second cutout circuit 32 is turned off. In step S9, the FPN correction circuit 18a is turned on, the first cutting circuit 26 is turned on, and the second cutting circuit 32 is turned off. In step S11, the FPN correction circuit 18a is turned on to turn off the first cutout circuit 26, while the second cutout circuit 32 is turned on. Then, it returns to step S1.

  If the determination result in step S1 is NO, that is, “No FPN”, the process proceeds to step S13. In step S13, as in step S3, whether or not the camera shake correction mode is in the ON state is determined based on the camera shake correction mode flav 84a. If the determination result in step S13 is YES, the process proceeds to step S15, and if the determination result is NO, the process proceeds to step S17.

  In step S15, as in step S7, the FPN correction circuit 18a is turned off, the first cutting circuit 26 is turned on, and the second cutting circuit 32 is turned off. In step S17, the FPN correction circuit 18a is turned off to turn off the first cutout circuit 26, while the second cutout circuit 32 is turned on. Then, it returns to step S1.

  Therefore, the state relating to FPN correction and clipping of the digital movie camera 10 with FPN is a state corresponding to the period in which step S7 is executed (hereinafter referred to as “S7 state”), and the period in which step S9 is executed. The state transitions between three states: a state corresponding to (S9 state) and a state corresponding to the period during which step S11 is executed (S11 state).

  On the other hand, the state relating to FPN correction and extraction of the digital movie camera 10 without FPN is a state corresponding to the period in which step S15 is executed (S15 state) and a state corresponding to the period in which step S17 is executed (S17). Transition between two states.

  First, the operation in the “S7 state” will be described. Referring to FIG. 1 again, the optical image of the object scene that has passed through the optical lens 12 is irradiated to the light receiving surface, that is, the image pickup surface of the image sensor 14, and the image pickup surface corresponds to the optical image of the object scene by photoelectric conversion. Electric charges, ie raw image signals, are generated. The image sensor 14 repeatedly executes the exposure over the exposure time T and the reading of the raw image signal generated thereby, for example, at a period of 1/60 seconds. A raw image signal corresponding to the optical image of the object scene is output from the image sensor 14. The output raw image signal is subjected to preprocessing such as A / D conversion and clamping by the signal preprocessing circuit 16, thereby generating raw image data. The generated raw image data is given to the FPN correction circuit 18a.

  At this time, the FPN correction circuit 18a is in a halt state, and the writing process (described later) to the line memory 18b by the CPU 24 is not executed. The given raw image data is written in the first raw image area 60 (FIG. 2) of the DRAM 20 through the FPN correction circuit 18a.

  The raw image data created by the signal preprocessing circuit 16 is also given to the motion detection circuit 22. The motion detection circuit 22 detects a motion vector based on the given raw image data. Specifically, nine detection areas E11 to E33 are assigned to the object scene (see FIG. 6), and each detection area has a feature between the current frame (k) and the previous frame (k-1). The motion of a point, that is, a motion vector is repeatedly detected. The detection results of the motion detection circuit 22, that is, nine “k: k−1” corresponding to the nine detection areas E11 to E33 are written in the memory R built in the CPU 24 (see FIG. 3).

  Based on the motion vector stored in the memory R, the CPU 24 calculates, for each frame, a cutout position at which the motion of the object scene image caused by camera shake is canceled, and extracts the cutout position information indicating the calculation result as the first cutout information. The circuit 26 is notified.

  The raw image data stored in the first raw image area 60 is then given to the first cutout circuit 26. The cutout area E1 of the first cutout circuit 26 is movable (see FIG. 5A), and the first cutout circuit 26 moves the cutout area E1 based on the cutout position information notified from the CPU 24 while giving the cutout area E1. A process corresponding to the cut-out area E1 is cut out from each frame on the obtained raw image data. By the first cut-out process, the motion component in the time axis direction due to camera shake included in the raw image data is suppressed.

  The raw image data output from the first cutout circuit 26 is then provided to the 3DDNR circuit 28. At this time, the raw image data of the previous frame stored in the second raw image area 62 is further supplied to the 3DDNR circuit 28 as reference data, and the CPU 24 notifies the 3DDNR circuit 28 of the motion vector stored in the memory R. That is, the raw image data of the kth and (k−1) th frames and the nine motion vectors “k: k−1” between these two frames are input to the 3DDNR circuit 28 at a common timing. Is done.

  The 3DDNR circuit 28 performs weighted addition processing based on the nine motion vectors “k: k−1” on the raw image data of the kth frame and the raw image data of the (k−1) th frame. Here, the 3DDNR circuit 28 and the weighted addition process executed by the 3DDNR circuit 28 will be described in detail with reference to FIGS.

  7, 3DDNR circuit 28 includes a buffer 50a and controller 50b, a buffer 52a and controller 52b, a pair of multiplier circuits 54a and 54b, an adder circuit 56, a buffer 58a and controller 58b, and a CPU 60. Raw image data from the first cutout circuit 26 is written into the buffer 50a, and raw image data from the DRAM 20 is written into the buffer 52a.

  When the kth frame is stored in the buffer 50a and the (k-1) th frame is stored in the buffer 52a, the controller 50b and the controller 52b respectively convert the frames stored in the buffer 50a and the buffer 52a into 6 × 6 36 blocks. (B11 to B66: refer to FIG. 6), and these 36 blocks are sequentially read according to a block read command issued repeatedly by the CPU 60.

  The pair of read blocks, that is, the block Bij read from the buffer 50a and the block Bij read from the buffer 52a are input to the multiplication circuit 54a and the multiplication circuit 54b, respectively. At this time, the CPU 60 identifies the nine motion vectors “k: k−1” corresponding to the block Bij, and determines the coefficient α based on the identified motion vector.

  The multiplication circuit 54a is given a coefficient “α”, and the multiplication circuit 54a multiplies the input block Bij by the given coefficient “α”. On the other hand, the coefficient “1-α” is given to the multiplication circuit 54b, and the multiplication circuit 54b multiplies the input block Bij by the coefficient “1-α”.

  The multiplication result of the multiplication circuit 54 a and the multiplication result of the multiplication circuit 54 b are added together by the addition circuit 56. Then, the addition result of the addition circuit 56, that is, the block Bij after the weighted addition is written into the buffer 58a.

  When the last block of the kth frame, that is, the block B66 is written to the buffer 58a, the CPU 60 issues a frame read command to the controller 58b, and the controller 58b outputs the 36 blocks stored in the buffer 58a as one frame.

  The processing of the CPU 60 as described above follows the flowchart of FIG. Referring to FIG. 12, in the first step S31, it is determined whether or not frames are stored in the buffer 50a and the buffer 52a. If the determination result is NO, the process waits. If the determination result of step S31 is YES, it will move to step S33 and will issue a block read command to each of the controller 50b and the controller 52b. Thereafter, the process proceeds to step S35.

  In step S35, whether or not the movement between the pair of blocks read in response to the block read command, that is, the block Bij of the kth frame and the block Bij of the (k-1) th frame is large is determined from the CPU 24. It discriminate | determines based on the notified motion vector information.

  Specifically, first, a detection area corresponding to a pair of blocks is specified based on the correspondence information 86a to 86i stored in the correspondence information area 86. For example, when the pair of blocks is “B11”, “B11” is described in the correspondence information 86a, so that the corresponding detection area is “E11”.

  Next, the nine motion vectors “k: k−1” corresponding to the nine detection areas E11 to E33 notified from the CPU 24 are selected corresponding to the detection area specified as described above. Then, the size of the selected “k: k−1” is compared with the threshold value (Th2). If “| k: k−1 |> Th2”, it is determined that the movement is large, while if “| k: k−1 | ≦ Th2”, it is determined that the movement is small.

  If the determination result in step S35 is YES, that is, if the movement is large, the coefficient α is set to the maximum value αmax (for example, 0.8) in step S37. If the determination result in step S35 is NO, that is, if the movement is small, the coefficient α is calculated in step S39. This calculation process is based on a function or table that defines the relationship between motion, that is, | k: k-1 | and the coefficient α. An example of such a function is shown in FIG.

  Referring to FIG. 8, this function corresponds to a line segment with two points (0, 0.5) and (Th2, 0.8) at both ends, and α = 0.8 in a section where the motion exceeds Th2. Become. In general, any function may be used as long as α is minimum when the motion is “0”, α increases as the motion increases, and α is maximum when the motion is “Th2”. When using a table, discrete values calculated from a function, for example, (0, 0.5), (1, 0.6),..., (Th2, 0.8) are registered.

  Returning to FIG. 12, when the coefficient is determined in step S37 or S39, the process proceeds to step S41. In step S41, the coefficients “α” and “1-α” are notified to the multiplication circuits 54a and 54b, respectively. In subsequent step S43, it is determined whether or not the block read immediately before is the last block of the frame. If NO, the process returns to step S33. If “YES” in the step S43, the process shifts to a step S45 to issue a frame output command to the controller 58b, and then returns to the step S31.

  By such a weighted addition process, random noise included in the raw image data is suppressed.

  Referring again to FIG. 1, the raw image data output from the 3DDNR circuit 28 is written in the second raw image area 62 (see FIG. 2) of the RAM 20. The raw image data stored in the second raw image area 62 in this manner is then given to the 3DDNR circuit 28 as reference data.

  The raw image data output from the 3DDNR circuit 28 is also provided to the signal post-processing circuit 30. The signal post-processing circuit 30 performs post-processing such as color separation, gamma correction, and YUV conversion on the given raw image data. Thereby, the raw image data is converted into YUV image data.

  The YUV image data obtained in this way is written in the YUV image area 64 (see FIG. 2) in the DRAM 20. The YUV image data stored in the YUV image area 64 is then given to the second cutout circuit 32. At this time, the second cut-out circuit 32 is in a resting state, and the applied YUV image data passes through the second cut-out circuit 32 and is supplied to each of the display system circuit 34 and the recording system circuit 36.

  In the display system circuit 34, processing for driving an LCD monitor (not shown) with the given YUV image data is executed. In the recording system circuit 36, processing for compressing the applied YUV image data, processing for recording the compressed image data on a recording medium (not shown), and the like are executed.

  Next, the operation in the “S9 state” will be described. Referring to FIG. 1 again, the image sensor 14 repeatedly executes exposure and readout of the raw image signal generated thereby, as in the “S7 state”. The raw image signal output from the image sensor 14 is preprocessed by the signal preprocessing circuit 16, and the raw image data created thereby is given to the FPN correction circuit 18a in units of lines.

  The CPU 24 reads the size, generation position & correction value 80b (see FIG. 4) from the flash memory 38, and writes this to the line memory 18b. This write processing is based on the line memory write control program 82b (see FIG. 4). The FPN correction circuit 18a performs FPN correction on the given raw image data based on the size, generation position & correction value 80b stored in the line memory 18b. Thereby, the FPN included in the raw image data is suppressed.

  The raw image data stored in the first raw image area 60 is then given to the first cutout circuit 26 and subjected to the same first cutout process.

  At this time, as in the “S7 state”, the raw image data of the previous frame stored in the second raw image area 62 is further given to the 3DDNR circuit 28 as reference data. The motion detection circuit 22 performs similar motion vector detection, and the detection result is notified to the 3DDNR circuit 28 through the CPU 24. The 3DDNR circuit 28 executes weighted addition processing based on the motion vector notified to the given raw image data.

  The raw image data output from the 3DDNR circuit 28 is written in the second raw image area 62 of the RAM 20 and then given to the 3DDNR circuit 28 as reference data. The raw image data output from the 3DDNR circuit 28 is also supplied to the signal post-processing circuit 30 and converted into YUV image data. The YUV image data is written in the YUV image area 64 in the DRAM 20 and then given to the second cutout circuit 32.

  At this time, as in the “S7 state”, the second cut-out circuit 32 is in a resting state, and the given YUV image data passes through the second cut-out circuit 32 to display the display system circuit 34 and the recording system circuit 36. Given to each of the. The display system circuit 34 performs LCD driving processing and the like, and the recording system circuit 36 performs compression processing and recording processing.

  Here, the state transition from the “S7 state” to the “S9 state” is shown in the timing chart of FIG. Referring to FIG. 9, while the magnitude of the motion vector is equal to or less than the threshold value, the first cutout, FPN correction, and 3DDNR processes are continued, and the second cutout process is paused. When the magnitude of the motion vector exceeds the threshold value, the FPN correction process is stopped. The first cutout process and the 3DDNR process are continued, and the second cutout process is stopped.

  Next, the operation in the S11 state will be described. The “S11 state” is a state in which, in the “S9 state”, the first cutting circuit 26 is stopped and the second cutting circuit 32 is activated. Therefore, in the “S11 state”, the cutout position calculation and notification processing by the CPU 24 is not executed. For this reason, the raw image data in the first raw image area 60 passes through the first cutout circuit 26 and is supplied to the 3DDNR circuit 28. On the other hand, YUV image data from the YUV image area 64 is given to each of the display system circuit 34 and the recording system circuit 36 after being subjected to the second cutting process of the second cutting circuit 32.

  The cut-out area E2 of the second cut-out circuit 32 is fixed (FIG. 5B), and the second cut-out circuit 32 cuts out the portion corresponding to the cut-out area E2 from each frame for the given YUV image data. Cut out. The change in the image size due to the stop / restart of the first cropping is eliminated by executing such a second cropping process. Except this point, the operation is the same as in the “S9 state”.

  Here, the state transition from the “S7 state” to the “S11 state” is shown in the timing chart of FIG. Referring to FIG. 10, while the camera shake correction is ON, the first cutout and 3DDNR processes are continued, and the second cutout and FPN correction processes are paused. When camera shake correction is turned off, the first cutout process is stopped, and the FPN correction and second cutout processes are started. The 3DDNR process is continued.

  Finally, the operation in each state of S15 and S17 will be briefly described. The “S15 state” is the same state as the “S7 state”, and the operations performed are common between both states.

  The “S17 state” is a state in which the FPN correction circuit 18a is further suspended in the “S11 state”. Therefore, in the “S17 state”, the raw image data from the signal preprocessing circuit 16 passes through the FPN correction circuit 18a and is written in the first raw image area 60 of the DRAM 20. Except this point, the operation is the same as in the “S11 state”.

  As is clear from the above, in this embodiment, an optical image of the object scene is repeatedly captured by the image sensor 14, and a plurality of object scene images according to time series are output from the image sensor 14. The FPN correction circuit 18a performs FPN correction processing on the plurality of object scene images, and the motion detection circuit 22 detects the motion of the feature points among the plurality of object scene images.

  The first cutout circuit 26 cuts out each of the plurality of object scene images at a position based on the detection result of the motion detection means. That is, the cutting position of the first cutting circuit 26 moves according to the movement of the feature point. In this way, the movement between the plurality of object scene images is suppressed by making the cutout position follow the movement of the feature point.

  Then, for each of the plurality of object scene images according to the time series after the cut-out process by the first cut-out circuit 26, an addition process for adding the immediately preceding object scene image to the object scene image, It is applied by the 3DDNR circuit 28. As a result, random noise included in each of the plurality of object scene images is suppressed.

  On the other hand, the CPU 24 repeatedly determines whether or not the motion detected by the motion detection means exceeds the threshold (S5), and invalidates the FPN correction circuit 18a when the determination result is affirmative (S7). When the determination result is negative, the FPN correction circuit 18a is validated (S9).

  Therefore, the power consumption can be reduced because the FPN correction circuit 18a is invalidated during a period in which the motion exceeds the threshold. Thus, even if the FPN correction circuit 18a is invalidated, the FPN is suppressed through the cutting process by the first cutting circuit 26 and the adding process by the subsequent 3DDNR circuit 28. This is because the movement of the cutout position results in the FPN having the property of random noise and is suppressed by the 3DDNR circuit 28.

  Also in this embodiment, in the 3DDNR circuit 28, the buffer 50a and the controller 50b and the buffer 52a and the controller 52b divide each of the pair of object scene images to be added to the blocks B11 to B66 (common partial images). Then, the multiplication circuits 54a and 54b and the addition circuit 56 weight the division results with coefficients for each block and add them to each other. The coefficient (α) for weighting is determined for each block by the CPU 60 based on the motion vector notified from the motion detection circuit 22 via the CPU 24. In this way, by determining the coefficient for each common partial image, it is possible to avoid blurring in a portion with large motion.

  The blocks B11 to B66 may be a single pixel or m pixels × n pixels (m and n are integers of 1 or more), but by appropriately selecting the block size (for example, about several pixels × several pixels) ), The processing amount for determining the coefficient can be suppressed. Thereby, random noise can be appropriately suppressed with a small amount of processing.

  In the 3DDNR circuit 28 of this embodiment, the CPU 60 determines the coefficient α on the basis of the motion vector notified from the motion detection circuit 22 via the CPU 24, but instead, a pair output from the buffer 50a and the buffer 52a. The difference between the blocks Bij and Bij may be obtained, and the coefficient may be determined based on this difference. Such an embodiment is shown in FIGS.

  Referring to FIG. 13, the digital movie camera 10A is the same as the digital movie camera 10 of FIG. 1, except that the motion vector notification from the CPU 24 to the CPU 60 is omitted and the 3DDNR circuit 28 is replaced with a 3DDNR circuit 28A.

  Referring to FIG. 14, the 3DDNR circuit 28A includes a difference calculation circuit 53 that calculates a difference between blocks Bij and Bij output from the buffers 50a and 52a in the 3DDNR circuit 28 of FIG. Based on this, a coefficient calculation circuit 55 for calculating the coefficient α is added.

  The calculation result of the difference calculation circuit 53 is given to each of the coefficient calculation circuit 55 and the CPU 60. The coefficient calculation circuit 55 calculates the coefficient α by the following equation (1) based on the given difference.

α = {difference} × β + η (1)
Here, β and η are parameters corresponding to the slope and intercept of the straight line, and are set by the CPU 60.

  The CPU 60 determines the parameters β and η in the above equation (1) based on the given difference, and sets the determination result in the difference calculation circuit 53. The operation flow of the CPU 60 in this case is shown in FIG.

  The flow of FIG. 15 includes steps S34 and S36 instead of steps S35 to S41 in the flow of FIG. In step S34, the parameters β and η are determined based on the difference from the difference calculation circuit 53. In step S36, the determination result is set in the coefficient calculation circuit 55. In response to the coefficient setting process, the coefficient α and “1-α” are output from the coefficient calculation circuit 55 to the multiplication circuits 54a and 54b.

  The parameters β and η may be determined independently of the difference or may be fixed values. In such a case, it is not necessary to notify the CPU 60 of the calculation result of the difference calculation circuit 53.

  Thereby, random noise can be suppressed more appropriately.

  1 and FIG. 13, the number of detection areas of the motion detection circuit 22 is nine, and these nine detection areas are arranged in a 3 × 3 matrix form on the frame (FIG. 6). The number and arrangement are not limited to this. For example, the four detection areas E11, E31, E13, and E33 arranged at the four corners of the frame may be omitted, and the coefficient α may be fixed to a minimum value (for example, 0.5) in a portion where no detection area is allocated. However, the first cutout circuit 26 appropriately removes motion components between frames caused by camera shake, and the 3DDNR circuits 28 and 28A include random noise (this includes motion components within the frames caused by camera shake). ) Is appropriately removed, a matrix mode of m rows and n columns (m is an integer of 3 or more and n is an integer of 3 or more) is preferable.

  In each embodiment, there is a gap between the detection areas E11 to E33, but such a gap may be eliminated. Adjacent detection areas may partially overlap. The shape is not limited to a rectangle, and may be a circle or a polygon.

  In each embodiment, the motion detection circuit 22 detects the motion of the feature point between the current frame (k) and the immediately preceding frame (k−1), and the first cutout circuit 26 cuts out based on the detection result. However, after the detection process of the motion detection circuit 22, the CPU 24 predicts the motion of the feature point between the current frame (k) and the immediately following frame (k + 1) based on the detection result. The extraction circuit 26 may perform extraction based on the prediction result. In the example of FIG. 13, the 3DDNR circuit 28 may also use the prediction results.

  In each embodiment, the FPN information is fixed. For example, the CPU 24 may detect the FPN of the image sensor 14 at any time through the FPN correction circuit 18a, and update the FPN information based on the detection result.

  Although the digital movie camera (10, 10A) has been described above as an example, the present invention can be applied to an image processing apparatus that processes a plurality of scene images according to a time series captured by an image sensor. The image processing apparatus in this case is not only a device having an image sensor as a component (for example, a digital still camera, a portable terminal with a camera), but also a device having no image sensor (for example, an external image sensor). Also included are personal computers and digital video players that can capture scene images.

It is a block diagram which shows the structure of one Example of this invention. It is an illustration figure which shows a part of memory map of DRAM. It is an illustration figure which shows a part of memory map of the built-in memory of CPU24. It is an illustration figure which shows a part of memory map of flash memory. (A) is an illustration figure which shows the extraction area of a 1st extraction circuit, (B) is an illustration figure which shows the extraction area of a 2nd extraction circuit. It is an illustration figure which shows the correspondence between the detection area of a motion detection circuit, and the division | segmentation block of a 3DDNR circuit. FIG. 2 is a block diagram illustrating a configuration example of a 3DDNR circuit applied to the embodiment in FIG. 1. It is an illustration figure which shows the function utilized for the weighted addition process in a 3DDNR circuit. It is a timing chart which shows the operation example of each element of FIG. 1 Example. It is a timing chart which shows the other operation example of each element of FIG. 1 Example. It is a flowchart which shows a part of operation | movement of CPU24 applied to the FIG. 1 Example. It is a flowchart which shows a part of operation | movement of CPU60 applied to the FIG. 1 Example. It is a block diagram which shows the structure of the other Example of this invention. FIG. 14 is a block diagram illustrating a configuration example of a 3DDNR circuit applied to the embodiment in FIG. 13; It is a flowchart which shows a part of operation | movement of CPU60 applied to the FIG. 13 Example.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10,10A ... Digital movie camera 14 ... Image sensor 16 ... Signal pre-processing circuit 18a ... FPN correction circuit 18b ... Line memory 20 ... DRAM
22 ... Motion detection circuit 24, 60 ... CPU
DESCRIPTION OF SYMBOLS 26 ... 1st cutting-out circuit 28, 28A ... 3DDNR circuit 30 ... Signal post-processing circuit 32 ... 2nd cutting-out circuit 38 ... Flash memory 40 ... Key input device 50a, 52a, 58a ... Buffer 50b, 52b, 58b ... Controller 54a, 54b ... multiplication circuit 53 ... difference calculation circuit 55 ... coefficient calculation circuit 56 ... adder circuit

Claims (15)

An image processing apparatus that performs image processing on a plurality of object scene images according to a time series output from an imaging unit that repeatedly captures an optical image of the object scene,
A motion detecting means for detecting a motion of a feature point between the plurality of object scene images;
First cutout means for cutting out each of the plurality of scene images at a position based on a detection result of the motion detection means; and the plurality of scene images after passing through the first cutout means For each, an addition means for performing addition processing for adding one or more object scene images that are temporally close to the object scene image,
The adding means includes a dividing means for dividing each set of object scene images to be added to each other into a common partial image, and the division result of the dividing means is weighted by a coefficient for each common partial image. An image processing apparatus including weighted addition means for adding. The detection means assigns a plurality of detection areas to each of the plurality of object scene images, performs motion detection for each detection area,
The adding means further includes coefficient determining means for determining the coefficient for each common partial image,
The coefficient determination means specifies a detection area corresponding to each common partial image from the plurality of detection areas, and performs coefficient determination based on the movement of the detection area specified among the detection results of the detection means. The image processing apparatus according to claim 1. The adding means further includes a difference calculating means for calculating a difference between the division results of the dividing means for each common partial image, and a coefficient determining means for determining the coefficient for each common partial image,
The image processing apparatus according to claim 1, wherein the coefficient determining unit determines a coefficient based on a calculation result of the difference calculating unit. FPN correction means for performing FPN correction processing on the plurality of object scene images,
A first discriminating unit that repeatedly discriminates whether or not the motion detected by the motion detecting unit exceeds a threshold value, and invalidates the FPN correction unit when the discrimination result of the first discriminating unit is positive, 4. The image processing apparatus according to claim 1, further comprising a first control unit that activates the FPN correction unit when a determination result of the first determination unit is negative. 5. A first memory in which FPN information unique to the imaging means is written;
The image processing apparatus according to claim 4, wherein the FPN correction unit performs FPN correction processing based on FPN information stored in the first memory.   The image processing apparatus according to claim 5, wherein the FPN information includes size information indicating a size of the FPN, and the threshold is a value related to the size information.   The image processing apparatus according to claim 6, wherein the threshold value is larger than a value indicated by the size information. The FPN information includes presence / absence information indicating presence / absence of FPN,
A second memory in which mode information indicating validity / invalidity of the camera shake correction mode is written; and update means for updating the mode information stored in the second memory in response to a mode switching operation;
The said 1st discrimination | determination means performs a discrimination | determination process, when the presence / absence information stored in the said 1st memory shows that FPN exists, and the mode information stored in the said 2nd memory shows camera shake correction mode effective. Image processing apparatus.   The image processing apparatus according to claim 8, wherein the first control unit validates the first cutout unit regardless of a determination result of the first determination unit.   When the presence / absence information stored in the first memory indicates that the FPN is present and the mode information stored in the second memory indicates that the camera shake correction mode is invalid, the FPN correction unit is enabled and the first cutout unit is disabled. The image processing apparatus according to claim 8, further comprising a second control unit configured to perform the control.   Third control means for disabling the FPN correction means and enabling the first cutout means when the presence / absence information indicates no FPN and the mode information stored in the second memory indicates that the camera shake correction mode is valid The image processing apparatus according to claim 10, comprising:   And fourth control means for invalidating each of the FPN correction means and the first cut-out means when the presence / absence information indicates no FPN and the mode information stored in the second memory indicates invalid camera shake correction mode. The image processing apparatus according to claim 11. A second cutting means for cutting out each of the plurality of object scene images at a fixed position;
Each of the first control means and the third control means further invalidates the second cutting means in response to the activation process of the first cutting means,
The image processing apparatus according to claim 12, wherein each of the second control unit and the fourth control unit further enables the second cutout unit in response to the invalidation process of the first cutout unit.   The image processing apparatus according to claim 1, wherein the imaging unit includes a CMOS.   The image processing apparatus according to claim 1, further comprising the imaging unit.

Images