USRE43491E1 - Image shake correction image processing apparatus and program - Google Patents
Image shake correction image processing apparatus and program Download PDFInfo
- Publication number
- USRE43491E1 USRE43491E1 US12/981,316 US98131610A USRE43491E US RE43491 E1 USRE43491 E1 US RE43491E1 US 98131610 A US98131610 A US 98131610A US RE43491 E USRE43491 E US RE43491E
- Authority
- US
- United States
- Prior art keywords
- image
- frames
- blocks
- motion vector
- moving image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
- 238000012545 processing Methods 0.000 title claims abstract description 24
- 238000012937 correction Methods 0.000 title description 11
- 230000033001 locomotion Effects 0.000 claims abstract description 109
- 239000013598 vector Substances 0.000 claims abstract description 63
- 230000006870 function Effects 0.000 claims description 3
- 238000003672 processing method Methods 0.000 claims 1
- 238000006073 displacement reaction Methods 0.000 abstract description 38
- 230000015654 memory Effects 0.000 description 16
- 208000019300 CLIPPERS Diseases 0.000 description 13
- 208000021930 chronic lymphocytic inflammation with pontine perivascular enhancement responsive to steroids Diseases 0.000 description 13
- 230000006835 compression Effects 0.000 description 4
- 238000007906 compression Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000000034 method Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 238000005303 weighing Methods 0.000 description 1
- 230000003936 working memory Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/681—Motion detection
- H04N23/6811—Motion detection based on the image signal
Definitions
- the present invention relates to image processing apparatus and program for correcting an image shake in a moving image.
- An image shake preventing apparatus which detects an image shake due to camera shake with an image sensor such as a CCD or a CMOS and corrects the image shake (for example, see Unexamined Japanese Patent Publication 2004-15376).
- This apparatus calculates a range of magnitudes of motion vectors to be selected in respective areas, which compose each of image frames which in turn compose a captured image, based on the average value and standard deviation of the motion vectors and selects only motion vectors present in the selected range of magnitudes. Furthermore, the apparatus calculates a contrast coefficient for motion vectors in each area, a weighing coefficient based on the contrast coefficient, and an average of the motion vectors weighted with the weighting coefficient in each area, which is used as a total motion vector in that area. Thus, the reliability of the total motion vector is evaluated and the total motion vector can be calculated appropriately.
- a conventional image capture apparatus 200 comprises an image capturer 201 that captures a moving image and converts the image to electronic image data, a feature detector 202 that detects the features of an image based on the image data and calculates the amount of the features, a block selector 203 that selects a block appropriate for calculating a displacement of the frame, a motion calculator 204 that calculates a motion vector from each of the selected blocks, a displacement corrector 205 that corrects the displacement of that frame based on the calculated motion vectors, an encoder 206 that encodes resulting corrected image data, and a display 207 that displays a resulting captured moving image.
- a series of processing steps such as amount-of-features calculation, block selection, moving vector calculation and displacement correction, ranging from the exposing step to the encoding and displaying steps (steps A 1 -A 6 ), must be performed in a predetermined time (see FIG. 8 ).
- steps A 1 -A 6 a series of processing steps
- a considerable processing time is required from determination of an offset in one frame to clipping of predetermined image data.
- the drive frequency of a CPU concerned must be increased and that a working memory must be provided additionally.
- the present invention provides an image processing apparatus comprising: a feature detector that detects features, and calculates the amount of features, of an image in each of blocks that compose one of a plurality of frames which in turn compose a moving image of a subject; a block selector that selects blocks that contains a amount of features, larger than a predetermined amount of features, detected by the feature detector from among the plurality of blocks of the particular frame; a motion calculator that calculates a motion of each of blocks of a second particular one of the plurality of frames next to the particular frame and corresponding to a respective one of the blocks selected by the block selector; a total motion vector calculator that calculates a total motion vector of the above-mentioned next frame based on the motions of the respective blocks of the second particular frame calculated by the motion calculator; and an image shake corrector that corrects an image shake in the moving image based on the total motion vector calculated by the total motion vector calculator.
- At least one of the feature detector and the motion calculator may perform its operation such that a received amount of image data to be processed does not exceed the amount of the image data to be processed.
- the present invention provides an image processing program that causes a computer to perform the functions of detecting features, and calculating the amount of features, of an image in each of blocks that compose one of a plurality of frames which in turn compose a moving image of a subject; selecting blocks that contains a calculated amount of features, larger than a predetermined amount of features, from among the plurality of blocks of the particular frame; calculating a motion of each of blocks of a second particular one of the plurality of frames next to the first-mentioned particular frame and corresponding to a respective one of the selected blocks; calculating a total motion vector of the second particular frame based on the calculated motions of the respective selected blocks of the second particular frame; and correcting an image shake in the moving image based on the calculated total motion vector.
- FIG. 1 is a block diagram of an essential part of an image capture apparatus illustrated as an embodiment of an image processing apparatus according to the present invention
- FIG. 2 schematically illustrates a frame involving macroblocks to be processed in an image shake correction by the image capture apparatus of FIG. 1 ;
- FIG. 3 schematically illustrates calculation of motion vectors
- FIG. 4 schematically illustrates relations of each steps of image shake correction on continuous two frames
- FIG. 5 is a sequence chart of the image shake correction to be performed by the FIG. 1 image capture apparatus
- FIG. 6 illustrates a modification of the image shake correction to be performed by the FIG. 1 image capture apparatus
- FIG. 7 is a block diagram indicative of the composition of an essential part of a conventional image capture apparatus.
- FIG. 8 schematically illustrates an image shake correction to be performed by the FIG. 7 conventional image capture apparatus.
- FIG. 1 is a block diagram of an essential part of an image capture apparatus 100 illustrated as an embodiment of an image processing apparatus according to the present invention.
- the image capture apparatus 100 is applied, for example, to a digital camera, and captures a moving image of a subject and encodes a captured moving image in accordance with an information compression system such as MPEG 4 which is encoding standards to record the captured moving image. More specifically, as shown in FIG. 1 , the image capture apparatus 100 comprises an image capture unit 1 that captures a still or moving image of a subject by converting its captured optical image to electric image data, a feature detector 2 that calculates the amount of features of the image data, a block selector 3 that selects from among a plurality of blocks that compose a frame F a block B 1 (see FIG.
- a memory 4 that stores information on the selected block B 1 , a motion calculator 5 that calculates a motion vector from the information on the block B 1 , a displacement corrector 6 that calculates a displacement amount of the frame F based on the motion vector calculated by the motion calculator 5 and corrects the displacement amount, an encoder 7 that encodes the image data, which is corrected by the displacement corrector 6 , in accordance with a predetermined information compression system, and a display 8 that displays a resulting captured moving image.
- the image capturer 1 captures one of a still image and a moving image by operating a changeover switch (not shown). More specifically, the image capturer 1 comprises an image capture lens group having focus and zoom functions, an electronic image capturer that comprises a CCD or a CMOS that converts an optical image of a subject that has passed through the image capture lens group due to exposure to two-dimensional image data, a signal processor that performs a predetermined image processing step on the image data received from the electronic image capturer, and an image capture controller that controls the electronic image capturer and the signal processor, although these elements are not shown.
- the image data is forwarded from the signal processor to the feature detector 2 and the motion calculator 5 under control of the image capture controller.
- the feature detector 2 detects features of each of frames that compose a moving image received from the image capturer 1 by performing a predetermined operation on the frame F, and then calculates the amount of features of the frame. More specifically, the feature detector 2 reads each of a plurality of blocks that compose a respective one (for example, F 1 ) of a plurality of frames F, which compose a captured image, stored in the memory 4 , which may, for example, include an image memory. The feature detector 2 then detects the amount of features, which includes, for example, intersections of lines, patterns and/or vertexes of an object, in a specified image area (or block) of the image, which can easily specify a difference between the frame F 1 and a next frame.
- the block selector 3 selects each block B 1 (see FIG. 2 ) which contains an amount of features larger than a predetermined amount of features from among a plurality of blocks that compose a frame F. As shown in FIG. 2 , the block B 1 selected by the block selector 3 contains an image of a tip of a subject and hence is appropriate for calculation of a motion vector by the motion calculator 5 . A block B 0 , whose amount of features is small, such as an expressionless background is not appropriate for calculation of a motion vector because a wrong motion vector may be calculated. The block selector 3 forwards block information including the coordinates of the selected block B 1 to an image clipper 9 (see FIG. 5 ).
- the image clipper 9 is composed, for example, of a memory controller that controls outputting image data on the frame F stored in the memory 4 to the motion calculator 5 , encoder 7 and display 8 . More specifically, the image clipper 7 calculates an image area of a block B 2 corresponding to the block B 1 from a frame F 2 (hereinafter referred to as present frame F 2 ) next to the frame F 1 (hereinafter referred to as reference frame F 1 ) selected by the block selector 3 and corresponding to the block B 1 of the frame 1 based on the block information on the reference frame F 1 received from the block sector 3 (see FIG. 5 ). The information on the calculated image area is clipped and forwarded to the memory 4 .
- the memory 4 comprises, for example, an image memory that stores image data received from the image capturer 1 after subjected to predetermined image processing by the image capturer 1 .
- the memory 4 also stores information on the image area received from the image clipper 9 . Further, the memory 4 temporarily stores image data on the present frame F 2 subjected to displacement correction by a displacement corrector 6 to be described later in more detail.
- the motion calculator 5 calculates a motion vector (or motion) of the block B 2 of the present frame F 2 corresponding to the block B 1 of the frame F 1 selected by the block selector 3 of the reference frame F 1 .
- a motion vector or motion
- the block B 1 of the preceding reference frame F 1 is used.
- the motion calculator 5 detects a block B 2 of the present frame F 2 where a difference between the blocks B 2 and B 1 is minimum in the image area of the present frame F 2 corresponding to that of the selected block B 1 , based on the image data on the selected block B 1 of the preceding reference frame F 1 and image data on a corresponding image area of the present frame F 2 read from the memory 4 . Then, the motion calculator 5 calculates a motion vector indicative of the amount of movement of the detected block B 2 in horizontal and vertical directions (see FIG. 3 ).
- the displacement corrector 6 comprises a displacement calculator 61 that calculates a displacement indicative of a total motion vector of the present frame F 2 by averaging the motion vectors of the plurality of blocks of the present frame F 2 calculated by the motion calculator 5 , and an offset calculator 62 that calculates an offset of the present frame F 2 relative to the reference frame 1 based on the calculated displacement.
- Data on the calculated offset of the present frame F 2 is then forwarded to the image clipper 9 , which moves (or corrects) the present frame F 2 by the offset relative to the preceding frame F 1 , and clips an image area of the present frame F 2 equal to the display area of the display 8 .
- the displacement corrector 6 and the image clipper 9 compose a displacement corrector that corrects a displacement of a moving image based on the respective motion vectors of the feature blocks B 1 calculated by the motion calculator 5 .
- the image clipper 9 forwards the corrected image data of the present frame F 2 to the memory 4 , which stores the image data temporarily, and then forwards the image data to the encoder 7 and display 8 at predetermined times.
- the encoder 7 encodes moving image data in accordance with an information compression system that performs motion compensation such as, for example, MPEG 4. More specifically, the encoder 7 performs motion compensation that reduces the code amount of moving image data received from the displacement corrector 6 , using a time correlation between the frames F composing the moving image, and an information compression/encoding step such as DCT (Discreet Cosine Transform) that reduces the code amount of the image data using a spatial correlation between the frames F.
- the encoder 7 may quantize a DCT conversion coefficient calculated in DCT with a predetermined Q value in consideration of a visual characteristic.
- the display 8 is composed, for example, of a liquid crystal display that displays a moving image based on the moving image data, whose displacement is corrected, received from the displacement corrector 6 .
- the display 8 may also receive uncorrected moving image data, and display a corresponding moving image.
- FIG. 4 schematically illustrates one example of the displacement correcting step.
- FIG. 5 is another illustration of the displacement correcting step.
- the motion vector calculating step, the displacement correcting step, the encoding step and the displaying step for the reference frame F 1 are substantially the same as those for the present frame F 2 , and further description thereof will be omitted.
- the horizontal axis represents a time axis and each of divisions of the time axis represents a processing period for a respective frame F.
- a feature detecting step is performed on the reference frame F 1 (steps S 1 ). More particularly, when one (hereinafter referred to as reference frame F 1 ) of frames F that compose a moving image captured by the image capturer 1 and stored in the memory 4 is read and forwarded to the feature detector 2 , the feature detector 2 detects, and calculates the amount of, features of each of blocks that compose the reference frame F 1 and then calculates the amount of features. Then, the block selector 3 selects a predetermined number of blocks B 1 , which each contain an amount of features larger than a predetermined amount of features from among the blocks that compose the reference frame F 1 (step S 2 ).
- the image clipper 9 When the image clipper 9 receives block information including coordinates of each selected block B 1 from the block selector 3 , the image clipper 9 calculates or clips an image area that contains a block B 2 of the present frame F 2 next to the reference frame F 1 corresponding to the selected block B 1 of the frame F 1 (step S 3 ). When the image area is calculated, the image clipper 9 forwards the information on the image area to the memory 4 , which then stores the information temporarily (step S 4 ).
- the motion calculator 5 then performs a motion calculating step on the present frame F 2 (step S 5 ). More particularly, the motion calculator 5 calculates a motion vector of a block B 2 of the present frame F 2 corresponding to each selected block B 1 of the preceding reference frame F 1 read from the memory 4 based on the image data on the selected block B 1 and the image data on the image area of the block B 2 .
- the displacement calculator 61 of the displacement corrector 6 averages the motion vectors of a predetermined number of blocks B 2 , thereby calculating a displacement of the present frame F 2 from the reference frame F 1 (step S 6 ).
- the offset calculator 62 calculates an offset of the present frame F 2 relative to the frame F 1 based on the calculated displacement of the frame F 2 (step S 7 ).
- the image clipper 9 moves or corrects the present frame F 2 by the amount of the offset relative to the reference frame F 1 , and clips an image area of the present frame F 2 equal to the display area of the display 8 (step S 8 ).
- step S 9 data on the present frame F 2 whose displacement is corrected at a predetermined time is forwarded from the image clipper 9 to the encoder 7 and the display 8 , which then perform encoding and displaying steps, respectively, on the present frame F 2 data (step S 9 ).
- the above processing steps are sequentially performed on the respective ones of the plurality of frames F that compose the moving image, a moving image whose image shake is corrected is obtained.
- a motion vector of a block B 2 of the present frame F 2 corresponding to each selected block B 1 of a reference frame F 1 is calculated and the total motion vector of the frame F 2 is calculated. That is, the total motion vector of the present frame F 2 can be calculated based on the calculated amount of features of the blocks of the reference frame F 1 .
- the time required from the reception of the image data of the present frame F 2 to the termination of the displacement correcting step is reduced compared to the conventional image processing apparatus which sequentially performs the respective processing steps ranging from the exposing step to the encoding step.
- the motion calculating step and the amount of features calculating step involving each of the plurality of frames F can be performed in parallel.
- the time required for performing the processing steps ranging from the start of the image capture to the termination of the displacement correcting step of all of the frames F that compose the moving image is reduced.
- the drive frequency of the image capture apparatus 100 and hence its power consumption are reduced.
- the image capture apparatus 100 also can appropriately accommodate to an increasing frame rate of the moving image.
- the motion calculator 5 is illustrated as calculating the motion vectors of the blocks of the present frame F 2 after the exposing step is terminated, or all the image data of the present frame F 2 is acquired, the motion calculating step by the motion calculator 5 and the exposing step may be performed in parallel, for example, as shown in FIG. 6 .
- the motion calculator 5 performs a motion vector calculating step such that a received amount (or, for example, count) of image data of a block on which the motion vector calculating step is performed is not exceeded by an amount (or, for example, count) of image data processed by the motion calculator 5 .
- the motion calculator 5 can perform the motion calculating step on that image data.
- the motion calculating step can start slightly delayed from the beginning of the exposing step without waiting for the termination of the exposing step, thereby bringing forward the respective ending times of the motion vector calculating step and subsequent processing steps by a time indicated by a two-dot-chain line arrow G in FIG. 6 .
- the time required from the beginning of the exposing step to the termination of the encoding step/displaying step of the present frame F 2 is reduced.
- the motion calculating step and the exposing step are illustrated as performed in parallel in the displacement correcting step of FIG. 6
- the exposing step and the amount of features calculating step may be performed in parallel or the amount of features calculating step and the motion calculating step may be performed in parallel with the exposing step.
- the step in parallel with which one or both of the motion calculating step and the amount of features calculating step can be performed is not limited to the exposing step, but may be any step which is performed before the motion calculating step and the amount of features calculating step are performed.
- the image data decoding step may be performed in parallel with the motion calculating step and the amount of features calculating step, thereby reducing the time required from decoding the encoded moving image file to reproduction of a moving image whose image shake is corrected.
- the image capture apparatus 100 that captures an image of a subject and displays or reproduces the captured still or moving image
- any apparatus capable of performing at least a predetermined image processing operation on a captured moving image may be employed instead.
- the present invention may be embodied by a personal computer as an image processing apparatus that captures a moving image file, calculates the amount of features, selects blocks that contain an amount of features larger than a predetermined amount of features, calculates motion vectors of the blocks, calculates a total motion vector and corrects an image shake in the moving image.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Studio Devices (AREA)
Abstract
An image processing apparatus comprising: a block selector that selects one of blocks that contain an amount of features, larger than a predetermined amount of features, from among a plurality of blocks which compose a particular one of a plurality of frames of a moving image of a subject; a motion calculator that calculates a motion vector of each of blocks of a second particular one of the plurality of frames next to the first-mentioned particular frame and corresponding to a respective one of the selected blocks; a displacement calculator that calculates a displacement of the second particular frame based on the calculated motion vectors; and an image shake corrector that corrects an image shake in the moving image based on the calculated displacement.
Description
1. Field of the Invention
The present invention relates to image processing apparatus and program for correcting an image shake in a moving image.
2. Related Art
An image shake preventing apparatus is hitherto known which detects an image shake due to camera shake with an image sensor such as a CCD or a CMOS and corrects the image shake (for example, see Unexamined Japanese Patent Publication 2004-15376). This apparatus calculates a range of magnitudes of motion vectors to be selected in respective areas, which compose each of image frames which in turn compose a captured image, based on the average value and standard deviation of the motion vectors and selects only motion vectors present in the selected range of magnitudes. Furthermore, the apparatus calculates a contrast coefficient for motion vectors in each area, a weighing coefficient based on the contrast coefficient, and an average of the motion vectors weighted with the weighting coefficient in each area, which is used as a total motion vector in that area. Thus, the reliability of the total motion vector is evaluated and the total motion vector can be calculated appropriately.
However, since in this apparatus it is determined whether or not the calculated motion vectors are appropriate for calculating the total motion vector, the motion vectors of a subject image inappropriate for calculating the total motion vector will also be calculated necessarily. Thus, the efficiency of the image shake correction is low.
We can use a method of calculating motion vectors of macroblocks of a frame estimated to be appropriate for calculating a total motion vector as a result of analysis of the distinctive attributes of a subject and then calculating a total motion vector based on the motion vectors of the macroblocks. This method is improved in prevention of needless vector calculation. Since in this method analysis of the subject, or determination of whether or not each macroblock is appropriate for calculating the total motion vector, is still required as a prestep, it is difficult to reduce the series of processing times greatly.
For example, as shown in FIGS. 7 and 8 , a conventional image capture apparatus 200 comprises an image capturer 201 that captures a moving image and converts the image to electronic image data, a feature detector 202 that detects the features of an image based on the image data and calculates the amount of the features, a block selector 203 that selects a block appropriate for calculating a displacement of the frame, a motion calculator 204 that calculates a motion vector from each of the selected blocks, a displacement corrector 205 that corrects the displacement of that frame based on the calculated motion vectors, an encoder 206 that encodes resulting corrected image data, and a display 207 that displays a resulting captured moving image.
In the image capture apparatus 200, in order to ensure a predetermined frame rate when a moving image is captured, a series of processing steps such as amount-of-features calculation, block selection, moving vector calculation and displacement correction, ranging from the exposing step to the encoding and displaying steps (steps A1-A6), must be performed in a predetermined time (see FIG. 8 ). However, if such series of processing steps are performed sequentially, a considerable processing time is required from determination of an offset in one frame to clipping of predetermined image data. Thus, there is the problem with this apparatus that the drive frequency of a CPU concerned must be increased and that a working memory must be provided additionally.
It is therefore an object of the present invention to provide an image processing apparatus and program that reduces the time required from the start of the moving image capture to the termination of the image shake correction in each of frames that composes the moving image, thereby accommodating appropriately to an increasing frame rate of the moving image.
In one aspect, the present invention provides an image processing apparatus comprising: a feature detector that detects features, and calculates the amount of features, of an image in each of blocks that compose one of a plurality of frames which in turn compose a moving image of a subject; a block selector that selects blocks that contains a amount of features, larger than a predetermined amount of features, detected by the feature detector from among the plurality of blocks of the particular frame; a motion calculator that calculates a motion of each of blocks of a second particular one of the plurality of frames next to the particular frame and corresponding to a respective one of the blocks selected by the block selector; a total motion vector calculator that calculates a total motion vector of the above-mentioned next frame based on the motions of the respective blocks of the second particular frame calculated by the motion calculator; and an image shake corrector that corrects an image shake in the moving image based on the total motion vector calculated by the total motion vector calculator.
At least one of the feature detector and the motion calculator may perform its operation such that a received amount of image data to be processed does not exceed the amount of the image data to be processed.
In another aspect, the present invention provides an image processing program that causes a computer to perform the functions of detecting features, and calculating the amount of features, of an image in each of blocks that compose one of a plurality of frames which in turn compose a moving image of a subject; selecting blocks that contains a calculated amount of features, larger than a predetermined amount of features, from among the plurality of blocks of the particular frame; calculating a motion of each of blocks of a second particular one of the plurality of frames next to the first-mentioned particular frame and corresponding to a respective one of the selected blocks; calculating a total motion vector of the second particular frame based on the calculated motions of the respective selected blocks of the second particular frame; and correcting an image shake in the moving image based on the calculated total motion vector.
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate presently preferred embodiment of the present invention and, together with the general description given above and the detailed description of the preferred embodiments given below, serve to explain the principles of the present invention in which:
An embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram of an essential part of an image capture apparatus 100 illustrated as an embodiment of an image processing apparatus according to the present invention.
The image capture apparatus 100 is applied, for example, to a digital camera, and captures a moving image of a subject and encodes a captured moving image in accordance with an information compression system such as MPEG 4 which is encoding standards to record the captured moving image. More specifically, as shown in FIG. 1 , the image capture apparatus 100 comprises an image capture unit 1 that captures a still or moving image of a subject by converting its captured optical image to electric image data, a feature detector 2 that calculates the amount of features of the image data, a block selector 3 that selects from among a plurality of blocks that compose a frame F a block B1 (see FIG. 2 ) that contains an amount of features larger than a predetermined amount of features appropriate for calculating a displacement of the frame F, a memory 4 that stores information on the selected block B1, a motion calculator 5 that calculates a motion vector from the information on the block B1, a displacement corrector 6 that calculates a displacement amount of the frame F based on the motion vector calculated by the motion calculator 5 and corrects the displacement amount, an encoder 7 that encodes the image data, which is corrected by the displacement corrector 6, in accordance with a predetermined information compression system, and a display 8 that displays a resulting captured moving image.
The image capturer 1 captures one of a still image and a moving image by operating a changeover switch (not shown). More specifically, the image capturer 1 comprises an image capture lens group having focus and zoom functions, an electronic image capturer that comprises a CCD or a CMOS that converts an optical image of a subject that has passed through the image capture lens group due to exposure to two-dimensional image data, a signal processor that performs a predetermined image processing step on the image data received from the electronic image capturer, and an image capture controller that controls the electronic image capturer and the signal processor, although these elements are not shown. The image data is forwarded from the signal processor to the feature detector 2 and the motion calculator 5 under control of the image capture controller.
The feature detector 2 detects features of each of frames that compose a moving image received from the image capturer 1 by performing a predetermined operation on the frame F, and then calculates the amount of features of the frame. More specifically, the feature detector 2 reads each of a plurality of blocks that compose a respective one (for example, F1) of a plurality of frames F, which compose a captured image, stored in the memory 4, which may, for example, include an image memory. The feature detector 2 then detects the amount of features, which includes, for example, intersections of lines, patterns and/or vertexes of an object, in a specified image area (or block) of the image, which can easily specify a difference between the frame F1 and a next frame.
The block selector 3 selects each block B1 (see FIG. 2 ) which contains an amount of features larger than a predetermined amount of features from among a plurality of blocks that compose a frame F. As shown in FIG. 2 , the block B1 selected by the block selector 3 contains an image of a tip of a subject and hence is appropriate for calculation of a motion vector by the motion calculator 5. A block B0, whose amount of features is small, such as an expressionless background is not appropriate for calculation of a motion vector because a wrong motion vector may be calculated. The block selector 3 forwards block information including the coordinates of the selected block B1 to an image clipper 9 (see FIG. 5 ).
The image clipper 9 is composed, for example, of a memory controller that controls outputting image data on the frame F stored in the memory 4 to the motion calculator 5, encoder 7 and display 8. More specifically, the image clipper 7 calculates an image area of a block B2 corresponding to the block B1 from a frame F2 (hereinafter referred to as present frame F2) next to the frame F1 (hereinafter referred to as reference frame F1) selected by the block selector 3 and corresponding to the block B1 of the frame 1 based on the block information on the reference frame F1 received from the block sector 3 (see FIG. 5 ). The information on the calculated image area is clipped and forwarded to the memory 4.
The memory 4 comprises, for example, an image memory that stores image data received from the image capturer 1 after subjected to predetermined image processing by the image capturer 1. The memory 4 also stores information on the image area received from the image clipper 9. Further, the memory 4 temporarily stores image data on the present frame F2 subjected to displacement correction by a displacement corrector 6 to be described later in more detail.
The motion calculator 5 calculates a motion vector (or motion) of the block B2 of the present frame F2 corresponding to the block B1 of the frame F1 selected by the block selector 3 of the reference frame F1. In this case, it is assumed that there is no significant change between corresponding blocks of adjacent frames appropriate for calculating a total motion vector (or displacement) of a particular frame F. Thus, when the total motion vector of the present frame F2 is calculated, the block B1 of the preceding reference frame F1 is used. More specifically, the motion calculator 5 detects a block B2 of the present frame F2 where a difference between the blocks B2 and B1 is minimum in the image area of the present frame F2 corresponding to that of the selected block B1, based on the image data on the selected block B1 of the preceding reference frame F1 and image data on a corresponding image area of the present frame F2 read from the memory 4. Then, the motion calculator 5 calculates a motion vector indicative of the amount of movement of the detected block B2 in horizontal and vertical directions (see FIG. 3 ).
The displacement corrector 6 comprises a displacement calculator 61 that calculates a displacement indicative of a total motion vector of the present frame F2 by averaging the motion vectors of the plurality of blocks of the present frame F2 calculated by the motion calculator 5, and an offset calculator 62 that calculates an offset of the present frame F2 relative to the reference frame 1 based on the calculated displacement. Data on the calculated offset of the present frame F2 is then forwarded to the image clipper 9, which moves (or corrects) the present frame F2 by the offset relative to the preceding frame F1, and clips an image area of the present frame F2 equal to the display area of the display 8. The displacement corrector 6 and the image clipper 9 compose a displacement corrector that corrects a displacement of a moving image based on the respective motion vectors of the feature blocks B1 calculated by the motion calculator 5.
The image clipper 9 forwards the corrected image data of the present frame F2 to the memory 4, which stores the image data temporarily, and then forwards the image data to the encoder 7 and display 8 at predetermined times.
The encoder 7 encodes moving image data in accordance with an information compression system that performs motion compensation such as, for example, MPEG 4. More specifically, the encoder 7 performs motion compensation that reduces the code amount of moving image data received from the displacement corrector 6, using a time correlation between the frames F composing the moving image, and an information compression/encoding step such as DCT (Discreet Cosine Transform) that reduces the code amount of the image data using a spatial correlation between the frames F. The encoder 7 may quantize a DCT conversion coefficient calculated in DCT with a predetermined Q value in consideration of a visual characteristic.
The display 8 is composed, for example, of a liquid crystal display that displays a moving image based on the moving image data, whose displacement is corrected, received from the displacement corrector 6. The display 8 may also receive uncorrected moving image data, and display a corresponding moving image.
The displacement correcting step will be described with reference to FIGS. 4 and 5 . FIG. 4 schematically illustrates one example of the displacement correcting step. FIG. 5 is another illustration of the displacement correcting step. In the following description, the motion vector calculating step, the displacement correcting step, the encoding step and the displaying step for the reference frame F1 are substantially the same as those for the present frame F2, and further description thereof will be omitted. In FIG. 4 , the horizontal axis represents a time axis and each of divisions of the time axis represents a processing period for a respective frame F.
As shown in FIGS. 4 and 5 , first, a feature detecting step is performed on the reference frame F1 (steps S1). More particularly, when one (hereinafter referred to as reference frame F1) of frames F that compose a moving image captured by the image capturer 1 and stored in the memory 4 is read and forwarded to the feature detector 2, the feature detector 2 detects, and calculates the amount of, features of each of blocks that compose the reference frame F1 and then calculates the amount of features. Then, the block selector 3 selects a predetermined number of blocks B1, which each contain an amount of features larger than a predetermined amount of features from among the blocks that compose the reference frame F1 (step S2).
When the image clipper 9 receives block information including coordinates of each selected block B1 from the block selector 3, the image clipper 9 calculates or clips an image area that contains a block B2 of the present frame F2 next to the reference frame F1 corresponding to the selected block B1 of the frame F1 (step S3). When the image area is calculated, the image clipper 9 forwards the information on the image area to the memory 4, which then stores the information temporarily (step S4).
The motion calculator 5 then performs a motion calculating step on the present frame F2 (step S5). More particularly, the motion calculator 5 calculates a motion vector of a block B2 of the present frame F2 corresponding to each selected block B1 of the preceding reference frame F1 read from the memory 4 based on the image data on the selected block B1 and the image data on the image area of the block B2. When the motion vectors of the blocks B2 are calculated, the displacement calculator 61 of the displacement corrector 6 averages the motion vectors of a predetermined number of blocks B2, thereby calculating a displacement of the present frame F2 from the reference frame F1 (step S6). Then, the offset calculator 62 calculates an offset of the present frame F2 relative to the frame F1 based on the calculated displacement of the frame F2 (step S7).
Then, when the image clipper 9 receives data on the calculated offset from the offset calculator 62, the image clipper 9 moves or corrects the present frame F2 by the amount of the offset relative to the reference frame F1, and clips an image area of the present frame F2 equal to the display area of the display 8 (step S8).
Then, data on the present frame F2 whose displacement is corrected at a predetermined time is forwarded from the image clipper 9 to the encoder 7 and the display 8, which then perform encoding and displaying steps, respectively, on the present frame F2 data (step S9). When the above processing steps are sequentially performed on the respective ones of the plurality of frames F that compose the moving image, a moving image whose image shake is corrected is obtained.
As described above, according to the image capture apparatus 100 of this embodiment, among a plurality of frames F that compose a moving image of a subject a motion vector of a block B2 of the present frame F2 corresponding to each selected block B1 of a reference frame F1 is calculated and the total motion vector of the frame F2 is calculated. That is, the total motion vector of the present frame F2 can be calculated based on the calculated amount of features of the blocks of the reference frame F1. Thus, the time required from the reception of the image data of the present frame F2 to the termination of the displacement correcting step is reduced compared to the conventional image processing apparatus which sequentially performs the respective processing steps ranging from the exposing step to the encoding step.
The motion calculating step and the amount of features calculating step involving each of the plurality of frames F can be performed in parallel. Thus, when these processing steps are performed sequentially on the respective ones of the plurality of frames, the time required for performing the processing steps ranging from the start of the image capture to the termination of the displacement correcting step of all of the frames F that compose the moving image is reduced. Thus, the drive frequency of the image capture apparatus 100 and hence its power consumption are reduced. The image capture apparatus 100 also can appropriately accommodate to an increasing frame rate of the moving image.
While in the embodiment the motion calculator 5 is illustrated as calculating the motion vectors of the blocks of the present frame F2 after the exposing step is terminated, or all the image data of the present frame F2 is acquired, the motion calculating step by the motion calculator 5 and the exposing step may be performed in parallel, for example, as shown in FIG. 6 .
More specifically, the motion calculator 5 performs a motion vector calculating step such that a received amount (or, for example, count) of image data of a block on which the motion vector calculating step is performed is not exceeded by an amount (or, for example, count) of image data processed by the motion calculator 5. Thus, each time image data of a predetermined number of blocks is acquired in the exposing step from among all the blocks that compose the present frame F2, the motion calculator 5 can perform the motion calculating step on that image data. That is, the motion calculating step can start slightly delayed from the beginning of the exposing step without waiting for the termination of the exposing step, thereby bringing forward the respective ending times of the motion vector calculating step and subsequent processing steps by a time indicated by a two-dot-chain line arrow G in FIG. 6 . As a result, the time required from the beginning of the exposing step to the termination of the encoding step/displaying step of the present frame F2 is reduced.
While the motion calculating step and the exposing step are illustrated as performed in parallel in the displacement correcting step of FIG. 6 , the exposing step and the amount of features calculating step may be performed in parallel or the amount of features calculating step and the motion calculating step may be performed in parallel with the exposing step. The step in parallel with which one or both of the motion calculating step and the amount of features calculating step can be performed is not limited to the exposing step, but may be any step which is performed before the motion calculating step and the amount of features calculating step are performed.
Furthermore, when an encoded moving image file stored in the memory 4 is decoded and reproduced for displaying purposes without being subjected to the displacement correction, the image data decoding step may be performed in parallel with the motion calculating step and the amount of features calculating step, thereby reducing the time required from decoding the encoded moving image file to reproduction of a moving image whose image shake is corrected.
While in the embodiment the image capture apparatus 100 that captures an image of a subject and displays or reproduces the captured still or moving image is illustrated, any apparatus capable of performing at least a predetermined image processing operation on a captured moving image may be employed instead. For example, the present invention may be embodied by a personal computer as an image processing apparatus that captures a moving image file, calculates the amount of features, selects blocks that contain an amount of features larger than a predetermined amount of features, calculates motion vectors of the blocks, calculates a total motion vector and corrects an image shake in the moving image.
Various modifications and changes may be made thereunto without departing from the broad spirit and scope of this invention. The above-described embodiments are intended to illustrate the present invention, not to limit the scope of the present invention. The scope of the present invention is shown by the attached claims rather than the embodiments. Various modifications made within the meaning of an equivalent of the claims of the invention and within the claims are to be regarded to be in the scope of the present invention.
This application is based on Japanese Patent Application No. 2005-323575 filed on Nov. 8, 2005 and including specification, claims, drawings and summary. The disclosure of the above Japanese patent application is incorporated herein by reference in its entirety.
Claims (4)
1. An image processing apparatus comprising:
a feature detector that detects features of an image in each of a plurality of blocks that form each of a plurality of frames which in turn compose a moving image of a subject;
a block selector that selects blocks of a first one of the frames that contain an amount of the detected features that is larger than a predetermined amount;
a motion calculator that calculates a motion of respective blocks of a following second one of the frames which correspond to the selected blocks of the first one of the frames;
a total motion vector calculator that calculates a total motion vector of the second one of the frames based on the calculated motions of the respective blocks;
an image shake corrector that corrects an image shake in the moving image based on the total motion vector calculated by the total motion vector calculator; and
an encoder which encodes the moving image corrected by the image shake corrector for display;
wherein the feature detector and the encoder are controlled to operate in parallel.
2. The image processing apparatus of claim 1 , wherein at least one of the feature detector and the motion calculator is controlled to operate such that a received amount of image data to be processed does not exceed a predetermined amount.
3. An image processing method comprising:
detecting, using an image processor, features of an image in each of a plurality of blocks that form each of a plurality of frames which in turn compose a moving image of a subject;
selecting blocks of a first one of the frames that contain an amount of the detected features that is larger than a predetermined amount;
calculating a motion of respective blocks of a following second one of the frames which correspond to the selected blocks of the first one of the frames;
calculating a total motion vector of the second one of the frames based on the calculated motions of the respective blocks;
correcting an image shake in the moving image based on the calculated total motion vector;
encoding the corrected moving image for display; and
controlling the detecting and encoding operations to be performed in parallel.
4. A computer readable medium having stored thereon a program for performing functions comprising:
detecting, using an image processor, features of an image in each of a plurality of blocks that form each of a plurality of frames which in turn compose a moving image of a subject picked up by an image capture apparatus;
selecting blocks of a first one of the frames that contain an amount of the detected features that is larger than a predetermined amount;
calculating a motion of respective blocks of a following second one of the frames which correspond to the selected blocks of the first one of the frames;
calculating a total motion vector of the second one of the frames based on the calculated motions of the respective blocks;
correcting an image shake in the moving image based on the calculated total motion vector;
encoding the corrected moving image for display; and
controlling the detecting and encoding operations to be performed in parallel.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US12/981,316 USRE43491E1 (en) | 2005-11-08 | 2010-12-29 | Image shake correction image processing apparatus and program |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2005-323575 | 2005-11-08 | ||
JP2005323575A JP5044922B2 (en) | 2005-11-08 | 2005-11-08 | Imaging apparatus and program |
US11/592,828 US7595818B2 (en) | 2005-11-08 | 2006-11-03 | Image shake correction image processing apparatus and program |
US12/981,316 USRE43491E1 (en) | 2005-11-08 | 2010-12-29 | Image shake correction image processing apparatus and program |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/592,828 Reissue US7595818B2 (en) | 2005-11-08 | 2006-11-03 | Image shake correction image processing apparatus and program |
Publications (1)
Publication Number | Publication Date |
---|---|
USRE43491E1 true USRE43491E1 (en) | 2012-06-26 |
Family
ID=38003336
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/592,828 Ceased US7595818B2 (en) | 2005-11-08 | 2006-11-03 | Image shake correction image processing apparatus and program |
US12/981,316 Active 2027-12-01 USRE43491E1 (en) | 2005-11-08 | 2010-12-29 | Image shake correction image processing apparatus and program |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/592,828 Ceased US7595818B2 (en) | 2005-11-08 | 2006-11-03 | Image shake correction image processing apparatus and program |
Country Status (2)
Country | Link |
---|---|
US (2) | US7595818B2 (en) |
JP (1) | JP5044922B2 (en) |
Families Citing this family (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5098081B2 (en) * | 2007-07-19 | 2012-12-12 | オリンパス株式会社 | Image processing method and image processing apparatus |
JP4935930B2 (en) * | 2008-06-10 | 2012-05-23 | 富士通株式会社 | Image correction apparatus and image correction method |
JP5374220B2 (en) * | 2009-04-23 | 2013-12-25 | キヤノン株式会社 | Motion vector detection device, control method therefor, and imaging device |
TW201230805A (en) * | 2011-01-04 | 2012-07-16 | Aptos Technology Inc | Video playback apparatus and method |
FR2991785B1 (en) | 2012-06-06 | 2014-07-18 | Astrium Sas | STABILIZATION OF A VIEWING LINE OF AN IMAGING SYSTEM ON BOARD A SATELLITE |
CN107911615B (en) * | 2017-12-26 | 2019-07-02 | 北京理工大学 | A kind of Gray Projection digital image stabilization method and system |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5497191A (en) * | 1993-12-08 | 1996-03-05 | Goldstar Co., Ltd. | Image shake compensation circuit for a digital video signal |
US5614945A (en) * | 1993-10-19 | 1997-03-25 | Canon Kabushiki Kaisha | Image processing system modifying image shake correction based on superimposed images |
US5926212A (en) * | 1995-08-30 | 1999-07-20 | Sony Corporation | Image signal processing apparatus and recording/reproducing apparatus |
JP2004015376A (en) | 2002-06-06 | 2004-01-15 | Canon Inc | Apparatus for preventing image shake and camera |
US7012635B2 (en) * | 1997-01-31 | 2006-03-14 | Kabushiki Kaisha Toshiba | Solid state image sensor and video system using the same |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3626210B2 (en) * | 1991-11-14 | 2005-03-02 | 中部電力株式会社 | Camera three-dimensional motion detection method and apparatus based on moving image |
JP2004343483A (en) * | 2003-05-16 | 2004-12-02 | Acutelogic Corp | Device and method for correcting camera-shake and device for detecting camera shake |
-
2005
- 2005-11-08 JP JP2005323575A patent/JP5044922B2/en not_active Expired - Fee Related
-
2006
- 2006-11-03 US US11/592,828 patent/US7595818B2/en not_active Ceased
-
2010
- 2010-12-29 US US12/981,316 patent/USRE43491E1/en active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5614945A (en) * | 1993-10-19 | 1997-03-25 | Canon Kabushiki Kaisha | Image processing system modifying image shake correction based on superimposed images |
US5497191A (en) * | 1993-12-08 | 1996-03-05 | Goldstar Co., Ltd. | Image shake compensation circuit for a digital video signal |
US5926212A (en) * | 1995-08-30 | 1999-07-20 | Sony Corporation | Image signal processing apparatus and recording/reproducing apparatus |
US7012635B2 (en) * | 1997-01-31 | 2006-03-14 | Kabushiki Kaisha Toshiba | Solid state image sensor and video system using the same |
JP2004015376A (en) | 2002-06-06 | 2004-01-15 | Canon Inc | Apparatus for preventing image shake and camera |
Also Published As
Publication number | Publication date |
---|---|
JP5044922B2 (en) | 2012-10-10 |
US7595818B2 (en) | 2009-09-29 |
US20070103556A1 (en) | 2007-05-10 |
JP2007134788A (en) | 2007-05-31 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7880769B2 (en) | Adaptive image stabilization | |
US7171052B2 (en) | Apparatus and method for correcting motion of image | |
KR100864999B1 (en) | Camera shake correcting device | |
USRE43491E1 (en) | Image shake correction image processing apparatus and program | |
JP4887750B2 (en) | Image processing apparatus, control method, and program | |
KR100941285B1 (en) | Image processing device and computer readable medium | |
JP2008193530A (en) | Image recorder, image recording method and program | |
US7956898B2 (en) | Digital image stabilization method | |
US20110176027A1 (en) | Method and device for compressing moving image | |
US20090202163A1 (en) | Determination of optimal frame types in video encoding | |
JP4533089B2 (en) | Movie data generator | |
JP2008104223A (en) | Hand shaking correction device | |
JP2004187023A (en) | Moving image encoding device and moving image decoding device | |
JP2018191136A (en) | Encoding device, encoding method and program | |
US20100104011A1 (en) | Moving picture coding method | |
JPH06105299A (en) | Dynamic image compressor | |
JP2001024932A (en) | Device and method for inputting image | |
JP2006041648A (en) | Moving image processing unit | |
JP2005175891A (en) | Recorder | |
JP2011029690A (en) | Electronic camera and image encoding method | |
JP2010124369A (en) | Frame interpolation device, image encoder, and image decoder |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
FPAY | Fee payment |
Year of fee payment: 8 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 12 |