CN1976399A - Image processing apparatus and image processing method - Google Patents

Image processing apparatus and image processing method Download PDF

Info

Publication number
CN1976399A
CN1976399A CN 200610160944 CN200610160944A CN1976399A CN 1976399 A CN1976399 A CN 1976399A CN 200610160944 CN200610160944 CN 200610160944 CN 200610160944 A CN200610160944 A CN 200610160944A CN 1976399 A CN1976399 A CN 1976399A
Authority
CN
China
Prior art keywords
relevance values
reduction
vector
block
approaching
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN 200610160944
Other languages
Chinese (zh)
Other versions
CN100559843C (en
Inventor
仓田徹
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Corp
Original Assignee
Sony Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Corp filed Critical Sony Corp
Publication of CN1976399A publication Critical patent/CN1976399A/en
Application granted granted Critical
Publication of CN100559843C publication Critical patent/CN100559843C/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Image Analysis (AREA)

Abstract

The present invention provides an image processing apparatus for identifying a motion vector by executing the steps of setting a plurality of reference blocks, searching a specific reference block, and detecting the motion vector. The image processing apparatus includes correlation-value computation means, table generation means, and motion-vector computation means.

Description

Image processing apparatus and image processing method
Technical field
The present invention relates to be used to detect the image processing apparatus and the image processing method of two motion vectors between the different pictures.More specifically, the present invention is suitable for detecting the situation of the motion vector between the following picture, and described picture is comprising in the image information that is obtained in the image pickup process of carrying out such as the image pick-up device of digital camera or video camera etc. that by using so-called hand tremor trembles (hand-trembling) component.
Background technology
Generally speaking, hand-held during such as digital camera image pick up equipment carries out image pick-up operations such as (or video cameras) by using, the hand tremor of image pick up equipment during owing to image pick-up operation shaken, and causes the shake of each picture of captured image.A kind of known at trembling the shake that causes by hand tremor as the shake of captured image, the method that captured image is compensated is to detect the motion vector of each picture of captured image, and based on detected motion vector, the position of reading the position as image pickup data that exists in the video memory is offset, so that the shake at trembling of hand causes compensates image.
In addition, as the method for the motion vector of each picture that detects captured image, the block-matching technique of the correlation between known a kind of captured image that is used to find out two pictures.Because block-matching technique need be such as the mechanical component of the gyro sensor that serves as angular-rate sensor (gyro sensor) etc., so this technology provides and can realize that volume is little, the advantage of lightweight image pick up equipment.
The diagrammatic sketch of reference when Figure 42 and Figure 43 are the overviews of description block matching technique.Figure 44 shows the flow chart that the exemplary process of block-matching technique is adopted in expression.
Block-matching technique is such method, wherein, by the motion vector between the same units chunk on the reference picture of units chunk on the raw frames of the captured image of image pickup units output and captured image by with raw frames on piece and the correlation between the same block on the reference picture be associated.Reference picture by the captured image of image pickup units output is defined as current just processed picture, and the raw frames that is also referred to as target picture is defined as the last picture of adjacent reference picture or before reference picture every the picture of a picture interval.
Should be noted that in this case the image that picture is made up of a frame or field picture data.But in specification of the present invention, picture is defined as the image be made up of a frame image data, so that make the explanation easy to understand.Therefore, picture is also referred to as frame.That is to say that reference picture and raw frames also are called as reference frame and primitive frame respectively.
For example, the view data of reference frame is such view data, and it is exported by image pickup units, and as the view data of present frame, from manifesting of present frame, has stored in frame memory and a frame corresponding time of delay occurs.On the other hand, the view data of primitive frame is such view data, and it is as the image of the former frame that is close to, from manifesting of present frame, has stored in frame memory and two frames corresponding time of delay occurs.
As mentioned above, Figure 42 and Figure 43 are reference graph when the overview of description block matching technique.Figure 44 shows the flow chart that the exemplary process of block-matching technique is adopted in representative.
According to block-matching technique, as shown in figure 42, any pre-position on primitive frame 101 (being also referred to as target frame 101) is provided with object block 103.Object block 103 is the rectangular areas with pre-sizing.Object block 103 has a plurality of pixels, and these pixels are arranged as in the horizontal direction and form row, and a plurality of this row are arranged in vertical direction.
On the other hand, position (or identical coordinate) identical with the position (or coordinate) of object block 103 on reference frame 102 is located, if hand does not tremble, then object block projected image piece 104 will serve as object block 103.In Figure 42, object block projected image piece 104 is plotted as the piece that is centered on by dotted line.So hunting zone 105 is set to its center and overlaps with object block projected image piece 104.In Figure 42, hunting zone 105 is plotted as the piece that is centered on by chain-dotted line.In addition, reference block 106 is assumed to be the piece that will move to another position from a position on hunting zone 105, as below describing.When reference block 106 is positioned at the following position that exists on the reference frame 102, reference block 106 that moves and the correlation between the object block 103 become the strongest: described position is not have owing to hand tremor is trembled and the position of the offset of object block 103 from the primitive frame 101, as shown in the figure.
Then, the position of the reference block 106 on the reference frame 102 changes on hunting zone 105, make great efforts search hunting zone 105 so as to find out the view data that comprises in the reference block 106 that shows this position and object block 103 in comprise the position of the strongest correlation between the view data, that is, the correlation of finding out between mobile reference block 106 and the object block 103 becomes the strongest position.The position that shows strongest correlation is detected as the strongest correlation position of reference block 106, perhaps the physical location of object block 103 on reference frame 102.The strongest correlation position of detected reference block 106 or the physical location of object block 103 on reference frame 102 are detected as motion vector 110 from the offset value of the position of object block projected image piece 104, and it comprises moving direction.
The search hunting zone 105 with the process of finding out the position that shows strongest correlation in, reference block 106 in the position on the reference frame 102 on hunting zone 105 generally with once corresponding to the distance of a pixel or a plurality of pixels, on vertical and horizontal direction, changing.Therefore, each in a plurality of reference block locations all is set in advance in hunting zone 105, the position that will be moved to during search procedure as reference piece 106.
In the reference block 106 that moves to another position on the hunting zone 105 from a position and the strongest correlation between the object block 103 is to calculate on the basis of the respective pixel of the pixel of the reference block 106 of each current position of reference block 106 and object block 103 basically.As calculating the mobile reference block 106 and the method for the strongest correlation between the object block 103, multiple technologies have been proposed, comprise all methods.By use will be in the back as being used for storage finds dependency expression result's the table of the process of strongest correlation to be described as the typical method of SAD (absolute difference sum) by employing table, above-mentioned motion vector 110 be detected as representative from the position of object block projected image piece 104 reference vector to the size and Orientation of the distance of the strongest correlation position of reference block 106.SAD be all pixels in the reference block 106 and the brightness value between all respective pixel in the object block 103 difference absolute value and.Strongest correlation is by the minimum SAD representative of the indicated strongest correlation position of the motion vector that is used for beginning from the position of object block projected image piece 104 110.
If sad value is used to represent relevance values, then sad value is more little, and correlation is strong more.Therefore, reference block 106 is being moved to the process of another position from a position on hunting zone 105, position in the hunting zone 105 is detected as the strongest correlation position of reference block 106, promptly, as the position that shows the strongest correlation between reference block 106 and the object block 103, or has the position of the minimum sad value between reference block 106 and the object block 103.Motion vector 110 be representative from the position of the object block on the primitive frame 101 103 or the position of the object block projected image piece 104 on the reference frame 102 to the detected position that shows strongest correlation or have a vector of skew of the detected position of minimum sad value.
According to block-matching technique, a plurality of positions that reference block 106 will move on hunting zone 105 are set in advance, these positions are searched, to find out the ad-hoc location that shows the strongest correlation between reference block 106 and the object block 103, or has an ad-hoc location of the minimum sad value between reference block 106 and the object block 103, and the reference vector 107 that comprises the offset direction be used as representative from the position of the object block on the primitive frame 101 103 or the position of the object block projected image piece 104 on the reference frame 102 to the ad-hoc location that shows the strongest correlation reference block 106 and the object block 103, or to the vector of the skew of ad-hoc location, as shown in figure 42 with the minimum sad value between reference block 106 and the object block 103.Therefore, the reference vector 107 that points to reference block 106 has the value of being determined by the strongest correlation position of the reference block on the reference frame 102 106, and under the situation of block-matching technique, the strongest correlation position of reference block 106 is the positions that show minimum sad value.
According to block-matching technique, each position in a plurality of positions that will on hunting zone 105, move to for reference block 106, reference block 106 that calculates and the sad value between the object block 103 are registered as the table element in the relevance values table 108 that is stored in the memory by being associated by the reference vector 107 with the position of pointing to reference block 106, as shown in figure 43.For the purpose of simplifying the description, hereinafter, the sad value between reference block 106 and the object block 103 is also referred to as the reference block sad value, and the reference vector 107 of the position of sensing reference block 106 is also referred to as motion vector 110.Therefore, can come to find the motion vector 110 that is associated with minimum reference block sad value by all reference block sad values of in memory, storing of search to find out minimum reference block sad value from relevance values table 108.
As mentioned above, each position in a plurality of positions that will on hunting zone 105, move to for reference block 106, by being associated with reference vector 107, reference block sad value between reference block 106 and the object block 103 is registered as the table element in the relevance values table 108, and relevance values table 10g is also referred to as SAD table 108 later on.The reference block sad value is represented the correlation between reference block 106 and the object block 103.Since the reference block sad value be all pixels in the reference block 106 and the brightness value between all respective pixel in the object block 103 difference absolute value and, therefore be used to write down this difference absolute value each and relevance values table 108 be also referred to as SAD table 108.
As shown in figure 43, each element in the relevance values table 108 all is a relevance values, and this relevance values is the relevance values with the reference block 106 of the corresponding position, address of this element, or the reference block sad value of this position.
Should be noted that hereinbefore the position of object block 103 or reference block 106 is respectively the position of the specific part of object block 103 or reference block 106.The example of specific part is the center of object block 103 or reference block 106.Similarly, as described above, the reference vector 107 that comprises the offset direction be representative from the position of the object block on the primitive frame 101 103 or the position of the object block projected image piece 104 on the reference frame 102 to the position that shows strongest correlation or have a vector of side-play amount of the position of minimum sad value.In Figure 42 and example shown in Figure 43, object block 103 and object block projected image piece 104 all are positioned at the center of frame.
The reference vector 107 that points to reference block 106 and comprise the offset direction is representatives from the position of the position of object block 103 or object block projected image piece 104 to the position that shows strongest correlation or has the vector of side-play amount of the position of minimum sad value.Therefore, if identified the position that shows the strongest correlation between reference block 106 and the object block 103 or have reference block 106 and object block 103 between the position of minimum sad value, so also just identified the value of reference vector 107.That is to say,, so also just identified the value of reference vector 107 if identified the address of the element of the relevance values table 108 in the memory.
Below by with reference to flow chart shown in Figure 44, illustrate in greater detail above-mentioned of the prior art matching treatment.
This flow chart starts from step S1, and here, the reference block Ii that the label 106 among Figure 42 is indicated is designated as have coordinate in hunting zone 105 (vx, position vy).In hunting zone 105, specify the operation of reference block Ii to be equal to the operation of specifying with the corresponding reference vector 107 of reference block Ii.In the exemplary process of flow chart representative shown in Figure 44, (vx, vy) be is being under the situation of origin position coordinate with coordinate (0,0) to coordinate, the coordinate of the position that the reference vector of appointment 107 is pointed.Coordinate (0,0) is the position coordinates of the position coordinates of the object block 103 on the primitive frame 101 or the object block projected image piece 104 on the reference frame 102.The horizontal direction skew that on behalf of reference vector 107 positions pointed of appointment, coordinate vx begin from origin position, and the vertical offset that on behalf of reference vector 107 positions pointed of appointment, coordinate vy begin from the origin position with coordinate (0,0).
(vx, vy) all be is the amount that unit expresses with the pixel to each side-play amount.For example, expression formula vx=+1 has expressed the position that equals the distance of a pixel from origin position (0,0) skew to the right in the horizontal direction.On the other hand, expression formula vx=-1 has expressed the position that equals the distance of a pixel from origin position (0,0) skew left in the horizontal direction.In addition, expression formula vy=+1 has expressed the position that equals the distance of a pixel from origin position (0,0) in the vertical downward direction skew.On the other hand, expression formula vy=-1 has expressed the position that equals the distance of a pixel from origin position (0,0) in direction skew vertically upward.
As mentioned above, (vx is as the coordinate with respect to the position of being pointed to by reference vector 107 of the position of origin position (0,0) vy) to coordinate.Hereinafter, be called the position that reference vector 107 points to for short, so that make the explanation easy to understand as the position of pointing to by reference vector 107 with respect to the position of origin position (0,0).Each position that reference vector 107 points to is said to be and is and reference vector 107 corresponding positions.That is, (vx, it is own that vy) (wherein mark vx and vy are integers) represents reference vector 107 for amount.Therefore, hereinafter, point to the position (vx, vy) (reference vector 107 with coordinate (vx, position vy)) be expressed as reference vector (vx, vy).
As previously mentioned, with the center of hunting zone 105 center or origin position (0,0) as object block projected image piece 104.In the horizontal direction according to the distance among the scope ± Rx of the horizontal limit that has defined hunting zone 105, and in vertical direction according to the distance among the scope ± Ry of the vertical bounds that has defined hunting zone 105, reference block 106 moves to another position from a position on hunting zone 105.In this case, amount (vx, vy) satisfy following relation:
-Rx≤vx≤+ Rx also-Ry≤vy≤+ Ry
In next step S2, (x, point (or pixel) y) are designated as the point among the object block Io that is indicated by label 103 among Figure 42 to have coordinate.Make mark Io that (x, (x, the pixel value of y) locating make mark Ii that (x+vx y+vy) is indicated among the step S1 (vx, vy) point (x+vx, the pixel value of y+vy) locating among the reference block Ii that locates to be provided with in the piece position y) to indicate the point of appointment.Hereinafter, the point among the reference block Ii (x+vx, y+vy) be said to be with object block Io in point (x, y) corresponding point.Then, in next step S3, according to following equation (1) calculating pixel value Io (x, y) and pixel value Ii (x+vx, y+vy) the absolute value α of the difference between:
α＝|Io(x，y)-Ii(x+vx，y+vy)| ——(1)
For (the x that has a few among the object block Io, y) and their all respective point (x+vx in reference block Ii, y+vy), calculate the absolute value α of above-mentioned difference, and the absolute value of the difference that representative is calculated object block Io and reference block Ii α's and sad value be stored in and point to reference vector (vx, the place, address of the table element that vy) is associated of the current location of reference block Ii.That is to say that SAD is stored as the element of relevance values table 108 as the reference value table element 109 that is associated with reference block Ii.In order to calculate such sad value, at next step S4, the absolute value α of the difference that finds at step S3 is added to cumulatively to be stored as and reference vector (vx, vy) the interim sad value of the reference value table element 109 that is associated of reference block Ii pointed is with as the sad value that calculates so far.Result as the process that the absolute value α accumulation of all differences is sued for peace, obtain final sad value SAD (vx, vy), described all the difference absolute value α be as described above to the (x that has a few among the object block Io, y) and their all respective point in reference block Ii (x+vx y+vy) calculates.Therefore, final sad value SAD (vx, vy) can express by following equation (2):
SAD(vx，vy)＝∑α＝∑|Io(x，y)-Ii(x+vx，y+vy)| ——(2)
Then, the flow process of of the prior art matching treatment proceeds to next step S5, generation about whether among the object block Io have a few (x, y) and their all respective point in reference block Ii (x+vx y+vy) has carried out the judged result of the process of step S3 and S4.If the judged result indication is not as yet to the (x that has a few among the object block Io, y) and their all respective point (x+vx in reference block Ii, y+vy) judged result of the process of execution in step S3 and S4 all, the flow process of a matching treatment then of the prior art is returned step S2, at step S2, (x, another some y) is designated as another point among the object block Io to have coordinate.Then, step S3 after the repeating step S2 and the process of S4.
If indicate to the (x that has a few among the object block Io in the judged result that step S5 produces, y) and their all respective point in reference block Ii (x+vx y+vy) has carried out the process of step S3 and S4, then reference vector (vx, final sad value SAD vy) (vx, vy) found.The flow process of of the prior art matching treatment proceeds to step S6, produce about whether to all reference block locations in the hunting zone 105 (promptly to all reference vectors (vx, vy)) all executed the judged result of step S2 to the process of S5.
If the judged result that produces at step S6 place indication is not as yet to all reference vector (vx, vy) execution in step S2 is to the process of S5, then relevant have the flow process of the piece matching treatment in the technology to return step S1, at step S1, another reference vector (vx, vy) another reference block Ii pointed is set at another piece position in the hunting zone 105 (vx vy) locates.Then, the process of repeating step S1 and subsequent step.
If indicate to all reference block locations in the hunting zone 105 or to all reference vector (vx in the judged result that step S6 produces, vy) all executed the process of step S2 to S5, then all elements of relevance values table 108 all be filled with final sad value (vx, vy).The flow process of of the prior art matching treatment proceeds to step S7.(vx, vy) minimum value among is detected as the minimum value of representing the strongest correlation between object block 103 and the reference block 106 to the final sad value of all of storing in all elements of relevance values table 108.Then, at next step S8, point to the conduct be included in the relevance values table 108 and be used to store minimum final sad value (vx, (vx vy) is identified as aforementioned movement vector 110 to the reference vector of the element address of element vy).Make mark SAD (mx, my) indicate minimum final sad value (vx, vy), and mark vector (mx, my) indicate to point to the conduct storage SAD that is included in the relevance values table 108 (mx, the reference vector of the element address of element my) (vx, vy) or indicate motion vector 110.
As mentioned above, carry out of the prior art matching treatment being used for object block 103, with the vector of determining object block 103 (mx, my).
Of the prior art matching treatment technology has very long history.Conduct is used for coming the sensorless technology of compensating images pick device to be introduced at the influence that hand tremor is trembled this technology in late 1980s at first.Therefore, before the invention digital consumer, this technology has been used for a long time itself.
After this, though proposed a lot of inventions, comprise such as USP 3303312 (patent documentation 1) and Japanese patent application disclosing the invention of describing in the No.Hei 6-86149 documents such as (patent documentations 2) in early days, but the influence of trembling at hand tremor comes the successful result that the sensorless technology of compensating images pick device is developed provides video camera to a certain extent.But, since have better performance and the more low-cost gyro sensor of small size come into the market, so be used at present the technology of coming the market place of the sensorless technology of compensating images pick device almost completely to be used as to utilize gyro sensor at the influence that hand tremor is trembled be used for influence the technology of compensating images pick device to replace at this.
Be used at the maximum reason that the influence that hand tremor is trembled comes the market place of the sensorless technology of compensating images pick device gradually to lose be: be difficult to improve from the precision of the process by adopting the motion vector detection error that block-matching technique identifies, and by utilizing significantly improving of convenience that gyro sensor obtains.
Another reason is the following fact: be considered to the common accuracy of detection of shortcoming that is used for coming at the influence that hand tremor is trembled the technology of compensating images pick device as the technology of utilizing gyro sensor, do not throw into question in the moving picture photographing field as the main application of this technology.That is to say,, still, detect the process of motion vector and need not carry out with the high accuracy that pixel precision was provided even can not provide the characteristic that demonstrates high accuracy (for example pixel precision) such as transducers such as gyro sensors.On the other hand, in the operating period of taking moving picture, the influence of trembling at hand tremor under not having the situation of transducer comes in the process of compensating images pick device, and the mistake that causes of the motion vector of being discerned does not mistakenly throw into question sometimes.
On the other hand, in recent years, progress at full speed in the trend of the pixel count of the universal rapidly and raising composition picture of digital camera, begun to cause the new problem that is caused by following needs: need compensate in the environment with low light photograph the still picture of taking (therefore long time for exposure of needs) at the influence that hand tremor is trembled, be available although utilize the image pick-up device such as transducers such as gyro sensors.That is to say that the new problem that exposes gradually in industry is the problem that shortcoming caused that before illustrates as the shortcoming of gyro sensor in recent years, the shortcoming of gyro sensor becomes problem gradually in other words.
Compensate in the process of obtainable consumption device in the current market in the influence of trembling at hand tremor, the amount that hand tremor is trembled is measured by using roll gyroscope transducer or acceleration transducer, and be fed back to the mechanical system of carrying out High-speed Control, tremble to prevent the image that is incident upon such as on the imagers such as CCD (charge coupled device) imager or CMOS (complementary metal oxide semiconductors (CMOS)) imager.
As mechanical system, proposed to comprise system's (or comprising the module that wherein is integrated with imager) of lens, prism and imager.Lens, prism and imager are hereinafter referred to as lens skew, prism skew and imager skew.
If come the compensating images pick device by the influence of adopting said method to tremble at hand tremor, then this process will produce the delay error of the feedback of mechanical system, or be used to avoid the evaluated error and the departure of feedback delay.Feedback delay sum of errors departure is superimposed on the aforementioned trueness error as the error of gyro sensor itself.Therefore, coming in the process of compensating images pick device in the influence of trembling at hand tremor, is unusual difficulty based on the compensation of pixel precision.
Although there is serious problem, promptly, usually, aforementioned by only carrying out by utilizing coeval transducer to come to come the process of compensating images pick device to be difficult to obtain precision at the influence that hand tremor is trembled, if but can not break away from the influence that hand tremor is trembled, the image pick-up device through compensation of market preferr it that way is because it can reduce the influence that hand tremor is trembled.
But, along with pixel size reduces to increase with the increasing pixel count of tackling following expection, market awareness only is a matter of time to the following fact, the described fact is, by utilizing coeval transducer to come to come at the influence that hand tremor is trembled the restriction of the process of compensating images pick device, and pixel size reduces and pixel count increases and the pixel precision that the causes gap between improving is increasing.
Tremble in no transducer hand tremor and to experience under the situation of pained failure when influence that compensation technique attempts not use transducer to tremble at hand tremor compensates video camera, on the other hand, usually, detection that vector is trembled in the hand tremor with high pixel precision that comprises the rotational component on the axis of rolling (roll-axis) direction can be realized, transducer can also be removed and such as machineries such as lens skews.Therefore, from the cost viewpoint of measures, it is very outstanding that compensation technique is trembled in no transducer hand tremor.
However, in the expansion of the technology of the block-matching technique that depends on prior art, forming the number of elements of aforementioned relevance values table 108 (or SAD table 108) and the pixel count on picture increases pro rata.Therefore, the circuit that is difficult to have by use attainable scale realizes being used for the process that detected magnitude surpasses the motion vector of the still picture that manifests in the contemporary display frame of five mega pixels.
Image pick-up device manufacturer attempts various effort and attempts to remove to be used for detecting and be no more than 170 in size in the past, meet with under the background of pained failure when the circuit of vector is trembled in the hand tremor of the NTSC that manifests in the display frame of 000 pixel (national television system committee) motion picture, in trembling the process of vector, the hand tremor of the NTSC motion picture that the speed that detects with 60fps (frame per second) produces to use narrow hand tremor to tremble the hunting zone, but under the situation of still picture, the speed of 3fps is taken as prerequisite, make hand tremor tremble the hunting zone and become very big that this feasible problem that has existed becomes and more is difficult to solve.This is because the increase that the hunting zone is trembled in pixel count on the number of elements of forming relevance values table 108 and the picture and hand tremor with being in proportion.
Some documents disclose the method that compensation technique is trembled in the no transducer hand tremor that realizes being used for still picture, and particularly, described document is Japanese patent application No.Hei 7-283999 (patent documentation 3) openly in early days.According to disclosed technology in the patent documentation 3, a kind of algorithm is disclosed, wherein during not taking place, the short exposure time that hand tremor trembles takes some continuous still pictures.It is found that vector is trembled in hand tremor between the static images, and a plurality of still pictures of taking continuously between exposure period trembled according to their hand tremor vector parallel mobile in by stack each other (or being superimposed on the mean value of the still picture of taking continuously between exposure period), do not have the influence that hand tremor trembles and do not have the final high-quality rest image of low light with generation according to noise.
Japanese patent application discloses No.2005-38396 (patent documentation 4) in early days and has proposed the realized technology on the level that can realize.Patent documentation 4 disclosed technology comprise being used to find and are used for as the result of the conversion process of reduction original image and the device of the motion vector of the picture size that obtains, and allow public SAD table by a plurality of shared devices.The public SAD table of reduction original image and permission is extraordinary methods that realization reduces the size of SAD table 108 by a plurality of share technique, and this method also is used for such as other fields such as motion vector that detects MPEG (Motion Picture Experts Group) picture compression system and the changes of detection picture.
But disclosed algorithm has such problem in the patent documentation 4, and promptly its expensive time is carried out the transfer process and the very visit of carrying out using of the memory of size that has of reduction original image in this process.An example of memory is DRAM (dynamic ram (random access storage device)).Because this algorithm uses the device that is conducted interviews by a plurality of shared relevance values tables 108 (or SAD table 108) on the basis that is configured to share in the time, therefore the time that memory is conducted interviews becomes very long.The very long time that memory is conducted interviews has also increased the time of carrying out based on the processing of this algorithm inevitably.Because in order to shorten the system delay time, the influence of trembling at hand tremor comes the process of compensating images pick device to carry out with real-time mode, therefore carry out based on this algorithm processing spent has caused problem for a long time.
In addition,, must before conversion process, use low pass filter to carry out preliminary treatment, so that removal folding (aliasing) and low light are according to noise in order to carry out the conversion process of reduction original image.Because the characteristic of low pass filter changes according to the reduction factor of transfer process, particularly, under the situation of vertical direction low pass filter, many tap number character filter is used, but several line storages and processor logic are necessary, the problem that has caused circuit size to increase.
On the other hand, openly No.Hei 6-86149 (patent documentation 5) and Japanese patent application disclose the algorithm that No.2004-343843 documents such as (patent documentations 6) has proposed not use block-matching technique in early days in early days such as Japanese patent application.In the algorithm that is proposed each is all utilized and is configured to detect a plurality of points (each point all is taken as the characteristic point on two two field pictures for some reason) and on the basis of detected characteristic point two two field pictures is relative to each other connection so that find the device of global vector, and described global vector is that vector is trembled in the hand tremor that is used for the whole aspect of every two field picture.Perhaps, detect the characteristic point of one of two two field pictures, and all center on a plurality of zones of a detected characteristic point, with respect to another two field picture execution block matching treatment for each.
Patent documentation 5 and 6 disclosed algorithms have all reduced the size of treatment circuit, and all are also to be desirable therefore very effectively.But these algorithm effects depend on the characteristic point that identifies what really serve as the public characteristic of the characteristic of whole aspect of two two field pictures and two two field pictures and can be reduced by high level ground efficiently very much.As long as all things in the universe all are taken as the image pickup object of consumption image pick-up device, block-matching technique will be considered to slim lead in patent documentation 5 and 6 disclosed algorithms aspect robustness.
Summary of the invention
As previously mentioned, in such as image pick-up devices such as digital cameras, when anticipating better performance demands, will drop into the picture element density of more and more making great efforts to increase image future.In the case, by adopting the sensorless technology that does not use gyroscope (or accelerator) transducer to realize that coming the process of compensating images pick device at the influence that the hand tremor that takes place in the operation of taking still picture is trembled is highly significant.
In order to realize such process, as previously mentioned, adopt as trembling motion vector and come the method for compensating images pick device to be considered to up-and-coming method by the next influence of trembling of vector that use identifies at hand tremor by adopting block-matching technique to discern hand tremor with no approach sensor.But, under present case, adopt the solution of block-matching technique to have such problem, promptly do not make the motion of all requirements of satisfying little treatment circuit scale, high processing rate and outstanding robustness as yet.
The greatest problem of block-matching technique is that the size by the increase of relevance values table causes.In above-mentioned example, the relevance values table is the SAD table.As previously mentioned, must have under this prerequisite of size of at least five mega pixels at the image that in digital camera, generates at present, increasing pro rata with the pixel quantity of composition diagram picture inevitably of the size of relevance values table, in addition, under the situation of still picture, adopt the speed of about 3fps.Therefore, for still picture, it is necessary that the hunting zone is trembled in 10 times the hand tremor that the size of hunting zone is trembled in the hand tremor that size is approximately the motion picture that generates with the speed of 60fps.The size increase that the hunting zone is trembled in hand tremor is equal to the increase of relevance values table size, and the size increase of relevance values table is considered to the greatest problem that block-matching technique occurs.
The assessment result that many users provide is clearly indicated, and supposes that the whole area of frame is 100, then hand tremor tremble the size of hunting zone approximately be ± 10%.Under the situation of high-performance image pick device, the pixel quantity of composition diagram picture has been assumed to be 12,000,000, and former state utilizes adopted prior art, and the size of necessary relevance values table is estimated as about 80 megabits.In addition, if attempt to satisfy desirable processing speed, then must use SRAM (static RAM (SRAM) (random access storage device)) as the memory that is used for memory dependency value table.Although the semiconductor technology rules have allegedly obtained progress, the size of about 80 megabits still head and shoulders above real level, it is than high about 3 numerical digits of the value of reality.
In order to address the above problem, inventor of the present invention provides such image processing method and device, and it allows significantly dwindling by the size that adopts block-matching technique to discern employed relevance values table in the process of the motion vector between two frames.
In addition, in the method suggested in the prior art in piece coupling field, about as reduce in patent documentation 4 the disclosed technology of image with the technology of the size of dwindling the relevance values table by the carries out image transfer process, the inventor has pointed out following two problems.A problem is to have increased the processing time and increased memory size, and they are to be caused by the image conversion process that reduces image.Another problem is the increase that the filter of the suitable robust that is achieved as follows causes the treatment circuit size, and this filter is used for avoiding folding at the image conversion process of carrying out the reduction image.Therefore, wish to address these problems.
In order to address the above problem, as invention according to embodiment, a kind of image processing apparatus execution following steps that are used to discern motion vector:
A plurality of reference blocks are set as such reference block in the hunting zone that is provided with on reference picture, in the described reference block each has the size that equates with object block, and the object block that described object block conduct has the pre-sizing that covers a plurality of pixels is set at the pre-position on the target picture;
Search for specific reference block, described specific reference block has the strongest correlation in the correlation between reference block and object block; And
Based on specific reference block on the hunting zone on the reference picture from target picture on the skew of corresponding position, precalculated position of object block, detect motion vector.This image processing apparatus has:
The relevance values calculation element, it is configured to pixel value and the pixel value that is positioned at a plurality of pixels at the respective point place on the object block based on a plurality of pixels that are positioned at the some place on each reference block, calculate relevance values, each relevance values is represented the correlation between in the reference block and the object block;
Table creating device, it is configured to generate through reduction relevance values table by following operation:
To be stored in by the relevance values former state that the relevance values calculation element calculates at reference vector in the table element that is included in reduction relevance values table, as with as the table element that is associated through the reduction reference vector that obtains with the result of predetermined reduction factor reduction reference vector, described reference vector representative reference block on the reference picture from target picture on the skew of corresponding position, precalculated position of object block; Perhaps
By dividing by the relevance values calculation element at the relevance values that is calculated with the reference vector that produces through reducing reference vector by reduction, find a plurality of component relevance values, in described a plurality of component relevance values each and position are near being associated through in the reduction reference vector specific one of reduction reference vector a plurality of approaching, and in the component relevance values each is stored in the table element that is included in reduction relevance values table, as with the specific approaching table element that is associated through the reduction reference vector; And
Motion vector calculation device, it is configured to find the motion vector between target picture and the reference picture based on the component relevance values that is stored in reduction relevance values table.
Second invention according to another embodiment is invention according to the foregoing description.Table creating device also has:
Component relevance values calculation element, if it is configured to location a plurality of approaching through the reduction reference vector near through the reduction reference vector, then by dividing by the relevance values calculation element at the relevance values that is calculated with the above-mentioned reference vector that produces through reducing reference vector by reduction, find a plurality of component relevance values, in the described component relevance values each and position near through reduction reference vector a plurality of approaching in the reduction reference vector specific one be associated, and in the component relevance values each is stored in the table element that is included in reduction relevance values table, as with the specific approaching table element that is associated through the reduction reference vector; And
Final component relevance values calculation element, it is configured to find final component relevance values, described final component relevance values representative by will be added to by the new component relevance values that component relevance values calculation element newly calculates the accumulation that obtains on the previous component relevance values that has been stored in as the component relevance values that had before calculated by component relevance values calculation element in the table element of reduction relevance values table with.
Have the relevance values calculation element that adopts in the image processing apparatus according to the configuration of the invention described above in the mode identical, calculate the object block represented on the target frame and the relevance values of the correlation between each reference block in the hunting zone on the reference frame with image processing apparatus of the prior art.
But, the image processing apparatus that has according to configuration of the present invention comprises table creating device, be used to calculate and the component relevance values that is associated through the reduction reference vector that obtains as the result of the process that original reference vector before reducing is reduced, and the component relevance values that calculates that storage obtains as the result of the process that original relevance values table before reducing is reduced, rather than the storage relevance values of represent correlation between object block and the reference block as and the table element that joins corresponding to original reference vector correlation before the reduction of reference block.
The process that original relevance values table before reducing is reduced is the process of dwindling the size of table, and it has obtained table element still less.On the other hand, the process that original reference vector before reducing is reduced is the process of dwindling the preceding original reference vector of reduction, and it has obtained with this process original reference vector before as many through the reduction reference vector.Therefore, each table element through reducing the relevance values table is all with approaching through reducing a plurality of approaching being associated through one of reduction reference vector of reference vector through reduction reference vector or position.Table element quantity in reduction relevance values table be by the process that the relevance values table is reduced in the employed reduction factor determine.
In other words, be the table that is used for through the two field picture of reduction through reduction relevance values table, this will obtain as the result with the supposition process of coming through the reduction factor of reduction reference vector the primitive frame image is reduced through two field picture of reduction.But in fact, the size of target frame image, reference frame image, object block and each reference block does not change.The size of relevance values table and the length of each reference vector have been dwindled.
Searched through reduction relevance values table, to find out the particular table element of the final component relevance values that is used to store the strongest correlation of representing between object block and the reference block.Usually, representing the final component relevance values of the strongest correlation between object block and the reference block is minimum value in all final component relevance values that are stored in reduction relevance values table.As mentioned above, each table element through reducing the relevance values table is with reference vector or position are approaching to be associated through one of reduction reference vector through a plurality of of reduction reference vector through reducing.Therefore, from be associated with specific table element through the reduction reference vector or be associated with specific table element approaching through one of reduction reference vector, select requiredly, and reduce the factor and find the original motion vector by will be required multiply by through the reduction reference vector through the reduction reference vector.
According to the present invention, the process of reduction image is unnecessary.Therefore, both do not reduced the process of image, also the required memory span of this process does not increase.
As mentioned above, each table element through reducing the relevance values table is with approaching through reducing a plurality of approaching being associated through one of reduction reference vector of reference vector through reduction reference vector or position.Therefore, through the reduction reference vector not on one-to-one basis with corresponding through the table element of reduction relevance values table.If table element is not with corresponding through the reduction reference vector, promptly, if table element and position near through reduction reference vector a plurality of in the reduction reference vector specific one corresponding, then the table creating device of the present invention according to another embodiment calculates for this specific approaching component relevance values through the reduction reference vector.Then, table creating device is stored in the component relevance values that calculates specific approaching in the table element that the reduction reference vector is associated with this.
In the mode identical with the block-matching technique of prior art, the relevance values calculation element is by carrying out cumulative process, repeatedly the value that will calculate be added to interim and, up to all being considered, thereby find any relevance values through the reduction reference vector with all pixels in the reference block that is associated of reduction reference vector.
As mentioned above, by carry out the value that will calculate be added to interim and cumulative process, any specific at what be associated through the reduction reference vector with reference block, calculated the object block represented in the primitive frame and the relevance values of the correlation between the reference block in the reference frame.Replacement will be stored in the table element of reduction relevance values table at the specific relevance values former state of calculating through the reduction reference vector, can be by dividing at specific relevance values through the calculating of reduction reference vector, come at the position near specific through the reduction reference vector through the reduction reference vector, find a plurality of component relevance values.By dividing at the specific relevance values that calculates through the reduction reference vector to find the process of component relevance values itself to be equal to the suitable filter process that be adapted to reduce the factor through the reduction reference vector through the reduction reference vector near specific at the position.Therefore, being similar to the low pass filter that image is reduced necessary filter realizes dispensable.
According to the present invention, representing reference frame from the process of the motion vector of the skew of target frame by adopting block-matching technique to detect, the size of the relevance values table of Shi Yonging can obviously be dwindled during the course.In addition because the process that image is reduced is dispensable, so both processing time of causing of the process of image not being reduced increase, also the memory span that causes of not reduction process increases.In addition, owing to can on the basis of all pixels in the object block of target frame, find relevance values, realize it being unnecessary so be similar to the low pass filter that image is reduced necessary filter, therefore can not produce the treatment circuit scale increases such problem.
Description of drawings
Fig. 1 is the key diagram of reference when describing the overview of image processing method according to an embodiment of the invention;
Fig. 2 A and Fig. 2 B are the key diagrams of reference when describing the overview of image processing method according to an embodiment of the invention;
Fig. 3 is the key diagram of reference when describing the overview of image processing method according to an embodiment of the invention;
Fig. 4 A and Fig. 4 B are at the key diagram of describing reference when detecting being used to of realizing the process of accurate motion vector according to first typical case of image processing method according to an embodiment of the invention;
Fig. 5 is at the key diagram of describing reference when detecting being used to of realizing the process of accurate motion vector according to first typical case of image processing method according to an embodiment of the invention;
Fig. 6 is the key diagram of reference when describing the overview of image processing method according to an embodiment of the invention;
Fig. 7 is the key diagram of reference when describing the overview of image processing method according to an embodiment of the invention;
Fig. 8 A and Fig. 8 B are at the key diagram of describing reference when detecting being used to of realizing the process of accurate motion vector according to first typical case of image processing method according to an embodiment of the invention;
Fig. 9 is at the key diagram of describing reference when detecting being used to of realizing the process of accurate motion vector according to first typical case of image processing method according to an embodiment of the invention;
Figure 10 is at the key diagram of describing reference when detecting being used to of realizing the process of accurate motion vector according to first typical case of image processing method according to an embodiment of the invention;
Figure 11 A and Figure 11 B are at the key diagram of describing reference when detecting being used to of realizing the process of accurate motion vector according to first typical case of image processing method according to an embodiment of the invention;
Figure 12 A, Figure 12 B, Figure 12 C and Figure 12 D are at the key diagram of describing reference when detecting being used to of realizing the process of accurate motion vector according to first typical case of image processing method according to an embodiment of the invention;
Figure 13 is at the key diagram of describing reference when detecting being used to of realizing the process of accurate motion vector according to first typical case of image processing method according to an embodiment of the invention;
Figure 14 is at the key diagram of describing reference when detecting being used to of realizing the process of accurate motion vector according to first typical case of image processing method according to an embodiment of the invention;
Figure 15 A and Figure 15 B are at the key diagram of describing reference when detecting being used to of realizing the process of accurate motion vector according to second typical case of image processing method according to an embodiment of the invention;
Figure 16 is at the key diagram of describing reference when detecting being used to of realizing the process of accurate motion vector according to second typical case of image processing method according to an embodiment of the invention;
Figure 17 is at the key diagram of describing reference when detecting being used to of realizing the process of accurate motion vector according to second typical case of image processing method according to an embodiment of the invention;
Figure 18 A, Figure 18 B, Figure 18 C and Figure 18 D are at the key diagram of describing reference when detecting being used to of realizing the process of accurate motion vector according to second typical case of image processing method according to an embodiment of the invention;
Figure 19 is the key diagram of reference when describing the performance of image processing method according to an embodiment of the invention;
Figure 20 is the key diagram of reference when describing the overview of image processing method according to another embodiment of the present invention;
Figure 21 is the key diagram that the characteristic with the characteristic of image processing method according to an embodiment of the invention and method of the prior art compares;
Figure 22 is the key diagram that the characteristic with the characteristic of image processing method according to an embodiment of the invention and method of the prior art compares;
Figure 23 is the key diagram that the characteristic with the characteristic of image processing method according to an embodiment of the invention and method of the prior art compares;
Figure 24 shows the block diagram according to the Typical Disposition of the image processing apparatus of the first embodiment of the present invention;
Figure 25 shows and represents the flow chart of realizing according to according to the typical case of first in the image processing apparatus of the first embodiment of the present invention that is performed the processing that detects motion vector;
Figure 26 shows the continuation part of representing according to the flow chart of realizing according to the typical case of first in the image processing apparatus of the first embodiment of the present invention that is performed the processing that detects motion vector;
Figure 27 shows and represents the flow chart of realizing according to according to the typical case of second in the image processing apparatus of the first embodiment of the present invention that is performed the processing that detects motion vector;
Figure 28 shows the continuation part of representing according to the flow chart of realizing according to the typical case of second in the image processing apparatus of the first embodiment of the present invention that is performed the processing that detects motion vector;
Figure 29 shows and represents the flow chart of realizing according to according to the typical case of the 3rd in the image processing apparatus of the first embodiment of the present invention that is performed the processing that detects motion vector;
Figure 30 shows the continuation part of representing according to the flow chart of realizing according to the typical case of the 3rd in the image processing apparatus of the first embodiment of the present invention that is performed the processing that detects motion vector;
Figure 31 shows another continuation part of representing according to the flow chart of realizing according to the typical case of the 3rd in the image processing apparatus of the first embodiment of the present invention that is performed the processing that detects motion vector;
Figure 32 shows the another continuation part of representing according to the flow chart of realizing according to the typical case of the 3rd in the image processing apparatus of the first embodiment of the present invention that is performed the processing that detects motion vector;
Figure 33 shows the block diagram of the Typical Disposition of image processing apparatus according to a second embodiment of the present invention;
The key diagram of reference when Figure 34 is the processing of the detection motion vector in the image processing apparatus of describing according to a second embodiment of the present invention;
The key diagram of reference when Figure 35 is the processing of the detection motion vector in the image processing apparatus of describing according to a second embodiment of the present invention;
Figure 36 shows representative is performed the processing that detects motion vector in image processing apparatus according to a second embodiment of the present invention flow chart;
Figure 37 shows representative is performed the flow chart of the processing that detects motion vector in image processing apparatus according to a second embodiment of the present invention continuation part;
Figure 38 shows the block diagram of Typical Disposition of the image processing apparatus of a third embodiment in accordance with the invention;
Figure 39 be the influence of in describing the image processing apparatus of a third embodiment in accordance with the invention, trembling at hand tremor come compensating images processing the time reference key diagram;
Figure 40 shows the block diagram of Typical Disposition of the image processing apparatus of a fourth embodiment in accordance with the invention;
Figure 41 is the key diagram of reference when the image processing apparatus of describing according to another embodiment of the present invention;
Figure 42 is at the key diagram of describing reference when detecting the processing of motion vector by the block-matching technique that adopts prior art;
Figure 43 is at the key diagram of describing reference when detecting the processing of motion vector by the block-matching technique that adopts prior art; And
Figure 44 is at the key diagram of describing reference when detecting the processing of motion vector by the block-matching technique that adopts prior art.
Embodiment
Realize the embodiment of image processing method provided by the invention and image processing apparatus below with reference to description of drawings.
[realizing the overview of the embodiment of image processing method provided by the invention]
In the embodiment that realizes image processing method provided by the invention, the motion vector between two frames is also by adopting above-mentioned block-matching technique identification.Not that the relevance values that will represent the correlation between object block and the reference block is stored as the table element that is associated with reference vector corresponding to reference block, but the relevance values table is reduced to less table element, and each reference vector is reduced to through the reduction reference vector.Through the process of reduction relevance values table and each reference vector obtained less table element and with this process before the vector as much through the reduction reference vector.Therefore, not being stored in the table element at the specific relevance values former state that calculates through the reduction reference vector, but by cutting apart relevance values, a plurality of component relevance values may at the position near specific through the reduction reference vector through the reduction reference vector and found, and as being stored in the different table elements with the approaching element that is associated through the reduction reference vector.
It is quite little comparing according to the size of the relevance values table that uses in the size of the relevance values table of embodiment and the image processing apparatus of the prior art.Should be noted that relevance values is a sad value among the embodiment of explanation hereinafter, therefore, the SAD table is generated as the relevance values table.Therefore, in an embodiment, the relevance values of representing the strongest correlation between object block and the reference block is minimum sad value.
Fig. 1 is figure in explanation reference during according to the overview of the image processing method of embodiment to Fig. 3.More specifically, Fig. 1 show relevance values table TBLo of the prior art (be in the present embodiment SAD table) and the conduct that generates by image processing method according to embodiment through the graph of a relation of reduction relevance values table TBLs through reduction SAD table.
As described in earlier in respect of figures 42, equally in the present embodiment, the center that the hunting zone is set to this scope overlaps with the center of reference frame, and the center of reference frame is with corresponding with the center of reference vector target frame relatively.The center of target frame also with target frame in the conduct that comprises to overlap with the center of the object block of the reference block that on the hunting zone, moves to another position piece relatively from a position.It at the sad value of the reference block of a position absolute value sum of the difference between the pixel value of the pixel value of each pixel on the object block and the respective pixel on the reference block.
In the image processing apparatus of prior art, the relevance values that calculates is stored in SAD table TBLo as the table element tb1 that is arranged in place, the corresponding address of the reference vector RV of just processed reference block, as shown in Figure 1.
Therefore, in block-matching technique, the reference vector RV of the value of the skew on the representative frame image from the object block to the reference block is associated with relevance values, and described relevance values is being stored as the table element that is used for reference block on the one-to-one basis in relevance values table TBLo.That is to say that the quantity of forming the table element of relevance values table TBLo in the prior art equals the quantity of the reference vector RV that can be provided with in the hunting zone.
On the other hand, in the block-matching technique according to present embodiment, as Fig. 1, shown in Fig. 2 A and Fig. 2 B, each the reference vector RV that points to just processed reference block is reduced to through reduction reference vector CV with reduction factor 1/n, and wherein mark n indicates an integer.
Hereinafter, in order to make the explanation easy to understand, the horizontal direction reduction factor is assumed to be and equals the vertical direction reduction factor.But the horizontal direction reduction factor and the vertical direction reduction factor also can be set to the value that differs from one another independently of one another.In addition, as mentioned below, the horizontal direction reduction factor and the vertical direction reduction factor even be set to any mark (for example 1/m and 1/n, wherein mark m and n indicate integer) independently of one another are so that provide the flexibility of higher degree and the convenience of higher degree.
In addition, very similar with image processing method of the prior art in the present embodiment, the position of object block is used as the origin position (0,0) of the center of hunting zone.The horizontal direction of reference vector RV and vertical direction component (vx, the integer of the horizontal direction that vy) to be respectively representative measure from origin position (0,0) and the value of vertical direction.Hereinafter, have horizontal direction and vertical direction component (vx, reference vector RV vy) be called as reference vector RV (vx, vy).
Reference vector RV (vx, vy) with reduction factor 1/n be reduced to through reduction reference vector CV (vx/n, vy/n).Therefore, (vx/n, (vx/n vy/n) may not be integer for horizontal direction vy/n) and vertical direction component through reduction reference vector CV.That is to say that in some cases, they can be the values that comprises fractional part.Therefore, iff will at the original reference vector RV relevance values that calculate before the reduction as with present embodiment in non-integer vx/n and the immediate element that is associated through the reduction reference vector of vy/n value through reduction reference vector CV be stored as the element that in reducing the relevance values table, comprises, then will generated error.In addition, though the number of elements in reduction relevance values table less than the number of elements in the original relevance values table, the quantity of the original reference vector RV before the quantity through reducing reference vector CV equals to reduce.Therefore, not to be associated with element in reduction relevance values table on the man-to-man basis through reduction reference vector CV.
Approaching reference vector NV (vx/n, vy/n) be defined as such through the reduction reference vector: it has and the reference vector RV (vx/n that is reduced, vy/n) the immediate integer vx/n of non-integer vx/n value value, or with by reference vector RV (vx/n, the immediate integer vy/n of non-integer vy/n value value vy/n) of being reduced.There are a plurality of approaching reference vector NV in each reference vector RV that is reduced.Because as mentioned above, CV is not associated with element on one-to-one basis through the reduction reference vector, therefore in the present embodiment, the sad value of calculating at the original reference vector meter before the reduction of reduction reference vector CV is not stored in by former state in the element of reduction relevance values table.But by adopting the linear weighted function distribution technique, the sad value that calculates at the original reference vector before the reduction of reduction reference vector CV is split into and the approaching as many component sad value of approaching reference vector NV through reduction reference vector CV in position.
Have non-integer vx/n and vy/n value (vx/n vy/n) is not associated with table element tb1 through the reduction table element through reduction reference vector CV.But, each all have integer vx/n and vy/n value approaching reference vector NV (vx/n, vy/n) on one-to-one basis with through the reduction table element table element tb1 be associated.Therefore, in the present embodiment, with with according to the block-matching technique of prior art sad value is stored as with reduce before the cumulative process carried out of the identical mode of the process of element of original reference vector correlation connection in, the component sad value that calculates by employing linear weighted function distribution technique is stored among the table element tb1 that is associated with approaching reference vector NV.The linear weighted function distribution technique is based on by the position of pointing to through reduction reference vector CV with by the method for position near the distance between the position of the approaching reference vector NV sensing through reducing reference vector CV.
More specifically, being used in the weight that finds the process of the component relevance values of approaching reference vector NV to use by employing linear weighted function distribution technique is to determine by the position of pointing to through reduction reference vector CV with on by the basis of position near the distance between the position of the approaching reference vector NV sensing through reducing reference vector CV, as mentioned above.Therefore, with the identical mode of block-matching technique of aforementioned prior art to interim and the current value that calculates that adds up, find the final component relevance values of each approaching reference vector NV.
Should be noted that if (vx/n, (vx/n vy/n) is integer to value vy/n), and then (vx/n, vy/n) this is associated with element through reduction relevance values table on one-to-one basis through reduction reference vector CV through reduction reference vector CV.Therefore, with (vx/n in the table element that vy/n) is associated, can store (vx, vy) corresponding sad value itself with reference vector RV through reduction reference vector CV.In addition, (vx/n under situation vy/n), there is no need to determine (vx/n, a plurality of approaching reference vector NV vy/n) through reduction reference vector CV through reduction reference vector CV this.
Next by providing following example above-mentioned processing is described.As previously mentioned, object block is prevented and treated at origin position (0,0) and is located.In this case, suppose that the reference vector RV shown in Fig. 2 A (3 ,-5) reduces on level and vertical both direction with reduction factor 1/n (=1/4), with obtain shown in Fig. 2 B through reducing reference vector CV (0.75 ,-1.25).
As mentioned above, each value through reduction reference vector CV that obtains all comprises fractional part, and indication is necessary to determine through reducing a plurality of approaching reference vector NV of reference vector CV.
In the example depicted in fig. 3, at through reduction reference vector CV (0.75 ,-1.25), determine four approaching reference vector NV1 (1 ,-1), NV2 (1 ,-2), NV3 (0 ,-1) and NV4 (0 ,-2).Can find out clearly that from the example of Fig. 3 a plurality of this approaching reference vectors are selected, thus the value of each approaching reference vector all be with through the reduction reference vector the immediate integer of value.
In the example depicted in fig. 3, four approaching reference vector NV1, NV2, NV3 and NV4 are the vectors that point to position P1, P2, P3 and P4 (each all is illustrated as a circle) from origin position (0,0) respectively.On the other hand, be the vector that points to the some P0 that is shown mark X from origin position (0,0) through reduction reference vector CV.
Therefore, under the situation of present embodiment, by adopting above-mentioned linear weighted function distribution technique to calculate among four approaching reference vector NV1, NV2, NV3 and NV4 the component relevance values of each, and with the identical mode of block-matching technique to the prior art of interim and the current value that calculates that adds up, find the final component relevance values of each approaching reference vector NV.
Next, determine to find the weight of using in the process of component relevance values of approaching reference vector NV1, NV2, NV3 and NV4 as described below.As mentioned above, point to some P0 (0.75 ,-1.25) through reduction reference vector CV, and approaching reference vector NV1, NV2, NV3 and NV4 point to position P1 (1 ,-1), P2 (1 ,-2), P3 (0 ,-1) and P4 (0 ,-2) respectively.Therefore, (distance between position P0 and the P1): (distance between position P0 and the P2): (distance between position P0 and the P3): (distance between position P0 and the P4)=1: 3: 3: 9.Because weight is inversely proportional to distance, so exemplary weights 9/16,3/16,3/16 and 1/16 is assigned to four approaching reference vector NV1, NV2, NV3 and NV4 respectively.
Suppose that the relevance values that calculates at the original reference vector RV before the reduction of serving as the original value through reducing reference vector CV is S α.In this case, it is found as follows to point to component relevance values SADp1, SADp2, SADp3 and the SADp4 of approaching reference vector NV1, NV2, NV3 and NV4 of position P1, P2, P3 and P4 respectively:
SADp1＝Sα×9/16
SADp2＝Sα×3/16
SADp3＝Sα×3/16
SADp4＝Sα×1/16
With with the identical mode of block-matching technique to the prior art of interim and the current value that calculates that adds up, the final component relevance values of component relevance values SADp1, SADp2, SADp3 and the SADp4 of four approaching reference vector NV1, NV2, NV3 and NV4 is calculated, and be stored in the table element that in reduction relevance values table, comprises, as for pointing to four elements that approaching reference vector NV1, NV2, NV3 and NV4 provides of position P0, P1, P2, P3 and P4 respectively.
In the present embodiment, for each all points to all reference vectors of the reference block that is provided with in the hunting zone, execution is reduced to reference vector through the process of reduction reference vector and is the process at included element calculating component SAD as the element that is associated with approaching reference vector in reduction relevance values table.
Can be clear that very much from the description that has provided, in the present embodiment, with reduction factor 1/n execution each reference vector RV is reduced to through reducing the process of reference vector CV, following process on level and vertical both direction to relevance values table TBLo reduction with identical reduction factor 1/n, so as to generate have the size of being reduced through reduction relevance values table TBLs.Relevance values table TBLo has original size, and is included in the element that is associated with reference vector RV on the one-to-one basis.Then, for each element through reduction relevance values table TBLs calculates the component sad value, this calculating is to carry out by cutting apart at the sad value that is calculated by the reference vector RV reference block pointed that serves as the original value through reducing reference vector CV that is associated with this element.The reader can be with reference to figure 1 to obtain more information.
Therefore, in the present embodiment, the number of elements of forming through reduction relevance values table TBLs is to form (the 1/n of the number of elements of original relevance values table TBLo 2) doubly.That is to say that the size of relevance values table can significantly reduce.
According to the foregoing description, for for each element of reduction relevance values table TBLs, approaching four approaching reference vector NV in reduction reference vector CV are selected, so, from at by serving as the relevance values that calculates through the reference vector RV processed reference block pointed of the original value of reduction reference vector CV, find the component relevance values of selected approaching reference vector NV.In the process of cutting apart based on so-called linear weighted function distribution technique at the relevance values that calculates by the reference vector RV reference block pointed that serves as original value, found the component relevance values of position near approaching reference vector NV through reducing reference vector CV through reducing reference vector CV.But should be noted that select near through the method for the approaching reference vector NV of reduction reference vector CV and for each element through reduction relevance values table TBLs find the linear weighted function distribution technique of component relevance values never be limited to of the present invention those.
For example, as an alternative, corresponding each element through reduction relevance values table TBLs, near selected through 9 of reduction reference vector CV or 16 approaching reference vector NV, then, from at by serving as the relevance values that calculates through the reference vector RV processed reference block pointed of the original value of reduction reference vector CV, find the component relevance values of selected approaching reference vector NV.But, in this case, in the process of cutting apart based on so-called cubic interpolation (cubicinterpolation) technology at the relevance values that calculates by the reference vector RV reference block pointed that serves as original value, find the component relevance values of position near approaching reference vector NV through reducing reference vector CV through reducing reference vector CV.By carrying out these processes, the precision of component relevance values has improved.If emphasize real-time necessity more and reduce the treatment circuit number, it will be more effective then finding the process of the component relevance values of four approaching reference vector NV.
In addition, under the situation of present embodiment, the component relevance values is in the cumulative process of carrying out in mode same as the prior art, be stored as element through reduction relevance values table (through reduction SAD table), in the prior art, for in the position that comprises in the hunting zone of the position that will be moved to as the reference piece each, relevance values is stored as the element of original relevance values table.
But, in the block-matching technique of prior art, reference vector is associated with the address of the element of forming the relevance values table on one-to-one basis, thereby be each reference block calculating relevance values, and described relevance values only is stored in the relevance values table as the element that is associated with reference vector corresponding to reference vector.On the other hand, in the situation according to the technology of present embodiment, reference vector is not associated with the address of forming through the element of reduction relevance values table (or through reduction SAD table) on one-to-one basis.Therefore, the relevance values that calculates for reference block is divided into a plurality of component reference values, and each component reference value all is stored in reduction relevance values table then, as with one of the approaching reference vector that is associated with the component reference value corresponding element.Be very similar to each element in the relevance values table, each each of memory location that all is used for the component reference value that storage computation obtains also is being initialized to 0 at first.
As previously mentioned, be sad value according to the block-matching technique of prior art as the relevance values that the element of relevance values table is stored in the relevance values table.Then, in the situation of the block-matching technique of prior art, the relevance values table is searched, with the table element of the minimum sad value of finding out the object block that is used to store on the indicating target frame and the strongest correlation between the reference block on the reference frame.Then, the reference vector that points to the address of the table element be used to store minimum sad value is taken as and has represented from the position of target frame to the motion vector that moves of the position of reference frame.
On the other hand, in the situation of embodiment, be the component relevance values as the relevance values that the element through reduction relevance values table is stored in reduction relevance values table according to embodiment, it also is a sad value.Then, searched through reduction relevance values table, with find out that the table element that is used for storing minimum sad value and reference frame comprise as a plurality of reference blocks by their approaching reference vectors separately piece pointed, identify motion vector from these approaching reference vectors, because each in these approaching reference vectors needs not to be motion vector accurately.
As the most rational technology from the approaching reference vector identification motion vector that is associated with this table element through reduction relevance values table, by the size through reduction relevance values table being multiply by Integer n (inverse of reduction factor 1/n), be resumed through reduction relevance values table and be original relevance values table.Then, identification as with the element in the original relevance values table of being included in through the corresponding element of detected element of reduction relevance values table.At last, determine the motion vector of the selected element of the original relevance values table of sensing.But this technology can be used for the image processing apparatus of admissible error to a certain extent.
For the accuracy with higher degree detects motion vector, must on the element value that is stored in reduction relevance values table, carry out one of following typical interpolation process.By carrying out one of typical interpolation process, can be with the accuracy detection of original degree motion vector accurately.
[the first typical interpolation process that is used to detect motion vector] with accuracy of higher degree
The such technology of the first typical interpolation process employing that is used to detect motion vector with accuracy of higher degree is wherein by using quadratic surface to be similar to a plurality of relevance values of storing in the element of reduction relevance values table.This technology is the technology that obtains by to the method for using description in the patent documentation of quoting previously 1 through reduction relevance values table.
In the present embodiment, because sad value is used as relevance values, so relevance values is more little, and the indicated correlation of relevance values is strong more.Therefore, in the present embodiment, searched through reduction relevance values table, with the particular table element of finding out the minimum relatedness value that is used to store the indication strongest correlation with as a plurality of reference blocks that in reference frame, comprise of the piece of indicating by its approaching reference vector separately.Can be with the table address precision, search is through the table element of reduction relevance values table, and described table address precision is other precision of integer level.In addition, also be identified with other precision of integer level as in a plurality of approaching table element in the following table section in the zone in reduction relevance values table each, described table section is the center with the particular table element that is detected with other precision of integer level.Then, by adopting least square method, find quadratic surface as representative as the particular table element and in table section detected approaching table element and be stored in the curved surface of the relevance values in reduction relevance values table.Next, represent quadric minimum value of relevance values to be determined, and the position that is confirmed as the relevance values of minimum value is identified as from the position of origin position (0,0) skew.The position of the relevance values that is confirmed as minimum value that identifies is with corresponding as the position that comprises in the region of search on reference frame that demonstrates with the position of the reference block of the strongest correlation of object block.The position of identifying is the position that is included in as the position at the place, address with other precision of mark level in reduction relevance values table (or through reduction SAD table).That at last, points to the position identify is detected as the vector of the position that sensing identifies with other precision of mark level through the reduction reference vector.
Fig. 4 A or Fig. 4 B show the example that quadric process is set.In arbitrary example therein, mark tm indicates the particular table element that identifies with other precision of integer level as the table element of representing the minimum relatedness value.On the other hand, all to be indicated in particular table element tm be the table element that also identifies with other precision of integer level in the table section at center to each among mark t1, t2, t3 and the t4.Need on both direction, clamp at least four table elements of particular table element tm.
Then, as shown in Figure 5, coordinate space is supposed in the scope of reduction reference vector (or in scope of reduction relevance values table).Scope through the reduction reference vector is corresponding with the hunting zone of reference frame.The position of target frame (strictly speaking, being the position of object block projected image piece 104 shown in Figure 42) is taken as the origin position (0,0) of coordinate space.Vertical Z axle (or relevance values axle) is taken as the axle of representing relevance values, and it is inverse ratio ground with the correlation between reference block and the object block and reduces.Horizontal x axle (or vx/n axle) is taken as represents reference block axle from the skew of object block on the x direction, or representative is through the axle of the value vx/n of reduction reference vector.Similarly, horizontal y axle (or vy/n axle) is taken as represents reference block axle from the skew of object block on perpendicular to the y direction of x direction, or representative is through the axle of the value vy/n of reduction reference vector.
Therefore, from the relevance values of the minimum value table element tm that identifies with other precision of integer level and as two table element t1 that also identify with other precision of integer level of the table element of clamping minimum value table element tm at specific direction and the relevance values of t3, in coordinate space shown in Figure 5, created conic section.Similarly, from the relevance values of minimum value table element tm and on perpendicular to the other direction of this specific direction as two table element t2 that also identify with other precision of integer level of the table element of clamping minimum value table element tm and the relevance values of t4, in coordinate space, created another conic section.Then, by adopting least square method, in coordinate space shown in Figure 5, find the approximate quadratic surface 201 that comprises these two conic sections.
Next, the conduct on X-Y plane has coordinate (vx/n, 203 places, position of position vy/n) detects the minimum point 202 of approximate quadratic surface 201, as shown in Figure 5.The position (vx/n vy/n) is position as the table element (or table element address) with the minimum relatedness value in reduction relevance values table, with other precision of mark level identify the position.At last, the position that definite sensing identifies with other precision of mark level (vx/n, minimum value vector 204 vy/n), and by taking advantage of minimum value vector 204 with the reciprocal value n that reduces the factor, calculate motion vector 205, as shown in Figure 6 with original size and its original orientation.
For example, by reducing original relevance values table to follow the process of reducing with 1/4 pair of reference vector of the reduction factor, obtain shown in Figure 7 through reduction relevance values table TBLs, and from the address of the minimum value table element that identifies with other precision of mark level, find motion vector 204 (0.777 ,-1.492).In this case, minimum value vector 204 is multiplied by 4, to obtain original motion vector 205 (3.108 ,-5.968).Motion vector 205 is motion vectors of the original scale of image.
According to the invention described above, searched through reduction relevance values table, with the particular table element tm that finds out the minimum relatedness value that is used to store the indication strongest correlation with particular table element tm four approaching table elements in the table section at center.But,, preferably find out the approaching table element of the greater number in such table section for the approximate quadratic surface of relevance values is set.Therefore, usually, find out the approaching table element in the following rectangular tables zone, described rectangular area comprises m * m the table element (wherein mark m indicates and equals 3 integer at least) on level and the vertical direction, the particular table element tm place that its center is going out with other accuracy detection of integer level.
But the approaching table element of bigger quantity is not necessarily better.This is because the approaching table element in this big table section causes treating capacity to increase.In addition, if the quantity of approaching table element increases, then probably detect the local minimum of the mistake that depends on image model inevitably.Therefore, select to comprise table element in the rectangular tables zone of approaching table element of right quantity.
Two examples that are included in the rectangular tables zone in reduction relevance values table as the zone of the approaching table element that comprises right quantity hereinafter have been described.An example according to this embodiment is such rectangular tables zone, and its center is at the minimum value table element tm place that finds with other precision of integer level, and comprises in the horizontal and vertical directions 3 * 3 approaching table elements around this minimum value table element tm.Another example according to this embodiment is the rectangular tables zone as the zone that comprise 4 * 4 approaching table elements in the horizontal and vertical directions centering on this minimum value table element tm of center at the minimum value table element tm place that finds with other precision of integer level.
[the rectangular tables zone that comprises 3 * 3 table elements]
Fig. 8 A and Fig. 8 B show by using following rectangular tables zone to find the figure of the technology of motion vector, this holds regional center at the minimum value table element tm place that finds with other precision of integer level, and comprises 3 * 3 approaching table elements that center on this minimum value table element tm in the horizontal and vertical directions.In Fig. 8 A, table section is illustrated as grey block.
According to the technology shown in Fig. 8 A and Fig. 8 B, by adopting, the approximate quadratic surface 201 shown in Fig. 8 B is set based on the minimum value table element tm that finds with other precision of integer level with around the least square method of the relevance values of 8 approaching table elements (shown in Fig. 8 A) of minimum value table element tm.Next, shown in Fig. 8 B, (vx/n, 203 places, position on the X-Y plane of position vy/n) detect the minimum point 202 that is similar to quadratic surface 201 to have coordinate in conduct.The position (vx/n, vy/n) be as and the corresponding position of table element (or table element address) with the minimum relatedness value in reduction relevance values table, the position of identifying with other precision of mark level.
At last, determine to point to minimum value vector 204 as the position of identifying with other precision of mark level 203 of the position of table element, and take advantage of minimum value vector 204 by reciprocal value n with the reduction factor, calculate motion vector 205 (or minimum value vector), as shown in Figure 6 with original size and its original orientation.
By adopt method hereinafter described, carry out find with approximate quadratic surface 201 on the process of minimum point 202 corresponding positions 203.As shown in Figure 9, and coordinate (x, y) system is designed to such system, and wherein the position at the center of the minimum value table element tm that finds with other precision of integer level is taken as origin position (0,0).In this case, be positioned at around 8 approaching table elements of the minimum value table element tm that finds with other precision of integer level and have in the horizontal direction, except the position of coordinate (x=0 and y=0) by the x axial coordinate of x=-1, x=0 and x=+1 representative and in the position of vertical direction by the y axial coordinate of y=-1, y=0 and y=+1 representative.That is to say, be positioned at coordinate (1 ,-1), (0 ,-1), (1 ,-1), (1,0), (0,1), (1,1), (0,1) and (1,1) around 8 approaching table elements of the minimum value table element tm that finds with other precision of integer level and locate.
Make mark Sxy indicate the relevance values of the table element in the coordinate system shown in Figure 9.For example, the relevance values of the minimum value table element tm that finds at origin position (0,0) with other precision of integer level is by symbol S 00Mark, and on the right side of minimum value table element tm and the relevance values that is lower than the approaching table element of the position (1,1) of minimum value table element tm locating by symbol S 11Mark.
Therefore, according to equation shown in Figure 10 (A) and (B), can find utilization with other precision of integer level at (x, y) origin position of coordinate system (0,0) the minimum value table element tm that locates to find, with other precision of mark level at (x, y) coordinate of observed position in the coordinate system (dx, dy).
At equation shown in Figure 10 (A) with (B), the value of Kx and Ky provides as follows:
For x=-1, Kx=-1;
For x=0, Kx=0;
For x=1, Kx=1;
For y=-1, Ky=-1;
For y=0, Ky=0; And
For y=1, Ky=1.
Coordinate (dx, dy) be position (dx from other precision of mark level, dy) with other precision of integer level at origin position (0,0) locates the position of the minimum value table element tm that finds, utilization with other precision of integer level at origin position (0,0) locate the minimum value table element tm that finds, with other precision of mark level at (x, y) coordinate of observed position in the coordinate system.Position 203 can be detected as the position of separating with the center of the minimum value table element tm that identifies.
[the rectangular tables zone that comprises 4 * 4 table elements]
Figure 11 A and Figure 11 B are by using following rectangular tables zone to find the figure of the technology of motion vector, the center of this rectangular area is at the minimum value table element tm place that finds with other precision of integer level, and comprises in the horizontal and vertical directions 4 * 4 approaching table elements around this minimum value table element tm.In Figure 11 A, table section is illustrated as grey block.
(comprise m * m approaching table element in m * m table section, wherein m is an odd number) under the situation of (for example above-mentioned table section that comprises the table element that 9 (=3 * 3) are approaching and comprise the table section of 25=5 * 5 an approaching table element), the minimum value table element tm that finds with other precision of integer level is positioned at the table element place, center as approaching table element.Therefore, can be provided for determining the rectangular tables zone of motion vector at an easy rate.
On the other hand, (comprise m * m approaching table element in m * m table section, wherein m is an even number) under the situation of (for example comprising the minimum value table element tm that finds with other precision of integer level and the table section that comprises 4 * 4 table elements of 15 approaching table elements), minimum value table element tm is not positioned at the table element place, center as approaching table element.Therefore, be difficult to easily be provided for determining the rectangular tables zone of motion vector, therefore some effort that can carry out designing below.
In this case, in that relevance values is table is compared to each other with the relevance values (in the present embodiment, they all are final component sad values) of the approaching table element that comprises minimum value table element tm in the delegation with minimum value table element tm that find with other precision of integer level through reduction.Result as a comparison, such rectangular tables zone is set up, and wherein minimum value table element tm serves as second table element in this row.The table element with minimum relatedness value in four that comprise minimum value table element tm adjacent approaching table elements serves as the 4th approaching table element of this row.Similarly, be compared to each other through the reduction relevance values relevance values with the same approaching table element that comprises minimum value table element tm that lists of minimum value table element tm that find with other precision of integer level table.Result as a comparison, such rectangular tables zone is set up, and wherein minimum value table element tm serves as second table element in these row.The table element with minimum relatedness value in four that comprise minimum value table element tm adjacent approaching table elements serves as the 4th approaching table element of these row.
In the example shown in Figure 11 A and Figure 11 B, the minimum value table element tm that finds with other precision of integer level is clamped with two that have relevance values 177 and 173 in the delegation respectively adjacent approaching table elements.In this case, minimum value table element tm is taken as second table element of this row, and the approaching table element on the approaching table element right side with less relevance values 173 is taken as the 4th approaching table element of this row.Similarly, the minimum value table element tm that finds with other precision of integer level is clamped by same two the adjacent approaching table elements that have sad value 168 and 182 respectively that list.In this case, minimum value table element tm is taken as second table element of these row, and is taken as the 4th approaching table element of these row having approaching table element above the approaching table element of less relevance values 168.
Then, in the example shown in Figure 11 A and Figure 11 B, based on the minimum value table element tm that finds with other precision of integer level with around the relevance values of 15 approaching table elements (shown in Figure 11 A) of minimum value table element tm, by adopting least square method, the approximate quadratic surface 201 shown in Figure 11 B is set.Next, shown in Figure 11 B, (vx/n, 203 places, position on the X-Y plane of position vy/n) detect the minimum point 202 that is similar to quadratic surface 201 to have coordinate in conduct.The position (vx/n, vy/n) be as and the corresponding position of table element (or table element address) with the minimum relatedness value in reduction relevance values table, with other precision of mark level identify the position.
At last, determine to point to minimum value vector 204 as the position of identifying with other precision of mark level 203 of the position of table element, and take advantage of minimum value vector 204 by reciprocal value n with the reduction factor, calculate motion vector 205 (or minimum value vector), as shown in Figure 6 with original size and its original orientation.
By adopt hereinafter described method to carry out to find with approximate quadratic surface 201 on the process of minimum point 202 corresponding positions 203.As shown in Figure 9, and coordinate (x, y) system is designed to such system, and wherein the position at the center of the minimum value table element tm that finds with other precision of integer level is taken as origin position (0,0).
Under the situation of the example shown in Figure 11 A and Figure 11 B, shown in Figure 12 A, Figure 12 B, Figure 12 C and Figure 12 D, four rectangular tables zones that comprise 16 table elements of layout have produced the diverse location of the minimum value table element tm that finds with other precision of integer level by different way.It is approaching table element in right side or the left side of minimum value table element tm that the position of minimum value table element tm in the rectangular tables zone of finding with other precision of integer level depends on aforementioned the 4th table element that comprises the row of minimum value table element tm, and the aforementioned approaching table element of the 4th table element below above the minimum value table element tm still being that comprises the row of minimum value table element tm.
In this case, can find out clearly that the fixed-site of the minimum value table element tm that finds with other precision of integer level is located in the position (0,0) of coordinate system from Figure 12 A, Figure 12 B, Figure 12 C and Figure 12 D.The position near the position of 15 approaching table elements of minimum value table element tm have in the horizontal direction by the x axial coordinate of x=-2 or x=-1, x=0 and x=+1 or x=+2 representative and and vertical direction on the y axial coordinate by y=-2 or y=-1, y=0 and y=+1 or y=+2 representative.
Make the relevance values of the table element of mark Sxy sign picture 12A in the coordinate system shown in Figure 12 D.For example, the relevance values of the minimum value table element tm that locates to find at origin position (0,0) with other precision of integer level is by symbol S 00Indicate, and on the right side of minimum value table element tm and the relevance values that is lower than the table element of the position (1,1) of minimum value table element tm locating by symbol S 11Mark.
Therefore, according to equation shown in Figure 13 (C) and (D), can find utilization with other precision of integer level at (x, y) origin position of coordinate system (0,0) the minimum value table element tm that locates to find, with other precision of mark level at (x, y) coordinate of observed position in the coordinate system (dx, dy).(x, y) origin position of coordinate system (0,0) and covering comprise with other precision of integer level in that (x, y) origin position of coordinate system (0,0) the minimum value table element tm that locates to find overlaps at the center of the rectangular area of 16 interior table elements.
At equation shown in Figure 13 (C) with (D), the value of Kx and Ky is respectively by (Kx shown in Figure 14, Ky) value of the level of coordinate system and vertical axis representative, this coordinate system is as being placed on a coordinate system on the rectangular tables zone, shown in the rectangular tables zone comprise the minimum value table element tm that finds with other precision of integer level and position 15 approaching table elements near this minimum value table element tm.The center of this rectangular area is with (Kx, Ky) origin position of coordinate system (0,0) overlaps.The value of Kx and Ky is to depend on the value to 4 kinds of different layouts shown in Figure 12 D as Figure 12 A of the layout of table element.
Particularly, under the situation of the coordinate system shown in Figure 12 A, shown in Figure 14 (Kx, Ky) the coordinate Kx of coordinate system and Ky have following value:
For x=-2, Kx=-1.5;
For x=-1, Kx=-0.5;
For x=0, Kx=0.5;
For x=1, Kx=1.5;
For y=-2, Ky=-1.5;
For y=-1, Ky=-0.5;
For y=0, Ky=0.5; And
For y=1, Ky=1.5.
Under the situation of the coordinate system shown in Figure 12 B, shown in Figure 14 (Kx, Ky) the coordinate Kx of coordinate system and Ky have following value:
For x=-2, Kx=-1.5;
For x=-1, Kx=-0.5;
For x=0, Kx=0.5;
For x=1, Kx=1.5;
For y=-1, Ky=-1.5;
For y=0, Ky=-0.5;
For y=1, Ky=0.5; And
For y=2, Ky=1.5.
Under the situation of the coordinate system shown in Figure 12 C, shown in Figure 14 (Kx, Ky) the coordinate Kx of coordinate system and Ky have following value:
For x=-1, Kx=-1.5;
For x=0, Kx=-0.5;
For x=1, Kx=0.5;
For x=2, Kx=1.5;
For y=-2, Ky=-1.5;
For y=-1, Ky=-0.5;
For y=0, Ky=0.5; And
For y=1, Ky=1.5.
Under the situation of the coordinate system shown in Figure 12 D, shown in Figure 14 (Kx, Ky) the coordinate Kx of coordinate system and Ky have following value:
For x=-1, Kx=-1.5;
For x=0, Kx=-0.5;
For x=1, Kx=0.5;
For x=2, Kx=1.5;
For y=-1, Ky=-1.5;
For y=0, Ky=-0.5;
For y=1, Ky=0.5; And
For y=2, Ky=1.5.
The mark Δ x that uses in the equation shown in Figure 13 (C) is (x, y) the coordinate x of the position of the table element in the coordinate system (Kx, Ky) skew of the coordinate Kx in the coordinate system shown in Figure 14 relatively shown in Figure 12 A, Figure 12 B, Figure 12 C or Figure 12 D.Similarly, the mark Δ y that uses in the equation shown in Figure 13 (D) is (x, y) the coordinate y of the position of the table element in the coordinate system (Kx, Ky) skew of the coordinate Ky in the coordinate system shown in Figure 14 relatively shown in Figure 12 A, Figure 12 B, Figure 12 C or Figure 12 D.Shifted by delta x and Δ y have following value:
Under the situation of Figure 12 A, Δ x=-0.5, Δ y=-0.5;
Under the situation of Figure 12 B, Δ x=-0.5, Δ y=0.5;
Under the situation of Figure 12 C, Δ x=0.5, Δ y=-0.5; And
Under the situation of Figure 12 D, Δ x=0.5, Δ y=0.5.
Coordinate (dx, dy) be utilize with other precision of integer level (x, the y) origin position of coordinate system (0,0) the minimum value table element tm that locates to find, with other precision of mark level at (x, y) coordinate of observed position in the coordinate system.From with other precision of mark level at (x, y) observed position (dx in the coordinate system, dy) with other precision of integer level at (x, y) origin position (0 of coordinate system, 0) locates the position of the minimum value table element tm that finds, the position that the center of the minimum value table element tm that position 203 can be detected as and identify is separated.
[the second typical interpolation process of detection motion vector] with accuracy of higher degree
The second typical interpolation process of detection motion vector with accuracy of higher degree adopts following technology.A plurality of relevance values (wherein in the present embodiment, each all is final component sad value) of being stored in the element of arranging with horizontal direction on the row that comprises the minimum value table element tm that finds with other precision of integer level in reduction relevance values table are used to create the cubic curve on the plane that is oriented on the horizontal direction.Be used to create cubic curve on the plane that is oriented on the vertical direction at a plurality of relevance values stored in the element of arranging with vertical direction that list that in reduction relevance values table, comprise minimum value table element tm.Then, and the position of the minimum value of cubic curve (vx, vy) detected, and be taken as minimum value address with other precision of mark level.
Figure 15 A and Figure 15 B are the key diagrams of reference when being described below the second typical interpolation process of detection motion vector of the accuracy with higher degree.Be very similar to the first typical interpolation process of the detection motion vector of accuracy with higher degree, by using following rectangular tables zone, the second typical interpolation process is performed to find motion vector, described rectangular tables zone is the center with the minimum value table element tm that other precision of integer level finds, and comprises the approaching table element that centers on minimum value table element tm in the horizontal and vertical directions.In the example shown in Figure 15 A and Figure 15 B, the quantity of approaching table element is set to 16 (=4 * 4).In Figure 15 A, table section is illustrated as grey block.
Next, be very similar to the first typical interpolation process of the detection motion vector of the accuracy with higher degree, shown in Figure 15 B, coordinate space is assumed to be in the scope (or through reducing the scope of relevance values table) of reduction reference vector.Scope through reducing reference vector is corresponding to the hunting zone of reference frame.The position of target frame (position of object block projected image piece 104 shown in Figure 42 perhaps more strictly speaking) is taken as the origin position (0,0) of coordinate space.Vertical Z axle (or relevance values axle) is taken as the axle of representing relevance values, and it reduces inversely with the correlation between reference block and the object block.In the present embodiment, relevance values is final component sad value.Horizontal x axle (or vx/n axle) is taken as representative, and reference block is from the axle of the skew of object block on the x direction, or representative is through the axle of the value vx/n of reduction reference vector.Similarly, vertical y axle (or vy/n axle) is taken as representative, and reference block is from the axle of the skew of object block on perpendicular to the y direction of x direction, or representative is through the axle of the value vy/n of reduction reference vector.
Then, four table elements from the horizontal direction row that comprises minimum value table element tm near selection 16 table elements of the minimum value table element tm that finds with other precision of integer level.Next, the relevance values of four selecteed table elements (wherein each all is final component sad value) is used to create the horizontal cubic curve 206 on the plane that is oriented in the coordinate system on the horizontal direction.Then, in the zone of table element, select the horizontal direction position vx/n of the minimum value on the horizontal cubic curve 206 with other precision of mark level.
Similarly, from comprising four table elements that the vertical direction of minimum value table element tm lists near selection 16 table elements of the minimum value table element tm that finds with other precision of integer level.Next, the relevance values of four selecteed table elements (wherein each all is final component sad value) is used to create the vertical cubic curve 207 on the plane that is oriented in the coordinate system on the vertical direction.Then, in the zone of table element, select the vertical direction position vy/n of the minimum value on the vertical cubic curve 207 with other precision of mark level.
Then, from the horizontal direction position vx/n of other accuracy selection of mark level with the vertical direction position vy/n of other accuracy selection of mark level, find minimum value table address 208 with other precision of mark level.Fraction precision minimum value table address 208 be with horizontal cubic curve 206 and vertical cubic curve 207 on the corresponding table element of minimum value address.At last, determine the minimum value vector 209 of the fraction precision minimum value table address 208 that identifies with other precision of mark level as the sensing of the position of table element.Take advantage of minimum value vector 209 by reciprocal value n, calculate motion vector (or minimum value vector), as shown in Figure 6 with original size and its original orientation with the reduction factor.
That is to say that the second typical interpolation process adopts such technology, wherein by adopting the technology identical going and be oriented to four table elements of selection in each of row on the vertical direction on being oriented to horizontal direction with the first typical interpolation process.Then, four table elements upward selecting based on being expert at are created the cubic curve on the plane that is oriented on the horizontal direction, and based on the cubic curve of creating at four table elements that list selection on the plane that is oriented on the vertical direction, shown in Figure 15 B.
By following method carry out find with horizontal cubic curve 206 and vertical cubic curve 207 on the process of minimum point 202 corresponding fraction precision minimum value table addresses 208.Make mark S 0, S 1, S 2And S 3Be indicated in and be oriented to going or be oriented to the relevance values of four table elements that list selection on the vertical direction on the horizontal direction.As mentioned above, in the present embodiment, relevance values is final component sad value.Relevance values S 0, S 1, S 2And S 3Corresponding to the horizontal cubic curve 206 on the along continuous straight runs continuously or four consecutive points of vertical cubic curve 207 layouts on the vertical direction.As shown in figure 16, mark Ra, Rb and Rc indicate representative point S respectively 0And S 1Between line segment, the representative point S of axial distance 1And S 2Between the line segment and the representative point S of axial distance 2And the line segment of the axial distance between the S3.Line segment part u is the fractional part that comprises in the coordinate figure of position of minimum relatedness value.Comprise the equation of the line segment part u of fractional part included in the coordinate figure of the position of serving as the minimum relatedness value according to depending among shown in Figure 16 three line segment Ra, Rb and the Rc which, find line segment part u.
As mentioned above, line segment Ra is and relevance values (or sad value) S 0Corresponding position and with relevance values S 1Corresponding position between line segment.Line segment Rb is and relevance values S 1Corresponding position and with relevance values S 2Corresponding position between line segment.Line segment Rc is and relevance values S 2Corresponding position and with relevance values S 3Corresponding position between line segment.As mentioned above, in the present embodiment, relevance values is final component sad value.
If the fraction precision position of minimum relatedness value is present among the line segment Ra shown in Figure 16,, found as the representative of mark beginning from line segment Ra to the line segment part u of the distance of this position then by using equation (E) shown in Figure 17.
Similarly, if the fraction precision position of minimum relatedness value is present among the line segment Rb shown in Figure 16,, found as the representative of mark beginning from line segment Rb to the line segment part u of the distance of this position then by using equation (F) shown in Figure 17.
In an identical manner, if the fraction precision position of minimum relatedness value is present among the line segment Rc shown in Figure 16, then by using equation (G) shown in Figure 17, found as the representative of mark beginning from line segment Rc to the line segment part u of the distance of this position.
The technology which bar that is used for determining shown in Figure 16 three line segment Ra, Rb and Rc comprises fractional part u has hereinafter been described.
Figure 18 A is the key diagram of reference when description is used for determining which bar of shown in Figure 16 three line segment Ra, Rb and Rc comprises the technology of fractional part u to Figure 18 D.At first, mark Smin indicates the minimum relatedness value of the position that goes out with other accuracy detection of integer level.Mark Sn2 indicates the such relevance values that is arranged in the integer precision position: it is minimum with differing of minimum relatedness value Smin at the relevance values of the integer precision position of all four table elements.The real minimum relatedness value that symbol x among Figure 18 A, Figure 18 B and Figure 18 C is indicated may reside in such position of going out with other accuracy detection of mark level: the position between the position of the position of minimum relatedness value Smin and relevance values Sn2.Then, by distinguishing relevance values S shown in Figure 16 0, S 1, S 2And S 3In which serve as minimum relatedness value Smin and relevance values Sn2, can determine which bar among three line segment Ra, Rb and the Rc comprises fractional part u.
Should be noted that also there is this situation that wherein the integer precision position of minimum relatedness value Smin is the edge of scope of position that comprises the relevance values of four table elements, shown in Figure 18 D.In this case, the position of real minimum relatedness value x is difficult to determine, and this embodiment do not find out the position of real minimum relatedness value x, and this situation is treated to error.But,, also can find the position of real minimum relatedness value x even under the situation shown in Figure 18 D.
As mentioned above, according to the foregoing description, dwindle factor 1/n by using to utilize 2Dwindled size through reduction relevance values table, can detect the motion vector of original image ratio.Figure 19 shows the figure of the following fact, dwindles factor 1/n though promptly used to utilize 2Dwindled size through reduction relevance values table, but can obtain almost identical vector detection result with the image processing apparatus of prior art.
The one dimension reduction factor 1/n that uses when the representative of the trunnion axis of Figure 19 reduces the relevance values table on level or vertical direction.On the other hand, vertical axis representative vector error, the i.e. error of detected motion vector.The value of vector error shown in Figure 19 is expressed with pixel.
In Figure 19, curve 301 representatives are at the mean value of the different detected vector error of the reduction factor.Curve 302 representatives are at three times of values (3 σ [99.7%] value) of the variances sigma of the different detected vector error of the reduction factor.Curve 303 is curve of approximation of curve 302.
Curve representative shown in Figure 19 is with the different detected vector error of one dimension reduction factor 1/n.But because the relevance values table is bivariate table, therefore the size (number of elements of promptly forming the relevance values table) of table is dwindled with following ratio, the one dimension reduction factor 1/n that this ratio equals to use among Figure 19 square.But, not have to change and the curve of variance linear increase of vector error from the mean value of indication vector error with the reduction factor variations, can clearly find out effect according to the technology of present embodiment.
In addition, even for n=64 (or reduction factor 1/64), the mean value of vector error also is very little, has proved the failure that detection caused that does not have incorrect motion vector.Therefore, size that we can say the relevance values table can be utilized and dwindle the factor 1/4096 and dwindle.
In addition, as previously mentioned, come in the process of compensating motion picture real-time response and reduce time delay and be very important in the influence of trembling at hand tremor.But as long as the error of detected motion vector is not the failure that the detection by incomplete incorrect motion vector causes, this error is exactly tolerable to a certain extent.Therefore, the size of relevance values table can significantly reduce, and does not lead to the failure.Therefore, these embodiment can be described as very useful.
At the real system that is used for coming at the influence that hand tremor is trembled compensating images, reference frame 102 is divided into a plurality of regional areas, and detects motion vector 205 for each regional area.This is because the mobile object of image pickup might be included in the reference frame 102 very much.For example, in a reference frame 102, detect 16 motion vectors 205, as shown in figure 20.Then, when considering transformation (each changes by a motion vector 205 indications since the image in past), carry out statistic processes, so that determine the global vector of reference frame 102, i.e. motion vector is trembled in the hand tremor of reference frame 102.
In this case, as shown in figure 20, be provided with respectively in advance 16 motion vectors 205 will detecting with origin position PO1, PO2 ... and PO16 is 16 hunting zone SR1 at center, SR2 ..., and SR16, and object block projected image piece IB1, IB2 ... be assumed to be with IB16 and be present in hunting zone SR1 respectively, SR2 ... and the center of SR16.
Then, at hunting zone SR1, SR2 ... in among the SR16 each, will with object block projected image piece IB1, IB2 ... be set to respectively will be at hunting zone SR1 with reference block that among the IB16 each has an identical size, SR2 ... and SR16 goes up the piece that moves to another position from a position.Then, generate table through reduction SAD, so as with the identical mode of aforementioned technology provided by the invention, at hunting zone SR1, SR2 ... and find motion vector 205 among the SR16 each.
With as the comparing of the method for the motion vector that is used for detecting image in patent documentation 4 disclosed art methods with the size of having dwindled, have the following remarkable advantage that is different from art methods according to the image processing method of the foregoing description.
At first, different with patent documentation 4 disclosed art methods, according to the image processing method of the embodiment process of needs reduction image not.This is because according to the image processing method that embodiment provided, and is stored in as table element in the process of reduction relevance values table at the component relevance values that will calculate for reference block, carries out the process of switch element address simultaneously.As mentioned above, the relevance values that calculates for reference block is actually the final component sad value that calculates into reference block.
Therefore, compare with patent documentation 4 disclosed art methods, image processing method according to embodiment provides following advantage, for example removed the reduction image logic, be used for the memory storage by the required time of reduction image, be used in the memory storage by the bandwidth of the process of reduction image, and be used to store memory by the reduction image.
Secondly, patent documentation 4 disclosed art methods have produced another serious problems, and as previously mentioned, the folding and low light that this method needs low pass filter to eliminate and generates in the process of reduction image shines noise.That is to say that in the process of reduction image, view data may be provided for suitable low pass filter before resampling.Otherwise will fold, and use the precision of the motion vector that the image detection of reduction goes out significantly to worsen.
Be proved to be ideal characterisitics in theory as the function of in the process of reduction image, using that low pass filter represented of the function of similar sinc function for being low pass filter.The sinc function itself is the function with unlimited tap (infinite-tap) FIR (finite impulse response (FIR)) filter of the cut-off frequency f/2 that is expressed by sin (x ∏)/(x ∏).Under the situation of the low pass filter that has desirable cut-off frequency f/ (2n) for reduction factor 1/n, cut-off frequency is by sin (x ∏/n)/(x ∏/n) express, though it also can be used as the form of sinc function.
The figure of the top of Figure 21, Figure 22 and Figure 23 shows the shape (or ideal characterisitics of low pass filter) of the sinc function that is used to reduce the factor 1/2,1/4 and 1/8 respectively.Can find out clearly that from Figure 21 to Figure 23 the reduction factor is big more, the factor that function is expanded on tap axle (tap-axis) direction is big more.In other words, even for being similar under the situation of unlimited tap sinc function, we can say that also the tap number of FIR filter may increase with important coefficient.
In addition, if the cut-off frequency in the known frequency band is low more, then tap number is just big more than the influence of filter shape to the Effect on Performance of low pass filter.
Therefore, use generally demonstrates such contradiction according to the motion vector recognition methods by the reduction image that patent documentation 4 disclosed art methods generate: although the reduction factor of image is big more, the effect of size of dwindling the relevance values table is just big more, but we have run into such fact, and promptly cost increases pro rata with the increase of the reduction factor.
Usually, when realizing high-order tap FIR filter, the cost of processing logic square increases pro rata with number of taps, thereby causes very big problem.But, be used to realize that the quantity increase of the line storage of vertical filter has caused bigger problem.In the digital camera of making in recent years, for the size of dwindling line storage so that catch up with ever-increasing pixel quantity, carry out so-called band (strap) and handle.But, even every memory lines big or small reduced for example, but the quantity of line storage own has increased, if the physical layout zone is converted into cost, then total cost can significantly increase.
As mentioned above, known means of reducing based on the image according to patent documentation 4 disclosed art methods face very big obstacle, especially when realizing vertical low pass filters.On the other hand, the image processing method according to present embodiment has solved this problem in a completely different way effectively.
The image of the low pass filter of the image processing method that provides according to the embodiment of the invention is provided respectively to the figure of the below of Figure 23 Figure 21.According to the image processing method that the embodiment of the invention provides, do not carry out the reduction treatment of picture.But, generate the processing that comprises low pass filter through the process of reduction relevance values table, shown in arbitrary figure of its image in these figure.
The figure of the below from Figure 21 to Figure 23 can find out clearly that the characteristic of this low pass filter is the simple filter characteristic, and wherein the important coefficient of sinc function part can be by linear approximation, and number of taps increases in the mode with reduction factor interlocking.The increase mode of simple filter characteristic and tap number is suitable for such fact: cut-off frequency is low more, and tap number is big more to the Effect on Performance of low pass filter.That is to say, find the process of component relevance values (wherein each all is the component sad value) to be equal to according to the embodiment of the invention and be embodied as ball bearing made using will represent high performance low pass filter with the mode of reducing factor interlocking.As previously mentioned, finding the process of component relevance values is as the processing of carrying out according to embodiment based on the process of the linear weighted function distribution technique that is used to find the component relevance values.
The ball bearing made using that realizes low pass filter provides another advantage than patent documentation 4 disclosed art methods.That is to say that according to patent documentation 4 disclosed art methods, image is reduced in sampling process after by low pass filter.In the reduction process, a lot of image informations have been lost.More specifically, in the processing that low pass filter is carried out, the word length of the brightness value of image information was rounded off before image information is stored in the memory significantly.Therefore, most of low levels of Pixel Information are to the not influence of reduction image.
On the other hand, according to according to the image processing techniques of embodiment, the brightness value of all pixels in the object block is all used comparably being used for calculating being stored in as table element in the process of the final component relevance values of reduction relevance values table.That is to say, final component correlation be relevance values accumulation and, each in the described relevance values is that one of pixel in the object block is found.Therefore, by only increasing word length, can carry out such relevance values computational process: even the final relevance values that calculates does not at last comprise the process errors that rounds off yet through each element of reduction relevance values table.Very little owing to comparing with the size of frame memory through the size of reduction relevance values table, therefore composition is expanded through the word length of each element of reduction relevance values table and can not produced big problem.As a result, the processing through reduction relevance values table and definite motion vector can realize with the precision of higher degree.
[embodiment of image processing apparatus provided by the invention]
With reference to the accompanying drawings, illustrated hereinafter that wherein the embodiment as the image pick-up device of the image processing apparatus of the image processing method that adopts the embodiment of the invention to provide all has been provided for each.
[first embodiment]
First embodiment that has realized as the image pick-up device of the image processing apparatus of the image processing method that adopts the embodiment of the invention to provide is provided Figure 24.
As shown in figure 24, image pick-up device 10 according to present embodiment comprises that image pickup signal treatment system, CPU (CPU) 1, user operate input unit 3, video memory unit 4, with data recording/reproducing device 5, they are connected with each other by system bus 2.The image pickup signal treatment system comprises that image pickup lens 10L, image pickup device 11, timing signal generation unit 12, pretreatment unit 13, Date Conversion Unit 14, hand tremor tremble motion vector detecting unit 15, resolution conversion unit 16, coding/decoding unit 17, NTSC encoder 18, and monitors display unit 6.Should be noted that the CPU1 that describes in the patent specification comprises is used to store as the ROM (read-only memory) of the various softwares of carrying out for CPU1 of handling procedure with by the RAM (random access storage device) of CPU1 as the service area.
The order that receives image pickup and recording process to start with operate input unit 3 by the operational order of user's input via the user time, image pick-up device 10 shown in Figure 24 is carried out hereinafter the process of the recording image pick-up data that will describe.The order of the process that receives the image pickup data that reproduction to start with write down operate input unit 3 by the operational order of user's input via the user time, image pick-up device 10 shown in Figure 24 is carried out the process that is reproduced in the image pickup data that writes down on the recording medium that uses in the data recording/reproducing device 5.
As shown in figure 24, the light beam that enters from image pickup object via the camera optical system that uses image pickup lens 10L is irradiated to the image pickup device 11 that is used for carries out image pick process on the light beam.Should be noted that camera optical system itself does not illustrate in the drawings.In the present embodiment, image pickup device 11 is configured to CCD (charge coupled device) imager.Should be noted that image pickup device 11 also can be configured to CMOS (complementary metal oxide semiconductors (CMOS)) imager.
In image pick-up device according to present embodiment, when the user operates input unit 3 when image pick-up device 10 is imported the operational order of the order of image pickup and recording process to start with via the user, image pickup device 11 outputs comprise the primary signal of the bayer array of three primary colors (i.e. red (R), green (G) and blueness (B)).Primary signal as the analog image pickoff signals is the signal of conduct according to result's acquisition of the sampling process of the timing signal of timing signal generation unit 12 generations.Image pickup device 11 offers pretreatment unit 13 with the analog image pickoff signals, so that carry out such as detecting preliminary treatment such as compensation process and γ compensation process.Date Conversion Unit 14 outputs to Date Conversion Unit 14 with pretreated result.
The analog image pickoff signals that Date Conversion Unit 14 will offer it is converted to the digital picture pickoff signals (YC data) that comprises luminance signal component Y and chroma signal component Cb/Cr, by system bus 2 the digital picture pickoff signals is offered video memory unit 4.
In the embodiment shown in Figure 24, video memory unit 4 comprises two frame memories 41 and 42.At first, the digital picture pickoff signals that receives from Date Conversion Unit 14 is stored in the frame memory 41.Then, after past time corresponding to a frame, the digital picture pickoff signals that is stored in the frame memory 41 is sent to frame storage 42, and the new digital picture pickoff signals that receives from Date Conversion Unit 14 is stored in the frame memory 41.Therefore, be former frame just by the frame that is stored in the digital picture pickoff signals representative in the frame memory 42, it is than by the leading frame of frame that is stored in the digital picture pickoff signals representative in the frame memory 41.
Then, hand tremor is trembled motion vector detecting unit 15 by system bus 2 these two frame memories 41 and 42 of visit, so that from frame memory 41 and 42 reading number image pickup signals.Then, hand tremor is trembled motion vector detecting unit 15 and is carried out the process that is used for detecting as previously mentioned the motion vector between the digital picture pickoff signals.In this motion vector detection process, the frame that is stored in the digital picture pickoff signals representative in the frame storage 41 is taken as reference frame, and the frame that is stored in the digital picture pickoff signals representative in the frame memory 42 is taken as the primitive frame that serves as target frame.
Then, hand tremor tremble motion vector detecting unit 15 will be as the result of motion vector detection process and the detected motion vector that obtains offers resolution conversion unit 16 as control signal.
According to tremble the motion vector that motion vector detecting unit 15 receives from hand tremor, resolution conversion unit 16 is carried out the view data that is used for cutting out by the deferred frame of the digital picture pickoff signals representative that is stored in frame memory 42, and frame is converted to the process of the frame of the size with necessary resolution and necessity.Tremble the motion vector that motion vector detecting unit 15 receives by basis from hand tremor, cut out the view data of the frame of the digital picture pickoff signals representative that is stored in the frame memory 42, the image that obtains as the result of transfer process is the image that does not have the influence that hand tremor trembles.
The view data that is produced by resolution conversion unit 16 as the view data that does not comprise the influence that hand tremor is trembled is provided for NTSC (national television system committee) encoder 18, so that view data is converted to the standard colour-video signal that meets the NTSC system.Then, NTSC encoder 18 provides supervision display unit 6 with the standard colour-video signal, and this supervision display unit 6 serves as and be used for showing that captured image is for the electronic viewfinder that monitors purpose on screen.
With show captured image on the screen of display unit 6 side by side monitoring for the process that monitors purpose, resolution conversion unit 16 will not comprise that also hand tremor trembles the view data of influence and offer coding/decoding unit 17, so that carry out such as cataloged procedure such as modulation record grade.Then, coding/decoding unit 17 will offer data recording/reproducing device 5 as the view data that the result of cataloged procedure obtains, so as on recording medium recording image data.The example of recording medium is CD and the hard disk such as DVD (digital versatile disc) etc.
When the user operates input unit 3 when image pick-up device 10 is imported the operational order of the procedure command of reproducing the image pickup data that is recorded to start with via the user, from the recording medium reproducing data of data recording/reproducing device 5, and these data are provided for coding/decoding unit 17.At this moment, the 17 pairs of image pickup data in coding/decoding unit are carried out decoding/reproduction process.Then, the view data that coding/decoding unit 17 will obtain as the result of decoding/reproduction process offers by NTSC encoder 18 and monitors display unit 6, so that show reproduced image on screen.Should be noted that NTSC encoder 18 also can will offer the external signal recipient from the vision signal of its output by video output terminals, though this feature is not shown in Figure 24.
Motion vector detecting unit 15 is trembled in hand tremor can be implemented as hardware or DSP (digital signal processor).Perhaps, hand tremor is trembled motion vector detecting unit 15 and also can be implemented as the software of being carried out by CPU1.
[operation of motion vector detecting unit 15 is trembled in hand tremor]
＜the first typical case realizes 〉
Below with reference to Figure 25 and flow chart shown in Figure 26, illustrate and realize that hand tremor trembles the handling process that first typical case of the operation of motion vector detecting unit 15 realizes.
This flow chart starts from step S101, and here, the reference block Ii that the label 106 among Figure 42 is indicated is specified in has coordinate (vx, position vy) in the hunting zone 105.The operation of specifying reference block Ii in hunting zone 105 also is to specify and the corresponding reference vector of reference block Ii (vx, operation vy).As previously mentioned, (vx, vy) be is being under the situation of origin position coordinate with coordinate (0,0) to coordinate, the coordinate of the position that the reference vector of appointment 107 is pointed.Coordinate (0,0) is the position coordinates of the position coordinates of the object block 103 on the primitive frame 101 or the object block projected image piece 104 on the reference frame 102.Coordinate vx represents the horizontal direction skew of the reference vector 107 of appointment position pointed from origin position with coordinate (0,0).Coordinate vy represents the vertical offset of the reference vector 107 of appointment position pointed from origin position.Be very similar to aforementioned art methods, skew vx and vy are with the pixel formal representation.
As previously mentioned, with the center of hunting zone 105 center or origin position (0,0) as object block projected image piece 104.With the distance among the scope ± Rx of the horizontal limit that defined hunting zone 105 with in vertical direction with the distance among the scope ± Ry of the vertical bounds that defined hunting zone 105, reference block 106 moves to another position from a position on hunting zone 105 in the horizontal direction.In this case, amount (vx, vy) satisfy following relation:
-Rx≤vx≤+ Rx and-Ry≤vy≤+ Ry
In next step S102, (x, point (or pixel) y) are designated as the point among the object block Io that is indicated by label 103 among Figure 42 to have coordinate.Make mark Io (x, y) indicate appointment point (x, the pixel value of y) locating, make mark Ii (x+vx, y+vy) be indicated in step S101 in the piece position (vx, vy) point (x+vx, the pixel value of y+vy) locating among the reference block Ii that locates to be provided with.Hereinafter, the point among the reference block Ii (x+vx, y+vy) be said to be with object block Io in point (x, y) corresponding point.Then, in next step S103, according to following equation (1) calculating pixel value Io (x, y) and pixel value Ii (x+vx, y+vy) the absolute value α of the difference between:
α＝|Io(x，y)-Ii(x+vx，y+vy)| ——(1)
For (the x that has a few among the object block Io, y) and their all respective point (x+vx in reference block Ii, y+vy), calculate the absolute value α of above-mentioned difference, and the absolute value of the difference that representative is calculated object block Io and reference block Ii α's and sad value be stored in and point to reference vector (vx, the temporary storage position that vy) is associated of the current location of reference block Ii.In order to calculate such sad value, at next step S104, be accumulated on the interim sad value that is stored in the temporary storage position, with as the sad value that calculates at present at the absolute value α of the difference that step S103 finds.Result as the process that the absolute value α accumulation of all differences is sued for peace, obtain final sad value SAD (vx, vy), described all the difference absolute value α be as described above to the (x that has a few among the object block Io, y) and their all respective point in reference block Ii (x+vx y+vy) calculates.Therefore, final sad value SAD (vx, vy) can express by following equation (2):
SAD(vx，vy)＝∑α＝∑|Io(x，y)-Ii(x+vx，y+vy)| ——(2)
Then, flow process according to the first typical processing that realizes proceeds to next step S105, generation about whether to all the pixel (x among the object block Io, y) and their all respective pixel in reference block Ii (x+vx y+vy) has carried out the judged result of the process of step S103 and S104.If the judged result indication is not as yet to all the pixel (x among the object block Io, y) and their all respective pixel (x+vx in reference block Ii, y+vy) all carried out the process of step S103 and S104, then the flow process of the processing that realizes according to first typical case is returned step S102.(x, one other pixel y) is designated as the one other pixel among the object block Io to have coordinate.Then, step S103 after the repeating step S102 and the process of S104.
Just the step S1 with flow chart shown in Figure 44 is identical to the process of S5 respectively to the process of S105 for step S101, and except under the situation of step S103 and S104, sad value is stored in the temporary storage position.
On the other hand, if indicate to the (x that has a few among the object block Io in the judged result that step S105 produces, y) and their all respective point (x+vx in reference block Ii, y+vy) all carried out the process of step S103 and S104, if promptly current reference vector (vx, vy) (vx, vy) found, then the flow process according to the first typical processing that realizes proceeds to step S106 to final sad value SAD.By to reference vector (vx vy) reduces, calculate through the reduction reference vector (vx/n, vy/n).Particularly, by with reference vector (vx vy) multiply by reduction factor 1/n, calculate through the reduction reference vector (vx/n, vy/n).Usually, (vx/n, (vx/n vy/n) has fractional part for x direction vy/n) and the value of y direction through the reduction reference vector.
Then, at next step 107, recognizing site is approaching through reduction reference vector (vx/n, a plurality of approaching reference vector vy/n).Each approaching reference vector all be have integer vx/n value and integer vy/n value through the reduction reference vector.In the present embodiment, the quantity of approaching reference vector is set to 4.Then, at next step S108, aforementioned by adopting based on approaching reference vector position pointed with through reduction reference vector (vx/n, vy/n) the linear weighted function distribution technique of the relation between the position pointed, the sad value that is stored in temporary position at step S104 place is divided into four component sad values.Next, at next step S109, four component values distribute between four table elements that comprise in reduction relevance values table, as four table elements that are associated with four approaching reference vectors respectively.
After the process of step S109 is finished, proceed to the step S111 of flow chart shown in Figure 26 according to the flow process of the first typical processing that realizes, to produce about whether all reference block locations in the hunting zone 105 (promptly having been carried out the judged result of step S102 to the process of S109 to all reference vectors (vx, vy)).
If all reference blocks in the hunting zone 105 (promptly to the process of all reference vectors (vx, vy)) execution in step S102 to S109, are not then returned step S101 according to the flow process of the first typical processing that realizes as yet in the judged result indication that step S111 produces.(vx, vy) another reference block Ii pointed is set at another piece position in the hunting zone 105 (vx vy) locates another reference vector.Then, the process of repeating step S102 and subsequent step.
If indicate to all reference block locations in the hunting zone 105 or to all reference vector (vx in the judged result that step S111 produces, vy) all executed the process of step S102 to S109, promptly, if all elements through reduction relevance values table all has been filled final component sad value, then the flow process according to the first typical processing that realizes proceeds to step S112.In the table element address (mx, my) locate to detect be stored in all elements of reduction relevance values table the minimum value in the important final sad value.
Then, at next step S113, create quadratic surface as the curved surface of approximate following relevance values, described relevance values is at table element address (mx, my) locate detected minimum relatedness value and as near the table element address (mx, table element my) be stored in a plurality of relevance values in reduction relevance values table.As mentioned above, each relevance values all is a sad value.In the present embodiment, as (mx, the quantity that is stored in the relevance values in reduction relevance values table of table element my) is set to 15 near the table element address.Then, at next step S114, point to the position on the X-Y plane of other precision of mark level the minimum value vector (px, py) be detected as with quadratic surface on the corresponding vector of minimum sad value.The minimum value vector (px, py) position of Zhi Xianging be with quadratic surface on the corresponding position of minimum sad value.
Then, step S115 in the end, by with the minimum value vector (px py) multiply by the reciprocal value n of the reduction factor, calculate motion vector with original size and its original orientation (px * n, py * n), as shown in Figure 6.
Figure 25 and flow chart shown in Figure 26 have been represented the processing of carrying out according to the block-matching technique of realizing according to first typical case that is used to detect the motion vector that is used for an object block.For the frame of segmentation, as shown in figure 20, need detect a plurality of motion vectors to frame fragment.In this case, for wanting detected each motion vector or each frame fragment, hunting zone and reduction factor 1/n can be reset, so that the processing that execution Figure 25 and flow chart shown in Figure 26 are represented.
Note that self-evidently, except above-mentioned quadratic surface method, the cubic curve method also can be used as and be used to find minimum value vector (px, py) the technology of sensing with other precision of mark level detected position on the hunting zone.
＜the second typical case realizes 〉
In the situation that above-mentioned first typical case realizes, find the sad value that is used for reference block and reference vector, and by adopting the linear weighted function distribution technique, sad value is divided into a plurality of component sad values that are used for a plurality of approaching reference vectors then, the position of each in the described approaching reference vector near obtain as the result that reference vector is reduced through the reduction reference vector.That is to say that sad value is represented the relevance values between object block and the reference block.In the situation that above-mentioned first typical case realizes, find relevance values for the reference block that is associated with reference vector, and by adopting the linear weighted function distribution technique, relevance values is divided into a plurality of component relevance values that are used for a plurality of approaching reference vectors subsequently, the position of each in the described approaching reference vector near obtain as the result that reference vector is reduced through the reduction reference vector.
On the other hand, in the situation that second typical case realizes, relevance values is calculated as the poor of pixel on the object block and the pixel value between the respective pixel on the reference block.Therefore, the relevance values that calculates is not a sad value.Then, by adopting the linear weighted function distribution technique, the relevance values that calculates is divided into a plurality of component relevance values that are used for a plurality of approaching reference vectors, the position of each in the described approaching reference vector near obtain as the result that the reference vector that points to reference block is reduced through the reduction reference vector.To all pixels in the object block (or all respective pixel in the reference block), the process of double counting relevance values, be divided into the process of a plurality of component relevance values with the relevance values that will calculate,, find a plurality of final component relevance values with the technology that adds up by employing.When all pixels in the reference block having been finished the process of calculating relevance values and the relevance values that calculates have been divided into the process of a plurality of component relevance values, obtain through the state of reduction relevance values table (or obtain through reduction SAD table) with realize by first typical case generating through reduction relevance values table (or through reduction SAD epiphase with).
Tremble the second typical handling process that realizes of the operation of motion vector detecting unit 15 below by reference Figure 27 and flowchart text realization hand tremor shown in Figure 28.
Step S101 with flow chart shown in Figure 25 is identical to S103 respectively because the step S121 of flow chart shown in Figure 27 is to the process of S123, so does not describe the process of step S121 to S123 in detail.
In the next step S123 of the flow chart that second typical case realizes, according to equation (1) calculate pixel on the object block Io (x, pixel value Io y) (x, y) and pixel value Ii (x+vx, y+vy) the absolute value α of the difference between of the respective pixel on the reference block Ii.Then, at step S124, the reference vector by pointing to reference block Ii with reduction factor 1/n reduction (vx, vy), calculate through the reduction reference vector (vx/n, vy/n).
Next, at next step S125, recognizing site is approaching through reduction reference vector (vx/n, a plurality of approaching reference vector vy/n).In the approaching reference vector each all be have integer vx/n value and integer vy/n value through the reduction reference vector.In the present embodiment, the quantity of approaching reference vector is set to 4.Then, at next step S126, aforementioned by adopting based on approaching reference vector position pointed with through reduction reference vector (vx/n, vy/n) the linear weighted function distribution technique of the relation between the position pointed, it is poor to be divided into four components as the absolute value α that finds at step S123 of the absolute value of the difference of pixel value.
Next, at next step S127, between four table elements that in reduction relevance values table, comprise, distribute four components poor, as four table elements that are associated with four approaching reference vectors respectively.
After the process of step S127 is finished, flow chart according to the second typical processing that realizes proceeds to next step S128, whether to produce about to the (x that had a few among the object block Io, y) and their all respective point in reference block Ii (x+vx y+vy) has carried out the judged result of step S123 to the process of S127.If the judged result indication is not as yet to all the pixel (x among the object block Io, y) and their all respective pixel (x+vx in reference block Ii, y+vy) carried out the process of step S123, then returned step S122 according to the flow process of the second typical processing that realizes to S127.(x, another pixel y) is designated as another pixel among the object block Io to have coordinate.Then, the step S123 after the repeating step S122 is to the process of S127.
If the judged result that step S128 produces indication is to all the pixel (x among the object block Io, y) these processes have been carried out, promptly found and be used for current reference vector (vx, vy) final sad value SAD (vx, vy), then the flow process of Chu Liing proceeds to the step S131 of flow chart shown in Figure 28, to produce about whether (vx vy) has carried out the judged result of step S122 to the process of S128 to all reference vectors.
If (promptly to the process of all reference vectors (vx, vy)) execution in step S122 to S128, then the flow process of Chu Liing is not returned step S121 to all reference blocks as yet in judged result indication that step S131 produces.(vx vy) locates to be provided with by another reference vector (vx, vy) another reference block Ii of Zhi Xianging in another piece position of 105 in the hunting zone.Then, the process of repeating step S122 and subsequent step.
If indicate to all reference block locations or to all reference vector (vx in the judged result that step S131 produces, vy) these processes have been carried out, that is, if all be filled final component sad value through all elements of reduction relevance values table, then the flow process of Chu Liing proceeds to step S132.In the table element address (mx, my) locate to detect be stored in through reduction relevance values table or in all elements of reduction SAD table the minimum value in the important final sad value.
Then, at next step S133, create quadratic surface, as being similar at table element address (mx, my) locate detected minimum relatedness value and as near table element address (mx, table element my) and be stored in the curved surface of a plurality of relevance values in reduction relevance values table.As mentioned above, each relevance values all is a sad value.In the situation of present embodiment, as near table element address (mx, table element my) and the quantity that is stored in the relevance values in reduction relevance values table is set to 15.Then, at next step S134, point to the position on the X-Y plane of other precision of mark level the minimum value vector (px, py) be detected as with quadratic surface on the corresponding vector of minimum sad value.The minimum value vector (px, py) position pointed be with quadratic surface on the corresponding position of minimum sad value.
Then, step S135 in the end, by with the minimum value vector (px py) multiply by the reciprocal value n of the reduction factor, calculate motion vector with original size and its original orientation (px * n, py * n), as shown in Figure 6.
Figure 27 and flow chart shown in Figure 28 have been represented the processing of carrying out according to the block-matching technique of realizing according to second typical case that is used to detect the motion vector that is used for an object block.For fragmented frame, as shown in figure 20, need detect a plurality of motion vectors to frame fragment.In this case, for wanting detected each motion vector or each frame fragment, hunting zone and reduction factor 1/n can be reset, so that the processing that execution Figure 27 and flow chart shown in Figure 28 are represented.
Please note, self-evident, even in the situation that second typical case realizes, except above-mentioned quadratic surface method, also can be used as based on the cubic curve method of two cubic curves on the plane that is oriented on vertical and the horizontal direction respectively and to be used to find minimum value vector (px, py) the technology of sensing with other precision of mark level detected position on the hunting zone.
＜the three typical case realizes 〉
If adopt arbitrary method according to definite motion vector of embodiment, even then for the one dimension reduction factor 1/64 that is used to reduce reference vector, the process of determining motion vector can not finish for example to detect the such failure of complete incorrect motion vector, can clearly find out as the effect that represents from method shown in Figure 19.Therefore, in essence, the size that is used as the SAD table of canonical correlation value table can successfully be dwindled with the two dimension reduction factor 1/4096 (=1/64 * 1/64).
In fact, can attempt further dwindling the size of showing (being used as canonical correlation value table) through reduction SAD, described reduction process of reducing the factor 1/4096 or the one dimension reduction factor 1/64 as the use two dimension through reduction SAD table obtains.That is to say, at first, obtain table through reduction SAD by carrying out first process that detects motion vector with the one dimension reduction factor 1/64.Then, before second process of carrying out with the one dimension reduction factor (representative value is 1/8) detection motion vector, the size of hunting zone (corresponding to the relevance values table) is further dwindled, to obtain the new search scope that center and detected motion vector position pointed overlaps.Therefore, carry out the process that detects motion vector, so that the vector error that obtains is narrowed down to value within the marginal range by dwindling the one dimension reduction factor (promptly increasing the value of 1/n).By the one dimension reduction factor suitably being set so that second process detects motion vector with appropriate value, can be with the accuracy detection motion vector of high level.
Below by the flow chart of reference Figure 29 and Figure 32, illustrate and realize that hand tremor trembles the handling process that the 3rd typical case of the operation of motion vector detecting unit 15 realizes.
Basically be based on the processing of the detection motion vector of realizing according to first typical case as the processing of the Figure 29 of the processing that realizes according to the 3rd typical case and flow chart representative shown in Figure 32.Therefore, step S101 with flow chart shown in Figure 25 is just in time identical to the process of S109 respectively to the process of S149 for the step S141 of flow chart shown in Figure 29, and the step S111 with flow chart shown in Figure 26 is just in time identical to the process of S115 respectively and the step S151 of flow chart shown in Figure 30 is to the process of S155.
But under the situation that the 3rd typical case realizes, the processing that detects motion vector does not finish at the step S155 place of flow chart shown in Figure 30.But, be used as first motion vector at the detected motion vector of step S155.Then, at next step S156, has the center that has dwindled big or small new search scope by using by the position conduct pointed of detected motion vector, and narrow down to second by the 1/na that being used for of one dimension being reduced the factor uses from first handles detected motion vector and handle the 1/nb that uses (na＞nb) wherein, the size of the hunting zone in the same reference frame is further dwindled, to obtain the new search scope.
Particularly, detected motion vector points is included in the approximate piece scope that has the reference block of correlation in a way with object block in first motion vector detection is handled.Then, can newly be provided with to be included in a way the approximate piece scope that has a reference block of correlation with object block is the narrower hunting zone at center, as the second new piece scope of using in handling that will detect another motion vector.By using littler reduction vector, can expect, can detect second motion vector with littler error.
Like this, as mentioned above, at step S156, the narrower hunting zone and the one dimension reduction factor of having dwindled are used to carry out second processing of another motion vector of detection to handle identical mode to the step S171 of S168 and flow chart shown in Figure 32 to first motion vector detection of S174 with the step S161 of the step S157 of flow chart shown in Figure 30 and S158, flow chart shown in Figure 31.The step S101 of the process of these steps and flow chart shown in Figure 25 is identical to the process of S115 to the step S111 of the process of S109 and flow chart shown in Figure 26.
Handle by carrying out above-mentioned second motion vector detection, finally detect second motion vector, as required final motion vector at step S174.
By repeating the method for twice execution, realized method according to the detection motion vector of the 3rd typical case's realization according to the detection motion vector of first typical case's realization.But, self-evident, can utilize the hunting zone that reduces gradually (utilization reduce gradually the reduction factor) if necessary, repeat the method for the detection motion vector of realizing according to first typical case more than twice ground.
In addition, self-evident, at the detection motion vector (px that realizes realizing according to the 3rd typical case, during py) method, method according to the second typical detection motion vector of realizing can be performed, and replaces the detection motion vector (px, method py) that realize according to first typical case.In addition, the situation that realizes with aforementioned first and second typical cases is the same, cubic curve method based on two cubic curves on the plane that is oriented to respectively on vertical and the horizontal direction can replace above-mentioned quadratic surface method and be used as motion vector (px, method py) that is used to detect the position of pointing to other precision of mark level.
[second embodiment]
In first embodiment of the image processing apparatus of realizing conduct image pick-up device as shown in figure 24, as prerequisite, the video memory unit 4 of being trembled motion vector detecting unit 15 visits by hand tremor comprises two frame memories that are respectively applied for the image of storing primitive frame (serving as target frame) and reference frame.Therefore, utilize from target frame to have postponed timing, detect motion vector corresponding to the time period of a frame.
On the other hand, in a second embodiment, image processing apparatus has such configuration, and wherein the current image date from image pickup device 11 is taken as reference frame, thereby for the flow data of raster scan instrument, can calculate sad value with real-time mode.
Figure 33 shows the block diagram of realization as second embodiment of the image processing apparatus of image pick-up device.Can find out clearly that from Figure 33 the configuration of image pick-up device 10 is identical with the configuration of the image pick-up device 10 that first embodiment shown in Figure 24 realizes, except in a second embodiment, video memory unit 4 comprises a frame memory 43.
In a second embodiment, the primitive frame that serves as target frame is stored in the frame memory 43, and the frame of exporting as stream from Date Conversion Unit 14 serves as reference frame.The hand tremor of adopting among aforementioned first embodiment is trembled motion vector detecting unit 15 based on the view data that is stored in two frames in frame memory 41 and 42 respectively, carries out the process of calculating as the sad value of the canonical correlation value of representing the correlation between reference block and the reference block.On the other hand, in second embodiment shown in Figure 33, be stored in the image that view data in the frame memory 43 is taken as the primitive frame that serves as reference frame.The view data that is used as reference frame from the stream picture data of Date Conversion Unit 14, and the view data of motion vector detecting unit 15 based on this two frame trembled in hand tremor, carry out to calculate the process as the sad value of the canonical correlation value of representing object block on the target frame and the relevance values between the reference block on the reference frame.
Then, resolution conversion unit 16 cuts out frame image data based on trembled motion vector detecting unit 15 detected motion vectors by hand tremor from frame memory 43.In this way, can export the view data that does not have the influence that hand tremor trembles.Remaining configuration is identical with first embodiment with other operation.
As mentioned above, under the situation of this second embodiment, be used as the view data of reference frame from the stream picture data of Date Conversion Unit 14.Therefore, for certain pixel on the reference frame that imports into, comprise that a plurality of reference blocks of the pixel of the copy that serves as certain pixel exist simultaneously on reference frame.The existence of this reference block is described with reference to Figure 34.
Can find out clearly that from Figure 34 the hunting zone 105 on the reference frame 102 comprises input pixel Din, it is the copy pixel of the pixel D2 on the object block 103 on the primitive frame 101.Input pixel Din is the pixel that comprises on the left side of reference vector 1071 reference block 1061 pointed, also is the pixel that comprises in the upper right corner of reference vector 1072 reference block 1062 pointed.
Therefore, when handling reference block 1061 during the process of the margin of image element between calculating pixel, the pixel value quilt of input pixel Din compares with the pixel D1 on the object block 103.On the other hand, when handling reference block 1062 during the process of the margin of image element between calculating pixel, the pixel value quilt of input pixel Din compares with the pixel D2 on the object block 103.
In order to make explanation be convenient to understand, Figure 34 and Figure 35 are described below, they all show two reference frames.But, in fact, exist to comprise a plurality of reference blocks of importing pixel.
When the sad value that calculates according to this second embodiment between object block and the reference block, brightness value Y by finding the input pixel Din on the processed reference block 106 and as and the absolute value of the difference between the brightness value Y of the pixel value at the some place on the object block 103 of the corresponding point of point of input pixel Din, the calculating pixel value difference.Each absolute value that calculates this difference, the absolute value of difference just is accumulated to and before had been stored in interim and last in the table element, as the absolute value sum of these differences, described table element is included in the SAD table 108 as the table element according to the reference vector 107 that is associated with reference block 106.To each reference vector 107 that is associated with the reference frame 106 that comprises input pixel Din, the process of the absolute value of execution calculating pixel value difference and the process of in table element, storing absolute value.
For example, hypothetical reference piece 1061 is current processed reference blocks.In this case, the brightness value Y by finding the input pixel Din on the reference block 1061 and as and the absolute value of the difference between the brightness value Y of the pixel D1 at the some place on the object block 103 of the corresponding point of point of input pixel Din, the calculating pixel value difference.Then, the absolute value of the difference that calculates is accumulated to and before is stored in interim and last in the relevance values table element (or SAD table element) 1091, as the absolute value sum of these differences, described table element is included in the relevance values table shown in Figure 35 (or SAD table) 108 as the table element according to the reference block 1071 that is associated with reference block 1061.To each reference vector 107 that is associated with the reference frame 106 that comprises input pixel Din, carry out the calculating pixel value difference absolute value process and with the absolute value that calculates be added to calculate and formerly be stored in interim in the SAD table element 1091 and process.For example, reference vector 1072 is associated with the reference block 1062 that also comprises input pixel Din.In this case, as described below, to the reference vector reference vector 1072 that is associated with reference block 1062, carry out the calculating pixel value difference absolute value process and with the absolute value that calculates be added to calculate and formerly be stored in interim in the SAD table element 1092 and process, described table element is included in the relevance values table shown in Figure 35 (or SAD table) 108 as the table element according to reference vector 1072.
When reference block 1062 is processed, brightness value Y by finding the input pixel Din on the reference block 1062 and as and the absolute value of the difference between the brightness value Y of the pixel D2 at the some place on the object block 103 of the corresponding point of point of input pixel Din, the calculating pixel value difference.Then, the absolute value of the difference that calculates is accumulated to and before is stored in interim and last in the SAD table element 1092, as the absolute value sum of these differences, this table element is included in the SAD table 108 shown in Figure 35 as the table element according to the reference vector 1072 that is associated with reference block 1062.
Above-mentioned to wherein each all comprises the processing of all reference blocks 106 (for example reference block 1061 and 1062) execution of importing pixel Din to all the input pixel Din execution in the hunting zone 105, and, when all the input pixel Din in the hunting zone 105 are finished this processing, each table element 109 of SAD table 108 all comprises final sad value, and the establishment of SAD table 108 is finished.
Explanation with reference to Figure 35 is set up equally for following situation: the technology of prior art is applied to the process of calculating sad value with real-time mode.As described in earlier in respect of figures 35, each in the SAD table element 1091 and 1092 is the typical SAD table element 109 that comprises in SAD table 108 as the element that is associated with reference vector 1071 and reference vector 1072 respectively.On the other hand, in this second embodiment, each table element 109 of relevance values table (or SAD table) 108 is not final sad value, the accumulation of the absolute value that described final sad value is above-mentioned margin of image element and.But, being very similar to above-mentioned first embodiment, SAD table 108 is reduced to through reduction relevance values table (or through reduction SAD table), and each table element through reducing relevance values table (or through reduction SAD table) all is by carrying out the value that following steps obtain:
The input pixel in the hunting zone on the calculating reference frame 106 and the absolute difference of the margin of image element between the respective pixel on the target frame;
Point to the reference vector 107 of reference block 106 with reduction factor 1/n reduction;
By adopting the linear weighted function distribution technique, the absolute difference that calculates is divided into a plurality of component absolute differences; And
The component absolute difference is added to previous calculating and be stored in interim in a plurality of table elements and, described a plurality of table element and a plurality of corresponding approaching being associated through the reduction reference vector, described approaching through the reduction reference vector near existing of obtaining as the result that reference vector 107 is reduced through the reduction vector.
Each reference vector 107 that comprises the reference block 106 of importing pixel for sensing is carried out above-mentioned steps, to obtain to be stored in the value in the table element.Each input pixel is repeated pointing to the step of all reference vectors 107 execution of sharing the reference block 106 of importing pixel.When each the input pixel that comprises in to the hunting zone has all been finished the execution of these steps, be done through reduction relevance values table (or through reduction SAD table).
After reduction SAD table is finished, can be by adopting and the identical technology of first embodiment, carry out according to the detection of second embodiment process of motion vector accurately.As previously mentioned, the typical technology that adopts of first embodiment is the quadratic surface technology and technology based on the cubic curve on the plane that is oriented on vertical and the horizontal direction.
Figure 36 and Figure 37 show the flow chart that the performed processing of motion vector detecting unit 15 is trembled in hand tremor that the image pick-up device 10 according to second embodiment that representative is used for detecting motion vector adopts.
Flow chart starts from step S181, at step S181, hand tremor tremble motion vector detecting unit 15 receive arbitrfary point on the frame that comprises in the input picture as the reference frame (x, the pixel data Din of the pixel of y) locating (x, y).Then, at next step S182, specify to point to one of a plurality of reference block Ii reference vector (vx, vy), wherein each reference block Ii all comprise the position (x, the input pixel Din that y) locates (x, y).
Make mark Ii that (x y) indicates that (vx, vy) (x, the pixel value of the pixel of y) locating make mark Io that (x-vx y-vy) indicates point (x-vx, the pixel value of the pixel of y-vy) locating on the object block Io to the point on the reference block Ii pointed by reference vector.Hereinafter, the point among the object block Io (x-vx, y-vy) be said to be with reference block Ii in point (x, y) corresponding point.Then, at next step S183, according to following equation (3) calculating pixel value Ii (x, y) and pixel value Io (x-vx, y-vy) the absolute value α of the difference between:
α＝|Ii(x，y)-Io(x-vx，y-vy)| ——(3)
Then, at next step S184, the reference vector by pointing to reference block Ii with reduction factor 1/n reduction (vx, vy), calculate through the reduction reference vector (vx/n, vy/n).Usually, (vx/n vy/n) comprises fractional part for each x direction through reducing reference vector that obtains and y direction value.
Next, at next step S185, it is approaching through reduction reference vector (vx/n, approaching reference vector vy/n) to identify a plurality of positions.As previously mentioned, each approaching reference vector all be have integer vx/n value and integer vy/n value through the reduction reference vector.In the present embodiment, the quantity of approaching reference vector is set to 4.Then, at next step S186, foregoing by adopting based on the position of approaching reference vector sensing with through reduction reference vector (vx/n, vy/n) the linear weighted function distribution technique of the relation between the position of Zhi Xianging, it is poor that the absolute value α that finds at step S183 as margin of image element is divided into four components.Next, at next step S187, four component differences are distributed between four table elements that are included in reduction relevance values table, as four table elements that are associated with four approaching reference vectors respectively.
After the process of completing steps S187, flow process according to the processing of second embodiment proceeds to next step S188, to produce about judging whether to wherein each all points to and comprises input pixel Din (x, y) (vx vy) has carried out the judged result of step S183 to the process of S187 to all reference vectors of reference block Ii.If judged result indication as yet not to wherein each all point to comprise input pixel Din (x, all reference vectors of reference block Ii y) (vx, vy) all execution in step S183 is to the process of S187, then the flow process of Chu Liing is returned step S182.Specify to point to wherein each all comprise input pixel Din (x, another reference vector of one of a plurality of reference block Ii y) (vx, vy).Then, the step S183 after the repeating step S182 is to the process of S187.
On the other hand, if the judged result indication that produces at step S188 is to wherein each all points to and comprises input pixel Din (x, y) all reference vector (vx of reference block Ii, vy) all carried out the process of step S183 to S187, then proceed to the step S191 of flow chart shown in Figure 37 according to the processing stream of second embodiment, to produce about whether (x y) has carried out the judged result of step S182 to the process of S188 to all the input pixel Din in the hunting zone 105.If the judged result indication is not as yet to all the input pixel Din (x in the hunting zone 105, y) all execution in step S182 to the process of S188, then return step S181 according to the flow process of the processing of second embodiment, at step S181, receive another point (x on the frame, the pixel data Din of another pixel of y) locating (x, y).Then, carry out the process of subsequent step.
On the other hand, (x y) has carried out the process of step S182 to S188, and then the flow process according to the processing of second embodiment proceeds to step S192 if the judged result indication that produces at step S191 is to all the input pixel Din in the hunting zone 105.In the table element address (mx, my) locate to detect through reduction relevance values table or in all elements of reduction SAD table, store the minimum value in the important final sad value.
Then, at next step S193, create quadratic surface as the curved surface of approximate following relevance values, described relevance values is at table element address (mx, my) locate detected minimum relatedness value and as near the table element address (mx, table element my) be stored in a plurality of relevance values in reduction relevance values table.As mentioned above, each relevance values all is a sad value.In a second embodiment, as (mx, the quantity that is stored in the relevance values in reduction relevance values table of table element my) is set to 15 near the table element address.Then, at next step S194, point to the position on the X-Y plane of other precision of mark level the minimum value vector (px, py) be detected as with quadratic surface on the corresponding vector of minimum sad value.The minimum value vector (px, py) position pointed be with quadratic surface on the corresponding position of minimum sad value.
Then, step S195 in the end, by with the minimum value vector (px py) multiply by the reciprocal value n of the reduction factor, calculate motion vector with original size and its original orientation (px * n, py * n), as shown in Figure 6.
It should be noted that, same in the situation of second embodiment, the situation that realizes with first to the 3rd typical case of aforementioned first embodiment is the same, replace above-mentioned quadratic surface method, cubic curve method based on two cubic curves on the plane that is oriented to respectively on vertical and the horizontal direction also can be used as motion vector (px, method py) that detects the position of pointing to other precision of mark level.
In addition, self-evident, same in the situation of second embodiment, realize the same with the 3rd typical case of aforementioned first embodiment, can be in the constriction hunting zone and change if necessary in the reduction factor, repeat the processing that detects motion vector through reduction SAD table by using in two or more stages.
Second embodiment provides such advantage: compare with first embodiment, the size of frame memory can reduce a frame, therefore, can shorten the storage required time of input picture in frame memory.Self-evident, the effect that memory size is dwindled can be described.But the storage required short time of input picture also is considered to important feature in recent years in frame memory.Particularly, in the system that handles motion picture, the storage required short time of input picture itself makes system delay shorten in frame memory.Therefore, eliminate practical object and its incompatible sensation that difference caused between the image that manifests on the display floater that picture picks up and the interest of product is had very big effect causing the user.
[the 3rd embodiment]
In above-mentioned first and second embodiment, supposed to be used for coming the system of compensating motion picture at the influence that hand tremor is trembled, and by supposing to have provided description to embodiment as prerequisite.But the block-matching technique that the embodiment of the invention provides also can expand to this technology at an easy rate to being used for compensating at the influence that hand tremor is trembled the application of the system of still picture.The 3rd embodiment is the embodiment that applies the present invention to compensate at the influence that hand tremor is trembled the system of still picture.
In the 3rd embodiment, be very similar to second embodiment, utilization is taken as the input picture frame of the reference frame of wanting processed, detects such motion vector, its indication with taken the lead mobile corresponding to the relevant input picture frame of the target image frame of time period of a frame than input picture frame.But, in the 3rd embodiment, by when the influence of trembling at hand tremor comes compensating images, will be that a plurality of images of taking continuously in the image pick-up operation of 3fps superpose each other with the typical rate, carry out the process that the influence of trembling at hand tremor compensates still picture.
Because can be by when the influence of trembling at hand tremor comes compensating images, a plurality of images of taking continuously in the image pick-up operation are superposeed each other, carry out the process that compensates still picture according to the influence of trembling at hand tremor of the 3rd embodiment, it is necessary therefore not allowing the pixel precision greater than the error of a pixel.That is to say, in the situation of the 3rd embodiment, tremble motion vector, except the parallel mobile component on representative level and the vertical direction, also be necessary to detect rotational component as hand tremor.
Figure 38 is the block diagram according to the Typical Disposition of the image pick-up device of the 3rd embodiment.Tremble rotation adding device 19 between motion vector detecting unit 15 and the resolution conversion unit 16 by be added on hand tremor to configuration shown in Figure 33 as the configuration of second embodiment, and, obtained Typical Disposition shown in Figure 38 to the video memory unit 4 interpolation frame memories 44 that comprise frame memory 43.Frame memory 44 is used as rotational component that detects motion vector and the memory that two field picture is superposeed each other.Remainder is identical with configuration shown in Figure 33.
With with the identical mode of aforementioned second embodiment as the embodiment that handles motion picture, by the input pixel data that will receive from Date Conversion Unit 14 view data as the reference frame, and use the view data that is stored in the frame memory 43 as the view data of serving as the primitive frame of target frame, carry out the processing that detects motion vector.In the situation of the 3rd embodiment, for a frame, detection type is similar to a plurality of motion vectors shown in Figure 20.
In addition, in the situation of the 3rd embodiment, hand tremor is trembled the information that motion vector detecting unit 15 trembles motion vector detecting unit 15 detected a plurality of motion vectors with hand tremor and is offered rotation adding device 19.These motion vectors of rotation adding device 19 integrated treatments are trembled motion vector so that detect the hand tremor of the parallel mobile component of representative and the anglec of rotation.
Then, after a frame delay, rotation adding device 19 is trembled the motion vector and the anglec of rotation according to hand tremor, cut out the picture frame that is stored in the frame memory 43, simultaneously, in the frame that rotation cuts out, the frame that cuts out is added on the picture frame that is stored in the frame memory 44, or calculate the frame cut out and be stored in the average of picture frame in the frame memory 44.By repeating these processes, picture frame 120 can be stored in the frame memory 44, as do not show the influence that hand tremor trembles and have higher S/N than and than the rest image of high image resolution.Figure 39 shows the process that in rotation and translation stack frame frame is superposeed each other.
Then, the two field picture of resolution conversion unit 16 from be stored in frame memory 44, cut out image section, as previously mentioned this part is offered coding/decoding unit 17, as the data of the captured image that will write down with predetermined picture resolution and predetermined picture size.Resolution conversion unit 16 also offers NTSC encoder 18 with this image section, as the data of the image that will be monitored.
Same in the situation of the 3rd embodiment, by use through reduction relevance values table (or through reduction SAD table) detect the processing of each motion vector can be in the constriction hunting zone and change the reduction factor if necessary in, repeat in two or more stages.Tremble the influence that the process neutralization of motion vector trembles at hand tremor in the hand tremor that detects still picture according to the 3rd embodiment and compensate in the process of still picture, almost without limits the real-time characteristic of process.Pixel quantity increases the feasible accuracy detection motion vector that is necessary with high level, and still, for such necessity, the processing that detects the motion vector of layerings in a plurality of stages becomes very effective.
In above-mentioned the 3rd embodiment, detect motion vector by adopting the method identical with second embodiment.But should be noted that also and can in the 3rd embodiment, detect motion vector by adopting the method identical with first embodiment.
[the 4th embodiment]
Above-mentioned the 3rd embodiment adopts and compares the method that the motion vector and the anglec of rotation are trembled in hand tremor that detects by the image with input picture and leading input picture one frame.But in fact, first frame is taken as the basis, and subsequent frame is superimposed on this basis as shown in figure 39.Therefore, first frame is treated more modestly as the basis in the process that detects motion vector, so that reduce error.The 4th embodiment is the embodiment that considers this point.Figure 40 shows the block diagram according to the Typical Disposition of the image pick-up device of the 4th embodiment.
In the 4th embodiment, except frame memory 43 and 44, the extra frame memory 45 that video memory unit 4 also comprises.The view data of Date Conversion Unit 14 outputs is stored in frame memory 43 and 45.
The 4th embodiment is the system with such configuration, wherein frame memory 45 is used to store first frame and serves as target frame, target frame is also referred to as primitive frame, and detected with the motion vector of the vector that moves of the image-related input picture of basic frame as representative.In this configuration, the result that image adds process is stored in the frame memory 44.Remainder is identical with the 3rd embodiment.
By first frame of input picture is used as base image, the 4th embodiment has realized carrying out the system of unlimited interpolation process or on average unlimited/interpolation process.But it should be noted that, if memory is fully available or permission interim operation of preserving data in data recording/reproducing device 5, the all images that then will experience unlimited interpolation process or unlimited average/add process can be kept in memory or the data recording/reproducing device 5 in advance, then on contest (tournament) basis to the image implementation.
[the 5th embodiment]
Technology by combination comes the no sensor, method of compensating images according to the influence of trembling at hand tremor of first to the 4th embodiment and the influence of trembling at hand tremor on the up-to-date technology basis comes compensating images can obtain better effect.
The beginning of patent specification has illustrated that partly the influence by using gyro sensor to tremble at hand tremor comes the process of compensating images only to obtain rough compensation, and the rotation compensation technology is difficult to realize.On the other hand, come the no transducer process of compensating images that the precision of the high level that comprises rotation compensation is provided by the influence of adopting block-matching technique to tremble at hand tremor.But if the hunting zone broadens, then the cost of SAD table raises rapidly, perhaps, even employing according to the block matching method of embodiment, is carried out the process that detects motion vectors in a plurality of stages and has also been prolonged the execution required time of entire process.
In order to address the above problem, must provide the system that comes compensating images with the influence of trembling at hand tremor of precision low-cost, very high level and high processing rate by carrying out following steps:
Come compensating images at the optical effect that hand tremor is trembled, to obtain rough compensation;
Dwindle the size that is used for the hunting zone of when the influence that sensor-level safety pin adversary trembles comes the motion vector of compensating images, using in detection;
Detect the motion vector in the hunting zone; And
The influence of trembling at hand tremor with no approach sensor comes compensating images.
As mentioned above, by adopting block-matching technique, can solve the problem that prior art systems produced that the influence of trembling at hand tremor comes compensating images according to first to the 5th embodiment.The problem that prior art systems produces comprises that relevance values table scale is big and because relevance values table scale causes almost being difficult to carry out the rest image compensation greatly.But, according to embodiment, not only can significantly dwindle the size of relevance values table, and can realize at an easy rate that the influence of trembling at hand tremor compensates the method for still picture.
In addition, compare with the various sensorless technologies that are used for compensating still picture at the influence that hand tremor is trembled of present proposition, the realization of trembling compensation method according to the no transducer hand tremor based on block-matching technique of first to the 5th embodiment provides and has comprised following advantage: low-cost, the precision of high level, short processing time and good robustness.
Compensate as the influence of trembling that obtainable every kind of device all is the system that adopts the combination optical compensation technique that comprises gyro sensor method and lens offset method on the current market of device of rest image at hand tremor.But such system has introduced the unfavorable picture quality of big sum of errors.But, on the other hand,, can remove transducer and other mechanical components according to the technology that the embodiment of the invention provides, so as the permission system be implemented as with low-cost and very the influence of trembling of the precision of high level at hand tremor compensate the device of rest image.
[other embodiment]
In the situation of the foregoing description, reference vector is reduced on vertical and horizontal direction with the identical reduction factor.But reference vector also can be reduced on vertical and horizontal direction with the different reduction factors.
In addition, in the above-described embodiments, calculate sad value on the basis of the pixel value of all pixels in reference block and object block, it is used as the relevance values of representing the correlation between reference and the object block.But, calculating sad value on the basis of each pixel that also can be in reference block and the pixel value of every k the pixel (wherein k is an integer) in the object block, it is used as the relevance values of representing the correlation between reference and the object block.
In addition, the system that is used as with real-time mode detection motion vector has dwindled required processing cost and the time of execution processing.Therefore, in such system, usually necessary ferret out piece is to find out above-mentioned representational point, and on the basis of the pixel value at the pixel value at the representational some place in object block and the some place that in reference block, comprises, calculate the relevance values such as sad value.
Particularly, for example, object block 103 is divided into a plurality of sub-pieces, and each sub-piece comprises a * b pixel, and perhaps each comprises " a " individual pixel column and " b " individual pixel column, and as shown in figure 41, wherein a and b equal 1 integer at least.Then, one of a plurality of pixels in each sub-piece are taken as representational point (or impact point) TP of sub-piece.Then, on the basis of the pixel value at the some place that comprises in the pixel value of the selecteed representational some TP that in object block 103, comprises and the reference block 106, calculate the relevance values such as sad value.
But the pixel value of all pixels in the reference block 106 still is used in the process of calculating such as relevance values such as sad values.Particularly, reference block 106 is divided into a plurality of pixel coverage AR, each pixel coverage AR comprises a * b pixel as the sub-piece in the object block 103 (or impact point TP), and all a * b pixel among the pixel coverage AR all combines in the process of calculating such as relevance values such as sad values with pixel value corresponding to the impact point TP place in the sub-piece of pixel coverage AR and is used.
More specifically, calculate the impact point TP in the sub-piece on the object block 103 and the absolute value of the margin of image element between each pixel in a * b the pixel among the pixel coverage AR on the reference block 106, find then at the sub-piece of this absolute value of a * b pixel calculating with.To each sub-piece on the object block 103 (or each impact point TP) find this sub-piece and, then, calculate this seed block of finding at all sub-pieces on the object block 103 (or all impact point TP) and piece and.The piece that calculates and be the sad value of object block 103 and reference block 106, and be stored in the relevance values table (or SAD table) 108 element as table 108.
Then, to each reference block 106 in the hunting zone 105 that provides for object block 103, as described above on the basis of the impact point TP in object block 103, find piece and (or sad value) of object block 103 and reference block 106, and it is stored in the relevance values table (or SAD table) 108 element as table 108, so that ST Stuffing Table 108 is promptly so that finish the establishment of table 108.But, in the situation of this additional embodiments, because each in a plurality of reference blocks 106 that are provided with in hunting zone 105 all comprises a * b pixel, so the center of reference block 106 has been offset the corresponding distance of a multiple pixel with a * b pixel or a * b each other.
Representational point in the object block is used to calculate in the situation of device of the sad value that serves as the value of representing the correlation between object block and the reference block therein, for each the pixel coverage AR that comprises a plurality of pixels in the reference block, reference to storage once, to find out the impact point TP on the object block.Therefore, can be greatly reduced the access times of memory, accessed because the impact point TP on the object block needs.
In addition, in the situation of the device of the representational point in using object block, the view data of the pixel at the impact point TP place on the object block need be stored in the memory.That is to say, do not need to store the view data of all pixels on the object block.Therefore, the size that is used to store the frame memory of the primitive frame that serves as the target frame that comprises object block also can be dwindled greatly.
In addition, except frame memory, the representational some memory that is implemented as SRAM also can be provided in this locality, as the local storage that is used to store as the view data of the object block on the primitive frame of target frame.In this way, can reduce bandwidth to the visit of the video memory unit 4 that is implemented as DRAM.
By arriving Figure 44,, the process that adopts the technology of the representational point that uses object block has been described above at the technology that illustrates previously with reference to Figure 42.But, self-evident, the explanation of the process of technology that adopt to use representational point also is applicable to as the method for front by describing to Figure 37 with reference to Figure 34 according to the method for second embodiment.
By carrying out following steps, the technology of the representational point of use object block is applied to the method according to second embodiment:
For each the input pixel in the whole hunting zone, detect all reference blocks of the pixel coverage AR that shares the pixel (being called the input pixel) that has comprised the input reference frame; And
Determine a plurality of representational point on the object block as such point, wherein each point is corresponding to one of pixel coverage AR in each reference block in all detected reference blocks.Should be noted that the input locations of pixels among the pixel coverage AR is different with AR.
Then, for the input pixel among the pixel coverage AR, the pixel value of the pixel that one of representational point that is positioned at object block is located is read by the memory from the view data that is used to store the primitive frame that serves as target frame, and by and the pixel value one of input pixel be used from the absolute value of the difference of calculating between the pixel value of the pixel value of the pixel at representational some place and input pixel.Then, each component value of absolute value is accumulated to the component value of being stored in the element included in serving as the SAD table of relevance values table that had before calculated, as with the corresponding element of reference vector that points to reference block.
In above-mentioned processing, memory is conducted interviews to read wherein the pixel value that each all is positioned at the pixel that one of representational point locates.Therefore, the access times to memory can significantly reduce.
Note, self-evident, in the situation of the embodiment that has described, be applied to use in the situation of reduction relevance values table (or through reduction SAD table) based on the processing of representational point.
In addition, in the above-described embodiments, each is calculated the absolute value of the difference of pixel value and sad value as relevance values by handling brightness value Y.But in order to detect motion vector, processed pixel value is not limited to brightness value Y.That is to say that chromatic value Cb/Cr also can be taken as processed pixel value.In addition, be converted into brightness value Y and chromatic value Cb/Cr initial data before and also can in the processing that detects motion vector, be taken as processed pixel value.
In addition, as previously mentioned, the configuration that motion vector detecting unit 15 is not limited to be carried out by hardware the processing that detects motion vector is trembled in hand tremor.That is to say, detect the processing of motion vector and also can carry out by software.
Processing according to the detection motion vector of arbitrary embodiment is carried out on captured image.But notice that self-evident, the information that embodiments of the invention are applied to is not limited to captured image.
In addition, described as beginning to locate at specification of the present invention, represent the value of the correlation between object block and the reference block to be not limited to sad value.
In addition, the description that has provided has illustrated the example of trembling the situation application embodiment of the invention of motion vector to the hand tremor that detects the image of being taken by image pick-up device.But scope of the present invention is not limited to detect the hand tremor of the image of being taken by image pick-up device and trembles motion vector.That is to say that the present invention also can be applicable to detect the hand tremor of moving of the another location of representative from arbitrary picture position to picture and trembles motion vector.The picture of this position and another position is called target and reference picture (or target and reference frame).In this case, the picture formerly on any the be taken as time shaft in target picture or the reference picture, and another picture is taken as at the back picture.In the above-described embodiments, detected because motion vector is trembled in hand tremor, institute's target picture is taken as picture formerly.But, when reference picture is taken as formerly picture, also can detect motion vector at the picture of blotter.
In addition, it will be understood by those of skill in the art that depending on design needs and other factors, multiple modification, combination, sub-portfolio and replacement may take place, they drop in the scope of claims or its equivalent.
The present invention comprises and the Japanese patent application JP 2005-347437 theme of submitting to Japan Patent office respectively on December 1st, 2005 and on July 18th, 2006 relevant with JP 2006-195047, and the full content of these two parts of applications all is contained in this by reference.

Claims (34)

1. one kind is used for by carrying out the image processing apparatus of following steps identification motion vector:
A plurality of reference blocks are set as such reference block in the hunting zone that is provided with on reference picture, in the described reference block each has the size that equates with object block, and the object block that described object block conduct has the pre-sizing that covers a plurality of pixels is set at the pre-position on the target picture;
Search for specific reference block, described specific reference block has the strongest correlation in the correlation between described a plurality of reference blocks and described object block; And
Based on described specific reference block from described target picture on the skew of corresponding position, described precalculated position of described object block, detect described motion vector,
Described image processing apparatus comprises:
The relevance values calculation element, described relevance values calculation element is configured to the pixel value and the pixel value that is positioned at a plurality of pixels at the respective point place on the described object block based on a plurality of pixels that are positioned at the some place on each described reference block, calculate relevance values, each relevance values is represented the correlation between in the described reference block and the described object block;
Table creating device, described table creating device are configured to generate through reduction relevance values table by following operation:
To be stored in by the relevance values former state that described relevance values calculation element calculates at reference vector and be included in the described table element in reduction relevance values table, as with the table element that is associated through the reduction reference vector that obtains as reducing the result of described reference vector with the predetermined reduction factor, described reference vector representative reference block on the described reference picture from described target picture on the skew of corresponding position, described precalculated position of described object block; Perhaps
By dividing by described relevance values calculation element at being reduced to produce the described relevance values that calculates through the described reference vector of reduction reference vector, find a plurality of component relevance values, in described a plurality of component relevance values each and position are associated through in the reduction reference vector specific one of reduction reference vector a plurality of approaching near described, and each accumulation in the described component relevance values be stored in be included in the described table element in reduction relevance values table, as with the described specific approaching table element that is associated through the reduction reference vector; And
Motion vector calculation device, described motion vector calculation device are configured to find the motion vector between described target picture and the described reference picture based on being stored in described described component relevance values in reduction relevance values table.
2. image processing apparatus according to claim 1, wherein, described table creating device also comprises:
Component relevance values calculation element, if described component relevance values calculation element is configured to the location near described a plurality of approaching through the reduction reference vector through the reduction reference vector, then by dividing by described relevance values calculation element at being reduced to produce the described relevance values that calculates through the described reference vector of reduction reference vector, find a plurality of component relevance values, in the described component relevance values each and position near described through reduction reference vector a plurality of approaching in the reduction reference vector specific one be associated, and in the described component relevance values each is stored in is included in the described table element in reduction relevance values table, as with the described specific approaching table element that is associated through the reduction reference vector, described storage operation is usually carried out by adding described component relevance values to described list cell; And
Final component relevance values calculation element, described final component relevance values calculation element is configured to find final component relevance values, described final component relevance values representative by will be added to by the new component relevance values that described component relevance values calculation element newly calculates be stored in as the component relevance values that had before calculated by described component relevance values calculation element the accumulation that obtains on the described previous component relevance values in the table element of reduction relevance values table with.
3. image processing apparatus according to claim 1, wherein, described relevance values calculation element calculates relevance values at each described reference block, and described calculating is to be undertaken by the absolute value sum that finds the difference between the pixel value of a plurality of pixels at the some place on the described reference block and the pixel value in a plurality of pixels at the respective point place on the described object block.
4. one kind is used for by carrying out the image processing apparatus of following steps identification motion vector:
A plurality of reference blocks are set as such reference block in the hunting zone that is provided with on reference picture, in the described reference block each has the size that equates with object block, and the object block that described object block conduct has the pre-sizing that covers a plurality of pixels is set at the pre-position on the target picture;
Search for specific reference block, described specific reference block has the strongest correlation in the correlation between described a plurality of reference blocks and described object block; And
Based on described specific reference block from described target picture on the skew of corresponding position, described precalculated position of described object block, detect described motion vector,
Described image processing apparatus comprises:
The relevance values calculation element, described relevance values calculation element is configured to the pixel value and the pixel value that is arranged in a plurality of each pixel of pixel at the respective point place on the described object block at a plurality of each pixel of pixel that are arranged in the some place on each described reference block, calculates relevance values;
Table creating device, described table creating device are configured to generate through reduction relevance values table by following operation:
To be stored in by the relevance values former state that described relevance values calculation element obtains at the calculated for pixel values of the pixel on the reference vector reference block pointed and be included in the described table element in reduction relevance values table, as with the table element that is associated through the reduction reference vector that obtains as reducing the result of described reference vector with the predetermined reduction factor, described reference vector representative reference block on the described reference picture from described target picture on the skew of corresponding position, described precalculated position of described object block; Perhaps
By dividing by described relevance values calculation element at the relevance values that is obtained with the calculated for pixel values that produces described pixel on the described reference vector of reduction reference vector reference block pointed by reduction, find a plurality of component relevance values, in the described component relevance values each and position are associated through in the reduction reference vector specific one of reduction reference vector a plurality of approaching near described, and each accumulation in the described component relevance values be stored in be included in the described table element in reduction relevance values table, as with the described specific approaching table element that is associated through the reduction reference vector; And
Motion vector calculation device, described motion vector calculation device are configured to find the motion vector between described target picture and the described reference picture based on being stored in described described component relevance values in reduction relevance values table.
5. image processing apparatus according to claim 4, wherein, described table creating device also has:
Component relevance values calculation element, if described component relevance values calculation element is configured to the location near described a plurality of approaching through the reduction reference vector through the reduction reference vector, then by dividing by described relevance values calculation element at the relevance values that is obtained with the calculated for pixel values that produces described pixel on the described reference vector of reduction reference vector reference block pointed by reduction, find a plurality of component relevance values, in the described component relevance values each and position are associated through in the reduction reference vector specific one of reduction reference vector a plurality of approaching near described, and in the described component relevance values each is stored in is included in the described table element in reduction relevance values table, as with the described specific approaching table element that is associated through the reduction reference vector, described storage operation is usually carried out by adding described component relevance values to described list cell; And
Final component relevance values calculation element, described final component relevance values calculation element is configured to find final component relevance values, described final component relevance values representative by will be added to by the new component relevance values that described component relevance values calculation element newly calculates be stored in as the component relevance values that had before calculated by described component relevance values calculation element the accumulation that obtains on the described previous component relevance values in the table element of reduction relevance values table with.
6. image processing apparatus according to claim 4, wherein, described relevance values calculation element is at the pixel value and the pixel value that is arranged in a plurality of each pixel of pixel at the respective point place on the described object block of a plurality of each pixel of pixel that are arranged in the some place on each described reference block, calculate relevance values, described calculating is that the absolute value by the difference between the pixel value of the described pixel on pixel value that finds the described pixel on the described reference block and the described object block carries out.
7. image processing apparatus according to claim 1, wherein, each is the picture that conduct is obtained by the result of the image pick-up operation of image pick-up device execution for described target picture and reference picture, and described target picture is to lead over the picture of described reference picture.
8. image processing apparatus according to claim 4, wherein, each is the picture that conduct is obtained by the result of the image pick-up operation of image pick-up device execution for described target picture and reference picture, and described target picture is to lead over the picture of described reference picture.
9. image processing apparatus according to claim 1, wherein
Based on the described motion vector that is found by described motion vector calculation device, the described hunting zone on the described reference picture is by constriction, and the described predetermined reduction factor is reduced; And
Described by using by the region of search of constriction and the described reduced reduction factor, be repeated by described relevance values calculation element, described table creating device and the performed process of described motion vector calculation device.
10. image processing apparatus according to claim 4, wherein:
Based on the motion vector that is found by described motion vector calculation device, the described hunting zone on the described reference picture is by constriction, and the described predetermined reduction factor is reduced; And
Described by using by the region of search of constriction and the described reduced reduction factor, be repeated by described relevance values calculation element, described table creating device and the performed process of described motion vector calculation device.
11. image processing apparatus according to claim 2, wherein, described component relevance values calculation element based on described through reduction reference vector position pointed and described approaching distance between reduction reference vector position pointed, find described component relevance values, each in the described component relevance values and position are associated through in the reduction reference vector one of reduction reference vector a plurality of approaching near described.
12. image processing apparatus according to claim 5, wherein, described component relevance values calculation element based on described through reduction reference vector position pointed and described approaching distance between reduction reference vector position pointed, find described component relevance values, each in the described component relevance values and position are associated through in the reduction reference vector one of reduction reference vector a plurality of approaching near described.
13. image processing apparatus according to claim 1, wherein, described predetermined reduction is the vertical direction reduction factor and the horizontal direction reduction factor that can be provided with independently of one another.
14. image processing apparatus according to claim 4, wherein, described predetermined reduction is the vertical direction reduction factor and the horizontal direction reduction factor that can be provided with independently of one another.
15. image processing apparatus according to claim 1, wherein, described relevance values reduces inversely with the correlation of being represented by described relevance values, and described motion vector calculation device comprises:
The curved surface generating apparatus, described curved surface generating apparatus is configured to create approximate high-order curved surface based on the minimum value in the described relevance values of storing with as the approaching relevance values that is stored in described relevance values table with the approaching element of the element position that is used for storing described minimum value in reduction relevance values table;
Smallest point checkout gear, described smallest point checkout gear are configured to detect and the corresponding point of the minimum value of described approximate high-order curved surface; And
The vector detection device, described vector detection device be configured to detect representative from the corresponding position of described object block to as with the corresponding described detected point of described minimum value by the vector of the offset of the detected point of described smallest point checkout gear, and from described detected vector calculating kinematical vector.
16. image processing apparatus according to claim 4, wherein, described relevance values reduces inversely with the correlation of being represented by described relevance values, and described motion vector calculation device comprises:
The curved surface generating apparatus, described curved surface generating apparatus is configured to create approximate high-order curved surface based on the minimum value in the described relevance values of storing with as the approaching relevance values that is stored in described relevance values table with the approaching element of the element position that is used for storing described minimum value in reduction relevance values table;
Smallest point checkout gear, described smallest point checkout gear are configured to detect and the corresponding point of the minimum value of described approximate high-order curved surface; And
The vector detection device, described vector detection device be configured to detect representative from the corresponding position of described object block to as with the corresponding described detected point of described minimum value by the vector of the offset of the detected point of described smallest point checkout gear, and from described detected vector calculating kinematical vector.
17. image processing apparatus according to claim 15, wherein, described approximate high-order curved surface is by described curved surface generating apparatus, based on described minimum value and described approaching relevance values, by the quadratic surface that adopts least square method to create.
18. image processing apparatus according to claim 16, wherein, described approximate high-order curved surface is by described curved surface generating apparatus, based on described minimum value and described approaching relevance values, by the quadratic surface that adopts least square method to create.
19. image processing apparatus according to claim 17, wherein, described approaching relevance values is that the element that is stored in respectively to be used for storing described minimum value is 8 relevance values of 8 elements at center.
20. image processing apparatus according to claim 18, wherein, described approaching relevance values is that the element that is stored in respectively to be used for storing described minimum value is 8 relevance values of 8 elements at center.
21. image processing apparatus according to claim 19, wherein, described approaching relevance values is that the element that is stored in respectively to be used for storing described minimum value is 15 relevance values of 8 elements at center.
22. image processing apparatus according to claim 20, wherein, described approaching relevance values is that the element that is stored in respectively to be used for storing described minimum value is 15 relevance values of 8 elements at center.
23. according to the described image processing apparatus of claim 1, wherein, described relevance values reduces inversely with the correlation of being represented by described relevance values, and described motion vector calculation device comprises:
Horizontal direction curve generating apparatus, described horizontal direction curve generating apparatus is configured to based on the minimum value in the described relevance values of storing in reduction relevance values table and is stored in the approaching relevance values of horizontal direction of described relevance values table as the approaching element of the element with being used for storing described minimum value that comprises in the horizontal direction, creates the approximate luminance curve of horizontal direction;
Horizontal direction smallest point checkout gear, described horizontal direction smallest point checkout gear are configured to detect the corresponding point of horizontal direction minimum value that is similar to luminance curve with described horizontal direction;
Vertical direction curve generating apparatus, described vertical direction curve generating apparatus is configured to based on the minimum value in the described relevance values of storing in reduction relevance values table and is stored in the approaching relevance values of horizontal direction of described relevance values table as the approaching element of the element with being used for storing described minimum value that comprises in vertical direction, creates the approximate luminance curve of vertical direction;
Vertical direction smallest point checkout gear, described vertical direction smallest point checkout gear are configured to detect the corresponding point of vertical direction minimum value that is similar to luminance curve with described vertical direction; And
Device for detecting motion vector, described device for detecting motion vector be configured to detect representative from the corresponding position of described object block to the vector of the offset of the corresponding point of twocouese minimum value, and from described detected vector calculating kinematical vector, described twocouese minimum value be considered to based on as with the corresponding point of described horizontal direction minimum value by the detected point of described horizontal direction smallest point checkout gear and as with the corresponding point of described vertical direction minimum value by the level and the vertical direction minimum value of the detected point of described vertical direction smallest point checkout gear.
24. according to the described image processing apparatus of claim 4, wherein, described relevance values reduces inversely with the correlation of being represented by described relevance values, and described motion vector calculation device comprises:
Horizontal direction curve generating apparatus, described horizontal direction curve generating apparatus is configured to based on the minimum value in the described relevance values of storing in reduction relevance values table and is stored in the approaching relevance values of horizontal direction of described relevance values table as the approaching element of the element with being used for storing described minimum value that comprises in the horizontal direction, creates the approximate luminance curve of horizontal direction;
Horizontal direction smallest point checkout gear, described horizontal direction smallest point checkout gear are configured to detect the corresponding point of horizontal direction minimum value that is similar to luminance curve with described horizontal direction;
Vertical direction curve generating apparatus, described vertical direction curve generating apparatus is configured to based on the minimum value in the described relevance values of storing in reduction relevance values table and is stored in the approaching relevance values of horizontal direction of described relevance values table as the approaching element of the element with being used for storing described minimum value that comprises in vertical direction, creates the approximate luminance curve of vertical direction;
Vertical direction smallest point checkout gear, described vertical direction smallest point checkout gear are configured to detect the corresponding point of vertical direction minimum value that is similar to luminance curve with described vertical direction; And
Device for detecting motion vector, described device for detecting motion vector be configured to detect representative from the corresponding position of described object block to the vector of the offset of the corresponding point of twocouese minimum value, and from described detected vector calculating kinematical vector, described twocouese minimum value be considered to based on as with the corresponding point of described horizontal direction minimum value by the detected point of described horizontal direction smallest point checkout gear and as with the corresponding point of described vertical direction minimum value by the level and the vertical direction minimum value of the detected point of described vertical direction smallest point checkout gear.
25. image processing apparatus according to claim 23, wherein, the approximate luminance curve of described horizontal direction is the cubic curve of being created based on described minimum value and 3 approaching relevance values of described horizontal direction by described horizontal direction curve generating apparatus, and the approximate luminance curve of the described vertical direction cubic curve that to be described vertical direction curve generating apparatus create based on described minimum value and 3 approaching relevance values of described vertical direction.
26. image processing apparatus according to claim 24, wherein, the approximate luminance curve of described horizontal direction is the cubic curve of being created based on described minimum value and 3 approaching relevance values of described horizontal direction by described horizontal direction curve generating apparatus, and the approximate luminance curve of the described vertical direction cubic curve that to be described vertical direction curve generating apparatus create based on described minimum value and 3 approaching relevance values of described vertical direction.
27. one kind is used for by carrying out the image processing method of following steps identification motion vector:
A plurality of reference blocks are set as such reference block in the hunting zone that is provided with on reference picture, in the described reference block each has the size that equates with object block, and the object block that described object block conduct has the pre-sizing that covers a plurality of pixels is set at the pre-position on the target picture;
Search for specific reference block, described specific reference block has the strongest correlation in the correlation between described a plurality of reference blocks and described object block; And
Based on described specific reference block from described target picture on the skew of corresponding position, described precalculated position of described object block, detect described motion vector,
Described image processing method is further comprising the steps of:
Pixel value and the pixel value that is positioned at a plurality of pixels at the respective point place on the described object block based on a plurality of pixels that are positioned at the some place on each described reference block, calculate relevance values, each relevance values is represented the correlation between in the described reference block and the described object block;
Generate through reduction relevance values table by following operation:
To be stored in by the relevance values former state that described relevance values calculation element calculates at reference vector and be included in the described table element in reduction relevance values table, as with the table element that is associated through the reduction reference vector that obtains as reducing the result of described reference vector with the predetermined reduction factor, described reference vector representative reference block on the described reference picture from described target picture on the skew of corresponding position, described precalculated position of described object block; Perhaps
By dividing by described relevance values calculation element at being reduced to produce the described relevance values that calculates through the described reference vector of reduction reference vector, find a plurality of component relevance values, in described a plurality of component relevance values each and position are associated through in the reduction reference vector specific one of reduction reference vector a plurality of approaching near described, and each accumulation in the described component relevance values be stored in be included in the described table element in reduction relevance values table, as with the described specific approaching table element that is associated through the reduction reference vector; And
Based on being stored in described described component relevance values in reduction relevance values table, find the motion vector between described target picture and the described reference picture.
28. image processing method according to claim 27, wherein, described generation is further comprising the steps of through the step of reduction relevance values table:
If the location is near described a plurality of approaching through the reduction reference vector through the reduction reference vector, then by dividing by described relevance values calculation element at being reduced to produce the described relevance values that calculates through the described reference vector of reduction reference vector, find a plurality of component relevance values, in the described component relevance values each and position near described through reduction reference vector a plurality of approaching in the reduction reference vector specific one be associated, and in the described component relevance values each is stored in is included in the described table element in reduction relevance values table, as with the described specific approaching table element that is associated through the reduction reference vector, described storage operation is usually carried out by adding described component relevance values to described list cell; And
Find final component relevance values, described final component relevance values representative by will be added to by the new component relevance values that described component relevance values calculation element newly calculates be stored in as the component relevance values that had before calculated by described component relevance values calculation element the accumulation that obtains on the described previous component relevance values in the table element of reduction relevance values table with.
29. one kind is used for by carrying out the image processing method of following steps identification motion vector:
A plurality of reference blocks are set as such reference block in the hunting zone that is provided with on reference picture, in the described reference block each has the size that equates with object block, and the object block that described object block conduct has the pre-sizing that covers a plurality of pixels is set at the pre-position on the target picture;
Search for specific reference block, described specific reference block has the strongest correlation in the correlation between described a plurality of reference blocks and described object block; And
Based on described specific reference block from described target picture on the skew of corresponding position, described precalculated position of described object block, detect described motion vector,
Described image processing method is further comprising the steps of:
Pixel value and the pixel value that is arranged in a plurality of each pixel of pixel at the respective point place on the described object block at a plurality of each pixel of pixel that are arranged in the some place on each described reference block calculate relevance values;
Generate through reduction relevance values table by following operation:
To be stored in by the relevance values former state that described relevance values calculation element obtains at the calculated for pixel values of the pixel on the reference vector reference block pointed and be included in the described table element in reduction relevance values table, as with the table element that is associated through the reduction reference vector that obtains as reducing the result of described reference vector with the predetermined reduction factor, described reference vector representative reference block on the described reference picture from described target picture on the skew of corresponding position, described precalculated position of described object block; Perhaps
By dividing by described relevance values calculation element at the relevance values that is obtained with the calculated for pixel values that produces described pixel on the described reference vector of reduction reference vector reference block pointed by reduction, find a plurality of component relevance values, in the described component relevance values each and position are associated through in the reduction reference vector specific one of reduction reference vector a plurality of approaching near described, and each accumulation in the described component relevance values be stored in be included in the described table element in reduction relevance values table, as with the described specific approaching table element that is associated through the reduction reference vector; And
Based on being stored in described described component relevance values in reduction relevance values table, find the motion vector between described target picture and the described reference picture.
30. image processing method according to claim 29, wherein, described generation is further comprising the steps of through the step of reduction relevance values table:
If the location is near described a plurality of approaching through the reduction reference vector through the reduction reference vector, then by dividing by described relevance values calculation element at the relevance values that is obtained with the calculated for pixel values that produces described pixel on the described reference vector of reduction reference vector reference block pointed by reduction, find a plurality of component relevance values, in the described component relevance values each and position are associated through in the reduction reference vector specific one of reduction reference vector a plurality of approaching near described, and in the described component relevance values each is stored in is included in the described table element in reduction relevance values table, as with the described specific approaching table element that is associated through the reduction reference vector, described storage operation is usually carried out by adding described component relevance values to described list cell; And
Find final component relevance values, described final component relevance values representative by will be added to by the new component relevance values that described component relevance values calculation element newly calculates be stored in as the component relevance values that had before calculated by described component relevance values calculation element the accumulation that obtains on the described previous component relevance values in the table element of reduction relevance values table with.
31. image processing method according to claim 27, wherein:
Based on the motion vector that finds in described motion vector computation step, the described hunting zone on the described reference picture is by constriction, and the described predetermined reduction factor is reduced; And
Described by the region of search of constriction and the described reduced reduction factor by using, described relevance values calculates, described table generates and the step of described motion vector computation is repeated.
32. image processing method according to claim 29, wherein:
Based on the motion vector that finds in described motion vector computation step, the described hunting zone on the described reference picture is by constriction, and the described predetermined reduction factor is reduced; And
Described by the region of search of constriction and the described reduced reduction factor by using, described relevance values calculates, described table generates and the step of described motion vector computation is repeated.
33. one kind is used for by carrying out the image processing apparatus of following steps identification motion vector:
A plurality of reference blocks are set as such reference block in the hunting zone that is provided with on reference picture, in the described reference block each has the size that equates with object block, and the object block that described object block conduct has the pre-sizing that covers a plurality of pixels is set at the pre-position on the target picture;
Search for specific reference block, described specific reference block has the strongest correlation in the correlation between described a plurality of reference blocks and described object block; And
Based on described specific reference block from described target picture on the skew of corresponding position, described precalculated position of described object block, detect described motion vector,
Described image processing apparatus comprises:
The relevance values calculating section, described relevance values calculating section is configured to the pixel value and the pixel value that is positioned at a plurality of pixels at the respective point place on the described object block based on a plurality of pixels that are positioned at the some place on each described reference block, calculate relevance values, each relevance values is represented the correlation between in the described reference block and the described object block;
Table generates part, and described table generates part and is configured to generate through reduction relevance values table by following operation:
To be stored in by the relevance values former state that described relevance values calculating section calculates at reference vector and be included in the described table element in reduction relevance values table, as with the table element that is associated through the reduction reference vector that obtains as reducing the result of described reference vector with the predetermined reduction factor, described reference vector representative reference block on the described reference picture from described target picture on the skew of corresponding position, described precalculated position of described object block; Perhaps
By dividing by described relevance values calculating section at being reduced to produce the described relevance values that calculates through the described reference vector of reduction reference vector, find a plurality of component relevance values, in described a plurality of component relevance values each and position are associated through in the reduction reference vector specific one of reduction reference vector a plurality of approaching near described, and each accumulation in the described component relevance values be stored in be included in the described table element in reduction relevance values table, as with the described specific approaching table element that is associated through the reduction reference vector; And
Motion vector computation part, described motion vector computation partly are configured to find the motion vector between described target picture and the described reference picture based on being stored in described described component relevance values in reduction relevance values table.
34. one kind is used for by carrying out the image processing apparatus of following steps identification motion vector:
A plurality of reference blocks are set as such reference block in the hunting zone that is provided with on reference picture, in the described reference block each has the size that equates with object block, and the object block that described object block conduct has the pre-sizing that covers a plurality of pixels is set at the pre-position on the target picture;
Search for specific reference block, described specific reference block has the strongest correlation in the correlation between described a plurality of reference blocks and described object block; And
Based on described specific reference block from described target picture on the skew of corresponding position, described precalculated position of described object block, detect described motion vector,
Described image processing apparatus comprises:
The relevance values calculating section, described relevance values calculating section is configured to the pixel value and the pixel value that is arranged in a plurality of each pixel of pixel at the respective point place on the described object block at a plurality of each pixel of pixel that are arranged in the some place on each described reference block, calculates relevance values;
Table generates part, and described table generates part and is configured to generate through reduction relevance values table by following operation:
To be stored in by the relevance values former state that described relevance values calculating section obtains at the calculated for pixel values of the pixel on the reference vector reference block pointed and be included in the described table element in reduction relevance values table, as with the table element that is associated through the reduction reference vector that obtains as reducing the result of described reference vector with the predetermined reduction factor, described reference vector representative reference block on the described reference picture from described target picture on the skew of corresponding position, described precalculated position of described object block; Perhaps
By dividing by described relevance values calculating section at the relevance values that is obtained with the calculated for pixel values that produces described pixel on the described reference vector of reduction reference vector reference block pointed by reduction, find a plurality of component relevance values, in the described component relevance values each and position are associated through in the reduction reference vector specific one of reduction reference vector a plurality of approaching near described, and each accumulation in the described component relevance values be stored in be included in the described table element in reduction relevance values table, as with the described specific approaching table element that is associated through the reduction reference vector; And
Motion vector computation part, described motion vector computation partly are configured to find the motion vector between described target picture and the described reference picture based on being stored in described described component relevance values in reduction relevance values table.
CNB2006101609440A 2005-12-01 2006-12-01 Image processing apparatus and image processing method Expired - Fee Related CN100559843C (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2005347437 2005-12-01
JP2005347437 2005-12-01
JP2006195047 2006-07-18

Publications (2)

Publication Number Publication Date
CN1976399A true CN1976399A (en) 2007-06-06
CN100559843C CN100559843C (en) 2009-11-11

Family

ID=38126169

Family Applications (1)

Application Number Title Priority Date Filing Date
CNB2006101609440A Expired - Fee Related CN100559843C (en) 2005-12-01 2006-12-01 Image processing apparatus and image processing method

Country Status (1)

Country Link
CN (1) CN100559843C (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102047290A (en) * 2008-05-29 2011-05-04 奥林巴斯株式会社 Image processing device, image processing program, image processing method, and electronic device
CN102685371A (en) * 2012-05-22 2012-09-19 大连理工大学 Digital video image stabilization method based on multi-resolution block matching and PI (Portion Integration) control
CN101420613B (en) * 2007-10-22 2012-10-10 索尼株式会社 Image processing device and image processing method
WO2014084022A1 (en) * 2012-11-30 2014-06-05 富士フイルム株式会社 Image processing device and method, recording medium, program, and imaging device

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101420613B (en) * 2007-10-22 2012-10-10 索尼株式会社 Image processing device and image processing method
CN102047290A (en) * 2008-05-29 2011-05-04 奥林巴斯株式会社 Image processing device, image processing program, image processing method, and electronic device
CN102047290B (en) * 2008-05-29 2013-06-26 奥林巴斯株式会社 Image processing device, image processing program, image processing method, and electronic device
CN102685371A (en) * 2012-05-22 2012-09-19 大连理工大学 Digital video image stabilization method based on multi-resolution block matching and PI (Portion Integration) control
CN102685371B (en) * 2012-05-22 2015-04-08 大连理工大学 Digital video image stabilization method based on multi-resolution block matching and PI (Portion Integration) control
WO2014084022A1 (en) * 2012-11-30 2014-06-05 富士フイルム株式会社 Image processing device and method, recording medium, program, and imaging device

Also Published As

Publication number Publication date
CN100559843C (en) 2009-11-11

Similar Documents

Publication Publication Date Title
CN101079965A (en) Imager-created image signal-distortion compensation method, imager-created image signal-distortion compensation apparatus, image taking method and image taking apparatus
CN1175670C (en) Picture signal converting method and apparatus and picture providing system
CN1701614A (en) Image processing method and device, and program
CN100338495C (en) Zoom lens systems
CN100346636C (en) Image processing apparatus, image processing system, imaging apparatus and image processing method
CN1268135C (en) Method, device and recording medium for encoding, and method, device and recording medium for decoding
CN1248162C (en) Image processing apparatus and method and image pickup apparatus
CN1910907A (en) Control method, control device and control program for photographic device
CN1272286A (en) Block noise detector and block noise eliminator
CN1645241A (en) Imaging apparatus, method and device for processing images
CN1090455A (en) The coding/decoding method of dynamic image, device and storage medium thereof
CN1867940A (en) Imaging apparatus and image processing method therefor
CN1754384A (en) Image processing device and method, learning device and method, recording medium, and program
CN1488225A (en) Image processing method, image processing program, and image processor
CN1596547A (en) Moving picture coding apparatus, moving picture decoding apparatus, moving picture coding method, moving picture decoding method, program, and computer-readable recording medium containing the program
CN1106988A (en) Methods and apparatus for preventing rounding errors when transform coefficients representing a motion picture signal are inversely transformed
CN1706001A (en) Memory device, motion vector detection device, and detection method
CN1922630A (en) Image processing device, image processing system, image processing method, image processing program, and integrated circuit device
CN1874409A (en) Image signal processing apparatus, camera system and image signal processing method
CN1200571C (en) Orthogonal transformation, inverse orthogonal transformation method and device, and encoding and decoding method and device
CN1957616A (en) Moving picture encoding method and moving picture decoding method
CN1977205A (en) Zoom lens system, imaging device, and camera
CN101047822A (en) Thumbnail generating apparatus and image shooting apparatus
CN1582571A (en) Electronic camera
CN1799064A (en) Image processing device

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20091111

Termination date: 20211201

CF01 Termination of patent right due to non-payment of annual fee