CN100541535C - Image processing equipment and method - Google Patents

Image processing equipment and method Download PDF

Info

Publication number
CN100541535C
CN100541535C CNB2005800020184A CN200580002018A CN100541535C CN 100541535 C CN100541535 C CN 100541535C CN B2005800020184 A CNB2005800020184 A CN B2005800020184A CN 200580002018 A CN200580002018 A CN 200580002018A CN 100541535 C CN100541535 C CN 100541535C
Authority
CN
China
Prior art keywords
image
unit
point
value
pixel
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
CNB2005800020184A
Other languages
Chinese (zh)
Other versions
CN1910617A (en
Inventor
近藤哲二郎
渡边勉
大月知之
向井仁志
山口信行
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Corp
Original Assignee
Sony Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Corp filed Critical Sony Corp
Publication of CN1910617A publication Critical patent/CN1910617A/en
Application granted granted Critical
Publication of CN100541535C publication Critical patent/CN100541535C/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Abstract

The present invention relates to a kind of image processing equipment and method, recording medium and program that is used to provide the reliable tracking of trace point.When in frame n-1 as the right eye (502) of the people's of trace point face (504) when tracked, and when trace point (501) occurred in frame n, trace point (501) continued tracked.When in frame n+1, disappearing owing to the rotation of the object's face that will follow the tracks of (504) as the right eye (502) of trace point (501), trace point is transferred to left eye (503), and it is a difference that comprises the object's face (504) of right eye (502) with conduct.The present invention can be applied to security camera system.

Description

Image processing equipment and method
Technical field
The present invention relates to image processing equipment and method, recording medium and program, specifically, relate to image processing equipment and method, recording medium and the program of the required point in the moving image that can follow the tracks of time to time change reliably.
Background technology
Many methods of the required point in the automatic tracing moved image have been proposed.
For example, patent document 1 has proposed a kind of technology, wherein uses and follows the tracks of corresponding to the relevant motion vector of piece of wanting tracked object.
Patent document 2 has proposed a kind of technology, wherein estimates and wants the tracked relevant zone of object, and follow the tracks of this zone according to the estimated result of the motion in this zone.
[patent document 1] Japanese Unexamined Patent Application publication number 6-143235.
[patent document 2] Japanese Unexamined Patent Application publication number 5-304065.
But, in the technology that patent document 1 discloses, only use a motion vector to follow the tracks of.Thereby, enough robust performances can not be provided.In addition, for example owing to the image that comprises the object of wanting tracked rotates this object is disappeared from user's the ken, and when trace point occurs once more subsequently, can not follow the tracks of trace point again, this is a problem.
In patent document 2 described technology, utilize a zone.Thereby, increased robust performance.But, when in order to increase robustness and to make this zone too big, and for example tracked and when being trucked up when the image of children's face of catching with household video camera, it is tracked sometimes and trucked up to have the children's torso of the area bigger than face.
Thereby, in these two kinds of technology,,, realize that then robust tracking is difficult if perhaps for example temporarily disappear owing to scene change causes wanting tracked object if blocking (promptly if tracked object temporarily hidden by another object) has taken place.
Summary of the invention
The problem to be solved in the present invention
Thereby, the objective of the invention is to, even when being rotated, taking place blocking or occurrence scene at object and changing, also can provide the reliable tracking of trace point.
The means of dealing with problems
According to the present invention, a kind of image processing equipment comprises: position estimation device, be used for second the position that estimated statement is shown in the trace point of the image of next processing unit on the time, described second first point corresponding to the trace point in the image that is illustrated in last previous processing unit of time; Generation device when described position can not be estimated, is used to produce the point as the estimation of first candidate point at second; Determine device, be used in the time can estimating determining second point in next processing unit according to the estimated result of position estimation device in second position of next processing unit; And selecting arrangement, be used for when second position can not be estimated, selecting first point from the point of estimating.
Described processing unit can be a frame.
The precision that described position estimation device can also calculating location be estimated, and if the precision of calculating greater than reference value, then position estimation device determines that second position is estimable.
If second position in next processing unit is imponderable, then position estimation device is according to the position of second of first point estimation of being selected by selecting arrangement.
If second position is estimable, then position estimation device thinks that second position is the first new point, and can estimate the position of the trace point in the image of next processing unit.
Generation device can comprise regional estimation unit, be used for estimating that more at least group is at previous processing unit or than the previous processing unit target area of previous element more, described group belongs to and comprises first object, and the estimation point generation device, be used for producing the point of estimating according to the target area.
The zone estimation unit can find at least by prediction and as the target area position overlapped of wanting estimative object, can determine regional estimation range at the future position of the trace point that comprises the processing unit that is used for the estimating target zone, can in the regional estimation range of determining, sampling spot be set, and can estimate that by having same movement in the sampling spot and having the zone that maximum sized one group of sampling spot constitutes be the target area.
The shape of zone estimation range can be fixed.
The shape of zone estimation range can be variable.
The zone estimation unit can estimate than the previous processing unit target area in the processing unit of front more, and generation device can be created in than previous processing unit more the point in the target area of the estimation in the processing unit of front as the point of estimating.
The zone estimation unit can be estimated the target area in previous processing unit, and generation device can produce the point of formation target area as the point of estimating.
The zone estimation unit can be estimated adjacent with first and have with the point of first similar pixel value and the point adjacent with first adjacent point is the target area.
The zone estimation unit can have preliminary dimension and more comprise in the processing unit of front than previous processing unit in first the zone and extract sampling spot, and regional estimation unit can estimate to comprise regional mobile phase that passing through in the previous processing unit will have same movement and have a maximum sized sampling spot with the zone of the point that obtains of amount of movement be the target area.
Image processing equipment can also comprise the template generation device, be used to produce template, and correlation calculations device, be used for when determining at second, calculate than the processing unit of the piece correlativity between the piece of the presumptive area of the piece of the presumptive area in the next processing unit of expression in the processing unit of one or more processing unit of front and representation template more according to the point of estimating.When determining that according to the correlativity of calculating by the correlation calculations device correlativity is high, can determine that device detects trace point by using at least.
The template generation device can determine that the presumptive area around the trace point is a template.
The template generation device can produce template according to the target area.
When determining that according to the correlativity of calculating by the correlation calculations device correlativity is high, according to than the piece of the presumptive area of expression in the next the processing unit more piece of the presumptive area of representation template and the relation between the trace point and according to position in the processing unit of one or more processing unit of front, can determine second point with the piece that is defined as high correlativity.
The template generation device can determine that the zone and the presumptive area around sampling spot that are made of the sampling spot in the target area are template.
The correlation calculations device can by calculate in next processing unit piece and than the processing unit of this piece more the error between the piece of the template in the processing unit of one or more processing unit of front determine correlativity.
Image processing equipment can also comprise the pick-up unit that is used for determining scene change.When position estimation device and selecting arrangement can not select at second in the middle of the point of estimating, position estimation device and selecting arrangement stopped their processing according to predetermined condition, and changed this condition according to the appearance of scene change.
Determine that device can also comprise the estimated value calculation element, be used to calculate the estimated value of the correlativity between expression pixels of interest and the respective pixel, described pixels of interest is represented at least one pixel of first in the last previous processing unit of the time of being included in, described respective pixel is illustrated at least one pixel in the last next processing unit of time, and be determined according to the motion vector of pixels of interest, the variable value calculation element, be used to calculate the variable value of expression with respect to the change of the pixel value of pixels of interest, and the accuracy computation device, be used for the precision of calculating kinematical vector.
The quantity of pixels of interest can equal the quantity of the pixel of correspondence.
Described variable value can be the change that is used to indicate along the direction in space pixel value.
Described variable value can be indicated one of degree of scatter and dynamic range.
Described processing unit can be one of frame and field.
The accuracy computation device can be according to for the precision of variable value by the normalized value calculating kinematical vector of estimated value.
When variable value during greater than predetermined threshold, the accuracy computation device can determine that be the precision of motion vector for variable value by the normalized value of estimated value, and when variable value during less than predetermined threshold, the accuracy computation device can determine to indicate the low fixed value of precision of motion vector.
The estimated value calculation element can reckoner be shown in the pixel in the piece that comprises pixels of interest and comprise absolute difference between the pixel in the piece of corresponding pixel and estimated value.
The variable value calculation element can calculate variable value, this variable value represent by remove with the quantity of neighbor absolute difference and the value that obtains between pixels of interest and the neighbor adjacent in comprising the piece of pixels of interest with pixels of interest with.
The accuracy computation device can comprise comparison means, be used for comparison variable value and first reference value, the difference calculation element, be used to calculate second reference value and for variable value by the difference between the normalized value of estimated value, and output unit, the precision that is used for the difference calculating kinematical vector that calculates according to the comparative result of comparison means with by the difference calculation element, and the precision of output movement vector.
Image processing equipment also comprises device for detecting motion vector, be used for detecting motion vector from input picture, and motion vector offered the estimated value calculation element, motion compensator unit, be used for the motion vector motion compensation input picture that detects according to device for detecting motion vector, selecting arrangement is used for according to the image of the accuracy selection passive movement compensation system motion compensation of motion vector and does not have one of image of passive movement compensation, and code device, be used to the image of encoding and selecting by selecting arrangement.
Image processing equipment can also comprise the frequency distribution calculation element, be used to calculate frequency distribution with the weighting of motion vector precision, and the maximal value pick-up unit, be used to detect maximal value by the frequency distribution of frequency distribution calculation element calculating, and according to the maximal value detection background motion that detects.
Image processing equipment can also comprise average computing device, be used for calculating mean value in the motion vector precision of processing unit, and definite device is used for mean value and reference value that comparison is calculated by average computing device, and determines the appearance of scene change according to comparative result.
Average computing device can calculate one on average to a processing unit.
Image processing equipment can also comprise first point detection device, be used for detecting first point of the object that moves at image, the correcting area setting device, be used for being arranged on the correcting area that has preliminary dimension around the object of image according to estimated result, means for correcting, be used for the image in the correcting area of correcting image, and display control unit, be used for controlling the image that comprises by the image of the correcting area of correction and show.
Means for correcting can correcting image fuzzy.
Means for correcting can comprise conveyer, be used for transmitting the control signal of the image that is used to discern correcting area and be used for the parameter of the fuzzy grade of indicating image, the feature detection device, be used to detect feature according to the image in the correcting area of control signal identification, and the condition code of the feature of output expression detection, memory storage, be used for the parameter of the image blurring grade of storage representation and corresponding to coefficient by the condition code of feature detection device output, readout device, be used for reading described parameter and corresponding to coefficient by the condition code of feature detection device output from memory storage, the inner product calculation element, be used for inner product according to the pixel value of the coefficient calculations input picture of reading by readout device, and the selection output unit, be used to select the result of calculation of inner product calculation element and input picture pixel value export selected one in the lump.Image in correcting area can be corrected, thereby can remove the fuzzy of image.
First point detection device can comprise first extraction element, be used for to carry out pixel a plurality of pixels on every side that inner product is calculated in the predetermined first area extraction of input picture, second extraction element, be used for along a plurality of vertical and horizontal directions with a plurality of pixels of each zone extraction of a plurality of second areas of first area adjacency, piece difference calculation element, be used for by the absolute difference between the respective pixel value of calculating the pixel value that extracts by first extraction element and extracting by second extraction element with calculate a plurality of differences, and difference determines device, is used for determining that whether the piece difference is greater than predetermined threshold.
Described parameter can be the pixel of expression blurred picture and the parameter of the Gaussian function in the model tormulation formula of the relation between the pixel of blurred picture not.
By the coefficient of memory device stores can be the coefficient that the inverse matrix by the computation model expression formula obtains.
Select output unit can comprise first extraction element, be used to extract and carry out a plurality of pixels that inner product is calculated by the inner product calculation element, disperse calculation element, be used to calculate the dispersion degree of expression by the dispersion grade of a plurality of pixels of first extraction element extraction, and disperse to determine device, be used for whether determining by the dispersion degree that disperses calculation element to calculate greater than predetermined threshold.
Select output unit can also comprise the pixel selection device, be used for according to disperseing the definite result who determines device to select one of the result of calculation of inner product calculation element and pixel value of input picture output valve as pixel.
According to the present invention, a kind of image processing method comprises: estimating step, be used for second the position that estimated statement is shown in the trace point of the image of next processing unit on the time, described second first point corresponding to the trace point in the image that is illustrated in last previous processing unit of time; Produce step, when described position can not be estimated, be used to produce point at second as the estimation of first candidate point; Determining step is used in the time can estimating in second position of next processing unit determining second point in next processing unit according to the estimated result of location estimation step; And the selection step, be used for when second position can not be estimated, in the middle of the point of estimating, selecting first point.
Determining step can also comprise the estimated value calculation procedure, be used to calculate the estimated value of the correlativity between expression pixels of interest and the respective pixel, described pixels of interest is represented at least one pixel of first in the last previous processing unit of the time of being included in, described respective pixel is illustrated at least one pixel in the last next processing unit of time, and be determined according to the motion vector of pixels of interest, the variable value calculation procedure, be used to calculate the variable value of expression with respect to the change of the pixel value of pixels of interest, and the accuracy computation step, be used for the precision of calculating kinematical vector.
Image processing method of the present invention can also comprise that detect step at first, be used for detecting first point of the object that moves at image, correcting area is provided with step, be used for being arranged on the correcting area that has preliminary dimension around the object of image according to estimated result, aligning step, be used for the interior image of correcting area of correcting image, and show controlled step, be used for controlling the demonstration of the image of the image that comprises the correcting area of proofreading and correct by aligning step.
According to the present invention, a kind of recording medium of storage computation machine readable program, described computer-readable program comprises estimating step, be used for second the position that estimated statement is shown in the trace point of the image of next processing unit on the time, described second first point corresponding to the trace point in the image that is illustrated in last previous processing unit of time; Produce step, when described position can not be estimated, be used to produce point at second as the estimation of first candidate point; Determining step is used in the time can estimating in second position of next processing unit determining second point in next processing unit according to the estimated result of location estimation step; And the selection step, be used for when second position can not be estimated, selecting first point from the point of estimating.
According to the present invention, a kind of program that comprises program code, described program code is carried out computing machine: estimating step, be used for second the position that estimated statement is shown in the trace point of the image of next processing unit on the time, described second first point corresponding to the trace point in the image that is illustrated in last previous processing unit of time; Produce step, when described position can not be estimated, be used to produce estimative point at second as first candidate point; Determining step is used in the time can estimating in second position of next processing unit determining second point in next processing unit according to the estimated result of location estimation step; And the selection step, be used for when second position can not be estimated, selecting first point from the point of estimating.
According to the present invention,, then determine second point in processing unit subsequently according to definite result of this position if second position is estimable in processing unit subsequently.If second position in processing unit subsequently is imponderable, then from the point of the estimation that produces, select first point.
Advantage
According to the present invention, can be provided in the tracking of the trace point in the image.Particularly, can improve the robust performance of tracking.As a result, even when the rotation owing to the object that will follow the tracks of temporarily disappears trace point, perhaps even when blocking or scene change take place, also can follow the tracks of trace point reliably.
Description of drawings
Fig. 1 is the calcspar according to the exemplary configuration of object tracking equipment of the present invention;
Fig. 2 is the process flow diagram that is used to illustrate the tracking processing of being undertaken by object tracking equipment shown in Figure 1;
Fig. 3 represents the tracking processing when the object that will follow the tracks of rotates;
Fig. 4 represents the tracking processing when blocking takes place;
Fig. 5 represents the tracking processing when scene changes;
Fig. 6 is the process flow diagram that is used to illustrate the normal process of carrying out at step S1 shown in Figure 2;
Fig. 7 is the process flow diagram that is used to illustrate the initialization process of the normal process of carrying out at step S21 shown in Figure 6;
Fig. 8 is that the figure that candidate extracts processing is shifted in expression;
Fig. 9 is the calcspar that the exemplary configuration of relevant treatment unit is estimated in the zone;
Figure 10 is the process flow diagram that is illustrated in the zone estimation relevant treatment of step S26 shown in Figure 6;
Figure 11 is the process flow diagram that is illustrated in the zone estimation processing of step S61 shown in Figure 10;
Figure 12 A is the figure that is illustrated in the processing that is used for definite sampling spot of step S81 shown in Figure 11;
Figure 12 B is the figure that is illustrated in the processing that is used for definite sampling spot of step S81 shown in Figure 11;
Figure 13 A is the figure that is illustrated in the processing that is used for definite sampling spot of step S81 shown in Figure 11;
Figure 13 B is the figure that is illustrated in the processing that is used for definite sampling spot of step S81 shown in Figure 11;
Figure 14 A is the figure that is illustrated in the processing that is used for definite sampling spot of step S81 shown in Figure 11;
Figure 14 B is the figure that is illustrated in the processing that is used for definite sampling spot of step S81 shown in Figure 11;
Figure 15 is the figure that is illustrated in the processing that is used for definite sampling spot of step S81 shown in Figure 11;
Figure 16 is illustrated in the process flow diagram that step S86 shown in Figure 11 is used to upgrade the processing of regional estimation range;
Figure 17 A is the figure that expression is used to upgrade the processing of regional estimation range;
Figure 17 B is the figure that expression is used to upgrade the processing of regional estimation range;
Figure 17 C is the figure that expression is used to upgrade the processing of regional estimation range;
Figure 18 A is the figure that expression is used to upgrade the processing of regional estimation range;
Figure 18 B is the figure that expression is used to upgrade the processing of regional estimation range;
Figure 18 C is the figure that expression is used to upgrade the processing of regional estimation range;
Figure 19 A is the figure that expression is used to upgrade the processing of regional estimation range;
Figure 19 B is the figure that expression is used to upgrade the processing of regional estimation range;
Figure 19 C is the figure that expression is used to upgrade the processing of regional estimation range;
Figure 20 A is the figure that expression is used to upgrade the processing of regional estimation range;
Figure 20 B is the figure that expression is used to upgrade the processing of regional estimation range;
Figure 20 C is the figure that expression is used to upgrade the processing of regional estimation range;
Figure 21 is illustrated in the process flow diagram of another example that step S86 shown in Figure 11 is used to upgrade the processing of regional estimation range;
Figure 22 A is the figure that expression is used to upgrade the processing of regional estimation range;
Figure 22 B is the figure that expression is used to upgrade the processing of regional estimation range;
Figure 22 C is the figure that expression is used to upgrade the processing of regional estimation range;
Figure 22 D is the figure that expression is used to upgrade the processing of regional estimation range;
Figure 23 is that the transfer candidate that is illustrated in step S62 shown in Figure 10 extracts the process flow diagram of handling;
Figure 24 is that the template that is illustrated in step S63 shown in Figure 10 produces the process flow diagram of handling;
Figure 25 is the figure that representation template produces;
Figure 26 is the figure that representation template produces;
Figure 27 is the figure of the position relation between representation template and the trace point;
Figure 28 is the calcspar of another example of the zone shown in Figure 1 configuration of estimating the relevant treatment unit;
Figure 29 is a process flow diagram of estimating another example of processing in the zone of step S61 shown in Figure 10;
Figure 30 A is the figure of the growth in expression same color zone;
Figure 30 B is the figure of the growth in expression same color zone;
Figure 30 C is the figure of the growth in expression same color zone;
Figure 31 is the same color zone of expression trace point and the figure of regional estimated result;
Figure 32 is the process flow diagram that is illustrated in another example that the transfer candidate of the step S62 of Figure 10 extract to handle;
Figure 33 is the process flow diagram that is illustrated in the abnormality processing of step S2 shown in Figure 2;
Figure 34 is the process flow diagram of initialization process of abnormality processing that is illustrated in the step S301 of Figure 33;
Figure 35 is the figure that representation template is selected;
Figure 36 is the figure of expression region of search;
Figure 37 is the definite process flow diagram of handling of continuation that is illustrated in step S305 shown in Figure 33;
Figure 38 is the process flow diagram of another example that is illustrated in the normal process of step S1 shown in Figure 2;
Figure 39 is the process flow diagram that is illustrated in another example of the zone of step S61 shown in Figure 10 estimate handling;
Figure 40 is the process flow diagram that is illustrated in another example that the transfer candidate of step S62 shown in Figure 10 extract to handle;
Figure 41 is candidate is shifted in expression when carrying out normal process shown in Figure 6 figure;
Figure 42 is candidate is shifted in expression when carrying out normal process shown in Figure 38 figure;
Figure 43 is the calcspar of the exemplary configuration of motion estimation unit shown in Figure 1;
Figure 44 is the process flow diagram that the expression motion calculation is handled;
Figure 45 is the figure of expression frame time stream;
Figure 46 is the figure of expression frame piece;
Figure 47 is the figure of expression block matching method;
Figure 48 is the figure of expression motion vector;
Figure 49 is the process flow diagram that expression motion vector accuracy computation is handled;
Figure 50 is the figure that the method for estimated value is calculated in expression;
Figure 51 is the figure of expression activity computing;
Figure 52 is the figure that expression is used for the method for computational activity;
Figure 53 A is the figure that expression is used for the method for computing block activity;
Figure 53 B is the figure that expression is used for the method for computing block activity;
Figure 53 C is the figure that expression is used for the method for computing block activity;
Figure 53 D is the figure that expression is used for the method for computing block activity;
Figure 53 E is the figure that expression is used for the method for computing block activity;
Figure 53 F is the figure that expression is used for the method for computing block activity;
Figure 54 is the process flow diagram of expression threshold process;
Figure 55 is the figure that is illustrated in the relation between estimated value and the activity;
Figure 56 is the process flow diagram of expression normalized;
Figure 57 is the process flow diagram of expression Integral Processing;
Figure 58 is the calcspar of the exemplary configuration of background motion estimation unit;
Figure 59 is the process flow diagram that the expression background motion estimation is handled;
Figure 60 is the calcspar of the exemplary configuration of scene change detecting unit;
Figure 61 is that the expression scene change detects the process flow diagram of handling;
Figure 62 is the calcspar of the exemplary configuration of television receiver;
Figure 63 is the process flow diagram of the image display process of expression television receiver;
Figure 64 is the calcspar according to the exemplary configuration of image processing equipment of the present invention;
Figure 65 is the calcspar of the exemplary configuration of motion vector accuracy computation unit;
Figure 66 is the calcspar of the exemplary configuration of image processing equipment;
Figure 67 is the calcspar of the exemplary configuration of coding unit;
Figure 68 is the process flow diagram of the encoding process of presentation code unit;
Figure 69 is the calcspar of the exemplary configuration of DE Camera Shake ambiguity correction equipment;
Figure 70 is the calcspar of the exemplary configuration of background motion detecting unit;
Figure 71 is the process flow diagram that the DE Camera Shake ambiguity correction of expression DE Camera Shake ambiguity correction equipment is handled;
Figure 72 is the calcspar of the exemplary configuration of accumulation equipment;
Figure 73 is the calcspar of the exemplary configuration of scene change detecting unit;
Figure 74 is that the thumbnail of expression accumulation equipment produces the process flow diagram of handling;
Figure 75 is the process flow diagram that the image output of expression accumulation equipment is handled;
Figure 76 is the calcspar of the exemplary configuration of security camera system;
Figure 77 is the process flow diagram that the supervision of expression security camera system is handled;
Figure 78 is the calcspar of the another kind configuration of security camera system;
Figure 79 is the process flow diagram that the supervision of expression security camera system is handled;
Figure 80 is the calcspar according to the exemplary configuration of security camera system of the present invention;
Figure 81 is that expression monitors the process flow diagram of handling;
Figure 82 A is the figure of expression by an example of the image of security camera system demonstration;
Figure 82 B is the figure of expression by the example of security camera system demonstration;
Figure 82 C is the figure of expression by the example of security camera system demonstration;
Figure 83 is the figure of example of the motion of expression correcting area;
Figure 84 is the calcspar of the exemplary configuration of image correction unit;
Figure 85 is the calcspar of example of the control signal of image correction unit;
Figure 86 A is the figure of the fuzzy principle of presentation video;
The figure of the principle that Figure 86 B presentation video is fuzzy;
Figure 86 C is the figure of the fuzzy principle of presentation video;
Figure 87 is the figure of the fuzzy principle of presentation video;
Figure 88 is the figure of the fuzzy principle of presentation video;
Figure 89 is the figure of the fuzzy principle of presentation video;
Figure 90 is the figure of the example of expression parameter code combination;
Figure 91 is the figure of presentation video edge part;
Figure 92 is the process flow diagram that the expression ambiguity correction is handled;
Figure 93 is the process flow diagram of presentation video treatment for correcting;
Figure 94 is the process flow diagram that the presentation video feature detection is handled;
Figure 95 is the figure of the exemplary configuration of characteristics of image detecting unit;
Figure 96 A is the figure of expression by the image block of piece cut cells extraction;
Figure 96 B is the figure of expression by the image block of piece cut cells extraction;
Figure 96 C is the figure of expression by the image block of piece cut cells extraction;
Figure 96 D is the figure of expression by the image block of piece cut cells extraction;
Figure 96 E is the figure of expression by the image block of piece cut cells extraction;
Figure 97 is the process flow diagram of presentation video combined treatment;
Figure 98 is the calcspar of the exemplary configuration of presentation video assembled unit; And
Figure 99 is the figure of expression discrete calculation.
Embodiment
Below with reference to description of drawings example embodiment of the present invention.
Fig. 1 is the functional block diagram that comprises according to the object tracking equipment of image processing equipment of the present invention.Object tracking equipment 1 comprises template matches unit 11, motion estimation unit 12, scene change detecting unit 13, background motion estimation unit 14, relevant treatment unit 15 is estimated in the zone, shifts candidate storage unit 16, trace point determining unit 17, template storage unit 18, and control module 19.
Template matches unit 11 carries out at input picture and is stored in matching treatment between the template image in the template storage unit 18.Motion estimation unit 12 is estimated the motion of input picture and is estimated that to scene change detecting unit 13, background motion estimation unit 14, zone relevant treatment unit 15 and 17 outputs of trace point determining unit are by the motion vector of estimating to obtain and the precision of motion vector.The configuration of motion estimation unit 12 will be elaborated with reference to Figure 43 below.
Scene change detecting unit 13 detects scene change according to the precision that receives from motion estimation unit 12.The configuration of scene change detecting unit 13 will be elaborated with reference to Figure 50 below.
The motion that background motion estimation unit 14 comes estimated background according to the motion vector that receives from motion estimation unit 12 and precision, and estimated result is sent to the zone estimate relevant treatment unit 15.The configuration of background motion estimation unit 14 will be elaborated with reference to Figure 48 below.
The zone is estimated the motion of the background that relevant treatment unit 15 transmits according to the motion vector that transmits from motion estimation unit 12 and precision, by background motion estimation unit 14 and is come execution area to estimate to handle by the trace point information that trace point determining unit 17 transmits.The zone estimates that relevant treatment unit 15 also produces the transfer candidate also shifting the transfer candidate storage unit 16 that candidate sends the memory transfer candidate to according to input information.In addition, the zone estimates that relevant treatment unit 15 produces template according to input picture and template sent to the template storage unit 18 of storing template.The zone estimates that the configuration of relevant treatment unit 15 will be elaborated with reference to figure 9 below.
Trace point determining unit 17 is determined trace point according to the motion vector that is transmitted by motion estimation unit 12 and precision and by shifting the transfer candidate that candidate storage unit 16 transmits, and estimates the information of correlation unit 15 outputs about definite trace point to the zone.
Control module 19 is connected to from template matches unit 11 to template storage unit each unit of 18.Control module 19 makes to installing the result of (not shown) output tracking according to each unit of trace point instruction control by user's input.
The following describes the operation of object tracking equipment 1.
As shown in Figure 2, object tracking equipment 1 is carried out normal process and abnormality processing basically.That is, object tracking equipment 1 is carried out normal process at step S1.Referring to Fig. 6 normal process is described below.In this is handled, be used to follow the tracks of processing by user-defined trace point.If object tracking equipment 1 can not be transferred to new trace point to this trace point in the normal process of step S1, then handle in step S2 execute exception.Below with reference to Figure 33 abnormality processing is described.When trace point disappeared from image, abnormality processing was then carried out an operation, turned back to normal process by using the template matches operation.In abnormality processing, can not proceed (that is, processing can not be returned normal process) if determine to follow the tracks of operation, then handle being done.But,, determine that processing can return normal process, then handle and return step S1 once more if as the result who returns processing who uses template to carry out.Thereby, for each frame, alternately repeat in the normal process of step S1 with in the abnormality processing that goes on foot S2.
According to the present invention, as Fig. 3 to shown in Figure 5, by carrying out normal process and abnormality processing, even object tracking equipment 1 is owing to want the rotation of tracked object, when blocking and occurrence scene change taking place trace point temporarily being disappeared, also can follow the tracks of trace point.
That is, for example shown in Figure 3, be displayed among the frame n-1 as the people's who wants tracked object face 504.People's face 504 comprises right eye 502 and left eye 503.For example the user stipulates that right eye 502 (speaking by the book a pixel in the right eye 502) is as trace point 501.In example shown in Figure 3, in next frame n, the motion of the left of people Xiang Tu.In addition, in next frame n+1, people's face 504 clockwise rotates.As a result, the right eye 502 that can see has originally disappeared.Thereby, in known method, follow the tracks of and just can not carry out.Therefore, in the normal process of step S1, the left eye 503 of people's face is considered to and right eye 502 similar objects, and selected, makes trace point be transferred (setting) to left eye 503.Thereby, can continue to follow the tracks of.
In example shown in Figure 4, in frame n-1, ball 521 is from the left motion of people's face 504.In next frame n, ball 521 just in time covers people's face 504.In this state, the face 504 that comprises the people of the right eye 502 that is defined as trace point 501 is not shown.If this blocking takes place thereby the people's of the object that conduct will be tracked face 504 is not shown, then replace the branchpoint of trace point 501 to disappear.Thereby, be difficult to keep the tracking of trace point.But, according to the present invention, (more previous in fact, in time frame) stored as template in advance as the image of the right eye 502 of trace point 501 in frame n-1.When ball 521 moves further to the right and when occurring once more as the right eye 502 of trace point 501 in frame n+1, object tracking equipment 1 is shown once more by the right eye 502 that the abnormality processing at step S2 detects as trace point 501.Thereby right eye 502 is tracked as trace point 501 once more.
In example shown in Figure 5, in frame n-1, people's face 504 is shown, and but, in next frame n, automobile 511 covers the whole human body of the face that comprises the people.That is, in this case, change has taken place in scene.According to the present invention, even when when this scene change generation trace point 501 being disappeared from image, the right eye 502 that object tracking equipment 1 also can use template to detect as trace point 501 in the abnormality processing of step S2 when automobile 511 moves and show right eye 502 once more in frame n+1 is shown once more.Thereby right eye 502 can be tracked as trace point 501 once more.
Describe the normal process that the step S1 shown in Fig. 2 carries out in detail below with reference to process flow diagram shown in Figure 6.At step S21, trace point determining unit 17 is carried out the initialization process of normal process.Below with reference to flowchart text initialization process shown in Figure 7.In this initialization process, select regional estimation range about user-defined trace point.The scope of the point of the object that this regional estimation range is used to estimate to belong to identical with user-defined trace point (when for example trace point is eyes, the people who moves with eyes facial or as the health of rigid body).Select branchpoint in the point from regional estimation range.
At step S22, input next frame image is waited in control module 19 each unit of control.At step S23, motion estimation unit 12 is estimated the motion of trace point.That is, be the frame (next frame) that is included in the next frame of the frame (former frame) that goes on foot the user-defined trace point of S22 in time by receiving, control module 19 can obtain the image in two successive frames.Thereby, at step S23,, can estimate the motion of trace point by estimating in next frame the position of the trace point corresponding with trace point in the former frame.
Term used herein " former frame in time " refers to processing sequence (input sequence).In general, import several two field pictures according to the order of catching image.In this case, the frame of early catching is defined as former frame.But, when the frame of catching after was at first handled (input), the frame of catching after was defined as former frame.
At step S24, motion estimation unit 12 (Integral Processing unit 605 shown in Figure 43, it is illustrated below) determines whether to estimate trace point according to the result at step S23.For example by producing and the precision of the motion vector of output and predetermined threshold are compared and can be determined whether trace point can be estimated by motion estimation unit 12 (it is illustrated with reference to Figure 43 below).More particularly, if the precision of motion vector more than or equal to predetermined threshold, then trace point can be estimated.But, if the precision of motion vector, determines then that trace point can not be estimated less than predetermined threshold.That is, the possibility of the estimation is here relatively strictly determined.Even in the time in fact can estimating, if precision is low, described estimation also is confirmed as impossible.Thereby, can provide more reliable tracking to handle.
If the motion estimation result of the estimated result of the motion of trace point and near trace point point is consistent with dominant motion numerically, then can determine to estimate to be possible, otherwise estimation is impossible at step S24.
If determining the motion of trace point can estimate, promptly, if determine in the possibility that trace point correctly is set on the identical object (when right eye 502 is defined as trace point 501, correctly following the tracks of the possibility of right eye 502) highly relatively, then handle advancing to step S25.At step S25, trace point determining unit 17 is by means of motion (motion vector) the mobile tracking point of the estimation that obtains at step S23.That is, after carrying out this operation, the conduct in next frame can be determined corresponding to the trace point of the trace point in the former frame.
After the processing of carrying out step S25, carry out the zone at step S26 and estimate relevant treatment.This zone estimates that relevant treatment is elaborated with reference to Figure 10 below.Handle by carrying out this, the regional estimation range of being determined by the initialization process of normal process at step S21 is updated.In addition, in the time of for example can not showing trace point owing to the rotation of destination object, the candidate of the branchpoint that trace point will be transferred to (transfer candidate) (is promptly still keeping under the state of this trace point of tracking) being extracted (generation) in advance in this state.When in addition to the transfer of shifting candidate can not the time, follow the tracks of and temporarily stopped.But, be possible (being that trace point occurs once more) in order to confirm again secondary tracking, produce a template in advance.
After the zone of step S26 estimates that relevant treatment is finished, handle and return step S22, and repeat to go on foot S22 processing afterwards.
Promptly if the motion of user-defined trace point can be estimated, then repeat processing, thereby follow the tracks of from step S22 to step S26 for each frame.
But,, can not be estimated (estimation is impossible), that is,, then be handled advancing to step S27 if determine that for example the precision of motion vector is less than or equal to threshold value if determine the motion of trace point at step S24.At step S27, because estimated by the zone that at step S26 the transfer candidate that relevant treatment produces is stored in the transfer candidate storage unit 16, trace point determining unit 17 is selected a candidate that approaches original trace point most from stored candidate person during shifting candidate storage unit 16.At step S28, trace point determining unit 17 determines whether can select to shift candidate.If can select to shift candidate, then handle advancing to step S29, trace point is transferred to the transfer candidate that (changing into) selected at step S27 there.That is, the point of being represented by the transfer candidate is set to new trace point.After this, handle and return step S23, estimate there from the motion of the trace point that shifts the candidate selection.
At step S24, determine whether the motion of newly-installed trace point can be estimated.If estimate it is possible, then make trace point move the amount of the motion of an estimation at step S25.At step S26, carry out the zone and estimate relevant treatment.After this, handle and return step S22 once more, and repeat to go on foot S22 processing afterwards.
At step S24, determine that the motion of newly-installed trace point can not be estimated, handle and return step S27 once more.At step S27, select the transfer candidate of next the most approaching original trace point.At step S29, the transfer candidate of selection is set to new trace point.Repeat to go on foot S23 processing afterwards once more for newly-installed trace point.
If the motion of trace point can not be estimated after the transfer candidate of each preparation is set to new trace point, then determining to shift candidate at step S28 can not be selected.Thereby, finish normal process.After this, processing advances to the abnormality processing at step S2 shown in Fig. 2.
Describe the initialization operation of the normal process of the step S21 shown in Fig. 6 in detail below with reference to process flow diagram shown in Figure 7.
At step S41, control module 19 determine when pre-treatment whether be the processing of returning from abnormality processing.Be that control module 19 determines to finish to handle whether turned back to normal process once more after the abnormality processing of step S2.Because the abnormality processing of S2 is not performed as yet for first frame in the step, determine that therefore this processing is not the processing of returning from abnormality processing.Thereby, handle advancing to step S42.At step S42, it is the point that is designated as trace point that trace point determining unit 17 is provided with trace point.That is, the user is defined in a predetermined point in the input picture as the trace point of control module 19 by operation mouse or another kind of input block (not shown).According to this instruction, it is trace point to determine by user-defined point that control module 19 is controlled trace point determining units 17.In addition, can use another kind of method to determine trace point.The point that for example has maximum brightness can be confirmed as trace point.Trace point determining unit 17 estimates that to the zone relevant treatment unit 15 provides the information about the trace point of determining.
At step S43, estimation relevant treatment unit, zone 15 bases are determined regional estimation range in the position of the trace point that step S42 determines.The zone estimation range is a scope reference when the point on the entity that is comprising trace point is estimated.The zone estimation range is determined in advance, and makes to comprise that the entity of trace point occupies regional estimation range with preponderating.More particularly, regional estimation range is determined like this, makes position and size follow the entity that comprises trace point, and therefore, the part in presenting the regional estimation range of dominant motion numerically can be estimated as and belong to the entity that comprises trace point.At step S43, for example, wherein a trace point predetermined constant zone being positioned at its center is confirmed as regional estimation range as initial value.
Then, processing advances to step S22 shown in Figure 3.
On the contrary, if determine that at step S41 current processing is the processing of returning from the abnormality processing of step S2, then handles advancing to step S44.At step S44, trace point and regional estimation range are determined in the position of trace point determining unit 17 bases and the template matches in the processing of step S303 shown in Figure 33, and this is described below.For example and the point in the present frame of the trace point in template coupling be confirmed as trace point.In addition, the predetermined constant zone around this point is confirmed as regional estimation range.After this, processing advances to step S22 shown in Figure 3.
Below with reference to Fig. 8 above-mentioned processing is described.That is, at step S42 shown in Figure 7, as shown in Figure 8, if for example the eyes 502 of the people in frame n-1 are defined as trace point 501, then at step S43 comprising that the presumptive area of trace point 501 is appointed as regional estimation range 533.At step S24, determine whether a sampling spot in regional estimation range 533 can be estimated in next frame.In example shown in Figure 8, among the frame n+1 after frame n, hidden by ball 521 because comprise half zone 534, a left side of left eye 502, the motion of the trace point 501 in frame n can not be estimated in next frame n+1.Therefore, in this case, select a point from the point of regional estimation range 533 (as the face 504 of the entity that comprises right eye 502), this zone estimation range is prepared in advance to shifting candidate among the former frame n-1 in time.For example, be chosen in the left eye 503 that comprises in people's the face 504 here, or rather, a pixel in the left eye 503.The point of this selection is confirmed as trace point in frame n+1.
The zone estimates that relevant treatment unit 15 has configuration shown in Figure 9, estimates relevant treatment so that carry out the zone at step S26 shown in Figure 6.That is, the zone estimates that the regional estimation unit 41 of relevant treatment unit 15 receives motion vector and precision from motion estimation unit 12, and 14 receive background motion from the background motion estimation unit, and from the positional information of trace point determining unit 17 receptions about trace point.Shift candidate extraction unit 42 and receive motion vector and precision from motion estimation unit 12.Shift the also output of receiving area estimation unit 41 of candidate extraction unit 42.Template generation unit 43 receives the output of input picture and regional estimation unit 41.
Zone estimation unit 41 is imported the zone that estimation comprises the entity of trace point according to these, then estimated result is outputed to and shifts candidate extraction unit 42 and template generation unit 43.Shift candidate extraction unit 42 and extract the transfer candidate, follow, the transfer candidate that extracts is offered shift candidate storage unit 16 according to these inputs.Template generation unit 43 produces template according to these inputs, then, the template that produces is sent to template storage unit 18.
Figure 10 expression in detail estimates relevant processing (in the processing of step S26 shown in Figure 6) by the zone that estimation relevant treatment unit, zone 15 carries out.At step S61, estimate to handle by regional estimation unit 41 execution areas.Below with reference to its detailed operation of flowchart text shown in Figure 11.In this was handled, estimation was that the some points that belong in the image-region of the identical object of the object that belongs to trace point (entity that is synchronized with the movement with trace point) are extracted as regional estimation range (the regional estimation range 81 among Figure 17 the following describes).
At step S62, handle by shifting the 42 execution transfer candidates extractions of candidate extraction unit.Flow process below with reference to Figure 23 describes this processing in detail.By extracting the point that shifts candidate in the point of regional estimation unit 41 from the scope of being estimated to be regional estimation range.The point that extracts is stored in and shifts in the candidate storage unit 16.
At step S63, carry out template by template generation unit 43 and produce processing.Describe this processing in detail below with reference to process flow diagram shown in Figure 24.Handle the generation template by this.
Estimate to handle below with reference to the zone of flowchart text shown in Figure 11 step S61 shown in Figure 10.
At step S81, the sampling spot that regional estimation unit 41 is determined as candidate point, these points are the points that belongs to the object that comprises trace point by estimating.
For example, as shown in figure 12, sampling spot (being represented by black square) can be the pixel of the position that is spaced from each other intended pixel of along continuous straight runs and the vertical direction from fixed datum 541 beginnings.In example shown in Figure 12, the pixel in the upper left corner of every frame is defined as reference point 541 (in the drawings by symbol * expression).Sampling spot is to be spaced from each other 5 pixels and the pixel of 5 locations of pixels that are vertically spaced from each other from reference point 541 beginning along continuous straight runs.That is in this example, be sampling spot, with the pixel definition that disperses in the whole shielding.And in this example, the reference point in frame n and frame n+1 is the same point in the fixed position.
For example, as shown in figure 13, dynamically change reference point 541, thereby the reference point in frame n and datum in frame n+1 are in different positions.
In Figure 12 and example shown in Figure 13, for each frame, the distance between sampling spot is constant.But, as shown in figure 14, the distance between sampling spot can change for each frame.In example shown in Figure 14, in frame n, the distance between the sampling spot is 5 pixels, and in frame n+1, the distance between the sampling spot is 8 pixels.At this moment, the size of being estimated to belong to the zone of the object that comprises trace point can be used as reference range.More particularly, when the size of regional estimation range reduced, this distance also reduced.
In addition, as shown in figure 15, in a frame, the distance between the sampling spot can be different.At this moment, the distance between sampling spot and trace point can be used as reference range.That is, when sampling spot more approached trace point, the distance between sampling spot reduced.On the contrary, when sampling spot during further from trace point, the distance between the sampling spot increases.
Like this, determine sampling spot.Then, at step S82, regional estimation unit 41 is carried out the processing that is used for estimating in the motion of the interior sampling spot of regional estimation range (determining that at step S106, the S108 of step S43, S44 shown in Figure 7 and Figure 16 this is described below).That is, regional estimation unit 41 extracts point corresponding to sampling spot in the regional estimation range according to the motion vector that is transmitted by motion estimation unit 12 in next frame.
At step S83, regional estimation unit 41 is carried out the processing of removing some points according to the motion vector with the precision that is lower than predetermined threshold from the sampling spot of estimating at step S82.Being used to carry out this precision of handling required motion vector is provided by motion estimation unit 12.Thereby, the sampling spot in regional estimation range, only extract according to point with high-precision estimation of motion vectors.
At step S84, regional estimation unit 41 extracts full frame motion according to the estimated result of the motion in regional estimation range.Term used herein " full frame motion " refers to the motion that has maximum sized zone in having the zone of same movement.More particularly, for the motion of each sampling spot, and the proportional weighting of distance is assigned to the histogram that makes the generation motion between the sampling of sampling spot.The motion (motion vector) of extracting the maximization weighted frequency is as full frame motion.When producing histogram, for example, expression value that can the warm-up of considered pixel resolution.Motion with a pixel resolution difference can be added on the histogram.
At step S85, regional estimation unit 41 extracts sampling spot in the regional estimation range with full frame motion as regional estimated result.Wherein,, not only extract and have the sampling spot of the motion identical, and can extract and have the sampling spot that is less than or equal to predetermined threshold with the differences in motion of full frame motion with full frame motion as sampling spot with full frame motion.
At step S43, S44, S44 in the middle of the sampling spot in the regional estimation range that S106 or S108 determine, extracts (generation) at last and has the sampling spot of full frame motion as the point of being estimated to belong to the object that comprises trace point.
After this, at step S86, regional estimation unit 41 is carried out the processing that is used to upgrade regional estimation range.Handle then and advance to step S22 shown in Figure 6.
Figure 16 is illustrated in the processing of the renewal zone estimation range of step S86 shown in Figure 11 in detail.In step S101, the center of gravity of regional estimation unit 41 zonings.This zone refers to the zone (i.e. the zone that is limited by the point of being estimated to belong to the object that comprises trace point) that is limited by the sampling spot that extracts at the step S85 shown in Figure 11.That is, between motion vector (full frame motion) and this zone, has man-to-man correspondence.For example, shown in Figure 17 A, in the middle of the sampling spot of in regional estimation range 81, representing, extract the sampling spot represented by the black square as sampling spot with full frame motion at step S85 shown in Figure 11 by white square.The zone that extraction is estimated to be limited by these sampling spots is as zone 82.After this, the center of gravity 84 of zoning 82.More particularly, separate for each sampling spot according to the sampling spacing and join a weighting, and calculate sampling spot gravity as this regional center of gravity.Carry out this handle with find out in present frame should the zone the position.
At step S102, regional estimation unit 41 is according to the center of gravity of full frame motion moving area.This processing is carried out like this, makes regional estimation range 81 follow the motion of this regional location, and this zone is moved to the position of estimating in the next frame.Shown in Figure 17 B, when the trace point in present frame 83 occurred as the trace point in the next frame 93 according to the motion vector 88 of trace point 83, the motion vector 90 of full frame motion corresponded essentially to motion vector 88.Thereby, by the center of gravity 84 that moves in the present frame according to motion vector (full frame motion) 90, can obtain the point 94 in the frame identical with the frame (next frame) of trace point 93.By regional estimation range 91 being set to the scope that its center is a point 94, regional estimation range 81 can be followed the motion of the position in zone 82, thereby moves to the position of estimating in next frame.
At step S103, regional estimation unit 41 is determined the size of next regional estimation range according to regional estimated result.More particularly, the quadratic sum of the distance (distance between the black box in zone shown in Figure 17 82) between all sampling spots of being estimated in this zone is considered to the size in zone 82.The size of the regional estimation range 91 in next frame is determined like this, makes it be slightly larger than the size in zone 82.Promptly along with the increase of sampling spot quantity in zone 82, the size of regional estimation range 91 also increases.On the contrary, when the quantity of the sampling spot in zone 82 reduced, the size of regional estimation range 91 reduced.Thereby the size of regional estimation range 91 not only can be followed zone 82 increase and be dwindled, and can to avoid the full frame zone in regional estimation range 81 be the neighboring area of tracing object.
If the full frame motion of extracting at step S84 shown in Figure 11 equals background motion, then tracing object can not be distinguished by moving with background.Therefore, the processing (its details describes with reference to Figure 49 below) that is used for the estimated background motion is carried out in background motion estimation unit 14 always.At step S104, whether regional estimation unit 41 definite background motions that transmitted by background motion estimation unit 14 equal the full frame motion in step S84 extraction shown in Figure 11.If full frame motion equals background motion, then at step S105, the size of the next regional estimation range of regional estimation unit 41 restrictions makes that the size of current region estimation range is maximum.Thereby, can not be identified as tracing object to background mistakenly.Thereby the size of regional estimation range is controlled like this, makes can not be increased.
If S104 determines that full frame motion is not equal to background motion in the step, then need therefore, not skip processing in the processing of step S105 at step S105.
At step S106, regional estimation unit 41 is determined the size of next regional estimation ranges, the center of this scope be should the zone after moving center of gravity.Thereby regional estimation range is determined like this, make the center of gravity of regional estimation range equal this regional center of gravity of acquisition after moving, and the size of regional estimation range is proportional with this regional size.
In the example shown in Figure 17 B, determine the size of regional estimation range 91 according to zone 82 size, its center is the center of gravity 94 after moving according to motion vector (full frame motion) 90.
Should guarantee to have in the zone of the full frame motion of regional estimation range 91 inboards is zones (face 504 for example shown in Figure 8) of the object of wanting tracked.Therefore, at step S107, regional estimation unit 41 determines whether trace point is included in the next regional estimation range.If trace point is not included in the next regional estimation range, then carry out the processing of moving next regional estimation range at step S108 zone estimation unit 41, make next regional estimation range comprise trace point.If trace point is included in the next regional estimation range, then need be in the processing of step S108, therefore, the processing of S108 is skipped in the step.
More particularly, in this case, next regional estimation range can be moved like this, makes the displacement minimum.Perhaps, the center of gravity that next regional estimation range can be along a vector from regional estimation range moves to trace point by the distance of minimum, makes trace point be included in the next regional estimation range.
In order to keep the robust performance of tracking, can to skip is to comprise moving of zone that trace point carries out.
In the example shown in Figure 17 C, because regional estimation range 91 does not comprise trace point 93, regional estimation range 91 is moved to the position (position that comprises upper left corner trace point 93) by regional estimation range 101 indications.
Figure 17 A represents to 17C need be at the example of the mobile processing that goes on foot S108.On the contrary, represent to 18C need be at the example (that is the example when S107 determines that trace point is included in the next regional estimation range in the step) of the mobile processing of step S108 for Figure 18 A.
Shown in Figure 18 A-18C, when all sampling spots in regional estimation range 81 all are the point in zone, then do not need the mobile processing of step S108 shown in Figure 16.
Figure 17 A represents that to Figure 17 C and Figure 18 A-18C wherein regional estimation range is the example of rectangle.But, shown in Figure 19 A-19C and Figure 20 A-20C, regional estimation range can be circular.Figure 19 A-19C corresponds respectively to Figure 17 A-17C, wherein need be in the mobile processing of step S108.On the contrary, Figure 20 A-20C corresponds respectively to Figure 18 A-18C, wherein need not go on foot the mobile processing that S108 carries out.
Thereby, by carry out upgrading the processing (at step S86 shown in Figure 11) of regional estimation range shown in Figure 16, determined like this about the position and the size of the regional estimation range of next frame, make regional estimation range comprise trace point.
Upgrade the processing of regional estimation range in shown in Figure 16 being used for, the shape of regional estimation range is the rectangle or the circle of fixing.But, the shape of regional estimation range can be variable.In this example, be described in the processing that is used to upgrade regional estimation range of step S86 shown in Figure 11 below with reference to Figure 21.
At step S131, regional estimation unit 41 determines whether equal the background motion estimated by background motion estimation unit 14 in the full frame motion that step S84 shown in Figure 11 extracts.If the two does not wait, then handle and advance to step S133, go on foot regional estimation unit 41 at this and determine a zonule (that is, a point being determined a zonule) corresponding to each point of being estimated to belong to this zone (zone that constitutes by pixel) with the motion that equals full frame motion.In the example shown in Figure 22 A and the 22B, in regional estimation range 161, determine zonule 171 and 172, they are corresponding to the point in the zone of being indicated by black box.In the figure, label 171 expression is wherein corresponding to 4 examples that the zonule is overlapped of 4 points.The size of zonule can be determined like this, for example make and sampling spot between distance proportional.
At step S134, regional estimation unit 41 determines that the uniting of zonule in that step S133 determines is temporary transient regional estimation range.In the example shown in Figure 22 C, the zone 182 of the associating as regional 171 and 172 is confirmed as temporary transient regional estimation range.If after the associating that obtains the zonule, produce a plurality of discontinuous zones, then only have maximum sized zone and can be confirmed as temporary transient regional estimation range.
If S131 determines that full frame motion equals background motion in the step, then at step S132, regional estimation unit 41 determines that the current region estimation range is temporary transient regional estimation range.Its reason is, the current region estimation range is held constant because when the estimated result of background motion equals full frame motion, background can not by means of motion with to separate by tracked target area.
After step the finishing dealing with of S134 or S132, regional estimation unit 41 is determined next regional estimation range at step S135 by using full frame motion to move the temporary transient regional estimation range of determining at step S134 or S132.In the example shown in Figure 22 C, temporary transient regional estimation range 181 is moved according to the motion vector 183 of full frame motion, and is confirmed as temporary transient regional estimation range 182.
At step S136, regional estimation unit 41 determines whether trace point is included in the next regional estimation range that step S135 determines.If trace point is not included in the next regional estimation range, then handle advancing to step S137, go on foot regional estimation unit 41 at this and move next regional estimation range like this, make next regional estimation range comprise trace point.In the example shown in Figure 22 C and Figure 22 D, because regional estimation range 182 does not comprise trace point 184, regional estimation range 182 is moved like this, the feasible trace point 184 that comprises the upper left corner, and be confirmed as regional estimation range 191.
If S136 determines that trace point is included in the next regional estimation range in the step, then need be in the mobile processing of step S137, therefore, the mobile processing of S137 is skipped in the step.
The processing that is used to extract the transfer candidate of step S62 shown in Figure 10 is described below with reference to Figure 23.
At step S161, shift 42 pairs of candidate extraction units and estimated to belong to each point in the zone of full frame motion and keep the mobile result of the point that moves by the motion of estimating as shifting candidate.That is, directly do not use the point that obtains as regional estimated result.In order in next frame, to use these points, carry out the processing of extracting mobile result according to its motion estimation result.Then the transfer candidate that extracts is sent to and shift candidate storage unit 16, and be stored in the transfer candidate storage unit 16.
Below with reference to Fig. 8 this processing is described.That is in example shown in Figure 8, trace point 501 appears in frame n-1 and n.But, in frame n+1, trace point 501 quilts hide from the ball 521 of the left of this figure, and therefore, trace point 501 disappears.Thereby, in frame n+1, need transfer to trace point as a difference (for example transfer to left eye 503, or rather, near the point of right eye 502) of wanting in the tracked object's face 504.Therefore, before requiring to shift, reality in former frame, prepares the transfer candidate in advance.
More particularly, in example shown in Figure 8, can predict that in most of the cases, the estimated result of the motion from frame n to frame n+1 in regional estimation range 533 is not estimated correctly that this is to shift because require in regional estimation range 533.That is, in example shown in Figure 8, because the part of trace point and the object that comprises trace point disappears and shifts.Thereby, part 534 for regional estimation range 533 in frame n, object is hidden (part of being represented by cross hatch) in Fig. 8 in frame n+1 in this part, motion is not correctly estimated, therefore, the precision of motion is estimated as low or not low, thereby the estimated result of motion is insignificant.
In this case, because the motion estimation result that can be used for estimating in the zone reduces or is mixed with incorrect motion estimation result, thereby to have increased regional estimation be incorrect possibility.In addition, in general, be lower than zone from frame n to the frame n+1 possibility estimating in this possibility more region in front is estimated in time from frame n-1 to frame n.
Thereby, for the danger that reduces incorrect estimation with improve performance, need directly not use regional estimated result, but the regional estimated result of acquisition is used as the transfer candidate of moving target in frame n-1 (perhaps in time more the frame of front).
But, regional estimated result can directly be used.Below with reference to Figure 38 explanation processing in this case.
Figure 24 is illustrated in the detailed process that step S63 shown in Figure 10 is used to produce template.At step S181,43 pairs of template generation units are estimated to belong to each point in this zone (zone of full frame motion) and are determined a zonule.In example shown in Figure 25, determine a zonule 222 for the point 221 in this zone.
At step S182, template generation unit 43 determines that the uniting of zonule in that step S181 determines is the template zone.In example shown in Figure 25, uniting of zonule 222 is confirmed as template zone 231.
Then, at step S183, template generation unit 43 produces template by information and the image information about the template zone determined at step S182, and template is sent to template storage unit 18, this unit storing template.More particularly, the pixel data in template zone 231 is confirmed as template.
As shown in figure 26, corresponding to the zonule 241 of the point 221 in this zone greater than zonule shown in Figure 25 222.Thereby, as the template zone 251 of the associating of zonule 241 also greater than template zone 231 shown in Figure 25.
The size of zonule can and sampling spot between distance proportional.In this case, proportionality constant can be determined like this, make this size equal distance between the sampling spot square.Perhaps, proportionality constant can be determined like this, make this size be greater than or less than distance between the sampling spot square.
In addition, replace regional estimated result, can use have fixed measure and shape and its center in the zone of trace point as the template zone.
Figure 27 is illustrated in the position relation between template and the regional estimation range.Template zone 303 comprises trace point 305.The point in the upper left corner of the boundary rectangle 301 that surrounds around template zone 303 is confirmed as template fiducial point 304.The vector of the reference point 308 in 305 the vector 306 and the upper left corner from template fiducial point 304 to regional estimation range 302 is as the information about template zone 303 from template fiducial point 304 to trace point.Template is made of the pixel that comprises in the template zone 303. Vector 306 and 307 is used to return the processing of normal process when detecting the image identical with template.
In above-mentioned processing,, be confirmed as template corresponding to the scope and the pixel of present frame with to shift candidate different.But, candidate is identical with shifting, and the moving target point in next frame is used as template.
Thereby as shifting candidate, the template that is made of the pixel data that comprises trace point is produced during normal process in advance.
The zone estimation relevant treatment of step S26 shown in Fig. 6 can be estimated 15 execution of relevant treatment unit by the zone, and the zone estimates that relevant treatment unit 15 for example has configuration shown in Figure 28.
In this case, estimate relevant treatment unit 15 as zone shown in Figure 9, the zone estimates that relevant treatment unit 15 comprises regional estimation unit 41, shifts candidate extraction unit 42 and template generation unit 43.In this example, will be input to regional estimation unit 41 from trace point determining unit 17 about the information of trace point and input picture.Have only the output of regional estimation unit 41 to be imported into transfer candidate extraction unit 42.The output and the input picture of regional estimation unit 41 are input to template generation unit 43.
In this case, as processing shown in Figure 10, the estimation of S61 execution area is handled in the step, shifts the candidate extraction in step S62 execution and handles, and produce processing in step S63 execution template.Because the step S63 template of carrying out produce handle identical with processing shown in Figure 24, below only explanation estimate in the zone of step S61 to handle and extract processing at the transfer candidate of step S62.
At first be described in detail in the zone estimation processing of step S61 with reference to process flow diagram shown in Figure 29.At step S201, regional estimation unit shown in Figure 28 41 is determined sampling spots, so that estimate to belong to the zone in the image of the object that comprises trace point.This handles identical with the processing of step S81 shown in Figure 11.
But, be the frame (being included in the frame of following the tracks of the trace point after finishing) that has been determined of trace point wherein at step S201 frame to be processed.These are different with step S81 shown in Figure 11, are used for wherein determining that the frame of sampling spot is a former frame.
Then, at step S202, regional estimation unit 41 is carried out the processing along the direction in space application of low-pass filters of the image of next frame (wherein sampling spot be determined at step S201 frame).That is,, from image, remove radio-frequency component, make image smoothed by application of low-pass filters.Thereby, be convenient to handle in the generation in the same color zone of subsequently step S203.
At step S203, regional estimation unit 41 is included in sampling spot in the same color zone in the difference of pixel value and carries out the processing in zone that is used for being comprised by to start with trace point generation the same color of trace point under for the condition of the estimated result in zone less than threshold value Thimg and regulation.Be included in sampling spot in the same color zone of final generation and be used as the estimated result in zone.
More particularly, for example, shown in Figure 30 A, read pixel value with the trace point adjacent pixels along 8 directions.That is, along the top, the upper right side reads pixel value with the trace point adjacent pixels to, right, lower right, below, lower left, left and upper left side direction.Difference between the pixel value of pixel value that calculating is read and trace point 321.After this, determine that whether the difference of calculating is more than or equal to threshold value Thimg.In the example shown in Figure 30 A, along the pixel value of the pixel (promptly along top, upper right side, below, left and upper left pixel) of the direction shown in the arrow and each difference between the trace point 321 less than threshold value Thimg.On the contrary, along not by the pixel value of the pixel of the arrow indicated direction pixel of right-hand, lower right and lower left (promptly along) and the difference between the trace point 321 more than or equal to threshold value Thimg.
In this case, shown in Figure 30 B, have pixel less than the difference of threshold value Thimg (pixel of the arrow indication of origin autotracking point 321) and be used as pixel 322 and be recorded in the same color zone that comprises trace point 321.Carry out identical processing for the pixel 322 that is recorded in the same color zone.In the example shown in Figure 30 B, calculate the upper left side by the pixel value of the pixel 322 of white circle indication and and the pixel value of pixel 322 adjacent pixels (removing the pixel outside the pixel that is confirmed as the same color zone) between difference.Determine that then whether this difference is more than or equal to threshold value Thmig.In the example shown in Figure 30 B, carried out definite processing for the same color zone of the pixel of right-hand, lower right and below.Thereby, calculate along the top, upper right side, lower left, left and upper left difference.In addition, in this example, along the top, upper right side and upper left difference be less than threshold value Thmig.Shown in Figure 30 C, along the pixel of these directions as comprising that the pixel in the same color zone of trace point 321 is recorded.
Repeat this processing subsequently.Thereby, as shown in figure 31, in the middle of sampling spot, be included in the point that same color zone 331 interior points are estimated as the object that comprises trace point 321.
After zone shown in Figure 29 estimates that processing (step S61 shown in Figure 10) is done, carry out the extraction of transfer candidate by transfer candidate extraction unit 42 shown in Figure 28 at step S62 shown in Figure 10 and handle.This shifts candidate and extracts processing by shown in the process flow diagram of Figure 32.
That is,, shift candidate extraction unit 42 and determine to be estimated to be that the have a few in this zone (same color zone) is to shift candidate without change at step S231.Then, shift candidate extraction unit 42 and send transfer candidate storage unit 16, its memory transfer candidate to shifting candidate.
Estimate in the relevant treatment unit 15 in zone shown in Figure 28, the template generation is handled identical with processing shown in Figure 24, and this template produces and handles is to be carried out by template generation unit shown in Figure 28 43 at step S63 shown in Figure 10 after transfer candidate extraction processing (step S62 shown in Figure 10) shown in Figure 32 is done.Thereby, no longer it is carried out the explanation of repetition.
But, in this case, it is the template zone that the same color zone that comprises trace point can directly be determined.
Be described in detail in the abnormality processing that the above-mentioned normal process of step S1 shown in Figure 2 is done and carries out at step S2 afterwards below with reference to process flow diagram shown in Figure 33.As mentioned above, when the motion of determining trace point at step S24 shown in Figure 6 can not be estimated, and, carry out this processing when going on foot transfer candidate that S28 determines that trace point is transferred to can not be selected the time.
At step S301, control module 19 carries out the initialization process of abnormality processing.The details of this processing is by flowcharting shown in Figure 34.
At step S321, control module 19 determines that when it can not follow the tracks of trace point (when it can not estimate that the motion of trace point can not be selected transfer candidate that trace point will transfer to) determines whether the scene change generation.Whether scene change detecting unit 13 takes place according to the estimated result monitoring scene change from motion estimation unit 12 always.Control module 19 is determined at step S321 according to the testing result from scene change detecting unit 13.Detailed process below with reference to Figure 50 and 51 explanation scene change detecting units 13.
If scene change takes place, control module 19 estimates that the generation of scene change stops the tracking of trace point.Thereby at step S322, control module 19 patterns are set to scene change.On the contrary, if determine not have occurrence scene to change at step S321, then control module 19 is set to another kind of pattern in step S323 pattern.
The step S322 or S323 processing be done after, at step S324, the processing that is used to select in time template is the earliest carried out in template matches unit 11.More particularly, as shown in figure 35, for example, when frame n is changed to frame n+1 and execute exception processing, template matches unit 11 is chosen as frame n-m+1 and the template that produces, its be produce to frame n for frame n-m+1 and be stored in the middle of m the template in the template storage unit 18 in time the earliest template.
Thereby, replace being transformed into the template (template that in example shown in Figure 35, produces) of moment before the abnormality processing for frame n, and use the reason of the template of some time before conversion to be, when for example occurring to the conversion of abnormality processing owing to the blocking of wanting tracked object, most of objects moments before changing is hidden, therefore, template at this moment can not capture the enough big image of object probably.Thereby the template of the previous blink by being chosen in conversion can provide reliable tracking.
At step S325, the processing that is used for determining the template region of search is carried out in template matches unit 11.For example, the template region of search determined like this, makes before the being transformed into abnormality processing position of the trace point of moment become the center of template region of search.
That is, as shown in figure 36, suppose that the right eye 502 of the object's face 504 in frame n is defined as trace point 501.In frame n+1, cover in the face 504 that comprises trace point 501 from the ball 521 of left.In frame n+2, trace point 501 occurs again.In this case, be that the zone at center is confirmed as template region of search 312 with trace point 501 (be included in template zone 311 in).
At step S326, template matches unit 11 is reset at and is transformed into abnormality processing is 0 by the quantity of frame and the quantity of scene change afterwards.The quantity by frame and the quantity of scene change be used for determining to handle in the continuation of step S305 shown in Figure 33 (at step S361 shown in Figure 37, S363, S365 S367), is described below.
As mentioned above, finish the initialization process of abnormality processing.After this, at step S302 shown in Figure 33, control module 19 is carried out the processing of waiting for next frame.At step S303, the template matches that template matches unit 11 is carried out in the template region of search is handled.At step S304, template matches unit 11 determines whether to return normal process.
Specifically, in template matches is handled, calculate the difference between the pixel that will be mated in template (pixel in template zone 311 shown in Figure 36) in the frame of some frames in front and the template region of search absolute value and.Or rather, calculate the difference between the pixel of the pixel of the predetermined block in template zone 311 and the predetermined block in the template region of search absolute value and.The position of the piece in the movable platen zone 311 in order, and the value of this position absolute value of difference and that be added and be confirmed as to be positioned at template.After this, when template is moved in the template region of search in order, search have minimum absolute difference and the position and the value of this position.At step S304, compare with the minimum of poor absolute value with predetermined threshold.If the minimum of absolute difference and be less than or equal to this threshold value determines then to comprise that the image (being included in the template) of trace point occurs again, therefore, definitely can return normal process.Return normal process at step S1 shown in Figure 2 then.
Then, as mentioned above,, determine to have returned normal process at step S41 shown in Figure 7.At step S44, have absolute difference minimum and the position be considered to position with template matches.After this, according to the relation of the position in the regional estimation range of template position of storing explicitly and trace point, determine trace point and regional estimation range at matched position, with template.That is, as above in conjunction with Figure 27, determine regional estimation range 302 according to vector 306 and 307 with respect to trace point 305.
But, use when wherein not using the method for regional estimation range when estimate processing (zone for example shown in Figure 29 is estimated to handle) in the zone of step S61 shown in Figure 10 in, regional estimation range is not determined.
At step S304 shown in Figure 33, in order to determine whether to return normal process, the activity of template can be removed the value of the minimum of absolute difference and acquisition and a threshold ratio.In this case, can use the value calculated by activity computing unit 602 at step S532 shown in Figure 49 as activity.
Perhaps, in order to determine whether to return normal process, by the minimum of this absolute difference and divided by the value of the minimum of the absolute difference of front one frame and acquisition can with a threshold ratio.In this case, then do not need computational activity.
That is,, calculate the correlativity between template and template region of search at step S304.Determine according to relatively carrying out between this correlativity and threshold value is described.
At step S304, if determine to return normal process, then handle advancing to step S305, carry out and continue to determine to handle.Describing continuation in detail below with reference to process flow diagram shown in Figure 37 determines to handle.In this is handled, determine whether to proceed to follow the tracks of and handle.
At step S306, control module 19 determines that according to continuing the result's (according to following mark in step S366 shown in Figure 37 or S368 setting that will illustrate) who handles determines whether to continue to follow the tracks of trace point.Handle if can continue the tracking of trace point, then handle and return step S302, repeat to go on foot the later processing of S302.That is, repeatedly carry out this processing, occur again up to trace point.
But, (promptly determine after trace point disappears, to pass through the quantity of frame more than or equal to a threshold value THfr if determine to proceed the tracking processing of trace point at step S306 at step S365 shown in Figure 37, perhaps determine that at step S367 the scene change number is more than or equal to a threshold value THsc), then determine to follow the tracks of and handle and can not be performed.Thereby, finish to follow the tracks of and handle.
Figure 37 is illustrated in the continuation of the step S305 of Figure 33 in detail and determines to handle.At step S361, control module 19 is carried out the quantity of passing through frame that makes as variable increases by 1 processing.In the initialization process of the abnormality processing of step S301 shown in Figure 33 (at step S326 shown in Figure 34), the quantity by frame is reset to 0 in advance.
At step S362, control module 19 determines whether that occurrence scene changes.Because scene change detecting unit 13 is carried out the processing that is used to detect scene change always, it can determine whether that occurrence scene changes according to the testing result of scene change detecting unit 13.If occurrence scene changes, handle advancing to step S363, control module 19 makes the scene change number as variable add one.In the initialization process of step S326 shown in Figure 34, the scene change number is also reset to 0 in advance.Do not have occurrence scene to change under the situation of abnormality processing if be transferred to, then skip processing at step S363 in normal process.
Then, at step S364, control module 19 determines whether in the current pattern that is set up be the scene change pattern.This pattern is set up at step S322 or S323 shown in Figure 34.If the pattern of Set For Current is the scene change pattern, then handle advancing to step S367, wherein control module 19 determines that whether the scene change number is less than predetermined threshold THsc.If the scene change number less than predetermined threshold THsc, is then handled and advanced to step S366, wherein control module 19 is provided with the mark that indication can continue.If the scene change number more than or equal to predetermined threshold THsc, is then handled and advanced to step S368, wherein control module 19 is provided with the mark that expression can not continue.
On the contrary, going on foot S364, if determine that the pattern of Set For Current is not scene change pattern (is another pattern if determine), then handling advancing to step S365, wherein whether control module 19 determines to pass through the quantity of frame less than predetermined threshold THfr.Quantity by frame is also reset to 0 in advance in the initialization process of the step of abnormality processing shown in Figure 32 S326.If the quantity of determining to pass through frame less than predetermined threshold THfr, then is provided with the mark that expression can continue at step S366.But, if the quantity of determining to pass through frame more than or equal to predetermined threshold THfr, then is provided with the mark that expression can not continue at step S368.
As mentioned above, if the scene change number is more than or equal to threshold value THsc in template matches is handled, if the quantity of perhaps passing through frame more than or equal to threshold value THfr, is then determined can not continue to carry out to follow the tracks of to handle.
If pattern is another kind of pattern, then can when consideration scene change number be 0 condition, determines whether and to continue.
In the superincumbent explanation, carry out processing, and all frames are used to all handle according to the frame of image.But, can carry out processing according to the field.In addition, replace using all frames or all fields, can use by frame or field to be used for handling with a predetermined space rarefaction frame or an extraction.
In addition, in the explanation in front, the point of destination in the zone of estimating is used as the transfer candidate, can directly use the point in the zone of estimation, in this case, being changed in the normal process of step S1 shown in Figure 2 is that processing shown in Figure 38 replaces processing shown in Figure 6.
Basically the step S21 with shown in Figure 6 is identical to the processing of S29 in the processing from step S401 to step S410 shown in Figure 38.But, its difference is, then estimate relevant treatment, and replace the zone of step S26 shown in Figure 6 to estimate relevant treatment and carry out in the renewal of the regional estimation range of step S407 and handle in the zone that the processing that is used to wait for next frame corresponding to the step S402 shown in Figure 38 of step S22 shown in Figure 6 is inserted in step S403.Other processing and Fig. 6's is identical.Therefore will not repeat.
Estimate in the detailed zone of step S403 shown in Figure 38 relevant treatment with illustrate with reference to Figure 10 identical.Handle identical with reference to Figure 16 explanation in the renewal of the regional estimation range of step S407.
When carrying out normal process, estimate that in the zone of step S403 the zone of relevant treatment (relevant treatment is estimated in zone shown in Figure 10) is estimated to handle (zone of step S61 shown in Figure 10 is estimated to handle) by flowcharting shown in Figure 39 according to process flow diagram shown in Figure 38.
Processing from step S431 to the step S435 step S81 with shown in Figure 11 basically is identical to the processing of step S86.But, the renewal of the regional estimation range of step S86 shown in Figure 11 is handled and has been removed from processing shown in Figure 39.Other processing and Figure 11's is identical.That is,, in estimating to handle, zone shown in Figure 39 just do not need the renewal of regional estimation range to handle because handle in the renewal of step S407 execution area estimation range shown in Figure 38.
In addition, when carrying out normal process shown in Figure 38, estimate that in the zone of step S403 the transfer candidate extraction of relevant treatment (relevant treatment is estimated in zone shown in Figure 10) is handled (the transfer candidate of step S62 shown in Figure 10 extracts and handles) by shown in Figure 40.It is identical to extract processing at the processing of step S451 and the transfer candidate of step S231 shown in Figure 32.
As mentioned above, the difference between processing when carrying out normal process according to process flow diagram shown in Figure 38 and the processing when carrying out normal process according to process flow diagram shown in Figure 6 is shown in Figure 41 and 42.
When carrying out normal process according to process flow diagram shown in Figure 6 and the point of being represented by the black box in the regional estimation range 81 in the frame n when zone 82 as shown in figure 41 551 when constituting, the locational point 552 that is moved to according to the 82 interior points 551 of the zone among the motion vector 553 previous frame n is confirmed as shifting candidate (processing of the step S161 among Figure 23) in frame n+1.
The motion vector 553 of each point 551 equals the motion vector of full frame motion sometimes.But, the motion of the estimation of point is according to relating to whether the motion of determining each point equals the precision of full frame motion and slightly different.For example, be identical if determine to have the motion along continuous straight runs of difference a bit with vertical direction, then the motion of (0,0) comprises the motion of (1,1) and the motion of (1,0).In this case, when full frame motion was (0,0), each point 551 with the motion of (1,1) or (1,0) was moved the amount of this motion immediately.Replace direct application target point as shifting candidate, immediate point can be confirmed as shifting candidate in the sampling spot of Huo Deing in advance.Certainly, handle burden in order to reduce, each point 552 can be moved the amount of full frame motion.
Relative therewith, when carrying out normal process according to process flow diagram shown in Figure 38, the point 561 of regional estimation range 81 inside in frame n is confirmed as shifting candidate, as shown in figure 42.
The exemplary configuration of motion estimation unit shown in Figure 1 12 is described below with reference to Figure 43.Motion estimation unit 12 comprises motion vector detecting unit 606-1 and motion vector accuracy computation unit 606-2.In this embodiment, input picture is transmitted to estimated value computing unit 601, activity computing unit 602 and motion vector detecting unit 606-1.
Motion vector detecting unit 606-1 detects motion vector by input picture, and the motion vector and the input picture that detect are sent to motion vector accuracy computation unit 606-2.If input picture has comprised motion vector, then motion vector detecting unit 606-1 makes view data separate with motion vector, and view data and motion vector are sent to motion vector accuracy computation unit 606-2.If input data and motion vector are imported individually, then can not need motion vector detecting unit 606-1.
Motion vector accuracy computation unit 606-2 calculates the precision (hereinafter referred to as " motion vector precision ") of corresponding sports vector according to input picture (view data), and and exports the precision of acquisition together by the motion vector that motion vector detecting unit 606-1 transmits.
In this embodiment, motion vector accuracy computation unit 606-2 comprises estimated value computing unit 601, activity computing unit 602, and computing unit 606-3.Computing unit 606-3 comprises threshold value determining unit 603, normalized unit 604, and Integral Processing unit 605.
The motion vector that is transmitted by motion vector detecting unit 606-1 shown in Figure 43 is imported into estimated value computing unit 601.Input picture (view data) is imported into estimated value computing unit 601 and activity computing unit 602.
The estimated value of estimated value computing unit 601 calculating input images also sends this estimated value to normalized unit 604.The activity of activity computing unit 602 calculating input images also sends activity to the normalized unit 604 of threshold value determining unit 603 and computing unit 606-3.
The estimated value that normalized unit 604 comes normalization to be transmitted by estimated value computing unit 601 according to the activity that is transmitted by activity computing unit 602, and a value that obtains sends Integral Processing unit 605 to.Threshold value determining unit 603 compares activity and the predetermined threshold that is transmitted by activity computing unit 602, and definite result is sent to Integral Processing unit 605.Integral Processing unit 605 bases are come the calculating kinematical vector precision by the normalization information of normalized unit 64 transmission and the definite result who is transmitted by threshold value determining unit 603, thus the calculating kinematical vector precision.Integral Processing unit 605 is to the motion vector precision of equipment output acquisition then.At this moment, the motion vector that is transmitted by motion vector detecting unit 606-1 can also be exported in Integral Processing unit 605.
Describing the motion calculation of being undertaken by motion estimation unit 12 in detail below with reference to process flow diagram shown in Figure 44 handles.Motion vector detecting unit 606-1 obtains input picture at step S501, and S502 is divided into predetermined block to the frame of input picture in the step, at step S503 relatively this frame and the frame of (or front) subsequently in time, so that the detection motion vector.More particularly, by means of using block matching method to detect motion vector.The motion vector that detects is transmitted to estimated value computing unit 601.
To Figure 48 this processing is described below with reference to Figure 45.That is, at step S501 shown in Figure 44, for example, and as shown in figure 45, N frame F 1(first frame) is to F N(N frame) obtained in order.At step S502, the image in a frame is divided into square, and each square has the limit of 2L+1 pixel.Wherein, be located at frame F nIn any be piece Bp, and as shown in figure 46, the centre coordinate (pixel) of establishing piece Bp be a P (Xp, Yp).
At step S503, for example as shown in figure 47, at frame F N+1, it is frame F nNext frame, piece Bp scanning frame F N+1In predetermined scan area so that check absolute difference and position minimum make respective pixel.Like this, detect the piece (piece Bq) that is positioned at absolute difference and position minimum that makes respective pixel.(Xq Yq) is confirmed as some P (Xp, some YP) corresponding to piece Bp to detected central point Q.
As shown in figure 48, the center point P of piece Bp (Xp, Yp) and the central point Q of piece Bq (Xq, Yq) line between (arrow) be detected as motion vector V (Vx, Vy).Be motion vector V (Vx, Vy) by being calculated according to following formula:
V(vx，vy)＝Q(X q，Y q)-P(X p，Y p) ...(1)
At step S504 shown in Figure 44, attribute information storage unit 22 is carried out the motion vector accuracy computation and is handled.Below with reference to Figure 49 this processing is elaborated.Handle as a number magnitude calculation motion vector precision by means of this.
At step S505, motion vector accuracy computation unit 606-2 determines whether to be done for the calculating of the motion vector precision of all pieces in the frame.
If determine not finish, then handle and return step S504, and repeatedly carry out processing after the step S504 for the calculating of the motion vector precision of all pieces in this frame at step S505 motion vector accuracy computation unit 606-2.If motion vector accuracy computation unit 606-2 has determined to finish the calculating of the motion vector precision of all pieces, then the processing for this frame is done.All carry out above-mentioned processing for each frame.
The motion vector accuracy computation that is described in detail in step S504 shown in Figure 44 below with reference to process flow diagram shown in Figure 49 is handled.At step S531, estimated value computing unit 601 according to following formula calculate estimated value Eval (P, Q, i, j):
Eval(P，Q，i，j)＝∑∑|F j(X q+x，Y q+y)-Fi(X p+x，Y p+y)|
...(2)
Summation ∑ ∑ in formula (2) calculates for x in from-L to the scope of L, is calculating for y in the scope of L from-L.That is,, as shown in figure 50, suppose piece Bp and the piece Bq (limit of individual pixel of=2L+1=2 * 2+1) that has 5 in order to simplify.So, at frame F nIn the upper left corner coordinate of piece Bp be (some P 1The pixel value of the pixel 71 of (Xp-2, Yp-2)) and corresponding to pixel 771 at frame F N+1The coordinate of middle piece Bq is (point (Q 1Difference between the pixel value of the pixel 881 of (Xq-2, Yq-2)) is calculated.Equally, be positioned at a P 1(Xp-2, Yp-2) and P 25(Xp+2, Yp+2) pixel value of each pixel between and be positioned at Q 1(Xq-2, Yq-2) and Q 25(Xq+2, Yq+2) difference between the pixel value of the respective pixel of the piece Bq between is calculated.When L=2, obtain 25 differences, and calculate the summation of absolute difference.
(Xp, Yp) (it is above-mentioned at frame F to be positioned at P nThe quantity of pixel (pixels of interest) centre coordinate of interior piece Bp) is frame F with being positioned at N+1In the Q of centre coordinate of piece Bq (Xq Yq) and corresponding to the quantity of the pixel (respective pixel) of the central point of piece Bq can be at least one.But, when using a plurality of pixel, it is identical requiring this quantity.
This estimated value table be shown in its center in the frame for the piece of each point and in another frame its center be the estimated value (being the estimated value of motion vector) between the piece of this point.When this estimated value approached 0, then piece became more similar each other.Should be noted that in formula (2) F iAnd F jRepresent different in time frames.In the explanation in front, F nCorresponding to F i, F N+1Corresponding to F jIn formula (2), though absolute difference and as estimated value, the difference of two squares and also can be confirmed as estimated value.
Replace block matching method, can use gradient method or vector detection method.
Estimated value computing unit 601 transmits the estimated value that produces to normalized unit 604.
At step S532, activity computing unit 602 is by the input picture computational activity.Activity refers to the characteristic quantity of the complicacy of presentation video.Shown in Figure 51 and 52, (x (is neighbor Y (x-1 with 8 adjacent pixels y) being used for the pixels of interest Y of each pixel, y-1), and Y (x, y-1), Y (x+1, y-1), Y (x+1, y), and Y (x+1, y+1), Y (x, y+1), Y (x-1, y+1), and the difference between the Y (x-1, y)) absolute and on average calculated as the activity of pixels of interest according to following formula:
Activity ( x , y ) = Σ j = - 1 j Σ i = - 1 i | Y ( x + i , y + j ) - Y ( x , y ) | 8 · · · ( 3 )
In the example shown in Figure 52, and pixels of interest Y (x, value y) is 110, it is positioned at the center of 3 * 3 pixels.With pixels of interest Y (x, y) adjacent 8 pixels (neighbor Y (and x-1, y-1), Y (x, y-1), Y (x+1, y-1), Y (x+1, y), Y (x+1, y+1), Y (x, y+1), Y (x-1, y+1), and Y (x-1, y)) value is respectively 80,70,75,100,100,100,80 and 80.Thereby activity is expressed from the next:
Activity(x，y)＝{|80-110|+|70-110|+|75-110|+|100-110|+|100-110|+|100-110|+|80-110|+|80-110|}/8＝24.375
When according to pixel calculating kinematical vector precision, this activity is directly used in the calculating kinematical vector precision.When according to piece (piece comprises a plurality of pixels) calculating kinematical vector precision, the activity of piece is additionally calculated.
For according to piece calculating kinematical vector precision, for example, shown in Figure 53 A, (the piece Bp on the limit of=2L+1=2 * 2+1), the pixel 771 that is included in the center of activity zoning 851a is defined as pixels of interest for having 5.After this, use the value of pixel 771 and the value computational activity of 8 pixels adjacent with pixel 771.
In addition, to shown in the 53F, the pixel in piece Bp is scanned in order as Figure 53 B, to calculate with respect to the activity that is included in the neighbor pixels of interest of activity zoning 851b in each of 851f.The summation of the activity that all pixels among the piece Bp are calculated is defined as the activity of the piece of piece Bp.
Thereby, the summation of the activity of calculating for all pixels in the piece that is expressed from the next be defined as activity (piece activity) Blockactivity of piece (i, j):
Block activity(i，j)＝∑∑|Activity(x，y)| ...(4)
The summation that is provided by formula (4) is being calculated for x in the scope of L from-L, is being calculated for y in the scope of L from-L.The center that i in formula (4) and j represent piece, thus different with i and j in the formula (3).
Should be noted that the variation, dynamic range of piece or expression can be used for activity along other value of the change of direction in space pixel value.
At step S534, whether the piece activity that threshold value determining unit 603 is determined to be calculated by activity computing unit 602 at step S532 is greater than a predetermined threshold (threshold value Tha, it is illustrated with reference to Figure 53 below).Describe this processing in detail below with reference to the process flow diagram shown in Figure 54.In this is handled, an expression piece activity is set whether greater than the mark of threshold value THa.
At step S534, normalized is carried out in normalized unit 604.Describe this processing in detail below with reference to Figure 56.In this is handled, according to the estimated value of calculating at step S31, in the piece activity and threshold value (gradient of line 903, this describes with reference to Figure 55 below) the calculating kinematical vector precision that go on foot S532 calculating.
At step S535, Integral Processing is carried out in Integral Processing unit 605.Describe this processing in detail below with reference to Figure 57.In this is handled, according to the definite motion vector precision (step S552 or S553 shown in Figure 54) of the mark that is provided with at step S533 to the output of equipment (not shown).
Describe the threshold process of the step S533 shown in Figure 49 in detail below with reference to Figure 54.At step S551, threshold value determining unit 603 is according to determining that in the result of step S532 shown in Figure 49 whether the piece activity of calculating is greater than threshold value THa.
More particularly, experimental result shows that the piece activity is relevant as the estimated value of parameter with the use motion vector, shown in Figure 55.In Figure 55, transverse axis represents that (i, j), the longitudinal axis is represented estimated value Eval to piece activity blockactivity.If motion is correctly detected (if providing correct motion vector), then the value of the value of piece activity and estimated value is distributed in the lower area R1 of below of curve 901.In contrast, if provide wrong motion (motion vector of mistake), the value of piece activity and the value of estimated value are distributed on the left region R 2 (this value seldom is dispersed in the zone outside the region R 1 of the region R 2 of curve 902 tops and curve 901 belows) of curve 902.Curve 901 intersects at a P and curve 902.Value in the piece activity of a P is defined as threshold value THa.Threshold value THa represents that the value of if block activity is less than threshold value THa, and then having the corresponding sports vector is incorrect possibility (this describes in detail below).Threshold value determining unit 603 vector product divisional processing unit 605 output expression by the value of the piece activity of activity computing unit 602 inputs whether greater than the mark of threshold value THa.
If S551 determines the piece activity greater than threshold value THa (the corresponding sports vector is likely correct) in the step, then handle advancing to step S552.At step S552, threshold value determining unit 603 is provided with the mark of indicator dog activity greater than threshold value THa.
In contrast,, determine that the piece activity is not more than (promptly less than) threshold value THa (having the corresponding sports vector is incorrect possibility), then handle advancing to step S553, the mark that expression piece activity is not more than (promptly less than) threshold value THa is set if at step S551.
After this, whether threshold value determining unit 603 vector product divisional processing unit 605 output expression input block activity are greater than the mark of threshold value.
Describe the normalized of the step S534 shown in Figure 49 in detail below with reference to the process flow diagram shown in Figure 56.At step S571, piece activity that normalized unit 604 calculates according to the estimated value of calculating at step S531, in the step 532 and predetermined threshold (gradient of the line 903 shown in Figure 55) are according to following formula calculating kinematical vector precision VC:
VC=1-estimated value/piece activity ... (5)
In motion vector precision VC, the value that estimated value obtains divided by the piece activity is determined the position in the curve shown in Figure 55, and expression is with respect in initial point O and gradient being line 903 between 1 the some P, and this point is positioned at lower area or upper area.That is, the gradient of line 903 is 1.If the value that estimated value obtains divided by the piece activity is greater than 1, then distribute in the zone of online 903 tops corresponding to the point of this value.This means, when deduct 1 and during the motion vector precision VC less (bigger for negative value) that obtains, the possibility that respective point is distributed in region R 2 increases from this value.
On the contrary, if the value that estimated value is obtained divided by the piece activity less than 1, then distributes in the zone of online 903 belows corresponding to the point of this value.This means that when motion vector VC is big (approaching 0), the possibility that respective point is distributed in region R 1 increases.The motion vector precision VC that 604 vector product divisional processing unit, 605 outputs of normalized unit obtain in this way.
At step S572, whether the motion vector precision VC that normalized unit 604 is determined to calculate according to formula (5) is less than 0 (whether motion vector precision VC bears).If VC is more than or equal to 0 for the motion vector precision, then the processing of normalized unit 604 advances to step S573.At step S573, normalized unit 604 directly sends the motion vector precision VC that calculates at step S571 to Integral Processing unit 605.
But, if determine motion vector precision VC less than 0 (motion vector precision VC bears), then handle advancing to step S574 at step S572.At step S574, it is a fixed value 0 that normalized unit 604 is provided with motion vector precision VC, and motion vector precision VC is sent to Integral Processing unit 605.
Thereby, be incorrect possibility (motion vector is the vector of a mistake) (being that motion vector precision VC bears) if having motion vector, then the motion vector precision is set to 0.
Be described in detail in the Integral Processing of step S535 shown in Figure 49 below with reference to the process flow diagram shown in Figure 57.
At step S591, Integral Processing unit 605 determines whether the piece activity is less than or equal to threshold value THa.This determines to carry out according to the mark that is transmitted by threshold value determining unit 603.The if block activity is then directly exported motion vector precision VC and the motion vector that is calculated by normalized unit 604 in step S592 Integral Processing unit 605 greater than threshold value THa.
On the contrary, be less than or equal to threshold value THa if determine the piece activity, then the motion vector precision VC that is calculated by normalized unit 604 is set to 0 and be output at step S593.
This is because even the motion vector precision VC that calculates when normalization processing unit 604 is timing, if block activity value then has the possibility that can not obtain correct motion vector less than threshold value THa.Promptly, shown in Figure 55, between initial point O and some P, curve 202 extends through curve 901 (downwards by line 903) downwards in curve 901 and curve 902 area surrounded R3, wherein the piece activity is less than threshold value THa, the value that estimated value obtains divided by the piece activity is distributed in two region R 1 and the R2, therefore, can not obtain correct motion vector probably.Thereby, in this distribution, be that processing is carried out in low supposition according to the motion vector precision.Thereby, when motion vector precision VC when being negative and even when motion vector precision VC be timing, the if block activity is less than threshold value THa, then motion vector precision VC is set to 0.This design allows to obtain positive motion vector precision VC and represents the proper exercise vector reliably.In addition, when the value of motion vector precision VC increased, the possibility that obtains the proper exercise vector increased (possibility that distribution is included in the region R 1 increases).
This result is consistent with the experiment law, and these experiment laws show that in general, it is difficult obtaining reliable motion vector in brightness changes low zone (zone that activity is low).
Like this, calculating kinematical vector precision.As a result, the motion vector precision can be represented with a quantitative value, therefore, can detect reliable motion vector.Though the reference frame image is illustrated processing, these processing can be applied to field picture.
Figure 58 represents the exemplary configuration of background motion estimation unit 14 shown in Figure 1.In this example, background motion estimation unit 14 comprises frequency distribution computing unit 1051 and background motion determining unit 1052.
The frequency distribution of frequency distribution computing unit 1051 calculating kinematical vectors.Should be noted that by means of using the motion vector precision VC that transmits from motion estimation unit 12, make being reliable motion weighting to described frequency weighting.The motion that background motion determining unit 1052 determines to have maximum frequency according to the frequency distribution of being calculated by frequency distribution computing unit 1051 is a background motion.Then, background motion determining unit 1052 outputs to estimation relevant treatment unit, zone 15 to this motion.
Handle by the background motion estimation that background motion estimation unit 14 carries out referring now to Figure 59 explanation.
At step S651, frequency distribution computing unit 1051 calculates the frequency distribution of motion, more particularly, when in the scope of distance reference point ± 16 pixel, being expressed as the x coordinate of the motion vector of the candidate of background motion and y coordinate, frequency distribution computing unit 1501 is prepared 1089 (=16 * 2+1) * (16 * 2+1)) individual square frames, that is, corresponding to the square frame of the coordinate that may put of motion vector.When motion vector took place, frequency distribution computing unit 1051 made the coordinate corresponding to this motion vector add one.Thereby, frequency distribution that can calculating kinematical vector.
But, if when motion vector takes place added value 1, and if the frequency that takes place of the motion vector of low precision when high, the motion vector of this low precision may be confirmed as background motion.Therefore, when motion vector took place, frequency distribution computing unit 1051 did not add one for the square frame corresponding with this motion vector (coordinate), but this square frame increase by 1 be multiply by the value of motion vector precision VC (value of=motion vector precision VC).The value of this motion vector VC is normalized to the value in 0 to 1 scope.When this value approached 1, precision was higher.Thereby the frequency distribution of using said method to obtain becomes wherein according to the frequency distribution of motion vector precision to the motion vector weighting.Thereby the danger that the motion of low precision is confirmed as background motion is reduced.
At step S652, frequency distribution computing unit 1051 determines whether it finishes the processing of the frequency distribution of the motion of calculating all pieces.If there is not processed piece, then handle and return step S651, carry out in the processing that goes on foot S651 for next piece.
Thereby, full frame execution is calculated the processing of the frequency distribution of motion.If determine to be done, then handle advancing to step S653 for the processing of all pieces at step S652.At step S653, background motion determining unit 1052 is carried out the peaked processing that search rate distributes.That is, background motion determining unit 1052 is selected maximum frequency in the middle of the frequency of frequency distribution computing unit 1501 calculating, and determines that the motion vector corresponding to the frequency of selecting is the motion vector of background.The motion vector of this background is transmitted to the zone and estimates relevant treatment unit 15, and whether the motion that for example is used for determining background equals the full frame motion of step S104 shown in Figure 16 and step S131 shown in Figure 21.
Figure 60 describes the exemplary configuration of the scene change detecting unit 13 shown in Fig. 1 in detail.In this example, scene change detecting unit 13 comprises motion vector precision average calculation unit 1071 and threshold value determining unit 1072.
Motion vector precision average calculation unit 1071 is calculated from what motion estimation unit 12 transmitted and is used for the average of full frame motion vector precision VC, and this is on average outputed to threshold value determining unit 1072.Threshold value determining unit 1072 will be by on average comparing with predetermined threshold value that motion vector precision average calculation unit 1071 transmits.Threshold value determining unit 1072 determines according to comparative result whether scene change takes place also definite result to be outputed to control module 19 then.
Operation below with reference to the flowchart text scene change detecting unit 13 shown in Figure 61.The step S681, motion vector precision average calculation unit 1071 compute vectors precision and.More particularly, 1071 additions of motion vector precision average calculation unit are from the value of the motion vector precision VC that calculates for each piece of Integral Processing unit 605 output of motion estimation unit 12.The step S682, motion vector precision average calculation unit 1071 determine whether to all pieces all finished the calculating kinematical vector precision VC's and processing.If this processing is not finished as yet for all pieces, then motion vector precision average calculation unit 1701 repeats to go on foot the processing of S681.Handle by means of repeating this, calculate all pieces in a screen motion vector precision VC with.If step S682 determine to have finished all pieces of calculating in a screen the motion vector precision VC's and processing, then handle advancing to going on foot S683.At step S683, motion vector precision average calculation unit 1071 is carried out the average processing of motion vector precision VC.More particularly, the vector accuracy of a screen that calculates at step S681 VC's and divided by the piece number that is used for addition.Result's value is defined as mean value.
At step S684, threshold value determining unit 1072 will be compared with predetermined threshold the average of motion vector precision VC that step S683 is calculated by motion vector precision average calculation unit 1071, to determine that whether this threshold value is less than mean value.In general, if scene change takes place between two frames of different time at moving image, then respective image disappears.Therefore, even calculating kinematical vector, the precision of this motion vector also is low.Thereby, if motion vector precision VC on average less than threshold value, then change mark at step S685 threshold value determining unit 1072 Open Scenes.If on average being not less than of motion vector precision VC (promptly more than or equal to) threshold value, then threshold value determining unit 1072 is turn-offed the scene change mark at step S686.The scene change mark of opening represents to have taken place scene change, and the scene change mark that turn-offs represents that occurrence scene does not change.
Give control module 19 with this scene change token-passing, be used for determining at step S321 shown in Figure 34 and step S362 shown in Figure 37 whether scene change takes place.
The following describes the image processing equipment that comprises above-mentioned object tracking equipment.Figure 62 description object tracking equipment is applied to the example of television receiver 1700.Tuner 1701 receives the RF signal, and demodulation RF signal becomes picture signal and sound signal, and output image signal is to graphics processing unit 1702, and output audio signal is to audio treatment unit 1707.
Graphics processing unit 1702 demodulation are from the picture signal of tuner 1701 inputs.Graphics processing unit 1702 is exported the picture signal of demodulation then to subject tracking unit 1703, zoomed image generation unit 1704, and selected cell 1705.In fact subject tracking unit 1703 has the structure identical with above-mentioned object tracking equipment shown in Figure 11.Subject tracking unit 1703 is carried out the processing of following the tracks of in input picture by the trace point of user-defined object.Subject tracking unit 1703 is to the coordinate information of zoomed image generation unit 1704 outputs about trace point.Zoomed image generation unit 1704 is that the center produces a zoomed image with the trace point, and the output zoomed image is to selected cell 1705.Selected cell 1705 is selected the image that transmits from graphics processing unit 1702 according to user instruction and one of the image that transmitted by zoomed image generation unit 1704, and the image that output is selected is to the image display 1706 that is used to show this image.
Audio treatment unit 1707 demodulation from the sound signal of tuner 1701 input and the signal of exporting demodulation to loudspeaker 708.
By user's remote controller 1710.Telepilot 1710 is to the signal of control module 1709 outputs corresponding to user's operation.Control module 1709 for example comprises microcomputer, and response user's all elements of instruction control.Removable medium 1711 comprises semiconductor memory, disk, CD or magneto-optic disk.Removable medium 1711 is installed as required.Removable medium 1711 provides program and various types of data to control module 1709.
Processing below with reference to the flowchart text television receiver 1700 shown in Figure 63.
At step S701, tuner 1701 receives the signal of the channel of RF signal and demodulation user appointment by the antenna (not shown).Then, tuner 1701 is to graphics processing unit 1702 output image signals and to audio treatment unit 1707 output audio signals.Sound signal is by audio treatment unit 1707 demodulation, and output from loudspeaker 1708.
Graphics processing unit 1702 demodulation received image signals and to subject tracking unit 1703, zoomed image generation unit 1704 and selected cell 1705 output image signals.
At step S702, subject tracking unit 1703 determines whether the user starts tracking.If subject tracking unit 1703 determines that tracking is not activated, then subject tracking unit 1703 is skipped the processing at step S703 and S704.At step S705, selected cell 1705 is according to the picture signal of selecting from the control of control module 1709 to be transmitted by graphics processing unit 1702 with by one of picture signal of zoomed image generation unit 1704 inputs.In this case, because do not receive user's instruction, the picture signal that control module 1709 command selection unit 1705 are selected from graphics processing unit 1702.At step S706, image display 1706 shows the image of being selected by selected cell 1705.
At step S707, control module 1709 determines whether to finish the image display process according to user instruction.That is, in order to stop the image display process, user's remote controller 1710, command control unit 1709 termination image display process.If control module 1709 is not received user instruction, then handle and return step S701 and repeatedly carry out the later processing of step S701.
Like this, just carry out the normal process that directly shows the image corresponding with the signal of tuner 1701 receptions.
When image that explicit user on display 1706 is wanted to follow the tracks of, the user operates tuner 1701 and specifies this image.When carrying out this operation, control module 1709 determines that at step S702 tracking is activated, and controlling object tracking cell 1703.Under the control of control module 1709, subject tracking unit 1703 begins to follow the tracks of the trace point by user's appointment.This processing is identical with the processing of being carried out by above-mentioned object tracking equipment 1.
At step S704, zoomed image generation unit 1704 produces the zoomed image of its center at the trace point of being followed the tracks of by subject tracking unit 1703, and to selected cell 1705 output zoomed images.
This convergent-divergent is handled and can be carried out by using the adaptive classification technology that is proposed by the inventor.For example, the open 2002-196737 of Japanese Unexamined Patent Application has described a kind of technology, wherein uses the coefficient that is obtained by pre-training managing the conversion of signals of 525i to be become the signal of 1080i.This processing is actually and the identical processing of processing with 9/4 coefficient enlarged image of along continuous straight runs and vertical direction.But, the pixel count in image display 706 is fixed.Thereby, for example in order to produce 9/4 times big image, zoomed image generation unit 704 can be by being the 1080i signal with the 525i conversion of signals, and select its center of predetermined quantity to produce zoomed image in the pixel (corresponding to the pixel count of image display 706) of trace point.In order to reduce image, then carry out opposite operation.
Around this principle can produce image by any scale-up factor convergent-divergent.
If the trace command of receiving, then selected cell 1705 is at the zoomed image of step S705 selection by 1704 generations of zoomed image generation unit.As the result who selects, image display 1706 shows the zoomed image that is produced by zoomed image generation unit 1704 at step S706.
Its center just is shown on image-display units 1706 at the zoomed image of user-defined trace point like this.If scale-up factor is set to 1, then only follow the tracks of.
Figure 64 represents the functional structure according to image processing equipment 1801 of the present invention.Image processing equipment 1801 comprises motion vector detecting unit 1821 and motion vector accuracy computation unit 1822.
Motion vector detecting unit 1821 detects motion vector by input picture, and the motion vector and the input picture that detect are sent to motion vector accuracy computation unit 1822.In addition, when input picture contained motion vector, motion vector detecting unit 1821 divide image data and motion vector also sent view data and motion vector to motion vector accuracy computation unit 1822.If input data and motion vector are imported individually, then do not need motion vector detecting unit 1821.
Motion vector accuracy computation unit 1822 calculates the precision (below be called " motion vector precision ") of corresponding sports vector according to input picture (view data), and a precision that obtains is outputed to an equipment (not shown).
Figure 65 represents the exemplary configuration of the motion vector accuracy computation unit 1822 shown in Figure 64.In this embodiment, motion vector accuracy computation unit 1822 comprises estimated value computing unit 1841, activity computing unit 1842, and computing unit 1843.Computing unit 1843 comprises threshold value determining unit 1851, normalized unit 1852 and Integral Processing unit 1853.
Be transfused to estimated value computing unit 1841 from the motion vector of 1821 outputs of the motion vector detecting unit shown in Figure 64.Input picture (view data) is transfused to estimated value computing unit 1841 and activity computing unit 1842.
The estimated value of estimated value computing unit 1841 calculating input images also is sent to normalized unit 1852 to estimated value.The activity of activity computing unit 1842 calculating input images, and activity is sent to the normalized unit 1852 of threshold value determining unit 1851 and computing unit 1843.
The estimated value that normalized unit 1852 transmits from estimated value computing unit 1841 according to the activity normalization that is transmitted by activity computing unit 1842, and a value that obtains is sent to Integral Processing unit 1853.Threshold value determining unit 1851 will be compared with predetermined threshold from the activity that activity computing unit 1842 transmits, and definite result is sent to Integral Processing unit 1853.Integral Processing unit 1853 is according to the normalization information of 1852 transmission from the normalized unit and the definite precision of calculating kinematical vector as a result that transmits from threshold value determining unit 1851.Integral Processing unit 1853 is to the motion vector precision of equipment (not shown) output acquisition then.
Motion vector detecting unit 1821, motion vector accuracy computation unit 1822, estimated value computing unit 1841, accuracy computation unit 1842, computing unit 1843, threshold value determining unit 1851, normalized unit 1852 and Integral Processing unit 1853 have respectively and above-mentioned motion vector detecting unit 606-1 shown in Figure 43, motion vector accuracy computation unit 606-2, estimated value computing unit 601, activity computing unit 602, computing unit 606-3, threshold value determining unit 603, normalized unit 604, and Integral Processing unit 605 essentially identical structures.Therefore do not repeat their explanation.
Above-mentioned image processing equipment 1801 for example can be made of personal computer.
In this case, the configuration of image processing equipment 1 is for example shown in Figure 66.CPU (central processing unit) (CPU) 1931 is carried out various processing according to program stored in ROM (read-only memory) (ROM) 1932 or from the program that storage unit 1939 is packed into the random-access memory (ram) 1933.RAM 233 also stores CPU 1931 as required for carrying out the required data of various processing.
CPU 1931, ROM 1932 and RAM 1933 interconnect by bus 1934.Input/output interface 1935 also links to each other with bus 1934.
Following element links to each other with input/output interface 1935: input block 1936, for example comprise keyboard and mouse, display for example comprises cathode ray tube (CRT) or LCD (LCD), output unit 1937, for example comprise loudspeaker, communication unit 1938 for example comprises modulator-demodular unit or terminal adapter, and storage unit 1939, for example comprise hard disk.Communication unit 1938 carries out the processing of communicating by letter with distinct device by LAN or internet (not shown).
Drive unit 1940 also links to each other with input/output interface 1935.The removable medium that comprises disk, CD, magneto-optic disk or semiconductor memory is installed in the drive unit 1940 as required.The computer program of reading from these media is installed in the storage unit 1939 as required.
Illustrate according to coding unit 2261 of the present invention below with reference to Figure 67.
In coding unit 2261, input picture is sent to motion vector detecting unit 1821, motion compensation units 2272, and the selected cell 2273 of movement calculation unit 2271.Movement calculation unit 2271 has and the practically identical configuration of the above-mentioned image processing equipment 1801 shown in Figure 64.Motion vector detecting unit 1821 detects from the motion vector of input picture and the motion vector that detects and outputs to motion compensation units 2272 and extracode generation unit 2275.In addition, motion vector detecting unit 1821 is to motion vector accuracy computation unit 1822 output movement vector and input pictures.
Motion vector accuracy computation unit 1822 is according to motion vector and input picture calculating kinematical vector precision by motion vector detecting unit 1821 inputs, and the motion vector precision of calculating to control module 2274 outputs.Control module 2274 is according to the motion vector precision control selected cell 2273 and the extracode generation unit 2275 of input.
Motion compensation units 2272 is come compensating motion according to the input picture of transmission and the motion vector that is transmitted by motion vector detecting unit 1821, and the image of motion compensation is sent to selected cell 2273.Selected cell 2273 is selected the image of input picture or motion compensation, and the image of selecting to 2276 outputs of pixel value coding unit under the control of control module 2274.The image that pixel value coding unit 2276 codings receive also outputs to integral unit 2277.
Extracode generation unit 2275 produces extracode, and whether the motion of the every two field picture of this coded representation is compensated under the control of control module 2274, and with extracode and the motion vector combination of importing by motion vector detecting unit 1821.If desired, extracode generation unit 2275 adds the motion vector precision to image, and then, extracode generation unit 2275 is to the image of integral unit 2277 output combinations.
Integral unit 2277 integrations are from the code of pixel value coding unit 2276 input with by the extracode of extracode generation unit 2275 inputs, and the code of integration is outputed to the equipment (not shown).
Processing below with reference to the described flowchart text coding unit 2261 of Figure 68.To step S825, image is transfused to and every two field picture is cut into predetermined block at step S821.Detect motion vector according to the piece of shearing.Calculate the precision (motion vector precision) of each motion vector.It is all detected up to the motion vector precision of all pieces to repeat identical processing.
After this, at step S826, motion compensation units 2272 is according to input picture and compensation motion vector motion.That is, according to the difference between the image of motion vector computation two continuous frames and produce error image (motion compensated image).
At step S827, under the control of control module 2274, selected cell 2273 is selected input pictures and one of the motion compensated image that transmitted by motion compensation units 2272.That is, when the motion vector precision was enough high, control module 2274 command selection unit 2273 selected motion compensated image as the image that will be encoded.When the motion vector precision was not high enough, input picture was selected in 2274 command selection unit of control module 2273.Because according to one of motion vector accuracy selection input picture and motion compensated image, thereby can avoid using the image that compensates according to the little precision passive movement of reliability.Selected cell 2273 sends the image of selecting to pixel value coding unit 2276.
At step S828, pixel value coding unit 2276 is coded in the image (image of input picture or motion compensation) that step S828 selects.
At step S829, extracode generation unit 2275 produces extracodes, and whether this code required coded image that is used to represent to decode is the image of passive movement compensation under the control of control module 2274.This extracode can comprise the motion vector precision.
At step S830, integral unit 2277 integrations are in step S828 image encoded and going on foot the extracode that S829 produces.Integral unit 2277 is exported the image and the extracode of integration then to the equipment (not shown).
Like this, image is encoded like this, makes can avoid using according to being the image that incorrect motion vector (may be wrong vector) passive movement compensates.Thereby, can avoid carrying out motion compensation and the picture breakdown that causes by unserviceable motion vector, therefore, can when decoding, obtain high quality graphic.
Figure 69 represents that wherein the present invention is applied to the example of DE Camera Shake ambiguity correction equipment 2301.For example, DE Camera Shake ambiguity correction equipment 2301 is applied to digital video camcorder.
Input picture is imported into background motion detecting unit 2311 and output image generation unit 2314.Background motion detecting unit 2311 detects the background motion of input picture and the background motion that detects is outputed to displacement accumulative element 2312.Describe background motion detecting unit 2311 in detail below with reference to Figure 70.The displacement of the background motion of displacement accumulative element 2312 accumulation inputs, and blur the displacement of determining unit 2313 and the 2314 output accumulation of output image generation unit to DE Camera Shake.The fuzzy determining unit 2313 of DE Camera Shake determines according to predetermined threshold whether the displacement information of input is fuzzy corresponding to DE Camera Shake, and output determines that the result is to output image generation unit 2314.
Output image generation unit 2314 is according to producing output image by the displacement of displacement accumulative element 2312 inputs and the definite result who is imported by the fuzzy determining unit 2313 of DE Camera Shake by the input picture that transmits.Output image generation unit 2314 is recorded in output image on the recording medium 315 that can write then, for example on hard disk drive (HDD) and the video band.In addition, the image that output image generation unit 2314 produces to display unit 2316 outputs, display unit for example comprises LCD (LCD), it shows the image that produces.
Figure 70 represents the configuration of the background motion detecting unit 2311 shown in Figure 69 in detail.In this configuration, background motion detecting unit 2311 comprises movement calculation unit 2321, frequency distribution computing unit 2322, and background motion determining unit 2323.Movement calculation unit 2321 has and the practically identical configuration of the movement calculation unit 1801 shown in above-mentioned Figure 63.
Input picture is transmitted to the motion vector detecting unit 1821 of movement calculation unit 2321.Motion vector detecting unit 1821 detects the motion vector of input picture, and exports motion vector and the input picture that detects to motion vector accuracy computation unit 1822.Motion vector accuracy computation unit 1822 calculates precision (motion vector precision) corresponding to motion vector according to the motion vector of input and input picture, and to frequency distribution computing unit 2322 translatory movement vector accuracies.
The frequency distribution of frequency distribution computing unit 2322 calculating kinematical vectors.Should be noted that by using the motion vector precision VC that transmits by movement calculation unit 2321, make being reliable motion weighting to frequency weighting.The motion that background motion determining unit 2323 determines to have maximum frequency according to the frequency distribution of being calculated by frequency distribution computing unit 2322 is a background motion.
Describe the processing of being undertaken by DE Camera Shake ambiguity correction equipment 2301 in detail below with reference to the process flow diagram shown in Figure 71.To S834, input picture is acquired and the frame of image is cut into predetermined block at step S831.For example using, block matching method detects motion vector according to the piece of shearing.Calculate the precision (motion vector precision) of each motion vector then.
At step S835, frequency distribution computing unit 2322 calculates the frequency distribution of motion.More particularly, in as the x coordinate of the motion vector of the candidate of background motion and scope that the y coordinate is leaving reference point ± 16 pixel during expression, frequency distribution computing unit 2322 is prepared 1089 (=16 * 2+1) * (16 * 2+1) individual square frames, that is, corresponding to the square frame of the coordinate that may put of motion vector.When motion vector took place, frequency distribution computing unit 2322 made the coordinate corresponding to motion vector add one.Like this, frequency distribution that can calculating kinematical vector.
But, if when motion vector takes place added value 1, and the occurrence frequency of the motion vector of low precision is when high, then the motion vector of this low precision may be confirmed as background motion.Therefore, when motion vector took place, frequency distribution computing unit 2322 was to value added 1 corresponding to the square frame (coordinate) of this motion vector, and this square frame is added 1 value that multiply by motion vector precision VC (value of=motion vector precision VC).The value of motion vector precision VC is normalized to 0 to 1 value.This value approaches 1 more, and precision is high more.Thereby the frequency distribution of using said method to obtain becomes wherein according to the frequency distribution of motion vector precision to the motion vector weighting.Thereby the motion that can reduce low precision is confirmed as the danger of background motion.
At step S836, motion vector accuracy computation unit 1822 determines whether to finish the computing of frequency distribution of the motion of all pieces.If have untreated, then handle and return step S834, next piece is carried out in the processing that goes on foot S834 and S835.After computing, handle advancing to step S837 to the full frame frequency distribution of all having carried out motion.At step S837, background motion determining unit 2323 is carried out the peaked processing that search rate distributes.That is, background motion determining unit 2323 is selected maximum frequency in the middle of the frequency of being calculated by frequency distribution computing unit 2322, and determines that the motion vector corresponding to the frequency of selecting is a motion vectors of background motion.This motion vectors of background motion is transmitted to displacement accumulative element 2312.
At step S838, displacement accumulative element 2312 is each frame motion vectors of background motion of storage representation in order.
At step S839, whether the displacements (absolute value) that the fuzzy determining unit 2313 of DE Camera Shake is determined the expression motion vectors of background motion greater than predetermined threshold, thereby determines that input picture is whether owing to the shake of video camera fogs.If displacement greater than threshold value, is then determined the generation of rocking of hand.On the contrary, if displacement less than threshold value, then determines not take place rocking of hand.The fuzzy determining unit 2313 of DE Camera Shake transmits to output image generation unit 2314 determines the result.
If in the generation of rocking of the fuzzy determining unit 2313 definite hands of step S839 DE Camera Shake, then at step S840, output image generation unit 2314 produces and moves the image of described displacement in opposite direction and export this image.Thereby the user can write down or watch and wherein reduce the fuzzy image that produces owing to rocking of hand.
On the contrary, if at step S839, the fuzzy determining unit 2313 of DE Camera Shake determines not take place rocking of hand, then handles advancing to step S841, and output image generation unit 2314 is directly exported input picture.Output image is recorded on the recording medium 2315 and is shown on display unit 2316.
Thereby, the fuzzy detected and correction of DE Camera Shake.Use the motion vector precision to allow accurately detection background motion, so as to fuzzy minimum image is provided to the user.
Figure 72 represents according to exemplary accumulation equipment of the present invention.Accumulation equipment 2341 as hard drive (HDD) register comprises selected cell 2351, recording medium (HDD) 2352, index generating unit 2353, scene change detecting unit 2354, control module 2355, concordance list 2356, selected cell 2357, display image generation unit 2358, overhead control unit 2359, and instruction input block 2360.
Selected cell 2351 is selecting to be recorded in one of image on the recording medium 2352 and input picture under the control of overhead control unit 2359, and an image of selecting is sent to index generating unit 2353, scene change detecting unit 2354 and selected cell 2357.Under the control of overhead control unit 2359, image is recorded on the recording medium 2352 that is made of HDD.
Scene change detecting unit 2354 detects the scene change in the image that transmits, and testing result is sent to control module 2355.Control module 2355 is according to the testing result control index generating unit 2353 and the concordance list 2356 that transmit.
Index generating unit 2353 is extracted the additional information (time code, address etc.) that is recorded in the thumbnail on the recording medium 2352 and is used for the position of the thumbnail on the identification record medium 2352, and they is sent to concordance list 2356 under the control of control module 2355.Thumbnail is the downscaled images of the beginning image of each scene when determining that scene change takes place.
Thumbnail and corresponding additional information that concordance list 2356 storages transmit.The additional information that concordance list 2356 transmits corresponding to the thumbnail of storage to overhead control unit 2359 under the control of control module 2355.
Selected cell 2357 is selected the image that transmitted by selected cell 2351 and by one of thumbnail of concordance list 2356 inputs under the control of overhead control unit 2359, and the image of selecting to 2358 outputs of display image generation unit.The form that display image generation unit 2358 can be shown with image display device 2365 by the image that transmits under the control of overhead control unit 2359 produces image and exports this image.
Under the control of the scene change mark of exporting by scene change detecting unit 2354, and under the control of overhead control unit 2359, control module 2355 control index generating unit 2353 and concordance lists 2356.
Overhead control unit 2359 for example comprises microcomputer and controls each element.Instruction input block 2360 comprises a plurality of buttons and switch and telepilot.Instruction input block 2360 is to the signal of overhead control unit 2359 outputs corresponding to user instruction.
Figure 73 represents the exemplary configuration of the scene change detecting unit 2354 shown in Figure 72 in detail.In this example, scene change detecting unit 2354 comprises movement calculation unit 2371, motion vector precision average calculation unit 2372, and threshold value determining unit 2373.In fact movement calculation unit 2371 has the configuration identical with the above-mentioned image processing equipment 1801 shown in Figure 64.
The motion vector that motion vector detecting unit 1821 detects in the input picture, and the motion vector and the input picture that detect output to motion vector accuracy computation unit 1822.According to the motion vector and the image of input, motion vector accuracy computation unit 1822 calculates the precision (motion vector precision) of respective motion vectors, and exports the motion vector precision that obtains to motion vector precision average calculation unit 2372.
The motion vector precision VC's that 2372 pairs of full frame calculating of motion vector precision average calculation unit are transmitted by movement calculation unit 2371 is average, and this is on average outputed to threshold value determining unit 2373.Threshold value determining unit 2373 will be by on average comparing with predetermined threshold value that motion vector precision average calculation unit 2372 transmits.Then, threshold value determining unit 2373 determines according to comparative result whether scene change takes place, and output determines that the result is to control module 2355.
Describe the thumbnail of on recording medium 2352, carrying out during document image when accumulation equipment 2341 in detail below with reference to the process flow diagram shown in Figure 74 and produce processing.When just being recorded on the recording medium 2352, input carries out this processing.
Identical with the processing of arriving step S504 with reference to Figure 44 S501 of described step respectively to the processing of step S874 at step S871, that is, in these were handled, image was transfused to, and the frame of image is sheared into predetermined block.For example use block matching method to detect motion vector according to the piece of shearing.Calculate the precision (motion vector precision) of each motion vector then.
At step S875, motion vector precision average calculation unit 2372 calculate from the motion vector precision of the image (this image is recorded on the recording medium 2352) of selected cell 2351 inputs and.More particularly, motion vector precision average calculation unit 2372 is the value addition of the motion vector precision VC that calculates for each piece of the Integral Processing unit 1853 of the motion vector accuracy computation unit 1822 of movement calculation unit 2371 output.The step S876, motion vector accuracy computation unit 1822 determined whether to finish all pieces the calculating kinematical vector precision VC's and processing.If do not finish this processing of all pieces as yet, 1822 processing that repeat to go on foot S874 and S875 in motion vector accuracy computation unit.Handle by repeating these, calculate all pieces in the screen motion vector precision VC's and.If step S876 determined to finish calculate all pieces in the screen the motion vector precision VC's and processing, then handle to advance to going on foot S877.At step S877, motion vector precision average calculation unit 2372 is carried out the average processing of calculating kinematical vector precision VC.More particularly, the motion vector precision of a screen that calculates at step S875 VC's and divided by the piece number of addition.The value of gained is defined as on average.Thereby, obtain one on average for a screen (frame).
At step S878, threshold value determining unit 2373 is relatively going on foot the average and predetermined threshold of S877 by the motion vector precision VC of threshold value determining unit 2373 calculating, and to control module 2355 output comparative results.At step S879, whether control module 2355 is determined described on average less than threshold value.In general, if occurrence scene changes between two successive frames of moving image, then respective image disappears.Therefore, even calculating kinematical vector, the precision of motion vector also is low.Thereby, if motion vector precision VC on average less than threshold value, at step S880, control module 2355 control index generating unit 2353 make to produce thumbnail.
That is, at step S881, under the control of control module 2355, the picture size in the start frame of index generating unit 2353 minimizing new scenes is to produce thumbnail.For example, when the thumbnail that shows 3 * 3 on screen, this thumbnail is to produce by vertically with horizontal direction the size of original image being reduced to 1/3.In addition, at this moment, index generating unit 2353 is extracted the additional information (time code, address etc.) of the record images position that is used to be identified in the frame on the recording medium 2352.
At step S881, index generating unit 2353 is stored in thumbnail and the corresponding additional information that step S880 produces in concordance list 2356.
At step S879, if determine motion vector precision VC on average more than or equal to threshold value, then can not change by occurrence scene probably.Therefore, skipped, do not produced thumbnail in the processing of step S880 and S881.
Then, at step S882, whether control module 2355 definite users order and stop record.If order does not stop record as yet, then handle and return step S371, repeat S871 processing afterwards.If having ordered, the user stops record, then end process.
Thereby, during recording operation, automatically detect scene change, and automatically produce thumbnail.
Be used for handling below with reference to the flowchart text shown in Figure 75 to the image output of image display device 2365 output images of accumulation equipment 2341.Carry out this processing when user command playback image and when exporting this image.
At step S901, respond the operation of the instruction input block 2360 that is undertaken by the user, overhead control unit 2359 makes the image that is recorded on the recording medium 2352 by playback and output.The image that selected cell 2351 is exported from recording medium 2352 playback to display image generation unit 2358 by selected cell 2357.The form that display image generation unit 2358 becomes the image transitions of receiving display device 2356 to show, and to the image of the display device 2365 output conversions of image display image.
At step S902, the operation of the instruction input block 2360 that the response user carries out, overhead control unit 2359 determines whether the user orders the demonstration thumbnail.If the user does not order the demonstration thumbnail, then handle and return step S901, and repeat the processing after the step S901.That is, continue also to export the image that (demonstration) writes down on recording medium 2352 in playback on the image display device 2365.
On the contrary, if the user has ordered the demonstration thumbnail, then at step S903, the thumbnail of overhead control unit 2359 control concordance lists, 2356 outputs record in concordance list 2356.That is, concordance list 2356 is read the thumbnail tabulation, and exports these tabulations by selected cell 2357 to display image generation unit 2358.Display image generation unit 2358 is to the image display device 2356 output thumbnail tabulations that show this tabulation.Thereby, on screen, show the tabulation wherein be arranged with 3 * 3 thumbnail.
By operational order input block 2360, the user can select in the thumbnail (thumbnail tabulation) of a plurality of demonstrations.After this, at step S906, overhead control unit 2359 determines whether to be chosen in one of thumbnail that shows on the image display device 2365.If determine not have thumbnail selected, then handle and return step S903, repeat step S903 processing afterwards.That is, image display device 2365 shows the thumbnail tabulation continuously.
On the contrary, if determine to select a thumbnail (selecting required thumbnail in the thumbnail of user from tabulation), then begin the image of playback from image corresponding to the thumbnail of from recording medium 2352, selecting in step S905 overhead control unit 2359.The image of record is output to image display device 2365 by selected cell 2351, selected cell 2357 and display image generation unit 2358.Image display device 2365 shows this image.That is, if determine to have selected a thumbnail, just then overhead control unit 2359 is read from concordance list 2356 corresponding to the additional information (time code, address etc.) at the thumbnail that goes on foot the S904 selection.Overhead control unit 2359 controlling recording media 2352 make to begin replay image from the image corresponding to thumbnail, and image are outputed to the image display device 2365 that is used for display image then.
At step S906, whether overhead control unit 2359 definite users order and stop output image.It is by checking the operation of the instruction input block 2360 that is undertaken by the user, determines whether the user orders to stop output (demonstration) image.If determine that the user does not import halt instruction, then handle and return step S901, repeat S901 processing afterwards.But, do not import halt instruction if determine the user, then end process.
In addition, even when recording medium for example is DVD or video band, also can use accumulation equipment 2341.
Above-mentioned a series of processing not only can but also can be carried out by software by hardware.When above-mentioned a series of processing were carried out by software, software program is downloaded to from network or recording medium in the computing machine that is comprised in the specialized hardware maybe can be by installing in the computing machine that various programs carry out various functions (for example general purpose personal computer) therein.
In this explanation, be used to illustrate that the step of above-mentioned a series of processing not only comprises the processing of carrying out with above-mentioned order, and comprise by concurrently or the processing of carrying out independently.
Figure 76 represents that wherein the present invention is applied to the example of security camera system.In security camera system 2800, the image of being caught by the image capturing unit 2801 that comprises the CCD video camera is displayed on the image display 2802.Tracing object detecting unit 2803 detects from the image of image capturing unit 2801 inputs wants tracked object, and the output testing result is to subject tracking unit 2805.Subject tracking unit 2805 is operated like this, makes in the image that tracking image capturing unit 2801 transmits the tracked object of wanting by tracing object detecting unit 2803 regulations.Subject tracking unit 2805 has and above-mentioned object tracking equipment shown in Figure 11 essentially identical configuration.Camera driver unit 2804 drives image capturing unit 2801 under the control of subject tracking unit 2805, make to catch the image of its center at the trace point of wanting tracked object.
Control module 2806 for example comprises microcomputer and controls each element.The removable medium 2807 that comprises semiconductor memory, disk, CD or magneto-optic disk is connected to control module 2806 as required.Removable medium 1711 provides program and various types of data to control module 2806 as required.
Monitor the operation of handling below with reference to the flowchart text shown in Figure 77.When security camera system 2800 was switched on power supply, image capturing unit 2801 was caught the image of safety zone, and the image of catching is outputed to tracing object detecting unit 2803, subject tracking unit 2805 and image display 2802.At step S931, tracing object detecting unit 2803 is carried out the processing that detects the object of wanting tracked from the image of image capturing unit 2801 inputs.For example, when detecting the motion object, tracing object detecting unit 2803 detects the motion object as wanting tracked image.For example tracing object detecting unit 2803 detects the point with maximum brightness of the object of wanting tracked or central point as trace point, and the information that transmits about the trace point of determining to subject tracking unit 2805.
At step S932, subject tracking unit 2805 is carried out to follow the tracks of and is handled the trace point that detects at step S931 to follow the tracks of.This follow the tracks of to handle identical with the processing of above-mentioned object tracking equipment 1 shown in Figure 1.
At step S933, subject tracking unit 2805 detects the position of the trace point on the screen.At step S934, subject tracking unit 2805 detects poor between the position of the trace point that step S933 detects and picture centre.At step S935, subject tracking unit 2805 produces and is going on foot the corresponding camera driver signal of difference that S934 detects, and output camera driver signal arrives camera driver unit 2804.At step S936, camera driver unit 2804 drives image capturing unit 2801 according to the camera driver signal.Thereby image capturing unit 2801 such pan or inclinations make trace point be positioned at picture centre.
At step S937, control module 2806 determines whether to stop monitoring processing according to user's instruction.Do not stop to monitor processing if the user orders, then handle and return step S931, and repeat the processing after the S931.Stop to monitor processing if the user has ordered, then determine end process at step S937.Thereby control module 2806 finishes to monitor to be handled.
As mentioned above, in security camera system 2800, the object of motion is automatically detected is trace point, and shows the image of its central point at trace point on image display 2802.Thereby, can handle by simpler execution monitoring more reliably.
Figure 78 represents another example according to the structure of security camera system of the present invention.Security camera system 2900 comprises image capturing unit 2901, image display 2902, subject tracking unit 2903, camera driver unit 2904, control module 2905, instruction input block 2906 and removable medium 2907.
Similar with image capturing unit 2801, image capturing unit 2901 for example comprises the CCD video camera.Image capturing unit 2901 is exported the image of catching to image display 2902 and subject tracking unit 2903.Image-display units 2902 shows input picture.Subject tracking unit 2903 has the essentially identical structure of subject tracking unit with above-mentioned object tracking equipment 1 shown in Figure 1.Camera driver unit 2904 drives image capturing unit 2901 pans or tilts along predetermined direction under the control of subject tracking unit 2903.
Control module 2905 for example comprises microcomputer and controls each element.Instruction input block 2906 comprises various buttons and switch and telepilot.Instruction input block 2906 is to the signal of control module 2905 outputs corresponding to user instruction.The removable medium 2907 that comprises semiconductor memory, disk, CD or magneto-optic disk links to each other with control module 2905 as required.Removable medium 2907 provides program and various types of data to control module 2905 as required.
Operation below with reference to the flowchart text control module 2905 shown in Figure 79.
At step S961, control module 2905 determines whether the user has stipulated trace point.If trace point is not prescribed, then handle advancing to step S969, whether control module 2905 definite users order and stop to handle, and do not stop to handle if the user orders, and then handle and return step S961, and repeat S961 processing afterwards.
That is, during this was handled, the image of the image capture area of being caught by image capturing unit 2901 was output to image display 2902, and it shows this image.If user (observer) stops the processing of monitored secure areas, user's operational order input block 2906 then, command control unit 2905 stop to handle.When control module 2905 was stopped to handle by order, 205 of control modules stop to monitor to be handled.
On the other hand, if the user observes the concurrent incumbent thief how to hide of image who shows on image display 2902, then the user stipulates a point as trace point, shows the thief who hides at this point.The user stipulates this point by operational order input block 2906.When the user stipulated trace point, S961 determined that trace point is prescribed in the step, and handled and to advance to step S962, carried out to follow the tracks of and handled.Arrive the processing of step S967 execution with identical to the processing that S937 carries out at step S962 at the step S932 shown in Figure 77.That is, by carrying out this operation, image capturing unit 2901 is driven like this, makes the trace point of regulation be positioned at the center of screen.
At step S967, control module 2905 determines whether that order stops to monitor.If control module 2905 is stopped to monitor that then control module 2905 stops to handle by order.But, if control module 2909 is not stopped to monitor that then handle advancing to step S968, control module 2905 determines whether that it is stopped to follow the tracks of by order by order.For example, when the possible thief who goes out to be defined as trace point when User Recognition was not the thief, the user can operational order input block 2906, and command control unit 2905 stops to follow the tracks of.If at step S968, control module 2905 determines that it is not stopped to follow the tracks of by order, then handles and returns step S962, carries out the processing after step S962.That is, in this case, continue to follow the tracks of the operation of trace point.
If determine that at step S968 control module 2905 it is stopped to follow the tracks of by order, then stops to follow the tracks of operation.Then handle and return step S961, and repeat the processing after the step S961.
Thereby, in security camera system 2900, be shown at the center of image display 2902 by the image of user-defined trace point.Thereby the user can select any required image and can carefully monitor this image.
The present invention not only can be applied to television receiver and security camera system but also can be applied to various types of object tracking equipments.
Though top explanation is carried out with reference to Flame Image Process according to frame, the present invention can be applied to the Flame Image Process of carrying out according to the field.
Above-mentioned a series of processing not only can but also can be carried out by software by hardware.When above-mentioned a series of processing were carried out by software, software program is downloaded to from network or recording medium in the computing machine that is comprised in the specialized hardware maybe can be by installing in the computing machine that various programs carry out various functions (for example general purpose personal computer) therein.
Shown in Figure 76 or 78, the example of this recording medium comprises that not only slave unit distributed to user's removable medium 2807 or 2907 individually so that provide program to the user, for example disk (comprising floppy disk), CD (comprising compact read-only disk storer (CD-ROM) and digital universal disc (DVD)), magneto-optic disk (comprising mini disk (MD)), and semiconductor memory, also comprise ROM and be used for stored programme and be included in the hard disk of the equipment that is provided for the user.
In the present invention, the step of describing the program in the recording medium that is stored in not only comprises the processing of carrying out with above-mentioned order, and comprise can be by concurrently or the processing of carrying out independently.
In addition, as what use in this explanation, " system " refers to the logical combination of multiple arrangement; Described multiple arrangement needn't be included in the body.
Figure 80 represents the structure according to a kind of example of security camera system 3001 of the present invention.In security camera system 3001, the image of being caught by the image capturing unit 3021 that for example comprises the CCD video camera is shown on image display 3023.Tracing object detecting unit 3024 detects the object that will follow the tracks of from the image of image capturing unit 3021 inputs, and testing result is outputed to subject tracking unit 3026.Subject tracking unit 3026 has the essentially identical structure of subject tracking unit with above-mentioned object tracking equipment 1 shown in Figure 1.
Subject tracking unit 3026 is worked like this, makes to follow the tracks of the trace point of being stipulated by tracing object detecting unit 3024 in the image that is transmitted by image capturing unit 3021.The zone is provided with unit 3025 an object presumptive area on every side that comprises the trace point in the image of being caught by image capturing unit 3021 is set, and represents the positional information of these regional positions to image correction unit 3022 outputs.Image correction unit 3022 is proofreaied and correct in the image of being caught by image capturing unit 3021 and by the zone the interior image in zone that unit 3025 is provided with is set, and removes deblurring (defocusing blurring) in the feasible image from this zone, and to image display 3023 these images of output.Camera driver unit 3029 drives image capturing unit 3021, makes and catch the image of its center at trace point under the control of subject tracking unit 3026.
Control module 3027 for example comprises microcomputer, and controls each element.The removable medium 3028 that comprises semiconductor memory, disk, CD or magneto-optic disk links to each other with control module 3027 as required.Removable medium 3208 provides program and various types of data to control module 3027 as required.Control module 3027 also receives user instruction (for example order) by the input/output interface (not shown).
Monitor processing below with reference to the flowchart text shown in Figure 81.When security camera system 3001 is switched on power supply, image capturing unit 3021 is caught the image of safety zone, and the image of catching is outputed to image display 3023 by tracing object detecting unit 3024, subject tracking unit 3026 and image correction unit 3022.At step S1001, tracing object detecting unit 3024 is carried out the processing that detects the object of wanting tracked from the image of image capturing unit 3021 inputs.For example, when detecting the object of motion, tracing object detecting unit 3204 test example as the point of wanting tracked object or central point with maximum brightness as trace point, and to the information of subject tracking unit 3026 outputs about the trace point determined.
At step S1002, subject tracking unit 3206 is carried out to follow the tracks of and is handled the trace point that detects at step S1001 to follow the tracks of.Thereby the trace point (for example center of eyes or head) of wanting tracked object (for example human or animal) in the image of being caught by image capturing unit 8021 is tracked.The positional information of indicators track point is imported into the zone unit 3025 is set.
At step S1003, the zone is provided with unit 3025 and is arranged on a presumptive area (for example its center is at the rectangle with preliminary dimension of trace point) around the object that will follow the tracks of according to the output from subject tracking unit 3026 as correcting area.
At step S1004, image correction unit 3022 carries out image treatment for correcting are provided with image correcting area that unit 3025 be provided with in by the zone to proofread and correct in the image of being caught by image capturing unit 3021.Describe the image treatment for correcting in detail below with reference to Figure 93.This processing makes and produce unambiguous picture rich in detail in correcting area.
At step S1005, the image that image-display units 3023 outputs are proofreaied and correct at step S1004, that is, wherein only the image in correcting area is the image of clearly being caught by image capturing unit 3021 especially.
At step S1006, the motion of the tracking results detected object that subject tracking unit 3026 obtains according to the processing by step S1002, and produce the camera driver signal that is used for actuated camera makes the image of object that can capture movement.Subject tracking unit 3026 is then to control module 3027 output camera driver signals.At step S1007, camera driver unit 3027 drives image capturing unit 3021 according to the camera driver signal.Thereby image capturing unit 3021 pans or inclination make trace point always be positioned at the inside of screen.
At step S1008, control module 3027 determines whether to stop monitoring processing according to user instruction.Do not stop to monitor processing if the user orders, then handle and return step S1001, repeat S1001 processing afterwards.Stop to monitor processing if the user has ordered, then determine end process at step S1008.Thereby control module 3027 stops monitoring processing.
In addition, control signal is output to camera driver unit 3029, with actuated camera (image capturing unit 3021), the object of the detection that the information trace about trace point that makes the video camera basis be exported by tracing object detecting unit 3024 will be followed the tracks of, and at the inner trace point (trace point does not move to outside the screen) that shows of the screen of image display 3023.In addition, unit 3025 and control module 3027 output tracking results are set, for example about the positional information of the trace point on screen to the zone.
Figure 82 A-C is illustrated in the example of the time-series image that shows in this case on image display 3023.Figure 82 A represents the image of the object that will follow the tracks of 3051 of being caught by image capturing unit 3021.In these examples, the people's who runs to left image is hunted down as object 3051.In Figure 82 B, object 3051 from the position shown in Figure 82 A towards left movement.At Figure 82 C, object 3051 further moves towards left from the position shown in Figure 82 B.
Tracing object detecting unit 3024 is at the step S1001 detected object 3051 shown in Figure 81, and to the eyes of subject tracking unit 3026 object outputs 3051 (people) as trace point 3051A.At step S1002, subject tracking unit 3026 is carried out to follow the tracks of and is handled.At step S1003, the zone is provided with unit 3025, and object 3051 (trace point 3051A) presumptive area on every side that will follow the tracks of is set is correcting area 3052.
As mentioned above, subject tracking unit 3026 is according to trace point 3051A tracing object 3051.Thereby when object 3051 motions, trace point 3051A also moves, and tracking results (position) is output to the zone unit 3025 is set.Thereby, shown in Figure 82 A-82C, along with object 3051 to left movement, correcting area 3052 also is moved to the left.
For example, be pressed corresponding to the correcting area 3052 of object 3051 (trace point 3051A) of motion and state setting.Figure 83 represents wherein to be provided with to have a rectangular area of preliminary dimension as correcting area around trace point.In Figure 83, correcting area 3071A is set at first.For example, its center is set to the first correcting area 3071A in the presumptive area of trace point 3051A.If the user stipulates correcting area, this zone is set to the first correcting area 3071A.At this moment, the zone be provided with unit 25 within it the coordinate in the upper left corner of storage correcting area 3071A in portion's storer (X, Y).If the trace point 3051A of object 3051 motion, then subject tracking unit 3026 just begins such tracking, makes that to the zone unit 3025 being set as tracking results transmits along X-direction (horizontal direction among the figure) with along the information of Y direction (vertical direction among the figure) about the position (or displacement) of trace point 3051A.
Then, the above-mentioned coordinate according to the upper left corner is provided with correcting area.For example, when trace point 3051A moves x in screen upper edge X-direction, when Y direction moves y, the zone be provided with unit 3025 the coordinate in the upper left corner of correcting area 3071A (X adds x and y on Y), thus coordinates computed (X+x, Y+y).The zone is provided with the coordinate of unit 3025 these coordinates of storage as the upper left corner of new correcting area 3071B.And correcting area 3071B is set.If trace point 3051A moves a along X-direction again, move b along Y direction, then the zone be provided with unit 3025 a and b be added to the upper left corner of correcting area 3071A coordinate (X+x, Y+y), thereby coordinates computed (X+x+a, Y+y+b).The zone is provided with the coordinate of unit 3025 these coordinates of storage as the upper left corner of new correcting area 3071C, and correcting area 3071C is set.
Thereby along with object (trace point) motion, correcting area also moves.
In addition, as mentioned above, carry out image correction process (the step S1004 shown in Figure 81), make the fuzzy of removal of images by 3022 pairs of images of image correction unit in correcting area 3052 inside.This image is shown on image display 3023 then.Thereby the parts of images in the image shown in Figure 82 A-C of correcting area 3052 inside is clearly illustrated.On the contrary, shown dully with the image of comparing at the image of regional 3052 inside in the background 3053 of the outside of correcting area 3052.
Thereby the object in the correcting area 3052 of the image that shows on image display 3023 is always clearly illustrated.Therefore, watch the user of image display 3023 automatically to watch object 3051.As a result, for example, the user can find the object of thief or motion quickly.In addition, because object 3051 is clearly illustrated that what (people) object (for example people) that the user can correctly discern motion is.
As mentioned above, because subject tracking unit 3026 has the essentially identical structure of subject tracking unit with object tracking equipment 1 shown in Figure 1, no longer repeat its explanation.
By constituting the subject tracking unit 3026 shown in Figure 80 in the manner described above, even the object 3051 (Figure 82) that will follow the tracks of rotate or even when taking place to interdict, perhaps even as the trace point 3051A of object 3051 because scene change and when temporarily not being shown, the object 3051 that moves in image (trace point 3051A) also can accurately be followed the tracks of.
Thereby, be output to the zone about the positional information of the trace point 3051A of the object 3051 that will follow the tracks of as the tracking results of the subject tracking unit shown in Figure 80 3026 unit 3025 is set.Thereby the zone is provided with unit 3025 can be provided with above-mentioned correcting area 3052.After this, image correction unit 3022 is removed fuzzy (defocusing blurring) of the image in the zone 3052.
Describe the structure and the operation of the image correction unit 3022 shown in Figure 80 below in detail.Figure 84 is the calcspar of the detailed structure of image correction unit 3022.In this example, image correction unit 3022 comprises: control signal generating unit 3741, be used for being provided with the output signal generation control signal of unit 3025 according to the zone, and control signal sent to each element, characteristics of image detecting unit 3742, be used to detect the feature of input picture, address calculation 3743, be used for according to control signal calculated address, coefficients R OM 3744, be used for exporting the pre-determined factor of storage in advance according to the address of calculating by address calculation 3743, and extracted region unit 3745, be used for extracting a plurality of pixels corresponding to the presumptive area of input picture.
Image correction unit 3022 also comprises inner product computing unit 3746 and image assembled unit 3747.Inner product computing unit 3746 calculates by the pixel value of extracted region unit 3745 outputs with by the inner product of the coefficient of coefficients R OM 3744 outputs, and the pixel value revised of output.Image and background 3053 in the image assembled unit 3747 combination correction zones 3052, and the image of output combination.
Figure 85 is the control signal that expression is produced by control signal generating unit 3741.Control signal A is the signal that is used for being identified in the zone (correcting area 3052) that input picture will revise.The output that unit 3025 is set according to the zone produces control signal A, and sends it to extracted region unit 3745 and image assembled unit 3747.Control signal B is the signal that is used to discern the parameter σ that represents fog-level, and this is described below.Control signal B is transmitted to address calculation 3743.The value of parameter σ for example can be determined by user instruction by control module 3027, perhaps can be determined in advance.
Control signal C is used for the signal of weighting Wa that command conversion is used to find the solution the relational expression of fuzzy model tormulation formula, and this is described below.Control signal C is transmitted to address calculation 3743.Control signal D is used for the signal that command conversion is used for the threshold value of detected image feature.Control signal D is transmitted to characteristics of image detecting unit 742.Control signal C and D can consider that the feature of security camera system 3001 is determined in advance.Perhaps, control signal C and D can be produced according to user instruction by control module 3027.
The following describes image blurring principle.Suppose that focus of camera is provided with suitably, and the pixel value X that establishes unambiguous image is an actual value.If having the pixel value Y of the image of defocusing blurring is the value that is observed.Represent by x that when the coordinate of along continuous straight runs image vertically the coordinate of image is represented by y, with a plurality of pixels of recognition image, this actual value can be represented as X (x, y), observed reading can be represented as Y (x, y).
According to the present invention, use following formula (6) as fuzzy model tormulation.In formula (6), use Gaussian function by following formula (7) expression.By utilize Gaussian function convolution actual value X (x, y), can obtain observed reading Y (x, y).
Y ( x , y ) = &Sigma; - r < j < r - r < i < r [ W ( i , j ) &times; X ( x + i , y + j ) ] &CenterDot; &CenterDot; &CenterDot; ( 6 )
W ( j , i ) = 1 2 &pi; &sigma; 2 e j 2 + i 2 - 2 &sigma; &CenterDot; &CenterDot; &CenterDot; ( 7 )
In formula (6), parameter σ represents fuzzy value.
According to formula (6), by usage factor W weighting according to variable i and j (r＜i＜r, a plurality of actual value X of-r＜j＜r) change (x+i, y+j), can obtain an observed reading Y (x, y).Thereby, can obtain not have the value of a pixel of fuzzy image according to the value of a plurality of pixels with fuzzy image.
In addition, fuzzy value changes according to above-mentioned parameter σ.When relative hour of the value of parameter σ, can not disperse with respect to observed reading about the information of actual value widely.Thereby, obtain to have less fuzzy image.On the contrary, when parameter σ is big relatively, disperse with respect to observed reading about the information of actual value widely.Thereby, obtain to have big relatively fuzzy image.
As mentioned above, fuzzy value changes according to above-mentioned parameter σ.Therefore, for accurately correcting image is fuzzy, the value of parameter σ need be determined suitably.According to the present invention, the user stipulates the value of parameter σ.Perhaps, set in advance optimum value by the feature of considering security camera system 1.
Illustrate in greater detail image blurring principle below with reference to Figure 86-89.Figure 86 A is expressed as for simplicity the figure as pixel actual value X0-X8 of a given image when a dimension is flatly arranged.Figure 86 C represents the figure corresponding to the observed reading of Figure 86 A.Figure 86 B represents the figure of the size of coefficient W (i) with the form of bar chart.In these examples, the scope of variable i is-2＜i＜2.Middle bar is represented coefficient W (0).Represent W (2) from the most left to the rightest bar, W (1), W (0), W (1), and W (2).
Wherein the observed reading Y2 that can obtain among Figure 86 C according to formula (6) is as follows:
Y2＝W(-2)X2+W(-1)X3+W(0)X4+W(1)X5+W(2)X6
Similarly, in order to obtain the observed reading Y0 among Figure 86 C,, can obtain following observed reading Y0 by carrying out calculating about the actual value among the frame 3790-1 shown in Figure 87:
Y0＝W(-2)X0+W(-1)X1+W(0)X2+W(1)X3+W(2)X4
In addition, in order to obtain observed reading Y1, by carrying out the calculating about the actual value in frame 3790-2 shown in Figure 87, it is as follows to obtain observed reading Y1:
Y1＝W(-2)X1+W(-1)X2+W(0)X3+W(1)X4+W(2)X5
In addition, observed reading Y3 and Y4 can be obtained in a like fashion.
Figure 88 and 89 is illustrated in relation between Figure 86 A and Figure 86 C with bidimensional.That is, the value of each pixel is an observed reading among Figure 84, and obtained as actual value by the value of using each pixel among Figure 89.In this case, can obtain corresponding to the pixel A shown in Figure 88 observed reading Y (x, y) as follows:
(Y(x，y)＝W(-2，-2)X(x-2，y-2)+W(-1，-2)X(x-1，y-2)
+W(0，.2)X(x，y-2)...+W(2，2)X(x+2，y+2)
Promptly can obtain the observed reading corresponding to the pixel A shown in Figure 88 according to the actual value corresponding to the individual pixel in 25 (=5 * 5), described 25 pixels are represented at the frame of pixel A ' (corresponding to pixel A) by its center shown in Figure 89.Similarly, can be according to obtaining the observed reading shown in Figure 88 at the actual value of 25 pixels of the pixel B shown in Figure 89 ' (corresponding to pixel B) corresponding to pixel B (pixel) on pixel A the right corresponding to its center.Can be according to obtaining the observed reading shown in Figure 88 at the actual value of 25 pixels of the pixel C ' shown in Figure 89 (corresponding to pixel C) corresponding to pixel C corresponding to its center.Can utilize following formula obtain to correspond respectively to pixel B shown in Figure 88 and C observed reading Y (x+1, y) and Y (x+2, y):
Y(x+1，y)＝W(-2，-2)X(x-1，y-2)+W(-1，-2)X(x，y-2)
+W(0，-2)X(x-1，y-2)...+W(2，2)X(x+3，y+2)
Y(x+2，y)＝W(-2，-2)X(x，y-2)+W(-1，-2)X(x+1，y-2)
+W(0，-2)X(x+2，y-2)...+W(2，2)X(x+4，y+2)
After being calculated, can obtain determinant of a matrix by following formula (8)-(11) expression corresponding to the observed reading of all pixels shown in Figure 88:
Y f = Y ( x , y ) Y ( x + 1 , y ) Y ( x + 2 , y ) Y ( x + 3 , y ) &CenterDot; &CenterDot; &CenterDot; Y ( x , y + 1 ) Y ( x + 1 , y + 1 ) &CenterDot; &CenterDot; &CenterDot; Y ( x + 7 , y + 7 ) &CenterDot; &CenterDot; &CenterDot; ( 8 )
W f = W ( - 2 , - 2 ) W ( - 1 , - 2 ) &CenterDot; &CenterDot; &CenterDot; W ( 2,2 ) W ( - 2 , - 2 ) W ( - 1 , - 2 ) &CenterDot; &CenterDot; &CenterDot; W ( 2,2 ) &CenterDot; &CenterDot; &CenterDot; &CenterDot; &CenterDot; &CenterDot; &CenterDot; &CenterDot; &CenterDot; &CenterDot; &CenterDot; &CenterDot; W ( - 2 , - 2 ) W ( - 1 , y - 2 ) &CenterDot; &CenterDot; &CenterDot; W ( 2,2 ) &CenterDot; &CenterDot; &CenterDot; ( 9 )
X f = X ( x - 2 , y - 2 ) X ( x - 1 , y - 2 ) &CenterDot; &CenterDot; &CenterDot; X ( x , y - 2 ) X ( x - 1 , y - 2 ) X ( x , y - 2 ) &CenterDot; &CenterDot; &CenterDot; X ( x + 1 , y - 2 ) &CenterDot; &CenterDot; &CenterDot; &CenterDot; &CenterDot; &CenterDot; &CenterDot; &CenterDot; &CenterDot; &CenterDot; &CenterDot; &CenterDot; X ( x + 2 , y + 2 ) X ( x + 3 , y + 2 ) &CenterDot; &CenterDot; &CenterDot; X ( x + 9 , y + 9 ) &CenterDot; &CenterDot; &CenterDot; ( 10 )
Y f＝W fX f ...(11)
Wherein, if can find the solution the inverse matrix of the matrix W f in the formula (11), then can be according to observed reading Y fObtain actual value X fThat is, can obtain the pixel of unambiguous image according to the pixel of blurred picture, thereby the image of blur correction mode.
But, as described in reference Figure 86-89, comprise the pixel of many actual values with respect to the pixel of observed reading by the determinant of a matrix of formula (8)-(11) expression.Therefore, be difficult to obtain inverse matrix and (for example in the example shown in Figure 87, need 5 pixels of actual value for a pixel of observed reading.
Thereby, remove outside formula (8)-(11), introduce relational expression by following formula (12) to (15) expression:
W a(p 1)W 1(p 2)(X(x，y)-X(x，y-1))＝0...(12)
W a(p 1)W 2(p 2)(X(x，y)-X(x+1，y))＝0...(13)
W a(p 1)W 3(p 2)(X(x，y)-X(x，y+1))＝0...(14)
W a(p 1)W 4(p 2)(X(x，y)-X(x-1，y))＝0...(15)
Formula (12)-(15) are provided with restriction to the difference of two adjacent pixel values.When the actual value that will obtain is in the flat part (value between the neighbor does not have significant different part) of image, there is not contradiction.But, when the actual value that will obtain is in the edge part (value between the neighbor has significantly different part) of image, then has contradiction.Thereby the image that is corrected may deterioration.For this reason,, need use one of 4 formula (12)-(15) suitably, make neighbor not stride across the edge part of actual value each pixel for blur correction mode image suitably.
Therefore, characteristics of image detecting unit 3742 is determined edge part and flat (flat) part of image, produces the code p2 that presentation video flattens light along which direction (for example horizontal direction or vertical direction).Describe the operation of image characteristic detection unit 3742 in detail below with reference to Figure 94.According to the present invention, suppose that edge part in input picture (observed reading) and definite result of flat part equal the edge part of actual value and definite result of flat part.
In formula (12)-(15), be weighting function as the function W1-W4 of the function of code p2.According to the present invention,, can and use a relational expression to each pixel selection by control these functions W1-W4 according to code p2.Figure 90 represents the value corresponding to the function W1-W4 of code p2.Along with the increase of the value of this weighting function, this part becomes more flat.On the contrary, along with reducing of the value of this weighting function, this part becomes more not flat (possibility that becomes the edge increases).
Code p2 constitutes by 4.These by leftmost position rise represent respectively an image whether along the top, right-hand, below and left be flat.If logical image is flat along a direction, then corresponding position is set to " 1 ".For example, code p2 is flat for " 0001 " presentation video from pixels of interest along left, but along other direction not flat (promptly having the edge).Therefore, when code p2 was " 0001 ", the value of weighting function W4 increased, and other the weighting of formula (12)-(14) of the weighted sum of formula (15) is compared has big value.Thereby code p2 can change the weighting of 4 relational expressions.Thereby, can select suitably and use one of 4 relational expressions for each pixel, make adjacent pixels discord edge intersect.
For example, shown in Figure 91, suppose image from pixels of interest along the top and left flat, and image has the edge along right-hand and below.By changing the weighting of 4 formula (12)-(15), restriction " Xa-Xb=0 " and " Xa-Xc=0 " is applied to poor between the adjacent pixel values.But, not application limitations " Xa-Xd=0 " and " Xa-Xe=0 ".Note Xb, Xc, Xd and Xe represent respectively along right-hand, below, top and left and pixels of interest X adjacent pixels.
In addition, in formula (12)-(15), function Wa is different weighting function.The value of function Wa also changes according to code p1.By changing the value of function Wa, can control the overall noise and the details of the image of correction.When the value of function Wa was big, the user felt to have very little noise effect in the image of proofreading and correct, and therefore, the feeling of noise reduces.On the contrary, when the value of function Wa hour, the user feels to have the details influence of increase in the image of proofreading and correct, and therefore, the sensation of details increases.The code p1 that should be noted that the value that changes function Wa is corresponding to the control signal C shown in Figure 85.
As mentioned above, remove outside formula (8)-(11), also introduce relational expression by formula (12)-(15) expression.Thereby, can solve inverse matrix by formula (16) expression.As a result, can obtain actual value according to observed reading.
X s＝W s -1Y s ...(16)
According to the present invention, the coefficient Ws-1 that multiply by observed reading Ys is stored among the coefficients R OM 3744 in advance.Calculate the determinant of a matrix of representing by formula (16) (inner product) by 3746 pairs of input pictures that extract by extracted region unit 3745 of inner product computing unit.Thereby, all carry out the calculating of inverse matrix when not needing each image to be corrected.Can only calculate blur correction mode by inner product.But, because parameter σ and 4 above-mentioned relational expressions change according to input picture, all calculate inverse matrix for each possible combination of parameter σ and above-mentioned 4 relational expressions.After this, definite address corresponding to parameter σ and code p2.Different coefficient about these addresses is stored among the coefficients R OM 3744.
But, for example, if the combination of weighting function W1-W4 all is changed for each of the individual pixel in 25 (=5 * 5) in the frame shown in Figure 89 (t), and changes 4 relational expressions, then Zu He quantity is 25 powers (pixel count in the frame (t)) of 15 (group of function W1-W4 and numbers).If all calculate inverse matrix for each combination, then the quantity of coefficient becomes big.Because the finite capacity of coefficients R OM 3744, coefficients R OM 3744 can not store all coefficients.In this case, the code p2 that is in the center of frame (t) only is changed pixel Xt, makes conversion relational expression.For the pixel outside the pixel Xt in the frame (t), code p2 can be fixed in a pseudo-value for example " 1111 ".Thereby, can be the restricted number of combination coefficient 15.
In the superincumbent explanation, for fuzzy principle (model tormulation) is described, the territory of Gaussian function is confirmed as-2≤(x, y)≤2.In fact, the territory of Gaussian function is determined like this, the feasible parameter σ that supports enough big value.In addition, if relational expression can be described the feature of image, then be not limited to relational expression suc as formula (12)-(15) expression.Therefore, under the situation of the coefficients R OM 3744 with limited capacity, these relational expressions only (Xt) are converted for fuzzy center situation (center phase).But, the invention is not restricted to this.The method that is used for conversion relational expression can change according to the capacity of coefficients R OM 3744.
Handle below with reference to the ambiguity correction that Figure 92 explanation is undertaken by image correction unit 3022.At step S1801, image correction unit 3022 detects wants processed zone.Wanting processed zone is the zone that will carry out ambiguity correction, and promptly correcting area 3052.Should the zone according to the input that unit 3025 outputs are set from the zone.
At step S1802, the get parms value of σ of image correction unit 22.The value of parameter σ can or be determined in advance by user's regulation.At step S1803, image correction unit 3022 is the carries out image treatment for correcting also, and it will describe with reference to Figure 93 below.Handle by this, fuzzy image is corrected and is output.
Thereby therefore image blurring being removed in correcting area 3052, can obtain clearly image.
Describe the image correction process shown in Figure 92 in detail below with reference to Figure 93 at step S1803.
At step S1821, the 3742 carries out image feature extractions of characteristics of image detecting unit are handled, and it describes with reference to Figure 94 below.Thereby, determine with respect to pixels of interest which directional image to be flat along.The code p2 that generation illustrates with reference to Figure 90, and be output to address calculation 3743.
In step S1822, the address of address calculation 3743 design factor ROM 3744.For example, the address of coefficients R OM 3744 is by 4 formations corresponding to code p2 (output of characteristics of image detecting unit 3742), the value of described 4 bit representation parameter σ (the control signal B shown in Figure 85), 2 code p1 (the control signal C shown in Figure 85) corresponding to the weighting function Wa that is used to change 4 above-mentioned relational expressions.This address has 1024 values (2 of from 0 to 1023 10).Address calculation 3743 is calculated corresponding address according to output, control signal B and the control signal C of characteristics of image detecting unit 3742.
At step S1823, address calculation 3743 is read coefficient according to the address of calculating at step S1822 from coefficients R OM 3744, and the coefficient of reading is sent to inner product computing unit 3746.
At step S1824, inner product computing unit 3746 calculates inner product according to the coefficient of reading at step S1823 to each pixel, and to image assembled unit 3747 output inner product result of calculations.Thereby, as mentioned above, can obtain actual value by observed reading, therefore, image that can blur correction mode.
At step S1825, image assembled unit 3747 carries out image combined treatment, it describes with reference to Figure 97 below.Thereby, determine that the result of output inner product computing unit 3746 still is directly to export input picture for each pixel.At step S1826, post-processing unit 3747 output calibrations and image that select.
Detect processing below with reference to Figure 94 explanation at the characteristics of image that the step S1821 shown in Figure 93 carries out.At step S1841, characteristics of image detecting unit 3742 extracts piece.At step S1842, characteristics of image detecting unit 3742 calculates poor (its details describes with reference to Figure 96 below) between the piece that step S1841 extracts.At step S1843, piece difference and predetermined threshold value that characteristics of image detecting unit 3742 relatively calculates at step S1842.At step S1844, characteristics of image detecting unit 3742 is according to comparative result output code p2, and its expression is flat direction with respect to the pixels of interest image.
Illustrate in greater detail characteristics of image below with reference to Figure 95 and 96 and detect processing.Figure 95 is the calcspar of the detailed structure of presentation video characteristic detection unit 3742.In the left side of figure, provide piece cut cells 3841-1 to 3841-5.For example, shown in Figure 96 A-96E, piece cut cells 3841-1 extracts 5 pieces to 3841-5, and each comprises the individual pixel in 9 (=3 * 3), and one of them pixel is the pixels of interest (pixel that will be corrected at this moment) by black circle expression.
Piece 3881 shown in Figure 96 A is that its center is the intermediate mass of pixels of interest.Piece 3881 is extracted by piece cut cells 3841-5.Piece 3882 shown in Figure 96 B is by a top block that pixel obtains that piece 3881 is moved up.Piece 3882 is extracted by piece cut cells 3841-3.Piece 3883 shown in Figure 96 C is left pieces, and it is obtained by piece 3881 being moved to the left a pixel.Piece 3883 is extracted by piece cut cells 3841-4.
Piece 3884 shown in Figure 96 D is bottom biock, and it obtains by piece 3881 being moved down a pixel.Piece 3884 is extracted by piece cut cells 3841-1.Piece 3885 shown in Figure 96 E is right pieces, and it is by moving right piece 3881 pixel and obtained.Piece 3885 is extracted by piece cut cells 3841-2.At step S1841, extract 5 piece 3881-3885 for each pixels of interest.
Information about the pixel of each piece of being extracted to 3841-5 by piece cut cells 3841-1 is output to piece difference computing unit 3842-1 to 3842-4.For example, piece difference computing unit 3842-1 is to 3842-4 poor according between the pixel in each piece of following calculating.
In the middle of 9 pixels of piece 3881,3 pixels (value of pixel) in the most up are represented as a (3881) from the most left pixel, b (3881) and c (3881).3 pixels at the centre row are represented as d (3881) from the most left pixel, e (3881) and f (3881).3 pixels in the most descending are represented as g (3881) from the most left pixel, h (3881) and i (3881).Similarly, in the middle of 9 pixels of piece 3884,3 pixels (value of pixel) in the most up are represented as a (3884) from the most left pixel, b (3884) and c (3884).3 pixels at the centre row are represented as d (3884) from the most left pixel, e (3884) and f (3884).3 pixels in the most descending are represented as g (3884) from the most left pixel, h (3884) and i (3884).Piece difference computing unit 3842-1 is according to following computing block difference B (1):
B(1)＝|a(3881)-a(3884)|+|b(3881)-b(3884)|+|c(3881)-c(3884)|+...+|i(3881)-i(3884)|
That is, piece difference B (1) is in the pixel value of piece 3881 (centre) and the absolute difference sum between the respective pixel value in the piece 3884 (bottom).Similarly, piece difference computing unit 3842-2 calculates in the pixel value of piece 3881 (centre) and the absolute difference sum between the respective pixel value in the piece 3885 (right-hand), thereby obtains piece difference B (2).In addition, piece difference computing unit 3842-3 calculates in the pixel value of piece 3881 (centre) and the absolute difference sum between the respective pixel value in the piece 3882 (top), thereby obtains piece difference B (3).Piece difference computing unit 3842-3 calculates in the pixel value of piece 3881 (centre) and the absolute difference sum between the respective pixel value in the piece 3883 (left), thereby obtains piece difference B (4).
At step S1842, as mentioned above, computing block difference B (1)-B (4), they are intermediate mass and poor along between each piece of level and vertical 4 directions.Its result is output to corresponding threshold determining unit 3843-1 to 3843-4 and minimum direction determining unit 844.
Threshold value determining unit 3843-1 is to 3843-4 difference comparison block difference B (1) to B (4) and predetermined threshold value.Should be noted that these threshold values are converted according to control signal D.Respectively greater than predetermined threshold, then threshold value determining unit 3843-1 determines just that to 3843-4 this direction is the edge part to if block difference B (1) to B (4), and therefore, threshold value determining unit 3843-1 is to 3843-4 output " 0 ".Respectively less than predetermined threshold, then threshold value determining unit 3843-1 determines just that to 3843-4 this direction is flat part to if block difference B (1) to B (4), and therefore, threshold value determining unit 3843-1 is to 3843-4 output " 1 ".
At step S1843, make piece difference and threshold, as mentioned above.Threshold value determining unit 3843-1 is output to selector switch 845 to the output result of 3843-4 with the form of 4 codes.For example, if each piece difference B (1), B (3) and B (4) all less than threshold value, and piece difference B (2) is greater than threshold value, then output code " 1011 ".
In some cases, all piece difference B (1)-B (4) is greater than threshold value (being that image does not have flat part).In this case, from threshold value determining unit 3843-1 to 3843-4 output code " 0000 ".But, shown in Figure 90, when code p2 is " 0000 ", just corresponding weighting function W1-W4 can not be identified.Therefore, selector switch 3845 determines whether threshold value determining unit 3843-1 is " 0000 " to the output result of 3843-4.If selector switch 3845 determines that threshold value determining unit 3843-1 is " 0000 " to the output result of 3843-4, then selector switch is as the output of code p2 output from minimum direction determining unit 3844.
Minimum direction determining unit 3844 is determined the minimum value in the middle of piece difference B (1)-B (4), and exports corresponding to 4 codes determining the result to selector switch 3845 in the 3843-4 output code at threshold value determining unit 3843-1.For example, be minimum among piece difference B (1)-B (4) if determine piece difference B (1), then minimum direction determining unit 3844 is to selector switch 3845 output codes " 1000 ".
Even this design makes as threshold value determining unit 3843-1 during to 3843-4 output code " 0000 ", also allows as code p2 from minimum direction determining unit 3844 output codes " 1000 ".When threshold value determining unit 3843-1 when the output result of 3843-4 is not " 0000 ", as the output result of code p2 output threshold value determining unit 3843-1 to 3843-4.At step S3844, so just produce code p2 and be output to address calculation 743.
The image combined treatment of step S1825 shown in Figure 93 is described below with reference to Figure 97.At step S1861, image assembled unit 3747 is according to from the output of inner product computing unit 3746 degree of scatter of calculating pixel as a result.Thereby, can calculate the pixel degree of scatter around pixels of interest.At step S1862, whether the degree of scatter that image assembled unit 3747 is determined to calculate at step S1862 is greater than predetermined threshold.
If S1862 determines degree of scatter greater than threshold value in the step, then image assembled unit 3747 is set to " opening " (ON) at step S1863 input picture transformational marker.On the contrary, if determine that degree of scatter is not more than threshold value, then image assembled unit 3747 is set to " pass " (OFF) at step S1864 input picture transformational marker.
Carry out inner product calculating if the interior pixel of regional area of fuzzy input picture does not take place 3746 pairs of inner product computing units, then can be increased in the activity of this pixel image on every side, therefore, it is bad that picture quality may become.Wherein, if degree of scatter, determines then that this pixel is the pixel that becomes bad greater than predetermined threshold, and the input picture transformational marker is set to " opening ".When this pixel was output, the pixel that the pixel that its input picture transformational marker is set to " opening " is transfused to image replaced (being that pixel turns back to original pixels).
At step S1865, image assembled unit 3747 determines whether that all pixels all were examined.If determine not check out as yet all pixels, then handle and return step S1861, and repeat the processing after the step S1861.If S1865 determines to have checked all pixels in the step, then being combined at step S1866 image assembled unit 3747 not have the fuzzy image and an image of background 3053 in the correcting area 3052, and the image that makes up to image display 3023 outputs.
Thereby, determine that for each pixel output inner product result calculated still is directly to export the pixel of input picture.This design can make there is not the fuzzy picture degradation that stops in input picture by proofreading and correct topography.
Below with reference to Figure 98 and this phenomenon of 99 more detailed descriptions.Figure 98 is the exemplary configurations of image assembled unit 3747.The output result of inner product computing unit 3746 is inputs of piece cut cells 3091.Shown in Figure 99, it is the individual pixel a1-a9 in 9 (=3 * 3) of pixels of interest a5 that piece cut cells 3901 is sheared its center, and these pixels are outputed to dispersion computing unit 3802.Disperse computing unit 3802 according to following calculating degree of scatter:
v = &Sigma; * = 1 9 ( a * - m ) 2 &CenterDot; &CenterDot; &CenterDot; ( 17 )
Wherein m represents 9 pixels (pixel value) average in the piece, v represent each pixel and average between the variance sum, that is, and the degree of scatter of the pixel in piece.At step S1861, calculate degree of scatter like this, and result of calculation is outputed to threshold value determining unit 3903.
Threshold value determining unit 3903 is relatively disperseed the output result (degree of scatter) and the predetermined threshold of computing unit 3902.If determine degree of scatter greater than threshold value, then image assembled unit 3747 control selected cells 3904 are set to " opening " corresponding to the input picture transformational marker of pixels of interest.If determine that degree of scatter is not more than threshold value, then image assembled unit 3747 control selected cells 3904 are set to " pass " corresponding to the input picture transformational marker of pixels of interest.At step S1862-S1864, determine that so whether degree of scatter is greater than threshold value.According to definite result the input picture transformational marker is set.
Then, converting unit 3905 is changed between the pixel of the last result of selected cell 3904 and input picture.Converting unit 3905 outputs are selected one then.That is, the pixel of the image in correcting area 3052 is represented the last result of selected cell 3904, and the pixel of the image of background 3053 is represented the pixel of input picture.Image just is converted like this.
Thereby (Figure 82) is tracked for object 3051.Only the image in the correcting area 3052 that comprises object 3051 is updated (correction), so the fuzzy of image be removed, thereby shows clearly image.On the contrary, because the image of background 3053 just is shown without removing deblurring, the user can be automatically and the object of observation 3051 carefully.
In the explanation in front, image correction unit 3022 is proofreaied and correct the image in the correcting area 3052 of the image of being caught by image capturing unit 3021, makes and removes the fuzzy of image.But, image correction unit 3022 can be proofreaied and correct the image in the correcting area 3052 and do not removed the fuzzy of image, makes the brightness of each pixel in this zone and color setting be changed, and the image in this zone is by highlight simply.According to this design, though the user can not accurately watch object 3051, can be automatically and the object of observation 3051 carefully.In addition, and remove the fuzzy of image and compare, the configuration of image correction unit 3022 can be simplified.As a result, can realize object tracking equipment 1 with low cost.
Above-mentioned a series of processing not only can but also can be carried out by software by hardware.When above-mentioned a series of processing were carried out by software, software program is downloaded to from network or recording medium in the computing machine that is comprised in the specialized hardware maybe can be by installing in the computing machine that various programs carry out various functions (for example general purpose personal computer) therein.
In this explanation, be used for illustrating that the step in the recording medium program stored not only comprises the processing of carrying out with above-mentioned order, and comprise by concurrently or the processing of carrying out independently.
Label
1 object tracking device, 11 templates coupling unit, the unit is estimated in 12 motions, 13 scenes change detecting unit, and 14 background motions are estimated the unit, and relevant processing unit is estimated in 15 zones, 16 shift the candidate memory cell, 17 trace point determining units, 18 template memory cell, 19 control modules

Claims (45)

1. image processing equipment comprises:
Position estimation device is used for second the position that estimated statement is shown in the trace point of the image of next processing unit on the time, described second first point corresponding to the trace point in the image that is illustrated in last previous processing unit of time;
Generation device is used for when described second position can not be estimated, generation is as the point of the estimation of first candidate point;
Determine device, be used in the time can estimating, determine second point in next processing unit according to the estimated result of position estimation device in second position of next processing unit; And
Selecting arrangement is used for selecting first point according to the order that increases distance from original trace point from the point of estimating when second position can not be estimated.
2. image processing equipment as claimed in claim 1, wherein said processing unit is a frame.
3. image processing equipment as claimed in claim 1, wherein said position estimation device be the precision estimated of calculating location also, and if the precision of calculating greater than reference value, then position estimation device determines that second position is estimable.
4. image processing equipment as claimed in claim 1, if second position wherein in next processing unit is imponderable, then position estimation device is according to the position of second of first point estimation of being selected by selecting arrangement.
5. image processing equipment as claimed in claim 1, if wherein second position is estimable, then position estimation device thinks that second position is the first new point, and estimates the position of the trace point in the image of next processing unit.
6. image processing equipment as claimed in claim 1, wherein generation device comprises regional estimation unit, be used for estimating that more at least group is at previous processing unit or than the previous processing unit target area in the processing unit of front more, described group belongs to and comprises first object, and the estimation point generation device, be used for producing the point of estimating according to the target area.
7. image processing equipment as claimed in claim 6, wherein regional estimation unit finds at least by prediction and as the target area position overlapped of wanting estimative object, determine regional estimation range at the future position of the trace point that comprises the processing unit that is used for the estimating target zone, in the regional estimation range of determining, sampling spot is set, and estimates that by having same movement in the sampling spot and having the zone that maximum sized one group of sampling spot constitutes be the target area.
8. image processing equipment as claimed in claim 7, wherein the shape of regional estimation range is fixed.
9. image processing equipment as claimed in claim 7, wherein the shape of regional estimation range is variable.
10. image processing equipment as claimed in claim 7, wherein regional estimation unit estimates than the previous processing unit target area in the processing unit of front more, and wherein generation device is created in point in the target area of more estimating in the processing unit of front than previous processing unit as the point of estimating.
11. image processing equipment as claimed in claim 7, wherein regional estimation unit is estimated the target area in previous processing unit, and wherein generation device produces the point of formation target area as the point of estimating.
12. image processing equipment as claimed in claim 6, wherein regional estimation unit estimate adjacent with first and have the pixel value similar with first pixel value point and with first adjacent point adjacent point be the target area.
13. image processing equipment as claimed in claim 6, wherein regional estimation unit has preliminary dimension and is more comprising in the processing unit of front than previous processing unit in first the zone and extract sampling spot, and wherein regional estimation unit estimate to comprise in the previous processing unit by the regional mobile phase that will have same movement and have a maximum sized sampling spot with the zone of the point that obtains of amount of movement be the target area.
14. image processing equipment as claimed in claim 6, wherein image processing equipment also comprises:
The template generation device is used to produce template; And
The correlation calculations device, be used for when determining at second, calculate than the processing unit of the piece correlativity between the piece of the presumptive area of the piece of presumptive area and representation template in the next processing unit of expression in the processing unit of one or more processing unit of front more according to the point of estimating;
Wherein when determining that according to the correlativity of calculating by the correlation calculations device correlativity is high, determine that by using at least device detects trace point.
15. image processing equipment as claimed in claim 14, wherein the template generation device determines that the presumptive area around the trace point is a template.
16. image processing equipment as claimed in claim 14, wherein the template generation device produces template according to the target area.
17. image processing equipment as claimed in claim 14, wherein when determining that according to the correlativity of calculating by the correlation calculations device correlativity is high, according to than the piece of the presumptive area of expression in the next the processing unit more piece of the presumptive area of representation template and the relation between the trace point and according to position in the processing unit of one or more processing unit of front, determine second point with the piece that is defined as high correlativity.
18. image processing equipment as claimed in claim 14, wherein the template generation device determines that the zone and the presumptive area around sampling spot that are made of the sampling spot in the target area are template.
19. image processing equipment as claimed in claim 14, wherein the correlation calculations device by calculate in next processing unit piece and than the processing unit of this piece more the error between the piece of the template in the processing unit of one or more processing unit of front determine correlativity.
20. image processing equipment as claimed in claim 1 also comprises:
Be used to detect the pick-up unit of scene change;
Wherein when position estimation device and selecting arrangement can not select at second in the middle of the point of estimating, position estimation device and selecting arrangement stop their processing according to predetermined condition, and change this condition according to the appearance of scene change.
21. image processing equipment as claimed in claim 1 determines that wherein device also comprises:
The estimated value calculation element, be used to calculate the estimated value of the correlativity between expression pixels of interest and the respective pixel, described pixels of interest represents that the time of being included in goes up at least one pixel of first in the previous processing unit, and described respective pixel is illustrated in the time and goes up at least one pixel in the next processing unit and be determined according to the motion vector of pixels of interest;
The variable value calculation element is used to calculate the variable value of expression with respect to the change of the pixel value of pixels of interest; And
The accuracy computation device is used for the precision of calculating kinematical vector.
22. image processing equipment as claimed in claim 21, wherein the quantity of pixels of interest equals the quantity of respective pixel.
23. image processing equipment as claimed in claim 21, wherein said variable value indication is along the change of direction in space pixel value.
24. image processing equipment as claimed in claim 21, one of wherein said variable value indication degree of scatter and dynamic range.
25. image processing equipment as claimed in claim 21, wherein said processing unit are one of frame and field.
26. image processing equipment as claimed in claim 21, wherein accuracy computation device basis is for the precision of variable value by the normalized value calculating kinematical vector of estimated value.
27. image processing equipment as claimed in claim 21, wherein when variable value during greater than predetermined threshold, it is the precision of motion vector by the normalized value of estimated value that the accuracy computation device is determined with respect to variable value, and wherein when variable value during less than predetermined threshold, the accuracy computation device is determined the low fixed value of precision of indication motion vector.
28. image processing equipment as claimed in claim 21, wherein estimated value calculation element reckoner be shown in the pixel in the piece that comprises pixels of interest and comprise absolute difference between the pixel in the piece of respective pixel and estimated value.
29. image processing equipment as claimed in claim 21, wherein the variable value calculation element calculates variable value, this variable value represent by remove with the quantity of neighbor absolute difference and value that obtain between in comprising the piece of pixels of interest pixels of interest and the neighbor adjacent with pixels of interest with.
30. image processing equipment as claimed in claim 21, wherein the accuracy computation device comprises:
Comparison means is used for comparison variable value and first reference value;
The difference calculation element, be used to calculate second reference value and for variable value by the difference between the normalized value of estimated value; And
Output unit is used for the precision of the difference calculating kinematical vector that calculates according to the comparative result of comparison means with by the difference calculation element and the precision of output movement vector.
31. image processing equipment as claimed in claim 21 also comprises:
Device for detecting motion vector is used for detecting motion vector from input picture, and motion vector is offered the estimated value calculation element;
Motion compensator unit is used for the motion vector motion compensation input picture that detects according to device for detecting motion vector;
Be used for according to the image of the accuracy selection passive movement compensation system motion compensation of motion vector and do not have the device of one of the image of passive movement compensation; And
Code device is used to encode by the image of described accuracy selection passive movement compensation system motion compensation according to motion vector and the image that do not have the device of one of the image of passive movement compensation to select.
32. image processing equipment as claimed in claim 21 also comprises:
The frequency distribution calculation element is used to calculate the frequency distribution with the weighting of motion vector precision; And
The maximal value pick-up unit is used to detect the maximal value by the frequency distribution of frequency distribution calculation element calculating, and according to the maximal value detection background motion that detects.
33. image processing equipment as claimed in claim 21 also comprises:
Average computing device is used for calculating the mean value in the motion vector precision of processing unit; And
Be used for mean value and reference value that comparison is calculated by average computing device, and determine the device of the appearance of scene change according to comparative result.
34. image processing equipment as claimed in claim 33, wherein average computing device calculates one on average to a processing unit.
35. image processing equipment as claimed in claim 1 also comprises:
First point detection device is used for detecting first point at image motion object;
The correcting area setting device is used for being arranged on the correcting area that has preliminary dimension around the object that image will follow the tracks of according to estimated result;
Means for correcting is used for the image in the correcting area of correcting image, and
Display control unit is used for controlling the image that comprises by the image of the correcting area of correction and shows.
36. image processing equipment as claimed in claim 35, wherein the correction image is fuzzy.
37. image processing equipment as claimed in claim 36, wherein means for correcting comprises:
Conveyer is used for transmitting the control signal of the image that is used to discern correcting area and is used for the parameter of the fuzzy grade of indicating image;
The feature detection device is used to detect the feature according to the image in the correcting area of control signal identification, and the condition code of the feature of output expression detection;
Memory storage is used for the parameter of the image blurring grade of storage representation and corresponding to the coefficient by the condition code of feature detection device output;
Readout device is used for reading described parameter and corresponding to the coefficient by the condition code of feature detection device output from memory storage;
The inner product calculation element is used for the inner product according to the pixel value of the coefficient calculations input picture of being read by readout device; And
Select output unit, be used to select the result of calculation of inner product calculation element and input picture pixel value export selected one in the lump;
Wherein the image in correcting area is corrected to remove the fuzzy of image.
38. image processing equipment as claimed in claim 37, wherein first point detection device comprises:
First extraction element is used for will carrying out pixel a plurality of pixels on every side that inner product is calculated in the predetermined first area extraction of input picture;
Second extraction element, be used for along a plurality of vertical and horizontal directions with a plurality of pixels of each zone extraction of a plurality of second areas of first area adjacency;
Piece difference calculation element, be used for by the absolute difference between the respective pixel value of calculating the pixel value that extracts by first extraction element and extracting by second extraction element with calculate a plurality of differences; And
Difference is determined device, is used for determining that whether the piece difference is greater than predetermined threshold.
39. image processing equipment as claimed in claim 37, wherein said parameter are the pixel of expression blurred picture and the parameter of the Gaussian function in the model tormulation formula of the relation between the pixel of blurred picture not.
40. image processing equipment as claimed in claim 39 is the coefficient that the inverse matrix by the computation model expression formula obtains by the coefficient of memory device stores wherein.
41. image processing equipment as claimed in claim 37 wherein selects output unit to comprise:
Be used to extract the device that carries out a plurality of pixels of inner product calculating by the inner product calculation element;
Disperse calculation element, be used to calculate the dispersion grade of a plurality of pixels that the device of a plurality of pixels that inner product calculates extracts is carried out in expression by the inner product calculation element by described extraction dispersion degree; And
Disperse to determine device, be used for whether determining by the dispersion degree that disperses calculation element to calculate greater than predetermined threshold.
42. image processing equipment as claimed in claim 41, wherein select output unit also to comprise the pixel selection device, be used for according to disperseing the definite result who determines device to select one of the result of calculation of inner product calculation element and pixel value of input picture output valve as pixel.
43. an image processing method comprises:
Estimating step is used for second the position that estimated statement is shown in the trace point of the image of next processing unit on the time, described second first point corresponding to the trace point in the image that is illustrated in last previous processing unit of time;
Produce step, be used for when described second position can not be estimated, generation is as the point of the estimation of first candidate point;
Determining step is used in the time can estimating in second position of next processing unit, determines second point in next processing unit according to the estimated result of estimating step; And
Select step, be used for when second position can not be estimated, selecting first point from the point of estimating according to the order that increases distance from original trace point.
44. image processing method as claimed in claim 43, wherein determining step comprises:
The estimated value calculation procedure, be used to calculate the estimated value of the correlativity between expression pixels of interest and the respective pixel, described pixels of interest represents that the time of being included in goes up at least one pixel of first in the previous processing unit, and described respective pixel is illustrated in the time and goes up at least one pixel in the next processing unit and be determined according to the motion vector of pixels of interest;
The variable value calculation procedure is used to calculate the variable value of expression with respect to the change of the pixel value of pixels of interest; And
The accuracy computation step is used for the precision of calculating kinematical vector.
45. the image processing method as claim 43 also comprises:
First is detected step, is used for detecting first point at image motion object;
Correcting area is provided with step, is used for being arranged on the correcting area that has preliminary dimension around the object of image according to estimated result;
Aligning step is used for the image in the correcting area of correcting image; And
Show controlled step, the image that is used for controlling the image that comprises the correcting area of being proofreaied and correct by aligning step shows.
CNB2005800020184A 2004-01-06 2005-01-06 Image processing equipment and method Expired - Fee Related CN100541535C (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
JP2004000752 2004-01-06
JP000752/2004 2004-01-06
JP077398/2004 2004-03-18
JP077399/2004 2004-03-18

Related Child Applications (1)

Application Number Title Priority Date Filing Date
CN2008101610536A Division CN101393642B (en) 2004-01-06 2005-01-06 Image processing device and method

Publications (2)

Publication Number Publication Date
CN1910617A CN1910617A (en) 2007-02-07
CN100541535C true CN100541535C (en) 2009-09-16

Family

ID=37700853

Family Applications (2)

Application Number Title Priority Date Filing Date
CN2008101610536A Expired - Fee Related CN101393642B (en) 2004-01-06 2005-01-06 Image processing device and method
CNB2005800020184A Expired - Fee Related CN100541535C (en) 2004-01-06 2005-01-06 Image processing equipment and method

Family Applications Before (1)

Application Number Title Priority Date Filing Date
CN2008101610536A Expired - Fee Related CN101393642B (en) 2004-01-06 2005-01-06 Image processing device and method

Country Status (1)

Country Link
CN (2) CN101393642B (en)

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4881766B2 (en) * 2007-03-06 2012-02-22 パナソニック株式会社 Inter-camera link relation information generation device
JP4623122B2 (en) * 2008-04-03 2011-02-02 ソニー株式会社 Image signal processing apparatus, image signal processing method, and program
JP2010011441A (en) * 2008-05-26 2010-01-14 Sanyo Electric Co Ltd Imaging apparatus and image playback device
JP4513039B2 (en) * 2008-05-30 2010-07-28 ソニー株式会社 Image processing apparatus, image processing method, and program
JP4507129B2 (en) * 2008-06-06 2010-07-21 ソニー株式会社 Tracking point detection apparatus and method, program, and recording medium
JP4760918B2 (en) * 2009-01-23 2011-08-31 カシオ計算機株式会社 Imaging apparatus, subject tracking method, and program
JP2011257502A (en) * 2010-06-07 2011-12-22 Sony Corp Image stabilizer, image stabilization method, and program
EP3364655A4 (en) * 2015-11-11 2018-10-10 Samsung Electronics Co., Ltd. Method and apparatus for decoding video, and method and apparatus for encoding video
JP6961363B2 (en) * 2017-03-06 2021-11-05 キヤノン株式会社 Information processing system, information processing method and program

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1402551A (en) * 2001-08-07 2003-03-12 三星电子株式会社 Apparatus and method for automatically tracking mobile object

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3434979B2 (en) * 1996-07-23 2003-08-11 富士通株式会社 Local area image tracking device
US6067367A (en) * 1996-10-31 2000-05-23 Yamatake-Honeywell Co., Ltd. Moving direction measuring device and tracking apparatus

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1402551A (en) * 2001-08-07 2003-03-12 三星电子株式会社 Apparatus and method for automatically tracking mobile object

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
Three Dimensional Reconstruction of Room EnviromentbyCorrecting EPI from Image Sequence Taken withhandy Camara. Masao NAKAGAWA,Hideo SAITO,Shinji OZAWA.The Transactions of the Instisute of Electronics,Information and communication Engineers,Vol.J84-D-II No.2. 2001
Three Dimensional Reconstruction of Room EnviromentbyCorrecting EPI from Image Sequence Taken withhandy Camara. Masao NAKAGAWA,Hideo SAITO,Shinji OZAWA.The Transactions of the Instisute of Electronics,Information and communication Engineers,Vol.J84-D-II No.2. 2001 *
一种人头部实时跟踪方法. 徐一华，朱玉文，贾云得.中国图象图形学报，第7卷第1期. 2002
基于2-D模型的人体运动跟踪. 孙怡，王恩亮，安宏杰，胡家升.中国图象图形学报，第7(A)卷第7期. 2002

Also Published As

Publication number Publication date
CN1910617A (en) 2007-02-07
CN101393642B (en) 2011-05-18
CN101393642A (en) 2009-03-25

Similar Documents

Publication Publication Date Title
CN100541535C (en) Image processing equipment and method
EP1710751A1 (en) Image processing device and method, recording medium, and program
US20130335528A1 (en) Imaging device capable of producing three dimensional representations and methods of use
US8781162B2 (en) Method and system for head tracking and pose estimation
CN102184546B (en) Image processing apparatus, image processing method and image processing program
CN101354254B (en) Method for tracking aircraft course
CN103514432A (en) Method, device and computer program product for extracting facial features
CN101894375A (en) Person tracking method and person tracking apparatus
Sun et al. Bottom-up human pose estimation by ranking heatmap-guided adaptive keypoint estimates
CN114399829B (en) Posture migration method based on generative countermeasure network, electronic device and medium
CN114445853A (en) Visual gesture recognition system recognition method
WO2020213099A1 (en) Object detection/tracking device, method, and program recording medium
CN104683803A (en) Moving object detecting and tracking method applied to compressed domain
Zhang Innovation of English teaching model based on machine learning neural network and image super resolution
Jiang et al. A Probabilistic Attention Model with Occlusion-aware Texture Regression for 3D Hand Reconstruction from a Single RGB Image
CN113297963A (en) Multi-person posture estimation method and device, electronic equipment and readable storage medium
JP5703898B2 (en) Form management system, form image management method, and program
CN116797545A (en) Scoliosis detection method, computer equipment and storage medium
Chen et al. Repparser: end-to-end multiple human parsing with representative parts
Posl et al. Wavelet features for statistical object localization without segmentation
KR101373397B1 (en) Sampling method for random sample consensus based on constraint satisfaction problem in augmented reality
Mu Pose Estimation-Assisted Dance Tracking System Based on Convolutional Neural Network
JP4737576B2 (en) Stroke extraction device, stroke extraction method, and recording medium recording stroke extraction program
Essich et al. Learning to translate between real world and simulated 3d sensors while transferring task models
WO2022102236A1 (en) Information processing device, information processing method, and program

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant
C17 Cessation of patent right
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20090916

Termination date: 20140106