JP4725105B2 - Image processing apparatus and method, program, and recording medium - Google Patents

Description

  The present invention relates to an image processing apparatus and method, a program, and a recording medium, and in particular, makes it possible to easily display an object that moves in an image and to ensure a desired point in a moving image that changes from moment to moment. The present invention relates to an image processing apparatus and method, a program, and a recording medium that can be tracked.

  In the home security system, a monitoring image transmitted from the imaging device is displayed on a monitor TV (Television). In such a system, a method has been proposed in which a monitoring device is configured by combining a microwave sensor and an image sensor to improve the accuracy of intruder monitoring (see, for example, Patent Document 1).

  In addition, a method has been proposed in which a moving (moving) object is a tracking target in an image displayed as a moving image, and the tracking point is automatically tracked to display an image (for example, Patent Documents). 2).

Japanese Patent Application Laid-Open No. 2004-228620 proposes that a region related to a tracking target is estimated and the region is tracked based on a region motion estimation result.
Japanese Patent Laid-Open No. 11-161860 JP-A-6-143235 JP-A-5-304065

  However, in the conventional technology, even if an object (object) that moves in an image, such as an unauthorized intruder, is detected and tracked, which part of the image displayed to the user is There is a problem in that it is not possible to clearly display whether the object is a notable object, and as a result, it may take time for the user to find the notable object.

  Moreover, in the technique described in Patent Document 2, since tracking is performed using one motion vector, there is a problem of relatively lacking robustness. In addition, when the image including the tracking target is rotated so that the tracking target becomes invisible to the user, even if the tracking point becomes visible again, the tracking point is no longer displayed. There was a problem that could not be tracked.

  In the technique of Patent Document 3, a region is used. As a result, robustness is improved. However, if the area is too wide to improve robustness, for example, if you want to zoom and display the child's face in the image taken with the home video, There was a problem that a symptom of tracking and zoom display occurred.

  In any technique, robust tracking is performed when the tracking target is temporarily occluded by another object or when the tracking target is temporarily not displayed due to a scene change or the like. There was a problem that was difficult to do.

  The present invention has been made in view of such a situation, and makes it possible to easily display a moving object in an image and display it. In addition, the tracking point can be reliably tracked even when an object rotates, occlusion occurs, or a scene change occurs.

An image processing apparatus according to the present invention is an image processing apparatus that clearly displays a moving object, and is detected by a feature point detecting unit that detects a feature point of the moving object in the image, and the feature point detecting unit. A tracking unit that tracks the movement of an object in the image based on the feature points, and a correction that sets a correction area of a preset size around the object in the image based on the tracking result by the tracking unit An area setting unit, a correction unit that corrects an image in a correction area in the image, and a display control unit that controls display of an image in which the correction area is corrected by the correction unit . The correction means includes a control signal for specifying an image in the correction area, a supply means for supplying a parameter indicating a degree of blur of the image, and an image in the correction area specified based on the control signal. A feature detection unit that detects a feature and outputs a feature code representing the detected feature; a memory that stores a parameter that represents the degree of blur of the image; and a coefficient that corresponds to the feature code output by the feature detection unit Means for reading out the parameter and the coefficient corresponding to the feature code output by the feature detection means from the storage means, and the pixel of the input image based on the coefficient read by the readout means A product-sum operation means for performing a product-sum operation on the value, and a selection output for selecting and outputting the operation result by the product-sum operation means and the pixel value of the input image. And means to correct so as to eliminate image blurring in the correction area.

  The feature detection means includes: a first extraction means for extracting a plurality of pixels included in a preset first area around a pixel on which a product-sum operation is performed; A second extraction means for extracting a plurality of pixels included in a plurality of second regions, which are continuous in a plurality of vertical or horizontal directions, a pixel extracted by the first extraction means, a second In the pixel extracted by the extraction unit, a block difference calculation unit for calculating a plurality of block differences by calculating a sum of absolute values of corresponding pixel values, and whether the block difference is larger than a preset threshold value. Difference determining means for determining whether or not there may be provided.

  The parameter may be a parameter of a Gaussian function in a model expression representing a relationship between pixels of a blurred image and pixels of a non-blurred image.

  The coefficient stored by the storage means can be a coefficient obtained by calculating an inverse matrix of a model formula.

  The selection output unit includes a first extraction unit that extracts a plurality of pixels that have been subjected to the product-sum operation by the product-sum operation unit, and a variance that represents a degree of dispersion of the plurality of pixels extracted by the first extraction unit. Dispersion calculating means for calculating the degree and dispersion determining means for determining whether or not the degree of dispersion calculated by the dispersion calculating means is greater than a preset threshold value can be provided.

  The selection output unit further includes a pixel selection unit that selects, based on the determination result of the dispersion determination unit, a pixel value to be output from either a calculation result of the product-sum calculation unit or a pixel value of the input image. Can be.

An image processing method of the present invention is an image processing method of an image processing apparatus that clearly displays a moving object in an image, and a feature point detecting step for detecting a feature point of the moving object in the image; Based on the feature points detected by the feature point detection step, a tracking step for tracking the movement of the object in the image, and the surroundings of the object in the image based on the tracking result of the tracking step processing A correction area setting step for setting a correction area of a specified size, a correction step for correcting an image in the correction area in the image, and a display of an image in which the correction area is corrected by the correction step process Display control step . The correction step includes a control signal for specifying an image in the correction area, a supply step for supplying a parameter indicating a degree of blur of the image, and a feature of the image in the correction area specified based on the control signal. And a feature detection step for outputting a feature code representing the detected feature, a parameter representing a degree of blur of the image, and a coefficient corresponding to the feature code output by the processing of the feature detection step. A storage control step for controlling, a reading step for reading out the coefficient corresponding to the feature code output by the processing of the parameter and the feature detection step, the storage of which is controlled by the processing of the storage control step, and the reading step A product-sum operation that performs a product-sum operation on the pixel values of the input image based on the coefficients read by the processing A calculation step, and a selection output step of selecting and outputting a calculation result obtained by the processing of the product-sum calculation step and a pixel value of the input image, and correcting so as to remove the blur of the image in the correction region .

The program of the present invention is a program for an image processing apparatus that clearly displays a moving object in an image, and a feature point detection control step for controlling detection of a feature point of the moving object in the image; Based on the feature points detected by the process of the feature point detection control step, a tracking control step for controlling the movement of the object of the image is tracked, and the tracking result by the process of the tracking control step is included in the image. A correction area setting control step for controlling to set a correction area having a predetermined size around the object, a correction control step for controlling correction of an image in the correction area in the image, and a correction area. Causing the computer to execute a display control step for controlling the display of the image corrected by the processing of the correction control step . The correction control step includes a control signal for specifying an image in the correction area, a supply control step for supplying a parameter indicating a degree of blur of the image, and a correction signal in the correction area specified based on the control signal. A feature detection control step for detecting a feature of an image and outputting a feature code representing the detected feature, a parameter representing a degree of blur of the image, and a feature code whose output is controlled by the processing of the feature detection control step And a coefficient corresponding to the feature code whose output is controlled by the process of the parameter and the feature detection control step. Based on the read control step for reading out and the coefficient for which the read is controlled by the processing of the read control step. A product-sum operation control step for performing a product-sum operation on the pixel value of the input image, and a selection output for selecting and outputting the operation result by the processing of the product-sum operation control step and the pixel value of the input image And a control step for correcting so as to remove the blur of the image in the correction area.

The recording medium of the present invention is a recording medium on which a program of an image processing apparatus that clearly displays a moving object in an image is recorded, and controls the detection of feature points of the moving object in the image. Based on the feature point detection control step, the tracking control step for controlling the movement of the object of the image based on the feature point detected by the processing of the feature point detection control step, and the tracking result by the processing of the tracking control step And a correction area setting control step for controlling to set a correction area having a predetermined size around the object in the image, and a correction control for controlling correction of the image in the correction area in the image. And a display control step for controlling display of an image in which the correction area is corrected by the processing of the correction control step. Program that has been recorded. The correction control step includes a control signal for specifying an image in the correction area, a supply control step for supplying a parameter indicating a degree of blur of the image, and a correction signal in the correction area specified based on the control signal. A feature detection control step for detecting a feature of an image and outputting a feature code representing the detected feature, a parameter representing a degree of blur of the image, and a feature code whose output is controlled by the processing of the feature detection control step And a coefficient corresponding to the feature code whose output is controlled by the process of the parameter and the feature detection control step. Based on the read control step for reading out and the coefficient for which the read is controlled by the processing of the read control step. A product-sum operation control step for performing a product-sum operation on the pixel value of the input image, and a selection output for selecting and outputting the operation result by the processing of the product-sum operation control step and the pixel value of the input image And a control step for correcting so as to remove the blur of the image in the correction area.

In the present invention, the feature point of the moving object in the image is detected, the movement of the object of the image is tracked based on the detected feature point, and based on the tracking result, the feature point of the object is detected. A correction area having a preset size is set in the periphery, the image in the correction area in the image is corrected, and display of the image in which the correction area is corrected is controlled. For correction, a control signal for specifying an image in the correction area and a parameter indicating the degree of blur of the image are supplied, and the feature of the image in the correction area specified based on the control signal is detected, and the detected feature is detected. Is output, a parameter representing the degree of blur of the image and a coefficient corresponding to the feature code are stored, a parameter and a coefficient corresponding to the feature code are read, and based on the read coefficient, A product-sum operation is performed on the pixel value of the input image, and the calculation result and the pixel value of the input image are selected and output, and the image is corrected so as to eliminate blur in the correction region.

  According to the present invention, it is possible to easily find a noticeable object. In particular, it is possible to easily display an object that moves in an image.

  According to the present invention, it is possible to track a tracking point on an image. In particular, robustness in tracking can be improved. As a result, the tracking point can be reliably tracked even when the object rotates and the tracking point becomes temporarily invisible or an occlusion or a scene change occurs.

  Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows a configuration example when the present invention is applied to a surveillance camera system. In the surveillance camera system 1, an image captured by the imaging unit 21 including a CCD video camera is displayed on the image display 23. The tracking target detection unit 24 detects the tracking target from the image input from the imaging unit 21 and outputs the detection result to the object tracking unit 26.

  The object tracking unit 26 operates to track the tracking point specified by the tracking target detection unit 24 in the image supplied from the imaging unit 21. Based on the output result from the object tracking unit 26, the area setting unit 25 sets a predetermined area (area) around the object including the tracking point from the image captured by the imaging unit 21, and the area. Is output to the image correction unit 22. The image correction unit 22 performs correction to remove image blur (focus blur) on the area set by the area setting unit 25 in the image captured by the imaging unit 21 and outputs the corrected image to the image display 23. Based on the control from the object tracking unit 26, the camera driving unit 27 drives the imaging unit 21 so that the imaging unit 21 captures an image centered on the tracking point.

  The control unit 27 is configured by, for example, a microcomputer and controls each unit. The control unit 27 is connected to a removable medium 28 composed of a semiconductor memory, a magnetic disk, an optical disk, a magneto-optical disk, or the like as necessary, and a program and other various data are supplied as necessary. The control unit 27 also accepts input of instructions (commands, etc.) from the user via an input / output interface (not shown).

  Next, the operation of the monitoring process will be described with reference to the flowchart of FIG. When the power of the monitoring system 1 is turned on, the imaging unit 21 captures an area to be monitored, and an image obtained by capturing the image via the tracking target detection unit 24, the object tracking unit 26, and the image correction unit 22. To the image display 23. The tracking target detection unit 24 executes processing for detecting a tracking target from the image input from the imaging unit 21 in step S1. For example, when a moving object is detected, the tracking target detection unit 24 detects the moving object as a tracking target. The tracking target detection unit 24 detects, for example, the point with the highest luminance or the center point of the tracking target from the tracking targets, and outputs it to the object tracking unit 26.

  In step S2, the object tracking unit 26 executes a tracking process for tracking the tracking point detected in step S1. The details of the tracking process will be described later with reference to FIG. 6. With this process, a tracking point in an object (for example, a person, an animal, or the like) to be tracked in an image captured by the imaging unit 21 is obtained. (For example, the center of the eyes and the head) is tracked, and position information indicating the tracking position is output to the area setting unit 25.

  In step S <b> 3, the area setting unit 25 is represented by a predetermined area around the object to be tracked (for example, a square having a predetermined size centered on the tracking point) based on the output from the object tracking unit 26. Area) is set as the correction target area.

  In step S <b> 4, the image correction unit 22 executes image correction processing for correcting an image in the correction target area set by the area setting unit 25 among the images captured by the imaging unit 21. Details of the image correction process will be described later with reference to FIG. 65, but this process provides a clear video from which the blur of the image has been removed for the image in the correction target area.

  In step S5, the image display 23 outputs the image corrected in step S4, that is, the image captured by the imaging unit 21, and particularly corrected so that only the correction target area is clear. To do.

  In step S6, the object tracking unit 26 detects the movement of the object based on the tracking result of the process in step S2, generates a camera driving signal for driving the camera so that the moved object can be imaged, and the camera driving unit. 27. In step S7, the camera drive unit 27 drives the imaging unit 21 based on the camera drive signal. Thereby, the imaging unit 21 pans or tilts the camera so that the tracking point does not deviate from the screen.

  In step S8, the control unit 27 determines whether or not to end the monitoring process based on an instruction from the user. If the end is not instructed by the user, the control unit 27 returns to step S1 and performs the subsequent processes. Run repeatedly. When the end of the monitoring process is instructed by the user, it is determined to end in step S8, and the control unit 27 ends the monitoring process.

  3A to 3C are diagrams showing an example of an image displayed on the image display 23 at this time in time series. FIG. 3A is an example of an image in which the object 51 to be tracked is imaged by the imaging unit 21. In this example, a person who runs in the left direction in the figure and moves is imaged as the object 51. In FIG. 3B, the object 51 has moved to the left in the figure from the position of FIG. 3A, and in FIG. 3C, the object 51 has moved further to the left from the position of FIG. 3B.

  The tracking target detection unit 24 detects the object 51 in step S1 of FIG. 2, and outputs the eye of the object 51 (person) to the object tracking unit 26 as the tracking point 51A. In step S2, tracking processing is performed by the object tracking unit 26, and a predetermined area around the tracking target object 51 (tracking point 51A) is set as the correction target area 52 by the area setting unit 25 in step S3.

  As described above, the object tracking unit 26 tracks the object 51 based on the tracking point 51A. Therefore, when the object 51 moves, the tracking point 51A also moves, and the tracking result (position) is sent to the area setting unit 25. Is output. For this reason, as shown in FIGS. 3A to 3C, when the object 51 moves to the left in the figure, the correction target area 52 also moves to the left in the figure.

  The correction target area 52 corresponding to the moving object 51 (tracking point 51A) is set as follows, for example. FIG. 4 shows an example in which a rectangular area of a predetermined size is set around the tracking point as the correction target area. In the figure, it is assumed that the correction target area 71A is set first. As the first correction target area 71A, for example, a fixed range centered on the tracking point 51A is set. Of course, when designated by the user, the designated range is set as the correction target area 71A. At this time, the area setting unit 25 stores the coordinates (X, Y) of the upper left corner of the correction target area 71A in the built-in memory. When the tracking point 51A of the object 51 moves, the tracking by the object tracking unit 26 is performed, and the position of the tracking point 51A in the X-axis direction (left-right direction in the figure) and Y-axis direction (up-down direction in the figure) on the screen. Information on (or movement distance) is supplied to the area setting unit 25 as a tracking result.

  Then, the correction target area is set based on the coordinates of the upper left corner described above. For example, when the tracking point 51A moves on the screen by x in the X-axis direction and y in the Y-axis direction, the area setting unit 25 sets x and y to the upper left coordinates (X, Y) of the correction target area 71A. Are added, the coordinates (X + x, Y + y) are calculated, the coordinates are stored as the coordinates of the upper left corner of the new correction target area 71B, and the correction target area 71B is set. When the tracking point 51A further moves by a in the X-axis direction and b in the Y-axis direction, the area setting unit 25 adds a and b to the upper left coordinates (X + x, Y + y) of the correction target area 71A. Then, the coordinates (X + x + a, Y + y + b) are calculated, the coordinates are stored as the coordinates of the upper left corner of the new correction target area 71C, and the correction target area 71C is set.

  In this way, the correction target area moves with the movement of the object (tracking point).

  Further, as described above, the image in the correction target area 52 is subjected to the image correction processing (step S4 in FIG. 2) by the image correction unit 22, the image blur is removed, and the image is displayed on the image display 23. The 3A to 3C, the inside of the correction target area 52 is clearly displayed, and the image of the background 53 that is an area (background) outside the correction target area 52 is the same as the image in the correction target area 52. When compared, it is displayed relatively unclear.

  By doing so, the object 51 in the correction target area 52 is always clearly displayed in the image displayed on the image display 23. Therefore, the user who observes the image display 23 naturally selects the object 51. As a result, for example, intrusion of a suspicious person, movement of an object, and the like can be detected more quickly. Further, since the object 51 is clearly displayed, it is possible to make the user accurately recognize what (who) the moving object (for example, a person) is.

  Next, a detailed configuration example and operation of the object tracking unit 26 in FIG. 1 will be described. FIG. 5 is a block diagram illustrating a functional configuration example of the object tracking unit 26. In this example, the object tracking unit 26 includes a template matching unit 51, a motion estimation unit 52, a scene change detection unit 53, a background motion estimation unit 54, a region estimation related processing unit 55, a transfer candidate holding unit 56, and a tracking point determination unit 57. , A template holding unit 58 and a control unit 59.

  The template matching unit 51 performs a matching process between the input image and the template image held in the template holding unit 58. The motion estimation unit 52 estimates the motion of the input image, and obtains the motion vector obtained as a result of the estimation and the accuracy of the motion vector, the scene change detection unit 53, the background motion estimation unit 54, the region estimation related processing unit 55, And output to the tracking point determination unit 57. The scene change detection unit 53 detects a scene change based on the accuracy supplied from the motion estimation unit 52.

  The background motion estimation unit 54 executes processing for estimating the background motion based on the motion vector and the accuracy supplied from the motion estimation unit 52, and supplies the estimation result to the region estimation related processing unit 55. The region estimation related processing unit 55 is based on the motion vector and accuracy supplied from the motion estimation unit 52, the background motion supplied from the background motion estimation unit 54, and the tracking point information supplied from the tracking point determination unit 57. Perform region estimation processing. Further, the region estimation related processing unit 55 generates a transfer candidate based on the input information, supplies the transfer candidate to the transfer candidate holding unit 56, and holds it. Further, the region estimation related processing unit 55 creates a template based on the input image, supplies the template to the template holding unit 58, and holds it.

  The tracking point determination unit 57 determines a tracking point based on the motion vector and accuracy supplied from the motion estimation unit 52 and the transfer candidate supplied from the transfer candidate holding unit 56, and information on the determined tracking point is obtained. The result is output to the region estimation related processing unit 55.

  The control unit 59 controls each part of the template matching unit 51 to the template holding unit 58 based on the tracking point information output from the tracking target detection unit 24 to track the detected tracking target, and A control signal is output to the camera drive unit 27 so that the tracking point is displayed in the screen displayed on the display 23 (so that the tracking point is not outside the screen), and the camera (imaging unit 21) Control the drive. Further, the control unit 59 outputs a tracking result such as information on the position of the tracking point on the screen to the area setting unit 25, the control unit 27, and the like.

  Next, the operation of the object tracking unit 26 will be described. FIG. 6 is a flowchart for explaining the details of the tracking process executed by the object tracking unit 26 in step S2 of FIG.

  As shown in FIG. 7, the object tracking unit 26 basically executes normal processing and exception processing. That is, normal processing is performed in step S51. The details of this normal process will be described later with reference to FIG. 10, but the process of tracking the tracking point specified by the tracking target detection unit 24 is executed by this process. When the tracking point cannot be changed in the normal process of step S51, an exception process is executed in step S52. The details of this exception processing will be described later with reference to the flowchart of FIG. 37. When the tracking point becomes invisible from the image by this exception processing, the return processing to the normal processing is executed by template matching. If it is determined that the tracking process cannot be continued due to the exception process (cannot return to the normal process), the process is terminated. If it is determined that the return is possible, the process returns to step S51 again. In this way, the normal process in step S51 and the exception process in step S52 are sequentially repeated for each frame.

  In the present invention, as shown in FIG. 7 to FIG. 9, the tracking point is temporarily changed by the normal process and the exception process, such as the tracking target is rotated, the occlusion occurs, the scene change occurs, and the like. Even when it becomes invisible, tracking is possible.

  That is, for example, as shown in FIG. 7, a human face 104 as a tracking target (object) is displayed in the frame n−1, and this human face 104 has a right eye 102 and a left eye 103. ing. It is assumed that the user designates, for example, the right eye 102 (exactly one pixel therein) as the tracking point 101. In the example of FIG. 7, in the next frame n, the person moves to the left in the figure, and in the next frame n + 1, the person's face 104 rotates clockwise. As a result, the right eye 102 that has been visible until now is not displayed, and tracking cannot be performed with the conventional method. Therefore, in the normal process of step S51 described above, the left eye 103 on the face 104 as the same object as the right eye 102 is selected, and the tracking point is switched (set) to the left eye 103. This enables tracking.

  In the display example of FIG. 8, in the frame n−1, the ball 121 has moved from the left side of the face 104 in the drawing, and in the next frame n, the ball 121 has just covered the face 104. In this state, the face 104 including the right eye 102 designated as the tracking point 101 is not displayed. When such an occlusion occurs, the face 504 as the object is not displayed, so there is no transfer point to be tracked instead of the tracking point 101, and it becomes difficult to track the tracking point thereafter. However, in the present invention, the image of the frame n−1 (actually the earlier frame in time) is stored in advance as a template for the right eye 102 as the tracking point 101, and the ball 121 moves further to the right. When the right eye 102 designated as the tracking point 101 appears again in frame n + 1, it is confirmed that the right eye 102 as the tracking point 101 is displayed again by the exception processing in step S52 described above, and the right eye 102 is again tracked. The point 101 is tracked.

  In the example of FIG. 9, the face 104 is displayed in the frame n−1, but in the next frame n, the automobile 111 covers and covers the whole including the human face. That is, in this case, a scene change has occurred. In the present invention, even if the scene change occurs and the tracking point 101 no longer exists in the image, if the automobile 111 moves and the right eye 102 is displayed again in the frame n + 1, the exception processing in step S52 It is confirmed based on the template that the right eye 102 as the tracking point 101 has appeared again, and the right eye 102 can be tracked again as the tracking point 101.

  Next, the details of the normal processing in step S51 in FIG. 6 will be described with reference to the flowchart in FIG. In step S121, the tracking point determination unit 57 executes initialization processing of normal processing. The details will be described later with reference to the flowchart of FIG. 11, but the region estimation range based on the tracking point designated to be tracked by the user is designated by this processing. This area estimation range is the same object as the tracking point specified by the user (for example, when the tracking point is a human eye, a human face as a rigid body that moves like the eye, a human body, or the like) ) Is a range to be referred to when estimating the range of points belonging to. A transfer point is selected from points in the region estimation range.

  Next, in step S122, the control unit 59 controls each unit to wait for input of an image of the next frame. In step S123, the motion estimation unit 52 estimates the tracking point motion. That is, by capturing the frame (following frame) temporally after the frame including the tracking point designated by the user (previous frame) in the process of step S122, an image of two consecutive frames is eventually obtained. Therefore, in step S123, the movement of the tracking point is estimated by estimating the position of the tracking point of the subsequent frame corresponding to the tracking point of the previous frame.

  Note that “preceding in time” means the processing order (input order). Normally, images of each frame are input in the order of imaging. In this case, the frame captured earlier in time becomes the previous frame, but the frame captured later in time is processed (input) first. ), The frame imaged later in time becomes the previous frame.

  In step S124, the motion estimation unit 52 determines whether the tracking point can be estimated as a result of the process in step S123. Whether or not the tracking point can be estimated is determined, for example, by comparing the accuracy value of a motion vector (described later) generated and output by the motion estimation unit 52 with a preset threshold value. Specifically, it is possible to estimate if the accuracy of the motion vector is equal to or greater than a threshold, and it is determined that estimation is impossible if the accuracy is smaller than the threshold. In other words, the possibility here is determined relatively strictly, and if the estimation is not actually impossible but the accuracy is low, it is determined as impossible. As a result, more reliable tracking processing can be performed.

  In step S124, it is determined that the motion estimation result at the tracking point and the motion estimation result at a point near the tracking point can be estimated if they match the motions that occupy the majority, and that they cannot be estimated if they do not match. It is also possible to do so.

  When it is determined that the movement of the tracking point can be estimated (the probability that the tracking point is correctly set on the corresponding point on the same object (if the right eye 102 is designated as the tracking point 101, the right eye 102 If the probability of being correctly tracked is relatively high), the process proceeds to step S125, and the tracking point determination unit 57 shifts the tracking point by the estimated motion (motion vector) obtained in the process of step S123. In other words, this determines the tracking position in the subsequent frame after tracking the tracking point of the previous frame.

  After step S125, region estimation related processing is executed in step S126. The details of the area estimation related process will be described later with reference to FIG. 14. By this process, the area estimation range designated in the initialization process of the normal process in step S121 is updated. Furthermore, if the tracking point is not displayed because the target object rotates, for example, the candidate as a switching point (transfer candidate) as a point to be switched to is the state (can still be tracked). In the state), it is extracted (created) in advance. Further, when the transfer to the transfer candidate cannot be performed, the tracking is temporarily interrupted, but a template is created in advance in order to confirm that the tracking is possible again (the tracking point appears again).

  After the region estimation related process in step S126 is completed, the process returns to step S122 again, and the subsequent processes are repeatedly executed.

  That is, as long as the movement of the tracking point designated by the user can be estimated, the processing from step S122 to step S126 is repeatedly executed for each frame, and tracking is performed.

  On the other hand, if it is determined in step S124 that the movement of the tracking point cannot be estimated (that is, impossible), that is, as described above, for example, the accuracy of the motion vector is equal to or less than the threshold value. The process proceeds to step S127. In step S127, since the transfer candidate generated by the region estimation related process in step S126 is held in the transfer candidate holding unit 116, the tracking point determination unit 57 selects the transfer candidate closest to the original tracking point. Select one. The tracking point determination unit 57 determines whether or not a transfer candidate has been selected in step S128. If a transfer candidate has been selected, the tracking point determination unit 57 proceeds to step S129 and sets the tracking point as the transfer candidate selected in the process of step S127. Change (change). That is, the transfer candidate point is set as a new tracking point. Thereafter, the process returns to step S123, and the process of estimating the movement of the tracking point selected from the transfer candidates is executed.

  In step S124, it is determined again whether or not the movement of the newly set tracking point can be estimated. If it can be estimated, a process of shifting the tracking point by the estimated movement is performed in step S125, and step S126. In, the area estimation related process is executed. Thereafter, the process returns to step S122 again, and the subsequent processes are repeatedly executed.

  If it is determined in step S124 that the newly set tracking point cannot be estimated, the process returns to step S127, and the next transfer candidate closest to the original tracking point is selected from the transfer candidates. In step S129, the transfer candidate is set as a new tracking point. The process after step S123 is repeated again for the new tracking point.

  Even when all the prepared transfer candidates are used as new tracking points, if the movement of the tracking point cannot be estimated, it is determined in step S128 that the transfer candidate cannot be selected, and this normal process is performed. Is terminated. Then, the process proceeds to the exception process in step S52 of FIG.

  Next, details of the initialization process of the normal process in step S121 of FIG. 10 will be described with reference to the flowchart of FIG.

  In step S141, the control unit 59 determines whether or not the current process is a process for returning from the exception process. That is, after the exception process in step S52 is completed, it is determined whether or not the process has returned to the normal process in step S51. In the process of the first frame, since the exception process of step S52 has not been executed yet, it is determined that the process has not returned from the exception process, and the process proceeds to step S142. In step S142, the tracking point determination unit 57 executes processing for setting the tracking point to the position of the tracking point instruction. That is, for example, the point with the highest luminance in the input image is set as the tracking point.

  The tracking point may be set by other methods such as designation by the user. The designation by the user is performed, for example, by designating a predetermined point in the input image as a tracking point to the control unit 59 by operating a mouse or other input unit (not shown). The tracking point determination unit 57 supplies the set tracking point information to the region estimation related processing unit 55.

  In step S143, the region estimation related processing unit 55 sets a region estimation range based on the position of the tracking point set in the processing of step S142. This region estimation range is a reference range when estimating a point on the same rigid body as the tracking point, and more specifically, tracking so that the same rigid body portion as the tracking point occupies most of the region estimation range in advance. By setting the position and size of the estimated area range to follow the same rigid body part as the point, the part that shows the most movement in the estimated area range can be estimated as the same rigid body part as the tracking point It is for doing so. In step S143, as a default value, for example, a predetermined range centered on the tracking point is set as the region estimation range.

  Thereafter, the processing proceeds to step S122 in FIG.

  On the other hand, if it is determined in step S141 that the current process is a process for returning from the exception process in step S52, the process proceeds to step S144, and the tracking point determination unit 57 is described later with reference to FIG. By this processing, the tracking point and the area estimation range are set based on the position that matches the template. For example, a point on the current frame that matches the tracking point on the template is set as the tracking point, and a certain range set in advance from the point is set as the region estimation range. Thereafter, the processing proceeds to step S122 in FIG.

  The above process will be described with reference to FIG. That is, in step S142 of FIG. 11, for example, as shown in FIG. 12, when the human eye 102 of the frame n−1 is designated as the tracking point 101, a predetermined region including the tracking point 101 is specified in step S143. Is designated as the area estimation range 133. In step S124, it is determined whether or not the sample points within the region estimation range 133 can be estimated in the next frame. In the case of the example of FIG. 12, in the frame n + 1 next to the frame n, the left half region 134 in the drawing including the left eye 102 in the region estimation range 133 is hidden by the ball 121. 101 motion cannot be estimated in the next frame n + 1. Therefore, in such a case, one of the points in the area designation range 133 (in the face 104 as a rigid body including the right eye 102) prepared in advance as a transfer candidate in the previous frame n-1 in time is selected. One point (for example, the left eye 103 included in the face 104 (exactly one pixel therein)) is selected, and that point is set as the tracking point in the frame n + 1.

  The region estimation related processing unit 55 has a configuration as shown in FIG. 13 in order to execute the region estimation related processing in step S126 of FIG. That is, the motion vector and the accuracy are input from the motion estimation unit 52, the background motion is input from the background motion estimation unit 54, and the tracking point from the tracking point determination unit 57 is input to the region estimation unit 161 of the region estimation related processing unit 55. Position information is input. In addition to the motion vector and the accuracy supplied from the motion estimation unit 52, the transfer candidate extraction unit 162 is also supplied with the output of the region estimation unit 161. In addition to the input image being input to the template creation unit 163, the output of the region estimation unit 161 is input.

  The region estimation unit 161 estimates a rigid region including a tracking point based on the input, and outputs an estimation result to the transfer candidate extraction unit 162 and the template creation unit 163. The transfer candidate extraction unit 162 extracts transfer candidates based on the input, and supplies the extracted transfer candidates to the transfer candidate holding unit 56. The template creation unit 163 creates a template based on the input, and supplies the created template to the template holding unit 58.

  FIG. 14 shows details of the area estimation related process (the process of step S126 in FIG. 10) executed by the area estimation related processing unit 55. First, in step S161, region estimation processing is executed by the region estimation unit 161. The details thereof will be described later with reference to the flowchart of FIG. 15, but this process allows the region on the image that is estimated to belong to the same object (the rigid body that moves in synchronization with the tracking point) to which the tracking point belongs. Are extracted as points in the region estimation range.

  In step S162, the transfer candidate extraction unit 162 executes a transfer candidate extraction process. The details of the processing will be described later with reference to the flowchart of FIG. 27, but transfer candidate points are extracted from the points of the range estimated by the region estimation unit 161 as the region estimation range and held in the transfer candidate holding unit 56. .

  In step S163, the template creation unit 163 executes template creation processing. Details thereof will be described later with reference to the flowchart of FIG. 28, and a template is created by this processing.

  Next, details of the region estimation processing in step S161 in FIG. 14 will be described with reference to the flowchart in FIG.

  First, in step S181, the region estimation unit 161 determines a sample point as a candidate point of a point estimated to belong to the same target as the tracking point.

  For example, as shown in FIG. 16, the sample points are predetermined in the horizontal direction and the vertical direction with reference to the fixed reference point 201 among the pixels in the entire screen of the frame indicated by the white square in the figure. Pixels at positions separated by the number of pixels can be used as sample points (represented by black squares in the figure). In the example of FIG. 16, the upper left pixel of each frame is set as a reference point 201 (the reference point 201 is indicated by a cross in the figure), and is separated by 5 in the horizontal direction and 5 in the vertical direction. The pixel at the position is taken as the sample point. That is, in the case of this example, pixels at positions dispersed throughout the entire screen are taken as sample points. In this example, the reference point is a point at the same position fixed in each frame n, n + 1.

  For example, as shown in FIG. 17, the reference point 201 can be dynamically changed so as to be a point at a different position for each frame n, n + 1.

  In the example of FIGS. 16 and 17, the interval between the sample points is a fixed value in each frame. However, for example, as shown in FIG. 18, the interval between the sample points may be variable for each frame. it can. In the example of FIG. 18, the interval between sample points is 5 pixels in frame n, whereas it is 8 pixels in frame n + 1. As an interval reference at this time, an area of a region estimated to belong to the same object as the tracking point can be used. Specifically, if the area of the region estimation range is narrowed, the interval is also shortened.

  Alternatively, as shown in FIG. 19, the interval between sample points can be made variable in one frame. The distance from the tracking point can be used as a reference for the interval at this time. That is, the sample point closer to the tracking point has a smaller interval, and the farther from the tracking point, the larger the interval.

  When the sample points are determined as described above, in step S182, the region estimation unit 161 then determines the region estimation range (steps S143 and S144 in FIG. 11 or steps S206 and S208 in FIG. 20 described later). The process of estimating the movement of the sample points within (determined in the process of (1)) is executed. That is, the region estimation unit 161 extracts the corresponding point of the next frame corresponding to the sample point within the region estimation range based on the motion vector supplied from the motion estimation unit 52.

  In step S183, the region estimation unit 161 executes processing for excluding points based on motion vectors whose accuracy is lower than a preset threshold value from the sample points estimated in step S182. The accuracy of the motion vector necessary for this processing is supplied from the motion estimation unit 52. Thereby, only the points estimated based on the motion vector with high accuracy are extracted from the sample points within the region estimation range.

  In step S184, the region estimation unit 161 extracts the full screen motion in the motion estimation result within the region estimation range. The full screen motion means a motion that takes the area corresponding to the same motion and maximizes the area. Specifically, a motion histogram is generated by assigning a weight proportional to the sample point interval at each sample point to the motion of each sample point, and one motion (one motion vector) having the maximum weighting frequency is generated. Extracted as full screen motion. When generating a histogram, for example, a representative value of motion can be prepared with pixel accuracy, and a motion having a value with one pixel accuracy can be added to the histogram.

  In step S185, the region estimation unit 161 extracts sample points within the region estimation range having the full screen motion as a result of region estimation. In this case, the sample points having the full screen motion include not only the sample points having the same motion as the full screen motion but also the difference in motion from the full screen motion is equal to or less than a predetermined threshold value set in advance. The sample point can also be a sample point having full screen motion here.

  In this way, among the sample points within the area estimation range determined by the processing of steps S143, S144, S206, and S208, the sample point having the full screen motion is estimated to belong to the same target as the tracking point. Is finally extracted (generated).

  Next, in step S186, the region estimation unit 161 executes a region estimation range update process. Thereafter, the processing proceeds to step S122 in FIG.

  FIG. 20 shows details of the region estimation range update processing in step S186 of FIG. In step S201, the region estimation unit 161 calculates the center of gravity of the region. This region means a region composed of sample points extracted in the process of step S185 in FIG. 15 (region composed of points estimated to belong to the same target as the tracking point). That is, one motion vector (full screen motion) corresponds to this area. For example, as shown in FIG. 21A, among sample points indicated by white squares in the figure, sample points having full-screen motion in the process of step S185 in FIG. 21A, sample points indicated by black squares are extracted, and a region constituted by the sample points is extracted (estimated) as a region 222. Then, the center of gravity 224 of the region 222 is further calculated. Specifically, the sample point centroid obtained by assigning the weight of the sample point interval to each sample point is obtained as the centroid of the region. This process has the meaning of obtaining the position of the region in the current frame.

  Next, in step S202, the region estimation unit 161 executes a process of shifting the center of gravity of the region by full screen motion. This process has the meaning of causing the area estimation range 221 to follow the movement of the position of the area and to move to the estimated position in the next frame. As shown in FIG. 21B, when the tracking point 223 in the current frame appears as the tracking point 233 in the next frame based on the motion vector 238, the full-screen motion vector 230 substantially corresponds to the tracking point motion vector 238. Therefore, the point 234 on the same frame (next frame) as the tracking point 233 is obtained by shifting the center of gravity 224 in the current frame based on the motion vector 230 (full screen motion). If the area estimation range 231 is set around the point 234, the area estimation range 221 is moved to the estimated position in the next frame by following the movement of the position of the area 222.

  In step S203, the region estimation unit 161 determines the size of the next region estimation range based on the region estimation result. Specifically, the sum of squares of sample point intervals (intervals of points indicated by black squares in the region 222 in FIG. 21A) regarding all sample points estimated as the region is regarded as the area of the region 222. The size of the region estimation range 231 in the next frame is determined so that the size is slightly larger than the area. That is, the size of the region estimation range 231 increases when the number of sample points in the region 222 is large and decreases when the number of sample points is small. In this way, not only can the enlargement / reduction of the area 222 be followed, but also the entire screen area within the area estimation range 221 can be prevented from becoming a peripheral area to be tracked.

  When the full screen motion extracted in step S184 in FIG. 15 matches the background motion, the background and the tracking target cannot be distinguished by the motion. Therefore, the background motion estimation unit 54 always performs the background motion estimation process. In step S204, the region estimation unit 161 extracts the background motion supplied from the background motion estimation unit 54 and the process of step S184 in FIG. It is determined whether or not the full screen motion matches. If the full screen motion matches the background motion, in step S205, the region estimation unit 161 limits the size of the next region estimation range so that the current region estimation range is maximized. Thereby, it is suppressed that the background is erroneously recognized as the tracking target and the size of the area estimation range is enlarged.

  If it is determined in step S204 that the full screen motion and the background motion do not match, the processing in step S205 is not necessary and is skipped.

  Next, in step S <b> 206, the region estimation unit 161 determines the size of the next region estimation range around the shifted region center of gravity. As a result, the area estimation range is determined such that the center of gravity coincides with the already obtained area centroid after the shift, and the size thereof is proportional to the area width.

  In the example of FIG. 21B, the area estimation range 231 is determined to have a size corresponding to the area 222 area with the shifted center of gravity 234 based on the motion vector (full screen motion) 230 as the center.

  It is necessary to ensure (ensure) that the region having the full screen motion within the region estimation range 231 is the region of the tracking target (for example, the face 104 in FIG. 12). Therefore, in step S207, the region estimation unit 161 determines whether or not the tracking point is included in the next region estimation range. If the tracking point is not included, in step S208, the region estimation unit 161 determines that the tracking point is included in the next region estimation range. A process for shifting the region estimation range is executed. If the tracking point is included in the next area estimation range, the process of step S208 is not necessary and is skipped.

  As a specific method of shifting in this case, a method of minimizing the moving distance, a minimum distance at which the tracking point is included along a vector from the center of gravity of the region estimation range before the shift to the tracking point A way to move only is considered.

  In order to emphasize the robustness of tracking, a method of not performing a shift to include a tracking point in the region is also conceivable.

  In the example of FIG. 21C, since the area estimation range 231 does not include the tracking point 233, the area estimation range 241 is shifted to a position indicated as the area estimation range 241 (a position including the tracking point 233 at the upper left).

  21A to 21C show the case where the shift process of step S208 is necessary, but FIGS. 22A to 22C show the case where the shift process of step S208 is not necessary (the tracking point is the next region estimation range in step S207). Example).

  As shown in FIGS. 22A to 22C, when all the sample points in the area estimation range 221 are area points, the shift process in step S208 in FIG. 20 is not necessary.

  21A to 21C and FIGS. 22A to 22C show an example in which the region estimation range is rectangular, but the region estimation range is circular as shown in FIGS. 25A to 25C and FIGS. 26A to 26C. It is also possible. FIGS. 25A to 25C are diagrams corresponding to FIGS. 21A to 21C, showing a case where the shift process of step S208 (FIG. 20) is necessary, and FIGS. 26A to 26C correspond to FIGS. 22A to 22C. It is a figure and represents the case where the shift process of step S108 is not required.

  As described above, the position estimation range position and size for the next frame are determined to include the tracking point by the region estimation range update process in FIG. 20 (step S186 in FIG. 15).

  In the update process of the area estimation range in FIG. 20, the area estimation range is a rectangular (or circular) fixed shape, but may be a variable shape. An example of the region estimation range update processing in step S186 in FIG. 15 in this case will be described with reference to FIG.

  In step S231, the region estimation unit 161 determines whether or not the full screen motion extracted by the processing in step S184 in FIG. 15 matches the background motion estimated by the background motion estimation unit 164. If the movements of the two do not match, the process proceeds to step S233, where the area estimation unit 161 applies a small value corresponding to each of the points estimated as the area (area composed of pixels that match the full screen movement). A region is determined (one small region is determined for one point). In the example of FIGS. 26A and 26B, small regions 271 and 272 corresponding to the points of the regions indicated by the black squares in the drawing are determined in the region estimation range 261. The small area 271 in the figure shows an example in which small areas corresponding to four points overlap. The size of the small area may be determined so as to be proportional to the interval between the sample points, for example.

  Next, in step S234, the region estimation unit 161 sets the sum of the small regions determined in the process of step S233 as the provisional region estimation range. In the example of FIG. 26C, a region 281 that is the sum of the small region 271 and the small region 272 is set as the provisional region estimation range. When a plurality of discontinuous regions are formed as a result of taking the sum of the small regions, only the region having the largest area among them can be set as the provisional region estimation range.

  If it is determined in step S231 that the full screen motion matches the background motion, in step S232, the region estimation unit 161 sets the current region estimation range as the provisional region estimation range. The current region estimation range is used as the provisional estimation region range because the background and the tracking target cannot be distinguished from each other by the motion when the background motion estimation result matches the full screen motion. This is to prevent changes.

  After the process of step S234 or step S232, in step S235, the region estimation unit 161 determines the next region estimation range by shifting the provisional region estimation range determined in step S234 or step S232 by the full screen motion. In the example of FIG. 26D, the provisional region estimation range 281 is shifted based on the motion vector 283 due to the full screen motion to be a provisional region estimation range 282.

  In step S236, the region estimation unit 161 determines whether or not the tracking point is included in the next region estimation range determined in the process of step S235. If not included, the processing proceeds to step S237. The next region estimation range is shifted to include. In the example of FIG. 26C and FIG. 26D, the region estimation range 282 does not include the tracking point 284, so that the region estimation range 291 is shifted to include the tracking point 284 in the upper left.

  If it is determined in step S236 that the tracking point is included in the next region estimation range, the shift processing in step S237 is not necessary and is skipped.

  Next, the transfer candidate extraction process in step S162 of FIG. 14 will be described with reference to the flowchart of FIG.

  In step S261, the transfer candidate extraction unit 162 holds, as transfer candidates, the point shift results of the estimated motion corresponding to each of the points estimated as the full screen motion region. That is, instead of using the points obtained as region estimation results as they are, a process for extracting the results shifted based on the respective motion estimation results is performed for use in the next frame. The changed transfer candidates are supplied to and held in the transfer candidate holding unit 56.

  This process will be described with reference to FIG. That is, in the example of FIG. 12, the tracking point 101 exists in the frames n−1 and n, but in the frame n + 1, it is hidden by the ball 121 flying from the left side in the drawing, and the tracking point 101 does not exist. Therefore, in frame n + 1, it is necessary to change the tracking point to another point on the face 104 as a tracking target (for example, the left eye 103 (actually closer to the right eye 102)). Therefore, a transfer candidate is prepared in advance in a frame before transfer is actually required.

  Specifically, in the case of the example in FIG. 12, the motion estimation result in the region estimation range 133 from frame n to frame n + 1 needs to be changed in the region estimation range 133, so that there is a high probability that it cannot be estimated correctly. Is expected. That is, in the example of FIG. 12, the transfer occurs because the tracking point and a part of the same object are hidden. As a result, in the region estimation range 133 in the frame n, regarding the portion 134 where the target is hidden in the frame n + 1 (the shaded portion in FIG. 12) 134, the motion is not estimated correctly, and it is estimated that the accuracy of the motion is low. It is estimated that the accuracy is not low and a motion estimation result is meaningless.

  In such a case, there is a high possibility that the region estimation is erroneous due to a decrease in motion estimation results that can be used in region estimation or a mixture of erroneous motion estimation results. On the other hand, such a possibility is generally greater in the region estimation between the previous frame n−1 and the frame n in comparison with the estimation between the frame n and the frame n + 1. Expected to be lower.

  Therefore, in order to reduce the risk, the region estimation result is not used as it is, but the region estimation result obtained in the previous frame n-1 (or a frame earlier in time) is used as the movement destination in the next frame. It is desirable to use as a transfer candidate for improving the performance.

  However, the region estimation result can be used as it is. The processing in this case will be described with reference to FIG.

  FIG. 28 shows details of the template creation processing in step S163 of FIG. In step S <b> 281, the template creation unit 163 determines a small region corresponding to each of the points estimated as the region (region of full screen motion). In the example of FIG. 29, the small area 322 is determined corresponding to the point 321 of the area.

  In step S282, the template creation unit 163 sets the area of the sum of the small areas determined in the process of step S281 as the template range. In the example of FIG. 29, the sum area of the small areas 322 is the template range 331.

  In step S283, the template creation unit 163 creates a template from the template range information and image information set in step S282, supplies the template to the template holding unit 58, and holds the template. Specifically, pixel data in the template range 331 is used as a template.

  In FIG. 30, the small area 341 corresponding to the area point 321 has a larger area than the small area 322 in FIG. As a result, the template range 351 of the sum area of the small areas 341 is also wider than the template range 331 of FIG.

  It is conceivable that the size of the small region is proportional to the interval between the sample points, but the proportionality constant at that time can be determined so that the area becomes the square of the interval between the sample points, or larger or smaller than that. It is also possible.

  For example, a fixed size or shape range centered on the tracking point may be used as the template range without using the region estimation result.

  FIG. 31 shows the positional relationship between the template and the area estimation range. The template range 403 includes a tracking point 405. The upper left point of the circumscribed rectangle 401 circumscribing the template range 403 is a template reference point 404. The vector 406 from the template reference point 404 toward the tracking point 405 and the vector 407 from the template reference point 404 toward the upper left reference point 408 in the region estimation range 402 in the drawing are used as information of the template range 403. The template is composed of pixels included in the template range 403. The vectors 406 and 407 are used for returning to normal processing when the same image as the template is detected.

  In the above processing, unlike the case of the transfer candidate, the example in which both the range and the pixel correspond to the current frame is used as the template. However, as in the case of the transfer candidate, the destination in the next frame is set as the template. Can also be used.

  As described above, a template made up of pixel data including a tracking point is created in advance during normal processing in the same manner as a transfer candidate.

  The region estimation related process in step S126 of FIG. 10 can be processed by configuring the region estimation related processing unit 55 as shown in FIG. 32, for example.

  Even in this case, the region estimation related processing unit 55 includes the region estimation unit 161, the transfer candidate extraction unit 162, and the template creation unit 163 as in the case of FIG. 13, but in the case of this embodiment, The tracking point information and the input image are input to the region estimation unit 161 from the tracking point determination unit 57. Only the output of the region estimation unit 161 is supplied to the transfer candidate extraction unit 162. The template creation unit 163 is supplied with the output of the region estimation unit 161 and the input image.

  Also in this case, as in the case shown in FIG. 14, the basic processing is performed in step S161, in which region estimation processing is performed, in step S162, transfer candidate extraction processing is performed, and in step S163, template creation is performed. Processing is performed. Of these, the template creation process in step S163 is the same as that shown in FIG. 28, and therefore only the region estimation process in step S161 and the transfer candidate extraction process in step S162 will be described below.

  First, the details of the region estimation processing in step S161 will be described with reference to the flowchart in FIG. In step S301, the region estimation unit 161 in FIG. 32 determines sample points in order to estimate a region on the image that belongs to the same target as the tracking point. This process is the same as the process of step S181 in FIG.

  However, the target frame in the process of step S301 is a frame for which a tracking point has been obtained (a frame including a tracking point after tracking), and this point is a frame for obtaining a sample point in step S181 of FIG. Different from the previous frame.

  Next, in step S302, the region estimation unit 161 executes a process of applying a spatial low-pass filter to the image of the next frame (the frame whose sample point has been determined in step S301). That is, by applying a low pass filter, the high frequency component is removed and the image is smoothed. This facilitates the same color region growth process in the next step S303.

  Next, in step S303, the area estimation unit 161 grows the same color area of the tracking point from the tracking point as a starting point under the condition that the pixel value difference is less than the threshold THimg, and the sample points included in the same color area A process is executed with the region estimation result as. As the region estimation result, sample points included in the same color region as the growth result are used.

  Specifically, for example, as shown in FIG. 34A, pixel values of pixels in eight directions adjacent to the tracking point 421 are read. That is, the pixel values of pixels adjacent to the eight directions of the upper direction, the upper right direction, the right direction, the lower right direction, the lower direction, the lower left direction, the left direction, and the upper left direction are read. The difference between the read pixel value and the pixel value of the tracking point 421 is calculated. Then, it is determined whether or not the calculated difference value is greater than or equal to a threshold value THimg. In the case of FIG. 34A, the pixel value in the direction indicated by the arrow, that is, the difference between the pixel value in the upward direction, the upper right direction, the downward direction, the left direction, and the upper left direction and the tracking point 321 is less than the threshold THimg. It is assumed that the difference between the pixel value in the direction indicated without the middle arrow, that is, the right direction, the lower right direction, and the lower left direction, and the tracking point 421 is greater than or equal to the threshold THimg.

  In this case, as shown in FIG. 34B, a pixel whose difference is less than the threshold THimg (a pixel in a direction indicated by an arrow with respect to the tracking point 421 in FIG. 34A) is registered as a pixel 422 in the same color area as the tracking point 421. Is done. Similar processing is performed for each pixel 422 registered in the same color area. In the example shown in FIG. 34B, the difference between the pixel values of the pixel 422 indicated by the white circle on the upper left in the figure and the adjacent pixel (excluding the pixel that has already been determined to be the same color area) is calculated. It is determined whether the difference is greater than or equal to a threshold THimg. In the example of FIG. 34B, the pixels in the right direction, the lower right direction, and the lower direction are directions in which the determination process for the same color region has already been completed, so the upper direction, the upper right direction, the lower left direction, the left direction, and the upper left direction The difference at is calculated. In this example, the difference between the three directions of the upper direction, the upper right direction, and the upper left direction is less than the threshold THimg, and as shown in FIG. 34C, the pixels in that direction are pixels in the same color area as the tracking point 421. be registered.

  By sequentially repeating the above processing, as shown in FIG. 35, the points included in the same color region 431 among the sample points are estimated as points on the same object as the tracking point 421.

  Following the region estimation process shown in FIG. 33 (step S161 in FIG. 14), the transfer candidate extraction process in step S162 in FIG. 14 executed in the transfer candidate extraction unit 162 in FIG. 32 is shown in the flowchart in FIG. It becomes like this.

  That is, in step S331, the transfer candidate extraction unit 162 uses all points estimated as regions (same color regions) as transfer candidates as they are, and supplies them to the transfer candidate holding unit 56 for holding.

  32, following the region estimation process in FIG. 33 (step S161 in FIG. 14) and the transfer candidate extraction process in FIG. 36 (step S162 in FIG. 14), the template creation unit 163 in FIG. The template creation process executed in step S163 in FIG. 14 is the same as in the case shown in FIG. 28, and a description thereof will be omitted.

  However, in this case, the same color area of the tracking point can be used as the template range as it is.

  Details of the exception processing in step S52 performed following the normal processing in step S51 of FIG. 6 described above will be described with reference to the flowchart of FIG. As described above, this process is performed when it is determined in step S124 in FIG. 10 that it is impossible to estimate the movement of the tracking point, and in step S128, it is determined that the transfer candidate for changing the tracking point cannot be selected. Will be executed.

  In step S401, the control unit 59 executes an exception process initialization process. Details of this processing are shown in the flowchart of FIG.

  In step S321, the control unit 59 causes a scene change when it becomes impossible to track the tracking point (when it is impossible to estimate the movement of the tracking point and a transfer candidate for changing the tracking point cannot be selected). It is determined whether it has been. The scene change detection unit 53 constantly monitors whether or not there is a scene chain based on the estimation result of the motion estimation unit 52, and the control unit 59 performs step S421 based on the detection result of the scene change detection unit 53. Execute the judgment. Specific processing of the scene change detection unit 53 will be described later with reference to FIGS. 54 and 55.

  If a scene change has occurred, it is presumed that the reason why tracking has become impossible is due to the occurrence of a scene change, and in step S422, the control unit 19 sets the mode to scene change. In contrast, if it is determined in step S421 that no scene change has occurred, the control unit 59 sets the mode to another mode in step S423.

  After the process of step S422 or step S423, the template matching unit 51 executes a process of selecting the oldest template in time in step S424. Specifically, as shown in FIG. 39, for example, when transition from frame n to frame n + 1 is performed, if exception processing is to be executed, a frame n−m + 1 is generated for frame n and is stored in the template holding unit 58. A template generated with respect to the frame mn + 1, which is the oldest template in time, is selected from the held m-frame templates.

  In this way, the template just before the transition to exception processing (the template generated for frame n in the case of the example in FIG. 39) is used in order to select a template a little earlier in time, such as occlusion to be tracked. This is because when the transition to exception processing occurs, the tracking target is already considerably hidden immediately before the transition, and it is highly likely that the tracking target cannot be captured sufficiently large in the template at that time. Therefore, reliable tracking is possible by selecting a template in a frame slightly before in time.

  Next, in step S425, the template matching unit 51 executes processing for setting a template search range. For example, the template search range is set such that the position of the tracking point immediately before the transition to the exception process is the center of the template search range.

  That is, as shown in FIG. 40, when the right eye 102 of the subject's face 104 is designated as the tracking point 101 in the frame n, the ball 121 flies from the left direction in the figure, and the tracking point 101 in the frame n + 1. Is assumed, and the tracking point 101 appears again in the frame n + 2. In this case, a region centered on the tracking point 101 (included in the template range 411) is set as the template search range 412.

  In step S426, the template matching unit 51 resets the number of elapsed frames and the number of scene changes after shifting to exception processing to zero. The number of frames and the number of scene changes are used in a continuation determination process (steps S461, S463, S465, and S467 in FIG. 41) in step S405 in FIG.

  After the exception process initialization process is completed as described above, in step S402 in FIG. 37, the control unit 59 executes a process of waiting for the next frame. In step S403, the template matching unit 51 performs template matching processing within the template search range. In step S404, the template matching unit 51 determines whether it is possible to return to normal processing.

  Specifically, the absolute value sum of the difference between the template several frames before (a pixel in the template range 411 in FIG. 40) and the matching target pixel in the template search range is calculated by the template matching process. More specifically, the sum of absolute values of differences between the pixels in the predetermined block in the template range 411 and the predetermined block in the template search range is calculated. The position of the block is sequentially moved within the template range 411, and the sum of absolute values of the differences of the respective blocks is added to obtain a value at the position of the template. Then, the position where the sum of absolute values of the differences becomes the smallest when the template is sequentially moved within the template search range and its value are searched. In step S404, the absolute sum of the minimum differences is compared with a predetermined threshold value set in advance. If the sum of absolute values of the differences is less than or equal to the threshold value, an image including the tracking point (included in the template) has appeared again, so it is determined that the normal processing can be restored, and the processing Returns to the normal processing of step S51 of FIG.

  Then, as described above, in step S141 in FIG. 11, it is determined that the return from the exception processing is performed, and in step S144, the position where the difference absolute value sum is minimum is set as the matched position of the template, The tracking point and area estimation range are set based on the positional relationship between the template position and the tracking point area estimation range held corresponding to the template. That is, as described with reference to FIG. 31, the region estimation range 402 is set based on the vectors 406 and 407 with the tracking point 405 as a reference.

  However, in the region estimation process in step S161 of FIG. 14, when a method that does not use the region estimation range is used (for example, when the region estimation process shown in FIG. 33 is used), the region estimation range is set. Absent.

  The determination as to whether or not it is possible to return to normal processing in step S404 in FIG. 37 is performed by comparing the value obtained by dividing the minimum sum of absolute differences by the activity of the template with a threshold value. Also good. As the activity in this case, the value calculated in step S603 of FIG. 48 by the activity calculation unit 602 of FIG. 47 described later can be used.

  Alternatively, it is possible to return to the normal processing by comparing a value obtained by dividing the current minimum absolute difference sum by the minimum absolute difference sum one frame before with a predetermined threshold. It may be determined whether or not. In this case, it is not necessary to calculate the activity. That is, in step S404, the correlation between the template and the template search range is calculated, and determination is performed based on the comparison between the correlation value and the threshold value.

  If it is determined in step S404 that it is not possible to return to the normal process, the process proceeds to step S405, and the continuation determination process is executed. The details of the continuation determination process will be described later with reference to the flowchart of FIG. 41, whereby it is determined whether or not the tracking process can be continued.

  In step S406, the control unit 59 determines whether or not tracking of the tracking point can be continued based on the result of the continuation determination process (based on a flag set in steps S466 and S468 of FIG. 41 described later). To do. If the tracking point tracking process can be continued, the process returns to step S302, and the subsequent processes are repeatedly executed. That is, the process of waiting until the tracking point appears again is repeatedly executed.

  On the other hand, when it is determined in step S406 that the tracking point tracking process cannot be continued (in step S465 of FIG. 41 described later, it is determined that the number of elapsed frames after the tracking point disappears is greater than or equal to the threshold value THfr. Or when it is determined in step S467 that the number of scene changes is equal to or greater than the threshold value THsc), the tracking process is terminated as the tracking process is no longer possible.

  FIG. 41 shows the details of the continuation determination process in step S405 of FIG. In step S461, the control unit 59 executes a process of adding 1 to the number of elapsed frames as a variable. The number of elapsed frames is previously reset to 0 in the exception process initialization process (step S426 in FIG. 38) in step S401 in FIG.

  Next, in step S462, the control unit 59 determines whether there is a scene change. Whether or not there is a scene change can be determined based on the detection result of the scene change detection unit 53 always executing the detection process. If there is a scene change, the process proceeds to step S463, and the control unit 59 adds 1 to the number of scene changes as a variable. The number of scene changes is also reset to 0 in the initialization process of step S426 in FIG. If no scene change has occurred during the transition from the normal process to the exception process, the process of step S463 is skipped.

  Next, in step S464, the control unit 59 determines whether or not the currently set mode is a scene change. This mode is set in steps S422 and S423 in FIG. When the currently set mode is a scene change, the process proceeds to step S467, and the control unit 59 determines whether or not the number of scene changes is smaller than a preset threshold value THsc. If the number of scene changes is smaller than the threshold value THsc, the process proceeds to step S466, and the control unit 59 sets a flag that allows continuation. If the number of scene changes is equal to or greater than the threshold value THsc, the process proceeds to step S468 and cannot be continued. Set the flag.

  On the other hand, when it is determined in step S464 that the mode is not a scene change (when it is determined that the mode is other), the process proceeds to step S465, and the control unit 59 determines whether or not the number of elapsed frames is smaller than the threshold value THfr. Determine whether. The number of elapsed frames is also reset to 0 in advance in step S426 of the exception process initialization process of FIG. If it is determined that the number of elapsed frames is smaller than the threshold value THfr, a continuation flag is set in step S466, and if it is determined that the number of elapsed frames is greater than or equal to the threshold value THfr, the process continues in step S468. A disabled flag is set.

  As described above, when the number of scene changes during the template matching process is equal to or greater than the threshold value THsc, or when the number of elapsed frames is equal to or greater than the threshold value THfr, no further tracking process is possible.

  When the mode is other, a condition that the number of scene changes is 0 may be added to determine whether or not continuation is possible.

  In the above, it is assumed that the frame of the image is used as a processing unit and all frames are used. However, processing is not performed in units of fields or using all frames or fields, but is extracted by thinning out at a predetermined interval. It is also possible to use a modified frame or field.

  In the above description, the estimated movement destination of the point in the area is used as the transfer candidate, but it is also possible to use the point in the area as it is. In this case, the normal process of step S51 of FIG. 6 is changed to the process of FIG. 42 instead of the process of FIG.

  The processes in steps S501 through S510 in FIG. 42 are basically the same as the processes in steps S21 through S29 in FIG. 10, but the next frame after step S502 in FIG. 42 corresponding to step S122 in FIG. Next to the process of waiting for the process, the area estimation related process in step S503 is inserted, and the area estimation range update process in step S507 is executed instead of the area estimation related process in step S126 in FIG. Is different. The other processes are the same as those in FIG.

  The details of the area estimation related process in step S503 in FIG. 42 are the same as those described with reference to FIG. 10, and the area estimation range update process in step S507 is the same as in the case described with reference to FIG. It becomes processing.

  When the normal processing is executed as shown in the flowchart of FIG. 42, the region estimation processing (region estimation processing in step S161 of FIG. 10) of the region estimation related processing of step S503 (region estimation related processing of FIG. 10) is As shown in the flowchart of FIG.

  The processing from step S531 to step S535 is basically the same as the processing from step S181 to step S186 in FIG. However, the region estimation range update processing in step S186 in FIG. 15 is omitted in FIG. Other processes are the same as those in FIG. That is, the area estimation range update process is executed in step S507 of FIG. 42, and is thus unnecessary in the area estimation process of FIG.

  Furthermore, the transfer candidate extraction process (transfer candidate extraction process in step S162 of FIG. 14) of the area estimation related process (area estimation related process of FIG. 16) in step S503 when the normal process of FIG. 42 is performed is shown in FIG. It comes to be. The process in step S551 is the same as the transfer candidate extraction process in step S261 in FIG.

  As described above, the difference in processing between the case where the normal processing is performed as shown in the flowchart of FIG. 42 and the case where the normal processing is performed as shown in FIG. 10 will be described as shown in FIGS. 45 and 46. become.

  When the normal processing shown in the flowchart of FIG. 10 is performed, as shown in FIG. 45, in the frame n, the region 222 is constituted by the points 451 indicated by the black squares in the region estimation range 221. In the next frame n + 1, a point 452 on the frame n + 1 at a position obtained by shifting each point 451 of the region 222 in the previous frame n based on the respective motion vector 453 is set as a transfer candidate (FIG. 23). Step S161).

  The motion vector 453 of each point 451 may be the same as the motion vector of the full screen motion, but there are some variations in the estimated motion of each point depending on the accuracy considered as the same motion as the full screen motion. For example, assuming that the difference of ± 1 pixel is the same in both the horizontal direction and the vertical direction, the movement of (0, 0) includes the movement of (-1, 1) and (1, 0). In this case, even if the full screen motion is (0, 0), if each point 451 has a motion such as (-1, 1), (1, 0), etc., Only shifted. Instead of using the destination point as a transfer candidate as it is, it is also possible to set the closest point among sample points obtained in advance as a transfer candidate. Of course, in order to reduce the processing load, each point 451 may be shifted by the amount corresponding to the full screen movement.

  On the other hand, when the normal process is executed as shown in the flowchart of FIG. 42, as shown in FIG. 46, a point 461 in the region estimation range 221 in the frame n is set as a transfer candidate.

  Next, a configuration example of the motion estimation unit 52 in FIG. 5 will be described with reference to FIG. In this embodiment, the input image is supplied to the evaluation value calculation unit 601, the activity calculation unit 602, and the motion vector detection unit 606. The evaluation value calculation unit 601 calculates an evaluation value related to the degree of coincidence between the two objects associated by the motion vector, and supplies the evaluation value to the normalization processing unit 604. The activity calculation unit 602 calculates the activity of the input image and supplies it to the threshold value determination unit 603 and the normalization processing unit 604. The motion vector detection unit 606 detects a motion vector from the input image and supplies it to the evaluation value calculation unit 601 and the integration processing unit 605.

  The normalization processing unit 604 normalizes the evaluation value supplied from the evaluation value calculation unit 601 based on the activity supplied from the activity calculation unit 602, and supplies the obtained value to the integration processing unit 605. The threshold determination unit 603 compares the activity supplied from the activity calculation unit 602 with a predetermined threshold and supplies the determination result to the integration processing unit 605. The integration processing unit 605 calculates the accuracy of the motion vector based on the normalization information supplied from the normalization processing unit 604 and the determination result supplied from the threshold value determination unit 603, and the obtained accuracy is detected by the motion vector detection. This is output together with the motion vector supplied from the unit 606.

  Next, the motion estimation process of the motion estimation unit 52 will be described with reference to the flowchart of FIG. Although the motion vector is obtained for a point, the accuracy is calculated using image data of a small block in the vicinity of two points associated by the motion vector, for example, centering on the point. In step S601, the motion vector detection unit 606 detects a motion vector from the input image. For this detection, for example, a block matching method or a gradient method is used. The detected motion vector is supplied to the evaluation value calculation unit 601 and the integration processing unit 605.

In step S602, the evaluation value calculation unit 601 calculates an evaluation value. Specifically, for example, the sum of absolute differences of pixel values of two blocks centered on two points associated with motion vectors is calculated. That is, the motion vector V (vx, vy) detected by the motion vector detection unit 606 in step S601, the point P (Xp, Yp) on the image Fi of the previous frame based on the motion vector V (vx, vy), and the time later The relationship of the point Q (Xq, Yq) on the image Fj of the frame is expressed by the following equation.
Q (Xq, Yq) = P (Xp, Yp) + V (vx, vy) (1)

  The evaluation value calculation unit 601 calculates an evaluation value Eval (P, Q, i, j) for a block centered on the point P and a block centered on the point Q based on the following equation.

  Each block is a square having 2L + 1 pixels on one side. The summation ΣΣ in the above equation is performed between corresponding pixels when x is from −L to L and y is from −L to L. Therefore, for example, when L = 2, nine differences are obtained, and the sum of the absolute values is calculated. The evaluation value indicates that the two blocks match well as the value approaches zero.

  The evaluation value calculation unit 601 supplies the generated evaluation value to the normalization processing unit 604.

  In step S603, the activity calculation unit 602 calculates an activity from the input image. The activity is a feature amount representing the complexity of the image, and as shown in FIG. 49, the difference between the pixel of interest Y (x, y) and the adjacent 8 pixels Y (x + i, y + j) for each pixel. The average value of the sum of absolute values is calculated based on the following formula as activity Activity (x, y) at the target pixel position.

  In the case of the example of FIG. 49, the value of the pixel of interest Y (x, y) located at the center among the 3 × 3 pixels is 110, and the values of the eight pixels adjacent thereto are 80, 70, and 75, respectively. , 100, 100, 100, 80, 80, the activity Activity (x, y) is expressed by the following equation.

  Activity (x, y) = {| 80-110 | + | 70-110 | + | 75-110 | + | 100-110 | + | 100-110 | + | 80-110 | + | 80−110 |} /8=24.375.

  Similar processing is performed for all pixels in the frame.

  In order to calculate the motion vector accuracy in units of blocks, the sum of the activities of all the pixels in the block expressed by the following equation is defined as the activity (block activity) Blockactivity (i, j) of the block.

  In addition, the activity may be a variance value, a dynamic range, or the like.

  In step S604, the threshold determination unit 603 compares the block activity calculated by the activity calculation unit 602 with a predetermined threshold set in advance. Then, a flag indicating whether or not the input block activity is larger than the threshold value is output to the integration processing unit 605.

  Specifically, as a result of the experiment, the block activity and the evaluation value have the relationship shown in FIG. 50 using the motion vector as a parameter. In FIG. 50, the horizontal axis represents the block activity Blockactivity (i, j), and the vertical axis represents the evaluation value Eval. When the motion is correctly detected (when the correct motion vector is given), the block activity and the value of the evaluation value are distributed in the region R1 below the curve 621 in the figure. On the other hand, if an incorrect motion (incorrect motion vector) is given, the block activity and the evaluation value are distributed in the region R2 on the left side of the figure from the curve 622 (the region above the curve 622). There is almost no distribution in the region other than R2 and the region other than the region R1 below the curve 621). Curves 621 and 622 intersect at point P. The value of the block activity at this point P is set as the threshold value THa. The threshold value THa means that if the value of the block activity is smaller than that, the corresponding motion vector may be incorrect (this will be described in detail later). The threshold determination unit 603 outputs a flag indicating whether the block activity value input from the activity calculation unit 602 is greater than the threshold THa to the integration processing block 605.

  In step S605, the normalization processing unit 604 executes normalization processing. Specifically, the normalization processing unit 604 calculates the motion vector accuracy VC according to the following equation.

  VC = 1-evaluation value / block activity (5)

  However, if the value of the motion vector accuracy VC is less than 0, the value is replaced with 0. Of the motion vector accuracy VC, the value obtained by dividing the evaluation value by the block activity is a straight line 623 where the position on the graph of FIG. 50 defined by the value is the slope connecting the origin O and the point P is 1. Therefore, it indicates whether the region is in the lower region or the upper region in the figure. That is, if the slope of the straight line 623 is 1, and the value obtained by dividing the evaluation value by the block activity is larger than 1, the points corresponding to the value are points distributed in the area above the straight line 623. Means that. The motion vector accuracy VC obtained by subtracting this value from 1 means that the smaller the value, the higher the possibility that the corresponding points are distributed in the region R2.

  On the other hand, if the value obtained by dividing the evaluation value by the block activity is smaller than 1, it means that the points corresponding to the value are distributed in the lower area of the straight line 623 in the figure. The motion vector accuracy VC at that time means that the larger the value (closer to 0), the corresponding points are distributed in the region R1. The normalization processing unit 604 outputs the motion vector accuracy VC obtained by the calculation in this way to the integration processing unit 605.

  In step S606, the integration processing unit 605 executes integration processing. Details of this integration processing are shown in the flowchart of FIG.

  In step S631, the integration processing unit 605 determines whether the block activity is equal to or less than the threshold value THa. This determination is performed based on the flag supplied from the threshold determination unit 603. If the block activity is equal to or less than the threshold THa, the integration processing unit 605 sets the value of the motion vector accuracy VC calculated by the normalization processing unit 604 to 0 in step S632. If it is determined in step S631 that the activity value is greater than the threshold value THa, the processing in step S632 is skipped, and the value of the motion vector accuracy VC generated by the normalization processing unit 604 is output as it is together with the motion vector. Is done.

  This is because there is a possibility that a correct motion vector is not obtained if the block activity value is smaller than the threshold value THa even if the value of the motion vector accuracy VC calculated by the normalization processing unit 604 is positive. Because there is. That is, as shown in FIG. 50, between the origin O and the point P, the curve 622 protrudes below the curve 621 in the drawing (below the straight line 623). In a region R3 in which the value of the block activity is smaller than the threshold value Tha and surrounded by the curves 621 and 622, the value obtained by dividing the evaluation value by the block activity is distributed in both the regions R1 and R2. There is a high possibility that a correct motion vector is not obtained. Therefore, in such a distribution state, processing is performed assuming that the accuracy of the motion vector is low. For this reason, in step S632, the motion vector accuracy VC is set to 0 if the value is positive even if the value is positive. In this way, when the value of the motion vector accuracy VC is positive, it is possible to reliably represent that the correct motion vector is obtained. In addition, the larger the value of the motion vector accuracy VC, the higher the probability that a correct motion vector is obtained (the probability that the distribution is included in the region R1 increases).

  This coincides with an empirical rule that, in general, it is difficult to detect a motion vector with high reliability in a region where the luminance change is small (region where the activity is small).

  FIG. 52 illustrates a configuration example of the background motion estimation unit 54 of FIG. In this configuration example, the background motion estimator 54 includes a frequency distribution calculator 651 and a background motion determiner 652.

  The frequency distribution calculation unit 651 calculates a frequency distribution of motion vectors. However, this frequency is weighted so that a likely motion is weighted by using the motion vector accuracy VC supplied from the motion estimation unit 12. Based on the frequency distribution calculated by the frequency distribution calculation unit 651, the background motion determination unit 652 performs a process of determining a motion with the maximum frequency as a background motion, and outputs the background motion to the region estimation related processing unit 55.

  The background motion estimation process of the background motion estimation unit 54 will be described with reference to FIG.

  In step S651, the frequency distribution calculation unit 651 calculates a motion frequency distribution. Specifically, the frequency distribution calculation unit 651 assumes that the x and y coordinates of the motion vector as a background motion candidate are expressed in a range of ± 16 pixels from the reference point, respectively (1089 (= 16 × 2 + 1) × (16 × 2 + 1)), that is, a box for coordinates corresponding to possible values of a motion vector, and when a motion vector is generated, 1 is added to the coordinates corresponding to the motion vector. In this way, the motion vector frequency distribution can be calculated.

  However, when one motion vector is generated, if 1 is added, if the frequency of occurrence of a motion vector with low accuracy is high, a motion vector with low certainty may be determined as the background motion. . Therefore, when a motion vector is generated, the frequency distribution calculation unit 651 does not add a value 1 to a box (coordinates) corresponding to the motion vector, but a value obtained by multiplying the value 1 by the motion vector accuracy VC (= Motion vector accuracy VC). The value of the motion vector accuracy VC is normalized as a value between 0 and 1, and the closer the value is to 1, the higher the accuracy. Therefore, the frequency distribution obtained in this way is a frequency distribution obtained by weighting motion vectors based on their accuracy. This reduces the risk that a motion with low accuracy is determined as the background motion.

  Next, in step S652, the frequency distribution calculation unit 651 determines whether or not the process of calculating the motion frequency distribution has been completed for all blocks. If there is a block that has not yet been processed, the process returns to step S651, and the process of step S651 is executed for the next block.

  As described above, the motion frequency distribution calculation process is performed on the entire screen, and when it is determined in step S652 that the processing of all blocks has been completed, the process proceeds to step S653, and the background motion determination unit 652 A process for searching for the maximum value of the distribution is executed. That is, the background motion determination unit 652 selects the frequency having the maximum frequency from the frequencies calculated by the frequency distribution calculation unit 651, and determines the motion vector corresponding to the frequency as the motion vector of the background motion. The motion vector of the background motion is supplied to the region estimation related processing unit 55, and is used for, for example, a determination process of whether or not the full screen motion and the background motion in step S204 in FIG. 20 and step S231 in FIG. .

  FIG. 54 shows a detailed configuration example of the scene change detection unit 53 of FIG. In this example, the scene change detection unit 53 is configured by the motion vector accuracy average calculation unit 671 and the threshold determination unit 672.

  The motion vector accuracy average calculation unit 671 calculates the average value of all the screens of the motion vector accuracy VC supplied from the motion estimation unit 52 and outputs the average value to the threshold determination unit 672. The threshold value determination unit 672 compares the average value supplied from the motion vector accuracy average calculation unit 671 with a predetermined threshold value, and determines whether it is a scene change based on the comparison result. The result is output to the control unit 59.

  Next, the operation of the scene change detection unit 53 will be described with reference to the flowchart of FIG. In step S681, the motion vector accuracy average calculation unit 671 calculates the sum of vector accuracy. Specifically, the motion vector accuracy average calculation unit 671 executes a process of adding the value of the motion vector accuracy VC calculated for each block output from the integration processing unit 605 of the motion estimation unit 52. In step S682, the motion vector accuracy average calculation unit 671 determines whether or not the processing for calculating the sum of the vector accuracy VC has been completed for all the blocks. If the processing has not yet been completed, the processing of step S681 is repeated. . By repeating this process, the sum of motion vector accuracy VC of each block for one screen is calculated. If it is determined in step S682 that the calculation process of the sum of motion vector accuracy VCs for all one screen has been completed, the process advances to step S683, and the motion vector accuracy average calculation unit 671 calculates the average value of the vector accuracy VC. Execute. Specifically, a value obtained by dividing the sum of the vector accuracy VC for one screen calculated in the process of step S681 by the number of added blocks is calculated as an average value.

  In step S684, the threshold value determination unit 672 compares the average value of the motion vector accuracy VC calculated by the motion vector accuracy average calculation unit 671 in the process of step S683 with a preset threshold value, and whether or not the threshold value is smaller than the threshold value. Determine whether. In general, when a scene change occurs between two frames having different times in a moving image, there is no corresponding image, so even if a motion vector is calculated, the motion vector is not certain. Therefore, if the average value of the vector accuracy VC is smaller than the threshold value, the threshold value determination unit 672 turns on the scene change flag in step S685. If the average value of the vector accuracy VC is not smaller than the threshold value (if it is greater than or equal to the threshold value), in step S586 Turn off the change flag. When the scene change flag is on, it indicates that there is a scene change, and when the scene change flag is off, it indicates that there is no scene change.

  This scene change flag is supplied to the control unit 59, and is used for determining whether or not there is a scene change in step S421 in FIG. 38 and for determining whether or not there is a scene change in step S462 in FIG.

  As described above, by configuring the object tracking unit 26 in FIG. 1, the object 51 (FIG. 3) to be tracked rotates, occlusion occurs, or the tracking point 51A of the object 51 is changed by a scene change. Even when the image is temporarily not displayed, the moving object 51 (tracking point 51A) in the image can be accurately tracked.

  The position information of the tracking point 51A of the object 51 tracked in this way is output to the area setting unit 25 as a tracking result by the object tracking unit 26 in FIG. A correction target area 52 is set. Then, the image correction unit 22 removes the blur (focus blur) of the image in the correction target area 52.

  Next, a detailed configuration example and operation of the image correction unit 22 in FIG. 1 will be described. FIG. 56 is a block diagram illustrating a detailed configuration example of the image correction unit 22. In this example, the image correction unit 22 generates a control signal based on the output signal of the area setting unit 25 and supplies the control signal to each unit. The image feature detection unit detects the feature of the input image. 742, an address calculation unit 743 that calculates an address based on the control signal, a coefficient ROM 744 that outputs a predetermined coefficient stored in advance based on the address calculated by the address calculation unit 743, and an input image A region extraction unit 745 that extracts a plurality of pixels corresponding to a predetermined region is provided.

  Also, a product-sum operation unit 746 that performs a product-sum operation based on the coefficient output from the coefficient ROM 744 on the pixel level output from the region extraction unit 745 and outputs a newly corrected pixel level, and a product An image composition unit 747 that synthesizes and outputs the image in the correction target area 52 and the background 53 based on the output result of the sum calculation unit 746 and the control signal is provided.

  FIG. 57 is a diagram illustrating an example of a control signal generated by the control signal generation unit 741. The control signal A is a signal that specifies a portion to be corrected (correction target area 52) in the input image, is generated based on the output of the area setting unit 25, and is supplied to the region extraction unit 745 and the image composition unit 747. Is done. The control signal B is a signal that specifies a parameter σ representing the degree of blur described later, and is supplied to the address calculation unit 743. The value of the parameter σ may be specified based on a user's designation performed via the control unit 27, or may be set in advance.

  The control signal C is a signal for designating switching of the weight Wa of the relational expression used for solving a blur model expression described later, and is supplied to the address calculation unit 743. The control signal D is a signal that specifies switching of a threshold value used when detecting the feature of the image, and is supplied to the image feature detection unit 742. The control signals C and D may be set in advance in consideration of the characteristics of the monitoring camera system 1 or the like, or may be generated on the basis of a user's designation performed via the control unit 27. May be.

  Next, the principle of image blur will be described. Now, the pixel level X of the image in which the camera is properly set and the subject is not out of focus is set as a true value, and the pixel level Y of the image in which the subject is out of focus and the subject is out of focus is set as an observation value. . In order to express a plurality of pixels constituting an image, if the horizontal coordinate of the image is represented by x and the vertical coordinate is represented by y, the true value is represented by X (x, y), and the observed value Can be represented by Y (x, y).

  In the present invention, Expression (6) is applied as a blur model expression. In equation (6), the observed value Y (x, y) is obtained by convolution of the Gaussian function with the true value X (x, y) using the Gaussian function shown in equation (7).

  In equation (6), the parameter σ is a parameter representing the degree of blur.

  According to the equation (6), one observation value Y (x, y) is a plurality of true values X (x + i, r) that vary depending on variables i and j (−r <i <r, −r <j <r). y + j) is obtained by weighting with a coefficient W. Therefore, the level of one pixel of the image having no blur is obtained based on the levels of a plurality of pixels of the blurred image.

  Further, the degree of blurring of the image changes depending on the value of the parameter σ described above. When the value of the parameter σ is relatively small, the true value information is not diffused over a wide range in the observed value, and the image is relatively blurred. On the other hand, when the value of the parameter σ is relatively large, true value information is diffused over a wide range in the observed value, resulting in an image with relatively large blur.

  As described above, the degree of blurring of the image changes due to the change in the value of the parameter σ. For this reason, in order to correct the blur of the image accurately, it is necessary to appropriately obtain the value of the parameter σ. In the present invention, the user specifies the value of the parameter σ. Alternatively, an optimum value may be set in advance in consideration of the characteristics of the monitoring camera system 1 and the like.

  The principle of image blur will be described in more detail with reference to FIGS. FIG. 58A is a diagram showing true values X0 to X8 in an image on the assumption that pixels are arranged one-dimensionally in the horizontal direction for the sake of simplicity. FIG. 58C is a diagram illustrating observation values corresponding to FIG. 58A. FIG. 58B is a graph showing the magnitude of the coefficient W (i) in a bar graph form. In this example, the variable i is set to −2 <i <2, the center bar graph is set to the coefficient W (0), and the coefficients W (−2), W (−1), W in order from the leftmost bar graph. (0), W (1), and W (2).

  Here, based on the equation (6), the observed value Y2 in FIG. 58C is obtained as follows.

  Y2 = W (-2) X2 + W (-1) X3 + W (0) X4 + W (1) X5 + W (2) X6

  Similarly, when the observed value Y0 in FIG. 58C is obtained, the observed value Y0 is obtained as follows by performing an operation based on the portion indicated by the frame 790-1 in FIG. 59 in the true value. It is done.

  Y0 = W (-2) X0 + W (-1) X1 + W (0) X2 + W (1) X3 + W (2) X4

  Further, when the observed value Y1 is obtained, the observed value Y1 is obtained as follows by performing an operation based on the portion indicated by a frame 790-2 in FIG.

  Y1 = W (-2) X1 + W (-1) X2 + W (0) X3 + W (1) X4 + W (2) X5

  Y3 and Y4 can be similarly determined.

  FIGS. 60 and 61 are two-dimensional representations of the relationship between FIGS. 58A and 58C. That is, the level of each pixel constituting FIG. 60 is an observed value, and the level of each pixel constituting FIG. 61 is obtained as a true value. In this case, the observed value Y (x, y) corresponding to the pixel A in FIG. 60 is obtained as follows.

(Y (x, y) = W (-2, -2) X (x-2, y-2) + W (-1, -2) X (x-1, y-2) + W (0, .2) X (x, y-2) ... + W (2, 2) X (x + 2, y + 2)

  That is, the observed values corresponding to the pixel A in FIG. 60 are true values corresponding to 25 (= 5 × 5) pixels indicated by the frame a with the pixel A ′ (corresponding to the pixel A) in FIG. 61 as the center. Based on. Similarly, the observation values corresponding to the pixel B in FIG. 60 (the pixel on the right side of the pixel A in the drawing) correspond to 25 pixels centered on the pixel B ′ (corresponding to the pixel B) in FIG. The observed value corresponding to the pixel C in FIG. 60 is based on the true value corresponding to 25 pixels centered on the pixel C ′ (corresponding to the pixel C) in FIG. Is required. Equations for obtaining observed values Y (x + 1, y) and Y (x + 2, y) corresponding to pixels B and C in FIG. 60 are as follows.

Y (x + 1, y) = W (-2, -2) X (x-1, y-2) + W (-1, -2) X (x, y-2) + W (0,- 2) X (x-1, y-2) ... + W (2,2) X (x + 3, y + 2)

Y (x + 2, y) = W (-2, -2) X (x, y-2) + W (-1, -2) X (x + 1, y-2) + W (0,- 2) X (x + 2, y-2) ... + W (2,2) X (x + 4, y + 2)

  In this way, when the observation values corresponding to the respective pixels in FIG. 60 are obtained, determinants as shown in Expressions (8) to (11) are obtained.

  Here, in the determinant shown in Expression (11), if the inverse matrix of the matrix Wf can be obtained, the true value Xf can be obtained based on the observed value Yf. That is, based on the pixels of the blurred image, the pixels of the image without blur can be obtained, and the blurred image can be corrected.

  However, as described above with reference to FIGS. 58 to 61, the determinants shown in the equations (8) to (11) have more true value pixels than the observed value pixels. A matrix cannot be obtained (for example, in the example of FIG. 59, five true-value pixels are required for one observed-value pixel).

  Therefore, in addition to the expressions (8) to (11), the relational expressions shown in the expressions (12) to (15) are introduced.

  Expressions (12) to (15) limit the difference between the levels of adjacent pixels, and the true value to be obtained is a flat part of the image (no significant difference from the level of adjacent pixels). In some cases there is no contradiction. However, when the true value to be obtained is an edge portion (there is a large difference from the level of adjacent pixels), a contradiction occurs, and the corrected image may be deteriorated. For this reason, in order to appropriately correct a blurred image, it is necessary to properly use the four relational expressions (12) to (15) for each pixel so as not to cross the true edge portion.

  Therefore, the image feature detection unit 742 determines an edge portion and a flat portion in the input image, and generates a code p2 that indicates in which direction (for example, up, down, left, and right) flat. The detailed operation of the image feature detection unit 742 will be described later with reference to FIG. In the present invention, it is assumed that the determination result of the edge portion and the flat portion in the input image (observed value) is equal to the determination result of the edge portion and the flat portion in the true value.

  In Expressions (12) to (15), functions W1 to W4 that are functions of the code p2 are weight functions. In the present invention, the weighting functions W1 to W4 are controlled according to the code p2, so that the relational expressions for each pixel are properly used. FIG. 62 shows the values of the weight functions W1 to W4 corresponding to the code p2. When the value of the weight function is large, the meaning of being flat in the equations (12) to (15) is strong, and when the value of the weight function is small, the meaning is weak (meaning that is an edge is strong). .

  The code p2 is composed of 4 bits, and each bit indicates whether it is flat in the upper, right, lower or left direction in order from the left, and when it is flat in that direction, The corresponding bit is set to “1”. For example, when the code p2 is “0001”, it indicates that the pixel is flat in the left direction of the target pixel and the other directions are not flat (there is an edge). For this reason, when the code p2 is “0001”, the value of the weighting function W4 increases, and the weight of Expression (15) among the four relational expressions of Expressions (12) to (15) increases. In this way, the weights of the four relational expressions can be changed by the code p2, and the four relational expressions can be used properly for each pixel so as not to cross the edge.

  For example, as shown in FIG. 63, when the upper direction and the left direction of the target pixel Xa are flat and the right direction and the lower direction are edges, 4 of Expressions (12) to (15) is expressed by the code p2. By changing the weights of two relational expressions, the difference between adjacent pixel levels is limited to “Xa−Xb = 0” and “Xa−Xc = 0”, but “Xa−Xd = 0”, The limitation “Xa−Xe = 0” is not added. Xb, Xc, Xd, and Xe represent pixels adjacent to the right, bottom, top, or left of the target pixel Xa, respectively.

  In Expressions (12) to (15), the function Wa is another weight function, and the value of the weight function Wa changes according to the code p1. By changing the value of the weight function Wa, it is possible to control the noise and details of the entire corrected image. When the value of the weight function Wa is large, the influence of noise is felt small in the corrected image, and the noise feeling is reduced. Further, when the value of the weight function Wa is small, it is felt that the detail is emphasized in the corrected image, and the feeling of detail is improved. Note that the code p1 for changing the value of the weight function Wa corresponds to the control signal C in FIG.

  Thus, in addition to the equations (8) to (11), the relational expressions shown in the equations (12) to (15) are introduced. Thereby, it becomes possible to calculate an inverse matrix as shown in Expression (16), and as a result, a true value can be obtained based on the observed value.

  In the present invention, the coefficient Ws-1 related to the observed value Ys is stored in the coefficient ROM 744 in advance, and the determinant (product-sum) operation of Expression (16) is performed on the input image extracted by the region extracting unit 745. This is performed by the product-sum operation unit 746. By doing so, it is not necessary to perform an inverse matrix operation every time an image is corrected, and it is possible to correct blur only by a product-sum operation. However, since the parameter σ and the above-described four relational expressions are different depending on the input image, inverse matrix calculation is performed in advance for all possible combinations thereof, and the parameter σ, the code p2, and the like are supported. An address is determined, and a coefficient different for each address is stored in the coefficient ROM 744.

  However, for example, when the combination of the weight functions W1 to W4 is changed in all 25 (= 5 × 5) pixels in the frame (t) shown in FIG. (= Combination of functions W1 to W4 shown in FIG. 62) is a combination of 25 (the number of pixels in the frame (t)), and when the inverse matrix operation is performed for each combination, the number of coefficients is enormous. Therefore, since the capacity of the coefficient ROM 744 is limited, all the coefficients may not be stored. In such a case, for only Xt that is the central pixel in the frame (t), the relational expression is switched by changing the code p2 based on the feature, and the relational expression of the pixels other than the pixel Xt in the frame (t) is changed. For example, the code p2 may be fixed to “1111” in a pseudo manner. By doing so, the combinations of coefficients can be limited to 15.

  In the above, in order to explain the principle of blur (model equation), the domain of the Gaussian function is set to -2≤ (x, y) ≤2, but in reality the value of parameter σ is sufficiently large A range that can be handled is set. In addition, the relational expressions shown in Expressions (12) to (15) are not limited to these as long as they are expressions describing image characteristics. Further, as an example in the case where the capacity of the coefficient ROM 744 is limited, the example in which the relational expression is switched only to the blur center phase (Xt) has been described. However, the present invention is not limited to this, and the capacity of the coefficient ROM 744 is limited. The relational expression switching method may be changed accordingly.

  Next, the blur correction process by the image correction unit 22 will be described with reference to FIG. In step S801, the image correction unit 22 detects a processing target area. This processing target region is a region where correction of blur is to be performed, that is, a correction target area 52, and is detected based on a signal output from the area setting unit 25.

  In step S802, the image correction unit 22 acquires the value of the parameter σ. The parameter σ may be specified by the user, or a preset value may be acquired. In step S803, the image correction unit 22 performs an image correction process described later with reference to FIG. Thereby, the blurred image is corrected and output.

  In this way, the image in the correction target area 52 becomes a clear image by removing the blur of the image.

  Next, details of the image correction processing in step S803 in FIG. 64 will be described with reference to FIG.

  In step S821, the image feature detection unit 742 executes an image feature detection process described later with reference to FIG. Thereby, it is determined in which direction the target pixel is flat, and the code p2 described above with reference to FIG. 62 is generated and output to the address calculation unit 743.

  In step S822, the address calculation unit 743 calculates the address of the coefficient ROM 744. The address of the coefficient ROM 744 is, for example, 4 bits corresponding to the code p2 (output of the image feature detecting unit 742), 4 bits indicating the value of the parameter σ (control signal B in FIG. 57), and the above-described four relational expressions. 2 (corresponding to the control signal C in FIG. 57) corresponding to the code p1 for switching the weight function Wa of 1024 (2 to the 10th power) addresses. The address calculation unit 743 calculates a corresponding address based on the output of the image feature detection unit 742, the control signal B, and the control signal C.

  In step S823, the address calculation unit 743 reads a coefficient from the coefficient ROM 744 based on the address calculated in step S822 and supplies the coefficient to the product-sum calculation unit 746.

  In step S824, the product-sum operation unit 746 performs product-sum operation for each pixel based on the coefficient read in step S823, and outputs the result to the post-processing unit 747. Thereby, as described above, the true value is obtained from the observed value, and the blurred image is corrected.

  In step S825, the image composition unit 747 executes image composition processing described later with reference to FIG. Thus, it is determined for each pixel whether the processing result of the product-sum operation unit 746 is output or the input image is output as it is. In step S826, the post-processing unit 747 performs post-correction processing and outputs the selected image.

  Next, the image feature detection processing in step S821 in FIG. 65 will be described with reference to FIG. In step S841, the image feature detection unit 742 extracts blocks, and in step S842, calculates a difference between the blocks extracted in step S841 (details will be described later with reference to FIG. 68). In step S843, the image feature detection unit 742 compares the block difference calculated in step S842 with a preset threshold, and based on the comparison result, in step S844, sets a flat direction with respect to the target pixel. The code p2 which is the code to represent is output.

  With reference to FIGS. 67 and 68, the image feature detection processing will be described in more detail. FIG. 67 is a block diagram illustrating a detailed configuration example of the image feature detection unit 742. On the left side of the figure, block cutout units 841-1 to 841-5 for extracting predetermined blocks from the input image are provided. For example, as illustrated in FIGS. 68A to 68E, the block cutout units 841-1 to 841-5 include the target pixel around the target pixel (the pixel to be corrected) indicated by a black circle in the drawing. Five blocks composed of (= 3 × 3) pixels are extracted.

  A block 881 shown in FIG. 68A is a central block having a target pixel at the center thereof, and is extracted by the block cutout unit 841-5. A block 882 shown in FIG. 68B is an upper block obtained by moving the block 881 upward in the drawing by one pixel, and is extracted by the block cutout unit 841-3. A block 883 shown in FIG. 68C is a left side block obtained by moving the block 881 to the left in the drawing by one pixel, and is extracted by the block cutout unit 841-4.

  A block 884 shown in FIG. 68D is a lower block obtained by moving the block 881 downward in the drawing by one pixel, and is extracted by the block cutout unit 841-1. A block 885 shown in FIG. 68E is a right block obtained by moving the block 881 to the right in the drawing by one pixel, and is extracted by the block cutout unit 841-2. In step S841, five blocks 881 to 885 are extracted for each pixel of interest.

  Information on the pixels constituting each block extracted by the block cutout units 841-1 to 841-5 is output to the block difference calculation units 842-1 to 842-4. The block difference calculation units 842-1 to 842-4 calculate the pixel difference of each block as follows, for example.

  Now, the three pixels (levels) in the top row among the nine pixels of the block 881 are a (881), b (881), and c (881) from the left. The three pixels in the center row are d (881), e (881), and f (881) from the left. Let the three pixels in the bottom row be g (881), h (881), and i (881) from the left. Similarly, for the nine pixels in the block 884, the three pixels (levels) in the top row are a (884), b (884), and c (884) from the left, The three pixels are d (884), e (884), f (884) from the left, and the three pixels in the top row are g (884), h (884), i (884) from the left. And The block difference calculation unit 842-1 calculates the block difference B (1) as follows.

  B (1) = │a (881) -a (884) │ ＋ │b (881) -b (884) │ ＋ │c (881) -c (884) │ ＋ ・ ・ ・ + │i (881) -i (884) │

  That is, the block difference B (1) is the sum of absolute values of the difference in level of each corresponding pixel in the block 881 (center) and the block 884 (bottom). Similarly, the block difference calculation unit 842-2 calculates a block difference B (2) by calculating the sum of absolute values of the differences between the levels of the corresponding pixels in the block 881 (center) and the block 885 (right). . Further, the block difference calculation unit 842-3 corresponds to the block 881 (center) and the block 882 (upper), and the block difference calculation unit 842-3 corresponds to each of the block 881 (center) and the block 883 (left). The sum of absolute values of pixel level differences is obtained, and block differences B (3) and B (4) are calculated.

  In step S842, the block differences B (1) to B (4), which are the differences between the central block and the four blocks in the up, down, left, and right directions, are calculated in this way, and the result is the corresponding threshold value determination unit 843-1. To 843-4, and simultaneously supplied to the minimum direction determination unit 844.

  The threshold value determination units 843-1 to 843-4 compare the block differences B (1) to B (4) with preset threshold values, respectively, and determine their magnitudes. The threshold value is switched based on the control signal D. The threshold value determination units 843-1 to 843-4 determine that the direction is an edge portion and output “0” when the block differences B (1) to B (4) are larger than a preset threshold value, respectively. If it is smaller than the threshold, it is determined that the direction is a flat portion, and “1” is output.

  In step S843, the block difference is compared with the threshold value in this way. The output results of the threshold determination units 843-1 to 843-4 are output to the selector 845 as a 4-bit code. For example, when the block differences B (1), B (3), and B (4) are smaller than the threshold and the block difference B (2) is larger than the threshold, “1011” is output as the code.

  By the way, the case where the block differences B (1) to B (4) are all larger than the threshold value (when there is no flat portion) is also conceivable. In this case, “0000” is output as a code from the threshold determination units 843-1 to 843-4. However, as shown in FIG. 62, when the code p2 is “0000”, the corresponding weight functions W1 to W4 cannot be specified. Therefore, the selector 845 determines whether the output results from the threshold determination units 843-1 to 843-4 are “0000”, and the output results from the threshold determination units 843-1 to 843-4 are “0000”. When it is determined that there is, the output from the minimum direction determination unit 844 is output as the code p2.

  The minimum direction determination unit 844 determines a minimum value among the block differences B (1) to B (4), and uses a 4-bit code corresponding to the determination result as a threshold determination unit 843-1 to 843-4. Is output to the selector 845 at the same timing as that outputs the code. For example, when it is determined that B (1) is the smallest among the block differences B (1) to B (4), the minimum direction determination unit 844 outputs “1000” as a code to the selector 845.

  In this way, even if the code “0000” is output from the threshold determination units 843-1 to 843-4, the code “1000” output from the minimum direction determination unit 844 is output as the code p2. can do. Of course, when the output results from the threshold determination units 843-1 to 843-4 are not “0000”, the output results from the threshold determination units 843-1 to 843-4 are output as the code p2. In step S844, the code p2 is generated in this way and output to the address calculation unit 743.

  Next, the image composition processing in step S825 in FIG. 65 will be described with reference to FIG. In step S <b> 861, the image composition unit 747 calculates the degree of pixel dispersion based on the output result from the product-sum operation unit 746. Thereby, the degree of dispersion of pixels around the target pixel is calculated. In step S862, the image composition unit 747 determines whether the degree of dispersion calculated in step S862 is greater than a preset threshold value.

  If it is determined in step S862 that the degree of dispersion is greater than the threshold, the image composition unit 747 sets the input image replacement flag corresponding to the target pixel to ON in step S863. On the other hand, if it is determined that the degree of dispersion is not greater than the threshold, the image composition unit 747 sets the input image replacement flag corresponding to the target pixel to OFF in step S64.

  If the product-sum operation is performed by the product-sum operation unit 746 on a pixel in the input image that is not originally blurred, the activity of the image around the pixel increases, and the image may deteriorate instead. . Here, when the degree of dispersion is larger than the threshold value, it is determined that the pixel is a deteriorated pixel, and the input image replacement flag is set to ON. When the pixel whose input image replacement flag is set to ON is output, it is replaced with the pixel of the input image (returned to the original state) and output.

  In step S865, the image composition unit 747 determines whether or not all the pixels have been checked. If it is determined that all the pixels have not been checked yet, the process returns to step S861, and the subsequent processing is repeatedly executed. The When it is determined in step S865 that all the pixels have been checked, in step S866, the image composition unit 747 corrects the image in the correction target area 52 from which the image is corrected and the blur is removed, and the image of the background 53. The images are combined and output to the image display 23.

  In this way, it is determined for each pixel whether the product-sum operation result is output or the pixel of the input image is output as it is. By doing so, it is possible to prevent the image from being deteriorated by correcting the image with respect to the originally unblurred portion of the input image.

  This point will be described in more detail with reference to FIGS. 70 and 71. FIG. FIG. 70 is a block diagram illustrating a configuration example of the image composition unit 747. The output result from the product-sum operation unit 746 is input to the block cutout unit 901, and the block cutout unit 901 has 9 (= 3 × 3) pixels centered on the pixel of interest a5 as shown in FIG. A1 to a9 are cut out and output to the variance calculation unit 802. The variance calculation unit 802 calculates the degree of dispersion as follows.

  Here, m represents an average value of (levels) of nine pixels in the block, v is a sum of squares of differences from the average values of the respective pixels, and the degree of dispersion of the pixels in the block Represents. In step S861, the degree of dispersion is calculated in this way, and the calculation result is output to the threshold value determination unit 903.

  The threshold determination unit 903 compares the output result (dispersion degree) from the variance calculation unit 902 with a preset threshold value. If it is determined that the dispersion degree is greater than the threshold value, the image composition unit 747 corresponds to the target pixel. The selection unit 904 is controlled to set the input image replacement flag to be turned ON. If it is determined that the degree of dispersion is not greater than the threshold value, the image composition unit 747 controls the selection unit 804 to set the input image replacement flag corresponding to the target pixel to OFF. In steps S862 to S864, in this way, it is determined whether or not the degree of dispersion is greater than the threshold value, and the input image replacement flag is set based on the determination result.

  Then, the switching unit 905 switches the final processing result by the selection unit 904 and the pixel of the input image and outputs the result. That is, the pixels of the image in the correction target area 52 are set as the final processing result by the selection unit 904, and the pixels of the image of the background 53 are switched to become the pixels of the input image.

  In this way, the object 51 (FIG. 3) is tracked, and only the image in the correction target area 52 including the object 51 is corrected (corrected) so that the blur of the image is removed and displayed clearly. On the other hand, since the image of the background 53 is displayed without removing the blur of the image, the user can naturally focus on the moving object 51.

  In the above description, the example in which the image correction unit 22 corrects the image in the correction target area 52 in the image captured by the imaging unit 21 so as to remove the image blur has been described. The correction unit 22 does not remove the blur of the image, and for example, the setting of the brightness and color of each pixel constituting the image is changed and the highlight image is simply highlighted for the image in the correction target area 52. As described above, the image may be corrected. By doing so, the user may not be able to visually recognize the object 51 accurately, but still, the user can naturally make the user pay attention to the moving object 51, and the image is blurred. Compared with the case where correction is made so as to be removed, the image correction unit 22 can have a simpler configuration, and as a result, the surveillance camera system 1 can be realized at a lower cost.

  Next, the image processing apparatus to which the object tracking apparatus described above is applied will be further described. FIG. 72 shows an example where the object tracking device is applied to the television receiver 1000. The tuner 1001 receives an RF signal, demodulates it and separates it into an image signal and an audio signal, outputs the image signal to the image processing unit 1002, and outputs the audio signal to the audio processing unit 1007.

  The image processing unit 1002 demodulates the image signal input from the tuner 1001 and outputs the demodulated signal to the object tracking unit 1003, the zoom image creation unit 1004, and the selection unit 1005. The object tracking unit 1003 has substantially the same configuration as the object tracking device 26 in FIG. 5 described above. The object tracking unit 1003 executes processing for tracking the tracking point of the object designated by the user from the input image, and outputs coordinate information regarding the tracking point to the zoom image creation unit 1004. The zoom image creation unit 1004 creates a zoom image centered on the tracking point and outputs it to the selection unit 1005. The selection unit 1005 selects one of the image supplied from the image processing unit 1002 or the image supplied from the zoom image creation unit 1004 based on an instruction from the user, and outputs the selected image to the image display 1006 for display.

  The audio processing unit 1007 demodulates the audio signal input from the tuner 1001 and outputs it to the speaker 1008.

  The remote controller 1010 is operated by the user and outputs a signal corresponding to the operation to the control unit 1009. The control unit 1009 is configured by a microcomputer, for example, and controls each unit based on a user instruction. The removable medium 1011 includes a semiconductor memory, a magnetic disk, an optical disk, a magneto-optical disk, and the like. The removable medium 1011 is mounted as necessary, and provides the control unit 1009 with programs and various other data.

  Next, processing of the television receiver 1000 will be described with reference to the flowchart of FIG.

  In step S901, the tuner 1001 demodulates a signal of a channel designated by the user from an RF signal received via an antenna (not shown), outputs the image signal to the image processing unit 1002, and performs audio processing on the audio signal. Output to the unit 1007. The audio signal is demodulated by the audio processing unit 1007 and then output from the speaker 1008.

  The image processing unit 1002 demodulates the input image signal and outputs the demodulated image signal to the object tracking unit 1003, the zoom image creation unit 1004, and the selection unit 1005.

  In step S902, the object tracking unit 1003 determines whether tracking is instructed. If it is determined that tracking is not instructed, the processing of steps S903 and S904 is skipped. In step S 905, the selection unit 1005 selects one of the image signal supplied from the image processing unit 1002 and the image signal input from the zoom image creation unit 1004 based on control from the control unit 1009. In this case, since there is no specific instruction from the user, the control unit 1009 causes the selection unit 1005 to select the image signal from the image processing unit 1002. In step S906, the image display 1006 displays the image selected by the selection unit 1005.

  In step S907, the control unit 1009 determines whether to end the image display process based on a user instruction. That is, when ending the image display process, the user operates the remote controller 1010 and instructs the control unit 1009 to do so. If the user has not instructed termination, the process returns to step S901, and the subsequent processes are repeatedly executed.

  As described above, the normal processing for displaying the image corresponding to the signal received by the tuner 1001 as it is is executed.

  When an image desired to be tracked is displayed by looking at the image displayed on the image display 1006, the user designates the image by operating the remote controller 1010. When this operation is performed, in step S902, the control unit 1009 determines that tracking is instructed, and controls the object tracking unit 1003. Based on this control, the object tracking unit 1003 starts tracking processing of the tracking point specified by the user. This process is the same as the process of the object tracking device 26 described above.

  In step S <b> 904, the zoom image creation unit 1004 generates a zoom image centered on the tracking point tracked by the object tracking unit 1003 and outputs the zoom image to the selection unit 1005.

  This zoom image creation processing can be performed using the class classification adaptation processing previously proposed by the present applicant. For example, Japanese Patent Laid-Open No. 2002-196737 discloses a process for converting a 525i signal into a 1080i signal using a coefficient obtained by learning in advance. This process is substantially the same as the process of enlarging the image 9/4 times in both the vertical direction and the horizontal direction. However, since the image display 1006 has a fixed number of pixels, the zoom image creation unit 1004 converts the 525i signal into a 1080i signal and then centers the tracking point when creating a 9 / 4-fold image, for example. A zoom image can be created by selecting a predetermined number of pixels (a number of pixels corresponding to the image display 1006). The process of reducing is the reverse process.

  Based on this principle, a zoom image with an arbitrary magnification can be generated.

  If tracking is instructed, the selection unit 1005 selects the zoom image created by the zoom image creation unit 1004 in step S905. As a result, in step S906, the image display 1006 displays the zoom image created by the zoom image creation unit 1004.

  As described above, a zoom image centered on the tracking point designated by the user is displayed on the image display 1006. When the magnification is set to 1, only tracking is performed.

  FIG. 74 shows an example where the present invention is applied to a surveillance camera system. In this surveillance camera system 1100, an image captured by an imaging unit 1101 made up of a CCD video camera or the like is displayed on an image display 1102. The tracking target detection unit 1103 detects the tracking target from the image input from the imaging unit 1101, and outputs the detection result to the object tracking unit 1105. The object tracking unit 1105 operates to track the tracking target designated by the tracking target detection unit 1103 in the image supplied from the imaging unit 1101. The object tracking unit 1105 has basically the same configuration as the object tracking device 26 of FIG. 1 described above. Based on the control from the object tracking unit 1105, the camera driving unit 1104 drives the imaging unit 1101 so that the imaging unit 1101 captures an image centered on the tracking target tracking point.

  The control unit 1106 is configured by, for example, a microcomputer and controls each unit. The control unit 1106 is connected to a removable medium 1107 formed of a semiconductor memory, a magnetic disk, an optical disk, a magneto-optical disk, or the like as necessary, and a program and various other data are supplied as necessary.

  Next, the operation of the monitoring process will be described with reference to the flowchart in FIG. When the power of the monitoring system 1100 is turned on, the imaging unit 1101 images the monitoring area, and outputs an image obtained by the imaging to the tracking target detection unit 1103, the object tracking unit 1105, and the image display 1102. Yes. In step S931, the tracking target detection unit 1103 executes processing for detecting a tracking target from the image input from the imaging unit 1101. For example, when a moving object is detected, the tracking target detection unit 1103 detects the moving object as a tracking target. The tracking target detection unit 1103 detects, for example, the point with the highest luminance or the center point of the tracking target from the tracking target, and outputs the detected tracking point to the object tracking unit 1105.

  In step S932, the object tracking unit 1105 executes a tracking process for tracking the tracking point detected in step S931. This tracking process is the same as the process of the object tracking device 26 in FIG. 5 described above.

  In step S933, the object tracking unit 1105 detects the position of the tracking point on the screen. In step S934, the object tracking unit 1105 detects the difference between the position of the tracking point detected by the process in step S933 and the center of the image. In step S935, the object tracking unit 1105 generates a camera drive signal corresponding to the difference detected in the process of step S934, and outputs the camera drive signal to the camera drive unit 1104. In step S936, the camera drive unit 1104 drives the imaging unit 1101 based on the camera drive signal. As a result, the imaging unit 1101 pans or tilts so that the tracking point is located at the center of the screen.

  In step S937, the control unit 1106 determines whether or not to end the monitoring process based on an instruction from the user. If the end instruction is not received from the user, the control unit 1106 returns to step S931 and performs the subsequent processes. Run repeatedly. When the end of the monitoring process is instructed by the user, it is determined in step S937 that the monitoring process ends, and the control unit 1106 ends the monitoring process.

  As described above, in the surveillance camera system 1100, a moving object is automatically detected as a tracking point, and an image centered on the tracking point is displayed on the image display 1102. This makes it possible to perform the monitoring process more easily and reliably.

  FIG. 76 shows a configuration example of another surveillance camera system to which the present invention is applied. The surveillance camera system 1200 includes an imaging unit 1201, an image display 1202, an object tracking unit 1203, a camera driving unit 1204, a control unit 1205, an instruction input unit 1206, and a removable medium 1207.

  Similar to the imaging unit 1101, the imaging unit 1201 is configured by a CCD video camera or the like, and outputs the captured image to the image display 1202 and the object tracking unit 1203. The image display 1202 displays the input image. The object tracking unit 1203 has basically the same configuration as the object tracking device 26 in FIG. 5 described above. The camera driving unit 1204 drives the imaging unit 1201 to pan and tilt in a predetermined direction based on control from the object tracking unit 1203.

  The control unit 1205 is configured by a microcomputer, for example, and controls each unit. The instruction input unit 1206 includes various buttons, switches, a remote controller, and the like, and outputs a signal corresponding to an instruction from the user to the control unit 1205. The removable medium 1207 includes a semiconductor memory, a magnetic disk, an optical disk, a magneto-optical disk, and the like, connected as necessary, and appropriately supplies necessary programs and data to the control unit 1205.

  Next, the operation will be described with reference to the flowchart of FIG.

  In step S961, the control unit 1205 determines whether a tracking point is specified by the user. If the tracking point is not specified, the process proceeds to step S969, and the control unit 1205 determines whether or not the user has instructed the end of the process. If the end is not instructed, the process returns to step S961, and the subsequent steps Repeat the process.

  That is, during this time, an image obtained by the imaging unit 1201 imaging the imaging area is output to the image display 1202 and displayed. When viewing the image and ending the process of monitoring the monitoring area, the user (monitoring person) operates the instruction input unit 1206 to instruct the end. When the termination is instructed, the control unit 1205 terminates the monitoring process.

  On the other hand, the user looks at the image displayed on the image display 1202, and when, for example, a suspicious person is displayed, a predetermined point of the suspicious person is designated as a tracking point. This designation is performed by operating the instruction input unit 1206. When the user designates a tracking point, it is determined in step S961 that the tracking point has been designated, and the process proceeds to step S962 where a tracking process is executed. Hereinafter, the processing executed in steps S962 to S967 is the same as the processing performed in steps S932 to S937 in FIG. In other words, this causes the imaging unit 1201 to be driven so that the specified tracking point is at the center of the screen.

  In step S967, the control unit 1205 determines whether or not monitoring has been instructed. If instructed, the control unit 1205 terminates the processing. If not instructed, the control unit 1205 proceeds to step S968 and cancels tracking from the user. Whether or not is instructed is determined. For example, when it is determined that the user who has designated tracking is not a suspicious person, the user can operate the instruction input unit 1206 to instruct cancellation of tracking. When it is determined in step S968 that the tracking cancellation is not instructed, the control unit 1205 returns to step S962 and executes the subsequent processing. That is, in this case, the process of tracking the designated tracking point is continued.

  If it is determined in step S968 that tracking cancellation is instructed, the tracking process is canceled, the process returns to step S961, and the subsequent processes are repeatedly executed.

  As described above, in the surveillance camera system 1200, the image of the tracking point designated by the user is displayed at the center of the image display 1202. Therefore, the user can select a desired image arbitrarily and perform fine monitoring.

  The present invention can be applied not only to a television receiver and a surveillance camera system but also to various image processing apparatuses.

  In the above, the image processing unit is a frame, but the present invention can also be applied to a case where a field is a processing unit.

  It does not matter whether the above-described series of processing is realized by hardware or software. When the above-described series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. It is installed on a general-purpose personal computer or the like from a recording medium such as a network or a removable medium.

  In addition, the steps of executing the series of processes described above in this specification are performed in parallel or individually even if they are not necessarily processed in time series, as well as processes performed in time series in the order described. The processing to be performed is also included.

It is a block diagram which shows the structural example of the surveillance camera system to which this invention is applied. It is a flowchart explaining a monitoring process. It is a figure which shows the example of the image displayed by the surveillance camera system of FIG. It is a figure which shows the example of a movement of the correction object area of FIG. It is a block diagram which shows the structural example of the object tracking part of FIG. It is a flowchart figure explaining a tracking process. It is a figure explaining tracking when a tracking object rotates. It is a figure explaining the tracking when an occlusion occurs. It is a figure explaining the tracking when a scene change occurs. It is a flowchart explaining a normal process. It is a flowchart explaining the initialization process of a normal process. It is a figure explaining a transfer candidate extraction process. It is a block diagram which shows the structural example of an area | region estimation related process part. It is a flowchart explaining an area | region estimation related process. It is a flowchart explaining an area | region estimation process. It is a figure explaining the process which determines a sample point. It is a figure explaining the process which determines a sample point. It is a figure explaining the process which determines a sample point. It is a figure explaining the process which determines a sample point. It is a flowchart explaining the update process of an area estimation range. It is a figure explaining the update of a region estimation range. It is a figure explaining the update of a region estimation range. It is a figure explaining the update of a region estimation range. It is a figure explaining the update of a region estimation range. It is a flowchart explaining the other example of the update process of an area | region estimated range. It is a flowchart explaining the update of a region estimation range. It is a flowchart explaining a transfer candidate extraction process. It is a flowchart explaining a template creation process. It is a figure explaining template creation. It is a figure explaining template creation. It is a figure explaining the positional relationship of a template and a tracking point. It is a block diagram which shows the other structural example of an area | region estimation related process part. It is a flowchart explaining the other example of an area | region estimation process. It is a figure explaining growth of the same color area. It is a figure explaining the same color area | region of a tracking point and an area | region estimation result. It is a flowchart explaining the other example of a transfer candidate extraction process. It is a flowchart explaining exception processing. It is a flowchart explaining the initialization process of an exception process. It is a figure explaining selection of a template. It is a figure explaining the setting of a search range. It is a flowchart explaining a continuation determination process. It is a flowchart explaining the other example of another normal process. It is a flowchart explaining the other example of an area | region estimation process. It is a flowchart explaining the other example of a transfer candidate extraction process. It is a figure explaining the transfer candidate at the time of performing a normal process. It is a figure explaining the transfer candidate in the case of performing a normal process. It is a block diagram which shows the structural example of a motion estimation part. It is a flowchart explaining a motion estimation process. It is a figure explaining calculation of activity. It is a figure explaining the relationship between an evaluation value and activity. It is a flowchart explaining an integration process. It is a block diagram which shows the structural example of a background motion estimation part. It is a flowchart explaining a background motion estimation process. 5 is a block diagram illustrating a configuration example of a scene change detection unit 53. FIG. It is a flowchart explaining a scene change detection process. It is a block diagram which shows the structural example of the image correction part of FIG. It is a figure which shows the example of the control signal of an image correction part. It is a figure explaining the principle of a blur of an image. It is a figure explaining the principle of a blur of an image. It is a figure explaining the principle of a blur of an image. It is a figure explaining the principle of a blur of an image. It is a figure which shows the example of the combination of a parameter code. It is a figure explaining the edge part of an image. It is a flowchart explaining a blur correction process. It is a flowchart explaining an image correction process. It is a flowchart explaining an image feature detection process. It is a block diagram which shows the structural example of an image feature detection part. It is a figure explaining the block extracted by a block cutout part. It is a flowchart explaining an image composition process. It is a block diagram which shows the structural example of an image synthetic | combination part. It is a figure explaining a distributed calculation. It is a block diagram which shows the structural example of a television receiver. It is a flowchart explaining the image display process of a television receiver. It is a block diagram which shows the structural example of a surveillance camera system. It is a flowchart explaining the monitoring process of a monitoring camera system. It is a block diagram which shows the other structural example of the surveillance camera system. It is a flowchart explaining the monitoring process of a monitoring camera system.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Surveillance camera system, 21 Image pick-up part, 22 Image correction part, 24 Tracking object detection part, 25 Area setting part, 26 Object tracking part, 51 Template matching part, 52 Motion estimation part, 53 Scene change detection part, 54 Background motion estimation , 55 region estimation related processing unit, 56 transfer candidate holding unit, 57 tracking point determination unit, 58 template holding unit, 741 control signal generation unit, 742 image feature detection unit, 743 address calculation unit, 744 coefficient ROM, 745 region extraction Section, 746 product-sum operation section, 747 image composition section

Claims (9)

An image processing apparatus that clearly displays a moving object,
Feature point detecting means for detecting feature points of a moving object in an image;
Tracking means for tracking the movement of the object in the image based on the feature points detected by the feature point detection means;
Correction area setting means for setting a correction area of a preset size around the object in the image based on the tracking result by the tracking means;
Correction means for correcting an image in the correction region in the image;
Display control means for controlling the display of the image in the correction area corrected by the correction means ,
The correction means includes
Supply means for supplying a control signal for specifying an image in the correction area, and a parameter representing a degree of blur of the image;
Feature detection means for detecting a feature of the image in the correction region identified based on the control signal and outputting a feature code representing the detected feature;
Storage means for storing a parameter representing the degree of blur of the image and a coefficient corresponding to the feature code output by the feature detection means;
Reading means for reading out the coefficient and the coefficient corresponding to the feature code output by the feature detection means from the storage means;
Product-sum operation means for performing product-sum operation on the pixel values of the input image based on the coefficient read by the reading means;
A selection output means for selecting and outputting a calculation result by the product-sum calculation means and a pixel value of the input image;
Correct so as to remove the blur of the image in the correction area
Image processing device. The feature detection means includes
A first extraction means for extracting a plurality of pixels included in a preset first area around the pixel to be subjected to the product-sum operation from the input image;
A second extraction means for extracting a plurality of pixels included in a plurality of second regions, which are continuous in a plurality of vertical or horizontal directions, and the first region;
A block for calculating a plurality of block differences by calculating a sum of absolute values of differences between corresponding pixel values in the pixels extracted by the first extracting unit and the pixels extracted by the second extracting unit. Difference calculation means;
The image processing apparatus according to claim 1 , further comprising: a difference determination unit that determines whether the block difference is larger than a preset threshold value. The image processing apparatus according to claim 1 , wherein the parameter is a parameter of a Gaussian function in a model formula representing a relationship between a pixel of a blurred image and a pixel of a non-blurred image. The image processing apparatus according to claim 3 , wherein the coefficient stored by the storage unit is a coefficient obtained by calculating an inverse matrix of the model formula. The selection output means includes
First extraction means for extracting a plurality of pixels on which product-sum operation has been performed by the product-sum operation means;
Dispersion calculating means for calculating a degree of dispersion representing a degree of dispersion of the plurality of pixels extracted by the first extracting means;
The image processing apparatus according to claim 1 , wherein a dispersion determination unit that determines whether or not the degree of dispersion calculated by the dispersion calculation unit is greater than a preset threshold value. The selection output unit is a pixel selection unit that selects a pixel value to be output from either a calculation result by the product-sum calculation unit or a pixel value of the input image based on the determination result of the variance determination unit. The image signal processing apparatus according to claim 5 , further comprising: An image processing method of an image processing apparatus for clearly displaying a moving object in an image,
A feature point detecting step for detecting a feature point of a moving object in the image;
A tracking step of tracking the movement of the object in the image based on the feature points detected by the processing of the feature point detection step;
A correction area setting step for setting a correction area of a predetermined size around the object in the image based on the tracking result of the tracking step;
A correction step of correcting an image in the correction area of the image;
And a display control step of the correction area to control the display of the corrected image by the processing in the correction step,
The correction step includes
Supplying a control signal for specifying an image in the correction area, and a parameter representing a degree of blur of the image;
A feature detection step of detecting a feature of the image in the correction region specified based on the control signal and outputting a feature code representing the detected feature;
A parameter indicating the degree of blur of the image, and a storage control step for controlling storage of a coefficient corresponding to the feature code output by the processing of the feature detection step;
A step of reading out a parameter corresponding to the feature code output by the processing of the parameter and the feature detection step, the storage of which is controlled by the processing of the storage control step;
A product-sum operation step for performing a product-sum operation on the pixel values of the input image based on the coefficients read out by the processing in the reading step;
A selection output step of selecting and outputting a calculation result by the process of the product-sum calculation step and a pixel value of the input image;
An image processing method for performing correction so as to remove blur in an image in the correction area . An image processing apparatus program that clearly displays a moving object in an image,
A feature point detection control step for controlling detection of feature points of a moving object in the image;
A tracking control step for controlling to track the movement of the object in the image based on the feature point detected by the processing of the feature point detection control step;
A correction area setting control step for controlling to set a correction area of a preset size around the object in the image based on the tracking result by the processing of the tracking control step;
A correction control step for controlling correction of an image in the correction region in the image;
A display control step for controlling display of an image in which the correction area is corrected by the processing of the correction control step;
Including
The correction control step includes
A supply control step of supplying a control signal for specifying an image in the correction area, and a parameter representing a degree of blur of the image;
A feature detection control step of detecting a feature of the image in the correction region identified based on the control signal and outputting a feature code representing the detected feature;
A parameter representing the degree of blur of the image, and a storage control step for controlling storage of a coefficient corresponding to the feature code whose output is controlled by the processing of the feature detection control step;
A read control step for reading out the coefficient corresponding to the feature code whose output is controlled by the process of the parameter and the feature detection control step whose storage is controlled by the process of the storage control step;
A product-sum operation control step for performing a product-sum operation on the pixel value of the input image based on the coefficient whose reading is controlled by the processing of the reading control step;
A selection output control step of selecting and outputting a calculation result by the process of the product-sum calculation control step and a pixel value of the input image;
A program for causing a computer to execute a process of correcting so as to remove blurring of an image in the correction area . A recording medium in which a program of an image processing device for clearly displaying a moving object in an image is recorded,
A feature point detection control step for controlling detection of feature points of a moving object in the image;
A tracking control step for controlling to track the movement of the object in the image based on the feature point detected by the processing of the feature point detection control step;
A correction area setting control step for controlling to set a correction area of a preset size around the object in the image based on the tracking result by the processing of the tracking control step;
A correction control step for controlling correction of an image in the correction region in the image;
A display control step for controlling display of an image in which the correction area is corrected by the processing of the correction control step;
Including
The correction control step includes
A supply control step of supplying a control signal for specifying an image in the correction area, and a parameter representing a degree of blur of the image;
A feature detection control step of detecting a feature of the image in the correction region identified based on the control signal and outputting a feature code representing the detected feature;
A parameter representing the degree of blur of the image, and a storage control step for controlling storage of a coefficient corresponding to the feature code whose output is controlled by the processing of the feature detection control step;
A read control step for reading out the coefficient corresponding to the feature code whose output is controlled by the process of the parameter and the feature detection control step whose storage is controlled by the process of the storage control step;
A product-sum operation control step for performing a product-sum operation on the pixel value of the input image based on the coefficient whose reading is controlled by the processing of the reading control step;
A selection output control step of selecting and outputting a calculation result by the process of the product-sum calculation control step and a pixel value of the input image;
A recording medium on which is recorded a program that causes a computer to execute a correction process so as to remove blur in an image in the correction area .

Images