JP2007272731A - Image processing apparatus and method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily designate an object as the object of tracking, and to estimate the tracking position of a designated object. <P>SOLUTION: A detection object region generation part 172 acquires offset quantity for area designation for the current tracking point or cursor position in a picked-up image, and generates a detection object region with predetermined size in a user designation direction on the basis of the offset quantity. A feature value calculation part 173 uses the pixels of a predetermined pixel interval as sample points in the detection object region, and calculates the feature values of the sample points. A region estimation part 174 extracts pixels satisfying predetermined conditions among the feature values of respective sample points in the detection object region, generates a region configured of the pixel groups, and calculates the center of gravity position of the generated region. The calculated center of gravity position is estimated as a tracking point intended by the user. This invention can be applied to a monitor camera system. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, method, and program, and in particular, allows an object to be tracked to be easily specified and allows the tracking position of the specified object to be estimated. The present invention relates to an image processing apparatus and method, and a program.

  Conventionally, many methods for tracking an object extracted by obtaining a motion vector from an image or a designated object have been proposed in Patent Documents 1 to 5, for example.

In addition, as a method for extracting an object in an image, for example, as proposed in Patent Document 6 and Patent Document 7, a difference between adjacent frames is taken, and a portion having a large absolute value of the difference is set as a moving object. There are a method of extracting, a method of extracting a similar color portion as an object in the same frame, a method of extracting an object by designating a plurality of surroundings, and the like.
JP 2005-165791 A JP 2005-165929 A JP 2004-46647 A JP 2000-105835 A JP-A-6-89342 JP 2002-74375 A JP 2002-352248 A

  By the way, in the method of calculating the difference between adjacent frames, it is difficult to extract an object when there is no moving object or the entire image is moving. Also, in the method of extracting similar color portions as objects in the same frame, it is difficult to extract objects having a plurality of different colors as a group. Furthermore, in the method of extracting an object by specifying a plurality of points around the periphery, it is difficult to extract when the target object is moving.

  As described above, it is difficult to specify the tracking point when specifying the tracking target or when the target object is moving. In the case of accumulated images, the tracking point can be specified after pausing, but if tracking is lost due to erroneous detection of a motion vector, the tracking point is paused and the tracking point is corrected each time. Takes time. Furthermore, in tracking in a real-time image, the difficulty in specifying and correcting the tracking point may impair the value of an application called object tracking in the image.

  The present invention has been made in view of such circumstances, and makes it possible to easily specify an object to be tracked and to estimate the tracking position of the specified object. Is.

  An image processing apparatus according to an aspect of the present invention is an image processing apparatus that tracks a moving object, and detects a current tracking point or a cursor position on the object in an image based on a direction specified by a user. A generating unit that generates a target region; a calculating unit that calculates a feature amount of a pixel included in the detection target region generated by the generating unit; and a predetermined amount based on the feature amount calculated by the calculating unit Extraction means for extracting the pixels, and estimation means for estimating the position of the center of gravity of the region constituted by the predetermined pixels extracted by the extraction means as the tracking point intended by the user.

  Fine adjustment means for finely adjusting the tracking point estimated by the estimation means so as to be located at the center of gravity of the object may be further provided.

  The generation means can generate a detection target area corresponding to an offset amount designated by the user.

  The generation means can generate a detection target region centered on a current tracking point or cursor position on the object.

  When the predetermined pixel cannot be extracted by the extraction unit, the generation unit can generate a new detection target region in which the detection target region is enlarged.

  The extraction means can extract pixels having a feature quantity different from the feature quantity at the current tracking point or cursor position.

  The extraction means can extract pixels having a feature quantity different from the background feature quantity.

  The extraction unit can extract pixels having a feature amount designated by the user.

  The extraction means can extract pixels having the feature quantity of the rank frequency designated by the user from the calculated feature quantity frequency.

  The extraction means can extract pixels having a feature quantity with a frequency of the rank specified by the user from the frequency of feature quantities different from the feature quantity at the current tracking point or cursor position.

  The estimating means estimates a center of gravity position of an area composed of pixels having a feature amount having a reliability equal to or higher than a predetermined threshold among the extracted predetermined pixels as a tracking point intended by the user. be able to.

  The feature amount may be a motion vector, color, or luminance.

  An image processing method according to an aspect of the present invention is an image processing method of an image processing apparatus that tracks a moving object, in a direction specified by a user with respect to a current tracking point or cursor position on the object in the image. And generating a detection target region, calculating a feature amount of a pixel included in the generated detection target region, extracting a predetermined pixel based on the calculated feature amount, and extracting the extracted pixel Including a step of estimating a centroid position of an area composed of predetermined pixels as a tracking point intended by the user.

  The program according to one aspect of the present invention is a program that causes a computer to execute image processing of an image processing apparatus that tracks a moving object, and that is specified by a user with respect to a current tracking point or cursor position on the object in the image. A detection target region is generated based on the direction in which the detection target region is generated, a feature amount of a pixel included in the generated detection target region is calculated, and a predetermined pixel is extracted and extracted based on the calculated feature amount And a step of estimating a center of gravity position of an area formed by the predetermined pixels as a tracking point intended by the user.

  In one aspect of the present invention, a detection target region is generated based on a direction specified by a user with respect to a current tracking point or cursor position on an object, and a feature amount of a pixel included in the generated detection target region Is calculated, a predetermined pixel is extracted based on the calculated feature amount, and the barycentric position of an area composed of the extracted predetermined pixel is estimated as the tracking point intended by the user.

  According to the present invention, it is possible to easily specify an object to be tracked and estimate a tracking position of the specified object.

  Embodiments of the present invention will be described below. Correspondences between the constituent elements of the present invention and the embodiments described in the specification or the drawings are exemplified as follows. This description is intended to confirm that the embodiments supporting the present invention are described in the specification or the drawings. Therefore, even if there is an embodiment which is described in the specification or the drawings but is not described here as an embodiment corresponding to the constituent elements of the present invention, that is not the case. It does not mean that the form does not correspond to the constituent requirements. On the contrary, even if an embodiment is described herein as corresponding to the invention, this does not mean that the embodiment does not correspond to other than the configuration requirements. .

  An image processing apparatus according to one aspect of the present invention (for example, the monitoring camera system 1 in FIG. 1) is an image processing apparatus that tracks a moving object, and the current tracking point or cursor position on the object in the image. Based on the direction designated by the user, a generation unit (for example, a detection target region generation unit 172 in FIG. 27) that generates a detection target region, and pixels included in the detection target region generated by the generation unit A calculation unit (for example, a feature amount calculation unit 173 in FIG. 27) that calculates a feature amount, and an extraction unit (for example, an area in FIG. 27) that extracts a predetermined pixel based on the feature amount calculated by the calculation unit. An estimation unit 174) and an estimation unit (e.g., estimating a center of gravity position of an area composed of the predetermined pixels extracted by the extraction unit as a tracking point intended by the user) , And a region estimating unit 174 of FIG. 27).

  The image processing apparatus further includes fine adjustment means (for example, the tracking position correction unit 24 in FIG. 1) for finely adjusting the tracking point estimated by the estimating means so as to be the center of gravity position of the object. it can.

  A transmission method and a program according to one aspect of the present invention include an image processing method of an image processing apparatus that tracks a moving object, or a program that causes a computer to execute the image processing, and a current tracking point on the object in an image. Alternatively, a detection target area is generated with respect to the cursor position based on the direction specified by the user (for example, step S134 in FIG. 38), and the feature amount of the pixel included in the generated detection target area is calculated ( For example, in step S135 in FIG. 38, a predetermined pixel is extracted based on the calculated feature amount (for example, step S138 in FIG. 38), and the barycentric position of the region constituted by the extracted predetermined pixel Is estimated as the tracking point intended by the user (for example, step S140 in FIG. 38).

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a configuration example when the present invention is applied to a surveillance camera system.

  The imaging unit 21 includes, for example, a CCD (Charge Coupled Device) video camera or the like, and displays the captured image on the image display 25. The object position estimation unit 22 detects a tracking target from the image input from the imaging unit 21 and outputs the detection result to the object tracking unit 23. Further, the object position estimating unit 22 is configured such that when a predetermined direction is designated by the user via the instruction input unit 29 or the cursor is moved in a direction in which the object to be tracked exists, the cursor position and the offset position , The tracking position intended by the user is estimated, and the estimation result is output to the object tracking unit 23.

  The object tracking unit 23 operates to track the tracking point detected or estimated by the object position estimation unit 22 in the image supplied from the imaging unit 21. The object tracking unit 23 outputs the tracking result to the object position estimating unit 22 and outputs position information representing the tracking position as the tracking result to the tracking position correcting unit 24. Furthermore, the object tracking unit 23 controls the camera driving unit 26 so that the moved object can be imaged based on the tracking result. The tracking position correction unit 24 finely adjusts the position information representing the tracking position supplied from the object tracking unit 23 so that it becomes the barycentric position of the object to which the tracking point belongs, and outputs the correction result to the image display 25. At the same time, the correction result is supplied to the object tracking unit 23. As a result, the object tracking unit 23 can operate to track the corrected tracking point from the next frame. Based on the control from the object tracking unit 23, the camera driving unit 26 drives the imaging unit 21 so that the imaging unit 21 captures an image centered on the tracking point.

  The control unit 27 is configured by, for example, a microcomputer and controls each unit. The control unit 27 is connected to a removable medium 28 formed of a semiconductor memory, a magnetic disk, an optical disk, a magneto-optical disk, or the like as needed, and a program and other various data are supplied as needed. The instruction input unit 29 includes various buttons, switches, a remote controller using infrared rays or radio waves, and outputs a signal corresponding to an instruction from the user to the control unit 27.

  Next, monitoring processing executed by the monitoring camera system shown in FIG. 1 will be described with reference to the flowchart of FIG. In this process, when the power of the monitoring system 1 is turned on, an area to be monitored is imaged by the imaging unit 21, and an image obtained by imaging is imaged via the object position estimating unit 22 and the object tracking unit 23. It is output to the display 25 and started.

  In step S <b> 1, the object position estimation unit 22 executes a process of detecting a tracking target from the image input from the imaging unit 21. For example, when there is a moving object in the input image, the object position estimation unit 22 detects the moving object as a tracking target, and selects a point with the highest luminance or a center point of the tracking target from the tracking target. Detect as tracking point. In addition, when a predetermined direction is designated by the user via the instruction input unit 29, the object position estimation unit 22 estimates a tracking position intended by the user based on the designated direction, and the estimation result is obtained from the object tracking unit. To 23. Details of the processing for estimating the tracking position will be described with reference to FIG.

  In step S2, the object tracking unit 23 executes a tracking process for tracking the tracking point detected in the process of step S1. The details of the tracking process will be described later with reference to FIG. 5. With this process, a tracking point in an object (for example, a person, an animal, etc.) to be tracked in an image captured by the imaging unit 21 is obtained. (For example, the center of the eyes and head) is tracked, the tracking result is output to the control unit 27, and the position information indicating the tracking position is output to the tracking position correcting unit 24.

  In step S3, the control unit 27 causes the image display 25 to superimpose and display a mark representing the tracking position on the image captured by the imaging unit 21, based on the tracking result obtained in step S2.

  In step S4, the object tracking unit 23 detects the movement of the object based on the tracking result in the process of step S2, generates a camera driving signal for driving the camera so that the moved object can be imaged, and the camera driving unit. 26. In step S <b> 5, the camera drive unit 26 drives the imaging unit 21 based on the camera drive signal from the object tracking unit 23. Thereby, the imaging unit 21 pans or tilts the camera so that the tracking point does not deviate from the screen.

  In step S6, the control unit 27 determines whether or not to end the monitoring process based on an instruction from the user via the instruction input unit 29. If the end is not instructed by the user, the control unit 27 proceeds to step S1. Return and repeat the subsequent processing. If it is determined in step S6 that the user has instructed the end of the monitoring process, the control unit 27 ends the monitoring process.

  3A to 3C are diagrams showing examples of images displayed on the image display 25 at this time in time series. FIG. 3A is an example of an image in which an object 41 to be tracked is imaged by the imaging unit 21. In this example, a person who moves in the left direction in the figure is imaged as the object 41. In FIG. 3B, the object 41 has moved to the left in the figure from the position of FIG. 3A, and in FIG. 3C, the object 41 has moved further to the left from the position of FIG. 3B.

  The object position estimation unit 22 detects the object 41 in step S1 of FIG. 2, and outputs the human eye that is the object 41 to the object tracking unit 23 as the tracking point 41A. In step S2, the object tracking unit 23 performs a tracking process.

  Next, a detailed configuration example and operation of the object tracking unit 23 in FIG. 1 will be described.

  FIG. 4 is a block diagram illustrating a functional configuration example of the object tracking unit 23. As shown in FIG. 4, the object tracking unit 23 includes a template matching unit 51, a motion estimation unit 52, a scene change detection unit 53, a background motion estimation unit 54, a region estimation related processing unit 55, a transfer candidate holding unit 56, and a tracking. The point determination unit 57, the template holding unit 58, and the control unit 59 are configured.

  The template matching unit 51 performs a matching process between the input image and the template image held in the template holding unit 58. The motion estimation unit 52 estimates the motion of the input image, and obtains the motion vector obtained as a result of the estimation and the accuracy of the motion vector, the scene change detection unit 53, the background motion estimation unit 54, the region estimation related processing unit 55, And output to the tracking point determination unit 57. The scene change detection unit 53 detects a scene change based on the motion vector supplied from the motion estimation unit 52 and the accuracy of the motion vector.

  The background motion estimation unit 54 executes processing for estimating the background motion based on the motion vector supplied from the motion estimation unit 52 and the accuracy of the motion vector, and supplies the estimation result to the region estimation related processing unit 55. The area estimation related processing unit 55 includes the motion vector supplied from the motion estimation unit 52 and the accuracy of the motion vector, the background motion supplied from the background motion estimation unit 54, and the tracking point information supplied from the tracking point determination unit 57. Based on the above, region estimation processing is performed. The region estimation related processing unit 55 generates a transfer candidate based on the input information, supplies the transfer candidate to the transfer candidate holding unit 56, and holds it. Further, the region estimation related processing unit 55 creates a template based on the input image, supplies the template to the template holding unit 58, and holds it.

  The tracking point determination unit 57 determines the tracking point based on the motion vector supplied from the motion estimation unit 52, the accuracy of the motion vector, and the transfer candidate supplied from the transfer candidate holding unit 56. Information about the points is output to the region estimation related processing unit 55.

  The control unit 59 controls each part of the template matching unit 51 to the template holding unit 58 based on the tracking point information output from the object position estimation unit 22 to track the detected tracking target and to display an image display. A control signal is output to the camera drive unit 26 so that the tracking point is displayed in the screen displayed on the screen 25, and the drive of the imaging unit 21 is controlled. Thereby, the tracking point is controlled so as not to go out of the screen. Further, the control unit 59 outputs a tracking result such as information on the position of the tracking point on the screen to the tracking position correction unit 24 and the control unit 27.

  Next, the operation of the object tracking unit 23 will be described.

  FIG. 5 is a flowchart illustrating details of the tracking process executed by the object tracking unit 23 in step S2 of FIG.

  As shown in FIG. 5, the object tracking unit 23 basically executes normal processing and exception processing. That is, normal processing is performed in step S11. The details of this normal process will be described later with reference to the flowchart of FIG. 9, and the process of tracking the tracking point detected or estimated by the object position estimation unit 22 is executed by this process.

  When the tracking point cannot be tracked in the normal process of step S11, an exception process is executed in step S12. Details of this exception processing will be described later with reference to the flowchart of FIG. 12. When the tracking point is not visible from the image by this exception processing, a return processing to the normal processing is executed by template matching. If it is determined that the tracking process cannot be continued due to the exception process, that is, the process cannot be returned to the normal process, the process is terminated. If it is determined that it is possible to return, the process returns to step S11 again. In this manner, the normal process in step S11 and the exception process in step S12 are repeatedly executed sequentially for each frame.

  In the present embodiment, as shown in FIGS. 6 to 8, the tracking point is temporarily changed by the normal process and the exception process, such as the tracking target rotating, the occlusion occurring, the scene changing, etc. Tracking is possible even if it becomes invisible.

  In the example of FIG. 6, a person's face 74 as a tracking target is displayed in the frame n−1, and this person's face 74 has a right eye 72 and a left eye 73. It is assumed that the user designates, for example, the right eye 72 (exactly one pixel therein) as the tracking point 71 among them. In the example of FIG. 6, in the next frame n, the person to be tracked moves in the left direction in the figure, and in the next frame n + 1, the person's face 74 rotates in the clockwise direction. . As a result, the right eye 72 that has been visible until now is not displayed, and tracking cannot be performed. Therefore, in the above-described normal processing in step S <b> 11 of FIG. 5, the left eye 73 on the face 74 as the same object as the right eye 72 is selected as a new tracking point, and the tracking point is switched to the left eye 73. This enables tracking.

  In the example of FIG. 7, the ball 81 has moved from the left side of the figure of the human face 74 as the tracking target in the frame n−1, and the ball 81 just covers the face 74 in the next frame n. ing. In this state, the face 74 including the right eye 72 designated as the tracking point 71 is not displayed. When such an occlusion occurs, the face 74 as the object is not displayed, so there is no transfer point to be tracked instead of the tracking point 71, and it becomes difficult to track the tracking point thereafter. Therefore, in the present embodiment, an image of frame n−1 (actually a previous frame in time) including the right eye 72 as the tracking point 71 is stored in advance as a template, and the ball 81 moves further to the right. When the right eye 72 designated as the tracking point 71 appears again in the frame n + 1, it is confirmed based on the template that the right eye 72 as the tracking point 71 is displayed again by the exception processing in step S12 of FIG. Thus, the right eye 72 is tracked again as the tracking point 71.

  In the example of FIG. 8, the person's face 74 as a tracking target is displayed in the frame n−1. However, in the next frame n, the automobile 91 covers the entire face including the person's face 74. That is, in this case, a scene change has occurred. Therefore, in the present embodiment, even when the scene change occurs and the tracking point 71 no longer exists in the image, the vehicle 91 moves and the right eye 72 is displayed again in the frame n + 1. 5, it is confirmed based on the template that the right eye 72 as the tracking point 71 has reappeared, and the right eye 72 can be tracked again as the tracking point 71.

  Next, the details of the normal processing in step S11 in FIG. 5 will be described with reference to the flowchart in FIG.

  In step S21, the tracking point determination unit 57 executes an initialization process of a normal process. The details will be described later with reference to the flowchart of FIG. 10, but an area estimation range based on the tracking point designated to be tracked by the user is set by this processing. This area estimation range is the same object as the tracking point specified by the user (for example, when the tracking point is a human eye, a human face as a rigid body that moves like the eye, a human body, or the like) ) Is a range to be referred to when estimating the range of points belonging to. A transfer point is selected from points in the region estimation range.

  In step S22, the control unit 59 controls each unit so as to wait for input of an image of the next frame. In step S23, the motion estimation unit 52 estimates the tracking point motion. That is, by capturing a frame (hereinafter referred to as a subsequent frame) that is temporally later than a frame including a tracking point designated by the user (here, referred to as a previous frame) in the process of step S22, two consecutive frames are eventually obtained. Therefore, in step S23, the movement of the tracking point is estimated by estimating the position of the tracking point of the subsequent frame corresponding to the tracking point of the previous frame.

  Note that “before or after in time” means the order of input or the order of processing. Usually, since the images of each frame are input in the order of imaging, in that case, the frame captured earlier in time becomes the previous frame, but the frame captured later in time is processed first. In this case, a frame that is imaged later in time becomes the previous frame.

  In step S24, the motion estimation unit 52 determines whether the tracking point can be estimated as a result of the process in step S23. Whether or not the tracking point can be estimated is determined, for example, by comparing the accuracy value of a motion vector (described later) generated and output by the motion estimation unit 52 with a preset threshold value. Specifically, it is possible to estimate if the accuracy of the motion vector is equal to or greater than a threshold, and it is determined that estimation is impossible if the accuracy is smaller than the threshold. In other words, the possibility here is determined relatively strictly, and if the estimation is not actually impossible but the accuracy is low, it is determined as impossible. As a result, more reliable tracking processing can be performed.

  In step S24, it is determined that the motion estimation result at the tracking point and the motion estimation result at a point in the vicinity of the tracking point can be estimated if the motion occupies a large number of motions. It is also possible to do so.

  If it is determined in step S24 that the movement of the tracking point can be estimated, that is, if the probability that the tracking point is correctly set on the corresponding point on the same object is relatively high, the process proceeds to step S25. . More specifically, when the right eye 72 is designated as the tracking point 71 in the example of FIG. 6, it is determined that the movement of the tracking point can be estimated when the probability that the right eye 72 is correctly tracked is relatively high. Is done.

  In step S25, the tracking point determination unit 57 shifts the tracking point by the estimated motion (so-called motion vector) obtained in the process of step S23. As a result, the tracking position in the subsequent frame after the tracking of the tracking point in the previous frame is determined.

  After step S25, region estimation related processing is executed in step S26. Details of this region estimation related processing are disclosed in Japanese Patent Laid-Open No. 2005-303983 previously proposed by the present applicant. With this process, the area estimation range designated in the initialization process of the normal process in step S21 is updated. In addition, when the tracking point is not displayed due to rotation of the target object, candidates for transfer points (transfer candidates) as points to be transferred are previously extracted in a state where tracking is still possible. And is held by the transfer candidate holding unit 56. In addition, when the transfer to the transfer candidate becomes impossible, the tracking is temporarily interrupted, but a template for confirming that the tracking can be performed again when the tracking point appears again is created in advance, and the template holding unit 58.

  After the region estimation related processing in step S26 is completed, the processing returns to step S22 again, and the subsequent processing is repeatedly executed.

  As described above, as long as the movement of the tracking point designated by the user can be estimated, the processing from step S22 to step S26 is repeatedly executed for each frame, and tracking is performed.

  On the other hand, when it is determined in step S24 that the movement of the tracking point cannot be estimated, that is, as described above, for example, when the accuracy of the motion vector is equal to or less than the threshold value, the processing is performed. Proceed to step S27.

  In step S27, the tracking point determination unit 57 holds the transfer candidate generated in the region estimation-related processing in step S26 in the transfer candidate holding unit 56, and therefore, the transfer candidate closest to the original tracking point is among them. Select one. The tracking point determination unit 57 determines whether or not a transfer candidate has been selected in step S28. If a transfer candidate has been selected, the tracking point determination unit 57 proceeds to step S29 and sets the tracking point as the transfer candidate selected in step S27. Change (change). Thereby, the transfer candidate point is set as a new tracking point.

  After the process of step S29, the process returns to step S23, and the process of estimating the movement of the tracking point selected from the transfer candidates is executed. Then, it is determined again whether or not the motion of the newly set tracking point can be estimated in step S24. If it can be estimated, a process of shifting the tracking point by the estimated motion is performed in step S25. In step S26, region estimation related processing is executed. Thereafter, the process returns to step S22 again, and the subsequent processes are repeatedly executed.

  If it is determined in step S24 that the newly set tracking point cannot be estimated, the process proceeds again to step S27, and the next transfer candidate closest to the original tracking point is selected from the transfer candidates. In step S29, the transfer candidate is set as a new tracking point. The process after step S23 is repeated again for the new tracking point.

  Even if all the transfer candidates held in the transfer candidate holding unit 56 are used as new tracking points, if it is not possible to estimate the movement of the tracking point, it is determined in step S28 that no transfer candidate has been selected. This normal processing is terminated. Then, the process proceeds to the exception process in step S12 of FIG.

  Next, details of the initialization process of the normal process in step S21 of FIG. 9 will be described with reference to the flowchart of FIG.

  In step S41, the control unit 59 determines whether or not the current process is a process for returning from the exception process. That is, after the exception process in step S12 in FIG. 5 is terminated, it is determined whether or not the process has returned to the normal process in step S11. In the process of the first frame, since the exception process in step S12 has not been executed yet, it is determined that the process has not returned from the exception process, and the process proceeds to step S42.

  In step S42, the tracking point determination unit 57 executes processing for setting the tracking point to the position of the tracking point instruction. That is, for example, the point with the highest luminance in the input image is set as the tracking point.

  The tracking point may be set by other methods such as designation by the user. The designation by the user is, for example, performed by designating a predetermined point in the input image as a tracking point to the control unit 59 by operating the instruction input unit 29 by the user. The tracking point determination unit 57 supplies the set tracking point information to the region estimation related processing unit 55.

  In step S43, the region estimation related processing unit 55 sets a region estimation range based on the tracking point position information set in step S42. This region estimation range is a reference range when estimating a point on the same rigid body as the tracking point, and more specifically, tracking so that the same rigid body portion as the tracking point occupies most of the region estimation range in advance. By setting the position and size of the estimated area range to follow the same rigid body part as the point, the part that shows the most movement in the estimated area range can be estimated as the same rigid body part as the tracking point It is for doing so. In step S43, as a default value, for example, a predetermined fixed range centered on the tracking point is set as the region estimation range. Thereafter, the process proceeds to step S22 in FIG.

  On the other hand, if it is determined in step S41 that the current process is a return process from the exception process in step S12 of FIG. 5, the process proceeds to step S44. In step S44, the tracking point determination unit 57 sets a tracking point and a region estimation range based on a position that matches a template by exception processing described later with reference to the flowchart of FIG. For example, a point on the current frame that matches the tracking point on the template is set as the tracking point, and a certain range set in advance from the point is set as the region estimation range. Thereafter, the processing proceeds to step S22 in FIG.

  The above process will be described with reference to FIG.

  In step S42 of FIG. 10, for example, as shown in FIG. 11, when the right eye 72 of the person in the frame n-1 is designated as the tracking point 71, a predetermined region including the tracking point 71 is an area in step S43. Designated as the estimated range 101. In step S24 of FIG. 9, it is determined whether or not the sample points within the region estimation range 101 can be estimated in the next frame. In the case of the example in FIG. 11, in the frame n + 1 next to the frame n, the left half region 102 in the drawing including the right eye 72 in the region estimation range 101 is hidden by the ball 81. 71 motion cannot be estimated in the next frame n + 1. Therefore, in such a case, one of the points in the area designation range 101 (in the face 74 as a rigid body including the right eye 72) prepared in advance as a transfer candidate in the previous frame n-1 in time is selected. One point (for example, the left eye 73 included in the face 74 (exactly one pixel therein)) is selected, and that point is set as a new tracking point in the frame n + 1.

  Next, with reference to the flowchart of FIG. 12, the details of the exception process of step S12 performed following the normal process of step S11 of FIG. 5 will be described. As described above, this process is performed when it is determined in step S24 in FIG. 9 that it is impossible to estimate the movement of the tracking point, and in step S28, it is determined that the transfer candidate for changing the tracking point cannot be selected. Will be executed.

  In step S51, the control unit 59 executes an exception process initialization process. Here, the details of the exception process initialization process will be described with reference to the flowchart of FIG.

  In step S61, the control unit 59 determines whether or not a scene change has occurred when the movement of the tracking point cannot be estimated and the transfer candidate for changing the tracking point cannot be selected. The scene change detection unit 53 constantly monitors whether or not a scene change has occurred based on the estimation result of the motion estimation unit 52, and the control unit 59 performs step S61 based on the detection result of the scene change detection unit 53. The determination process is executed. Specific processing of the scene change detection unit 53 will be described later with reference to FIGS.

  If a scene change has occurred, the control unit 59 estimates that the reason why tracking has become impossible is that a scene change has occurred, and sets the mode to scene change in step S62. On the other hand, when it is determined in step S61 that no scene change has occurred, the control unit 59 sets the mode to another mode in step S63.

  After the process of step S62 or step S63, the template matching unit 51 executes a process of selecting the oldest template in time in step S64. Specifically, as shown in FIG. 14, for example, when transition from frame n to frame n + 1 is performed, if exception processing is to be executed, a frame n−m + 1 is generated for frame n and is stored in the template holding unit 58. A template generated with respect to the frame mn + 1, which is the oldest template in time, is selected from the held m-frame templates.

  In this way, the template immediately before the transition to exception processing (the template generated with respect to frame n in the case of the example in FIG. 14) is not used, and the template slightly before in time is selected for the following reason. It is. In other words, if a transition to exception processing occurs due to the tracking target occlusion, the tracking target is already quite hidden immediately before the transition, and the tracking target cannot be captured sufficiently large in the template at that time. This is because the possibility is high. Therefore, reliable tracking is possible by selecting a template in a frame slightly before in time.

  Returning to the description of FIG. In step S65, the template matching unit 51 executes a process for setting a template search range. For example, the template search range is set such that the position of the tracking point immediately before the transition to the exception process is the center of the template search range.

  That is, as shown in FIG. 15, when the right eye 72 of the subject's face 74 is designated as the tracking point 71 in the frame n, the ball 81 flies from the left direction in the figure, and the tracking point 71 in the frame n + 1. Is assumed, and the tracking point 71 appears again in the frame n + 2. In this case, an area centered on the tracking point 71 included in the template range 111 is set as the template search range 112.

  In step S66, the template matching unit 51 resets the number of elapsed frames and the number of scene changes after shifting to exception processing to zero. The number of frames and the number of scene changes are used in a continuation determination process (steps S71, S73, S75, and S77 in FIG. 16) in step S55 in FIG.

  After the exception process initialization process is completed as described above, in step S52 of FIG. 12, the control unit 59 executes a process of waiting for an input of an image of the next frame. In step S53, the template matching unit 51 performs a template matching process within the template search range.

  In step S54, the template matching unit 51 determines whether it is possible to return to normal processing. That is, by the template matching process, an absolute value sum of differences between a template several frames before (a pixel in the template range 111 in FIG. 15) and a matching target pixel in the template search range is calculated.

  More specifically, the absolute value sum of the difference between each pixel in the predetermined block in the template range 111 and the predetermined block in the template search range is calculated. The position of the block is sequentially moved within the template range 111, and the sum of absolute values of the differences of the respective blocks is added to obtain a value at the position of the template. Then, the position where the sum of absolute values of the differences becomes the smallest when the template is sequentially moved within the template search range and its value are searched. In step S54, the absolute sum of the searched minimum differences is compared with a predetermined threshold value set in advance. If the sum of the absolute values of the differences is equal to or less than the threshold value, the image including the tracking point included in the template has appeared again. Therefore, it is determined that the normal process can be restored, and the process is illustrated in FIG. The process returns to the normal process of step S11 in step 5.

  Then, as described above, in step S41 of FIG. 10, it is determined that the process returns from the exception process. In step S44, the position where the sum of absolute differences is minimum is set as the matched position of the template. The tracking point and the area estimation range are set based on the positional relationship between the template position held corresponding to the template and the tracking point area estimation range.

  In step S54 of FIG. 12, it is determined whether or not it is possible to return to normal processing by comparing the value obtained by dividing the minimum sum of absolute differences by the activity of the template with a threshold value. Also good. As the activity in this case, the value calculated in step S93 in FIG. 18 by the activity calculation unit 122 in FIG. 17 described later can be used.

  Alternatively, it is possible to return to the normal processing by comparing a value obtained by dividing the current minimum absolute difference sum by the minimum absolute difference sum one frame before with a predetermined threshold. It may be determined whether or not. In this case, it is not necessary to calculate the activity. That is, in step S54, the correlation between the template and the template search range is calculated, and determination is performed based on the comparison between the correlation value and the threshold value.

  If it is determined in step S54 that it is not possible to return to the normal process, the process proceeds to step S55, and the continuation determination process is executed. The details of the continuation determination process will be described later with reference to the flowchart of FIG. 16, and this process determines whether or not the exception process can be continued.

  In step S56, the control unit 59 determines whether or not tracking of the tracking point in the exception process by the continuation determination process in step S55 can be continued (in steps S76 and S78 of FIG. 16 described later). Determine based on the set flag). When it is determined that the tracking process of the tracking point in the exception process can be continued, the process returns to step S52, and the subsequent processes are repeatedly executed. That is, the process of waiting until the tracking point appears again is repeatedly executed.

  On the other hand, when it is determined in step S56 that the tracking process of the tracking point in the exception process cannot be continued (the number of elapsed frames after the tracking point disappears in the process of step S75 in FIG. 16 described later). If it is determined that the threshold value THfr or more is determined or the number of scene changes is determined to be greater than or equal to the threshold value THsc in the process of step S77), it is determined that exception processing is no longer possible, and the tracking process is terminated. Instead of ending the tracking process, it may be possible to return to the normal process again using the tracking point that has been held. The processing in this case will be described later with reference to the flowchart of FIG.

  Next, details of the continuation determination process in step S55 of FIG. 12 will be described with reference to the flowchart of FIG.

  In step S71, the control unit 59 executes a process of adding 1 to the number of elapsed frames as a variable. The number of elapsed frames is previously reset to 0 in the exception process initialization process (step S66 in FIG. 13) in step S51 in FIG.

  Next, in step S72, the control unit 59 determines whether or not there is a scene change. Whether or not there is a scene change can be determined based on the detection result of the scene change detection unit 53 always executing the detection process. If it is determined that there is a scene change, the process proceeds to step S73, and the control unit 59 adds 1 to the number of scene changes as a variable. The number of scene changes is also reset to 0 in the initialization process of step S66 in FIG. If it is determined that no scene change has occurred during the transition from the normal process to the exception process, the process of step S73 is skipped.

  Next, in step S74, the control unit 59 determines whether or not the currently set mode is a scene change. This mode is set in step S62 or S63 of FIG. When it is determined that the currently set mode is a scene change, the process proceeds to step S77, and the control unit 59 determines whether or not the number of scene changes is smaller than a preset threshold value THsc. When the control unit 59 determines in step S77 that the number of scene changes is smaller than the threshold value THsc, the control unit 59 proceeds to step S76, sets a continuation flag, and determines that the number of scene changes is equal to or greater than the threshold value THsc. The process proceeds to step S78, and a flag that cannot be continued is set.

  On the other hand, when it is determined in step S74 that the mode is not a scene change (when it is determined that the mode is other), the process proceeds to step S75, and the control unit 59 determines whether or not the number of elapsed frames is smaller than the threshold value THfr. Determine whether. This elapsed frame number is also reset to 0 in advance in step S66 of the exception process initialization process of FIG. When it is determined in step S75 that the number of elapsed frames is smaller than the threshold value THfr, the control unit 59 proceeds to step S76, sets a continuation flag, and is determined that the number of elapsed frames is equal to or greater than the threshold value THfr. In this case, the process proceeds to step S78, and a flag indicating that continuation is not possible is set.

  As described above, when the number of scene changes at the time of template matching processing is equal to or greater than the threshold value THsc, or when the number of elapsed frames is equal to or greater than the threshold value THfr, further exception processing is impossible.

  When the mode is other, a condition that the number of scene changes is 0 may be added to determine whether or not continuation is possible.

  In the above, it is assumed that the frame of the image is used as a processing unit and all frames are used. However, processing is not performed in units of fields or using all frames or fields, but is extracted by thinning out at a predetermined interval. It is also possible to use a modified frame or field.

  Next, a configuration example of the motion estimation unit 52 in FIG. 4 will be described with reference to FIG. As shown in FIG. 17, an input image is supplied to the evaluation value calculation unit 121, the activity calculation unit 122, and the motion vector detection unit 123.

  The evaluation value calculation unit 121 calculates an evaluation value related to the degree of coincidence between both objects associated by the motion vector, and supplies the evaluation value to the normalization processing unit 125. The activity calculation unit 122 calculates the activity of the input image and supplies it to the threshold value determination unit 124 and the normalization processing unit 125. The motion vector detection unit 123 detects a motion vector from the input image and supplies it to the evaluation value calculation unit 121 and the integration processing unit 126.

  The threshold determination unit 124 compares the activity supplied from the activity calculation unit 122 with a predetermined threshold and supplies the determination result to the integration processing unit 126. The normalization processing unit 125 normalizes the evaluation value supplied from the evaluation value calculation unit 121 based on the activity supplied from the activity calculation unit 122, and supplies the obtained value to the integration processing unit 126.

  The integration processing unit 126 calculates the accuracy of the motion vector based on the normalization information supplied from the normalization processing unit 125 and the determination result supplied from the threshold determination unit 124, and uses the obtained accuracy to detect the motion vector. The motion vector supplied from the unit 123 is output.

  Next, a motion estimation process executed by the motion estimation unit 52 will be described with reference to the flowchart of FIG. Although the motion vector is obtained for a point, the accuracy is calculated using image data of a small block in the vicinity of two points associated by the motion vector, for example, centering on the point.

  In step S91, the motion vector detection unit 123 detects a motion vector from the input image. For this detection, for example, a block matching method or a gradient method is used. The detected motion vector is supplied to the evaluation value calculation unit 121 and the integration processing unit 126.

  In step S92, the evaluation value calculation unit 121 calculates an evaluation value related to the degree of coincidence of both objects associated with the motion vector detected in the process of step S91. Specifically, for example, the sum of absolute differences of pixel values of two blocks centered on two points associated with motion vectors is calculated. That is, the motion vector V (vx, vy) detected by the motion vector detection unit 123 in the process of step S91, the point P (Xp, Yp) on the image Fi of the previous frame in time based on the motion vector V (vx, vy) The relationship between the point Q (Xq, Yq) on the image Fj of the subsequent frame is expressed by the following equation (1).

  The evaluation value calculation unit 121 calculates an evaluation value Eval (P, Q, i, j) based on the following equation (2) for a block centered on the point P and a block centered on the point Q.

  Each block is a square having 2L + 1 pixels on one side. The sum ΣΣ in the above equation (2) is performed between corresponding pixels when x is from −L to L and y is from −L to L. Therefore, for example, when L = 2, nine differences are obtained, and the sum of the absolute values is calculated. The evaluation value indicates that the two blocks match well as the value approaches zero.

  The evaluation value calculation unit 121 supplies the calculated evaluation value to the normalization processing unit 125.

  In step S93, the activity calculation unit 122 calculates an activity from the input image. The activity is a feature amount representing the complexity of the image, and as shown in FIG. 19, the difference between the target pixel Y (x, y) and the adjacent 8 pixels Y (x + i, y + j) for each pixel. The average value of the sum of absolute values is calculated based on the following equation (3) as activity Activity (x, y) at the target pixel position.

  In the case of the example in FIG. 19, the value of the pixel of interest Y (x, y) located in the center among 3 × 3 pixels is 110, and the values of eight pixels adjacent to it are 80, 70, and 75, respectively. , 100, 100, 100, 80, 80, the activity Activity (x, y) is expressed by the following equation.

  Activity (x, y) = {| 80-110 | + | 70-110 | + | 75-110 | + | 100-110 | + | 100-110 | + | 80-110 | + | 80−110 |} /8=24.375.

  Similar processing is performed for all pixels in the frame.

  In order to calculate the motion vector accuracy in units of blocks, the sum of the activities of all the pixels in the block expressed by the following equation (4) is defined as the activity (block activity) Blockactivity (i, j) of the block.

  In addition, the activity may be a variance value, a dynamic range, or the like.

  In step S94, the threshold determination unit 124 compares the block activity calculated in the process of step S93 with a predetermined threshold set in advance. Then, a flag indicating whether or not the input block activity is larger than the threshold value is output to the integration processing unit 126.

  Specifically, as a result of the experiment, the block activity and the evaluation value have the relationship shown in FIG. 20 with the motion vector as a parameter. In FIG. 20, the horizontal axis represents the block activity Blockactivity (i, j), and the vertical axis represents the evaluation value Eval.

  When the motion is correctly detected (when the correct motion vector is given), the block activity and the value of the evaluation value are distributed in the region R1 below the curve 131 in the figure. On the other hand, when an incorrect motion (incorrect motion vector) is given, the block activity and the value of the evaluation value are distributed in a region R2 on the left side in the figure from the curve 132. Note that there is almost no distribution in the region other than the region R2 above the curve 132 and the region other than the region R1 below the curve 131. The curve 131 and the curve 132 intersect at the point P, and the value of the block activity at the point P is set as the threshold value THa. The threshold value THa means that if the value of the block activity is smaller than that, the corresponding motion vector may be incorrect (this will be described in detail later). The threshold value determination unit 124 outputs a flag indicating whether or not the value of the block activity input from the activity calculation unit 122 is greater than the threshold value THa to the integration processing unit 126.

  In step S95, the normalization processing unit 125 normalizes the evaluation value calculated in the process of step S92 based on the activity calculated in the process of step S93. Specifically, the normalization processing unit 125 calculates the motion vector accuracy VC according to the following equation (5).

  However, if the value of the motion vector accuracy VC is less than 0, the value is replaced with 0. Of the motion vector accuracy VC, the value obtained by dividing the evaluation value by the block activity is a straight line 133 whose position on the graph of FIG. Therefore, it indicates whether the region is in the lower region or the upper region in the figure. That is, if the slope of the straight line 133 is 1, and the value obtained by dividing the evaluation value by the block activity is larger than 1, the points corresponding to the value are points distributed in the area above the straight line 133. Means that. The motion vector accuracy VC obtained by subtracting this value from 1 means that the smaller the value, the higher the possibility that the corresponding points are distributed in the region R2.

  On the other hand, if the value obtained by dividing the evaluation value by the block activity is smaller than 1, it means that the points corresponding to the value are distributed in the lower region of the straight line 133 in the figure. The motion vector accuracy VC at that time means that the larger the value (closer to 0), the corresponding points are distributed in the region R1. The normalization processing unit 125 outputs the motion vector accuracy VC obtained by the calculation in this way to the integration processing unit 126.

  In step S96, the integration processing unit 126 executes integration processing. Details of this integration processing are shown in the flowchart of FIG.

  In step S101, the integration processing unit 126 determines whether or not the block activity is equal to or less than a threshold value THa. This determination is performed based on the flag supplied from the threshold determination unit 124. When determining that the block activity is equal to or less than the threshold value THa, the integration processing unit 126 sets the value of the motion vector accuracy VC calculated by the normalization processing unit 125 to 0 in step S102. If it is determined in step S101 that the activity value is greater than the threshold value THa, the processing in step S102 is skipped, and the value of the motion vector accuracy VC generated by the normalization processing unit 125 is output as it is together with the motion vector. Is done.

  This is because there is a possibility that a correct motion vector is not obtained if the block activity value is smaller than the threshold value THa even if the value of the motion vector accuracy VC calculated by the normalization processing unit 125 is positive. Because there is. That is, as shown in FIG. 20, between the origin O and the point P, the curve 132 protrudes downward from the curve 131 in the drawing (below the straight line 133). In a region R3 in which the value of the block activity is smaller than the threshold Tha and is surrounded by the curves 131 and 132, the value obtained by dividing the evaluation value by the block activity is distributed in both the regions R1 and R2. There is a high possibility that a correct motion vector is not obtained.

  Therefore, in such a distribution state, processing is performed assuming that the accuracy of the motion vector is low. For this reason, in step S102, even if the motion vector accuracy VC is positive, it is set to 0 if it is smaller than the threshold value Tha. In this way, when the value of the motion vector accuracy VC is positive, it is possible to reliably represent that the correct motion vector is obtained. In addition, the larger the value of the motion vector accuracy VC, the higher the probability that a correct motion vector is obtained (the probability that the distribution is included in the region R1 increases).

  This coincides with an empirical rule that, in general, it is difficult to detect a motion vector with high reliability in a region where the luminance change is small (region where the activity is small).

  Next, a configuration example of the background motion estimation unit 54 in FIG. 4 will be described with reference to FIG. As illustrated in FIG. 22, the background motion estimation unit 54 includes a frequency distribution calculation unit 141 and a background motion determination unit 142.

  The frequency distribution calculation unit 141 calculates a frequency distribution of motion vectors. However, this frequency is weighted so that a probable motion is weighted by using the motion vector accuracy VC supplied from the motion estimation unit 52 of FIG. Based on the frequency distribution calculated by the frequency distribution calculation unit 141, the background motion determination unit 142 performs a process of determining a motion with the maximum frequency as a background motion, and outputs the background motion to the region estimation related processing unit 55 in FIG. .

  The background motion estimation process executed by the background motion estimation unit 54 will be described with reference to the flowchart of FIG.

  In step S111, the frequency distribution calculation unit 141 calculates a motion frequency distribution. Specifically, the frequency distribution calculation unit 141 assumes that the x and y coordinates of a motion vector as a background motion candidate are expressed in a range of ± 16 pixels from the reference point, respectively (1089 (= 16 × 2 + 1) × (16 × 2 + 1)), that is, a box for coordinates corresponding to possible values of a motion vector, and when a motion vector is generated, 1 is added to the coordinates corresponding to the motion vector. In this way, the motion vector frequency distribution can be calculated.

  However, when one motion vector is generated, if 1 is added, if the frequency of occurrence of a motion vector with low accuracy is high, a motion vector with low certainty may be determined as the background motion. . Therefore, when a motion vector occurs, the frequency distribution calculation unit 141 does not add the value 1 to the box (coordinates) corresponding to the motion vector, but instead of adding the value 1 to the motion vector accuracy VC (= Motion vector accuracy VC). The value of the motion vector accuracy VC is normalized as a value between 0 and 1, and the closer the value is to 1, the higher the accuracy. Therefore, the frequency distribution obtained in this way is a frequency distribution obtained by weighting motion vectors based on their accuracy. This reduces the risk that a motion with low accuracy is determined as the background motion.

  Next, in step S112, the frequency distribution calculation unit 141 determines whether or not the process of calculating the motion frequency distribution has been completed for all blocks. When it is determined that there is an unprocessed block, the process returns to step S111, and the motion frequency distribution is calculated for the next block.

  As described above, the motion frequency distribution calculation processing is performed on the entire screen, and when it is determined in step S112 that the processing of all blocks has been completed, the process proceeds to step S113, where the background motion determination unit 142 A process for searching for the maximum value of the distribution is executed. That is, the background motion determination unit 142 selects a frequency having the maximum frequency from the frequencies calculated by the frequency distribution calculation unit 141, and determines a motion vector corresponding to the frequency as a motion vector of the background motion. The motion vector of the background motion is supplied to the region estimation related processing unit 55 in FIG. 4 and is used for the determination process of whether or not the full screen motion matches the background motion.

  Next, a configuration example of the scene change detection unit 53 in FIG. 4 will be described with reference to FIG. As illustrated in FIG. 24, the scene change detection unit 53 includes a motion vector accuracy average calculation unit 151 and a threshold determination unit 152.

  The motion vector accuracy average calculation unit 151 calculates the average value of the entire motion vector accuracy VC supplied from the motion estimation unit 52 in FIG. 4 and outputs the average value to the threshold determination unit 152. The threshold determination unit 152 compares the average value supplied from the motion vector accuracy average calculation unit 151 with a predetermined threshold, and determines whether or not the scene change is based on the comparison result. The result is output to the control unit 59 in FIG.

  The scene change detection process executed by the scene change detection unit 53 will be described with reference to the flowchart of FIG.

  In step S121, the motion vector accuracy average calculation unit 151 calculates the sum of vector accuracy. Specifically, the motion vector accuracy average calculation unit 151 executes a process of adding the value of the motion vector accuracy VC calculated for each block output from the integration processing unit 126 of the motion estimation unit 52.

  In step S122, the motion vector accuracy average calculation unit 151 determines whether or not the processing for calculating the sum of the vector accuracy VC has been completed for all the blocks. If the processing has not yet been completed, the processing of step S121 is repeated. . By repeating this process, the sum of motion vector accuracy VC of each block for one screen is calculated.

  If it is determined in step S122 that the calculation of the sum of motion vector accuracy VC for one entire screen has been completed, the process proceeds to step S123, and the motion vector accuracy average calculation unit 151 calculates the average value of the vector accuracy VC. Execute the process. Specifically, a value obtained by dividing the sum of the vector accuracy VC for one screen calculated in the process of step S121 by the added number of blocks is calculated as an average value.

  In step S124, the threshold determination unit 152 compares the average value of the motion vector accuracy VC calculated by the motion vector accuracy average calculation unit 151 in the process of step S123 with a preset threshold value, and determines whether or not the threshold value is smaller than the threshold value. Determine whether. In general, when a scene change occurs between two frames having different times in a moving image, there is no corresponding image, so even if a motion vector is calculated, the motion vector is not certain. Therefore, the threshold determination unit 152 proceeds to step S125 when determining that the average value of the vector accuracy VC is smaller than the threshold, and turns on the scene change flag, and proceeds to step S126 when determined to be equal to or greater than the threshold. Turn off the scene change flag. When the scene change flag is turned on, it indicates that there is a scene change, and when the scene change flag is turned off, it indicates that there is no scene change.

  This scene change flag is supplied to the control unit 59, and is used for determining whether or not there is a scene change in step S61 in FIG. 13 and for determining whether or not there is a scene change in step S72 in FIG.

  As described above, by configuring the object tracking unit 23 in FIG. 1, the object 41 (FIG. 3) to be tracked rotates, occlusion occurs, or the tracking point 41A of the object 41 is changed due to a scene change. Even when the image is temporarily not displayed, it is possible to accurately track the tracking point 41A of the moving object 41 in the image.

  By the way, when a desired tracking result is not obtained even though the tracking processing is performed by the object tracking unit 23, that is, for example, as shown in FIG. 26, the tracking point 161 is separated from the object 162 to be tracked. In this case, the user designates a predetermined direction with respect to the position where the current tracking point 161 is located via the instruction input unit 29, or moves the cursor in a direction where the object 162 to be tracked exists. Thus, the object position estimation unit 22 that has received the instruction can estimate the tracking position intended by the user.

  In the example of FIG. 26, since the object 162 exists in the upper left direction with respect to the tracking point 161, the user designates the upper left direction using the instruction input unit 29, or By specifying the cursor locus 163 by moving the cursor upward, the object position estimation unit 22 that has received the instruction has a predetermined size with the tracking direction or the cursor position as a reference and the user specified direction as an offset. It is possible to generate a detection target region 164, calculate the feature amount of pixels at a predetermined interval in the region, extract pixels that meet the conditions, and estimate the tracking position intended by the user.

  Next, a detailed configuration example and operation of the object position estimation unit 22 in FIG. 1 will be described.

  FIG. 27 is a block diagram illustrating a functional configuration example of the object position estimation unit 22. As shown in FIG. 27, the object position estimation unit 22 includes a parameter control unit 171, a detection target region generation unit 172, a feature amount calculation unit 173, and a region estimation unit 174.

  The parameter control unit 171 is a parameter that represents the size of the detection target region generated by the detection target region generation unit 172, a parameter that represents the pixel interval of sample points when the feature amount is calculated by the feature amount calculation unit 173, and A parameter representing a predetermined condition when the tracking position is extracted by the region estimation unit 174 is controlled.

  The detection target area generation unit 172 specifies a predetermined direction via the instruction input unit 29 for the position where the current tracking point is located by the user, or moves the cursor in the direction where the object to be tracked exists. When the cursor trajectory is specified, an area designating offset amount with respect to the tracking point or the cursor position is acquired, and a detection target area having a predetermined size in the user designated direction is generated based on the offset amount.

  More specifically, for example, as shown in FIG. 28, the number of samples is 10 at a predetermined pixel interval (every 16 pixels in the example of FIG. 28) in the horizontal direction with reference to a reference point 182 including the tracking point 181. Detection target area having a size such that the number of samples is 10 at a predetermined pixel interval (every 8 pixels in the example of FIG. 28) in the vertical direction and offset by the user-specified direction (in this case, diagonally upward to the left). 183 is generated.

  Here, the position of the detection target region 183 with respect to the tracking point 181 is controlled by the offset amount acquired from the parameter control unit 171. For example, as shown in FIG. 29, the tracking point 181 and the reference point 182 are at the same position. In some cases, the detection target region 183 is generated around the tracking point 181 as shown in FIG. Further, when there is no feature amount satisfying the extraction condition in the detection target region 183 by the region estimation unit 174 to be described later, for example, as illustrated in FIG. 31, the reference point 182-1 is generated as a reference. The size of the detection target region 183-1 thus enlarged is enlarged, and a detection target region 183-2 having 14 samples in the horizontal direction and 14 samples in the vertical direction is defined with reference to the new reference point 182-2. It is also possible to generate it.

  Returning to the description of FIG. The feature amount calculation unit 173 calculates a feature amount of the sample point by using a pixel at a position separated by a predetermined number of pixels in the detection target region generated by the detection target region generation unit 172 as a sample point. Examples of the feature amount include a motion vector calculated by a block matching method.

  In the example of FIGS. 28 to 31, the sample points are represented by black circles. In the detection target region 183, sample points are set at intervals of 16 pixels in the horizontal direction and every 8 pixels in the vertical direction.

  FIG. 32 schematically shows a motion vector of each sample point in the detection target region 183 calculated by the feature amount calculation unit 173. In the figure, the tracking target motion vector is represented by black circles and arrows, and the background motion vector is represented by white circles and arrows. Here, for example, as shown in FIG. 33, the background motion vector is calculated as the mode value of the motion vectors of the pixels arranged uniformly on the entire screen. That is, the motion vector represented by white circles and arrows in FIG. 33 is the most frequent motion, and the pixel having the motion vector is the background motion. On the other hand, a pixel having a motion vector represented by a black circle and an arrow is a tracking target motion.

  Returning to the description of FIG. The region estimation unit 174 extracts a pixel satisfying a predetermined condition from the motion vectors as the feature amount of each sample point in the detection target region calculated by the feature amount calculation unit 173, and is a region configured by the pixels Is generated. Here, the predetermined condition is a pixel having a motion vector other than the background motion.

  FIG. 34 schematically shows a region 191 composed of a pixel group having a motion vector other than the background motion generated by the region estimation unit 174. In the figure, the tracking target motion vector is represented by black circles and arrows, and the background motion vector is represented by white circles and arrows. As described above, the region estimation unit 174 generates a region 191 composed of a pixel group having a motion vector to be tracked.

  Further, the region estimation unit 174 calculates the barycentric position of the generated region. Thereby, for example, as shown in FIG. 35, the barycentric position 201 of the region 191 is calculated. Furthermore, the region estimation unit 174 estimates the calculated center-of-gravity position 201 as a tracking point intended by the user, and outputs position information representing the tracking position as an estimation result to the object tracking unit 23. As a result, the tracking process is continued by the object tracking unit 23, and the image display 25 displays, as shown in FIG. 36, for example, a mark superimposed display representing the center of gravity position 201 as the tracking point intended by the user.

  As described above, the pixel having the motion vector other than the background motion vector, that is, the pixel having the motion vector to be tracked is extracted from the feature quantity of the motion vector, and the barycentric position of the region configured by the extracted pixel group And the center of gravity position can be estimated as the tracking point intended by the user.

  In addition to the motion vector, the feature amount includes color, brightness, and the like. The condition extracted by the region estimation unit 174 is, for example, a pixel having a feature quantity different from the feature quantity of the tracking point 181, a pixel having a feature quantity arbitrarily designated by the user, or a feature quantity Can be used as a pixel or the like whose reliability satisfies a threshold value.

  As an example of the reliability, it is determined how many pixels have the same feature quantity as the feature quantity of the target pixel in the peripheral pixels centered on the target pixel. For example, if five or more pixels have the same feature amount among the eight surrounding pixels centered on the target pixel, it is determined that the feature amount of the target pixel is reliable.

  Furthermore, it is possible to set the mode having the nth feature amount from the mode value and the mode value among the extraction conditions as the extraction condition. For example, in FIG. 35, it is possible to calculate the center of gravity of the region 191 by determining the frequency of the feature amount without extracting the background motion in advance and extracting the pixel having the second feature amount from the mode value. Can do. In the example of FIG. 35, the pixel having the mode feature value is represented by a white circle and an arrow, the pixel having the second frequency feature amount is represented by a black circle and an arrow, and the mode feature amount is The background motion vector is used.

  As shown in FIG. 37, when two objects (two cars in the figure) are in the detection target area 183, the pixel having the third feature amount from the mode value is set as the extraction condition. Thus, even a small object can be estimated as a tracking target. In the example of FIG. 37, the pixel having the mode value feature amount is represented by a white circle, the pixel having the second frequency feature amount is represented by a black circle, and the pixel having the third frequency feature amount is represented by It is represented by a white triangle.

  Next, tracking position estimation processing executed by the object position estimation unit 22 will be described with reference to the flowchart of FIG. This process is started when a predetermined direction is designated by the user via the instruction input unit 29 or when the cursor is moved in the direction in which the object to be tracked exists and the cursor locus is designated.

  In step S131, the detection target area generation unit 172 acquires a tracking point or a cursor position specified by the user, and in step S132, acquires an offset amount for area specification with respect to the acquired tracking point or cursor position.

  In step S133, the detection target region generation unit 172 acquires from the parameter control unit 171 a parameter that represents the size of the detection target region and a parameter that represents the pixel interval of the sample points in the detection target region. In step S134, the detection target area generation unit 172 obtains a detection target area based on the parameter acquired in the process of step S133. For example, as shown in FIG. 28, a detection target region 183 having a size of 10 samples at intervals of 16 pixels in the horizontal direction and 10 samples at intervals of 8 pixels in the vertical direction is generated. Is done.

  In step S135, the feature amount calculation unit 173 calculates a motion vector as the feature amount of the sample point in the detection target region 183 generated in the process of step S134. In step S136, the feature amount calculation unit 173 determines whether or not the feature amount of all the sample points in the detection target region 183 has been calculated, and when it is determined that there is a sample point for which the feature amount has not yet been calculated. Returning to step S135, the subsequent processing is repeated.

  If it is determined in step S136 that the feature amounts of all the sample points in the detection target region 183 have been calculated, the process proceeds to step S137, and the region estimation unit 174 obtains feature amounts that satisfy the condition. Here, motion vectors other than the background motion are extracted and the number of extractions is held.

  In step S138, the region estimation unit 174 extracts pixels having feature quantities that satisfy the conditions. Here, pixels having motion vectors other than the background motion are extracted.

  In step S139, the region estimation unit 174 determines whether or not the predetermined number has been checked, that is, whether or not only the number of extractions retained in the process of step S137 has been checked. Returning to S138, the subsequent processing is repeated.

  If it is determined in step S139 that the predetermined number has been examined, the process proceeds to step S140, where the region estimation unit 174 generates a region composed of the extracted pixel group, and calculates the barycentric position of the region. For example, as shown in FIG. 35, a region 191 composed of the extracted pixel group is generated, and the barycentric position 201 of the region 191 is calculated.

  In step S141, the region estimation unit 174 estimates the position of the center of gravity calculated in the process of step S140 as the tracking point intended by the user, and outputs the tracking point as the estimation result to the object tracking unit 23.

  Thus, when a desired tracking result is not obtained despite the tracking process being performed, the tracking position intended by the user can be estimated in real time based on the designation by the user. As a result, the object tracking unit 23 can continuously perform the tracking process of the tracking point estimated by the object position estimating unit 22, thereby reducing the load at the time of specifying or correcting the tracking point and more suitable tracking. The result can be obtained.

  Further, the object tracking unit 23 outputs position information representing the tracking position as a tracking result to the tracking position correcting unit 24. As a result, the tracking position correction unit 24 can finely adjust the position information indicating the tracking position so that it becomes the barycentric position of the object to which the tracking point belongs.

  Next, an example in which the tracking position correction unit 24 finely adjusts the tracking result output from the object tracking unit 23 will be described.

  For example, as shown in FIG. 39, when the tracking point intended by the user is estimated from the position x (t) to the position x ′ (t + 1) as described above, the estimated position x ′ (t + 1) Is calculated, and the tracking point is finely adjusted to the center of gravity x (t + 1) of the object. The fine adjustment result is supplied from the tracking position correcting unit 24 to the object tracking unit 23, and thereafter the tracking process is continued from the position of the finely adjusted tracking point x (t + 1).

  Here, a method for obtaining the range of the object to which the tracking point belongs based on the pixel value (color information) called color and calculating the center of gravity will be described.

(1) First, the RGB color (r ′, g ′, b ′) at the position x ′ (t + 1) is calculated. The color system may not be RGB.
(2) Regarding the peripheral pixel at the position x ′ (t + 1) where the RGB color is calculated, it is determined that the pixel having a color close to the color at the position x ′ (t + 1) belongs to the same object. For example, it is determined that a pixel in which Δc expressed by the following equation (6) is smaller than a predetermined value belongs to the same object as the position x ′ (t + 1). r, g, and b are the colors of each peripheral pixel.

（３）位置ｘ'（ｔ＋１）と同じオブジェクトに属している画素の位置から、次式（７）に基づいて、オブジェクトの重心ｘ（ｔ＋１）を算出する。Ｘｉは、同じオブジェクトに属している各画素の位置であり、ｎは、同じオブジェクトに属している画素の総数である。

(3) From the position of the pixel belonging to the same object as the position x ′ (t + 1), the center of gravity x (t + 1) of the object is calculated based on the following equation (7). Xi is the position of each pixel belonging to the same object, and n is the total number of pixels belonging to the same object.

  As described above, it is possible to fine-tune the tracking point to the center of gravity of the object to which the tracking point belongs by obtaining the range of the object to which the tracking point belongs by the method using the color information and calculating the center of gravity position thereof. It is.

  Note that the present invention is not limited to the method using color information, and for example, a range of an object may be obtained based on a motion vector, a luminance value, or the like, and the center position may be calculated. For example, when a motion vector is used, it is determined that a vector having the same or similar vector as the motion vector at the position x ′ (t + 1) belongs to the same object, and the range of the object is obtained as follows. The center of gravity can be calculated.

(1) Motion Vector Detection As shown in FIG. 40, a motion vector is detected at each sampling interval (sx, sy) within an area 222 centered on the tracking point 221. The size of the region 222 is sx · m × sy · n as the number of samples m and n. For the detection of the motion vector, for example, a block matching method or a gradient method is used.
(2) Motion vector frequency distribution calculation For example, motion candidates within the processing range are represented by integer values of −16 <= vx <= 16 and −16 <= vy <= 16 (vx is horizontal motion, vy is vertical motion) In this case, 33 (= 16 × 2 + 1) pixels × 33 pixels = 1089 boxes, that is, boxes corresponding to coordinates corresponding to possible values of the motion vector are prepared, and at a certain sample point (vx, vy) = When (2, 2), 1 is added to the box of (2, 2). By performing this processing on all sample points within the processing range, the frequency distribution of the motion vector can be calculated.
(3) Extraction of sample points to be tracked As shown in FIG. 41, sample points showing movement similar to the movement occupying a large number in the area 222 (in the example of FIG. 41, the movement in the diagonally downward left direction) are tracked. Extract as a point on the object.
(4) Gravity calculation of sample point Flag (i, j) (1 <= i <= m, 1 <= j <= n) indicating whether or not the sample point is a sample point on the tracking target in the region 222 Set as. If it is extracted as a sample point on the tracking target in (3), it is set to 1. If it is not extracted as a sample point on the tracking target in (3), it is set to 0. The center of gravity G (x, y) of the sample point is calculated as the sample point P (x, y) based on the following equation (8).

  As described above, the range of the object to which the tracking point belongs can be obtained by the method using the motion vector, and the center of gravity position can be calculated. In addition to the estimation of the tracking point intended by the user, the finely adjusted previous tracking point is characterized by high robustness against disturbance, that is, easy tracking processing.

  FIG. 42 is a diagram illustrating an example of an image displayed on the image display 25 after the tracking point has been finely adjusted by the tracking position correction unit 24. In FIG. 42, the range of the object (automobile in the example of FIG. 42) to which the tracking position correction unit 24 belongs from the position 221 estimated as the tracking point intended by the user by the object position estimation unit 22. It is shown that the center of gravity position 231 is calculated and finely adjusted there.

  In this way, in the tracking process, in addition to the estimation of the tracking point intended by the user, it can be finely adjusted to a position that is easy to perform the tracking process, that is, the position of the center of gravity of the tracking target, so that a more suitable tracking result can be obtained. It becomes.

  Next, another example of the exception process in step S12 performed following the normal process in step S11 of FIG. 5 will be described with reference to the flowchart of FIG. This process is basically the same as the process of FIG. 12, and it is determined in step S24 of FIG. 9 that it is impossible to estimate the movement of the tracking point, and further, the transfer candidate for changing the tracking point in step S28. This is executed when it is determined that cannot be selected.

  Since the processing of steps S151 to S155 is the same as the processing of steps S51 to S55 of FIG. 12 described above, the description thereof will be simplified. That is, in step S151, an exception process initialization process is executed. In step S152, an input of an image of the next frame is awaited. In step S153, a template matching process is performed within the template search range, and in step S154. When it is determined whether or not it is possible to return to the normal process, and when it is determined that the return to the normal process is possible, the process returns to the normal process of step S11 in FIG. If it is determined that it is not possible, a continuation determination process is executed in step S155.

  In step S156, the control unit 59 determines whether or not tracking of the tracking point in the exception process by the continuation determination process in step S155 can be continued (in steps S76 and 78 in FIG. 16). Determine based on the flag set. If it is determined that the tracking process of the tracking point can be continued, the process returns to step S52, and the subsequent processes are repeatedly executed. That is, the process of waiting until the tracking point appears again is repeatedly executed.

  On the other hand, when it is determined in step S156 that the tracking process of the tracking point in the exception process cannot be continued (the number of frames that have elapsed after the tracking point disappeared in the process of step S75 in FIG. 16 is the threshold THfr). If it is determined as above, or if the number of scene changes is determined to be greater than or equal to the threshold THsc in the process of step S77), the process proceeds to step S157, and the control unit 59 controls and holds the tracking point determination unit 57. Set the tracking point. Thereafter, the process returns to the normal process in step S11 of FIG.

  In this way, when tracking of the tracking point in exception processing becomes impossible, it is possible to return to normal processing again using the tracking point that has been held, so the tracking processing is continued. Can be performed.

  In the above, the tracking position can be estimated when the tracking position designation button is pressed or the cursor movement is stopped, without depending on the user interface.

  In the above description, an example in which the present invention is applied to a surveillance camera system has been described. However, the present invention is not limited thereto, and can be applied to various image processing apparatuses such as a television receiver.

  Further, in the above, the image processing unit is a frame, but it is of course possible to use a field as a processing unit.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  As shown in FIG. 1, a program recording medium for storing a program that is installed in a computer and can be executed by the computer is a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only). Memory), DVD (including Digital Versatile Disc), magneto-optical disc, or removable media 28, which is a package media composed of semiconductor memory, or a hard disk on which a program is temporarily or permanently stored . The program is stored in the program recording medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via an interface such as a router or a modem as necessary.

  In the present specification, the steps for describing a program are not only processes performed in time series in the order described, but also processes that are executed in parallel or individually even if they are not necessarily processed in time series. Is also included.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

It is a block diagram which shows the structural example of the surveillance camera system to which this invention is applied. It is a flowchart explaining a monitoring process. It is a figure which shows the example of the image displayed by the surveillance camera system of FIG. It is a block diagram which shows the structural example of the object tracking part of FIG. It is a flowchart figure explaining a tracking process. It is a figure explaining tracking when a tracking object rotates. It is a figure explaining the tracking when an occlusion occurs. It is a figure explaining the tracking when a scene change occurs. It is a flowchart explaining a normal process. It is a flowchart explaining the initialization process of a normal process. It is a figure explaining the initialization process of a normal process. It is a flowchart explaining exception processing. It is a flowchart explaining the initialization process of an exception process. It is a figure explaining selection of a template. It is a figure explaining the setting of a search range. It is a flowchart explaining a continuation determination process. It is a block diagram which shows the structural example of a motion estimation part. It is a flowchart explaining a motion estimation process. It is a figure explaining calculation of activity. It is a figure explaining the relationship between an evaluation value and activity. It is a flowchart explaining an integration process. It is a block diagram which shows the structural example of a background motion estimation part. It is a flowchart explaining a background motion estimation process. It is a block diagram which shows the structural example of a scene change detection part. It is a flowchart explaining a scene change detection process. It is a figure explaining estimation of a tracking position. It is a block diagram which shows the functional structural example of the object position estimation part of FIG. It is a figure which shows the example of a detection object area | region. It is a figure which shows the other example of a detection object area | region. It is a figure which shows the other example of a detection object area | region. It is a figure which shows the example which expanded the magnitude | size of the detection object area | region. It is a figure which shows typically the motion vector of each sample point in a detection object area | region. It is a figure explaining the motion of a background. It is a figure which shows typically the area | region comprised with the pixel which has a motion vector other than a background motion. It is a figure explaining the process which calculates the gravity center position of the area | region of FIG. It is a figure which shows the example of the output image after a tracking process. It is a figure explaining the other process which calculates the gravity center position of an area | region. It is a flowchart explaining a tracking position estimation process. It is a figure explaining the example which fine-tunes a tracking point. It is a figure explaining a motion vector detection process. It is a figure explaining the sample point extraction process of tracking object. It is a figure explaining tracking point fine adjustment processing. It is a flowchart explaining the exception process of another example.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Surveillance camera system, 21 Image pick-up part, 22 Object position estimation part, 23 Object tracking part, 24 Tracking position correction part, 51 Template matching part, 52 Motion estimation part, 53 Scene change detection part, 54 Background motion estimation part, 55 area | region Estimation related processing unit, 56 transfer candidate holding unit, 57 tracking point determination unit, 58 template holding unit, 171 parameter control unit, 172 detection target region generation unit, 173 feature quantity calculation unit, 174 region estimation unit

Claims (14)

In an image processing apparatus that tracks a moving object,
Generating means for generating a detection target region based on a direction designated by a user with respect to a current tracking point or a cursor position on the object in an image;
A calculation unit that calculates a feature amount of a pixel included in the detection target region generated by the generation unit;
Extraction means for extracting predetermined pixels based on the feature amount calculated by the calculation means;
An image processing apparatus comprising: an estimation unit that estimates a center of gravity position of an area composed of the predetermined pixels extracted by the extraction unit as a tracking point intended by the user. The image processing apparatus according to claim 1, further comprising a fine adjustment unit that finely adjusts the tracking point estimated by the estimation unit so that the tracking point becomes a center of gravity position of the object. The image processing apparatus according to claim 1, wherein the generation unit generates a detection target area according to an offset amount designated by the user. The image processing apparatus according to claim 1, wherein the generation unit generates a detection target region centered on a current tracking point or a cursor position on the object. The image processing apparatus according to claim 1, wherein when the predetermined pixel cannot be extracted by the extraction unit, the generation unit generates a new detection target region in which the detection target region is enlarged. The image processing apparatus according to claim 1, wherein the extraction unit extracts a pixel having a feature amount different from a feature amount of the current tracking point or cursor position. The image processing apparatus according to claim 1, wherein the extraction unit extracts a pixel having a feature quantity different from a background feature quantity. The image processing apparatus according to claim 1, wherein the extraction unit extracts a pixel having a feature amount designated by the user. The image processing apparatus according to claim 1, wherein the extraction unit extracts a pixel having a feature quantity having a frequency with a rank specified by the user from the calculated frequency of the feature quantity. 2. The image processing according to claim 1, wherein the extraction unit extracts a pixel having a feature quantity having a frequency with a rank specified by the user from a feature quantity frequency different from the feature quantity of the current tracking point or cursor position. apparatus. The estimation means estimates a barycentric position of an area composed of pixels having a feature amount having a reliability greater than or equal to a predetermined threshold among the extracted predetermined pixels as a tracking point intended by the user. Item 8. The image processing apparatus according to Item 1. The image processing apparatus according to claim 1, wherein the feature amount is a motion vector, a color, or luminance. In an image processing method of an image processing apparatus for tracking a moving object,
Based on the direction specified by the user for the current tracking point or cursor position on the object in the image,
Calculating a feature amount of a pixel included in the generated detection target region;
Based on the calculated feature value, a predetermined pixel is extracted,
An image processing method including a step of estimating a center of gravity position of an area composed of the extracted predetermined pixels as a tracking point intended by the user. In a program for causing a computer to execute image processing of an image processing apparatus that tracks a moving object,
Based on the direction specified by the user for the current tracking point or cursor position on the object in the image,
Calculating a feature amount of a pixel included in the generated detection target region;
Based on the calculated feature value, a predetermined pixel is extracted,
A program including a step of estimating a center of gravity position of an area composed of the extracted predetermined pixels as a tracking point intended by the user.

Images