JP2009010453A - Image processing apparatus and method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To correct tracking point at a predetermined position which corresponds according, to a correction indication from a user during tracking of an object. <P>SOLUTION: An imaging unit 21 images the circumference of the object, and a tracking object detection unit 22 detects the tracking point of the moving object. An object tracking unit 23 tracks the tracking point of the object and outputs the tracking result to a tracking position correcting unit 24. When the user indicates correction of the tracking position via an indication input unit 29, the tracking position correcting unit 24 reflects a correction value from the user on the tracking result from the object tracking unit 23, enlarges the image at an enlargement rate corresponding to a correction state, and outputs the image to an image display 25. The present invention is applicable to a surveillance camera system. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, method, and program, and more particularly to an image processing apparatus, method, and method capable of correcting a tracking point at a predetermined position according to a correction instruction from a user during tracking of an object. Regarding the program.

  Conventionally, many methods for tracking an object designated by a user among images displayed as moving images have been proposed. Most of them are to perform tracking processing fully automatically after first instructing the tracking target.

  However, in practice, a desired tracking result cannot be obtained by fully automatic processing due to disturbances such as image noise and long-time occlusion.

Therefore, several methods for correcting the tracking result by the user have been proposed. For example, Patent Document 1 automatically detects and tracks a moving body in an image, and when a desired result is not obtained, the user manually operates. For example, Patent Literature 2 accepts correction from the user for the tracking result and performs the tracking process again.
JP 2005-244562 A JP 2005-165929 A

  By the way, Patent Document 1 automatically detects a target, and cannot track a desired target. Further, the operation is merely switched from the automatic detection / tracking to the operation by the user, and the desired correction instruction by the user cannot be reflected in the subsequent tracking process.

  Further, in order to reflect the correction instruction in Patent Document 2, it is necessary to instruct the start of the tracking process again after interrupting the tracking process and reflecting the correction, and correction cannot be performed in real time.

  As described above, when there is a target correction instruction from the user during the tracking process, the tracking process cannot be continued while reflecting the instruction in real time.

  The present invention has been made in view of such circumstances, and enables tracking points to be corrected to a predetermined position according to a correction instruction from a user during tracking of an object.

  An image processing apparatus according to one aspect of the present invention is an image processing apparatus that tracks a moving object, the tracking unit that tracks the movement of the object in an image, and the tracking unit that is tracking the object. When the correction of the tracking point on the object is instructed, reflecting means for reflecting the correction instruction on the tracking result while continuing the tracking processing of the object, and when the correction is instructed, the enlargement ratio of the display of the image Is controlled by autonomous determination, and is provided with display control means for enlarging and displaying the image at the adjusted enlargement ratio.

  When the correction is started as the adjustment of the enlargement ratio, the display control means decreases the enlargement ratio from the first enlargement ratio to the second enlargement ratio, and sets the enlargement ratio to the second enlargement ratio. When the magnification is held and the correction is completed, the magnification is increased from the second magnification to the first magnification.

  The display control means adjusts the enlargement ratio in the first period from the first enlargement ratio to the second enlargement ratio based on the change rate of the enlargement ratio during the first period. An enlargement factor is calculated, and the adjustment of the enlargement factor in the second period from the second enlargement factor to the first enlargement factor is based on the change rate of the enlargement factor during the second period. Calculate the magnification.

  The first period and the second period are defined in advance, and the display control unit adjusts the change rate based on the first period for adjusting the enlargement ratio in the first period. And calculating the enlargement rate using the change rate, and adjusting the enlargement rate during the second change period, calculating the change rate based on the second period, and changing the change rate. Is used to calculate the magnification.

  The change rate during the first period and the change rate during the second period are defined in advance.

  The display control means adjusts the magnification rate by changing the second magnification rate.

  The display control means adjusts the enlargement ratio during the first period by changing a change rate of the enlargement ratio during the first period from the first enlargement ratio to the second enlargement ratio. The magnification rate change rate during the second period from the second magnification rate to the first magnification rate is varied to adjust the magnification rate during the second period.

  Calculation means for calculating a correction value based on the correction instruction is further provided.

  The reflecting means reflects the correction value calculated by the calculating means to the tracking result as a relative value with respect to the coordinate system of the image.

  The reflection unit reflects the correction value calculated by the calculation unit in the tracking result as a relative value with respect to the tracking target.

  After the correction instruction is reflected by the reflecting means, fine adjustment means is further provided for finely adjusting the tracking point so that it becomes the center of gravity of the object.

  An image processing method and program according to one aspect of the present invention are an image processing method and program corresponding to the above-described image processing apparatus according to one aspect of the present invention.

  In the image processing apparatus, method, and program according to one aspect of the present invention, when movement of an object in an image is tracked and correction of a tracking point on the object is instructed during tracking of the object, While the tracking process is continued, the correction instruction is reflected in the tracking result. Further, when the correction is instructed, the enlargement ratio of the image display is adjusted by the autonomous determination of the apparatus, and the image is enlarged and displayed at the adjusted enlargement ratio.

  As described above, according to one aspect of the present invention, a tracking point can be corrected to a predetermined position according to a correction instruction from a user during tracking of an object.

  Embodiments of the present invention will be described below. Correspondences between the configuration requirements of the present invention and the embodiments described in the detailed description of the present invention are exemplified as follows. This description is to confirm that the embodiments supporting the present invention are described in the detailed description of the invention. Accordingly, although there are embodiments that are described in the detailed description of the invention but are not described here as embodiments corresponding to the constituent elements of the present invention, It does not mean that the embodiment does not correspond to the configuration requirements. Conversely, even if an embodiment is described herein as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

  Further, this description does not mean that all the inventions corresponding to the specific examples described in the embodiments of the invention are described in the claims. In other words, this description is an invention corresponding to the specific example described in the embodiment of the invention, and the existence of an invention not described in the claims of this application, that is, in the future, a divisional application will be made. Nor does it deny the existence of an invention added by amendment.

An image processing apparatus according to one aspect of the present invention (for example, the surveillance camera system 1 in FIG. 1) includes:
An image processing apparatus for tracking a moving object,
Tracking means for tracking the movement of the object in the image (for example, the object tracking unit 23 in FIG. 1);
When correction of the tracking point on the object is instructed during tracking of the object by the tracking means, reflecting means for reflecting the correction instruction on the tracking result while continuing the tracking processing of the object (for example, the correction in FIG. 26) Value reflection unit 162),
When the correction is instructed, display control means (for example, the control unit 27 in FIG. 1) that adjusts an enlargement ratio of the display of the image by autonomous determination and enlarges and displays the image at the adjusted enlargement ratio; An image processing apparatus.

  When the correction is started (for example, at time t0 in FIG. 39) as the adjustment of the enlargement ratio, the display control means changes the enlargement ratio from the first enlargement ratio (for example, the enlargement ratio Za in FIG. 39). The size is reduced to a second enlargement rate (eg, enlargement rate Zb in FIG. 39) (see, for example, the period from time t0 to time t1 in FIG. 39), and the enlargement rate is held at the second enlargement rate (eg, 39 (see the period from time t1 to time t2 in FIG. 39), when the correction is completed (for example, at time t2 in FIG. 39), the enlargement ratio is increased from the second enlargement ratio to the first enlargement ratio. (For example, refer to the period from time t2 to time t3 in FIG. 39).

  Further, a calculation unit (for example, a correction value calculation unit 161 in FIG. 26) that calculates a correction value based on the correction instruction is further provided.

  Further, after the correction instruction is reflected by the reflection means, fine adjustment means (for example, a fine adjustment unit 163 in FIG. 26) for finely adjusting the tracking point so as to be located at the center of gravity of the object is further provided.

An image processing method according to one aspect of the present invention includes:
An image processing method of an image processing apparatus (for example, the surveillance camera system 1 in FIG. 1) that tracks a moving object,
Tracking the movement of the object in the image (eg, step S11 in FIG. 5);
When correction of the tracking point on the object is instructed during tracking of the object (for example, when it is determined YES in step S154 of FIG. 36), the correction is performed while continuing the tracking processing of the object. The instruction is reflected in the tracking result (for example, step S156 in FIG. 36),
When the correction is instructed, the enlargement ratio of the image display is adjusted by autonomous determination, and the image is enlarged and displayed at the adjusted enlargement ratio (for example, steps S157 and S158 in FIG. 36).
Includes steps.

  The program according to one aspect of the present invention is a program corresponding to the information processing method according to one aspect of the present invention described above, and is executed by, for example, the computer of FIG.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a configuration example when the present invention is applied to a surveillance camera system.

  The imaging unit 21 includes, for example, a CCD (Charge Coupled Device) video camera or the like, and displays the captured image on the image display 25. The tracking target detection unit 22 detects the tracking target from the image input from the imaging unit 21, and outputs the detection result to the object tracking unit 23.

  The object tracking unit 23 operates to track the tracking point designated by the tracking target detection unit 22 in the image supplied from the imaging unit 21. The object tracking unit 23 outputs the tracking result to the tracking position correcting unit 24 and controls the camera driving unit 26 so that the moved object can be imaged based on the tracking result. When the tracking position correction unit 24 is instructed by the user via the instruction input unit 29 to correct the tracking position, the tracking position correction unit 24 reflects the correction value from the user in the tracking result from the object tracking unit 23 and displays the correction result on the image display. The correction result is supplied to the object tracking unit 23. As a result, the object tracking unit 23 can operate to track the corrected tracking point from the next frame. Based on the control from the object tracking unit 23, the camera driving unit 26 drives the imaging unit 21 so that the imaging unit 21 captures an image centered on the tracking point.

  The control unit 27 is configured by, for example, a microcomputer and controls each unit. The control unit 27 is connected to a removable medium 28 formed of a semiconductor memory, a magnetic disk, an optical disk, a magneto-optical disk, or the like as needed, and a program and other various data are supplied as needed. The instruction input unit 29 includes various buttons, switches, a remote controller using infrared rays or radio waves, and outputs a signal corresponding to an instruction from the user to the control unit 27.

  Next, monitoring processing executed by the monitoring camera system shown in FIG. 1 will be described with reference to the flowchart of FIG. In this processing, when the power of the monitoring system 1 is turned on, an area to be monitored is imaged by the imaging unit 21, and an image obtained by the imaging is imaged via the tracking target detection unit 22 and the object tracking unit 23. It is output to the display 25 and started.

  In step S <b> 1, the tracking target detection unit 22 executes processing for detecting a tracking target from the image input from the imaging unit 21. For example, when there is a moving object in the input image, the tracking target detection unit 22 detects the moving object as a tracking target, and selects a point with the highest luminance or a center point of the tracking target from the tracking target. Detection is performed as a tracking point, and the detection result is output to the object tracking unit 23. The tracking target can of course be set by the user specifying a predetermined position via the instruction input unit 29.

  In step S2, the object tracking unit 23 executes a tracking process for tracking the tracking point detected in the process of step S1. The details of the tracking process will be described later with reference to FIG. 5. With this process, a tracking point in an object (for example, a person, an animal, etc.) to be tracked in an image captured by the imaging unit 21 is obtained. (For example, the center of the eyes and head) is tracked, the tracking result is output to the control unit 27, and the position information indicating the tracking position is output to the tracking position correcting unit 24.

  In step S3, the control unit 27 causes the image display 25 to superimpose and display a mark representing the tracking position on the image captured by the imaging unit 21, based on the tracking result obtained in step S2.

  In step S4, the object tracking unit 23 detects the movement of the object based on the tracking result in the process of step S2, generates a camera driving signal for driving the camera so that the moved object can be imaged, and the camera driving unit. 26. In step S <b> 5, the camera drive unit 26 drives the imaging unit 21 based on the camera drive signal from the object tracking unit 23. Thereby, the imaging unit 21 pans or tilts the camera so that the tracking point does not deviate from the screen.

  In step S6, the control unit 27 determines whether or not to end the monitoring process based on an instruction from the user via the instruction input unit 29. If the end is not instructed by the user, the control unit 27 proceeds to step S1. Return and repeat the subsequent processing. If it is determined in step S6 that the user has instructed the end of the monitoring process, the control unit 27 ends the monitoring process.

  3A to 3C are diagrams showing examples of images displayed on the image display 25 at this time in time series. FIG. 3A is an example of an image in which an object 41 to be tracked is imaged by the imaging unit 21. In this example, a person who moves in the left direction in the figure is imaged as the object 41. In FIG. 3B, the object 41 has moved to the left in the figure from the position of FIG. 3A, and in FIG. 3C, the object 41 has moved further to the left from the position of FIG. 3B.

  The tracking target detection unit 22 detects the object 41 in step S1 of FIG. 2, and outputs the human eye that is the object 41 to the object tracking unit 23 as the tracking point 41A. In step S2, the object tracking unit 23 performs a tracking process.

  Next, a detailed configuration example and operation of the object tracking unit 23 in FIG. 1 will be described.

  FIG. 4 is a block diagram illustrating a functional configuration example of the object tracking unit 23. As shown in FIG. 4, the object tracking unit 23 includes a template matching unit 51, a motion estimation unit 52, a scene change detection unit 53, a background motion estimation unit 54, a region estimation related processing unit 55, a transfer candidate holding unit 56, and a tracking. The point determination unit 57, the template holding unit 58, and the control unit 59 are configured.

  The template matching unit 51 performs a matching process between the input image and the template image held in the template holding unit 58. The motion estimation unit 52 estimates the motion of the input image, and obtains the motion vector obtained as a result of the estimation and the accuracy of the motion vector, the scene change detection unit 53, the background motion estimation unit 54, the region estimation related processing unit 55, And output to the tracking point determination unit 57. The scene change detection unit 53 detects a scene change based on the motion vector supplied from the motion estimation unit 52 and the accuracy of the motion vector.

  The background motion estimation unit 54 executes processing for estimating the background motion based on the motion vector supplied from the motion estimation unit 52 and the accuracy of the motion vector, and supplies the estimation result to the region estimation related processing unit 55. The area estimation related processing unit 55 includes the motion vector supplied from the motion estimation unit 52 and the accuracy of the motion vector, the background motion supplied from the background motion estimation unit 54, and the tracking point information supplied from the tracking point determination unit 57. Based on the above, region estimation processing is performed. The region estimation related processing unit 55 generates a transfer candidate based on the input information, supplies the transfer candidate to the transfer candidate holding unit 56, and holds it. Further, the region estimation related processing unit 55 creates a template based on the input image, supplies the template to the template holding unit 58, and holds it.

  The tracking point determination unit 57 determines the tracking point based on the motion vector supplied from the motion estimation unit 52, the accuracy of the motion vector, and the transfer candidate supplied from the transfer candidate holding unit 56. Information about the points is output to the region estimation related processing unit 55.

  The control unit 59 controls each part of the template matching unit 51 to the template holding unit 58 based on the tracking point information output from the tracking target detection unit 22 to track the detected tracking target and display an image display. A control signal is output to the camera drive unit 26 so that the tracking point is displayed in the screen displayed on the screen 25, and the drive of the imaging unit 21 is controlled. Thereby, the tracking point is controlled so as not to go out of the screen. Further, the control unit 59 outputs a tracking result such as information on the position of the tracking point on the screen to the tracking position correction unit 24 and the control unit 27.

  Next, the operation of the object tracking unit 23 will be described.

  FIG. 5 is a flowchart illustrating details of the tracking process executed by the object tracking unit 23 in step S2 of FIG.

  As shown in FIG. 5, the object tracking unit 23 basically executes normal processing and exception processing. That is, normal processing is performed in step S11. The details of this normal process will be described later with reference to the flowchart of FIG. 9, and the process of tracking the tracking point detected by the tracking target detection unit 22 is executed by this process.

  When the tracking point cannot be tracked in the normal process of step S11, an exception process is executed in step S12. Details of this exception processing will be described later with reference to the flowchart of FIG. 12. When the tracking point is not visible from the image by this exception processing, a return processing to the normal processing is executed by template matching. If it is determined that the tracking process cannot be continued due to the exception process, that is, the process cannot be returned to the normal process, the process is terminated. If it is determined that it is possible to return, the process returns to step S11 again. In this manner, the normal process in step S11 and the exception process in step S12 are repeatedly executed sequentially for each frame.

  In the present embodiment, as shown in FIGS. 6 to 8, the tracking point is temporarily changed by the normal process and the exception process, such as the tracking target rotating, the occlusion occurring, the scene changing, etc. Tracking is possible even if it becomes invisible.

  In the example of FIG. 6, a person's face 74 as a tracking target is displayed in the frame n−1, and this person's face 74 has a right eye 72 and a left eye 73. It is assumed that the user designates, for example, the right eye 72 (exactly one pixel therein) as the tracking point 71 among them. In the example of FIG. 6, in the next frame n, the person to be tracked moves in the left direction in the figure, and in the next frame n + 1, the person's face 74 rotates in the clockwise direction. . As a result, the right eye 72 that has been visible until now is not displayed, and tracking cannot be performed. Therefore, in the above-described normal processing in step S <b> 11 of FIG. 5, the left eye 73 on the face 74 as the same object as the right eye 72 is selected as a new tracking point, and the tracking point is switched to the left eye 73. This enables tracking.

  In the example of FIG. 7, the ball 81 has moved from the left side of the figure of the human face 74 as the tracking target in the frame n−1, and the ball 81 just covers the face 74 in the next frame n. ing. In this state, the face 74 including the right eye 72 designated as the tracking point 71 is not displayed. When such an occlusion occurs, the face 74 as the object is not displayed, so there is no transfer point to be tracked instead of the tracking point 71, and it becomes difficult to track the tracking point thereafter. Therefore, in the present embodiment, an image of frame n−1 (actually a previous frame in time) including the right eye 72 as the tracking point 71 is stored in advance as a template, and the ball 81 moves further to the right. When the right eye 72 designated as the tracking point 71 appears again in the frame n + 1, it is confirmed based on the template that the right eye 72 as the tracking point 71 is displayed again by the exception processing in step S12 of FIG. Thus, the right eye 72 is tracked again as the tracking point 71.

  In the example of FIG. 8, the person's face 74 as a tracking target is displayed in the frame n−1. However, in the next frame n, the automobile 91 covers the entire face including the person's face 74. That is, in this case, a scene change has occurred. Therefore, in the present embodiment, even when the scene change occurs and the tracking point 71 no longer exists in the image, the vehicle 91 moves and the right eye 72 is displayed again in the frame n + 1. 5, it is confirmed based on the template that the right eye 72 as the tracking point 71 has reappeared, and the right eye 72 can be tracked again as the tracking point 71.

  Next, the details of the normal processing in step S11 in FIG. 5 will be described with reference to the flowchart in FIG.

  In step S21, the tracking point determination unit 57 executes an initialization process of a normal process. The details will be described later with reference to the flowchart of FIG. 10, but an area estimation range based on the tracking point designated to be tracked by the user is set by this processing. This area estimation range is the same object as the tracking point specified by the user (for example, when the tracking point is a human eye, a human face as a rigid body that moves like the eye, a human body, or the like) ) Is a range to be referred to when estimating the range of points belonging to. A transfer point is selected from points in the region estimation range.

  In step S22, the control unit 59 controls each unit so as to wait for input of an image of the next frame. In step S23, the motion estimation unit 52 estimates the tracking point motion. That is, by capturing a frame (hereinafter referred to as a subsequent frame) that is temporally later than a frame including a tracking point designated by the user (here, referred to as a previous frame) in the process of step S22, two consecutive frames are eventually obtained. Therefore, in step S23, the movement of the tracking point is estimated by estimating the position of the tracking point of the subsequent frame corresponding to the tracking point of the previous frame.

  Note that “before or after in time” means the order of input or the order of processing. Usually, since the images of each frame are input in the order of imaging, in that case, the frame captured earlier in time becomes the previous frame, but the frame captured later in time is processed first. In this case, a frame that is imaged later in time becomes the previous frame.

  In step S24, the motion estimation unit 52 determines whether the tracking point can be estimated as a result of the process in step S23. Whether or not the tracking point can be estimated is determined, for example, by comparing the accuracy value of a motion vector (described later) generated and output by the motion estimation unit 52 with a preset threshold value. Specifically, it is possible to estimate if the accuracy of the motion vector is equal to or greater than a threshold, and it is determined that estimation is impossible if the accuracy is smaller than the threshold. In other words, the possibility here is determined relatively strictly, and if the estimation is not actually impossible but the accuracy is low, it is determined as impossible. As a result, more reliable tracking processing can be performed.

  In step S24, it is determined that the motion estimation result at the tracking point and the motion estimation result at a point in the vicinity of the tracking point can be estimated if the motion occupies a large number of motions. It is also possible to do so.

  If it is determined in step S24 that the movement of the tracking point can be estimated, that is, if the probability that the tracking point is correctly set on the corresponding point on the same object is relatively high, the process proceeds to step S25. . More specifically, when the right eye 72 is designated as the tracking point 71 in the example of FIG. 6, it is determined that the movement of the tracking point can be estimated when the probability that the right eye 72 is correctly tracked is relatively high. Is done.

  In step S25, the tracking point determination unit 57 shifts the tracking point by the estimated motion (so-called motion vector) obtained in the process of step S23. As a result, the tracking position in the subsequent frame after the tracking of the tracking point in the previous frame is determined.

  After step S25, region estimation related processing is executed in step S26. Details of this region estimation related processing are disclosed in Japanese Patent Laid-Open No. 2005-303983 previously proposed by the present applicant. With this process, the area estimation range designated in the initialization process of the normal process in step S21 is updated. In addition, when the tracking point is not displayed due to rotation of the target object, candidates for transfer points (transfer candidates) as points to be transferred are previously extracted in a state where tracking is still possible. And is held by the transfer candidate holding unit 56. In addition, when the transfer to the transfer candidate becomes impossible, the tracking is temporarily interrupted, but a template for confirming that the tracking can be performed again when the tracking point appears again is created in advance, and the template holding unit 58.

  After the region estimation related processing in step S26 is completed, the processing returns to step S22 again, and the subsequent processing is repeatedly executed.

  As described above, as long as the movement of the tracking point designated by the user can be estimated, the processing from step S22 to step S26 is repeatedly executed for each frame, and tracking is performed.

  On the other hand, when it is determined in step S24 that the movement of the tracking point cannot be estimated, that is, as described above, for example, when the accuracy of the motion vector is equal to or less than the threshold value, the processing is performed. Proceed to step S27.

  In step S27, the tracking point determination unit 57 holds the transfer candidate generated in the region estimation-related processing in step S26 in the transfer candidate holding unit 56, and therefore, the transfer candidate closest to the original tracking point is among them. Select one. The tracking point determination unit 57 determines whether or not a transfer candidate has been selected in step S28. If a transfer candidate has been selected, the tracking point determination unit 57 proceeds to step S29 and sets the tracking point as the transfer candidate selected in step S27. Change (change). Thereby, the transfer candidate point is set as a new tracking point.

  After the process of step S29, the process returns to step S23, and the process of estimating the movement of the tracking point selected from the transfer candidates is executed. Then, it is determined again whether or not the motion of the newly set tracking point can be estimated in step S24. If it can be estimated, a process of shifting the tracking point by the estimated motion is performed in step S25. In step S26, region estimation related processing is executed. Thereafter, the process returns to step S22 again, and the subsequent processes are repeatedly executed.

  If it is determined in step S24 that the newly set tracking point cannot be estimated, the process proceeds again to step S27, and the next transfer candidate closest to the original tracking point is selected from the transfer candidates. In step S29, the transfer candidate is set as a new tracking point. The process after step S23 is repeated again for the new tracking point.

  Even if all the transfer candidates held in the transfer candidate holding unit 56 are used as new tracking points, if it is not possible to estimate the movement of the tracking point, it is determined in step S28 that no transfer candidate has been selected. This normal processing is terminated. Then, the process proceeds to the exception process in step S12 of FIG.

  Next, details of the initialization process of the normal process in step S21 of FIG. 9 will be described with reference to the flowchart of FIG.

  In step S41, the control unit 59 determines whether or not the current process is a process for returning from the exception process. That is, after the exception process in step S12 in FIG. 5 is terminated, it is determined whether or not the process has returned to the normal process in step S11. In the process of the first frame, since the exception process in step S12 has not been executed yet, it is determined that the process has not returned from the exception process, and the process proceeds to step S42.

  In step S42, the tracking point determination unit 57 executes processing for setting the tracking point to the position of the tracking point instruction. That is, for example, the point with the highest luminance in the input image is set as the tracking point.

  The tracking point may be set by other methods such as designation by the user. The designation by the user is, for example, performed by designating a predetermined point in the input image as a tracking point to the control unit 59 by operating the instruction input unit 29 by the user. The tracking point determination unit 57 supplies the set tracking point information to the region estimation related processing unit 55.

  In step S43, the region estimation related processing unit 55 sets a region estimation range based on the tracking point position information set in step S42. This region estimation range is a reference range when estimating a point on the same rigid body as the tracking point, and more specifically, tracking so that the same rigid body portion as the tracking point occupies most of the region estimation range in advance. By setting the position and size of the estimated area range to follow the same rigid body part as the point, the part that shows the most movement in the estimated area range can be estimated as the same rigid body part as the tracking point It is for doing so. In step S43, as a default value, for example, a predetermined fixed range centered on the tracking point is set as the region estimation range. Thereafter, the process proceeds to step S22 in FIG.

  On the other hand, if it is determined in step S41 that the current process is a return process from the exception process in step S12 of FIG. 5, the process proceeds to step S44. In step S44, the tracking point determination unit 57 sets a tracking point and a region estimation range based on a position that matches a template by exception processing described later with reference to the flowchart of FIG. For example, a point on the current frame that matches the tracking point on the template is set as the tracking point, and a certain range set in advance from the point is set as the region estimation range. Thereafter, the processing proceeds to step S22 in FIG.

  The above process will be described with reference to FIG.

In step S42 of FIG. 10, for example, as shown in FIG. 11, when the right eye 72 of the person in the frame n-1 is designated as the tracking point 71, a predetermined region including the tracking point 71 is an area in step S43. Designated as the estimated range 101. In step S24 of FIG. 9, it is determined whether or not the sample points within the region estimation range 101 can be estimated in the next frame. In the case of the example in FIG. 11, in the frame n + 1 next to the frame n, the left half region 102 in the drawing including the right eye 72 in the region estimation range 101 is hidden by the ball 81. 71 motion cannot be estimated in the next frame n + 1. Therefore, in such a case, one of the points in the area designation range 101 (in the face 74 as a rigid body including the right eye 72) prepared in advance as a transfer candidate in the previous frame n-1 in time is selected. One point (for example, the left eye 73 included in the face 74 (exactly one pixel therein)) is selected, and that point is set as a new tracking point in the frame n + 1.

  Next, with reference to the flowchart of FIG. 12, the details of the exception process of step S12 performed following the normal process of step S11 of FIG. 5 will be described. As described above, this process is performed when it is determined in step S24 in FIG. 9 that it is impossible to estimate the movement of the tracking point, and in step S28, it is determined that the transfer candidate for changing the tracking point cannot be selected. Will be executed.

  In step S51, the control unit 59 executes an exception process initialization process. Here, the details of the exception process initialization process will be described with reference to the flowchart of FIG.

  In step S61, the control unit 59 determines whether or not a scene change has occurred when the movement of the tracking point cannot be estimated and the transfer candidate for changing the tracking point cannot be selected. The scene change detection unit 53 constantly monitors whether or not a scene change has occurred based on the estimation result of the motion estimation unit 52, and the control unit 59 performs step S61 based on the detection result of the scene change detection unit 53. The determination process is executed. Specific processing of the scene change detection unit 53 will be described later with reference to FIGS.

  If a scene change has occurred, the control unit 59 estimates that the reason why tracking has become impossible is that a scene change has occurred, and sets the mode to scene change in step S62. On the other hand, when it is determined in step S61 that no scene change has occurred, the control unit 59 sets the mode to another mode in step S63.

  After the process of step S62 or step S63, the template matching unit 51 executes a process of selecting the oldest template in time in step S64. Specifically, as shown in FIG. 14, for example, when transition from frame n to frame n + 1 is performed, if exception processing is to be executed, a frame n−m + 1 is generated for frame n and is stored in the template holding unit 58. A template generated with respect to the frame mn + 1, which is the oldest template in time, is selected from the held m-frame templates.

  In this way, the template immediately before the transition to exception processing (the template generated with respect to frame n in the case of the example in FIG. 14) is not used, and the template slightly before in time is selected for the following reason. It is. In other words, if a transition to exception processing occurs due to the tracking target occlusion, the tracking target is already quite hidden immediately before the transition, and the tracking target cannot be captured sufficiently large in the template at that time. This is because the possibility is high. Therefore, reliable tracking is possible by selecting a template in a frame slightly before in time.

  Returning to the description of FIG. In step S65, the template matching unit 51 executes a process for setting a template search range. For example, the template search range is set such that the position of the tracking point immediately before the transition to the exception process is the center of the template search range.

  That is, as shown in FIG. 15, when the right eye 72 of the subject's face 74 is designated as the tracking point 71 in the frame n, the ball 81 flies from the left direction in the figure, and the tracking point 71 in the frame n + 1. Is assumed, and the tracking point 71 appears again in the frame n + 2. In this case, an area centered on the tracking point 71 included in the template range 111 is set as the template search range 112.

  In step S66, the template matching unit 51 resets the number of elapsed frames and the number of scene changes after shifting to exception processing to zero. The number of frames and the number of scene changes are used in a continuation determination process (steps S71, S73, S75, and S77 in FIG. 16) in step S55 in FIG.

  After the exception process initialization process is completed as described above, in step S52 of FIG. 12, the control unit 59 executes a process of waiting for an input of an image of the next frame. In step S53, the template matching unit 51 performs a template matching process within the template search range.

  In step S54, the template matching unit 51 determines whether it is possible to return to normal processing. That is, by the template matching process, an absolute value sum of differences between a template several frames before (a pixel in the template range 111 in FIG. 15) and a matching target pixel in the template search range is calculated.

  More specifically, the absolute value sum of the difference between each pixel in the predetermined block in the template range 111 and the predetermined block in the template search range is calculated. The position of the block is sequentially moved within the template range 111, and the sum of absolute values of the differences of the respective blocks is added to obtain a value at the position of the template. Then, the position where the sum of absolute values of the differences becomes the smallest when the template is sequentially moved within the template search range and its value are searched. In step S54, the absolute sum of the searched minimum differences is compared with a predetermined threshold value set in advance. If the sum of the absolute values of the differences is equal to or less than the threshold value, the image including the tracking point included in the template has appeared again. Therefore, it is determined that the normal process can be restored, and the process is illustrated in FIG. The process returns to the normal process of step S11 in step 5.

  Then, as described above, in step S41 of FIG. 10, it is determined that the process returns from the exception process. In step S44, the position where the sum of absolute differences is minimum is set as the matched position of the template. The tracking point and the area estimation range are set based on the positional relationship between the template position held corresponding to the template and the tracking point area estimation range.

  In step S54 of FIG. 12, it is determined whether or not it is possible to return to normal processing by comparing the value obtained by dividing the minimum sum of absolute differences by the activity of the template with a threshold value. Also good. As the activity in this case, the value calculated in step S93 in FIG. 18 by the activity calculation unit 122 in FIG. 17 described later can be used.

  Alternatively, it is possible to return to the normal processing by comparing a value obtained by dividing the current minimum absolute difference sum by the minimum absolute difference sum one frame before with a predetermined threshold. It may be determined whether or not. In this case, it is not necessary to calculate the activity. That is, in step S54, the correlation between the template and the template search range is calculated, and determination is performed based on the comparison between the correlation value and the threshold value.

  If it is determined in step S54 that it is not possible to return to the normal process, the process proceeds to step S55, and the continuation determination process is executed. The details of the continuation determination process will be described later with reference to the flowchart of FIG. 16, and this process determines whether or not the exception process can be continued.

  In step S56, the control unit 59 determines whether or not tracking of the tracking point in the exception process by the continuation determination process in step S55 can be continued (in steps S76 and S78 of FIG. 16 described later). Determine based on the flag set. If it is determined that the tracking process of the tracking point in the exception process can be continued, the process returns to step S52, and the subsequent processes are repeatedly executed. That is, the process of waiting until the tracking point appears again is repeatedly executed.

  On the other hand, when it is determined in step S56 that the tracking process of the tracking point in the exception process cannot be continued (the number of elapsed frames after the tracking point disappears in the process of step S75 in FIG. 16 described later). If it is determined that the threshold value THfr or more is determined or the number of scene changes is determined to be greater than or equal to the threshold value THsc in the process of step S77), it is determined that exception processing is no longer possible, and the tracking process is terminated. Instead of ending the tracking process, it may be possible to return to the normal process again using the tracking point that has been held. The processing in this case will be described later with reference to the flowchart of FIG.

  Next, details of the continuation determination process in step S55 of FIG. 12 will be described with reference to the flowchart of FIG.

  In step S71, the control unit 59 executes a process of adding 1 to the number of elapsed frames as a variable. The number of elapsed frames is previously reset to 0 in the exception process initialization process (step S66 in FIG. 13) in step S51 in FIG.

  Next, in step S72, the control unit 59 determines whether or not there is a scene change. Whether or not there is a scene change can be determined based on the detection result of the scene change detection unit 53 always executing the detection process. If it is determined that there is a scene change, the process proceeds to step S73, and the control unit 59 adds 1 to the number of scene changes as a variable. The number of scene changes is also reset to 0 in the initialization process of step S66 in FIG. If it is determined that no scene change has occurred during the transition from the normal process to the exception process, the process of step S73 is skipped.

  Next, in step S74, the control unit 59 determines whether or not the currently set mode is a scene change. This mode is set in step S62 or S63 of FIG. When it is determined that the currently set mode is a scene change, the process proceeds to step S77, and the control unit 59 determines whether or not the number of scene changes is smaller than a preset threshold value THsc. When the control unit 59 determines in step S77 that the number of scene changes is smaller than the threshold value THsc, the control unit 59 proceeds to step S76, sets a continuation flag, and determines that the number of scene changes is equal to or greater than the threshold value THsc. The process proceeds to step S78, and a flag that cannot be continued is set.

  On the other hand, when it is determined in step S74 that the mode is not a scene change (when it is determined that the mode is other), the process proceeds to step S75, and the control unit 59 determines whether or not the number of elapsed frames is smaller than the threshold value THfr. Determine whether. This elapsed frame number is also reset to 0 in advance in step S66 of the exception process initialization process of FIG. When it is determined in step S75 that the number of elapsed frames is smaller than the threshold value THfr, the control unit 59 proceeds to step S76, sets a continuation flag, and is determined that the number of elapsed frames is equal to or greater than the threshold value THfr. In this case, the process proceeds to step S78, and a flag indicating that continuation is not possible is set.

  As described above, when the number of scene changes at the time of template matching processing is equal to or greater than the threshold value THsc, or when the number of elapsed frames is equal to or greater than the threshold value THfr, further exception processing is impossible.

  When the mode is other, a condition that the number of scene changes is 0 may be added to determine whether or not continuation is possible.

  In the above, it is assumed that the frame of the image is used as a processing unit and all frames are used. However, processing is not performed in units of fields or using all frames or fields, but is extracted by thinning out at a predetermined interval. It is also possible to use a modified frame or field.

  Next, a configuration example of the motion estimation unit 52 in FIG. 4 will be described with reference to FIG. As shown in FIG. 17, an input image is supplied to the evaluation value calculation unit 121, the activity calculation unit 122, and the motion vector detection unit 123.

  The evaluation value calculation unit 121 calculates an evaluation value related to the degree of coincidence between both objects associated by the motion vector, and supplies the evaluation value to the normalization processing unit 125. The activity calculation unit 122 calculates the activity of the input image and supplies it to the threshold value determination unit 124 and the normalization processing unit 125. The motion vector detection unit 123 detects a motion vector from the input image and supplies it to the evaluation value calculation unit 121 and the integration processing unit 126.

  The threshold determination unit 124 compares the activity supplied from the activity calculation unit 122 with a predetermined threshold and supplies the determination result to the integration processing unit 126. The normalization processing unit 125 normalizes the evaluation value supplied from the evaluation value calculation unit 121 based on the activity supplied from the activity calculation unit 122, and supplies the obtained value to the integration processing unit 126.

  The integration processing unit 126 calculates the accuracy of the motion vector based on the normalization information supplied from the normalization processing unit 125 and the determination result supplied from the threshold determination unit 124, and uses the obtained accuracy to detect the motion vector. The motion vector supplied from the unit 123 is output.

  Next, a motion estimation process executed by the motion estimation unit 52 will be described with reference to the flowchart of FIG. Although the motion vector is obtained for a point, the accuracy is calculated using image data of a small block in the vicinity of two points associated by the motion vector, for example, centering on the point.

  In step S91, the motion vector detection unit 123 detects a motion vector from the input image. For this detection, for example, a block matching method or a gradient method is used. The detected motion vector is supplied to the evaluation value calculation unit 121 and the integration processing unit 126.

  In step S92, the evaluation value calculation unit 121 calculates an evaluation value related to the degree of coincidence of both objects associated with the motion vector detected in the process of step S91. Specifically, for example, the sum of absolute differences of pixel values of two blocks centered on two points associated with motion vectors is calculated. That is, the motion vector V (vx, vy) detected by the motion vector detection unit 123 in the process of step S91, the point P (Xp, Yp) on the image Fi of the previous frame in time based on the motion vector V (vx, vy) The relationship between the point Q (Xq, Yq) on the image Fj of the subsequent frame is expressed by the following equation (1).

  The evaluation value calculation unit 121 calculates an evaluation value Eval (P, Q, i, j) based on the following equation (2) for a block centered on the point P and a block centered on the point Q.

  Each block is a square having 2L + 1 pixels on one side. The sum ΣΣ in the above equation (2) is performed between corresponding pixels when x is from −L to L and y is from −L to L. Therefore, for example, when L = 2, nine differences are obtained, and the sum of the absolute values is calculated. The evaluation value indicates that the two blocks match well as the value approaches zero.

  The evaluation value calculation unit 121 supplies the calculated evaluation value to the normalization processing unit 125.

  In step S93, the activity calculation unit 122 calculates an activity from the input image. The activity is a feature amount representing the complexity of the image, and as shown in FIG. 19, the difference between the target pixel Y (x, y) and the adjacent 8 pixels Y (x + i, y + j) for each pixel. The average value of the sum of absolute values is calculated based on the following equation (3) as activity Activity (x, y) at the target pixel position.

  In the case of the example in FIG. 19, the value of the pixel of interest Y (x, y) located in the center among 3 × 3 pixels is 110, and the values of eight pixels adjacent to it are 80, 70, and 75, respectively. , 100, 100, 100, 80, 80, the activity Activity (x, y) is expressed by the following equation.

  Activity (x, y) = {| 80-110 | + | 70-110 | + | 75-110 | + | 100-110 | + | 100-110 | + | 80-110 | + | 80−110 |} /8=24.375.

  Similar processing is performed for all pixels in the frame.

  In order to calculate the motion vector accuracy in units of blocks, the sum of the activities of all the pixels in the block expressed by the following equation (4) is defined as the activity (block activity) Blockactivity (i, j) of the block.

  In addition, the activity may be a variance value, a dynamic range, or the like.

  In step S94, the threshold determination unit 124 compares the block activity calculated in the process of step S93 with a predetermined threshold set in advance. Then, a flag indicating whether or not the input block activity is larger than the threshold value is output to the integration processing unit 126.

  Specifically, as a result of the experiment, the block activity and the evaluation value have the relationship shown in FIG. 20 with the motion vector as a parameter. In FIG. 20, the horizontal axis represents the block activity Blockactivity (i, j), and the vertical axis represents the evaluation value Eval.

  When the motion is correctly detected (when the correct motion vector is given), the block activity and the value of the evaluation value are distributed in the region R1 below the curve 131 in the figure. On the other hand, when an incorrect motion (incorrect motion vector) is given, the block activity and the value of the evaluation value are distributed in a region R2 on the left side in the figure from the curve 132. Note that there is almost no distribution in the region other than the region R2 above the curve 132 and the region other than the region R1 below the curve 131. The curve 131 and the curve 132 intersect at the point P, and the value of the block activity at the point P is set as the threshold value THa. The threshold value THa means that if the value of the block activity is smaller than that, the corresponding motion vector may be incorrect (this will be described in detail later). The threshold value determination unit 124 outputs a flag indicating whether or not the value of the block activity input from the activity calculation unit 122 is greater than the threshold value THa to the integration processing unit 126.

  In step S95, the normalization processing unit 125 normalizes the evaluation value calculated in the process of step S92 based on the activity calculated in the process of step S93. Specifically, the normalization processing unit 125 calculates the motion vector accuracy VC according to the following equation (5).

  However, if the value of the motion vector accuracy VC is less than 0, the value is replaced with 0. Of the motion vector accuracy VC, the value obtained by dividing the evaluation value by the block activity is a straight line 133 whose position on the graph of FIG. Therefore, it indicates whether the region is in the lower region or the upper region in the figure. That is, if the slope of the straight line 133 is 1, and the value obtained by dividing the evaluation value by the block activity is larger than 1, the points corresponding to the value are points distributed in the area above the straight line 133. Means that. The motion vector accuracy VC obtained by subtracting this value from 1 means that the smaller the value, the higher the possibility that the corresponding points are distributed in the region R2.

  On the other hand, if the value obtained by dividing the evaluation value by the block activity is smaller than 1, it means that the points corresponding to the value are distributed in the lower region of the straight line 133 in the figure. The motion vector accuracy VC at that time means that the larger the value (closer to 0), the corresponding points are distributed in the region R1. The normalization processing unit 125 outputs the motion vector accuracy VC obtained by the calculation in this way to the integration processing unit 126.

  In step S96, the integration processing unit 126 executes integration processing. Details of this integration processing are shown in the flowchart of FIG.

  In step S101, the integration processing unit 126 determines whether or not the block activity is equal to or less than a threshold value THa. This determination is performed based on the flag supplied from the threshold determination unit 124. When determining that the block activity is equal to or less than the threshold value THa, the integration processing unit 126 sets the value of the motion vector accuracy VC calculated by the normalization processing unit 125 to 0 in step S102. If it is determined in step S101 that the activity value is greater than the threshold value THa, the processing in step S102 is skipped, and the value of the motion vector accuracy VC generated by the normalization processing unit 125 is output as it is together with the motion vector. Is done.

  This is because there is a possibility that a correct motion vector is not obtained if the block activity value is smaller than the threshold value THa even if the value of the motion vector accuracy VC calculated by the normalization processing unit 125 is positive. Because there is. That is, as shown in FIG. 20, between the origin O and the point P, the curve 132 protrudes downward from the curve 131 in the drawing (below the straight line 133). In a region R3 in which the value of the block activity is smaller than the threshold Tha and is surrounded by the curves 131 and 132, the value obtained by dividing the evaluation value by the block activity is distributed in both the regions R1 and R2. There is a high possibility that a correct motion vector is not obtained.

  Therefore, in such a distribution state, processing is performed assuming that the accuracy of the motion vector is low. For this reason, in step S102, even if the motion vector accuracy VC is positive, it is set to 0 if it is smaller than the threshold value Tha. In this way, when the value of the motion vector accuracy VC is positive, it is possible to reliably represent that the correct motion vector is obtained. In addition, the larger the value of the motion vector accuracy VC, the higher the probability that a correct motion vector is obtained (the probability that the distribution is included in the region R1 increases).

  This coincides with an empirical rule that, in general, it is difficult to detect a motion vector with high reliability in a region where the luminance change is small (region where the activity is small).

  Next, a configuration example of the background motion estimation unit 54 in FIG. 4 will be described with reference to FIG. As illustrated in FIG. 22, the background motion estimation unit 54 includes a frequency distribution calculation unit 141 and a background motion determination unit 142.

  The frequency distribution calculation unit 141 calculates a frequency distribution of motion vectors. However, this frequency is weighted so that a probable motion is weighted by using the motion vector accuracy VC supplied from the motion estimation unit 52 of FIG. Based on the frequency distribution calculated by the frequency distribution calculation unit 141, the background motion determination unit 142 performs a process of determining a motion with the maximum frequency as a background motion, and outputs the background motion to the region estimation related processing unit 55 in FIG. .

  The background motion estimation process executed by the background motion estimation unit 54 will be described with reference to the flowchart of FIG.

  In step S111, the frequency distribution calculation unit 141 calculates a motion frequency distribution. Specifically, the frequency distribution calculation unit 141 assumes that the x and y coordinates of a motion vector as a background motion candidate are expressed in a range of ± 16 pixels from the reference point, respectively (1089 (= 16 × 2 + 1) × (16 × 2 + 1)), that is, a box for coordinates corresponding to possible values of a motion vector, and when a motion vector is generated, 1 is added to the coordinates corresponding to the motion vector. In this way, the motion vector frequency distribution can be calculated.

  However, when one motion vector is generated, if 1 is added, if the frequency of occurrence of a motion vector with low accuracy is high, a motion vector with low certainty may be determined as the background motion. . Therefore, when a motion vector occurs, the frequency distribution calculation unit 141 does not add the value 1 to the box (coordinates) corresponding to the motion vector, but instead of adding the value 1 to the motion vector accuracy VC (= Motion vector accuracy VC). The value of the motion vector accuracy VC is normalized as a value between 0 and 1, and the closer the value is to 1, the higher the accuracy. Therefore, the frequency distribution obtained in this way is a frequency distribution obtained by weighting motion vectors based on their accuracy. This reduces the risk that a motion with low accuracy is determined as the background motion.

  Next, in step S112, the frequency distribution calculation unit 141 determines whether or not the process of calculating the motion frequency distribution has been completed for all blocks. When it is determined that there is an unprocessed block, the process returns to step S111, and the motion frequency distribution is calculated for the next block.

  As described above, the motion frequency distribution calculation processing is performed on the entire screen, and when it is determined in step S112 that the processing of all blocks has been completed, the process proceeds to step S113, where the background motion determination unit 142 A process for searching for the maximum value of the distribution is executed. That is, the background motion determination unit 142 selects a frequency having the maximum frequency from the frequencies calculated by the frequency distribution calculation unit 141, and determines a motion vector corresponding to the frequency as a motion vector of the background motion. The motion vector of the background motion is supplied to the region estimation related processing unit 55 in FIG.

  Next, a configuration example of the scene change detection unit 53 in FIG. 4 will be described with reference to FIG. As illustrated in FIG. 24, the scene change detection unit 53 includes a motion vector accuracy average calculation unit 151 and a threshold determination unit 152.

  The motion vector accuracy average calculation unit 151 calculates the average value of the entire motion vector accuracy VC supplied from the motion estimation unit 52 in FIG. 4 and outputs the average value to the threshold determination unit 152. The threshold determination unit 152 compares the average value supplied from the motion vector accuracy average calculation unit 151 with a predetermined threshold, and determines whether or not the scene change is based on the comparison result. The result is output to the control unit 59 in FIG.

  The scene change detection process executed by the scene change detection unit 53 will be described with reference to the flowchart of FIG.

  In step S121, the motion vector accuracy average calculation unit 151 calculates the sum of motion vector accuracy. Specifically, the motion vector accuracy average calculation unit 151 executes a process of adding the value of the motion vector accuracy VC calculated for each block output from the integration processing unit 126 of the motion estimation unit 52.

  In step S122, the motion vector accuracy average calculation unit 151 determines whether or not the processing for calculating the sum of motion vector accuracy VC has been completed for all the blocks. If the processing has not yet been completed, the processing of step S121 is performed. repeat. By repeating this process, the sum of motion vector accuracy VC of each block for one screen is calculated.

  If it is determined in step S122 that the calculation of the sum of the motion vector accuracy VC for one entire screen has been completed, the process proceeds to step S123, and the motion vector accuracy average calculation unit 151 calculates the average value of the motion vector accuracy VC. Execute the process. Specifically, a value obtained by dividing the sum of motion vector accuracy VC for one screen calculated in the process of step S121 by the number of added blocks is calculated as an average value.

  In step S124, the threshold determination unit 152 compares the average value of the motion vector accuracy VC calculated by the motion vector accuracy average calculation unit 151 in the process of step S123 with a preset threshold value, and determines whether or not the threshold value is smaller than the threshold value. Determine whether. In general, when a scene change occurs between two frames having different times in a moving image, there is no corresponding image, so even if a motion vector is calculated, the motion vector is not certain. Therefore, the threshold determination unit 152 proceeds to step S125 when determining that the average value of the motion vector accuracy VC is smaller than the threshold, and turns on the scene change flag, and when determining that the average value of the motion vector accuracy VC is equal to or greater than the threshold, proceeds to step S126. Go ahead and turn off the scene change flag. When the scene change flag is turned on, it indicates that there is a scene change, and when the scene change flag is turned off, it indicates that there is no scene change.

  This scene change flag is supplied to the control unit 59, and is used for determining whether or not there is a scene change in step S61 in FIG. 13 and for determining whether or not there is a scene change in step S72 in FIG.

  As described above, by configuring the object tracking unit 23 in FIG. 1, the object 41 (FIG. 3) to be tracked rotates, occlusion occurs, or the tracking point 41A of the object 41 is changed due to a scene change. Even when the image is temporarily not displayed, it is possible to accurately track the tracking point 41A of the moving object 41 in the image.

  Further, the position information of the tracking point 41A of the object 41 tracked in this way is output to the tracking position correction unit 24 as a tracking result by the object tracking unit 23 of FIG. That is, when the user does not obtain a desired tracking result even though the tracking processing is performed by the object tracking unit 23, the user instructs the correction by instructing the correction of the tracking position via the instruction input unit 29. The received tracking position correction unit 24 can correct the position information as the tracking result supplied from the object tracking unit 23.

  Next, a detailed configuration example and operation of the tracking position correction unit 24 in FIG. 1 will be described.

  FIG. 26 is a block diagram illustrating a functional configuration example of the tracking position correction unit 24. As shown in FIG. 26, the tracking position correction unit 24 includes a correction value calculation unit 161, a correction value reflection unit 162, and a fine adjustment unit 163.

  When the correction value of the tracking position is instructed by the user via the instruction input unit 29, the correction value calculation unit 161 calculates the correction value. For example, the instruction input unit 29 includes cross-direction buttons that can indicate up, down, right, and left directions, and a predetermined correction amount is set in advance in response to one press of the buttons. If so, the user's correction value Δu is calculated according to the number of times the button is pressed. Further, for example, when the instruction input unit 29 is configured with a lever capable of instructing a predetermined direction, and a predetermined correction amount is set in advance according to the inclination angle of the lever, according to the degree of tilting of the lever A user correction value Δu is calculated.

  The correction value reflecting unit 162 reflects the user correction value Δu supplied from the correction value calculating unit 161 in the position information as the tracking result supplied from the object tracking unit 23.

  Here, an example in which the correction value Δu of the user is reflected on the tracking point as the tracking target calculated by the block matching method or the like will be described.

  As shown in FIG. 27, according to the block matching method, a difference vector Δx between the current frame t = t and the next frame t = t + 1 is calculated. When normal tracking processing is performed by the object tracking unit 23 as described above, the tracking point x (t + 1) of the next frame t = t + 1 is calculated based on the following equation (6). x (t) is a tracking point of the current frame t = t.

  For example, in the case of the first method in which only the user correction value Δu is reflected without considering the calculated difference vector Δx, the tracking point x (t + 1) of the next frame t = t + 1 is based on the following equation (7). ) Is calculated.

  FIG. 28 shows an example in which the correction value of the user is reflected by the first method. As shown in FIG. 28, after the correction performed by the user between the position x (t) and the position x (t + 2) is completed, the corrected tracking point is set to the position x (t + 3). As described above, the correction value Δu by the user is reflected on the tracking point as a relative position with respect to the coordinate system of the image.

  In addition, for example, the tracking point using the calculated difference vector Δx and the tracking point based on the user correction value Δu are separately held and reflected in the block matching process when the correction by the user is completed. In this case, the tracking point u (t) at the start of correction is calculated based on the following equation (8).

  Further, the tracking point being corrected is calculated based on the following equation (9).

  Further, a tracking point x (t + 1) at the end of correction is calculated based on the following equation (10).

  In this case, the tracking point output during correction is x ′, but the point is to use x in the block matching process inside the object tracking unit 23.

  FIG. 29 shows an example in which the correction value of the user is reflected by the second method. As shown in FIG. 29, after the correction performed by the user between the position x (t) and the position x (t + 2) is completed, the corrected tracking point is the position of x (t + 3) + u (t + 3). This position is again x (t + 3). Thus, the correction value Δu by the user is reflected as a relative position with respect to the tracking target.

  By providing a correction amount change button in the instruction input unit 29, for example, the user greatly changes the preset correction amount by operating a cross direction button or lever while pressing the correction amount change button. be able to. As a result, first press the correction amount change button and operate the cross direction button or lever to roughly adjust the tracking position, then release the correction amount change button and operate the cross direction button or lever to track details. By correcting the position, it is possible to correct the desired tracking point in a short time. This is particularly effective when the image size is large.

  Returning to the description of FIG. The fine adjustment unit 163 further finely adjusts the tracking point on which the correction value Δu of the user is reflected by the correction value reflection unit 162 by the first method or the second method.

  Here, an example will be described in which the tracking point is finely adjusted to the position of the center of gravity of the object as the tracking target.

  For example, as shown in FIG. 30, when correction by the user is performed from the position x (t) to the position x ′ (t + 1) using the first method described above, the corrected position x ′ (t + 1) Is calculated, and the tracking point is finely adjusted to the center of gravity x (t + 1) of the object. Thereafter, the tracking process is continued from the position of the fine-tuned tracking point x (t + 1).

  Here, a method for obtaining the range of the object to which the tracking point belongs based on the pixel value (color information) called color and calculating the center of gravity will be described.

  (1) First, the RGB color (r ′, g ′, b ′) at the position x ′ (t + 1) is calculated. The color system may not be RGB.

  (2) For the peripheral pixel at the position x ′ (t + 1) where the RGB color is calculated, it is determined that a pixel having a color close to the color at the position x ′ (t + 1) belongs to the same object. ) Is smaller than a predetermined value, it is determined that the pixel belongs to the same object as the position x ′ (t + 1), where r, g, and b are the colors of the surrounding pixels.

  (3) The center of gravity x (t + 1) of the object is calculated from the position of the pixel belonging to the same object as the position x ′ (t + 1) based on the following equation (12). Xi is the position of each pixel belonging to the same object, and n is the total number of pixels belonging to the same object.

  As described above, it is possible to fine-tune the tracking point to the center of gravity of the object to which the tracking point belongs by obtaining the range of the object to which the tracking point belongs by the method using the color information and calculating the center of gravity position thereof. It is.

  Note that the present invention is not limited to the method using color information, and for example, a range of an object may be obtained based on a motion vector, a luminance value, or the like, and the center of gravity position may be calculated. For example, when a motion vector is used, it is determined that a vector having the same or similar vector as the motion vector at the position x ′ (t + 1) belongs to the same object, and the range of the object is obtained as follows. The center of gravity can be calculated.

(1) Motion Vector Detection As shown in FIG. 31, a motion vector is detected at each sampling interval (sx, sy) within a region 212 centered on the tracking point 211. The size of the region 212 is sx · m × sy · n as the number of samples m and n. For the detection of the motion vector, for example, a block matching method or a gradient method is used.

(2) Motion vector frequency distribution calculation For example, motion candidates within the processing range are represented by integer values of −16 <= vx <= 16 and −16 <= vy <= 16 (vx is horizontal motion, vy is vertical motion) In this case, 33 (= 16 × 2 + 1) pixels × 33 pixels = 1089 boxes, that is, boxes corresponding to coordinates corresponding to possible values of the motion vector are prepared, and at a certain sample point (vx, vy) = When (2, 2), 1 is added to the box of (2, 2). By performing this processing on all sample points within the processing range, the frequency distribution of the motion vector can be calculated.

(3) Extraction of sample points to be tracked As shown in FIG. 32, sample points showing movement similar to the movement occupying a large number in the area 212 (in the example of FIG. 32, the movement to the left obliquely downward) are tracked. Extract as a point on the object.

(4) Gravity calculation of sample point Flag (i, j) (1 <= i <= m, 1 <= j <= n) indicating whether or not the sample point is a sample point on the tracking target in the region 212 Set as. If it is extracted as a sample point on the tracking target in (3), it is set to 1. If it is not extracted as a sample point on the tracking target in (3), it is set to 0. Then, the centroid G (x, y) of the sample point is calculated as the sample point P (x, y) based on the following equation (13).

  As described above, the range of the object to which the tracking point belongs can be obtained by the method using the motion vector, and the center of gravity position can be calculated. In addition to the correction by the user, the finely adjusted previous tracking point is characterized by high robustness against disturbance, that is, easy tracking processing.

  The tracking correction process as described above will be described with reference to the flowchart of FIG.

  In step S131, the control unit 59 determines whether or not the instruction input unit 29 is used by the user to specify the tracking target position and the tracking start is instructed, and waits until the tracking start is instructed. If it is determined in step S131 that the tracking target position is specified and tracking start is instructed, the process proceeds to step S132.

  In step S132, the control unit 59 controls each unit so as to wait for input of an image of the next frame. In step S133, the tracking point determination unit 57 calculates the target position in the next frame. For example, as shown in FIG. 27, according to the block matching method, the difference vector Δx between the current frame and the next frame is calculated, and the target position of the next frame is calculated based on the above equation (6). A tracking point is calculated.

  In step S134, the control unit 27 determines whether or not position correction is instructed by the user via the instruction input unit 29. If it is determined that position correction is not instructed, the process proceeds to step S135. In step S135, the tracking point determination unit 57 directly uses the target position calculated in step S133 as the target position of the next frame.

  On the other hand, if it is determined in step S134 that the position correction has been instructed by the user, the process proceeds to step S136, and the control unit 59 outputs the tracking result from the tracking point determination unit 57 to the tracking position correction unit 24. The tracking position correction unit 24 sets the input tracking result reflecting the correction value by the user as the target position of the next frame.

  Specifically, the correction calculation unit 161 calculates a correction value according to the amount of operation of the instruction input unit 29 from the user (for example, the number of times the button is pressed or the degree of lever depression). Then, the correction value reflecting unit 162 reflects the correction value of the user in the tracking result by the first method described with reference to FIG. 28 or the second method described with reference to FIG. Further, the fine adjustment unit 163 finely adjusts the tracking point to the position of the center of gravity of the object as the tracking target.

  In step S137, the control unit 27 causes the image display 25 to superimpose and display a mark representing the tracking position on the image captured by the imaging unit 21, based on the tracking result obtained in step S135 or S136.

  The upper diagram of FIG. 34 shows an example of an input image captured by the imaging unit 21, and the lower diagram of FIG. 34 shows an example of an output image when the input image tracking process is performed. The upper side view of FIG. 34 shows an object 221 that is a tracking target imaged by the imaging unit 21. 34, the tracking process of the object 221 is performed, and the mark 221A is superimposed on the tracking point of the object 221.

  Returning to the description of FIG. In step S138, it is determined whether or not the user has instructed the end of tracking. If it is determined that the end of tracking has not been instructed yet, the process returns to step S132 and the above-described processing is repeated. If it is determined in step S138 that the end of tracking has been instructed, the tracking position correction process ends.

  In this way, in the tracking process, in addition to correction by the user, fine adjustment is made to the position of the center of gravity of the tracking target, so that robustness against disturbance is increased.

  By the way, as another example of the process of step S137, the control unit 27 cuts out an input image centered on the tracking point of the object 221 as shown in the upper diagram of FIG. It can also be displayed on the image display 25 as an enlarged zoom image as shown. That is, an enlarged display of the tracking target is also possible.

  Specifically, for example, when the enlarged display of the tracking target is performed, the control unit 27 and the like may execute the tracking correction process of FIG. That is, FIG. 36 is a flowchart illustrating an example of the tracking correction process and an example different from FIG.

  Note that the processing from steps S151 to S156 in FIG. 36 is basically the same as the processing from steps S131 to S136 in FIG. Therefore, the description is omitted here. Therefore, an example of processing after step S157 will be described below.

  In step S157, the control unit 27 calculates an enlargement ratio according to the correction status. Details of the enlargement ratio calculation method will be described later.

  In step S158, the control unit 27 enlarges and displays the target position. That is, the control unit 27 cuts out the image captured by the imaging unit 21 so as to include the target position based on the tracking result by the process of step S155 or S156 (the same process as step S135 or S136 of FIG. 33). The image (in the example of FIG. 35, the image in the dotted frame in the upper diagram) is enlarged at the magnification calculated in step S157 and displayed on the image display 25.

  Thereby, the process of step S158 is complete | finished and a process progresses to step S159. The processing after step S159 is basically the same processing as the processing after step S138 of FIG. Therefore, the description is omitted here.

  Further, details of the processing in steps S157 and S158 (hereinafter referred to as enlarged display processing) in FIG. 36 will be described.

  First, an example of a method for displaying a target position in enlarged display will be described.

  For example, as shown in the lower view of FIG. 35 described above, a method of superimposing and displaying the mark 221A on the tracking point that is the target position of the object 221 can be employed.

  Also, for example, as shown in FIG. 37, a method of displaying four marks 221L, 221R, 221U, and 221D at the four corners of the image can be employed to indicate the tracking point that is the target position of the object 221. That is, the intersection of the line segment connecting the two horizontal marks 221L and 221R and the line segment connecting the two vertical marks 221U and 221D indicates the tracking point that is the target position of the object 221.

  Needless to say, the display method of the target position in the enlarged display is not particularly limited to the above-described example. However, it is preferable that the method allows the user to easily visually recognize the target position in the image as in the example described above.

  Next, the enlargement ratio will be described.

  The enlargement ratio is defined as, for example, the following formula (14). According to this definition, the minimum value of the enlargement ratio is 1.

  Magnification factor = size of output image / size of cropped image (14)

  For example, when 4 is calculated as the enlargement ratio in the process of step S157 in FIG. 36, a zoom image that is four times the size of the clipped image is generated and displayed on the image display 25 in the process of the next step S158. Will be.

  This enlargement ratio can be set so that the user can set an arbitrary value when no correction instruction is given.

  Of course, the enlargement ratio can be fixed. However, if the enlargement ratio is always constant when the user corrects, the situation around the object 221 may be difficult to see. As a result, the correction operation may be difficult for the user.

  For example, in the upper side view of FIG. 35 described above, the user may desire to correct the position of the tracking target object 221 based on the relative position between the tracking target object 221 and another object 222. . In such a case, when the enlarged display as shown in the lower view of FIG. 35 is performed, the separate object 222 is completely invisible. As a result, the user cannot grasp the relative position with respect to the different object 222, and the correction operation becomes difficult.

  Therefore, the control unit 27 can automatically adjust the enlargement ratio at the time of correction in order to make it easy to see the surrounding situation and to facilitate the user's correction operation.

  Here, the automatic adjustment of the enlargement ratio means that the control unit 27 calculates the enlargement ratio by autonomous determination without involving the user's instruction operation, and displays an enlarged image at the enlargement ratio. .

  In other words, the adjustment for the manual adjustment in which the user manually adjusts the enlargement ratio while correcting the position is an automatic adjustment of the enlargement ratio. That is, manual adjustment can be adopted as the adjustment of the enlargement ratio. However, in this case, the user must perform a plurality of other operations simultaneously with the manual adjustment of the enlargement ratio. That is, the user must perform a very difficult and complicated operation. Therefore, when manual adjustment is adopted, it goes against the purpose of facilitating the user's correction operation. Therefore, in the present embodiment, automatic adjustment is adopted as adjustment of the enlargement ratio so as to save labor for complicated user operations.

  Hereinafter, the automatic adjustment will be described in more detail.

  FIG. 38 is a diagram for explaining the basic concept of automatic adjustment. The upper image in FIG. 38, that is, the image displayed at time ta shows an image when correction is not performed.

  It is assumed that correction is started after the time ta. In other words, it is assumed that YES is determined in step S154 of FIG. 36 for each frame. In this case, the control unit 27 automatically reduces the enlargement ratio in the enlargement display processing in steps S157 and S158 for each frame. That is, a series of images (images of each frame) are output while the enlargement ratio is reduced. Therefore, for the user, instead of the object 221 being reduced gradually, the surrounding situation becomes gradually visible. For example, the center image in FIG. 38, that is, the image displayed at time tb is an image in which the enlargement ratio is automatically reduced so that the surrounding situation such as another object 222 can be seen (an image of a predetermined frame). ) Is an example.

  It is assumed that the correction is finished after the time tb. In other words, it is assumed that NO is determined in the process of step S154 of FIG. In this case, the control unit 27 automatically increases the enlargement ratio in the enlargement display processing in steps S157 and S158 for each subsequent frame, and finally returns to the original enlargement ratio. That is, a series of images (images of each frame) are output while the enlargement ratio increases. Therefore, for the user, the object 221 is gradually enlarged and appears to return to the original size. For example, the lower image of FIG. 38, that is, the image displayed at time tc is an example of an image in which the size of the object 221 is restored (an image of a predetermined frame at a stage where the enlargement ratio is restored). Is shown.

  Specifically, for example, the enlargement ratio changes as shown in FIG. That is, FIG. 39 shows an example of the change over time of the enlargement ratio during automatic adjustment.

  In FIG. 39, the horizontal axis represents the time axis, and the vertical axis represents the enlargement ratio.

  As for the enlargement ratio, Za represents the enlargement ratio at the normal time, that is, the enlargement ratio without correction. Zb represents a specified value of the enlargement ratio at the time of automatic adjustment. That is, the enlargement ratio Zb indicates a lower limit value when the enlargement ratio is automatically changed. The enlargement factor Zb can be varied as will be described later, but here, a fixed value defined in advance is adopted for simplicity of explanation.

  Regarding time, t0 indicates the time when the user started correction. t1 indicates the time at which the enlargement rate reaches the specified value. t2 indicates the time when the user finished the correction. t3 indicates the time to return to the original enlargement ratio.

  In this case, as a process of each step S157 for each frame between times t0 and t1 and each frame between times t2 and t3, the control unit 27 enlarges each frame as follows, for example. Each rate can be calculated.

  For example, the control unit 27 can calculate the enlargement ratio of each frame by defining the time required for automatic adjustment.

  Specifically, for example, a predetermined time for a required time ΔT01 between times t0 and t1 shown in the following equation (15) and a required time ΔT23 between times t1 and t2 shown in the following equation (16): Is set in advance.

△ T01 = t1−t0 (15)
△ T23 = t3−t2 (16)

  Then, the control unit 27 calculates the change rate of the enlargement rate (change amount of the enlargement rate per unit time) ΔV01 and ΔV23 using the following formulas (17) and (18).

ΔV01 = (Zb − Za) / ΔT01 (17)
△ V23 = (Za-Zb) / △ T23 (18)

  Finally, the control unit 27 determines the enlargement ratio according to the calculated change rates ΔV01 and ΔV23. As a result, the change in the enlargement rate can be completed within the defined required time ΔT01 or ΔT23. However, it should be noted that the change rate of the enlargement factor depends on the enlargement factor Za before correction.

  Note that the method for calculating the enlargement ratio of each frame between times t0 and t1 and each frame between times t2 and t3 is not particularly limited to the method using Equations (15) to (18) described above. In addition, for example, it is possible to adopt a method in which a change rate of the enlargement rate (a change amount of the enlargement rate per unit time) is defined and the enlargement rate is determined based on the defined change rate. When such a method is employed, the enlargement ratio can be changed at a defined change rate. However, it should be noted that the required times ΔT01 and ΔT23 depend on the enlargement ratio Za before correction.

  Also, the change pattern of the enlargement ratio between the times t0 to t1 and between the times t2 to t3 is not particularly limited to the example of FIG. That is, the pattern is not particularly limited to a pattern in which the enlargement ratio is linearly changed. In addition, for example, it is possible to adopt a pattern that changes in a curved manner.

  Moreover, although the enlargement ratio Zb is fixed in the above-described example, it can be changed by employing the following various methods. The following various methods can be combined.

  As an example of a variable method of the enlargement factor Zb, for example, a method designated by the user, an automatic adjustment method according to an object, and a method such as an enlargement factor method according to a correction amount can be considered. Hereinafter, these methods will be described individually.

  First, the designation method by the user will be described.

  The designation method by the user means a method by which the user can directly specify the enlargement ratio Zb. As a specific example of the designation method by the user, for example, a designation method based on an absolute value or a designation method based on a relative value can be adopted.

  The designation method using the absolute value is a method in which the user directly designates a desired enlargement ratio Zb by a numerical value itself (absolute value) such as “1.0”, or a numerical value (absolute value) indicating the number of pixels in the coordinate system of the input image. A method specified by (value).

  The designation method based on the relative value is that the user designates the desired enlargement factor Zb by a relative value with respect to the original enlargement factor Za, for example, “1/2 times the original enlargement factor Za”, A method of designating by a relative value such as a ratio to the number of pixels in the coordinate system.

  Note that a numerical value that is less than 1 in terms of absolute value can be designated as the numerical value of the enlargement ratio Zb. However, the image display in that case does not seem to make much sense from the viewpoint of ease of operation. That is, when there is a purpose other than the ease of operation, it is preferable that a numerical value that is less than 1 in terms of absolute value can be designated as the numerical value of the enlargement ratio Zb.

  In the above-described example, it was assumed that Zb ≦ Za. Therefore, if this premise is not satisfied, it should be noted that exceptional measures such as forcibly changing the enlargement ratio Zb = Za are necessary.

  Next, an automatic adjustment method according to the object will be described.

  The automatic adjustment method according to the object refers to a method of automatically adjusting the enlargement ratio Zb according to the size of the object currently being tracked.

  For example, FIG. 40 is a diagram illustrating an example of an automatic adjustment method according to an object. The upper image in FIG. 40, that is, the image displayed at time tα shows an image when no correction is made. That is, it is assumed that the object 221 protrudes from the output image in the original state of the enlargement ratio Za when no correction is made.

  It is assumed that correction is started after the time tα. That is, assume that YES is determined in step S154 of FIG. 36 for each frame. In this case, the control unit 27 automatically reduces the enlargement ratio in the enlargement display processing in steps S157 and S158 for each frame. At that time, the control unit 27 displays the center image of FIG. 40 at the time tβ of FIG. 40 at the time of correction, that is, the object 221 is displayed at the full output image at the time of correction. Automatically adjusts the zoom factor Zb.

  It is assumed that the correction is finished after the time tβ. In other words, it is assumed that NO is determined in the process of step S154 of FIG. In this case, the control unit 27 automatically increases the enlargement ratio in the enlargement display processing in steps S157 and S158 for each subsequent frame, and finally returns to the original enlargement ratio Za. That is, a series of images (images of each frame) are output while the enlargement ratio increases. Therefore, for the user, the object 221 is gradually enlarged and appears to return to the original size. For example, the lower image of FIG. 40, that is, the image displayed at time tγ is an image in which the size of the object 221 is restored (an image of a predetermined frame at the stage when the original magnification factor Za is restored). An example is shown.

  In this case, in combination with the designation method by the user, the enlargement ratio Zb may be designated by a relative value with respect to the size of the object 221 such as 1/2 of the size of the object 221.

  In addition, as an automatic adjustment method according to the object, for example, a method of automatically adjusting the enlargement ratio Zb according to the movement amount Δx of the tracking point (the object 221 in the example of FIG. 40) can be employed. Specifically, for example, a method of reducing the enlargement factor Zb when the motion amount Δx is large and increasing the enlargement factor Zb when the motion amount is small can be employed. This method is based on the reason that if the movement is large (fast), it would be easier to operate by displaying a wider range of images, and conversely if the movement is slow, it would not be necessary to change the magnification rate much.

  Next, an enlargement ratio method according to the correction amount will be described.

  The enlargement factor method according to the correction amount is a method in which the user automatically adjusts the enlargement factor Zb according to the correction amount Δu when the target position as the tracking point is corrected. For example, the method of reducing the enlargement factor Zb when the correction amount Δu is large and increasing the enlargement factor Zb when the correction amount Δu is small is an example of the enlargement factor method according to the correction amount. The method of this example is because it is easier to operate when displaying a wider range of images when performing large corrections, and it is better not to change the enlargement ratio too much when performing small corrections. It is a method based on this.

  The method for changing the enlargement ratio Zb has been described above.

  Similar to the change of the enlargement factor Zb described above, the change amount of the enlargement factor can be changed. That is, the change amount of the enlargement ratio is fixed in the above-described example, but can be changed by employing the following various methods. The following various methods can be combined.

  As an example of the variable method of the change rate of the enlargement ratio, similar to the variable method of the enlargement ratio Zb, there are methods such as a user designation method, an automatic adjustment method according to the object, and an enlargement rate method according to the correction amount. It is done.

  For example, as the designation method by the user, a method in which the user can select from a plurality of options such as a method of changing linearly and a method of changing smoothly (for example, a spline curve) can be adopted.

  For example, as an automatic adjustment method according to the object, when the enlargement factor Zb corresponding to the size of the object is calculated by the variable method of the enlargement factor Zb, enlargement is performed according to the difference between the enlargement factor Zb and the enlargement factor Za. A technique of adjusting the rate change rate can be adopted. Specifically, for example, when the difference (Za−Zb) is large, a method of increasing the change amount of the enlargement ratio, and when the difference is small, a method of reducing the change amount of the enlargement ratio can be employed. This method is based on the reason that if it is difficult to see the whole image of the object (the difference is large), it will be easier to operate by making it look faster, and vice versa. .

  For example, as an automatic adjustment method according to the object, a method of automatically adjusting the change amount of the enlargement ratio according to the movement amount Δx of the tracking point (object) can be adopted. Specifically, for example, a method of increasing the amount of change in the enlargement ratio when the amount of movement Δx is large and reducing the amount of change in the enlargement ratio when the amount of movement Δx is small can be employed. This method is based on the reason that if the movement is large (fast), it would be easier to operate if a wider range of images were displayed faster, and vice versa.

  For example, as an enlargement ratio method according to the correction amount, when the correction amount Δu in the correction of the tracking position of the user is large, the change amount of the enlargement factor is increased, and when the correction amount Δu is small, the change of the enlargement factor is A method of reducing the amount can be adopted. This method is based on the reason that when a large correction is performed, it is easier to operate if a wider range of images is displayed faster, and vice versa.

  Next, another example of the exception process at step S12 performed following the normal process at step S11 in FIG. 5 will be described with reference to the flowchart in FIG. This process is basically the same as the process of FIG. 12, and it is determined in step S24 of FIG. 9 that it is impossible to estimate the movement of the tracking point, and further, the transfer candidate for changing the tracking point in step S28. This is executed when it is determined that cannot be selected.

  Since the processing of steps S171 to S175 is the same as the processing of steps S51 to S55 of FIG. 12 described above, the description thereof will be simplified. That is, in step S171, exception process initialization processing is executed. In step S172, an input of an image of the next frame is awaited. In step S173, template matching processing is performed within the template search range. In step S174. When it is determined whether or not it is possible to return to the normal process, and when it is determined that the return to the normal process is possible, the process returns to the normal process of step S11 in FIG. If it is determined that it is not possible, a continuation determination process is executed in step S175.

  In step S176, the control unit 59 determines whether or not the tracking point tracking in the exception process by the continuation determination process in step S175 can be continued (in steps S76 and 78 in FIG. 16). Determine based on the flag set. If it is determined that the tracking process of the tracking point can be continued, the process returns to step S172, and the subsequent processes are repeatedly executed. That is, the process of waiting until the tracking point appears again is repeatedly executed.

  On the other hand, when it is determined in step S176 that the tracking process of the tracking point in the exception process cannot be continued (the number of elapsed frames after the tracking point disappeared in the process of step S75 in FIG. 16 is the threshold THfr). If it is determined as above, or if the number of scene changes is determined to be greater than or equal to the threshold THsc in the process of step S77), the process proceeds to step S177, and the control unit 59 controls and holds the tracking point determination unit 57. Set the tracking point. Thereafter, the process returns to the normal process in step S11 of FIG.

  In this way, when tracking of the tracking point in exception processing becomes impossible, it is possible to return to normal processing again using the tracking point that has been held, so the tracking processing is continued. Can be performed.

  In the above, when it is determined that the desired result is not obtained by the tracking process, the user correction is reflected in real time based on the correction instruction from the user, and therefore the tracking process can be continuously performed. Thereby, the load at the time of designating and correcting the tracking point is reduced, and a more suitable tracking result can be obtained.

  In the above description, an example in which the present invention is applied to a surveillance camera system has been described. However, the present invention is not limited thereto, and can be applied to various image processing apparatuses such as a television receiver.

  Further, in the above, the image processing unit is a frame, but it is of course possible to use a field as a processing unit.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 42 is a block diagram showing an example of the configuration of a personal computer that executes the above-described series of processing by a program.

  42, a CPU (Central Processing Unit) 301 executes various processes according to a program stored in a ROM (Read Only Memory) 302 or a storage unit 308. A RAM (Random Access Memory) 303 appropriately stores programs executed by the CPU 301 and data. The CPU 301, ROM 302, and RAM 303 are connected to each other by a bus 304.

  An input / output interface 305 is also connected to the CPU 301 via the bus 304. The input / output interface 305 is connected to an input acquisition unit 306 including a keyboard, a mouse, and a microphone, and an output unit 307 including a display and a speaker. The CPU 301 executes various processes in response to commands input from the input acquisition unit 306. Then, the CPU 301 outputs the processing result to the output unit 207.

  The storage unit 308 connected to the input / output interface 305 includes, for example, a hard disk, and stores programs executed by the CPU 301 and various data. The communication unit 309 communicates with an external device via a network such as the Internet or a local area network.

  A program may be acquired via the communication unit 309 and stored in the storage unit 308.

  A drive 310 connected to the input / output interface 305 drives a removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives the programs and data recorded therein. Get etc. The acquired program and data are transferred to and stored in the storage unit 308 as necessary.

  As shown in FIG. 42, a program recording medium that stores a program that is installed in a computer and is ready to be executed by the computer includes a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read). 1), a removable medium 311 which is a package medium composed of a magneto-optical disk, a semiconductor memory or the like, or the removable medium 28 of FIG. 1, or a program is temporarily or permanently stored. ROM 302, a hard disk constituting the storage unit 308, and the like. The program is stored in the program recording medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via a communication unit 309 that is an interface such as a router or a modem as necessary. Done.

  In this specification, the steps for describing a program are not only processes performed in time series in the order described, but also processes that are executed in parallel or individually even if they are not necessarily processed in time series. Is also included.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

It is a block diagram which shows the structural example of the surveillance camera system to which this invention is applied. It is a flowchart explaining the example of a monitoring process. It is a figure which shows the example of the image displayed by the surveillance camera system of FIG. It is a block diagram which shows the structural example of the object tracking part of FIG. It is a flowchart figure explaining the example of a tracking process. It is a figure explaining tracking when a tracking object rotates. It is a figure explaining the tracking when an occlusion occurs. It is a figure explaining the tracking when a scene change occurs. It is a flowchart explaining the example of a normal process. It is a flowchart explaining the example of the initialization process of a normal process. It is a figure explaining the example of the initialization process of a normal process. It is a flowchart explaining the example of exception processing. It is a flowchart explaining the example of the initialization process of an exception process. It is a figure explaining selection of a template. It is a figure explaining the setting of a search range. It is a flowchart explaining the example of a continuation determination process. It is a block diagram which shows the structural example of a motion estimation part. It is a flowchart explaining the example of a motion estimation process. It is a figure explaining calculation of activity. It is a figure explaining the relationship between an evaluation value and activity. It is a flowchart explaining the example of an integration process. It is a block diagram which shows the structural example of a background motion estimation part. It is a flowchart explaining the example of a background motion estimation process. It is a block diagram which shows the structural example of a scene change detection part. It is a flowchart explaining the example of a scene change detection process. It is a block diagram which shows the functional structural example of the tracking position correction | amendment part of FIG. It is a figure explaining a block matching system. It is a figure explaining the example which reflects a user's correction value. It is a figure explaining the other example in which a user's correction value is reflected. It is a figure explaining the example which fine-tunes a tracking point. It is a figure explaining the example of a motion vector detection process. It is a figure explaining the example of the sample point extraction process of tracking object. It is a flowchart explaining the example of a tracking correction process. It is a figure which shows the example of the output image after an input image and a tracking process. It is a figure which shows the example of the output image after an input image and a tracking process and an enlarged display process. It is a flowchart explaining another example of a tracking correction process. It is a figure explaining the enlarged display process in a tracking correction process. It is a figure explaining the enlarged display process in a tracking correction process. It is a figure explaining the enlarged display process in a tracking correction process. It is a figure explaining the enlarged display process in a tracking correction process. It is a flowchart explaining the example of the exception process of another example. It is a block diagram which shows the structural example at the time of comprising at least one part of the surveillance camera system to which this invention is applied with the personal computer.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Surveillance camera system, 21 Imaging part, 22 Tracking object detection part, 23 Object tracking part, 51 Template matching part, 52 Motion estimation part, 53 Scene change detection part, 54 Background motion estimation part, 55 Area estimation related process part, 56 Transfer candidate holding unit, 57 tracking point determining unit, 58 template holding unit, 161 correction value calculating unit, 162 correction value reflecting unit, 163 fine adjustment unit

Claims (13)

In an image processing apparatus that tracks a moving object,
Tracking means for tracking the movement of the object in the image;
When correction of the tracking point on the object is instructed during tracking of the object by the tracking means, reflecting means for reflecting the correction instruction on the tracking result while continuing the tracking processing of the object;
An image processing apparatus comprising: a display control unit that adjusts an enlargement ratio of the display of the image by autonomous determination when the correction is instructed, and displays the image in an enlarged manner with the adjusted enlargement ratio. When the correction is started as the adjustment of the enlargement ratio, the display control means decreases the enlargement ratio from the first enlargement ratio to the second enlargement ratio, and sets the enlargement ratio to the second enlargement ratio. The image processing apparatus according to claim 1, wherein the image processing apparatus maintains the enlargement ratio and increases the enlargement ratio from the second enlargement ratio to the first enlargement ratio when the correction is completed. The display control means includes
For the adjustment of the enlargement ratio in the first period from the first enlargement ratio to the second enlargement ratio, the enlargement ratio is calculated based on the change rate of the enlargement ratio during the first period,
Regarding the adjustment of the enlargement factor in the second period from the second enlargement factor to the first enlargement factor, the enlargement factor is calculated based on the change rate of the enlargement factor during the second period. Item 3. The image processing apparatus according to Item 2. The first period and the second period are predefined,
The display control means includes
For the adjustment of the enlargement ratio in the first period, calculate the change rate based on the first period, calculate the enlargement ratio using the change rate,
The image according to claim 3, wherein the adjustment of the enlargement ratio in the second change period is calculated based on the second period, and the enlargement ratio is calculated using the change speed. Processing equipment. The image processing apparatus according to claim 3, wherein the change rate during the first period and the change rate during the second period are defined in advance. The image processing apparatus according to claim 2, wherein the display control unit adjusts the enlargement ratio by changing the second enlargement ratio. The display control means includes
Varying the magnification rate change rate during the first period from the first magnification rate to the second magnification rate, and adjusting the magnification rate during the first period;
The zoom ratio during the second period is adjusted by varying a change rate of the zoom ratio during the second period from the second magnification ratio to the first magnification ratio. Image processing device. The image processing apparatus according to claim 1, further comprising a calculation unit that calculates a correction value based on the correction instruction. The image processing apparatus according to claim 8, wherein the reflection unit reflects the correction value calculated by the calculation unit in the tracking result as a relative value with respect to a coordinate system of the image. The image processing apparatus according to claim 8, wherein the reflecting unit reflects the correction value calculated by the calculating unit in the tracking result as a relative value with respect to the tracking target. The image processing apparatus according to claim 1, further comprising: a fine adjustment unit that finely adjusts the tracking point so as to be a center of gravity position of the object after the correction instruction is reflected by the reflection unit. In an image processing method of an image processing apparatus for tracking a moving object,
Track the movement of the object in the image,
When correction of the tracking point on the object is instructed during tracking of the object, the correction instruction is reflected in the tracking result while continuing the tracking processing of the object,
An image processing method including a step of adjusting an enlargement ratio of display of the image by autonomous determination when the correction is instructed, and causing the image to be enlarged and displayed at an adjusted enlargement ratio. In a program for causing a computer to execute image processing of an image processing apparatus that tracks a moving object,
Track the movement of the object in the image,
When correction of the tracking point on the object is instructed during tracking of the object, the correction instruction is reflected in the tracking result while continuing the tracking processing of the object,
A program comprising the steps of: adjusting the enlargement ratio of the display of the image by autonomous determination when the correction is instructed, and displaying the image in an enlarged manner with the adjusted enlargement ratio.

Images