JP2007272735A - Image processor and image processing method, program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To smoothly display a tracking object even when the tracking object performs specific movement such as vertical vibration and horizontal vibration. <P>SOLUTION: Tracking results and moving vectors calculated for each frame from a tracking processing part are stored in a tracking object position memory 802 and a tracking object movement memory 801, respectively. The presentation positions to the previous frame are stored in a presentation position memory 803. A presentation position calculation part 805 calculates the presentation position of a current frame by carrying out filtering in a time direction by using weighting factors calculated by a weighting factor calculation part 804 to information as the object of filtering among the information stored in the tracking object movement memory 801, the tracking object position memory 802 and the presentation position memory 803. This invention can be applied to a television receiver for tracking and enlarging and displaying an object. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and method, a program, and a recording medium, and in particular, smoothly displays a tracking display that becomes unnatural when a tracking target makes a specific movement such as vertical and horizontal vibrations. The present invention relates to an image processing apparatus and method, a program, and a recording medium.

  There have been many techniques for tracking an object designated by a user in a moving image, and the applicant has also proposed in Patent Document 1 filed earlier.

JP 2005-303983 A

  However, in the conventional technique, the tracking result obtained by tracking is displayed as it is. For this reason, for example, when the tracking target moves, such as vertical vibration or horizontal vibration in ski competitions, or sudden movements with different accelerations, the movement of the background portion is particularly unnatural in the displayed image. Therefore, the user who is viewing the image may feel uncomfortable.

  The present invention has been made in view of such a situation, and it is possible to smoothly display a tracking display that becomes unnatural when a tracking target makes a specific movement such as vertical and horizontal vibrations. It is something that can be done.

  An image processing apparatus according to an aspect of the present invention includes: a tracking unit that tracks the movement of the object by estimating the movement of the object in the image in the image processing apparatus that displays the moving object; and the tracking unit. It is calculated by the presentation position calculation means for calculating the presentation position by performing filtering in the time direction on the tracked tracking result or the estimated motion corresponding to the tracking result, and the presentation position calculation means. Display image creating means for creating a display image based on the presentation position; and display control means for controlling display of the display image created by the display image creating means.

  The display image creation means can create a zoomed display image with the presentation position calculated by the presentation position calculation means as a center.

  The presentation position calculation means includes coefficient calculation means for calculating a coefficient used for filtering in the time direction according to the magnitude of movement corresponding to the tracking result, and corresponds to the tracking result or the tracking result. The presentation position can be calculated by performing filtering in the time direction using the coefficient calculated by the coefficient calculation unit with respect to the estimated motion.

The presentation position calculation unit includes a determination unit that determines whether or not a sum of movements corresponding to the tracking results of a plurality of processing units is smaller than a predetermined threshold, and the plurality of processing units estimated by the determination unit. When it is determined that the sum of movements corresponding to the tracking result is smaller than a predetermined threshold, the presentation position calculated in the previous processing unit in time is used as it is, and the presentation position in the subsequent processing unit is used as it is. Can be calculated.

  An image processing method according to one aspect of the present invention is an image processing method of an image processing apparatus that displays a moving object, and the movement of the object is tracked and tracked by estimating the movement of the object in the image. For the tracking result or the estimated motion corresponding to the tracking result, time-direction filtering is performed to calculate a presentation position, and a display image is created based on the calculated presentation position. Controlling the display of the displayed image.

  A program according to one aspect of the present invention is a program that causes a computer to perform a process of displaying a moving object, and tracks the movement of the object by estimating the movement of the object in the image. For the tracking result or the estimated motion corresponding to the tracking result, time-direction filtering is performed to calculate a presentation position, and a display image is created based on the calculated presentation position. Controlling the display of the displayed image.

  A program recorded on a recording medium according to one aspect of the present invention is a program that causes a computer to perform a process of displaying a moving object, and the movement of the object is estimated by estimating the movement of the object in an image. , And by performing filtering in the time direction on the tracked tracking result or the estimated motion corresponding to the tracking result, the presentation position is calculated, and based on the calculated presentation position Creating a display image and controlling display of the created display image.

  In one aspect of the present invention, the movement of the object is tracked by estimating the movement of the object in the image, and the tracking result that is tracked or the estimated movement corresponding to the tracking result is detected. The presentation position is calculated by filtering in the time direction. Then, a display image is created based on the calculated presentation position, and display of the created display image is controlled.

  According to the present invention, even if the tracking target moves in a specific manner such as vertical and horizontal vibrations, the tracking target can be displayed smoothly.

  Embodiments of the present invention will be described below. Correspondences between constituent elements of the present invention and the embodiments described in the specification or the drawings are exemplified as follows. This description is intended to confirm that the embodiments supporting the present invention are described in the specification or the drawings. Therefore, even if there is an embodiment which is described in the specification or the drawings but is not described here as an embodiment corresponding to the constituent elements of the present invention, that is not the case. It does not mean that the form does not correspond to the constituent requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

  An image processing apparatus according to one aspect of the present invention estimates the movement of an object in an image in an image processing apparatus (for example, the television receiver 1 in FIG. 1) that displays a moving object. The tracking means for tracking the movement (for example, the tracking processing unit 41 in FIG. 3) and the tracking result tracked by the tracking means or the estimated motion corresponding to the tracking result are filtered in the time direction. By performing, a presentation position calculation unit that calculates a presentation position (for example, the presentation position calculation unit 42 in FIG. 3) and a display image creation that creates a display image based on the presentation position calculated by the presentation position calculation unit Means (for example, the zoom image creating unit 24 in FIG. 1) and display control means (for example, controlling display of the display image created by the display image creating means) , And an image display 26 of FIG. 1).

  The presentation position calculation means includes coefficient calculation means (for example, a weight coefficient calculation unit 841 in FIG. 47) that calculates a coefficient used for filtering in the time direction according to the magnitude of the movement corresponding to the tracking result. Calculating the presentation position by performing filtering in the time direction using the coefficient calculated by the coefficient calculation unit for the tracking result or the estimated motion corresponding to the tracking result; it can.

The presentation position calculation means includes determination means (for example, a movement determination unit 861 in FIG. 50) for determining whether or not a sum of movements corresponding to the tracking results of a plurality of processing units is smaller than a predetermined threshold. When it is determined that the sum of movements corresponding to the tracking results of the plurality of processing units estimated by the determination unit is smaller than a predetermined threshold, the presentation position calculated in the previous processing unit in time is used as it is. The presentation position of the later processing unit can be calculated in terms of time.

  An image processing method or program according to one aspect of the present invention tracks the movement of an object by estimating the movement of the object in the image in the image processing method or program of the image processing apparatus that displays the moving object. (For example, step S21 in FIG. 4), the presentation position is calculated by performing time-direction filtering on the tracked tracking result or the estimated motion corresponding to the tracking result (for example, FIG. 4 (step S22), a display image is created based on the calculated presentation position (for example, step S4 in FIG. 2), and the display of the created display image is controlled (for example, step S6 in FIG. 2). Includes steps.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows a configuration example when the present invention is applied to a television receiver. The television receiver 1 includes a tuner 21, an image processing unit 22, an object tracking unit 23, a zoom image creation unit 24, a selection unit 25, an image display 26, an audio processing unit 27, a speaker 28, a control unit 29, and a remote controller 30. , And a removable medium 31.

  The tuner 21 receives the RF signal, demodulates it and separates it into an image signal and an audio signal, outputs the image signal to the image processing unit 22, and outputs the audio signal to the audio processing unit 27.

  The image processing unit 22 demodulates the image signal input from the tuner 21 and outputs the demodulated signal to the object tracking unit 23, the zoom image creation unit 24, and the selection unit 25. The object tracking unit 23 performs a process of tracking the tracking point of the object specified by the user from the input image, and uses the tracking result to obtain a presentation position as a reference for presenting the tracked object. The coordinate information regarding the presentation position is output to the zoom image creation unit 24. The zoom image creation unit 24 creates a zoom image (that is, a zoomed display image) based on the presentation position. The selection unit 25 selects one of the image supplied from the image processing unit 22 or the image supplied from the zoom image creation unit 24 based on an instruction from the user, and outputs the selected image to the image display 26 for display.

  The audio processing unit 27 demodulates the audio signal input from the tuner 21 and outputs it to the speaker 28.

  The control unit 29 is configured by, for example, a microcomputer and controls each unit based on a user instruction. The remote controller 30 is operated by the user and outputs a signal corresponding to the operation to the control unit 29. The removable medium 31 includes a semiconductor memory, a magnetic disk, an optical disk, a magneto-optical disk, and the like. The removable medium 31 is mounted as necessary, and provides the control unit 29 with a program and various other data.

  Next, processing of the television receiver 1 will be described with reference to the flowchart of FIG.

  In step S1, the tuner 21 demodulates the signal of the channel designated by the user from the RF signal received via the antenna (not shown), outputs the image signal to the image processing unit 22, and processes the sound signal as sound processing. To the unit 27. The audio signal is demodulated by the audio processing unit 27 and then output from the speaker 28.

  The image processing unit 22 demodulates the input image signal and outputs the demodulated image signal to the object tracking unit 23, zoom image creation unit 24, and selection unit 25.

  In step S2, the object tracking unit 23 determines whether or not tracking is instructed. If it is determined that tracking is not instructed, the processing of steps S3 and S4 is skipped. In step S <b> 5, the selection unit 25 selects one of the image signal supplied from the image processing unit 22 and the image signal input from the zoom image creation unit 24 based on control from the control unit 29. In this case, since there is no specific instruction from the user, the control unit 29 causes the selection unit 25 to select the image signal from the image processing unit 22. In step S <b> 6, the image display 26 displays the image selected by the selection unit 25.

  In step S7, the control unit 29 determines whether or not to end the image display process based on a user instruction. That is, when ending the image display process, the user operates the remote controller 30 and instructs the control unit 29 to do so. If the user has not instructed termination, the process returns to step S1, and the subsequent processes are repeatedly executed.

  As described above, normal processing for displaying an image corresponding to the signal received by the tuner 21 as it is is executed.

  When an image desired to be tracked is displayed by looking at the image displayed on the image display 26, the user designates the image by operating the remote controller 30. When this operation is performed, in step S2, the control unit 29 determines that tracking is instructed, and controls the object tracking unit 23. Based on this control, the object tracking unit 23 starts tracking processing of the tracking point designated by the user.

  Details of the tracking process will be described later with reference to FIG. 4. By this process, the tracking point designated by the user, for example, an object to be tracked in the image (for example, a person, an animal, etc.) Tracking point (eyes, center of head, etc.) is tracked, and using the tracking result, the presentation position that is the reference for the presentation is calculated when the tracked object is presented and represents the presentation position The position information is output to the zoom image creation unit 24 as presentation position information.

  In step S <b> 4, the zoom image creation unit 24 generates a zoom image based on the presentation position obtained by the object tracking unit 23 and outputs the zoom image to the selection unit 25. That is, the zoom image creating unit 24 generates a zoom image with the presentation position obtained by the object tracking unit 23 as the center.

  This zoom image creation processing can be performed using the class classification adaptation processing previously proposed by the present applicant. For example, Japanese Patent Laid-Open No. 2002-196737 discloses a process for converting a 525i signal into a 1080i signal using a coefficient obtained by learning in advance. This process is substantially the same as the process of enlarging the image 9/4 times in both the vertical direction and the horizontal direction. However, since the image display 26 has a fixed number of pixels, the zoom image creation unit 24 converts the 525i signal into a 1080i signal and then centers the presentation position when creating a 9/4 times image, for example. A zoom image can be created by selecting a predetermined number of pixels (number of pixels corresponding to the image display 26).

  Based on this principle, a zoom image with an arbitrary magnification can be generated.

  If tracking is instructed, the selection unit 25 selects the zoom image created by the zoom image creation unit 24 in step S5. As a result, in step S6, the image display 26 displays the zoom image created by the zoom image creation unit 24.

  As described above, the image display 26 displays a zoom image centered on the presentation position obtained using the tracking result of the tracking point designated by the user. When the magnification is set to 1, a display image without zooming is created and displayed based on the presentation position.

  Next, a detailed configuration example and operation of the object tracking unit 23 in FIG. 1 will be described. FIG. 3 is a block diagram illustrating a functional configuration example of the object tracking unit 23. In this example, the object tracking unit 23 includes a tracking processing unit 41 and a presentation position calculation unit 42.

  The tracking processing unit 41 estimates the motion of the input image and calculates a tracking point based on the motion vector obtained as a result of the estimation. Then, the tracking processing unit 41 supplies the tracking result such as the calculated position information of the tracking point and the motion vector corresponding to the tracking result obtained as a result of the estimation to the presentation position calculating unit 42. For example, the tracking processing unit 41 performs tracking using a tracking method such as a point tracking method with transfer described in Patent Document 1 or a tracking method such as a motion region centroid tracking method. Details of the tracking processing unit 41 will be described later with reference to FIG.

  The presentation position calculation unit 42 uses the tracking result and the motion vector supplied by the tracking processing unit 41 to calculate a presentation position serving as a reference when presenting the tracking of the object, and displays information regarding the calculated presentation position as a zoom. The data is output to the image creation unit 24, the control unit 29, and the like.

  Next, the operation of the object tracking unit 23 will be described. FIG. 4 is a flowchart for explaining the details of the tracking process executed by the object tracking unit 23 in step S3 of FIG.

  First, in step S21, the tracking processing unit 41 of the object tracking unit 23 executes tracking point calculation processing for calculating a tracking point by tracking the tracking point specified in step S2. The tracking point calculation process will be described later with reference to FIG. 6 or 41. By this process, the motion of the input image is estimated and specified by the user based on the motion vector obtained as a result of the estimation. The tracking point is obtained, and the motion vector information obtained as a result of the estimation, the tracking result that is the position information of the tracking point obtained using the motion vector, and the like are supplied to the presentation position calculation unit 42.

  In step S22, the presentation position calculation unit 42 uses the tracking result determined in step S21 and its motion vector to further provide a presentation position as a reference when presenting object tracking, that is, a zoom image creation. When the unit 24 creates a zoom image, a presentation position calculation process for calculating a presentation position as the center is executed.

  The details of this presentation position calculation process will be described later with reference to FIG. 45, but by this process, a plurality of motion vectors estimated from each frame in step S21 or a plurality of motion vectors obtained using the motion vectors are obtained. Filtering in the time direction is performed on the tracking results for the frames, so that the presentation position serving as a reference when presenting the tracking of the object, that is, the presentation position serving as the zoom center when creating a zoom image is obtained. The obtained information is supplied to the zoom image creating unit 24.

  Then, the process returns to step S3 in FIG. 2, and in step S4, the zoom image creation unit 24 generates a zoom image based on the presentation position obtained by the object tracking unit 23, and the generated zoom image is displayed. If it is selected by the selection unit 25 in step S5, it is displayed on the image display 26 in step S6.

  That is, an image is displayed on the image display 26 based on the presentation position obtained by performing time-direction filtering on the motion vector and the tracking result obtained by the tracking process.

  As a result, for example, even if the tracking target has a fine movement such as vertical and horizontal vibrations or a sudden movement with different acceleration, the image display 26 is not a movement as it is, but a fine movement or a sudden movement is alleviated. An image is displayed. Therefore, the conventional method of displaying the position of the tracking point as a reference as it is can reduce the discomfort felt by the user because the fine movement or the sudden movement is displayed as it is.

  Next, a detailed configuration example and operation of the tracking processing unit 41 in FIG. 3 will be described. FIG. 5 is a block diagram illustrating a functional configuration example of the tracking processing unit 41 that performs tracking by the point tracking method with transfer. In this example, the tracking processing unit 41 includes a template matching unit 51, a motion estimation unit 52, a scene change detection unit 53, a background motion estimation unit 54, a region estimation related processing unit 55, a transfer candidate holding unit 56, and a tracking point determination unit 57. , A template holding unit 58 and a control unit 59.

  The template matching unit 51 performs a matching process between the input image and the template image held in the template holding unit 58. The motion estimation unit 52 estimates the motion of the input image, and obtains the motion vector obtained as a result of the estimation and the accuracy of the motion vector, the scene change detection unit 53, the background motion estimation unit 54, the region estimation related processing unit 55, And output to the tracking point determination unit 57. The scene change detection unit 53 detects a scene change based on the accuracy supplied from the motion estimation unit 52.

  The background motion estimation unit 54 executes processing for estimating the background motion based on the motion vector and the accuracy supplied from the motion estimation unit 52, and supplies the estimation result to the region estimation related processing unit 55. The region estimation related processing unit 55 is based on the motion vector and accuracy supplied from the motion estimation unit 52, the background motion supplied from the background motion estimation unit 54, and the tracking point information supplied from the tracking point determination unit 57. Perform region estimation processing. Further, the region estimation related processing unit 55 generates a transfer candidate based on the input information, supplies the transfer candidate to the transfer candidate holding unit 56, and holds it. Further, the region estimation related processing unit 55 creates a template based on the input image, supplies the template to the template holding unit 58, and holds it.

  The tracking point determination unit 57 determines a tracking point based on the motion vector and accuracy supplied from the motion estimation unit 52 and the transfer candidate supplied from the transfer candidate holding unit 56, and information on the determined tracking point is obtained. The result is output to the region estimation related processing unit 55.

  The control unit 59 controls each part of the template matching unit 51 to the template holding unit 58 based on the tracking point information specified by the control unit 29 to track the specified tracking target, and the tracking point screen. The tracking result such as the position information above and the motion vector corresponding to the determined tracking point are output to the presentation position calculation unit 42 and the like. That is, the tracking result such as information on the tracking point determined by the tracking point determination unit 57 and the motion vector from the motion estimation unit 52 corresponding to the determined tracking point are supplied to the presentation position calculation unit 42.

  Next, details of the tracking point calculation processing in step S21 in FIG. 4 will be described with reference to the flowchart in FIG. The tracking point calculation process basically includes a normal process and an exception process, as shown in FIG.

  That is, the tracking processing unit 41 first executes normal processing in step S51. The details of this normal process will be described later with reference to FIG. 10, but a process for tracking a tracking point designated by the user is executed by this process. When the tracking point cannot be changed in the normal processing in step S51, the tracking processing unit 41 executes exception processing in step S52. Details of the exception processing will be described later with reference to FIG. 25. When the tracking point becomes invisible from the image by this exception processing, the return processing to the normal processing is executed by template matching.

  If it is determined that the tracking process cannot be continued due to the exception process (cannot return to the normal process), the process is terminated. If it is determined that the return is possible, the process returns to step S51 again. In this way, the normal process in step S51 and the exception process in step S52 are sequentially repeated for each frame.

  With this normal processing and exception processing, in the present invention, as shown in FIGS. 7 to 9, the tracking point is temporarily changed such that the tracking target rotates, occlusion occurs, a scene change occurs, etc. Even when it becomes invisible, tracking is possible.

  That is, for example, as shown in FIG. 7, a human face 104 as a tracking target (object) is displayed in the frame n−1, and this human face 104 has a right eye 102 and a left eye 103. ing. It is assumed that the user designates, for example, the right eye 102 (exactly one pixel therein) as the tracking point 101. In the example of FIG. 7, in the next frame n, the person moves to the left in the figure, and in the next frame n + 1, the person's face 104 rotates clockwise. As a result, the right eye 102 that has been visible until now is not displayed, and tracking cannot be performed with the conventional method. Therefore, in the normal process of step S51 described above, the left eye 103 on the face 104 as the same object as the right eye 102 is selected, and the tracking point is switched (set) to the left eye 103. This enables tracking.

  In the display example of FIG. 8, in the frame n−1, the ball 121 has moved from the left side of the face 104 in the drawing, and in the next frame n, the ball 121 has just covered the face 104. In this state, the face 104 including the right eye 102 designated as the tracking point 101 is not displayed. When such an occlusion occurs, the face 104 as the object is not displayed, so there is no transfer point to be tracked instead of the tracking point 101, and thereafter it becomes difficult to track the tracking point. However, in the present invention, the image of the frame n−1 (actually the earlier frame in time) is stored in advance as a template for the right eye 102 as the tracking point 101, and the ball 121 moves further to the right. When the right eye 102 designated as the tracking point 101 appears again in frame n + 1, it is confirmed that the right eye 102 as the tracking point 101 is displayed again by the exception processing in step S52 described above, and the right eye 102 is again tracked. The point 101 is tracked.

  In the example of FIG. 9, the face 104 is displayed in the frame n−1, but in the next frame n, the automobile 111 covers and covers the whole including the human face. That is, in this case, a scene change has occurred. In the present invention, even if the scene change occurs and the tracking point 101 no longer exists in the image, if the automobile 111 moves and the right eye 102 is displayed again in the frame n + 1, the exception processing in step S52 It is confirmed based on the template that the right eye 102 as the tracking point 101 has appeared again, and the right eye 102 can be tracked again as the tracking point 101.

  Next, the details of the normal processing in step S51 in FIG. 6 will be described with reference to the flowchart in FIG. In step S121, the tracking point determination unit 57 executes initialization processing of normal processing. The details will be described later with reference to FIG. 11, but an area estimation range based on the tracking point designated to be tracked by the user is designated by this processing. This area estimation range is the same object as the tracking point specified by the user (for example, when the tracking point is a human eye, a human face as a rigid body that moves like the eye, a human body, or the like) ) Is a range to be referred to when estimating the range of points belonging to. A transfer point is selected from points in the region estimation range.

  Next, in step S122, the control unit 59 controls each unit to wait for input of an image of the next frame. In step S123, the motion estimation unit 52 estimates the tracking point motion. That is, by capturing the frame (following frame) temporally after the frame including the tracking point designated by the user (previous frame) in the process of step S122, an image of two consecutive frames is eventually obtained. Therefore, in step S123, the movement of the tracking point is estimated by estimating the position of the tracking point of the subsequent frame corresponding to the tracking point of the previous frame.

  Note that “preceding in time” means the processing order (input order). Normally, images of each frame are input in the order of imaging. In this case, the frame captured earlier in time becomes the previous frame, but the frame captured later in time is processed (input) first. ), The frame imaged later in time becomes the previous frame.

  In step S124, the motion estimation unit 52 determines whether the tracking point can be estimated as a result of the process in step S123. Whether or not the tracking point can be estimated is determined, for example, by comparing the accuracy value of a motion vector (described later) generated and output by the motion estimation unit 52 with a preset threshold value. Specifically, it is possible to estimate if the accuracy of the motion vector is equal to or greater than a threshold, and it is determined that estimation is impossible if the accuracy is smaller than the threshold. In other words, the possibility here is determined relatively strictly, and if the estimation is not actually impossible but the accuracy is low, it is determined as impossible. As a result, more reliable tracking processing can be performed.

  In step S124, it is determined that the motion estimation result at the tracking point and the motion estimation result at a point near the tracking point can be estimated if they match the motions that occupy the majority, and that they cannot be estimated if they do not match. It is also possible to do so.

  When it is determined that the movement of the tracking point can be estimated (the probability that the tracking point is correctly set on the corresponding point on the same object (if the right eye 102 is designated as the tracking point 101, the right eye 102 If the probability of being correctly tracked is relatively high), the process proceeds to step S125, and the tracking point determination unit 57 shifts the tracking point by the estimated motion (motion vector) obtained in the process of step S123. In other words, this determines the tracking position in the subsequent frame after tracking the tracking point of the previous frame.

  Then, information on the determined tracking point (for example, tracking point position information) is output to the presentation position calculation unit 42 together with the corresponding motion vector, and the presentation position in step S22 of FIG. 4 (FIG. 45 to be described later). Used in the calculation process.

  After step S125, region estimation related processing is executed in step S126. The details of the area estimation related process will be described later with reference to FIG. 14. By this process, the area estimation range designated in the initialization process of the normal process in step S121 is updated. Furthermore, if the tracking point is not displayed because the target object rotates, for example, the candidate as a switching point (transfer candidate) as a point to be switched to is the state (can still be tracked). In the state), it is extracted (created) in advance. Further, when the transfer to the transfer candidate cannot be performed, the tracking is temporarily interrupted, but a template is created in advance in order to confirm that the tracking is possible again (the tracking point appears again).

  After the region estimation related process in step S126 is completed, the process returns to step S122 again, and the subsequent processes are repeatedly executed.

  That is, as long as the movement of the tracking point designated by the user can be estimated, the processing from step S122 to step S126 is repeatedly executed for each frame, and tracking is performed.

  On the other hand, if it is determined in step S124 that the movement of the tracking point cannot be estimated (that is, impossible), that is, as described above, for example, the accuracy of the motion vector is equal to or less than the threshold value. The process proceeds to step S127. In step S127, the tracking point determination unit 57 holds the transfer candidate generated in the region estimation-related processing in step S126 in the transfer candidate holding unit 56, so that the transfer candidate that is closest to the original tracking point is among them. Select one. The tracking point determination unit 57 determines whether or not a transfer candidate has been selected in step S128. If a transfer candidate has been selected, the tracking point determination unit 57 proceeds to step S129 and sets the tracking point as the transfer candidate selected in the process of step S127. Change (change). That is, the transfer candidate point is set as a new tracking point. Thereafter, the process returns to step S123, and the process of estimating the movement of the tracking point selected from the transfer candidates is executed.

  In step S124, it is determined again whether or not the movement of the newly set tracking point can be estimated. If it can be estimated, a process of shifting the tracking point by the estimated movement is performed in step S125, and step S126. In, the area estimation related process is executed. Thereafter, the process returns to step S122 again, and the subsequent processes are repeatedly executed.

  If it is determined in step S124 that the newly set tracking point cannot be estimated, the process returns to step S127, and the next transfer candidate closest to the original tracking point is selected from the transfer candidates. In step S129, the transfer candidate is set as a new tracking point. The process after step S123 is repeated again for the new tracking point.

  Even when all the prepared transfer candidates are used as new tracking points, if the movement of the tracking point cannot be estimated, it is determined in step S128 that the transfer candidate cannot be selected, and this normal process is performed. Is terminated. Then, the process proceeds to the exception process in step S52 of FIG.

  Next, the details of the initialization process of the normal process in step S121 of FIG. 10 will be described with reference to the flowchart of FIG.

  In step S141, the control unit 59 determines whether or not the current process is a process for returning from the exception process. That is, after the exception process in step S52 of FIG. 6 is terminated, it is determined whether or not the process has returned to the normal process in step S51. In the process of the first frame, since the exception process of step S52 has not been executed yet, it is determined that the process has not returned from the exception process, and the process proceeds to step S142. In step S142, the tracking point determination unit 57 performs a process of setting the tracking point to the position of the tracking point instruction that is a predetermined point in the input image designated by the user.

  In addition, when there is no designation | designated by a user, the point with the highest brightness | luminance can also be set as a tracking point in an input image, for example. The tracking point determination unit 57 supplies the set tracking point information to the region estimation related processing unit 55.

  In step S143, the region estimation related processing unit 55 sets a region estimation range based on the position of the tracking point set in the processing of step S142. This region estimation range is a reference range when estimating a point on the same rigid body as the tracking point, and more specifically, tracking so that the same rigid body portion as the tracking point occupies most of the region estimation range in advance. By setting the position and size of the estimated area range to follow the same rigid body part as the point, the part that shows the most movement in the estimated area range can be estimated as the same rigid body part as the tracking point It is for doing so. In step S143, as a default value, for example, a predetermined range centered on the tracking point is set as the region estimation range.

  Thereafter, the processing proceeds to step S122 in FIG.

  On the other hand, if it is determined in step S141 that the current process is a process for returning from the exception process in step S52 of FIG. 6, the process proceeds to step S144, and the tracking point determination unit 57 refers to FIG. Then, the tracking point and the area estimation range are set based on the position matched with the template by the process described later. For example, a point on the current frame that matches the tracking point on the template is set as the tracking point, and a certain range set in advance from the point is set as the region estimation range. Thereafter, the processing proceeds to step S122 in FIG.

  The above process will be described with reference to FIG. That is, in step S142 of FIG. 11, for example, as shown in FIG. 12, when the human eye 102 of the frame n−1 is designated as the tracking point 101, a predetermined region including the tracking point 101 is specified in step S143. Is designated as the area estimation range 133. In step S124, it is determined whether or not the sample points within the region estimation range 133 can be estimated in the next frame. In the case of the example of FIG. 12, in the frame n + 1 next to the frame n, the left half region 134 in the drawing including the left eye 102 in the region estimation range 133 is hidden by the ball 121. 101 motion cannot be estimated in the next frame n + 1. Therefore, in such a case, one of the points in the area designation range 133 (in the face 104 as a rigid body including the right eye 102) prepared in advance as a transfer candidate in the previous frame n-1 in time is selected. One point (for example, the left eye 103 included in the face 104 (exactly one pixel therein)) is selected, and that point is set as the tracking point in the frame n + 1.

  The region estimation related processing unit 55 has a configuration as shown in FIG. 13 in order to execute the region estimation related processing in step S126 of FIG. That is, the motion vector and the accuracy are input from the motion estimation unit 52, the background motion is input from the background motion estimation unit 54, and the tracking point from the tracking point determination unit 57 is input to the region estimation unit 161 of the region estimation related processing unit 55. Position information is input. In addition to the motion vector and the accuracy supplied from the motion estimation unit 52, the transfer candidate extraction unit 162 is also supplied with the output of the region estimation unit 161. In addition to the input image being input to the template creation unit 163, the output of the region estimation unit 161 is input.

  The region estimation unit 161 estimates a rigid region including a tracking point based on the input, and outputs an estimation result to the transfer candidate extraction unit 162 and the template creation unit 163. The transfer candidate extraction unit 162 extracts transfer candidates based on the input, and supplies the extracted transfer candidates to the transfer candidate holding unit 56. The template creation unit 163 creates a template based on the input, and supplies the created template to the template holding unit 58.

  FIG. 14 shows details of the area estimation related process (the process of step S126 in FIG. 10) executed by the area estimation related processing unit 55. First, in step S161, region estimation processing is executed by the region estimation unit 161. The details thereof will be described later with reference to FIG. 15. By this processing, the points of the region on the image that are estimated to belong to the same object (the rigid body that moves in synchronization with the tracking point) to which the tracking point belongs. Are extracted as points of the region estimation range.

  In step S162, the transfer candidate extraction unit 162 executes a transfer candidate extraction process. Details of the processing will be described later with reference to FIG. 20, but transfer candidate points are extracted from the points in the range estimated by the region estimation unit 161 as the region estimation range, and are held in the transfer candidate holding unit 56.

  In step S163, the template creation unit 163 executes template creation processing. Details thereof will be described later with reference to FIG. 21, but a template is created by this processing.

  Next, details of the region estimation processing in step S161 in FIG. 14 will be described with reference to the flowchart in FIG.

  First, in step S181, the region estimation unit 161 determines a sample point as a candidate point of a point estimated to belong to the same target as the tracking point.

  For example, as shown in FIG. 16, the sample points are predetermined in the horizontal direction and the vertical direction with reference to the fixed reference point 201 among the pixels in the entire screen of the frame indicated by the white square in the figure. Pixels at positions separated by the number of pixels can be used as sample points (represented by black squares in the figure). In the example of FIG. 16, the upper left pixel of each frame is set as a reference point 201 (the reference point 201 is indicated by a cross in the figure), and is separated by 5 in the horizontal direction and 5 in the vertical direction. The pixel at the position is taken as the sample point. That is, in the case of this example, pixels at positions dispersed throughout the entire screen are taken as sample points. In this example, the reference point is a point at the same position fixed in each frame n, n + 1.

  Note that the reference point 201 can be dynamically changed so as to be a point at a different position for each of the frames n and n + 1.

  In the example of FIG. 16, the interval between sample points is a fixed value in each frame, but the interval between sample points for each frame is, for example, 5 pixels in frame n and 8 in frame n + 1. It can also be variable with the pixel. As an interval reference at this time, an area of a region estimated to belong to the same object as the tracking point can be used. Specifically, if the area of the region estimation range is narrowed, the interval is also shortened.

  Alternatively, the interval between sample points can be made variable in one frame. The distance from the tracking point can be used as a reference for the interval at this time. That is, the sample point closer to the tracking point has a smaller interval, and the farther from the tracking point, the larger the interval.

  When the sample points are determined as described above, in step S182, the region estimation unit 161 then determines the region estimation range (steps S143 and S144 in FIG. 11 or steps S206 and S208 in FIG. 17 described later). The process of estimating the movement of the sample points within (determined in the process of (1) is executed. That is, the region estimation unit 161 extracts the corresponding point of the next frame corresponding to the sample point within the region estimation range based on the motion vector supplied from the motion estimation unit 52.

  In step S183, the region estimation unit 161 executes processing for excluding points based on motion vectors whose accuracy is lower than a preset threshold value from the sample points estimated in step S182. The accuracy of the motion vector necessary for this processing is supplied from the motion estimation unit 52. Thereby, only the points estimated based on the motion vector with high accuracy are extracted from the sample points within the region estimation range.

  In step S184, the region estimation unit 161 extracts the full screen motion in the motion estimation result within the region estimation range. The full screen motion means a motion that takes the area corresponding to the same motion and maximizes the area. Specifically, a motion histogram is generated by assigning a weight proportional to the sample point interval at each sample point to the motion of each sample point, and one motion (one motion vector) having the maximum weighting frequency is generated. Extracted as full screen motion. When generating a histogram, for example, a representative value of motion can be prepared with pixel accuracy, and a motion having a value with one pixel accuracy can be added to the histogram.

  In step S185, the region estimation unit 161 extracts sample points within the region estimation range having the full screen motion as a result of region estimation. In this case, the sample points having the full screen motion include not only the sample points having the same motion as the full screen motion but also the difference in motion from the full screen motion is equal to or less than a predetermined threshold value set in advance. The sample point can also be a sample point having full screen motion here.

  In this way, among the sample points within the area estimation range determined by the processing of steps S143, S144, S206, and S208, the sample point having the full screen motion is estimated to belong to the same target as the tracking point. Is finally extracted (generated).

  Next, in step S186, the region estimation unit 161 executes a region estimation range update process. Thereafter, the processing proceeds to step S122 in FIG.

  FIG. 17 shows details of the region estimation range update processing in step S186 of FIG. In step S201, the region estimation unit 161 calculates the center of gravity of the region. This region means a region composed of sample points extracted in the process of step S185 in FIG. 15 (region composed of points estimated to belong to the same target as the tracking point). That is, one motion vector (full screen motion) corresponds to this area. For example, as shown in FIG. 18A, among sample points indicated by white squares in the drawing, sample points having full-screen motion in the process of step S185 in FIG. In FIG. 18A, sample points indicated by black squares are extracted, and an area constituted by the sample points is extracted (estimated) as an area 222. Then, the center of gravity 224 of the region 222 is further calculated. Specifically, the sample point centroid obtained by assigning the weight of the sample point interval to each sample point is obtained as the centroid of the region. This process has the meaning of obtaining the position of the region in the current frame.

  Next, in step S202, the region estimation unit 161 executes a process of shifting the center of gravity of the region by full screen motion. This process has the meaning of causing the area estimation range 221 to follow the movement of the position of the area and to move to the estimated position in the next frame. As shown in FIG. 18B, when the tracking point 223 in the current frame appears as the tracking point 233 in the next frame based on the motion vector 238, the full-screen motion vector 230 substantially corresponds to the tracking point motion vector 238. Therefore, the point 234 on the same frame (next frame) as the tracking point 233 is obtained by shifting the center of gravity 224 in the current frame based on the motion vector 230 (full screen motion). If the area estimation range 231 is set around the point 234, the area estimation range 221 is moved to the estimated position in the next frame by following the movement of the position of the area 222.

  In step S203, the region estimation unit 161 determines the size of the next region estimation range based on the region estimation result. Specifically, the sum of squares of sample point intervals (intervals of points indicated by black squares in the region 222 in FIG. 18A) for all sample points estimated as the region is regarded as the area of the region 222. The size of the region estimation range 231 in the next frame is determined so that the size is slightly larger than the area. That is, the size of the region estimation range 231 increases when the number of sample points in the region 222 is large and decreases when the number of sample points is small. In this way, not only can the enlargement / reduction of the area 222 be followed, but also the entire screen area within the area estimation range 221 can be prevented from becoming a peripheral area to be tracked.

  When the full screen motion extracted in step S184 in FIG. 15 matches the background motion, the background and the tracking target cannot be distinguished by the motion. Therefore, the background motion estimation unit 54 always performs background motion estimation processing. In step S204, the region estimation unit 161 extracts the background motion supplied from the background motion estimation unit 54 and the processing in step S184 of FIG. It is determined whether or not the full screen motion matches. If the full screen motion matches the background motion, in step S205, the region estimation unit 161 limits the size of the next region estimation range so that the current region estimation range is maximized. Thereby, it is suppressed that the background is erroneously recognized as the tracking target and the size of the area estimation range is enlarged.

  If it is determined in step S204 that the full screen motion and the background motion do not match, the processing in step S205 is not necessary and is skipped.

  Next, in step S <b> 206, the region estimation unit 161 determines the size of the next region estimation range around the shifted region center of gravity. As a result, the area estimation range is determined such that the center of gravity coincides with the already obtained area centroid after the shift, and the size thereof is proportional to the area width.

  In the example of FIG. 18B, the area estimation range 231 is determined to have a size corresponding to the area of the area 222 with the shifted center of gravity 234 based on the motion vector (full screen motion) 230 as the center.

  It is necessary to ensure (ensure) that the region having the full screen motion within the region estimation range 231 is the region of the tracking target (for example, the face 104 in FIG. 12). Therefore, in step S207, the region estimation unit 161 determines whether or not the tracking point is included in the next region estimation range. If the tracking point is not included, in step S208, the region estimation unit 161 determines that the tracking point is included in the next region estimation range. A process for shifting the region estimation range is executed. If the tracking point is included in the next area estimation range, the process of step S208 is not necessary and is skipped.

  As a specific method of shifting in this case, a method of minimizing the moving distance, a minimum distance at which the tracking point is included along a vector from the center of gravity of the region estimation range before the shift to the tracking point A way to move only is considered.

  In order to emphasize the robustness of tracking, a method of not performing a shift to include a tracking point in the region is also conceivable.

  In the example of FIG. 18C, since the region estimation range 231 does not include the tracking point 233, the region estimation range 241 is shifted to a position indicated as the region estimation range 241 (a position including the tracking point 233 at the upper left).

  18A to 18C show the case where the shift process of step S208 is necessary, but FIGS. 19A to 19C show the case where the shift process of step S208 is not necessary (the tracking point is the next region estimation range in step S207). Example).

  As shown in FIGS. 19A to 19C, when all the sample points in the area estimation range 221 are area points, the shift process in step S208 in FIG. 17 is not necessary.

  18A to 18C and FIGS. 19A to 19C show an example in which the area estimation range is rectangular, but the area estimation range may be circular.

  As described above, the area estimation range update process in FIG. 17 (step S186 in FIG. 15) determines the position and size of the area estimation range for the next frame to include the tracking point.

  In the update process of the area estimation range in FIG. 17, the area estimation range is a rectangular (or circular) fixed shape, but may be a variable shape.

  Next, the transfer candidate extraction process in step S162 of FIG. 14 will be described with reference to the flowchart of FIG.

  In step S261, the transfer candidate extraction unit 162 holds, as transfer candidates, the point shift results of the estimated motion corresponding to each of the points estimated as the full screen motion region. That is, instead of using the points obtained as region estimation results as they are, a process for extracting the results shifted based on the respective motion estimation results is performed for use in the next frame. The changed transfer candidates are supplied to and held in the transfer candidate holding unit 56.

  This process will be described with reference to FIG. That is, in the example of FIG. 12, the tracking point 101 exists in the frames n−1 and n, but in the frame n + 1, it is hidden by the ball 121 flying from the left side in the drawing, and the tracking point 101 does not exist. Therefore, in frame n + 1, it is necessary to change the tracking point to another point on the face 104 as a tracking target (for example, the left eye 103 (actually closer to the right eye 102)). Therefore, a transfer candidate is prepared in advance in a frame before transfer is actually required.

  Specifically, in the case of the example in FIG. 12, the motion estimation result in the region estimation range 133 from frame n to frame n + 1 needs to be changed in the region estimation range 133, so that there is a high probability that it cannot be estimated correctly. Is expected. That is, in the example of FIG. 12, the transfer occurs because the tracking point and a part of the same object are hidden. As a result, in the region estimation range 133 in the frame n, regarding the portion 134 where the target is hidden in the frame n + 1 (the shaded portion in FIG. 12) 134, the motion is not estimated correctly, and it is estimated that the accuracy of the motion is low. It is estimated that the accuracy is not low and a motion estimation result is meaningless.

  In such a case, there is a high possibility that the region estimation is erroneous due to a decrease in motion estimation results that can be used in region estimation or a mixture of erroneous motion estimation results. On the other hand, such a possibility is generally greater in the region estimation between the previous frame n−1 and the frame n in comparison with the estimation between the frame n and the frame n + 1. Expected to be lower.

  Therefore, in order to reduce the risk, the region estimation result is not used as it is, but the region estimation result obtained in the previous frame n-1 (or a frame earlier in time) is used as the movement destination in the next frame. It is desirable to use as a transfer candidate for improving the performance.

  However, the region estimation result can be used as it is.

  FIG. 21 shows details of the template creation processing in step S163 of FIG. In step S <b> 281, the template creation unit 163 determines a small region corresponding to each of the points estimated as the region (region of full screen motion). In the example of FIG. 22, the small area 322 is determined corresponding to the area point 321.

  In step S282, the template creation unit 163 sets the area of the sum of the small areas determined in the process of step S281 as the template range. In the example of FIG. 22, the sum area of the small areas 322 is a template range 331.

  In step S283, the template creation unit 163 creates a template from the template range information and image information set in step S282, supplies the template to the template holding unit 58, and holds the template. Specifically, pixel data in the template range 331 is used as a template.

  In FIG. 23, the small area 341 corresponding to the area point 321 has a larger area than the small area 322 in FIG. As a result, the template range 351 of the sum area of the small areas 341 is also wider than the template range 331 of FIG.

  It is conceivable that the size of the small region is proportional to the interval between the sample points, but the proportionality constant at that time can be determined so that the area is the square of the sample point interval, or larger or smaller than that. It is also possible.

  For example, a fixed size or shape range centered on the tracking point may be used as the template range without using the region estimation result.

  FIG. 24 shows the positional relationship between the template and the area estimation range. The template range 403 includes a tracking point 405. The upper left point of the circumscribed rectangle 401 circumscribing the template range 403 is a template reference point 404. The vector 406 from the template reference point 404 to the tracking point 405 and the vector 407 from the template reference point 404 to the upper left reference point 408 in the region estimation range 402 in the drawing are used as information of the template range 403. The template is composed of pixels included in the template range 403. The vectors 406 and 407 are used for returning to normal processing when the same image as the template is detected.

  In the above processing, unlike the case of the transfer candidate, the example in which both the range and the pixel correspond to the current frame is used as the template. However, as in the case of the transfer candidate, the destination in the next frame is set as the template. Can also be used.

  As described above, a template made up of pixel data including a tracking point is created in advance during normal processing in the same manner as a transfer candidate.

  Details of the exception processing in step S52 performed following the normal processing in step S51 of FIG. 6 described above will be described with reference to the flowchart of FIG. As described above, this process is performed when it is determined in step S124 in FIG. 10 that it is impossible to estimate the movement of the tracking point, and in step S128, it is determined that the transfer candidate for changing the tracking point cannot be selected. Will be executed.

  In step S401, the control unit 59 executes an exception process initialization process. Details of this processing are shown in the flowchart of FIG.

  In step S421, the control unit 59 causes a scene change when the tracking point cannot be tracked (when the movement of the tracking point cannot be estimated and a transfer candidate for changing the tracking point cannot be selected). It is determined whether it has been. The scene change detection unit 53 constantly monitors whether or not there is a scene chain based on the estimation result of the motion estimation unit 52, and the control unit 59 performs step S421 based on the detection result of the scene change detection unit 53. Execute the judgment. Specific processing of the scene change detection unit 53 will be described later with reference to FIGS. 38 and 39.

  If a scene change has occurred, it is presumed that the reason why tracking has become impossible is due to the occurrence of a scene change, and in step S422, the control unit 59 sets the mode to scene change. In contrast, if it is determined in step S421 that no scene change has occurred, the control unit 59 sets the mode to another mode in step S423.

  After the process of step S422 or step S423, the template matching unit 51 executes a process of selecting the oldest template in time in step S424. Specifically, as shown in FIG. 27, for example, when transition from frame n to frame n + 1 is performed, if exception processing is to be executed, a frame n−m + 1 is generated for frame n and is stored in the template holding unit 58. A template generated with respect to the frame mn + 1, which is the oldest template in time, is selected from the held m-frame templates.

  In this way, the template just before the transition to exception processing (the template generated for the frame n in the case of FIG. 27) is not used, but the template slightly before in time is selected for the occlusion to be tracked or the like. This is because when the transition to exception processing occurs, the tracking target is already considerably hidden immediately before the transition, and it is highly likely that the tracking target cannot be captured sufficiently large in the template at that time. Therefore, reliable tracking is possible by selecting a template in a frame slightly before in time.

  Next, in step S425, the template matching unit 51 executes processing for setting a template search range. For example, the template search range is set such that the position of the tracking point immediately before the transition to the exception process is the center of the template search range.

  That is, as shown in FIG. 28, when the right eye 102 of the subject's face 104 is designated as the tracking point 101 in the frame n, the ball 121 flies from the left direction in the figure, and the tracking point 101 in the frame n + 1. Is assumed, and the tracking point 101 appears again in the frame n + 2. In this case, a region centered on the tracking point 101 (included in the template range 411) is set as the template search range 412.

  In step S426, the template matching unit 51 resets the number of elapsed frames and the number of scene changes after shifting to exception processing to zero. The number of frames and the number of scene changes are used in a continuation determination process (steps S461, S463, S465, and S467 in FIG. 30) in step S405 in FIG.

  After the exception process initialization process is completed as described above, in step S402 in FIG. 25, the control unit 59 executes a process of waiting for the next frame. In step S403, the template matching unit 51 performs template matching processing within the template search range. In step S404, the template matching unit 51 determines whether it is possible to return to normal processing.

  Specifically, the absolute value sum of the difference between the template several frames before (the pixel in the template range 411 in FIG. 28) and the matching target pixel in the template search range is calculated by the template matching process. More specifically, the sum of absolute values of differences between the pixels in the predetermined block in the template range 411 and the predetermined block in the template search range is calculated. The position of the block is sequentially moved within the template range 411, and the sum of absolute values of the differences of the respective blocks is added to obtain a value at the position of the template. Then, the position where the sum of absolute values of the differences becomes the smallest when the template is sequentially moved within the template search range and its value are searched. In step S404, the absolute sum of the minimum differences is compared with a predetermined threshold value set in advance. If the sum of absolute values of the differences is less than or equal to the threshold value, an image including the tracking point (included in the template) has appeared again, so it is determined that the normal processing can be restored, and the processing Returns to the normal processing of step S51 of FIG.

  Then, as described above, in step S141 in FIG. 11, it is determined that the return from the exception processing is performed, and in step S144, the position where the difference absolute value sum is minimum is set as the matched position of the template, The tracking point and area estimation range are set based on the positional relationship between the template position and the tracking point area estimation range held corresponding to the template. That is, as described above with reference to FIG. 24, the region estimation range 402 is set based on the vectors 406 and 407 with the tracking point 405 as a reference.

  However, in the region estimation process in step S161 of FIG. 14, when a method that does not use the region estimation range is used, the region estimation range is not set.

  The determination as to whether or not it is possible to return to normal processing in step S404 in FIG. 25 is performed by comparing the value obtained by dividing the minimum sum of absolute differences by the activity of the template with a threshold value. Also good. As the activity in this case, the value calculated in step S603 in FIG. 32 by the activity calculation unit 602 in FIG.

  Alternatively, it is possible to return to the normal processing by comparing a value obtained by dividing the current minimum absolute difference sum by the minimum absolute difference sum one frame before with a predetermined threshold. It may be determined whether or not. In this case, it is not necessary to calculate the activity. That is, in step S404, the correlation between the template and the template search range is calculated, and determination is performed based on the comparison between the correlation value and the threshold value.

  If it is determined in step S404 that it is not possible to return to the normal process, the process proceeds to step S405, and the continuation determination process is executed. Details of the continuation determination process will be described later with reference to FIG. 30, and thereby, it is determined whether or not the exception process can be continued.

  In step S406, the control unit 59 sets whether exception processing (tracking of the tracking point in exception processing) can be continued based on the result of the continuation determination processing (steps S466 and S468 in FIG. 30 described later). (Based on the flag that was set). If the exception process can be continued, the process returns to step S402, and the subsequent processes are repeatedly executed. That is, the process of waiting until the tracking point appears again is repeatedly executed.

  On the other hand, if it is determined in step S406 that the exception processing cannot be continued (whether it is determined in step S465 in FIG. 30 described later that the number of frames that have elapsed after the tracking point has disappeared is equal to or greater than the threshold value THfr. (Or if it is determined in step S467 that the number of scene changes is equal to or greater than the threshold value THsc), the tracking process is terminated as the exception process is no longer possible. Instead of ending the tracking process, it may be possible to return to the normal process again using the tracking point that has been held. The exception handling in this case is shown in FIG. Note that the processing in steps S441 through S445 in FIG. 29 is the same as the processing in steps S401 through S405 in FIG.

  That is, when it is determined by the continuation determination process in step S445 whether or not the exception process can be continued, thereafter, in step S446, the control unit 59 performs the exception process (tracking the tracking point in the exception process). Is determined based on the result of the continuation determination process (based on a flag set in steps S466 and S468 in FIG. 30 described later). If the exception process can be continued, the process returns to step S442, and the subsequent processes are repeatedly executed. That is, the process of waiting until the tracking point appears again is repeatedly executed.

  On the other hand, if it is determined in step S446 that the exception processing cannot be continued (whether it is determined in step S465 in FIG. 30 described later that the number of elapsed frames after the tracking point disappears is equal to or greater than the threshold value THfr. Or, when it is determined in step S467 that the number of scene changes is equal to or greater than the threshold value THsc), the exception processing is no longer possible and the processing returns to the normal processing in step S1051 in FIG.

  In this case, as described above, it is determined in step S141 in FIG. 11 that the process has returned from the exception process, and in step S144, based on the position of the tracking point immediately before the transition to the exception process held. The tracking point and the area estimation range are set.

  FIG. 30 shows details of the continuation determination process in step S405 of FIG. 25 (or step S445 of FIG. 29). In step S461, the control unit 59 executes a process of adding 1 to the number of elapsed frames as a variable. The number of elapsed frames is previously reset to 0 in the exception process initialization process (step S426 in FIG. 26) in step S401 in FIG.

  Next, in step S462, the control unit 59 determines whether there is a scene change. Whether or not there is a scene change can be determined based on the detection result of the scene change detection unit 53 always executing the detection process. If there is a scene change, the process proceeds to step S463, and the control unit 59 adds 1 to the number of scene changes as a variable. The number of scene changes is also reset to 0 in the initialization process of step S426 in FIG. If no scene change has occurred during the transition from the normal process to the exception process, the process of step S463 is skipped.

  Next, in step S464, the control unit 59 determines whether or not the currently set mode is a scene change. This mode is set in steps S422 and S423 in FIG. When the currently set mode is a scene change, the process proceeds to step S467, and the control unit 59 determines whether or not the number of scene changes is smaller than a preset threshold value THsc. If the number of scene changes is smaller than the threshold value THsc, the process proceeds to step S466, and the control unit 59 sets a flag that allows continuation. If the number of scene changes is equal to or greater than the threshold value THsc, the process proceeds to step S468 and cannot be continued. Set the flag.

  On the other hand, when it is determined in step S464 that the mode is not a scene change (when it is determined that the mode is other), the process proceeds to step S465, and the control unit 59 determines whether or not the number of elapsed frames is smaller than the threshold value THfr. Determine whether. The number of elapsed frames is also reset to 0 in advance in step S426 of the exception process initialization process of FIG. If it is determined that the number of elapsed frames is smaller than the threshold value THfr, a continuation flag is set in step S466, and if it is determined that the number of elapsed frames is greater than or equal to the threshold value THfr, the process continues in step S468. A disabled flag is set.

  As described above, when the number of scene changes at the time of template matching processing is equal to or greater than the threshold value THsc, or when the number of elapsed frames is equal to or greater than the threshold value THfr, further exception processing is impossible.

  When the mode is other, a condition that the number of scene changes is 0 may be added to determine whether or not continuation is possible.

  In the above, it is assumed that the frame of the image is used as a processing unit and all frames are used. However, processing is not performed in units of fields or using all frames or fields, but is extracted by thinning out at a predetermined interval. It is also possible to use a modified frame or field.

  Further, in the above, the estimated movement destination of the point in the area is used as the transfer candidate. However, in this case, even if the whole screen motion is (0, 0), each point in the area is , (-1, 1), (1, 0), etc., the movement is shifted by the amount of each movement. Instead of using the destination point as a transfer candidate as it is, it is also possible to set the closest point among sample points obtained in advance as a transfer candidate. Of course, in order to reduce the processing load, each point may be shifted by the amount corresponding to the full screen movement.

  Further, instead of using the estimated destination of the point in the area as the transfer candidate, it is possible to use the point in the area as it is.

  Next, a configuration example of the motion estimation unit 52 in FIG. 3 will be described with reference to FIG. In this embodiment, the input image is supplied to the evaluation value calculation unit 601, the activity calculation unit 602, and the motion vector detection unit 606. The evaluation value calculation unit 601 calculates an evaluation value related to the degree of coincidence between the two objects associated by the motion vector and supplies the evaluation value to the normalization processing unit 604. The activity calculation unit 602 calculates the activity of the input image and supplies it to the threshold value determination unit 603 and the normalization processing unit 604. The motion vector detection unit 606 detects a motion vector from the input image and supplies it to the evaluation value calculation unit 601 and the integration processing unit 605.

  The normalization processing unit 604 normalizes the evaluation value supplied from the evaluation value calculation unit 601 based on the activity supplied from the activity calculation unit 602, and supplies the obtained value to the integration processing unit 605. The threshold determination unit 603 compares the activity supplied from the activity calculation unit 602 with a predetermined threshold and supplies the determination result to the integration processing unit 605. The integration processing unit 605 calculates the accuracy of the motion vector based on the normalization information supplied from the normalization processing unit 604 and the determination result supplied from the threshold value determination unit 603, and the obtained accuracy is detected by the motion vector detection. This is output together with the motion vector supplied from the unit 606.

  Next, the motion estimation process of the motion estimation unit 52 will be described with reference to the flowchart of FIG. Although the motion vector is obtained for a point, the accuracy is calculated using image data of a small block in the vicinity of two points associated by the motion vector, for example, centering on the point. In step S601, the motion vector detection unit 606 detects a motion vector from the input image. For this detection, for example, a block matching method or a gradient method is used. The detected motion vector is supplied to the evaluation value calculation unit 601 and the integration processing unit 605.

  In step S602, the evaluation value calculation unit 601 calculates an evaluation value. Specifically, for example, the sum of absolute differences of pixel values of two blocks centered on two points associated with motion vectors is calculated. That is, the motion vector V (vx, vy) detected by the motion vector detection unit 606 in step S601, the point P (Xp, Yp) on the image Fi of the previous frame based on the motion vector V (vx, vy), and the time later The relationship of the point Q (Xq, Yq) on the image Fj of the frame is expressed by the following equation.

  The evaluation value calculation unit 601 calculates an evaluation value Eval (P, Q, i, j) for a block centered on the point P and a block centered on the point Q based on the following equation.

  Each block is a square having 2L + 1 pixels on one side. The summation ΣΣ in the above equation is performed between corresponding pixels when x is from −L to L and y is from −L to L. Therefore, for example, when L = 2, nine differences are obtained, and the sum of the absolute values is calculated. The evaluation value indicates that the two blocks match well as the value approaches zero.

  The evaluation value calculation unit 601 supplies the generated evaluation value to the normalization processing unit 604.

  In step S603, the activity calculation unit 602 calculates an activity from the input image. The activity is a feature amount representing the complexity of the image. As shown in FIG. 33, the difference between the pixel of interest Y (x, y) and the adjacent 8 pixels Y (x + i, y + j) for each pixel is shown in FIG. The average value of the sum of absolute values is calculated based on the following formula as activity Activity (x, y) at the target pixel position.

  In the case of the example of FIG. 33, the value of the pixel of interest Y (x, y) located at the center among 3 × 3 pixels is 110, and the values of eight pixels adjacent to it are 80, 70, and 75, respectively. , 100, 100, 100, 80, 80, the activity Activity (x, y) is expressed by the following equation.

  Activity (x, y) = {| 80-110 | + | 70-110 | + | 75-110 | + | 100-110 | + | 100-110 | + | 80-110 | + | 80−110 |} /8=24.375.

  Similar processing is performed for all pixels in the frame.

  In order to calculate the motion vector accuracy in units of blocks, the sum of the activities of all the pixels in the block expressed by the following equation is defined as the activity (block activity) Blockactivity (i, j) of the block.

  In addition, the activity may be a variance value, a dynamic range, or the like.

  In step S604, the threshold determination unit 603 compares the block activity calculated by the activity calculation unit 602 with a predetermined threshold set in advance. Then, a flag indicating whether or not the input block activity is larger than the threshold value is output to the integration processing unit 605.

  Specifically, as a result of the experiment, the block activity and the evaluation value have the relationship shown in FIG. 34 using the motion vector as a parameter. In FIG. 34, the horizontal axis represents block activity Blockactivity (i, j), and the vertical axis represents the evaluation value Eval. When the motion is correctly detected (when the correct motion vector is given), the block activity and the value of the evaluation value are distributed in the region R1 below the curve 621 in the figure. On the other hand, if an incorrect motion (incorrect motion vector) is given, the block activity and the evaluation value are distributed in the region R2 on the left side of the figure from the curve 622 (the region above the curve 622). There is almost no distribution in the region other than R2 and the region other than the region R1 below the curve 621). Curves 621 and 622 intersect at point P. The value of the block activity at this point P is set as the threshold value THa. The threshold value THa means that if the value of the block activity is smaller than that, the corresponding motion vector may be incorrect (this will be described in detail later). The threshold determination unit 603 outputs a flag indicating whether the block activity value input from the activity calculation unit 602 is greater than the threshold THa to the integration processing block 605.

  In step S605, the normalization processing unit 604 executes normalization processing. Specifically, the normalization processing unit 604 calculates the motion vector accuracy VC according to the following equation.

  However, if the value of the motion vector accuracy VC is less than 0, the value is replaced with 0. Of the motion vector accuracy VC, the value obtained by dividing the evaluation value by the block activity is a straight line 623 whose position on the graph of FIG. Therefore, it indicates whether the region is in the lower region or the upper region in the figure. That is, if the slope of the straight line 623 is 1, and the value obtained by dividing the evaluation value by the block activity is larger than 1, the points corresponding to the value are points distributed in the area above the straight line 623. Means that. The motion vector accuracy VC obtained by subtracting this value from 1 means that the smaller the value, the higher the possibility that the corresponding points are distributed in the region R2.

  On the other hand, if the value obtained by dividing the evaluation value by the block activity is smaller than 1, it means that the points corresponding to the value are distributed in the lower area of the straight line 623 in the figure. The motion vector accuracy VC at that time means that the larger the value (closer to 0), the corresponding points are distributed in the region R1. The normalization processing unit 604 outputs the motion vector accuracy VC obtained by the calculation in this way to the integration processing unit 605.

  In step S606, the integration processing unit 605 executes integration processing. Details of this integration processing are shown in the flowchart of FIG.

  In step S631, the integration processing unit 605 determines whether the block activity is equal to or less than the threshold value THa. This determination is performed based on the flag supplied from the threshold determination unit 603. If the block activity is equal to or less than the threshold THa, the integration processing unit 605 sets the value of the motion vector accuracy VC calculated by the normalization processing unit 604 to 0 in step S632. If it is determined in step S631 that the activity value is greater than the threshold value THa, the processing in step S632 is skipped, and the value of the motion vector accuracy VC generated by the normalization processing unit 604 is output as it is together with the motion vector. Is done.

  This is because there is a possibility that a correct motion vector is not obtained if the block activity value is smaller than the threshold value THa even if the value of the motion vector accuracy VC calculated by the normalization processing unit 604 is positive. Because there is. That is, as shown in FIG. 34, between the origin O and the point P, the curve 622 protrudes below the curve 621 in the drawing (downward from the straight line 623). In a region R3 in which the value of the block activity is smaller than the threshold value Tha and surrounded by the curves 621 and 622, the value obtained by dividing the evaluation value by the block activity is distributed in both the regions R1 and R2. There is a high possibility that a correct motion vector is not obtained.

  Therefore, in such a distribution state, processing is performed assuming that the accuracy of the motion vector is low. For this reason, in step S632, the motion vector accuracy VC is set to 0 if the value is positive even if the value is positive. In this way, when the value of the motion vector accuracy VC is positive, it is possible to reliably represent that the correct motion vector is obtained. In addition, the larger the value of the motion vector accuracy VC, the higher the probability that a correct motion vector is obtained (the probability that the distribution is included in the region R1 increases).

  This coincides with an empirical rule that, in general, it is difficult to detect a motion vector with high reliability in a region where the luminance change is small (region where the activity is small).

  FIG. 36 illustrates a configuration example of the background motion estimation unit 54 of FIG. In this configuration example, the background motion estimator 54 includes a frequency distribution calculator 651 and a background motion determiner 652.

  The frequency distribution calculation unit 651 calculates a frequency distribution of motion vectors. However, this frequency is weighted so that a likely motion is weighted by using the motion vector accuracy VC supplied from the motion estimation unit 52. Based on the frequency distribution calculated by the frequency distribution calculation unit 651, the background motion determination unit 652 performs a process of determining a motion with the maximum frequency as a background motion, and outputs the background motion to the region estimation related processing unit 55.

  The background motion estimation process of the background motion estimation unit 54 will be described with reference to FIG.

  In step S651, the frequency distribution calculation unit 651 calculates a motion frequency distribution. Specifically, the frequency distribution calculation unit 651 assumes that the x and y coordinates of the motion vector as a background motion candidate are expressed in a range of ± 16 pixels from the reference point, respectively (1089 (= 16 × 2 + 1) × (16 × 2 + 1)), that is, a box for coordinates corresponding to possible values of a motion vector, and when a motion vector is generated, 1 is added to the coordinates corresponding to the motion vector. In this way, the motion vector frequency distribution can be calculated.

  However, when one motion vector is generated, if 1 is added, if the frequency of occurrence of a motion vector with low accuracy is high, a motion vector with low certainty may be determined as the background motion. . Therefore, when a motion vector is generated, the frequency distribution calculation unit 651 does not add a value 1 to a box (coordinates) corresponding to the motion vector, but a value obtained by multiplying the value 1 by the motion vector accuracy VC (= Motion vector accuracy VC). The value of the motion vector accuracy VC is normalized as a value between 0 and 1, and the closer the value is to 1, the higher the accuracy. Therefore, the frequency distribution obtained in this way is a frequency distribution obtained by weighting motion vectors based on their accuracy. This reduces the risk that a motion with low accuracy is determined as the background motion.

  Next, in step S652, the frequency distribution calculation unit 651 determines whether or not the process of calculating the motion frequency distribution has been completed for all blocks. If there is a block that has not yet been processed, the process returns to step S651, and the process of step S651 is executed for the next block.

  As described above, the motion frequency distribution calculation process is performed on the entire screen, and when it is determined in step S652 that the processing of all blocks has been completed, the process proceeds to step S653, and the background motion determination unit 652 A process for searching for the maximum value of the distribution is executed. That is, the background motion determination unit 652 selects the frequency having the maximum frequency from the frequencies calculated by the frequency distribution calculation unit 651, and determines the motion vector corresponding to the frequency as the motion vector of the background motion. The motion vector of the background motion is supplied to the region estimation related processing unit 55, and is used for, for example, a determination process of whether or not the full screen motion matches the background motion in step S204 of FIG.

  FIG. 38 shows a detailed configuration example of the scene change detection unit 53 of FIG. In this example, the scene change detection unit 53 is configured by the motion vector accuracy average calculation unit 671 and the threshold determination unit 672.

  The motion vector accuracy average calculation unit 671 calculates the average value of all the screens of the motion vector accuracy VC supplied from the motion estimation unit 52 and outputs the average value to the threshold determination unit 672. The threshold value determination unit 672 compares the average value supplied from the motion vector accuracy average calculation unit 671 with a predetermined threshold value, and determines whether it is a scene change based on the comparison result. The result is output to the control unit 59.

  Next, the operation of the scene change detection unit 53 will be described with reference to the flowchart of FIG. In step S681, the motion vector accuracy average calculation unit 671 calculates the sum of vector accuracy. Specifically, the motion vector accuracy average calculation unit 671 executes a process of adding the value of the motion vector accuracy VC calculated for each block output from the integration processing unit 605 of the motion estimation unit 52.

  In step S682, the motion vector accuracy average calculation unit 671 determines whether or not the processing for calculating the sum of the vector accuracy VC has been completed for all the blocks. If the processing has not yet been completed, the processing of step S681 is repeated. . By repeating this process, the sum of motion vector accuracy VC of each block for one screen is calculated. If it is determined in step S682 that the calculation process of the sum of motion vector accuracy VCs for all one screen has been completed, the process advances to step S683, and the motion vector accuracy average calculation unit 671 calculates the average value of the vector accuracy VC. Execute. Specifically, a value obtained by dividing the sum of the vector accuracy VC for one screen calculated in the process of step S681 by the number of added blocks is calculated as an average value.

  In step S684, the threshold value determination unit 672 compares the average value of the motion vector accuracy VC calculated by the motion vector accuracy average calculation unit 671 in the process of step S683 with a preset threshold value, and whether or not the threshold value is smaller than the threshold value. Determine whether. In general, when a scene change occurs between two frames having different times in a moving image, there is no corresponding image, so even if a motion vector is calculated, the motion vector is not certain.

  Therefore, if the average value of the vector accuracy VC is smaller than the threshold value, the threshold value determination unit 672 turns on the scene change flag in step S685. If the average value of the vector accuracy VC is not smaller than the threshold value (if it is greater than or equal to the threshold value), in step S586 Turn off the change flag. When the scene change flag is on, it indicates that there is a scene change, and when the scene change flag is off, it indicates that there is no scene change.

  The scene change flag is supplied to the control unit 59, and is used for determining whether or not there is a scene change in step S421 in FIG.

  As described above, by configuring the tracking processing unit 41 in FIG. 3, an object to be tracked is rotated (FIG. 7), occlusion occurs (FIG. 8), or a scene change point causes an object tracking point. Even when the image is temporarily not displayed (FIG. 9), the moving object (tracking point) in the image can be accurately tracked.

  Next, another detailed configuration example and the operation of the tracking processing unit 41 in FIG. 3 will be described. FIG. 40 is a block diagram illustrating a functional configuration example of the tracking processing unit 41 that performs tracking by the motion region center-of-gravity tracking method. In this example, the tracking processing unit 41 includes a motion vector detection unit 701, a frequency distribution calculation unit 702, a sample point extraction unit 703, a centroid calculation unit 704, and a tracking point update unit 705.

  An input image from the image processing unit 22 is input to the motion vector detection unit 701 and the sample point extraction unit 703. The motion vector detection unit 701 detects a motion vector in an area centered on the tracking point in the input image. The frequency distribution calculation unit 702 uses the motion vector detected by the motion vector detection unit 701 to calculate the frequency distribution of the motion vector in the region.

  Based on the motion vector frequency distribution, the sample point extraction unit 703 extracts a sample point that shows a movement similar to a movement that occupies a large number in an area centered on the tracking point in the input image, and extracts the sample point on the tracking target. This point. The centroid calculation unit 704 calculates the centroid of the sample point based on whether or not the point in the region is the sample point to be tracked extracted by the sample point extraction unit 703.

  The tracking point update unit 705 updates the tracking point by adding the motion with the maximum frequency to the center of gravity calculated by the center of gravity calculation unit 704, and uses the updated tracking point information as the tracking result as the presentation position calculation unit. Output to 42. At this time, the added motion with the maximum frequency is also output as a motion vector corresponding to the tracking result.

  Next, the operation of the tracking processing unit 41 in FIG. 40 will be described. FIG. 41 is a flowchart for explaining the details of the tracking point calculation process executed by the tracking processing unit 41 in step S21 of FIG.

  In step S701, the motion vector detection unit 701 waits for input of an image of the next frame, and in step S702, detects a motion vector in an area centered on the tracking point in the input image.

  That is, by capturing the next (after) frame (next frame) temporally from the frame including the tracking point (previous frame) in the process of step S701, two consecutive frames of images are obtained. .

  The motion vector detection unit 701 tracks the tracking point specified by the user in step S2 of FIG. 4, for example, the tracking target in the input image 711 of the previous frame input in time as shown in FIG. A region 721 centering on the tracking point P in the object 722 (for example, a person, an animal, or the like) is obtained, and the corresponding next sample point in the sampling interval (Sx, Sy) is obtained in the region 721. A motion vector is detected by estimating a sample point of the frame. The size of the area 721 is m * Sx × n * Sy (* represents multiplication), where m and n are the number of samples.

  In step S <b> 703, the frequency distribution calculation unit 702 calculates the frequency distribution of the motion vector in the region 721 using the motion vector detected by the motion vector detection unit 701.

  For example, if the motion candidates in the region 721 are Vx (horizontal motion: −16 ≦ Vx ≦ 16) and Vy (vertical motion: −16 ≦ Vy ≦ 16), 33 × 33 = 1089 box, that is, a motion vector Boxes for coordinates corresponding to possible values are prepared, and when a motion vector is generated, 1 is added to the coordinates corresponding to the motion vector. For example, when (Vx, Vy) = (2, 2) at a certain sample point, 1 is added to the box of (2, 2). By performing this for all the sample points in the region 721, the motion vector frequency distribution in the region 721 is calculated.

  In step S <b> 704, the sample point extraction unit 703 is similar to the motion occupying a large number in the region 721 centered on the tracking point P in the input image based on the frequency distribution of the motion vector calculated by the frequency distribution calculation unit 702. A sample point indicating the movement to be extracted is extracted and used as a point on the tracking target.

  That is, as shown in FIG. 43 by enlarging the area 721 in FIG. 42, the tracking point P, which is the position of the tracking target set so that the human object 722 in the input image 711 of the previous frame is set as the tracking target. Since the human object 722 has a high ratio in the center area 721, motion vectors (thick arrows) detected from sample points on the human object 722 occupy a large number.

  Therefore, the sample point extraction unit 703 extracts sample points indicating movement similar to the movement occupying a large number in the area 721 centered on the tracking point P, and sets it as a point on the tracking target.

  In step S705, the centroid calculation unit 704 determines the centroid of the sample point Sa (x, y) based on whether or not the point in the region 721 is the sample point to be tracked extracted by the sample point extraction unit 703. G (x, y) is calculated. This calculation formula is expressed by the following formula (6).

  Here, flag (i, j) (1 ≦ i ≦ m, 1 ≦ j ≦ n) is a flag indicating whether or not the sample point is a tracking target, and is 1 when it is a sample point. If it is not a sample point, it is 0.

  In step S706, the tracking point update unit 705 updates the tracking point by adding the motion with the highest frequency to the center of gravity G (x, y) calculated by the center of gravity calculating unit 704.

  Then, the updated tracking point information is output as a tracking result to the presentation position calculation unit 42 together with the corresponding motion vector (that is, the above-described maximum frequency motion), and the tracking process is terminated. Returning to step S21 in step 4, the position calculation process by the presentation position calculation unit 42 is then executed in step S22.

  Next, a detailed configuration example and operation of the presentation position calculation unit 42 of FIG. 3 will be described with reference to FIG. FIG. 44 is a block diagram illustrating a functional configuration example of the presentation position calculation unit 42. In this example, the presentation position calculation unit 42 includes a tracking target motion memory 801, a tracking target position memory 802, a presentation position memory 803, a weighting coefficient calculation unit 804, and a presentation position calculation unit 805.

  The tracking target motion memory 801 stores the tracking target motion vector obtained for each frame from the tracking processing unit 41. That is, the tracking target motion memory 801 includes the tracking target motion vector of the current frame input last in time and the tracking target motion vectors of a plurality of past frames input temporally before the current frame. It is remembered. The tracking target motion memory 801 also stores motion vectors of a plurality of past frames that have been filtered when the target of filtering by the presentation position calculation unit 805 is a motion vector.

  The tracking target position memory 802 stores information on the position of the tracking point (hereinafter also referred to as the tracking target position), which is the tracking result obtained for each frame from the tracking processing unit 41. That is, the tracking target position memory 802 stores the tracking target position of the current frame input last in time and the tracking target positions of a plurality of past frames input temporally before the current frame. .

  The presentation position memory 803 stores the presentation positions of a plurality of past frames that have been filtered as necessary.

  The weighting factor calculation unit 804 calculates a weighting factor used for filtering performed by the presentation position calculation unit 805, and supplies the calculated weighting factor to the presentation position calculation unit 805. In the example of FIG. 44, a fixed value (for example, 0.8) is used as the weighting coefficient.

  The presentation position calculation unit 805 performs filtering in the time direction on the information to be filtered among the information stored in the tracking target motion memory 801, the tracking target position memory 802, and the presentation position memory 803. The presentation position of the current frame is obtained, and the information is output to the zoom image creation unit 24. If necessary, the result of filtering is stored in the corresponding memory (any one of the tracking target motion memory 801 to the presentation position memory 803) and used for the calculation of the next processing unit.

  Next, details of the presentation position calculation process in step S22 of FIG. 4 will be described with reference to the flowchart of FIG. In the example of FIG. 45, a case where processing at time t is performed will be described.

  Here, the tracking target motion vector v (t) obtained by the tracking processing unit 41 estimated for two frames at time t and time t−1 (hereinafter referred to as motion vector v (t) at time t), The tracking target position x (t) at time t determined by the tracking processing unit 41 and the presentation position X (t) determined at time t. At this time, the relationship between the tracking target position x (t) and the tracking target motion vector v (t) is expressed by the following equation (7). That is, the tracking target position x (t) at time t is obtained by adding the motion vector v (t) at time t to the tracking target position x (t-1) at time t-1 immediately before time t. It is what was sought.

  In step S801, the control unit 29 initializes each memory. That is, 0 is set for all of the motion vector stored in the tracking target motion memory 801, the tracking target position stored in the tracking target position memory 802, and the presentation position stored in the presentation position memory 803.

  When the tracking point is determined by the tracking processing unit 41 in step S125 of FIG. 10, the tracking target position x (t), which is the tracking result at time t, is stored in the tracking target position memory 802, and the motion vector v ( t) is stored in the tracking target motion memory 801.

  In step S802, the presentation position calculation unit 805 reads the tracking target position x (t) at time t from the tracking target position memory 802, and uses the weighting coefficient for the tracking target position x (t) in the time direction. Perform filtering. When there is a presentation position obtained in the past, the past presentation position is read from the presentation position memory 803, and the tracking target position x (t) and the past presentation position are used with weighting factors. Time-direction filtering is performed.

  That is, the presentation position calculation unit 805 uses the weighting factor specified by the weighting factor calculation unit 804 for the tracking target position x (t) and the past presentation position (for example, the presentation position X (t−1)). To calculate the presentation position X (t) at time t. The presentation position X (t) at time t is obtained by the following equation (8).

  Here, · represents multiplication. Α0 is a weighting factor for the current frame (in this case, a frame at time t), and α1,..., Αn are weighting factors for past frames (for example, frames at time t−1,..., Time tn). It is. The same applies to other expressions thereafter. In Expression (8), α0 is used for weighting the tracking target position x (t) at time t. In Expression (8), the presentation position X at time t−1,..., Time tn. (t-1) ..., used for weighting the presentation position X (tn). In the first processing, since the past presentation position is initialized in step S801, X (t) = α0 · x (t) is calculated in step S802 from equation (8).

  In step S <b> 803, the presentation position calculation unit 805 stores the obtained presentation position X (t) in the presentation position memory 803 and outputs information on the presentation position X (t) to the zoom image creation unit 24.

  Thereafter, the process returns to step S802, and the process for the next frame at time t + 1 is executed. That is, the tracking processing unit 41 stores the tracking target position x (t + 1), which is the tracking result at time t + 1, in the tracking target position memory 802, and the target motion memory 801 stores time t + 1. A motion vector v (t + 1) at time t + 1 estimated for two frames at time t is stored.

  In step S802, the presentation position calculation unit 805 reads the tracking target position x (t + 1) at time t + 1 from the tracking target position memory 802, and the time obtained in the previous step S802 from the presentation position memory 803. The presentation position X (t) of t is read out, and weighting coefficients α0 and α1 (for example, α0 = 0.2, α1 = 0.8) are used for the tracking target position x (t + 1) and the presentation position X (t). A weighting calculation is performed to obtain the presentation position X (t + 1) at time t + 1. That is, in this case, X (t + 1) = α0 · x (t + 1) + α1 · X (t) is calculated in step S802 from Equation (8).

  In step S <b> 803, the presentation position calculation unit 805 stores the obtained presentation position X (t + 1) in the presentation position memory 803, and sends information about the presentation position X (t + 1) to the zoom image creation unit 24. The process returns to step S802, and the subsequent processing is repeated.

  Next, the presentation position calculation process, that is, the filtering process in the time direction will be described in detail with reference to FIG. In the example of FIG. 46, the configuration of the presentation position calculation unit 805 of FIG. 44 is shown. In this case, the presentation position calculation unit 805 includes a calculator 821, an adder 822, a delay unit 823, and a calculator 824. In the calculator 821 and the calculator 824, weighting factors α0 and α1 (α0 + α1 = 1) are set by the weighting factor calculator 804, respectively.

  When the tracking target position x (t) at time t is input by the tracking processing unit 41, the computing unit 821 multiplies the input tracking target position x (t) by the weighting coefficient α0, and the result (ie, α0 · x (t)) is output to the adder 822. At this time, the adder 822 is obtained by multiplying the presentation target position X (t−1) obtained at the time t−1 immediately before the time t by the arithmetic unit 824 by the weighting coefficient α1 (α1 · X (t-1)) is also input.

  The adder 822 adds the input from the calculator 821 and the input from the calculator 824, and the result (α 0 · x (t) + α 1 · X (t−1)) is given as the presentation target position X at time t. (t) is output to the zoom image creation unit 24 and the delay unit 823 in the subsequent stage.

  The delay unit 823 delays the presentation target position X (t) at time t by one frame and outputs the delayed result to the calculator 824. The computing unit 824 multiplies the presentation target position X (t) at time t by the weighting coefficient α1, and outputs the result (ie, α1 · X (t)) to the adder 822.

  At this time, α0 · x (t + 1) at the next time t + 1 is also input to the adder 822 from the calculator 821, so that the adder 822 is input from the calculator 821 and input from the calculator 824. And the result (α0 · x (t + 1) + α1 · X (t)) is set as the presentation target position X (t + 1) at time t + 1 and the zoom image creation unit 24 and the delay in the latter stage Output to the unit 823.

  As described above, the presentation position at time t is calculated by filtering the tracking target position in the time direction, that is, by calculating with weighting the tracking target position at time t and the presentation position at time t-1. Is done.

  Thereby, for example, even when the tracking target performs a motion that vibrates vertically and horizontally, such as a ski competition, or a sudden motion of acceleration, the motion can be displayed smoothly.

  The presentation position at time t-1 used for the calculation of the presentation position at time t is further calculated by weighting the tracking target position at time t-1 and the presentation position at time t-2. It is calculated by doing. That is, in the example of FIG. 46, the case where the presentation position calculation unit 805 is configured by an IIR (Infinite Impulse Response) filter is shown, and the presentation position at time t obtained by this is calculated from the start of calculation. The tracking target position in the past frame has been inherited.

  On the other hand, the presentation position calculation unit 805 can also be configured with a FIR (Finite Impulse Response) filter. By configuring the presentation position calculation unit 805 with an FIR filter, the presentation position at time t can be obtained without inheriting the tracking target position in the past frame from the calculation start. Although the illustration of the configuration of the presentation position calculation unit 805 in this case is omitted, the presentation position X (t) at time t is obtained by the following equation (9).

  In the above description, the case where the presentation position is calculated by filtering in the time direction for the tracking target position x (t) stored in the tracking target position memory 802 has been described. The presentation position can also be calculated by performing filtering in the time direction similar to the case of the tracking target position x (t) described above for the tracking target motion vector v (t) stored in 801.

  In this case, the presentation position calculation unit 805 reads the motion vector v (t) at time t from the tracking target motion memory 801, and performs filtering in the time direction on the motion vector v (t). That is, the presentation position calculation unit 805 performs a weighting operation on the motion vector v (t) using the weighting factor specified by the weighting factor calculation unit 804, and performs a motion vector V (t after filtering (weighting calculation). ) And the presentation position X (t) at time t is obtained using the motion vector V (t). When the presentation position calculation unit 805 is configured with an IIR filter, the presentation position X (t) at time t is obtained by the following equation (10).

  That is, the motion vector v (t) at time t is weighted by the weighting coefficient α0, and the motion vector V (t after filtering at time t−1,..., Time tn by the weighting coefficients α1,. -1),..., V (tn) are weighted and calculated to obtain the motion vector V (t) at time t, and further, the motion vector V (t) at time t and the time t-1 The presentation position X (t) at time t is obtained from the tracking target position x (t-1). The motion vector V (t) after filtering (weighting calculation) obtained at this time is stored in the tracking target motion memory 801.

  Referring to FIG. 45 again, for example, a case where the motion vector at time t + 1 is filtered in the time direction will be briefly described. That is, the tracking processing unit 41 stores the time t + in the tracking target position memory 802. The tracking target position x (t + 1) that is the tracking result of 1 is stored, and the tracking target motion memory 801 stores the motion vector v (at time t + 1 estimated for two frames at time t + 1 and time t. t + 1) is stored.

  In step S802, the presentation position calculation unit 805 extracts the motion vector v (t + 1) from the tracking target motion memory 801 and the motion vector V (t) filtered (weighted) in step S802 of the previous time (time t). , A weighting operation is performed on the motion vector v (t + 1) and the motion vector V (t) using weighting coefficients α0 and α1 (for example, α0 = 0.2, α1 = 0.8), and time t + 1 The motion vector V (t + 1) that has been filtered (weighted) is obtained. The presentation position calculation unit 805 reads the tracking target position x (t) at time t stored in the previous time (time t) from the tracking target position memory 802, and uses the motion vector V (t + 1). The presentation position X (t + 1) at time t + 1 is obtained.

  That is, in this case, V (t + 1) = α0 · v (t + 1) + α1 · V (t) is obtained in step S802 from equation (10), and X (t + 1) = x (t) + V (t + 1) is obtained. At this time, instead of the tracking target position x (t) at time t, the presentation position X (t) at time t obtained in step S802 of the previous time (time t) is read, and X (t + 1) = X (t) + V (t + 1) can also be obtained.

  As described above, the presentation position can be calculated by performing filtering in the time direction on the tracking target motion vector v (t) stored in the tracking target motion memory 801.

  As described above, also when the motion vector is filtered in the time direction, the presentation position calculation unit 805 can be configured by an FIR filter. By configuring the presentation position calculation unit 805 with an FIR filter, the presentation position at time t can be obtained without inheriting the movement (vector) of the tracking target in the past frame from the start of calculation. Although illustration in this case is omitted, the presentation position X (t) at time t is obtained by the following equation (11).

  In the above, when the filtering target is the tracking target position and the filter configuration is the IIR filter, and when the filtering target is the tracking target position and the filter configuration is the FIR filter, the filtering target is the tracking target. 4 types of variations have been described, when the filter configuration is an IIR filter, and the filtering target is a tracking target motion vector and the filter configuration is an FIR filter. Even if it uses, the effect similar to the case where the object of filtering mentioned above is a tracking object position and a filter structure is an IIR filter is mention | raise | lifted.

  That is, for example, even when the tracking target performs a motion that vibrates up and down, left and right, such as a ski competition, or a sudden motion of acceleration, those motions can be displayed smoothly.

  FIG. 47 is a block diagram illustrating another example of the functional configuration of the presentation position calculation unit 42. 47 differs from the presentation position calculation unit 42 in FIG. 44 in that the weight coefficient calculation unit 804 is replaced with the weight coefficient calculation unit 841, but the tracking target motion memory 801, the tracking target position memory 802, and the presentation The point which has the position memory 803 and the presentation position calculating part 805 is common.

  In the example of FIG. 47, the weighting coefficient calculation unit 841 monitors the tracking target motion vector of the current frame stored in the tracking target motion memory 801, and determines the weighting coefficient according to the magnitude of the motion vector of the current frame. A value is calculated, and the calculated weighting coefficient is supplied to the presentation position calculation unit 805.

  Next, details of the presentation position calculation process in step S22 of FIG. 4 will be described with reference to the flowchart of FIG. Note that the processing in steps S841, S845, and S846 in FIG. 48 is the same as the processing in steps S801 to S803 in FIG. 45, and thus detailed description thereof will be omitted as appropriate.

  In step S841, the control unit 29 initializes each memory. That is, 0 is set for all of the motion vector stored in the tracking target motion memory 801, the tracking target position stored in the tracking target position memory 802, and the presentation position stored in the presentation position memory 803.

  When the tracking point is determined by the tracking processing unit 41 in step S125 of FIG. 10, the tracking target position x (t), which is the tracking result at time t, is stored in the tracking target position memory 802, and the motion vector v ( t) is stored in the target motion memory 801.

  In step S842, the weight coefficient calculation unit 841 determines whether or not the motion vector v (t) at time t is smaller than the threshold, and determines that the motion vector v (t) at time t is larger than the threshold. In step S843, a weighting coefficient (that is, for the current frame) attached to the tracking target position x (t) at time t is obtained according to the absolute value of the motion vector v (t) at time t.

  This will be specifically described with reference to FIG. In the example of FIG. 49, the horizontal axis represents the absolute value of the motion vector v (t) at time t, the vertical axis represents the weighting coefficient α0 for the current frame (time t), and 5 is set as the threshold value. The case is shown. As shown in FIG. 49, when the absolute value of the motion vector v (t) at time t is 0, the weighting coefficient α0 for the current frame is 0.5, and the absolute value of the motion vector v (t) at time t is When 5, the product of the absolute value of the motion vector v (t) and 0.1 is added to 0.5 so that the weighting coefficient α0 for the current frame becomes 1, and the weighting coefficient α0 for the current frame is calculated. . The weight coefficient α1 for the past frame is calculated according to the weight coefficient α0 for the current frame.

  That is, when the motion vector is large, the weighting factor for the current frame is increased. The weighting coefficient α0 for the current frame and the weighting coefficient α1 for the past frame are calculated by the following equation (12).

  On the other hand, when it is determined that the motion vector v (t) at time t is smaller than the threshold value, the process proceeds to step S844, and the weighting coefficient calculation unit 841 sets 0 as the weighting coefficient for the current frame (ie, time t). α0 is set, and 1 is a weighting factor α1 for the past frame. That is, α0 = 0 and α1 = 1, and the tracking target position x (t) at time t is not reflected in the presentation position at time t.

  In step S845, the presentation position calculation unit 805 reads the tracking target position x (t) at time t from the tracking target position memory 802 and presents the presentation position X (t−t−t-1) obtained in the previous step S845. 1) is read out, and filtering in the time direction is performed on the tracking target position x (t) and the presentation position X (t−1). That is, the presentation position calculation unit 805 performs weighting calculation using the weighting coefficient calculated by the weighting coefficient calculation unit 841 in step S843 or S844, and obtains the presentation position X (t) at time t.

  In step S846, the presentation position calculation unit 805 stores the obtained presentation position X (t) at time t in the presentation position memory 803, and outputs information on the presentation position X (t) to the zoom image creation unit 24. To do.

  Thereafter, the process returns to step S842, and the process for the next frame at time t + 1 is executed.

  As described above, when the movement of the current frame is small, the tracking result of the current frame is not reflected in the presentation position. When the movement of the current frame is large, the tracking of the current frame is performed according to the size. By making use of the result at the presentation position, it is possible to more smoothly display the above-described tracking target, such as a skiing competition, such as a motion with vertical and horizontal vibrations or a sudden acceleration.

  As another example, for example, when the presentation position calculation process by weighting is performed, if the movement of the current frame is larger than a certain threshold value, the obtained tracking point is set as the presentation position as it is. Alternatively, the presentation position calculation process by weighting described above may be performed only when the motion of the current frame becomes smaller than a certain threshold value.

  FIG. 50 is a block diagram illustrating still another functional configuration example of the presentation position calculation unit 42. In the example of FIG. 50, the presentation position calculation unit 805 is different from the presentation position calculation unit 42 of FIG. 44 in that the presentation position calculation unit 805 is replaced with the presentation position calculation unit 862 and the movement determination unit 861 is added. It is common to have a target motion memory 801, a tracking target position memory 802, a presentation position memory 803, and a weighting coefficient calculation unit 804.

  In the example of FIG. 50, the motion determination unit 861 adds the motion vector of the tracking target of each frame stored in the tracking target motion memory 801, and stores the sum in a memory (not shown) containing the sum. The motion determination unit 861 determines whether or not the sum of motion vectors stored in the built-in memory is equal to or greater than a threshold value, and supplies the determination result to the presentation position calculation unit 862.

  If the motion determination unit 861 determines that the sum of the tracking target motion vectors of each frame is equal to or greater than the threshold, the presentation position calculation unit 862 follows the tracking target motion memory 801, the tracking target position memory 802, and the presentation position memory 803. The information to be filtered is subjected to filtering in the time direction with respect to the information to be filtered, and the present frame presentation position is obtained, and the information is output to the zoom image creation unit 24.

  On the other hand, when it is determined by the motion determination unit 861 that the sum of the motion vectors to be tracked for each frame is not equal to or greater than the threshold, it is assumed that the motion between the frames is vertical or horizontal vibration, and the presentation position calculation unit 862 The presentation position is not changed from the presentation position of the previous frame. That is, tracking on the display is not performed.

  Next, details of the presentation position calculation process in step S22 of FIG. 4 will be described with reference to the flowchart of FIG. Note that the processing in steps S861, S865, and S867 in FIG. 51 is the same as the processing in steps S801 to S803 in FIG. 45, and thus detailed description thereof is omitted as appropriate.

  In step S861, the control unit 29 initializes each memory. That is, 0 is set for all of the motion vector stored in the tracking target motion memory 801, the tracking target position stored in the tracking target position memory 802, and the presentation position stored in the presentation position memory 803. At this time, the sum sum of motion vectors stored in a memory built in the motion determination unit 861 is also set to zero.

  When the tracking point is determined by the tracking processing unit 41 in step S125 of FIG. 10, the tracking target position x (t), which is the tracking result at time t, is stored in the tracking target position memory 802, and the motion vector v ( t) is stored in the tracking target motion memory 801.

  In step S862, the motion determination unit 861 adds the motion vector v (t) at time t to the sum sum of motion vectors stored in the built-in memory. In step S863, the absolute value of the sum sum is greater than the threshold value. It is determined whether or not it is small. If it is determined in step S863 that the absolute value of the sum sum of motion vectors is greater than the threshold, the process proceeds to step S864, and the motion vector determination unit 861 resets the sum sum (to 0).

  In step S865, the presentation position calculation unit 862 reads the tracking target position x (t) at time t from the tracking target position memory 802 in accordance with the determination result of the motion vector determination unit 861, and is obtained in the previous step S865. The presentation position X (t-1) at time t-1 is read out, and the time direction filtering is performed on the tracking target position x (t) and the presentation position X (t-1). Perform time-direction filtering. That is, the presentation position calculation unit 862 performs a weighting operation on the tracking target position x (t) using the weighting factor specified by the weighting factor calculation unit 804, and obtains the presentation position X (t) at time t. .

  On the other hand, if it is determined in step S863 that the absolute value of the sum sum of motion vectors is smaller than the threshold, the process proceeds to step S866, and the presentation position calculation unit 862 responds to the determination result of the motion vector determination unit 861. The presentation position X (t-1) obtained in the previous frame (previous time t-1) is read from the presentation position memory 803, and the presentation position X (t-1) in the previous frame (previous time t-1) is read. The presentation position X (t) at time t.

  In step S867, the presentation position calculation unit 862 stores the obtained presentation position X (t) in the presentation position memory 803 and outputs information on the presentation position X (t) to the zoom image creation unit 24.

  Thereafter, the process returns to step S862, and the process for the next frame at time t + 1 is executed.

  Note that the sum sum of motion vectors described above may be calculated independently of the x direction and the y direction of the motion vector v (t), or may be a norm.

  As described above, when the movement of a plurality of frames is small, the tracking result of the current frame is not reflected in the presentation position, and the presentation position of the previous frame is set as the presentation position of the current frame. It is possible to more smoothly display the above-described tracking target, such as a ski competition, in which there is vibration in the vertical and horizontal directions and a sudden acceleration.

  In the example of FIGS. 48 and 51 described above, the case where the filtering target is the tracking target position and the filter configuration is the IIR filter has been described. However, the present invention is not limited thereto, and the filtering target is the tracking target position. If the filter configuration is an FIR filter, the filtering target is the tracking target motion vector, the filtering configuration is the IIR filter, and the filtering target is the tracking target motion vector, and the filter configuration The same processing can be performed using other variations in the case where is an FIR filter.

  FIG. 52 shows a coordinate locus of a tracking target presentation position in a conventional method and a zoom image according to the present invention. The horizontal axis represents frame numbers 1 to 100 in the time direction, and the vertical axis represents 85 to 120 coordinate values.

  In the example of FIG. 52, the dotted line represents the locus of the coordinates of the presentation position according to the conventional method with the tracking point as the presentation position in one dimension for convenience of explanation, and the solid line represents the locus of the coordinates of the presentation position according to the present invention. For convenience of explanation, it is shown in one dimension.

  Comparing the trajectory of the dotted line and the solid line, the trajectory of the method of the present invention shown by the solid line is smoother and reduced than that of the conventional method shown by the dotted line. I understand.

  That is, by generating an image based on the presentation position obtained by the present invention and displaying it on the image display 26, the unnatural movement of the background portion as seen in the prior art is reduced, and thus occurs. The viewer's discomfort can be suppressed.

  In the above description, as the tracking method of the tracking processing unit 41, the point tracking method with transfer described in Patent Document 1 will be described with reference to FIG. 5 described above, and the motion region center of gravity will be described with reference to FIG. Although the tracking method has been described, the tracking method is not limited to the above two types. For example, a luminance waveform block matching method, a color waveform block matching method, or a method of performing extrapolation for a certain period of time with past motions, etc. It can also be used.

  The color waveform block matching method performs the same processing as the luminance waveform block matching method except that the information used is a color instead of a luminance. In the luminance waveform block matching method, if there is a change in luminance in the tracking target, there is a high possibility that it will be excluded from the target, but in the color waveform block matching method, the luminance information is eliminated by using the color information excluding the luminance component. Even if there is a change, there is a high possibility that it has been tracked correctly.

  The method of performing extrapolation for a certain period of time with the past motion predicts the target motion based on the past motion of a certain tracking method among the tracking methods described above. For example, the tracking result tracked by the luminance waveform block matching method or the like may show a locus as shown in FIG. 53 when the tracking target is moved to the foreground by occlusion.

  In the example of FIG. 53, the trace of the tracking result from time t-5 to time t + 4 is shown. Tracking position x (t-5) to tracking position x (t + 4) represent the tracking results from time t-5 to time t + 4, respectively, and the arrows between the tracking positions indicate movements between the times. Represents. As shown in the tracking position x (t-5) to the tracking position x (t), the gentle movement is continuous from the time t-5 to the time t. The movement from the tracking position x (t) to the tracking position x (t + 1) has become discontinuous due to a cause such as the tracking target moving.

  Therefore, in such a case, as shown in FIG. 54, based on the past movement history indicated by the solid line between time t-5 and time t, extrapolation movement indicated by the alternate long and short dash line, extrapolation The extrapolated motion determined based on the past motion history from the determined timing (in this case, after the time t in this case) is the actually determined motion indicated by the dotted line (FIG. 53). In place of), the system continues to substitute for a certain time.

  As the extrapolation timing determined at this time, for example, after the user designates the tracking target, the extrapolation timing may be set at regular intervals (for example, every 120 frames), or by looking at the past movement history, Is discontinuous (for example, when it differs greatly from the average motion of several frames in the past).

  Further, the extrapolation motion determined at this time includes a motion several frames before the extrapolation timing, an average motion of past and several frames, and the like. Note that the previous movement may be possible even if it is not several frames before the extrapolation timing, but the previous movement may be a movement at the boundary of the object and may not be appropriate.

  Although the case of tracking using the luminance waveform block matching method has been described, the tracking method is not limited, and any tracking method may be used.

  As described above, the motion estimated for the object or the tracking result tracked based on the motion is filtered in the time direction to calculate the presentation position, with the calculated presentation position as a reference, Since the display image is generated, as described above with reference to the graph of FIG. 52, the user is less likely to feel uncomfortable with the image displayed on the image display 26.

  In the prior art, a zoom image with a large zoom ratio is more uncomfortable than that of an image created with a zoom ratio of 1, so the present invention is created with a zoom ratio of 1. This is more effective for a zoom image having a large zoom ratio than for an image.

  The present invention can be applied not only to a television receiver but also to a surveillance camera system and various image processing apparatuses.

  In the above, the image processing unit is a frame, but the present invention can also be applied to a case where a field is a processing unit.

  It does not matter whether the above-described series of processing is realized by hardware or software. When the above-described series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. It is installed on a general-purpose personal computer or the like from a recording medium such as a network or a removable medium.

  In addition, the steps of executing the series of processes described above in this specification are performed in parallel or individually even if they are not necessarily processed in time series, as well as processes performed in time series in the order described. The processing to be performed is also included.

It is a block diagram which shows the structural example of the television receiver to which this invention is applied. It is a flowchart explaining the image display process of the television receiver of FIG. It is a block diagram which shows the structural example of the object tracking part of FIG. It is a flowchart explaining a tracking process. It is a block diagram which shows the structural example of the tracking process part of FIG. It is a flowchart explaining a tracking point calculation process. It is a figure explaining tracking when a tracking object rotates. It is a figure explaining the tracking when an occlusion occurs. It is a figure explaining the tracking when a scene change occurs. It is a flowchart explaining a normal process. It is a flowchart explaining the initialization process of a normal process. It is a figure explaining a transfer candidate extraction process. It is a block diagram which shows the structural example of an area | region estimation related process part. It is a flowchart explaining an area | region estimation related process. It is a flowchart explaining area | region estimation processing. It is a figure explaining the process which determines a sample point. It is a flowchart explaining the update process of an area estimation range. It is a figure explaining the update of a region estimation range. It is a figure explaining the update of a region estimation range. It is a flowchart explaining a transfer candidate extraction process. It is a flowchart explaining a template creation process. It is a figure explaining template creation. It is a figure explaining template creation. It is a figure explaining the positional relationship of a template and a tracking point. It is a flowchart explaining exception processing. It is a flowchart explaining the initialization process of an exception process. It is a figure explaining selection of a template. It is a figure explaining the setting of a search range. It is a flowchart explaining the other example of exception processing. It is a flowchart explaining a continuation determination process. It is a block diagram which shows the structural example of a motion estimation part. It is a flowchart explaining a motion estimation process. It is a figure explaining calculation of activity. It is a figure explaining the relationship between an evaluation value and activity. It is a flowchart explaining an integration process. It is a block diagram which shows the structural example of a background motion estimation part. It is a flowchart explaining a background motion estimation process. 5 is a block diagram illustrating a configuration example of a scene change detection unit 53. FIG. It is a flowchart explaining a scene change detection process. It is a block diagram which shows the structural example of the tracking process part of FIG. It is a flowchart explaining a tracking process. It is a figure explaining the area | region which detects a motion vector. It is a figure explaining the frequency of a motion vector. It is a block diagram which shows the structural example of the presentation position calculation part of FIG. It is a flowchart explaining a presentation position calculation process. It is a block diagram which shows the structural example of the presentation position calculating part of FIG. It is a block diagram which shows the other structural example of the presentation position calculation part of FIG. It is a flowchart explaining a presentation position calculation process. It is a figure explaining the process of step S843 of FIG. It is a block diagram which shows the further another structural example of the presentation position calculation part of FIG. It is a flowchart explaining a presentation position calculation process. It is a figure explaining the locus | trajectory of a presentation position. It is a figure explaining the system which extrapolates for a fixed time with a past motion. It is a figure explaining the system which extrapolates for a fixed time with a past motion.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Television receiver, 23 Object tracking part, 24 Zoom image creation part, 26 Image display, 29 Control part, 30 Remote controller, 41 Tracking processing part, 42 Presentation position calculation part, 801 Tracking object motion memory, 802 Tracking object position Memory, 803 presentation position memory, 804 weight coefficient calculation unit, 805 presentation position calculation unit, 841 weight coefficient calculation unit, 861 motion determination unit, 862 presentation position calculation unit

Claims (7)

In an image processing apparatus for displaying a moving object,
Tracking means for tracking the movement of the object by estimating the movement of the object in the image;
A presentation position tracking unit that calculates a presentation position by performing filtering in the time direction on the tracking result tracked by the tracking unit or the estimated motion corresponding to the tracking result;
Display image creation means for creating a display image based on the presentation position calculated by the presentation position calculation means;
An image processing apparatus comprising: display control means for controlling display of the display image created by the display image creating means. The image processing apparatus according to claim 1, wherein the display image creation unit creates a zoomed display image with the presentation position calculated by the presentation position calculation unit as a center. The presentation position calculation means includes
Coefficient calculating means for calculating a coefficient used for filtering in the time direction according to the magnitude of the movement corresponding to the tracking result,
The presentation position is calculated by performing filtering in the time direction using the coefficient calculated by the coefficient calculation unit with respect to the tracking result or an estimated motion corresponding to the tracking result. The image processing apparatus according to 1. The presentation position calculation means includes
Determining means for determining whether a sum of movements corresponding to the tracking results of a plurality of processing units is smaller than a predetermined threshold;
When it is determined that the sum of motions corresponding to the tracking results of the plurality of processing units estimated by the determination unit is smaller than a predetermined threshold, the presentation position calculated in the previous processing unit in time is used as it is. The image processing apparatus according to claim 1, wherein a presentation position of a later processing unit is calculated. In an image processing method of an image processing apparatus for displaying a moving object,
By tracking the movement of the object by estimating the movement of the object in the image,
By performing filtering in the time direction on the tracked tracking result or the estimated motion corresponding to the tracking result, the presentation position is calculated,
Create a display image based on the calculated presentation position,
An image processing method including a step of controlling display of the created display image. A program for causing a computer to display a moving object,
By tracking the movement of the object by estimating the movement of the object in the image,
By performing filtering in the time direction on the tracked tracking result or the estimated motion corresponding to the tracking result, the presentation position is calculated,
Create a display image based on the calculated presentation position,
A program including a step of controlling display of the created display image.   A recording medium on which the program according to claim 6 is recorded.

Images