JP4826316B2 - Image processing apparatus and method, program, and recording medium - Google Patents

Description

  The present invention relates to an image processing apparatus and method, a program, and a recording medium, and in particular, an image that can reduce a sense of incongruity caused by a transfer when a tracking point is switched to another candidate and displayed. The present invention relates to a processing apparatus and method, a program, and a recording medium.

  Many techniques for tracking an object designated by a user in a moving image have been proposed. However, in the conventional technique, when the occlusion that the tracking target is temporarily hidden by another object occurs or the tracking target is temporarily not displayed due to a scene change or the like, robust tracking is performed. It was difficult.

  Therefore, a tracking point candidate is obtained in advance, and the tracking point is determined in advance when the tracking target rotates and the tracking point temporarily disappears, or when an occlusion or scene change occurs. This applicant proposes in Patent Document 1 filed earlier that the tracking target is more reliably tracked by changing to the above candidate.

JP 2005-303983 A

  However, in the object tracking described in Patent Document 1, the tracking result is displayed as it is. Therefore, when the tracking point is changed to a tracking candidate, there may be a sense of incongruity due to the change in the display of the tracking result. This sense of incongruity is particularly noticeable when the object tracking described in Patent Document 1 is applied to an application that zooms and displays the tracking point as a center.

  The present invention has been made in view of such a situation, and makes it possible to reduce a sense of incongruity caused by a transfer when the tracking point is switched to another candidate for tracking and display. .

An image processing apparatus according to an aspect of the present invention is an image processing apparatus that displays a moving object, and is temporally subsequent to the first point as the tracking point of the moving object on the image of the previous processing unit. A motion estimation means for estimating the movement to the position of the second point as the tracking point in the processing unit; and if the movement to the position of the second point in the subsequent processing unit cannot be estimated, the first Selecting means for selecting another first point from the estimated points as transfer candidates for the point, and the position of the second point estimated for the other first point selected by the selecting means Based on the movement, the determining means for determining the second point in the subsequent processing unit, and calculating the presentation position based on the position of the first point and the position of the other first point Presenting position calculating means and the presenting Includes a display image generating means for generating a display image based on the presentation position calculated by置算detecting means, and display control means for controlling display of the display image created by the display image generation unit, the presentation The position calculation means includes a comparison determination means for determining whether or not a transfer amount obtained from the position of the first point and the position of the other first point is a predetermined amount or more, and the comparison determination means by the comparison determination means. When it is determined that the transfer amount is greater than or equal to a predetermined amount, only the predetermined amount of the transfer amount is reflected, and the movement to the position of the second point estimated for the other first point And a presentation position setting means for setting the presentation position .

An image processing method according to one aspect of the present invention is an image processing method for an image processing apparatus that displays a moving object. A motion estimation step for estimating the movement to the position of the second point as the tracking point in the subsequent processing unit, and the movement to the position of the second point in the subsequent processing unit cannot be estimated A selection step of selecting another first point from the estimated points as transfer candidates for the first point, and the position of the second point estimated for the selected other first point based on the movement of the, calculates a determination step of determining a second point in the processing unit after said, the presentation position based on the position and the position of the other of the first point of the first point Presentation position calculation step A display image generation step of generating a display image based on the calculated presentation position, seen including a display control step for controlling the display of the display image generated by the processing of the presentation position calculation step, the It is determined whether or not the transfer amount obtained from the position of the first point and the position of the other first point is a predetermined amount or more, and when it is determined that the transfer amount is a predetermined amount or more, Only the predetermined amount of the transfer amount is reflected, and the presentation position is set based on the movement to the position of the second point estimated for the other first point.

According to another aspect of the present invention, there is provided a program according to an image processing method of an image processing apparatus for displaying a moving object. A motion estimation step for estimating the movement to the position of the second point as the tracking point in the subsequent processing unit, and the movement to the position of the second point in the subsequent processing unit cannot be estimated, A selection step of selecting another first point from the estimated points as transfer candidates for the first point, and the position of the second point estimated for the selected other first point A determination step for determining the second point in the subsequent processing unit based on movement, and a presentation position for calculating a presentation position based on the position of the first point and the position of the other first point a calculation step A display image generation step of generating a display image based on the calculated presentation position, seen including a display control step for controlling the display of the display image generated by the processing of the presentation position calculation step, the second It is determined whether or not the transfer amount obtained from the position of one point and the position of the other first point is greater than or equal to a predetermined amount, and when it is determined that the transfer amount is greater than or equal to a predetermined amount, Only the predetermined amount of the amount is reflected, and the presentation position is set based on the movement to the position of the second point estimated for the other first point.

A program recorded on a recording medium according to one aspect of the present invention is used as a tracking point of a moving object on an image of a previous processing unit in terms of time in an image processing method of an image processing apparatus that displays a moving object. A motion estimation step for estimating the movement of the first point to the position of the second point as the tracking point in the subsequent processing unit; and the movement to the position of the second point in the subsequent processing unit Is not estimable, a selection step of selecting another first point from the estimated points as transfer candidates of the first point, and the first point estimated for the selected other first point A determination step of determining the second point in the subsequent processing unit based on the movement of the second point to the position of the second point, and based on the position of the first point and the position of the other first point to calculate the present position A presentation position calculating step, a display image generation step of generating a display image based on the calculated presentation position, seen including a display control step for controlling the display of the display image created, the presentation position calculating step By determining whether or not the transfer amount obtained from the position of the first point and the position of the other first point is greater than or equal to a predetermined amount, the determination is that the change amount is greater than or equal to the predetermined amount. If it is, the presentation position is set based on the movement to the position of the second point estimated for the other first point, reflecting only the predetermined amount of the transfer amount.

In one aspect of the present invention, the position of the second point as the tracking point in the temporally subsequent processing unit of the first point as the tracking point of the moving object on the image of the temporally previous processing unit. If the movement to the position of the second point in the subsequent processing unit cannot be estimated, another first point is selected from the estimated points as transfer candidates for the first point. Is selected, and the second point in the subsequent processing unit is determined based on the movement to the position of the second point estimated for the selected other first point. Then, a presentation position is calculated based on the position of the first point and the position of the other first point, a display image is created based on the calculated presentation position, and the created display image The display of is controlled. Further, when calculating the presentation position, it is determined whether or not the transfer amount obtained from the position of the first point and the position of the other first point is equal to or greater than a predetermined amount, and the transfer amount is If it is determined that it is greater than or equal to a predetermined amount, only the predetermined amount of the transfer amount is reflected, and based on the movement to the position of the second point estimated for the other first point, The presentation position is set.

  According to the present invention, when the tracking point is switched to another candidate for tracking and display, the uncomfortable feeling caused by the switching can be reduced. Thereby, even when the tracking point is zoomed and displayed, it is possible to perform display in which the uncomfortable feeling due to the transfer is reduced.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows a configuration example when the present invention is applied to a television receiver. The television receiver 1 includes a tuner 21, an image processing unit 22, an object tracking unit 23, a zoom image creation unit 24, a selection unit 25, an image display 26, an audio processing unit 27, a speaker 28, a control unit 29, and a remote controller 30. , And a removable medium 31.

  The tuner 21 receives the RF signal, demodulates it and separates it into an image signal and an audio signal, outputs the image signal to the image processing unit 22, and outputs the audio signal to the audio processing unit 27.

  The image processing unit 22 demodulates the image signal input from the tuner 21 and outputs the demodulated signal to the object tracking unit 23, the zoom image creation unit 24, and the selection unit 25. The object tracking unit 23 performs processing for tracking the tracking point of the object specified by the user from the input image, obtains a presentation position as a reference for presenting the tracked object based on the tracking point, The coordinate information regarding the position is output to the zoom image creation unit 24. The zoom image creation unit 24 creates a zoom image (that is, a zoomed display image) based on the presentation position. The selection unit 25 selects one of the image supplied from the image processing unit 22 or the image supplied from the zoom image creation unit 24 based on an instruction from the user, and outputs the selected image to the image display 26 for display.

  The audio processing unit 27 demodulates the audio signal input from the tuner 21 and outputs it to the speaker 28.

  The control unit 29 is configured by, for example, a microcomputer and controls each unit based on a user instruction. The remote controller 30 is operated by the user and outputs a signal corresponding to the operation to the control unit 29. The removable medium 31 includes a semiconductor memory, a magnetic disk, an optical disk, a magneto-optical disk, and the like. The removable medium 31 is mounted as necessary, and provides the control unit 29 with a program and various other data.

  Next, processing of the television receiver 1 will be described with reference to the flowchart of FIG.

  In step S1, the tuner 21 demodulates the signal of the channel designated by the user from the RF signal received via the antenna (not shown), outputs the image signal to the image processing unit 22, and processes the sound signal as sound processing. To the unit 27. The audio signal is demodulated by the audio processing unit 27 and then output from the speaker 28.

  The image processing unit 22 demodulates the input image signal and outputs the demodulated image signal to the object tracking unit 23, zoom image creation unit 24, and selection unit 25.

  In step S2, the object tracking unit 23 determines whether or not tracking is instructed. If it is determined that tracking is not instructed, the processing of steps S3 and S4 is skipped. In step S <b> 5, the selection unit 25 selects one of the image signal supplied from the image processing unit 22 and the image signal input from the zoom image creation unit 24 based on control from the control unit 29. In this case, since there is no specific instruction from the user, the control unit 29 causes the selection unit 25 to select the image signal from the image processing unit 22. In step S <b> 6, the image display 26 displays the image selected by the selection unit 25.

  In step S7, the control unit 29 determines whether or not to end the image display process based on a user instruction. That is, when ending the image display process, the user operates the remote controller 30 and instructs the control unit 29 to do so. If the user has not instructed termination, the process returns to step S1, and the subsequent processes are repeatedly executed.

  As described above, normal processing for displaying an image corresponding to the signal received by the tuner 21 as it is is executed.

  When an image desired to be tracked is displayed by looking at the image displayed on the image display 26, the user designates the image by operating the remote controller 30. When this operation is performed, in step S2, the control unit 29 determines that tracking is instructed, and controls the object tracking unit 23. Based on this control, the object tracking unit 23 starts tracking processing of the tracking point designated by the user.

  Details of the tracking process will be described later with reference to FIG. 4. By this process, the tracking point designated by the user, for example, an object to be tracked in the image (for example, a person, an animal, etc.) The tracking position (eye, center of head, etc.) is tracked, and based on the tracking point, when presenting the tracked object, the presentation position serving as the presentation reference is calculated, and the position representing the presentation position Information is output to the zoom image creation unit 24 as presentation position information.

  In step S <b> 4, the zoom image creation unit 24 generates a zoom image based on the presentation position obtained by the object tracking unit 23 and outputs the zoom image to the selection unit 25. That is, the zoom image creating unit 24 generates a zoom image with the presentation position obtained by the object tracking unit 23 as the center. That is, the presentation position can also be said to be a zoom presentation center point.

  This zoom image creation processing can be performed using the class classification adaptation processing previously proposed by the present applicant. For example, Japanese Patent Laid-Open No. 2002-196737 discloses a process for converting a 525i signal into a 1080i signal using a coefficient obtained by learning in advance. This process is substantially the same as the process of enlarging the image 9/4 times in both the vertical direction and the horizontal direction. However, since the image display 26 has a fixed number of pixels, the zoom image creation unit 24 converts the 525i signal into a 1080i signal and then centers the presentation position when creating a 9 / 4-fold image, for example. A zoom image can be created by selecting a predetermined number of pixels (number of pixels corresponding to the image display 26).

  Based on this principle, a zoom image with an arbitrary magnification can be generated.

  If tracking is instructed, the selection unit 25 selects the zoom image created by the zoom image creation unit 24 in step S5. As a result, in step S6, the image display 26 displays the zoom image created by the zoom image creation unit 24.

  As described above, the image display 26 displays a zoom image centered on the presentation position obtained based on the tracking point designated by the user. When the magnification is set to 1, a display image without zooming is created and displayed based on the presentation position.

  Next, a detailed configuration example and operation of the object tracking unit 23 in FIG. 1 will be described. FIG. 3 is a block diagram illustrating a functional configuration example of the object tracking unit 23. In this example, the object tracking unit 23 includes a tracking processing unit 41 and a presentation position calculation unit 42.

  The tracking processing unit 41 estimates the motion of the input image and calculates a tracking point based on the motion vector obtained as a result of the estimation. Then, the tracking processing unit 41 supplies the tracking result such as the calculated position information of the tracking point and the motion vector corresponding to the tracking result obtained as a result of the estimation to the presentation position calculating unit 42.

  The presentation position calculation unit 42 calculates a presentation position serving as a reference when presenting the tracking of the object (that is, the tracking result) based on the tracking result and the motion vector supplied from the tracking processing unit 41, and the calculated presentation Information about the position is output to the zoom image creation unit 24, the control unit 29, and the like.

  Next, the operation of the object tracking unit 23 will be described. FIG. 4 is a flowchart for explaining the details of the tracking process executed by the object tracking unit 23 in step S3 of FIG.

  First, in step S21, the tracking processing unit 41 of the object tracking unit 23 executes tracking point calculation processing for calculating a tracking point by tracking the tracking point specified in step S2. This tracking point calculation process basically includes a normal process and an exception process, as shown in FIG.

  That is, the tracking processing unit 41 first executes normal processing in step S51. The details of this normal process will be described later with reference to FIG. 10, but a process for tracking a tracking point designated by the user is executed by this process. When the tracking point cannot be changed in the normal processing in step S51, the tracking processing unit 41 executes exception processing in step S52. Details of the exception processing will be described later with reference to FIG. 25. When the tracking point becomes invisible from the image by this exception processing, the return processing to the normal processing is executed by template matching.

  If it is determined that the tracking process cannot be continued due to the exception process (cannot return to the normal process), the process is terminated. If it is determined that the return is possible, the process returns to step S51 again. In this way, the normal process in step S51 and the exception process in step S52 are sequentially repeated for each frame.

  With this normal processing and exception processing, in the present invention, as shown in FIGS. 6 to 8, the tracking point is temporarily changed such that the tracking target is rotated, occlusion occurs, a scene change occurs, and the like. Even when it becomes invisible, tracking is possible.

  That is, for example, as shown in FIG. 6, a human face 104 as a tracking target (object) is displayed in the frame n−1, and the human face 104 has a right eye 102 and a left eye 103. ing. It is assumed that the user designates, for example, the right eye 102 (exactly one pixel therein) as the tracking point 101. In the example of FIG. 6, in the next frame n, the person moves to the left in the figure, and in the next frame n + 1, the person's face 104 rotates clockwise. As a result, the right eye 102 that has been visible until now is not displayed, and tracking cannot be performed with the conventional method. Therefore, in the normal process of step S51 described above, the left eye 103 on the face 104 as the same object as the right eye 102 is selected, and the tracking point is switched (set) to the left eye 103. This enables tracking.

  In the display example of FIG. 7, in the frame n−1, the ball 121 has moved from the left side of the face 104 in the drawing, and in the next frame n, the ball 121 has just covered the face 104. In this state, the face 104 including the right eye 102 designated as the tracking point 101 is not displayed. When such an occlusion occurs, the face 104 as the object is not displayed, so there is no transfer point to be tracked instead of the tracking point 101, and thereafter it becomes difficult to track the tracking point. However, in the present invention, the image of the frame n−1 (actually the earlier frame in time) is stored in advance as a template for the right eye 102 as the tracking point 101, and the ball 121 moves further to the right. When the right eye 102 designated as the tracking point 101 appears again in frame n + 1, it is confirmed that the right eye 102 as the tracking point 101 is displayed again by the exception processing in step S52 described above, and the right eye 102 is again tracked. The point 101 is tracked.

  In the example of FIG. 8, the face 104 is displayed in the frame n−1, but in the next frame n, the automobile 111 covers and covers the whole including the human face. That is, in this case, a scene change has occurred. In the present invention, even if the scene change occurs and the tracking point 101 no longer exists in the image, if the automobile 111 moves and the right eye 102 is displayed again in the frame n + 1, the exception processing in step S52 It is confirmed based on the template that the right eye 102 as the tracking point 101 has appeared again, and the right eye 102 can be tracked again as the tracking point 101.

  Returning to FIG. 4, as described above, in the tracking point calculation process in step S <b> 21, the tracking point designated by the user or the point to be switched is changed until the tracking process cannot be continued by the exception process. The tracking point resulting from the transfer to the transfer point is determined. At this time, information on the determined tracking point (for example, tracking point position information and transfer history information) and a motion vector corresponding to the tracking point are supplied to the presentation position calculation unit 42.

  In step S22, the presentation position calculation unit 42 further provides a reference position when presenting the tracking of the object (that is, the tracking result) based on the information regarding the tracking point determined in step S21. When the zoom image creation unit 24 creates a zoom image, a presentation position calculation process is performed to calculate a presentation position as the center.

  The details of the presentation position calculation process will be described later with reference to FIG. 41, but the current tracking point obtained in step S21 is changed by this process, and the transfer is performed from the position of the past tracking point. If the transfer vector, which is the amount of movement to the point position, exceeds the predetermined amount of movement, only the predetermined amount of movement is reflected in the past presentation position, and the current presentation position is Calculated. In this case, the remaining motion amount that has not been reflected (hereinafter also referred to as a transfer remaining vector) is reflected in the processing of the next frame.

  On the other hand, if the current tracking point obtained in step S21 is not changed and there is no remaining transfer vector from the previous frame, the current tracking point obtained in step S21 is the current presentation point. Set as position.

  By this presentation position calculation process, a presentation position serving as a reference when presenting tracking of an object (that is, a tracking result), that is, a presentation position serving as a zoom center when creating a zoom image is obtained. In, a sense of incongruity that has conventionally occurred in the displayed image can be reduced by changing the tracking point.

  Next, a detailed configuration example and operation of the tracking processing unit 41 in FIG. 3 will be described. FIG. 9 is a block diagram illustrating a functional configuration example of the tracking processing unit 41. In this example, the tracking processing unit 41 includes a template matching unit 51, a motion estimation unit 52, a scene change detection unit 53, a background motion estimation unit 54, a region estimation related processing unit 55, a transfer candidate holding unit 56, and a tracking point determination unit 57. , A template holding unit 58 and a control unit 59.

  The template matching unit 51 performs a matching process between the input image and the template image held in the template holding unit 58. The motion estimation unit 52 estimates the motion of the input image, and obtains the motion vector obtained as a result of the estimation and the accuracy of the motion vector, the scene change detection unit 53, the background motion estimation unit 54, the region estimation related processing unit 55, And output to the tracking point determination unit 57. The scene change detection unit 53 detects a scene change based on the accuracy supplied from the motion estimation unit 52.

  The background motion estimation unit 54 executes processing for estimating the background motion based on the motion vector and the accuracy supplied from the motion estimation unit 52, and supplies the estimation result to the region estimation related processing unit 55. The region estimation related processing unit 55 is based on the motion vector and accuracy supplied from the motion estimation unit 52, the background motion supplied from the background motion estimation unit 54, and the tracking point information supplied from the tracking point determination unit 57. Perform region estimation processing. Further, the region estimation related processing unit 55 generates a transfer candidate based on the input information, supplies the transfer candidate to the transfer candidate holding unit 56, and holds it. Further, the region estimation related processing unit 55 creates a template based on the input image, supplies the template to the template holding unit 58, and holds it.

  The tracking point determination unit 57 determines a tracking point based on the motion vector and accuracy supplied from the motion estimation unit 52 and the transfer candidate supplied from the transfer candidate holding unit 56, and information on the determined tracking point is obtained. The result is output to the region estimation related processing unit 55.

  The control unit 59 controls each part of the template matching unit 51 to the template holding unit 58 based on the tracking point information specified by the control unit 29 to track the specified tracking target, and the tracking point screen. The tracking result such as the position information above and the motion vector corresponding to the determined tracking point are output to the presentation position calculation unit 42 and the like. That is, the tracking result such as information on the tracking point determined by the tracking point determination unit 57 and the motion vector from the motion estimation unit 52 corresponding to the determined tracking point are supplied to the presentation position calculation unit 42.

  Next, the details of the normal processing in step S51 in FIG. 5 will be described with reference to the flowchart in FIG. In step S121, the tracking point determination unit 57 executes initialization processing of normal processing. The details will be described later with reference to FIG. 11, but an area estimation range based on the tracking point designated to be tracked by the user is designated by this processing. This area estimation range is the same object as the tracking point specified by the user (for example, when the tracking point is a human eye, a human face as a rigid body that moves like the eye, a human body, or the like) ) Is a range to be referred to when estimating the range of points belonging to. A transfer point is selected from points in the region estimation range.

  Next, in step S122, the control unit 59 controls each unit to wait for input of an image of the next frame. In step S123, the motion estimation unit 52 estimates the tracking point motion. That is, by capturing the frame (following frame) temporally after the frame including the tracking point designated by the user (previous frame) in the process of step S122, an image of two consecutive frames is eventually obtained. Therefore, in step S123, the movement of the tracking point is estimated by estimating the position of the tracking point of the subsequent frame corresponding to the tracking point of the previous frame.

  Note that “preceding in time” means the processing order (input order). Normally, images of each frame are input in the order of imaging. In this case, the frame captured earlier in time becomes the previous frame, but the frame captured later in time is processed (input) first. ), The frame imaged later in time becomes the previous frame.

  In step S124, the motion estimation unit 52 determines whether the tracking point can be estimated as a result of the process in step S123. Whether or not the tracking point can be estimated is determined, for example, by comparing the accuracy value of a motion vector (described later) generated and output by the motion estimation unit 52 with a preset threshold value. Specifically, it is possible to estimate if the accuracy of the motion vector is equal to or greater than a threshold, and it is determined that estimation is impossible if the accuracy is smaller than the threshold. In other words, the possibility here is determined relatively strictly, and if the estimation is not actually impossible but the accuracy is low, it is determined as impossible. As a result, more reliable tracking processing can be performed.

  In step S124, it is determined that the motion estimation result at the tracking point and the motion estimation result at a point near the tracking point can be estimated if they match the motions that occupy the majority, and that they cannot be estimated if they do not match. It is also possible to do so.

  When it is determined that the movement of the tracking point can be estimated (the probability that the tracking point is correctly set on the corresponding point on the same object (if the right eye 102 is designated as the tracking point 101, the right eye 102 If the probability of being correctly tracked is relatively high), the process proceeds to step S125, and the tracking point determination unit 57 shifts the tracking point by the estimated motion (motion vector) obtained in the process of step S123. In other words, this determines the tracking position in the subsequent frame after tracking the tracking point of the previous frame.

  Then, information on the determined tracking point (for example, tracking point position information, transfer history information, etc.) is output to the presentation position calculation unit 42, and in step S22 of FIG. 4 (FIGS. 41 and 42 described later). Used in the presentation position calculation process.

  After step S125, region estimation related processing is executed in step S126. The details of the area estimation related process will be described later with reference to FIG. 14. By this process, the area estimation range designated in the initialization process of the normal process in step S121 is updated. Furthermore, if the tracking point is not displayed because the target object rotates, for example, the candidate as a switching point (transfer candidate) as a point to be switched to is the state (can still be tracked). In the state), it is extracted (created) in advance. Further, when the transfer to the transfer candidate cannot be performed, the tracking is temporarily interrupted, but a template is created in advance in order to confirm that the tracking is possible again (the tracking point appears again).

  After the region estimation related process in step S126 is completed, the process returns to step S122 again, and the subsequent processes are repeatedly executed.

  That is, as long as the movement of the tracking point designated by the user can be estimated, the processing from step S122 to step S126 is repeatedly executed for each frame, and tracking is performed.

  On the other hand, if it is determined in step S124 that the movement of the tracking point cannot be estimated (that is, impossible), that is, as described above, for example, the accuracy of the motion vector is equal to or less than the threshold value. The process proceeds to step S127. In step S127, the tracking point determination unit 57 holds the transfer candidate generated in the region estimation-related processing in step S126 in the transfer candidate holding unit 56, and from among these, the transfer candidate closest to the original tracking point is stored. Select one. The tracking point determination unit 57 determines whether or not a transfer candidate has been selected in step S128. If a transfer candidate has been selected, the tracking point determination unit 57 proceeds to step S129 and sets the tracking point as the transfer candidate selected in the process of step S127. Change (change). That is, the transfer candidate point is set as a new tracking point. At this time, for example, transfer history information such as a transfer flag indicating that transfer has been performed is stored. Thereafter, the process returns to step S123, and the process of estimating the movement of the tracking point selected from the transfer candidates is executed.

In step S124, it is determined again whether or not the movement of the newly set tracking point can be estimated. If it can be estimated, a process of shifting the tracking point by the estimated movement is performed in step S125, and step S126. In, the area estimation related process is executed. Thereafter, the process returns to step S122 again, and the subsequent processes are repeatedly executed. In addition,
The transfer history information stored in step S129 is reset when returning to step S122, for example.

  If it is determined in step S124 that the newly set tracking point cannot be estimated, the process returns to step S127, and the next transfer candidate closest to the original tracking point is selected from the transfer candidates. In step S129, the transfer candidate is set as a new tracking point. The process after step S123 is repeated again for the new tracking point.

  Even when all the prepared transfer candidates are used as new tracking points, if the movement of the tracking point cannot be estimated, it is determined in step S128 that the transfer candidate cannot be selected, and this normal process is performed. Is terminated. Then, the process proceeds to the exception process in step S52 of FIG.

  Next, details of the initialization process of the normal process in step S121 of FIG. 10 will be described with reference to the flowchart of FIG.

  In step S141, the control unit 59 determines whether or not the current process is a process for returning from the exception process. That is, after the exception process in step S52 of FIG. 5 is terminated, it is determined whether or not the process has returned to the normal process in step S51. In the process of the first frame, since the exception process of step S52 has not been executed yet, it is determined that the process has not returned from the exception process, and the process proceeds to step S142. In step S142, the tracking point determination unit 57 performs a process of setting the tracking point to the position of the tracking point instruction that is a predetermined point in the input image designated by the user.

  In addition, when there is no designation | designated by a user, the point with the highest brightness | luminance can also be set as a tracking point in an input image, for example. The tracking point determination unit 57 supplies the set tracking point information to the region estimation related processing unit 55.

  In step S143, the region estimation related processing unit 55 sets a region estimation range based on the position of the tracking point set in the processing of step S142. This region estimation range is a reference range when estimating a point on the same rigid body as the tracking point, and more specifically, tracking so that the same rigid body portion as the tracking point occupies most of the region estimation range in advance. By setting the position and size of the estimated area range to follow the same rigid body part as the point, the part that shows the most movement in the estimated area range can be estimated as the same rigid body part as the tracking point It is for doing so. In step S143, as a default value, for example, a predetermined range centered on the tracking point is set as the region estimation range.

  Thereafter, the processing proceeds to step S122 in FIG.

  On the other hand, if it is determined in step S141 that the current process is a process for returning from the exception process in step S52 of FIG. 5, the process proceeds to step S144, and the tracking point determination unit 57 refers to FIG. Then, the tracking point and the area estimation range are set based on the position matched with the template by the process described later. For example, a point on the current frame that matches the tracking point on the template is set as the tracking point, and a certain range set in advance from the point is set as the region estimation range. Thereafter, the processing proceeds to step S122 in FIG.

  The above process will be described with reference to FIG. That is, in step S142 of FIG. 11, for example, as shown in FIG. 12, when the human eye 102 of the frame n−1 is designated as the tracking point 101, a predetermined region including the tracking point 101 is specified in step S143. Is designated as the area estimation range 133. In step S124, it is determined whether or not the sample points within the region estimation range 133 can be estimated in the next frame. In the case of the example of FIG. 12, in the frame n + 1 next to the frame n, the left half region 134 in the drawing including the left eye 102 in the region estimation range 133 is hidden by the ball 121. 101 motion cannot be estimated in the next frame n + 1. Therefore, in such a case, one of the points in the area designation range 133 (in the face 104 as a rigid body including the right eye 102) prepared in advance as a transfer candidate in the previous frame n-1 in time is selected. One point (for example, the left eye 103 included in the face 104 (exactly one pixel therein)) is selected, and that point is set as the tracking point in the frame n + 1.

  The region estimation related processing unit 55 has a configuration as shown in FIG. 13 in order to execute the region estimation related processing in step S126 of FIG. That is, the motion vector and the accuracy are input from the motion estimation unit 52, the background motion is input from the background motion estimation unit 54, and the tracking point from the tracking point determination unit 57 is input to the region estimation unit 161 of the region estimation related processing unit 55. Position information is input. In addition to the motion vector and the accuracy supplied from the motion estimation unit 52, the transfer candidate extraction unit 162 is also supplied with the output of the region estimation unit 161. In addition to the input image being input to the template creation unit 163, the output of the region estimation unit 161 is input.

  The region estimation unit 161 estimates a rigid region including a tracking point based on the input, and outputs an estimation result to the transfer candidate extraction unit 162 and the template creation unit 163. The transfer candidate extraction unit 162 extracts transfer candidates based on the input, and supplies the extracted transfer candidates to the transfer candidate holding unit 56. The template creation unit 163 creates a template based on the input, and supplies the created template to the template holding unit 58.

  FIG. 14 shows details of the area estimation related process (the process of step S126 in FIG. 10) executed by the area estimation related processing unit 55. First, in step S161, region estimation processing is executed by the region estimation unit 161. The details thereof will be described later with reference to FIG. 15. By this processing, the points of the region on the image that are estimated to belong to the same object (the rigid body that moves in synchronization with the tracking point) to which the tracking point belongs. Are extracted as points of the region estimation range.

  In step S162, the transfer candidate extraction unit 162 executes a transfer candidate extraction process. Details of the processing will be described later with reference to FIG. 20, but transfer candidate points are extracted from the points in the range estimated by the region estimation unit 161 as the region estimation range, and are held in the transfer candidate holding unit 56.

  In step S163, the template creation unit 163 executes template creation processing. Details thereof will be described later with reference to FIG. 21, but a template is created by this processing.

  Next, details of the region estimation processing in step S161 in FIG. 14 will be described with reference to the flowchart in FIG.

  First, in step S181, the region estimation unit 161 determines a sample point as a candidate point of a point estimated to belong to the same target as the tracking point.

  For example, as shown in FIG. 16, the sample points are predetermined in the horizontal direction and the vertical direction with reference to the fixed reference point 201 among the pixels in the entire screen of the frame indicated by the white square in the figure. Pixels at positions separated by the number of pixels can be used as sample points (represented by black squares in the figure). In the example of FIG. 16, the upper left pixel of each frame is set as a reference point 201 (the reference point 201 is indicated by a cross in the figure), and is separated by 5 in the horizontal direction and 5 in the vertical direction. The pixel at the position is taken as the sample point. That is, in the case of this example, pixels at positions dispersed throughout the entire screen are taken as sample points. In this example, the reference point is a point at the same position fixed in each frame n, n + 1.

  Note that the reference point 201 can be dynamically changed so as to be a point at a different position for each of the frames n and n + 1.

  In the example of FIG. 16, the interval between sample points is a fixed value in each frame, but the interval between sample points for each frame is, for example, 5 pixels in frame n and 8 in frame n + 1. It can also be variable with the pixel. As an interval reference at this time, an area of a region estimated to belong to the same object as the tracking point can be used. Specifically, if the area of the region estimation range is narrowed, the interval is also shortened.

  Alternatively, the interval between sample points can be made variable in one frame. The distance from the tracking point can be used as a reference for the interval at this time. That is, the sample point closer to the tracking point has a smaller interval, and the farther from the tracking point, the larger the interval.

  When the sample points are determined as described above, in step S182, the region estimation unit 161 then determines the region estimation range (steps S143 and S144 in FIG. 11 or steps S206 and S208 in FIG. 17 described later). The process of estimating the movement of the sample points within (determined in the process of (1)) is executed. That is, the region estimation unit 161 extracts the corresponding point of the next frame corresponding to the sample point within the region estimation range based on the motion vector supplied from the motion estimation unit 52.

  In step S183, the region estimation unit 161 executes processing for excluding points based on motion vectors whose accuracy is lower than a preset threshold value from the sample points estimated in step S182. The accuracy of the motion vector necessary for this processing is supplied from the motion estimation unit 52. Thereby, only the points estimated based on the motion vector with high accuracy are extracted from the sample points within the region estimation range.

  In step S184, the region estimation unit 161 extracts the full screen motion in the motion estimation result within the region estimation range. The full screen motion means a motion that takes the area corresponding to the same motion and maximizes the area. Specifically, a motion histogram is generated by assigning a weight proportional to the sample point interval at each sample point to the motion of each sample point, and one motion (one motion vector) having the maximum weighting frequency is generated. Extracted as full screen motion. When generating a histogram, for example, a representative value of motion can be prepared with pixel accuracy, and a motion having a value with one pixel accuracy can be added to the histogram.

  In step S185, the region estimation unit 161 extracts sample points within the region estimation range having the full screen motion as a result of region estimation. In this case, the sample points having the full screen motion include not only the sample points having the same motion as the full screen motion but also the difference in motion from the full screen motion is equal to or less than a predetermined threshold value set in advance. The sample point can also be a sample point having full screen motion here.

  In this way, among the sample points within the area estimation range determined by the processing of steps S143, S144, S206, and S208, the sample point having the full screen motion is estimated to belong to the same target as the tracking point. Is finally extracted (generated).

  Next, in step S186, the region estimation unit 161 executes a region estimation range update process. Thereafter, the processing proceeds to step S122 in FIG.

  FIG. 17 shows details of the region estimation range update processing in step S186 of FIG. In step S201, the region estimation unit 161 calculates the center of gravity of the region. This region means a region composed of sample points extracted in the process of step S185 in FIG. 15 (region composed of points estimated to belong to the same target as the tracking point). That is, one motion vector (full screen motion) corresponds to this area. For example, as shown in FIG. 18A, among sample points indicated by white squares in the drawing, sample points having full-screen motion in the process of step S185 in FIG. In FIG. 18A, sample points indicated by black squares are extracted, and an area constituted by the sample points is extracted (estimated) as an area 222. Then, the center of gravity 224 of the region 222 is further calculated. Specifically, the sample point centroid obtained by assigning the weight of the sample point interval to each sample point is obtained as the centroid of the region. This process has the meaning of obtaining the position of the region in the current frame.

  Next, in step S202, the region estimation unit 161 executes a process of shifting the center of gravity of the region by full screen motion. This process has the meaning of causing the area estimation range 221 to follow the movement of the position of the area and to move to the estimated position in the next frame. As shown in FIG. 18B, when the tracking point 223 in the current frame appears as the tracking point 233 in the next frame based on the motion vector 238, the full-screen motion vector 230 substantially corresponds to the tracking point motion vector 238. Therefore, the point 234 on the same frame (next frame) as the tracking point 233 is obtained by shifting the center of gravity 224 in the current frame based on the motion vector 230 (full screen motion). If the area estimation range 231 is set around the point 234, the area estimation range 221 is moved to the estimated position in the next frame by following the movement of the position of the area 222.

  In step S203, the region estimation unit 161 determines the size of the next region estimation range based on the region estimation result. Specifically, the sum of squares of sample point intervals (intervals of points indicated by black squares in the region 222 in FIG. 18A) for all sample points estimated as the region is regarded as the area of the region 222. The size of the region estimation range 231 in the next frame is determined so that the size is slightly larger than the area. That is, the size of the region estimation range 231 increases when the number of sample points in the region 222 is large and decreases when the number of sample points is small. In this way, not only can the enlargement / reduction of the area 222 be followed, but also the entire screen area within the area estimation range 221 can be prevented from becoming a peripheral area to be tracked.

  When the full screen motion extracted in step S184 in FIG. 15 matches the background motion, the background and the tracking target cannot be distinguished by the motion. Therefore, the background motion estimation unit 54 always performs the background motion estimation process. In step S204, the region estimation unit 161 extracts the background motion supplied from the background motion estimation unit 54 and the process of step S184 in FIG. It is determined whether or not the full screen motion matches. If the full screen motion matches the background motion, in step S205, the region estimation unit 161 limits the size of the next region estimation range so that the current region estimation range is maximized. Thereby, it is suppressed that the background is erroneously recognized as the tracking target and the size of the area estimation range is enlarged.

  If it is determined in step S204 that the full screen motion and the background motion do not match, the processing in step S205 is not necessary and is skipped.

  Next, in step S <b> 206, the region estimation unit 161 determines the size of the next region estimation range around the shifted region center of gravity. As a result, the area estimation range is determined such that the center of gravity coincides with the already obtained area centroid after the shift, and the size thereof is proportional to the area width.

  In the example of FIG. 18B, the area estimation range 231 is determined to have a size corresponding to the area of the area 222 with the shifted center of gravity 234 based on the motion vector (full screen motion) 230 as the center.

  It is necessary to ensure (ensure) that the region having the full screen motion within the region estimation range 231 is the region of the tracking target (for example, the face 104 in FIG. 12). Therefore, in step S207, the region estimation unit 161 determines whether or not the tracking point is included in the next region estimation range. If the tracking point is not included, in step S208, the region estimation unit 161 determines that the tracking point is included in the next region estimation range. A process for shifting the region estimation range is executed. If the tracking point is included in the next area estimation range, the process of step S208 is not necessary and is skipped.

  As a specific method of shifting in this case, a method of minimizing the moving distance, a minimum distance at which the tracking point is included along a vector from the center of gravity of the region estimation range before the shift to the tracking point A way to move only is considered.

  In order to emphasize the robustness of tracking, a method of not performing a shift to include a tracking point in the region is also conceivable.

  In the example of FIG. 18C, since the region estimation range 231 does not include the tracking point 233, the region estimation range 241 is shifted to a position indicated as the region estimation range 241 (a position including the tracking point 233 at the upper left).

  18A to 18C show the case where the shift process of step S208 is necessary, but FIGS. 19A to 19C show the case where the shift process of step S208 is not necessary (the tracking point is the next region estimation range in step S207). Example).

  As shown in FIGS. 19A to 19C, when all the sample points in the area estimation range 221 are area points, the shift process in step S208 in FIG. 17 is not necessary.

  18A to 18C and FIGS. 19A to 19C show an example in which the area estimation range is rectangular, but the area estimation range may be circular.

  As described above, the area estimation range update process in FIG. 17 (step S186 in FIG. 15) determines the position and size of the area estimation range for the next frame to include the tracking point.

  In the update process of the area estimation range in FIG. 17, the area estimation range is a rectangular (or circular) fixed shape, but may be a variable shape.

  Next, the transfer candidate extraction process in step S162 of FIG. 14 will be described with reference to the flowchart of FIG.

  In step S261, the transfer candidate extraction unit 162 holds, as transfer candidates, the point shift results of the estimated motion corresponding to each of the points estimated as the full screen motion region. That is, instead of using the points obtained as region estimation results as they are, a process for extracting the results shifted based on the respective motion estimation results is performed for use in the next frame. The changed transfer candidates are supplied to and held in the transfer candidate holding unit 56.

  This process will be described with reference to FIG. That is, in the example of FIG. 12, the tracking point 101 exists in the frames n−1 and n, but in the frame n + 1, it is hidden by the ball 121 flying from the left side in the drawing, and the tracking point 101 does not exist. Therefore, in frame n + 1, it is necessary to change the tracking point to another point on the face 104 as a tracking target (for example, the left eye 103 (actually closer to the right eye 102)). Therefore, a transfer candidate is prepared in advance in a frame before transfer is actually required.

  Specifically, in the case of the example in FIG. 12, the motion estimation result in the region estimation range 133 from frame n to frame n + 1 needs to be changed in the region estimation range 133, so that there is a high probability that it cannot be estimated correctly. Is expected. That is, in the example of FIG. 12, the transfer occurs because the tracking point and a part of the same object are hidden. As a result, in the region estimation range 133 in the frame n, regarding the portion 134 where the target is hidden in the frame n + 1 (the shaded portion in FIG. 12) 134, the motion is not estimated correctly, and it is estimated that the accuracy of the motion is low. It is estimated that the accuracy is not low and a motion estimation result is meaningless.

  In such a case, there is a high possibility that the region estimation is erroneous due to a decrease in motion estimation results that can be used in region estimation or a mixture of erroneous motion estimation results. On the other hand, such a possibility is generally greater in the region estimation between the previous frame n−1 and the frame n in comparison with the estimation between the frame n and the frame n + 1. Expected to be lower.

  Therefore, in order to reduce the risk, the region estimation result is not used as it is, but the region estimation result obtained in the previous frame n-1 (or a frame earlier in time) is used as the movement destination in the next frame. It is desirable to use as a transfer candidate for improving the performance.

  However, the region estimation result can be used as it is.

  FIG. 21 shows details of the template creation processing in step S163 of FIG. In step S <b> 281, the template creation unit 163 determines a small region corresponding to each of the points estimated as the region (region of full screen motion). In the example of FIG. 22, the small area 322 is determined corresponding to the area point 321.

  In step S282, the template creation unit 163 sets the area of the sum of the small areas determined in the process of step S281 as the template range. In the example of FIG. 22, the sum area of the small areas 322 is a template range 331.

  In step S283, the template creation unit 163 creates a template from the template range information and image information set in step S282, supplies the template to the template holding unit 58, and holds the template. Specifically, pixel data in the template range 331 is used as a template.

  In FIG. 23, the small area 341 corresponding to the area point 321 has a larger area than the small area 322 in FIG. As a result, the template range 351 of the sum area of the small areas 341 is also wider than the template range 331 of FIG.

  It is conceivable that the size of the small region is proportional to the interval between the sample points, but the proportionality constant at that time can be determined so that the area becomes the square of the interval between the sample points, or larger or smaller than that. It is also possible.

  For example, a fixed size or shape range centered on the tracking point may be used as the template range without using the region estimation result.

  FIG. 24 shows the positional relationship between the template and the area estimation range. The template range 403 includes a tracking point 405. The upper left point of the circumscribed rectangle 401 circumscribing the template range 403 is a template reference point 404. The vector 406 from the template reference point 404 toward the tracking point 405 and the vector 407 from the template reference point 404 toward the upper left reference point 408 in the region estimation range 402 in the drawing are used as information of the template range 403. The template is composed of pixels included in the template range 403. The vectors 406 and 407 are used for returning to normal processing when the same image as the template is detected.

  In the above processing, unlike the case of the transfer candidate, the example in which both the range and the pixel correspond to the current frame is used as the template. However, as in the case of the transfer candidate, the destination in the next frame is set as the template. Can also be used.

  As described above, a template made up of pixel data including a tracking point is created in advance during normal processing in the same manner as a transfer candidate.

  Details of the exception processing in step S52 performed following the normal processing in step S51 of FIG. 5 described above will be described with reference to the flowchart of FIG. As described above, this process is performed when it is determined in step S124 in FIG. 10 that it is impossible to estimate the movement of the tracking point, and in step S128, it is determined that the transfer candidate for changing the tracking point cannot be selected. Will be executed.

  In step S401, the control unit 59 executes an exception process initialization process. Details of this processing are shown in the flowchart of FIG.

  In step S421, the control unit 59 causes a scene change when the tracking point cannot be tracked (when the movement of the tracking point cannot be estimated and a transfer candidate for changing the tracking point cannot be selected). It is determined whether it has been. The scene change detection unit 53 constantly monitors whether or not there is a scene chain based on the estimation result of the motion estimation unit 52, and the control unit 59 performs step S421 based on the detection result of the scene change detection unit 53. Execute the judgment. Specific processing of the scene change detection unit 53 will be described later with reference to FIGS. 38 and 39.

  If a scene change has occurred, it is presumed that the reason why tracking has become impossible is due to the occurrence of a scene change, and in step S422, the control unit 59 sets the mode to scene change. In contrast, if it is determined in step S421 that no scene change has occurred, the control unit 59 sets the mode to another mode in step S423.

  After the process of step S422 or step S423, the template matching unit 51 executes a process of selecting the oldest template in time in step S424. Specifically, as shown in FIG. 27, for example, when transition from frame n to frame n + 1 is performed, if exception processing is to be executed, a frame n−m + 1 is generated for frame n and is stored in the template holding unit 58. A template generated with respect to the frame mn + 1, which is the oldest template in time, is selected from the held m-frame templates.

  In this way, the template just before the transition to exception processing (the template generated for the frame n in the case of FIG. 27) is not used, but the template slightly before in time is selected for the occlusion to be tracked or the like. This is because when the transition to exception processing occurs, the tracking target is already considerably hidden immediately before the transition, and it is highly likely that the tracking target cannot be captured sufficiently large in the template at that time. Therefore, reliable tracking is possible by selecting a template in a frame slightly before in time.

  Next, in step S425, the template matching unit 51 executes processing for setting a template search range. For example, the template search range is set such that the position of the tracking point immediately before the transition to the exception process is the center of the template search range.

  That is, as shown in FIG. 28, when the right eye 102 of the subject's face 104 is designated as the tracking point 101 in the frame n, the ball 121 flies from the left direction in the figure, and the tracking point 101 in the frame n + 1. Is assumed, and the tracking point 101 appears again in the frame n + 2. In this case, a region centered on the tracking point 101 (included in the template range 411) is set as the template search range 412.

  In step S426, the template matching unit 51 resets the number of elapsed frames and the number of scene changes after shifting to exception processing to zero. The number of frames and the number of scene changes are used in a continuation determination process (steps S461, S463, S465, and S467 in FIG. 30) in step S405 in FIG.

  After the exception process initialization process is completed as described above, in step S402 in FIG. 25, the control unit 59 executes a process of waiting for the next frame. In step S403, the template matching unit 51 performs template matching processing within the template search range. In step S404, the template matching unit 51 determines whether it is possible to return to normal processing.

  Specifically, the absolute value sum of the difference between the template several frames before (the pixel in the template range 411 in FIG. 28) and the matching target pixel in the template search range is calculated by the template matching process. More specifically, the sum of absolute values of differences between the pixels in the predetermined block in the template range 411 and the predetermined block in the template search range is calculated. The position of the block is sequentially moved within the template range 411, and the sum of absolute values of the differences of the respective blocks is added to obtain a value at the position of the template. Then, the position where the sum of absolute values of the differences becomes the smallest when the template is sequentially moved within the template search range and its value are searched. In step S404, the absolute sum of the minimum differences is compared with a predetermined threshold value set in advance. If the sum of absolute values of the differences is less than or equal to the threshold value, an image including the tracking point (included in the template) has appeared again, so it is determined that the normal processing can be restored, and the processing Returns to the normal processing of step S51 of FIG.

  Then, as described above, in step S141 in FIG. 11, it is determined that the return from the exception processing is performed, and in step S144, the position where the difference absolute value sum is minimum is set as the matched position of the template, The tracking point and area estimation range are set based on the positional relationship between the template position and the tracking point area estimation range held corresponding to the template. That is, as described above with reference to FIG. 24, the region estimation range 402 is set based on the vectors 406 and 407 with the tracking point 405 as a reference.

  However, in the region estimation process in step S161 of FIG. 14, when a method that does not use the region estimation range is used, the region estimation range is not set.

  The determination as to whether or not it is possible to return to normal processing in step S404 in FIG. 25 is performed by comparing the value obtained by dividing the minimum sum of absolute differences by the activity of the template with a threshold value. Also good. As the activity in this case, the value calculated in step S603 in FIG. 32 by the activity calculation unit 602 in FIG.

  Alternatively, it is possible to return to the normal processing by comparing a value obtained by dividing the current minimum absolute difference sum by the minimum absolute difference sum one frame before with a predetermined threshold. It may be determined whether or not. In this case, it is not necessary to calculate the activity. That is, in step S404, the correlation between the template and the template search range is calculated, and determination is performed based on the comparison between the correlation value and the threshold value.

  If it is determined in step S404 that it is not possible to return to the normal process, the process proceeds to step S405, and the continuation determination process is executed. Details of the continuation determination process will be described later with reference to FIG. 30, and thereby, it is determined whether or not the exception process can be continued.

  In step S406, the control unit 59 sets whether exception processing (tracking of the tracking point in exception processing) can be continued based on the result of the continuation determination processing (steps S466 and S468 in FIG. 30 described later). (Based on the flag that was set). If the exception process can be continued, the process returns to step S402, and the subsequent processes are repeatedly executed. That is, the process of waiting until the tracking point appears again is repeatedly executed.

  On the other hand, if it is determined in step S406 that the exception processing cannot be continued (whether it is determined in step S465 in FIG. 30 described later that the number of frames that have elapsed after the tracking point disappears is equal to or greater than the threshold value THfr Or, when it is determined in step S467 that the number of scene changes is equal to or greater than the threshold value THsc), the tracking process is terminated as the tracking process is no longer possible. Instead of ending the tracking process, it may be possible to return to the normal process again using the tracking point that has been held. The exception handling in this case is shown in FIG. Note that the processing in steps S441 through S445 in FIG. 29 is the same as the processing in steps S401 through S405 in FIG.

  That is, when it is determined by the continuation determination process in step S445 whether or not the exception process can be continued, thereafter, in step S446, the control unit 59 performs the exception process (tracking the tracking point in the exception process). Is determined based on the result of the continuation determination process (based on a flag set in steps S466 and S468 in FIG. 30 described later). If the exception process can be continued, the process returns to step S442, and the subsequent processes are repeatedly executed. That is, the process of waiting until the tracking point appears again is repeatedly executed.

  On the other hand, if it is determined in step S446 that the exception processing cannot be continued (whether it is determined in step S465 in FIG. 30 described later that the number of elapsed frames after the tracking point disappears is equal to or greater than the threshold value THfr. Or, when it is determined in step S467 that the number of scene changes is equal to or greater than the threshold value THsc), the exception processing is no longer possible and the processing returns to the normal processing in step S1051 in FIG.

  In this case, as described above, it is determined in step S141 in FIG. 11 that the process has returned from the exception process, and in step S144, based on the position of the tracking point immediately before the transition to the exception process held. The tracking point and the area estimation range are set.

  FIG. 30 shows details of the continuation determination process in step S405 of FIG. 25 (or step S445 of FIG. 29). In step S461, the control unit 59 executes a process of adding 1 to the number of elapsed frames as a variable. The number of elapsed frames is previously reset to 0 in the exception process initialization process (step S426 in FIG. 26) in step S401 in FIG.

  Next, in step S462, the control unit 59 determines whether there is a scene change. Whether or not there is a scene change can be determined based on the detection result of the scene change detection unit 53 always executing the detection process. If there is a scene change, the process proceeds to step S463, and the control unit 59 adds 1 to the number of scene changes as a variable. The number of scene changes is also reset to 0 in the initialization process of step S426 in FIG. If no scene change has occurred during the transition from the normal process to the exception process, the process of step S463 is skipped.

  Next, in step S464, the control unit 59 determines whether or not the currently set mode is a scene change. This mode is set in steps S422 and S423 in FIG. When the currently set mode is a scene change, the process proceeds to step S467, and the control unit 59 determines whether or not the number of scene changes is smaller than a preset threshold value THsc. If the number of scene changes is smaller than the threshold value THsc, the process proceeds to step S466, and the control unit 59 sets a flag that allows continuation. If the number of scene changes is equal to or greater than the threshold value THsc, the process proceeds to step S468 and cannot be continued. Set the flag.

  On the other hand, when it is determined in step S464 that the mode is not a scene change (when it is determined that the mode is other), the process proceeds to step S465, and the control unit 59 determines whether or not the number of elapsed frames is smaller than the threshold value THfr. Determine whether. The number of elapsed frames is also reset to 0 in advance in step S426 of the exception process initialization process of FIG. If it is determined that the number of elapsed frames is smaller than the threshold value THfr, a continuation flag is set in step S466, and if it is determined that the number of elapsed frames is greater than or equal to the threshold value THfr, the process continues in step S468. A disabled flag is set.

  As described above, when the number of scene changes at the time of template matching processing is equal to or greater than the threshold value THsc, or when the number of elapsed frames is equal to or greater than the threshold value THfr, further exception processing is impossible.

  When the mode is other, a condition that the number of scene changes is 0 may be added to determine whether or not continuation is possible.

  In the above, it is assumed that the frame of the image is used as a processing unit and all frames are used. However, processing is not performed in units of fields or using all frames or fields, but is extracted by thinning out at a predetermined interval. It is also possible to use a modified frame or field.

  Further, in the above, the estimated movement destination of the point in the area is used as the transfer candidate. However, in this case, even if the whole screen motion is (0, 0), each point in the area is , (-1, 1), (1, 0), etc., the movement is shifted by the amount of each movement. Instead of using the destination point as a transfer candidate as it is, it is also possible to set the closest point among sample points obtained in advance as a transfer candidate. Of course, in order to reduce the processing load, each point may be shifted by the amount corresponding to the full screen movement.

  Further, instead of using the estimated destination of the point in the area as the transfer candidate, it is possible to use the point in the area as it is.

  Next, a configuration example of the motion estimation unit 52 in FIG. 3 will be described with reference to FIG. In this embodiment, the input image is supplied to the evaluation value calculation unit 601, the activity calculation unit 602, and the motion vector detection unit 606. The evaluation value calculation unit 601 calculates an evaluation value related to the degree of coincidence between the two objects associated by the motion vector, and supplies the evaluation value to the normalization processing unit 604. The activity calculation unit 602 calculates the activity of the input image and supplies it to the threshold value determination unit 603 and the normalization processing unit 604. The motion vector detection unit 606 detects a motion vector from the input image and supplies it to the evaluation value calculation unit 601 and the integration processing unit 605.

  The normalization processing unit 604 normalizes the evaluation value supplied from the evaluation value calculation unit 601 based on the activity supplied from the activity calculation unit 602, and supplies the obtained value to the integration processing unit 605. The threshold determination unit 603 compares the activity supplied from the activity calculation unit 602 with a predetermined threshold and supplies the determination result to the integration processing unit 605. The integration processing unit 605 calculates the accuracy of the motion vector based on the normalization information supplied from the normalization processing unit 604 and the determination result supplied from the threshold value determination unit 603, and the obtained accuracy is detected by the motion vector detection. This is output together with the motion vector supplied from the unit 606.

  Next, the motion estimation process of the motion estimation unit 52 will be described with reference to the flowchart of FIG. Although the motion vector is obtained for a point, the accuracy is calculated using image data of a small block in the vicinity of two points associated by the motion vector, for example, centering on the point. In step S601, the motion vector detection unit 606 detects a motion vector from the input image. For this detection, for example, a block matching method or a gradient method is used. The detected motion vector is supplied to the evaluation value calculation unit 601 and the integration processing unit 605.

  In step S602, the evaluation value calculation unit 601 calculates an evaluation value. Specifically, for example, the sum of absolute differences of pixel values of two blocks centered on two points associated with motion vectors is calculated. That is, the motion vector V (vx, vy) detected by the motion vector detection unit 606 in step S601, the point P (Xp, Yp) on the image Fi of the previous frame based on the motion vector V (vx, vy), and the time later The relationship of the point Q (Xq, Yq) on the image Fj of the frame is expressed by the following equation.

  The evaluation value calculation unit 601 calculates an evaluation value Eval (P, Q, i, j) for a block centered on the point P and a block centered on the point Q based on the following equation.

  Each block is a square having 2L + 1 pixels on one side. The summation ΣΣ in the above equation is performed between corresponding pixels when x is from −L to L and y is from −L to L. Therefore, for example, when L = 2, nine differences are obtained, and the sum of the absolute values is calculated. The evaluation value indicates that the two blocks match well as the value approaches zero.

  The evaluation value calculation unit 601 supplies the generated evaluation value to the normalization processing unit 604.

  In step S603, the activity calculation unit 602 calculates an activity from the input image. The activity is a feature amount representing the complexity of the image. As shown in FIG. 33, the difference between the pixel of interest Y (x, y) and the adjacent 8 pixels Y (x + i, y + j) for each pixel is shown in FIG. The average value of the sum of absolute values is calculated based on the following formula as activity Activity (x, y) at the target pixel position.

  In the case of the example of FIG. 33, the value of the pixel of interest Y (x, y) located at the center among 3 × 3 pixels is 110, and the values of eight pixels adjacent to it are 80, 70, and 75, respectively. , 100, 100, 100, 80, 80, the activity Activity (x, y) is expressed by the following equation.

  Activity (x, y) = {| 80-110 | + | 70-110 | + | 75-110 | + | 100-110 | + | 100-110 | + | 80-110 | + | 80−110 |} /8=24.375.

  Similar processing is performed for all pixels in the frame.

  In order to calculate the motion vector accuracy in units of blocks, the sum of the activities of all the pixels in the block expressed by the following equation is defined as the activity (block activity) Blockactivity (i, j) of the block.

  In addition, the activity may be a variance value, a dynamic range, or the like.

  In step S604, the threshold determination unit 603 compares the block activity calculated by the activity calculation unit 602 with a predetermined threshold set in advance. Then, a flag indicating whether or not the input block activity is larger than the threshold value is output to the integration processing unit 605.

  Specifically, as a result of the experiment, the block activity and the evaluation value have the relationship shown in FIG. 34 using the motion vector as a parameter. In FIG. 34, the horizontal axis represents block activity Blockactivity (i, j), and the vertical axis represents the evaluation value Eval. When the motion is correctly detected (when the correct motion vector is given), the block activity and the value of the evaluation value are distributed in the region R1 below the curve 621 in the figure. On the other hand, if an incorrect motion (incorrect motion vector) is given, the block activity and the evaluation value are distributed in the region R2 on the left side of the figure from the curve 622 (the region above the curve 622). There is almost no distribution in the region other than R2 and the region other than the region R1 below the curve 621). Curves 621 and 622 intersect at point P. The value of the block activity at this point P is set as the threshold value THa. The threshold value THa means that if the value of the block activity is smaller than that, the corresponding motion vector may be incorrect (this will be described in detail later). The threshold determination unit 603 outputs a flag indicating whether the block activity value input from the activity calculation unit 602 is greater than the threshold THa to the integration processing block 605.

  In step S605, the normalization processing unit 604 executes normalization processing. Specifically, the normalization processing unit 604 calculates the motion vector accuracy VC according to the following equation.

  However, if the value of the motion vector accuracy VC is less than 0, the value is replaced with 0. Of the motion vector accuracy VC, the value obtained by dividing the evaluation value by the block activity is a straight line 623 whose position on the graph of FIG. Therefore, it indicates whether the region is in the lower region or the upper region in the figure. That is, if the slope of the straight line 623 is 1, and the value obtained by dividing the evaluation value by the block activity is larger than 1, the points corresponding to the value are points distributed in the area above the straight line 623. Means that. The motion vector accuracy VC obtained by subtracting this value from 1 means that the smaller the value, the higher the possibility that the corresponding points are distributed in the region R2.

  On the other hand, if the value obtained by dividing the evaluation value by the block activity is smaller than 1, it means that the points corresponding to the value are distributed in the lower area of the straight line 623 in the figure. The motion vector accuracy VC at that time means that the larger the value (closer to 0), the corresponding points are distributed in the region R1. The normalization processing unit 604 outputs the motion vector accuracy VC obtained by the calculation in this way to the integration processing unit 605.

  In step S606, the integration processing unit 605 executes integration processing. Details of this integration processing are shown in the flowchart of FIG.

  In step S631, the integration processing unit 605 determines whether the block activity is equal to or less than the threshold value THa. This determination is performed based on the flag supplied from the threshold determination unit 603. If the block activity is equal to or less than the threshold THa, the integration processing unit 605 sets the value of the motion vector accuracy VC calculated by the normalization processing unit 604 to 0 in step S632. If it is determined in step S631 that the activity value is greater than the threshold value THa, the processing in step S632 is skipped, and the value of the motion vector accuracy VC generated by the normalization processing unit 604 is output as it is together with the motion vector. Is done.

  This is because there is a possibility that a correct motion vector is not obtained if the block activity value is smaller than the threshold value THa even if the value of the motion vector accuracy VC calculated by the normalization processing unit 604 is positive. Because there is. That is, as shown in FIG. 34, between the origin O and the point P, the curve 622 protrudes below the curve 621 in the drawing (downward from the straight line 623). In a region R3 in which the value of the block activity is smaller than the threshold value Tha and surrounded by the curves 621 and 622, the value obtained by dividing the evaluation value by the block activity is distributed in both the regions R1 and R2. There is a high possibility that a correct motion vector is not obtained.

  Therefore, in such a distribution state, processing is performed assuming that the accuracy of the motion vector is low. For this reason, in step S632, the motion vector accuracy VC is set to 0 if the value is positive even if the value is positive. In this way, when the value of the motion vector accuracy VC is positive, it is possible to reliably represent that the correct motion vector is obtained. In addition, the larger the value of the motion vector accuracy VC, the higher the probability that a correct motion vector is obtained (the probability that the distribution is included in the region R1 increases).

  This coincides with an empirical rule that, in general, it is difficult to detect a motion vector with high reliability in a region where the luminance change is small (region where the activity is small).

  FIG. 36 illustrates a configuration example of the background motion estimation unit 54 of FIG. In this configuration example, the background motion estimator 54 includes a frequency distribution calculator 651 and a background motion determiner 652.

  The frequency distribution calculation unit 651 calculates a frequency distribution of motion vectors. However, this frequency is weighted so that a likely motion is weighted by using the motion vector accuracy VC supplied from the motion estimation unit 52. Based on the frequency distribution calculated by the frequency distribution calculation unit 651, the background motion determination unit 652 performs a process of determining a motion with the maximum frequency as a background motion, and outputs the background motion to the region estimation related processing unit 55.

  The background motion estimation process of the background motion estimation unit 54 will be described with reference to FIG.

  In step S651, the frequency distribution calculation unit 651 calculates a motion frequency distribution. Specifically, the frequency distribution calculation unit 651 assumes that the x and y coordinates of the motion vector as a background motion candidate are expressed in a range of ± 16 pixels from the reference point, respectively (1089 (= 16 × 2 + 1) × (16 × 2 + 1)), that is, a box for coordinates corresponding to possible values of a motion vector, and when a motion vector is generated, 1 is added to the coordinates corresponding to the motion vector. In this way, the motion vector frequency distribution can be calculated.

  However, when one motion vector is generated, if 1 is added, if the frequency of occurrence of a motion vector with low accuracy is high, a motion vector with low certainty may be determined as the background motion. . Therefore, when a motion vector is generated, the frequency distribution calculation unit 651 does not add a value 1 to a box (coordinates) corresponding to the motion vector, but a value obtained by multiplying the value 1 by the motion vector accuracy VC (= Motion vector accuracy VC). The value of the motion vector accuracy VC is normalized as a value between 0 and 1, and the closer the value is to 1, the higher the accuracy. Therefore, the frequency distribution obtained in this way is a frequency distribution obtained by weighting motion vectors based on their accuracy. This reduces the risk that a motion with low accuracy is determined as the background motion.

  Next, in step S652, the frequency distribution calculation unit 651 determines whether or not the process of calculating the motion frequency distribution has been completed for all blocks. If there is a block that has not yet been processed, the process returns to step S651, and the process of step S651 is executed for the next block.

  As described above, the motion frequency distribution calculation process is performed on the entire screen, and when it is determined in step S652 that the processing of all blocks has been completed, the process proceeds to step S653, and the background motion determination unit 652 A process for searching for the maximum value of the distribution is executed. That is, the background motion determination unit 652 selects the frequency having the maximum frequency from the frequencies calculated by the frequency distribution calculation unit 651, and determines the motion vector corresponding to the frequency as the motion vector of the background motion. The motion vector of the background motion is supplied to the region estimation related processing unit 55, and is used for, for example, a determination process of whether or not the full screen motion matches the background motion in step S204 of FIG.

  FIG. 38 shows a detailed configuration example of the scene change detection unit 53 of FIG. In this example, the scene change detection unit 53 is configured by the motion vector accuracy average calculation unit 671 and the threshold determination unit 672.

  The motion vector accuracy average calculation unit 671 calculates the average value of all the screens of the motion vector accuracy VC supplied from the motion estimation unit 52 and outputs the average value to the threshold determination unit 672. The threshold value determination unit 672 compares the average value supplied from the motion vector accuracy average calculation unit 671 with a predetermined threshold value, and determines whether it is a scene change based on the comparison result. The result is output to the control unit 59.

  Next, the operation of the scene change detection unit 53 will be described with reference to the flowchart of FIG. In step S681, the motion vector accuracy average calculation unit 671 calculates the sum of vector accuracy. Specifically, the motion vector accuracy average calculation unit 671 executes a process of adding the value of the motion vector accuracy VC calculated for each block output from the integration processing unit 605 of the motion estimation unit 52.

  In step S682, the motion vector accuracy average calculation unit 671 determines whether or not the processing for calculating the sum of the vector accuracy VC has been completed for all the blocks. If the processing has not yet been completed, the processing of step S681 is repeated. . By repeating this process, the sum of motion vector accuracy VC of each block for one screen is calculated. If it is determined in step S682 that the calculation process of the sum of motion vector accuracy VCs for all one screen has been completed, the process advances to step S683, and the motion vector accuracy average calculation unit 671 calculates the average value of the vector accuracy VC. Execute. Specifically, a value obtained by dividing the sum of the vector accuracy VC for one screen calculated in the process of step S681 by the number of added blocks is calculated as an average value.

  In step S684, the threshold value determination unit 672 compares the average value of the motion vector accuracy VC calculated by the motion vector accuracy average calculation unit 671 in the process of step S683 with a preset threshold value, and whether or not the threshold value is smaller than the threshold value. Determine whether. In general, when a scene change occurs between two frames having different times in a moving image, there is no corresponding image, so even if a motion vector is calculated, the motion vector is not certain.

  Therefore, if the average value of the vector accuracy VC is smaller than the threshold value, the threshold value determination unit 672 turns on the scene change flag in step S685. If the average value of the vector accuracy VC is not smaller than the threshold value (if it is greater than or equal to the threshold value), in step S586 Turn off the change flag. When the scene change flag is on, it indicates that there is a scene change, and when the scene change flag is off, it indicates that there is no scene change.

  The scene change flag is supplied to the control unit 59, and is used for determining whether or not there is a scene change in step S421 in FIG.

  Next, a detailed configuration example and operation of the presentation position calculation unit 42 in FIG. 3 will be described with reference to FIG. FIG. 40 is a block diagram illustrating a functional configuration example of the presentation position calculation unit 42. In this example, the presentation position calculation unit 42 includes a transfer amount calculation unit 701, a tracking point storage unit 702, a reflected transfer amount calculation unit 703, a center point setting unit 705, and a center point storage unit 706.

  In the transfer amount calculation unit 701 and the center point setting unit 705 of the presentation position calculation unit 42, information on the tracking point (tracking point information) from the tracking processing unit 41 and a motion vector (that is, a motion amount) corresponding to the tracking point. Is entered. The information regarding the tracking point includes, for example, position information of the tracking point and history information of the transfer.

  Based on the input, the transfer amount calculation unit 701 determines whether or not the tracking point determined by the tracking point determination unit 57 of the tracking processing unit 41 has been changed to a transfer point, and according to the determination result. Position information of the tracking point (hereinafter also referred to as the current tracking point) determined in the current frame by the tracking point determination unit 57 and the tracking point (hereinafter referred to as the past) determined in the previous frame stored in the tracking point storage unit 702. A transfer vector is calculated based on the position information of the tracking point) and the motion vector corresponding to the current tracking point.

  That is, the transfer vector is the amount of movement from the past tracking point to the transfer point, which is obtained based on the past tracking point and the transfer point. In the example of FIG. 40, an example of calculating a transfer vector is shown. However, the transfer vector itself can be acquired from the tracking processing unit 41, and the transfer vector can be used.

  In the tracking point storage unit 702, the position information of the current tracking point input to the transfer amount calculation unit 701 is stored as the position information of the past tracking point for the calculation of the next frame.

  The reflected transfer amount calculation unit 703 is preset by the transfer vector calculated by the transfer amount calculation unit 701, the transfer remaining vector in the previous frame stored in the transfer remaining amount storage unit 704, the control unit 29, and the like. Based on the amount of motion that can be reflected in the frame calculation, that is, the step amount that is the maximum reflected transfer vector, the transfer change vector to be reflected in the current frame and the transfer remaining vector to be left in the next frame are calculated. To do.

  The remaining transfer vector calculated by the reflected transfer amount calculation unit 703 is stored in the remaining transfer amount storage unit 704 for the calculation of the next frame.

  The remaining transfer amount storage unit 704 stores the remaining transfer vector calculated in the previous frame by the reflected transfer amount calculation unit 703.

  The center point setting unit 705 creates a presentation position, that is, a zoom image, which serves as a reference when presenting the tracking of the object (that is, the tracking result) based on the reflected transfer vector calculated by the reflected transfer amount calculating unit 703. The presentation center point that is the center of zooming when calculating is calculated.

  Specifically, the center point setting unit 705 determines the current tracking point or the center determined by the tracking point determination unit 57 of the tracking processing unit 41 based on the reflection transfer vector calculated by the reflection transfer amount calculation unit 703. The presentation center point (hereinafter referred to as past center point) in the previous frame stored in the point storage unit 706, the reflection transfer vector calculated by the reflection transfer amount calculation unit 703, and the motion estimation unit of the tracking processing unit 41 Using the motion vector from 52, the presentation center point in the current frame (hereinafter referred to as the current center point) is calculated. The position information of the current center point calculated by the center point setting unit 705 is output to the zoom image creating unit 24 as presentation position information.

  The center point storage unit 706 stores the calculated position information of the current center point as the position information of the past center point for the calculation of the next frame.

  Next, details of the presentation position calculation process in step S22 of FIG. 4 will be described with reference to the flowcharts of FIGS.

  In step S701, the control unit 29 sets initial values for the remaining transfer vector f, the past center point Rn-1, and the past tracking point Pn-1. That is, 0 is set in the remaining transfer vector f in the remaining transfer amount storage unit 704, and the past center point Rn-1 in the center point storage unit 706 and the past tracking point Pn-1 in the tracking point storage unit 702 are initialized. The tracking set point is set.

  When the tracking point is determined by the tracking point determination unit 57 in step S125 of FIG. 10, information regarding the determined tracking point is input. That is, as described above, the transfer amount calculation unit 701 and the center point setting unit 705 transfer the tracking point position information and the determined tracking point in step S129 of FIG. The tracking point transfer history information indicating whether or not the data has been updated is input.

  In step S702, the transfer amount calculation unit 701 determines whether or not the current tracking point Pn determined by the tracking point determination unit 57 has been changed based on the input information regarding the tracking point. If it is determined in step S702 that the current tracking point Pn has been changed, the process proceeds to step S703, and the transfer amount calculation unit 701 is stored in the current tracking point Pn and the tracking point storage unit 702. The transfer vector d is calculated using the past tracking point Pn−1 and the motion vector m from the motion estimation unit 52.

  As shown in FIG. 43, when the current tracking point Pn has been changed, the current tracking point Pn is determined not to be able to be estimated in step S124 in FIG. At step S123, the motion vector m is estimated for the transfer point Q. In step S125, the motion vector m is shifted by the estimated motion vector m. This is obtained by subtracting the past tracking point Pn-1 from the current tracking point Pn and further subtracting the motion vector m.

  That is, since the current tracking point Pn is the transfer point Q shifted by the motion vector m, the transfer vector d is obtained based on the past tracking point Pn-1 and the transfer point Q. It can also be said that the amount of movement from the position of Pn-1 to the position of the transfer point Q.

  Thus, in step S703, the transfer vector d (= current tracking point Pn−past tracking point Pn−1−motion vector) is calculated.

  On the other hand, if the current tracking point Pn has not been changed, that is, if it is a tracking point obtained as a result of the processing in steps S122 to S125 of FIG. 10 being continuously performed, the process proceeds to step S704. The transfer amount calculation unit 701 sets 0 (0 vector) as the transfer vector d.

  The transfer vector d obtained in step S703 or S704 is input to the reflected transfer amount calculation unit 703.

  In step S705, the reflected transfer amount calculation unit 703 determines that the absolute value of the sum of the transfer vector d from the transfer amount calculation unit 701 and the transfer remaining vector fn-1 in the previous frame read from the transfer remaining amount storage unit 704 is It is determined whether or not the amount is greater than or equal to the amount. If it is determined in step S705 that the absolute value of the sum of the transfer vector d and the remaining transfer vector fn-1 is greater than or equal to the step amount, the process proceeds to step S706, and the reflected transfer amount calculation unit 703 further transfers the transfer amount. It is determined whether or not the sum of the vector d and the remaining transfer vector fn-1 is greater than zero.

  If it is determined in step S706 that the sum of the transfer vector d and the remaining transfer vector fn-1 is larger than 0 (that is, not a negative movement), the process proceeds to step S707, and the reflected transfer amount calculation unit 703 Uses the step amount, which is the maximum reflected transfer vector to be reflected this time, to determine the reflected transfer vector e, and uses the difference obtained by subtracting the step amount from the sum of the transfer vector d and the transfer remaining vector fn-1 of the previous frame. Then, a transfer remaining vector fn is obtained.

  That is, in step S707, the reflected transfer vector e (= step amount) and the remaining transfer vector fn (= transfer vector d + transfer remaining vector fn-1−step amount) are calculated.

  If it is determined in step S706 that the sum of the transfer vector d and the remaining transfer vector fn-1 is 0 or less (that is, the motion is negative), the process proceeds to step S708, and the reflected transfer amount calculation unit 703 Uses the step amount to determine the reflected transfer vector e, and uses the sum of the transfer vector d and the remaining transfer vector fn-1 of the previous frame plus the step amount to determine the remaining transfer vector fn.

  That is, in step S708, the reflected transfer vector e (= −step amount) and the remaining transfer vector fn (= transfer vector d + transfer remaining vector fn−1 + step amount) are calculated.

  On the other hand, if it is determined in step S705 that the absolute value of the sum of the transfer vector d and the remaining transfer vector fn-1 is smaller than the step amount, the process proceeds to step S709, where the transfer vector d and the remaining transfer vector of the previous frame are transferred. The reflected transfer vector e is obtained using the sum of fn−1, and 0 (0 vector) is set as the remaining transfer vector fn.

  That is, in step S709, the reflected transfer vector e (= transfer vector d + transfer remaining vector fn−1) and transfer remaining vector fn (= 0) are calculated.

  The reflected transfer vector e obtained in step S707, 708, or 709 is input to the center point setting unit 705. At this time, the remaining transfer vector fn is stored in the remaining transfer amount storage unit 704 as the remaining transfer vector fn-1 of the previous frame.

  In step S710 of FIG. 42, the center point setting unit 705 determines whether or not the reflected transfer vector e obtained by the reflected transfer amount calculation unit 703 is 0, and determines that the reflected transfer vector e is 0. In step S711, the current tracking point Pn is set as the current center point Rn.

  That is, since the current tracking point Pn has not been changed, there is no transfer vector d, and there is also no remaining transfer vector fn-1 from before the previous frame, so the current obtained by the tracking point determination unit 57 The tracking point Pn is set as it is as the current center point Rn.

  On the other hand, when it is determined that the reflected transfer vector e is not 0, in step S712, the center point setting unit 705 reflects the current center point Rn on the past center point Rn-1 read from the center point storage unit 706. This is obtained based on the motion vector m from the motion estimation unit 52 while reflecting the transfer vector e.

  That is, in step S712, the current center point Rn (= past center point Rn-1 + reflecting transfer vector e + motion vector m) is set.

  In the example of FIG. 43, a case where the current center point Rn = current tracking point Pn is set for the first time or in step S711 of the previous frame is shown. In this case, since the past tracking point Pn−1 = the past center point Rn−1, when the reflected transfer vector e is not 0, as shown in FIG. 43, the step amount is set to the past tracking point Pn−1. Alternatively, a point obtained by adding the reflected transfer vector e and the motion vector m obtained from the sum of the transfer vector d and the remaining transfer vector fn−1 of the previous frame is set as the current center point Rn.

  However, if the current center point Rn (= past center point Rn-1 + reflecting transfer vector e and motion vector m) is set in step S712 of the previous frame, the past tracking point Pn-1 is also set in step S712 of the current frame. Instead, a point obtained by adding the reflected transfer vector e and the motion vector m to the past center point Rn−1 is set as the current center point Rn.

  In step 711 or step S712, the set position information of the current center point Rn is output to the zoom image creating unit 24 as presentation position information.

  In step S713, the center point setting unit 705 stores the current center point Rn in the center point storage unit 706 as the past center point Rn-1, and the transfer amount calculation unit 701 stores the current tracking point Pn in the past tracking point. It is stored in the tracking point storage unit 702 as Pn-1. After step S713, the process returns to step S702, and the process for the next frame is repeated until the end of tracking is instructed by the user.

  In addition, as described above, the step amount is described as the preset maximum reflection transfer vector that reflects this time, but the step amount depends on how many steps later the tracking point and the zoom point coincide with each other. And can be variably calculated.

  Further, the step amount can be set to a size linked with the size of the area estimation range determined in the processing in steps S143 and S144 in FIG. 11 or the processing in steps S206 and S208 in FIG.

  Further, in the above description, the zoom point is calculated using a vector as the amount of motion, but the zoom point can also be calculated using an absolute distance.

  As described above, the object tracking unit 23 is provided with a presentation position that serves as a reference when presenting the tracking result (that is, when a zoom image is presented), separately from the tracking point, When changing tracking points, instead of reflecting all the amount of movement changed by the tracking point on the current frame presentation position, it is reflected step by step by a predetermined amount of the movement amount changed by the tracking point. Since it did in this way, the discomfort produced in the image displayed by changing the tracking point can be reduced.

  FIG. 44 shows a coordinate locus of a presentation position in a conventional method and a zoom image according to the present invention. The horizontal axis represents frame numbers 0 to 20 in the time direction, and the vertical axis represents coordinate values 0 to 450.

  In the example of FIG. 44, the dotted line represents the locus of the x and y coordinates of the presentation position according to the conventional method with the tracking point as the presentation position in order from the top, and the solid line represents the presentation position according to the present invention in order from the top The trajectories of the x-coordinate and y-coordinate are respectively shown.

  For both the x-coordinate and y-coordinate trajectories, as shown by the dotted line in the conventional method, there is a frame in which the coordinate value changes abruptly, whereas in the method of the present invention, the solid line shows As you can see, it has a gentle trajectory that does not change rapidly.

  That is, in the conventional method that uses the tracking point obtained in the tracking point calculation process in step S21 as it is as a presentation position that serves as a reference when generating an image (presentation position that becomes the center when a zoomed image is enlarged). When the tracking point is changed, the presentation position is changed as it is from the past tracking point to the current tracking point, so that the coordinate value changes abruptly as shown by the dotted line in FIG. When displaying, particularly when displaying enlarged, a sense of incongruity may occur.

  On the other hand, in the present invention, a presentation position is provided separately from the tracking point, and when the tracking point is changed, the presenting position is determined from the presenting position corresponding to the past tracking point. The presentation position corresponding to the tracking point can be changed step by step. As a result, in the present invention, as shown by the solid line in FIG. 44, the presentation position does not change abruptly. Therefore, the user can display the image displayed when switching the tracking point using the conventional method. The uncomfortable feeling felt can be reduced. This is particularly effective when an image is zoomed and displayed.

  As described above, by configuring the object tracking unit 23 in FIG. 1, the object to be tracked is rotated (FIG. 6), occlusion occurs (FIG. 7), or the object tracking point is changed by a scene change. Even when the image is temporarily not displayed (FIG. 8), the moving object (tracking point) in the image can be accurately tracked.

  Then, the presentation position is calculated based on the tracking point of the object accurately tracked in this manner, and the calculated presentation position information is sent to the zoom image creation unit 24 as a tracking result by the object tracking unit 23 of FIG. As a result of the output, the zoom image creation unit 24 creates a zoom image centered on the presentation position determined to be reflected step by step. As a result, as described above with reference to the graph of FIG. 44, the user feels less uncomfortable with the image displayed on the image display 26.

  That is, in the prior art, a zoom image with a large zoom ratio is more uncomfortable than an image created with a zoom ratio of 1. Therefore, the present invention provides an image created with a zoom ratio of 1. This is more effective than a zoom image with a large zoom ratio.

  The present invention can be applied not only to a television receiver but also to a surveillance camera system and various image processing apparatuses.

  In the above, the image processing unit is a frame, but the present invention can also be applied to a case where a field is a processing unit.

  It does not matter whether the above-described series of processing is realized by hardware or software. When the above-described series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. It is installed on a general-purpose personal computer or the like from a recording medium such as a network or a removable medium.

  In addition, the steps of executing the series of processes described above in this specification are performed in parallel or individually even if they are not necessarily processed in time series, as well as processes performed in time series in the order described. The processing to be performed is also included.

It is a block diagram which shows the structural example of the television receiver to which this invention is applied. It is a flowchart explaining the image display process of the television receiver of FIG. It is a block diagram which shows the structural example of the object tracking part of FIG. It is a flowchart explaining a tracking process. It is a flowchart explaining a tracking point calculation process. It is a figure explaining tracking when a tracking object rotates. It is a figure explaining the tracking when an occlusion occurs. It is a figure explaining the tracking when a scene change occurs. It is a block diagram which shows the structural example of the tracking process part of FIG. It is a flowchart explaining a normal process. It is a flowchart explaining the initialization process of a normal process. It is a figure explaining a transfer candidate extraction process. It is a block diagram which shows the structural example of an area | region estimation related process part. It is a flowchart explaining an area | region estimation related process. It is a flowchart explaining an area | region estimation process. It is a figure explaining the process which determines a sample point. It is a flowchart explaining the update process of an area estimation range. It is a figure explaining the update of a region estimation range. It is a figure explaining the update of a region estimation range. It is a flowchart explaining a transfer candidate extraction process. It is a flowchart explaining a template creation process. It is a figure explaining template creation. It is a figure explaining template creation. It is a figure explaining the positional relationship of a template and a tracking point. It is a flowchart explaining exception processing. It is a flowchart explaining the initialization process of an exception process. It is a figure explaining selection of a template. It is a figure explaining the setting of a search range. It is a flowchart explaining the other example of the initialization process of an exception process. It is a flowchart explaining a continuation determination process. It is a block diagram which shows the structural example of a motion estimation part. It is a flowchart explaining a motion estimation process. It is a figure explaining calculation of activity. It is a figure explaining the relationship between an evaluation value and activity. It is a flowchart explaining an integration process. It is a block diagram which shows the structural example of a background motion estimation part. It is a flowchart explaining a background motion estimation process. 5 is a block diagram illustrating a configuration example of a scene change detection unit 53. FIG. It is a flowchart explaining a scene change detection process. It is a block diagram which shows the structural example of the presentation position calculation part of FIG. It is a flowchart explaining a presentation position calculation process. It is a flowchart explaining a presentation position calculation process. It is a figure explaining presentation position calculation processing. It is a figure explaining the locus | trajectory of a presentation position.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Television receiver, 23 Object tracking part, 24 Zoom image creation part, 26 Image display, 29 Control part, 30 Remote controller, 41 Tracking process part, 42 Presentation position calculation part, 51 Template matching part, 52 Motion estimation part, 53 scene change detection unit, 54 background motion estimation unit, 55 region estimation related processing unit, 56 transfer candidate holding unit, 57 tracking point determination unit, 58 template holding unit, 59 control unit, 701 transfer amount calculation unit, 702 tracking point storage 703, reflected transfer amount calculation unit, 704 remaining transfer amount storage unit, 705 center point setting unit, 706 center point storage unit

Claims (4)

In an image processing apparatus for displaying a moving object,
Motion estimation for estimating the movement of the first point as the tracking point of the moving object on the image in the previous processing unit in time to the position of the second point as the tracking point in the subsequent processing unit in time Means,
If the movement to the position of the second point in the subsequent processing unit cannot be estimated, selection means for selecting another first point from the estimated points as transfer candidates for the first point;
Determining means for determining the second point in the subsequent processing unit based on the movement to the position of the second point estimated for the other first point selected by the selecting means;
Presenting position calculating means for calculating a presenting position based on the position of the first point and the position of the other first point;
Display image creation means for creating a display image based on the presentation position calculated by the presentation position calculation means;
Display control means for controlling the display of the display image created by the display image creating means ,
The presentation position calculation means includes
Comparison determination means for determining whether or not a transfer amount obtained from the position of the first point and the position of the other first point is equal to or greater than a predetermined amount;
When it is determined by the comparison determination means that the transfer amount is equal to or greater than a predetermined amount, only the predetermined amount of the transfer amount is reflected, and the second point estimated for the other first point is reflected. Presentation position setting means for setting the presentation position based on the movement of the point to the position
Image processing device. In an image processing method of an image processing apparatus for displaying a moving object,
Motion estimation for estimating the movement of the first point as the tracking point of the moving object on the image in the previous processing unit in time to the position of the second point as the tracking point in the subsequent processing unit in time Steps ,
When the movement to the position of the second point in the subsequent processing unit cannot be estimated, a selection step of selecting another first point from the estimated points as transfer candidates of the first point;
A determination step of determining the second point in the subsequent processing unit based on the movement to the position of the second point estimated for the selected other first point;
A presentation position calculating step of calculating a presentation position based on the position of the first point and the position of the other first point;
A display image creation step of creating a display image based on the calculated presentation position;
And a display control step for controlling the display of the display image generated seen including,
By the processing of the presentation position calculation step,
Determining whether the transfer amount obtained from the position of the first point and the position of the other first point is equal to or greater than a predetermined amount;
When it is determined that the transfer amount is equal to or greater than a predetermined amount, only the predetermined amount of the transfer amount is reflected, and the position of the second point estimated for the other first point is reflected. Set the presentation position based on movement
Image processing method. A program for causing a computer to display a moving object,
Motion estimation for estimating the movement of the first point as the tracking point of the moving object on the image in the previous processing unit in time to the position of the second point as the tracking point in the subsequent processing unit in time Steps ,
When the movement to the position of the second point in the subsequent processing unit cannot be estimated, a selection step of selecting another first point from the estimated points as transfer candidates of the first point;
A determination step of determining the second point in the subsequent processing unit based on the movement to the position of the second point estimated for the selected other first point;
A presentation position calculating step of calculating a presentation position based on the position of the first point and the position of the other first point;
A display image creation step of creating a display image based on the calculated presentation position;
And a display control step for controlling the display of the display image generated seen including,
By the processing of the presentation position calculation step,
Determining whether the transfer amount obtained from the position of the first point and the position of the other first point is equal to or greater than a predetermined amount;
When it is determined that the transfer amount is equal to or greater than a predetermined amount, only the predetermined amount of the transfer amount is reflected, and the position of the second point estimated for the other first point is reflected. Set the presentation position based on movement
program. A recording medium on which the program according to claim 3 is recorded.

Images