JP2007274543A - Image processing apparatus and method, program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily reset an object to be tracked while continuing tracking. <P>SOLUTION: An imaging unit 11 images an area to be monitored and inputs an image 31 in which an invader B is imaged, to a tracking device 12. The tracking device 12 uses the input image 31 to track the invader B as an object to be tracked in response to the instruction of a user A or performs tracking while resetting the invader B that has been deviated from the object to be tracked, as an object to be tracked in response to the instruction of the user A again and on the basis of the result of the tracking, a zoomed image 32 is generated, for example, and displayed on a display section 21. The present invention can be applied to a monitoring system in which an object moving on a captured image is tracked and monitored. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and method, a program, and a recording medium, and more particularly, to an image processing apparatus and method, a program, and a recording medium that can easily reset a tracking target.

  There have been many techniques for tracking an object designated by a user in a moving image, and the applicant has also proposed in Patent Document 1 filed earlier. Most of them, after first specifying a tracking target, perform tracking processing fully automatically.

  However, in reality, many of these tracking targets are subject to various disturbances such as severe deformation, relatively long occlusion, or noise on the image itself. It was difficult to obtain a desired tracking result by processing.

  On the other hand, in Patent Document 2, it is possible to select a tracking algorithm in advance, select an optimal algorithm for each scene, stop reproduction or pause, and reset a target to obtain a desired algorithm. It has been proposed to obtain tracking results.

JP 2005-303983 A JP 2001-111957 A

  However, in the proposal described in Patent Document 2, when a tracking algorithm is selected in advance, it must be repeated until an optimal algorithm is found, which places a heavy load on the user.

  Also, in order to stop or pause playback and reset the target, a huge storage device is required, which not only increases the cost, but also tracks the tracking result in synchronization with the input video. It was difficult to apply to the output application and it was not general purpose.

  The present invention has been made in view of such a situation, and makes it possible to easily reset a tracking target without stopping or pausing playback.

  An image processing apparatus according to an aspect of the present invention includes a tracking unit that tracks a moving object on an image as a tracking target in response to a user operation in the image processing apparatus that displays a moving object, and the tracking unit. The candidate calculation means for calculating the candidate position as the tracking target candidate according to the above, the display control means for controlling the display of the candidate position calculated by the candidate calculation means, and displayed in response to the user's operation Target setting means for setting the candidate position as the tracking target in the next frame of the tracking means.

  The candidate calculation means can read a predetermined position in a screen stored in advance and calculate the candidate position.

  The candidate calculation means can calculate the candidate position based on the feature amount of the image.

  The candidate calculation means can calculate the candidate position based on the tracking results of the plurality of tracking means.

  The plurality of tracking units can each track using a plurality of different types of tracking methods.

  The target setting unit can set the displayed candidate position as the tracking target in the next frame of the plurality of tracking units in response to a user operation.

  The plurality of tracking units can track a plurality of different neighboring positions on the object as tracking targets.

  The target setting unit is configured to select a plurality of different neighboring positions including the candidate position as the tracking target in the next frame of the plurality of tracking units based on the displayed candidate position in response to a user operation. Each can be set.

  Update means for updating the tracking results of some or all of the plurality of tracking means based on the tracking results of one of the plurality of tracking means can be further provided.

  The updating unit is configured to track a part or all of the plurality of tracking units based on a tracking result by one of the plurality of tracking units every time a predetermined time elapses. Results can be updated.

  The updating means is based on a result of tracking by one of the plurality of tracking means at a first timing when a predetermined time has elapsed, and is based on a part of the plurality of tracking means. The plurality of tracking means are updated with a tracking result by one tracking means of the plurality of tracking means at each second timing at which the predetermined time has elapsed, which is different from the first timing. The tracking result by some other tracking means can be updated.

  The updating unit is configured to perform a part or all of the plurality of tracking units based on a tracking result of one tracking unit among the plurality of tracking units when the tracking results of the plurality of tracking units are greatly different. The tracking result by the tracking means can be updated.

  The display control unit can control a list display of the candidate positions that are displayed on the image while the candidate positions being selected by a user operation are distinguished from other candidate positions.

  The display control means superimposes a first small image on the selected candidate position on the image, and differs from the first small image on the other candidate position on the image. The list display of the candidate positions can be controlled by superimposing two small images.

  The display control means further includes an image generation means for generating a zoom image centered on the candidate position, and can control the display of the zoom image centered on the candidate position generated by the image generation means. .

  The display control unit can control display of a plurality of zoom images centered on the plurality of candidate positions generated by the image generation unit.

  In the zoom image centered on the candidate position generated by the image generating means, the display control means shows the candidate position being selected by the user's operation on the image, distinguished from other candidate positions. The display on which the list of candidate positions is superimposed can be controlled.

  The display control means includes the candidate generated by the image generation means on a list display of the candidate positions, the candidate positions being selected by a user operation being distinguished from other candidate positions and displayed on the image. The display on which the zoom image centered on the position is superimposed can be controlled.

  An image processing method according to an aspect of the present invention is an image processing method of an image processing apparatus that displays a moving target, and is a tracking unit that tracks a moving object on an image as a tracking target in response to a user operation. The candidate position as the tracking target candidate is calculated, the display of the calculated candidate position is controlled, and the displayed candidate position in response to a user operation is displayed in the next frame of the tracking means. Including a step of setting as a tracking target.

  A program according to one aspect of the present invention is a program that causes a computer to perform a process of displaying a moving target, and is a tracking unit that performs tracking using a moving object on an image as a tracking target in response to a user operation. The candidate position as the tracking target candidate is calculated, the display of the calculated candidate position is controlled, and the displayed candidate position in response to a user operation is displayed in the next frame of the tracking means. Including a step of setting as a tracking target.

  A program recorded on a recording medium according to an aspect of the present invention is a program that causes a computer to perform a process of displaying a moving target, and that tracks a moving object on an image in response to a user operation. The candidate position as the tracking target candidate of the tracking means that performs tracking is calculated, the display of the calculated candidate position is controlled, and the candidate position displayed in response to a user operation is Setting as the tracking target in the next frame of the means.

  In one aspect of the present invention, in response to a user operation, a candidate position as the tracking target candidate of a tracking unit that performs tracking with a moving object on the image as a tracking target is calculated, and the calculated candidate The position display is controlled. In response to the user's operation, the displayed candidate position is set as the tracking target in the next frame of the tracking means.

  According to the present invention, it is possible to easily reset the tracking target while continuing tracking.

  Embodiments of the present invention will be described below. Correspondences between constituent elements of the present invention and the embodiments described in the specification or the drawings are exemplified as follows. This description is intended to confirm that the embodiments supporting the present invention are described in the specification or the drawings. Therefore, even if there is an embodiment which is described in the specification or the drawings but is not described here as an embodiment corresponding to the constituent elements of the present invention, that is not the case. It does not mean that the form does not correspond to the constituent requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

  According to an image processing apparatus of one aspect of the present invention, in an image processing apparatus (for example, the tracking apparatus 12 in FIG. 1) that displays a moving object, a moving object on the image is set as a tracking target in response to a user operation. Tracking means for performing tracking (for example, the tracking processing unit 71 in FIG. 3), candidate calculation means for calculating candidate positions as candidates for the tracking target by the tracking means (for example, the position calculating unit 82 in FIG. 3), Display control means for controlling the display of the candidate position calculated by the candidate calculation means (for example, the display image generation unit 54 in FIG. 2), and the candidate position displayed in response to a user operation, Target setting means (for example, a target position setting unit 83 in FIG. 3) that is set as the tracking target in the next frame of the tracking means.

  The candidate calculation means (for example, the position calculation unit 82 in FIG. 3 that performs the process of step S32 in FIG. 5) can read the predetermined position in the screen stored in advance and calculate the candidate position.

  The candidate calculation means (for example, the image feature amount calculation unit 131 in FIG. 7 that performs the process of step S52 in FIG. 8) can calculate the candidate position based on the feature amount of the image.

  The candidate calculation means (for example, the position calculation unit 82 in FIG. 9 that performs the processing in step S72 in FIG. 10) is tracked by a plurality of tracking means (for example, the tracking processing units 71-1 to 71-n in FIG. 9). Based on the result, the candidate position can be calculated.

  Update means (for example, the tracking result of FIG. 12) for updating the tracking results of some or all of the plurality of tracking means based on the tracking results of one tracking means of the plurality of tracking means. An updating unit 161) can be further provided.

  The display control means displays a list of candidate positions that are displayed on the image with candidate positions being selected by a user operation distinguished from other candidate positions (for example, display of candidate list images 241 in FIG. 25). Can be controlled.

  The display control means superimposes a first small image (for example, the cursor P in FIG. 25) on the selected candidate position on the image, and the other control position on the other candidate position on the image. A list display of the candidate positions can be controlled by superimposing a second small image (for example, point R in FIG. 25) different from the first small image.

  The display control unit further includes an image generation unit (for example, an enlarged signal processing unit 301 in FIG. 27) that generates a zoom image centered on the candidate position, and the candidate position generated by the image generation unit is centered. The display of the zoom image (for example, the zoom image 351-1 in FIG. 30) can be controlled.

  The display control unit can control display of a plurality of zoom images (for example, a plurality of zoom images 371-1 in FIG. 32) centered on the plurality of candidate positions generated by the image generation unit.

  In the zoom image centered on the candidate position generated by the image generating means, the display control means shows the candidate position being selected by the user's operation on the image, distinguished from other candidate positions. The display on which the list display of the candidate positions is superimposed (display of the display image 361-1 in FIG. 31) can be controlled.

  The display control means includes the candidate generated by the image generation means on a list display of the candidate positions, the candidate positions being selected by a user operation being distinguished from other candidate positions and displayed on the image. The display in which the zoom image centered on the position is superimposed (display of the display image 391-1 in FIG. 33) can be controlled.

  According to an image processing method or program of an aspect of the present invention, in the image processing method or program of an image processing apparatus for displaying a moving target, tracking is performed using a moving object on the image as a tracking target in response to a user operation. The candidate position as the tracking target candidate of the tracking means to be performed is calculated (for example, step S3 in FIG. 4), the display of the calculated candidate position is controlled (for example, step S9 in FIG. 4), and the user's operation The candidate position to be displayed is set as the tracking target in the next frame of the tracking means (for example, step S7 or S8 in FIG. 4).

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows a configuration example when the present invention is applied to a monitoring system. In this monitoring system, using an imaging device 11 made up of a CCD (Charge Coupled Devices) video camera or the like, and a tracking device 12 connected to the imaging device 11 and having a display unit 21 made up of an LCD (Liquid Crystal Display) or the like, The user A, who is a monitor, monitors whether there is a suspicious person in a predetermined space while viewing an image picked up by the image pickup device 11 and displayed on the display unit 21.

  The imaging device 11 captures an area to be monitored in the installed space and inputs the image 31 to the tracking device 12. For example, if there is an intruder B in the monitored area, an image 31 in which the intruder B is captured is input to the tracking device 12.

  The tracking device 12 uses the input image 31 to perform tracking for the intruder B as a tracking target in response to the instruction of the user A, or for the intruder B that has deviated from the tracking target again to the user A In response to this instruction, the tracking target is reset and tracking is performed. Based on the tracking result, for example, a zoomed image 32 is generated and displayed on the display unit 21.

  The display unit 21 displays the image 32 generated by the tracking device 12 based on the tracking result.

  In the monitoring system of FIG. 1, a camera driving unit that drives the imaging device 11 such that the imaging device 11 captures an image centered on the tracking target may be configured based on control from the tracking device 12. Is possible.

  FIG. 2 is a block diagram illustrating a configuration example of the tracking device 12 of FIG. The tracking device 12 includes an input terminal 51, an object tracking unit 52, an overall system control unit 53, a display image generation unit 54, a display unit 21, a remote controller 55, and a removable medium 56.

  An image captured by the imaging device 11 is input to the object tracking unit 52 and the display image generation unit 54 as an input image via the input terminal 51. The object tracking unit 52 executes processing for tracking an object designated as a tracking target by the user from the input image, and displays display tracking information based on the tracking result via the overall system control unit 53. To 54.

  The overall system control unit 53 is configured by, for example, a microcomputer, receives user operation information input via the remote controller 55, and supplies the user operation information to the object tracking unit 52 and the display image generation unit 54, thereby instructing the user. Each part is controlled based on the above. Further, the overall system control unit 53 supplies the display tracking information from the object tracking unit 52 to the display image generation unit 54.

  The display image generation unit 54 generates a display image using the input image according to the user operation information from the overall system control unit 53 and the display tracking information obtained by the tracking processing of the object tracking unit 52, and displays the display image. The image is displayed on the display unit 21.

  The remote controller 55 is operated by the user and transmits user operation information (for example, coordinate position and candidate selection information) corresponding to the user operation to the overall system control unit 53 using light such as infrared rays or radio waves. . Note that the remote controller 55 can be configured by, for example, a keyboard or a mouse.

  The removable medium 56 is composed of a semiconductor memory, a magnetic disk, an optical disk, a magneto-optical disk, and the like, and is mounted as necessary, and provides a program and various other data to the overall system control unit 53.

  FIG. 3 is a block diagram illustrating a detailed configuration example of the object tracking unit 52. The object tracking unit 52 of FIG. 3 includes a tracking processing unit 71 and a tracking processing control unit 72.

  The tracking processing unit 71 performs tracking based on the setting information generated by the tracking processing control unit 72 and outputs a tracking result to the tracking processing control unit 72. Although details of the tracking process will be described later, for example, as a tracking method by the tracking processing unit 71, a luminance waveform block matching method, a color waveform block matching method, a point tracking method with transfer described in Patent Document 1, and a motion region centroid tracking A method or a method of performing extrapolation for a certain period of time based on past movements can be used.

  The tracking processing control unit 72 includes, for example, a tracking result storage unit 81, a position calculation unit 82, and a target position setting unit 83. The tracking processing control unit 72 controls the tracking processing unit 71 on the basis of user operation information and the tracking processing unit 71. The display tracking information is generated on the basis of the tracking result of, and the display tracking information is output to the overall system control unit 53. The tracking information for display includes, for example, position information of a tracking target area or point being tracked and position information of at least one tracking target candidate area or point.

  The tracking result storage unit 81 stores the tracking result of the tracking processing unit 71. Based on the tracking result stored in the tracking result storage unit 81, the position calculation unit 82 inputs the tracking target position of the next frame (next frame) and the tracking target that are input after the current frame (current frame). Is calculated, the display tracking information is generated, and the display tracking information is output to the overall system control unit 53.

  Further, the position calculation unit 82 supplies candidate position information selected by the user and instructed to be changed to the target position setting unit 83 based on the generated display tracking information and user operation information. The target position setting unit 83 sets the position indicated by the user operation information, or the candidate position or the tracking target position from the position calculation unit 82 as the tracking target position of the next frame, and sets the setting information as the tracking processing unit. To 71.

  When the change is not instructed, the position calculation unit 82 supplies the tracking target position information calculated based on the tracking result to the target position setting unit 83. That is, in the target position setting unit 83, the tracking result of the tracking processing unit 71 is set as the tracking target position of the next frame as it is.

  Next, the processing of the tracking device 12 will be described with reference to the flowchart of FIG. When the power of the monitoring system in FIG. 1 is turned on, the imaging device 11 captures an area to be monitored. An image obtained by imaging is input to the tracking device 12 and displayed on the display unit 21 via the display image generation unit 54.

  The user A refers to the image displayed on the display unit 21 and operates the remote controller 55 to designate an object to be tracked on the image as a tracking target and instruct to start tracking. When this operation is performed, the overall system control unit 53 starts the process of FIG. That is, the overall system control unit 53 supplies user operation information to the tracking process control unit 72.

  In step S <b> 1, the target position setting unit 83 of the tracking processing control unit 72 sets the position indicated by the user operation information as the tracking target position, and outputs the setting information to the tracking processing unit 71. In step S2, the tracking processing unit 71 waits for the input of the next frame, and between the input next frame and the current frame, the tracking target designated by the user based on the setting information from the target position setting unit 83 The tracking process is started.

  The details of the tracking process will be described later with reference to FIG. 35 or 39. By this process, the tracking target specified by the user, that is, the object to be tracked in the image (for example, a person, an animal, etc.) ) Tracking point (or tracking area) is tracked, and the tracking result is output to the tracking processing control unit 72 and stored in the tracking result storage unit 81.

  In step S <b> 3, the position calculation unit 82 of the tracking process control unit 72 performs position calculation processing for calculating the positions of the tracking target and the tracking target candidate based on the tracking result by the tracking processing unit 71. The details of the position calculation process will be described later with reference to FIG. 5. With this process, the position of the tracking target and the position of the tracking target candidate are calculated, display tracking information is generated, and the display tracking information is The display image generation unit 54 is supplied via the overall system control unit 53.

  In step S4, the position calculation unit 82 determines whether or not there is an instruction for candidate position selection display from the user, and when it is determined that there is an instruction for candidate position selection display, that is, the candidate position selection display is instructed. In step S5, it is determined whether there is an instruction to select a candidate position from the user.

  If it is determined in step S5 that there is an instruction to select a candidate position, the process proceeds to step S6. In step S6, the position calculation unit 82 determines whether there is an instruction to change from the user to the candidate position. judge. If it is determined in step S6 that there is an instruction to change from the user to the candidate position, the target position setting unit 83 sets the candidate position as the tracking target position of the next frame in step S7.

  That is, according to a user's instruction for selecting and displaying a candidate position, the display unit 21 currently performs candidate position selection display, which is a display for selecting a candidate position, and the user selects one candidate position. Has been. Since the position calculation unit 82 supplies the selected candidate position information to the target position setting unit 83, the target position setting unit 83 sets the selected candidate position as the tracking target position of the next frame. The setting information is output to the tracking processing unit 71.

  On the other hand, when it is determined in step S4 that there is no instruction for candidate position selection display from the user, in step S5, when it is determined that there is no instruction for candidate position selection from the user, or in step S6, the candidate is selected from the user. If it is determined that there is no instruction to update the position, the process proceeds to step S8.

  In step S8, the target position setting unit 83 sets the tracking target position as it is as the tracking target position of the next frame.

  That is, at present, the candidate position selection display is not performed on the display unit 21. Even if the candidate position selection display is performed on the display unit 21, the candidate position is not selected. Even if the candidate position is selected, if there is no update instruction to the candidate position, the position calculation unit 82 supplies the target position setting unit 83 with the position information of the tracking target calculated in step S3. . Correspondingly, in step S8, the target position setting unit 83 sets the position of the tracking target calculated in step S3 as the position of the tracking target of the next frame as it is, and sets the setting information as the tracking processing unit 71. Output to.

  In step S <b> 9, the display image generation unit 54 generates a display image according to a user instruction and outputs the display image to the display unit 21.

  In other words, when there is no instruction for candidate position selection display from the user, the display image generation unit 54 generates a display image showing only the position of the tracking target using the input image, and displays the candidate position selection display from the user. When there is an instruction, a display image showing the candidate position is generated together with the position of the tracking target using the input image. In addition, when there is an instruction to select a candidate position from the user, the display image generation unit 54 uses the input image to superimpose a mark indicating the selection on the selected candidate position, for example, A display image showing the position and the candidate position is generated. Details of the display image will be described with reference to FIGS. 24 to 26 together with a method of operating the remote controller 55 by the user.

  In step S10, the overall system control unit 53 determines whether or not to end the process based on an instruction from the user. If the end of tracking is not instructed by the user, the process returns to step S2, and thereafter Repeat the process. That is, in step S2, the tracking target tracking process designated by the user based on the setting information in which the position of the tracking target set in step S7 or S8 is set between the input next frame and the current frame. Is started. When the end of tracking is instructed by the user, it is determined to end in step S10, and the overall system control unit 53 ends the process.

  Next, the position calculation process in step S3 in FIG. 4 will be described with reference to the flowchart in FIG. In the example of FIG. 5, an example in which a fixed position in the screen is calculated as a candidate position will be described.

  In this case, the coordinates of the fixed position in the screen are stored in advance as candidate positions in a register (not shown) built in the position calculation unit 82 of FIG. Specifically, for example, the coordinates of the center in the screen and the coordinates of the center in each divided screen that is obtained by dividing the screen into n (for example, four) are stored as candidate positions. When the candidate position is selected and changed by the user, one of these fixed positions (selected one) is set as the tracking target position.

  In step S31, the position calculation unit 82 calculates the position of the tracking target using the tracking result stored in the tracking result storage unit 81 as it is. In step S <b> 32, the position calculation unit 82 calculates a candidate position by reading the coordinates of the fixed position from a register (not shown).

  In step S <b> 33, the position calculation unit 82 generates display tracking information using the position of the tracking target calculated in step S <b> 31 and the candidate position calculated in step S <b> 32, and outputs the display tracking information to the overall system control unit 53. . The display tracking information output in step S <b> 33 is output to the display image generation unit 54 via the overall system control unit 53.

  Thereby, when the candidate position selection display is instructed by the user in step S9 of FIG. 4 described above, the display generated by the display image generation unit 54 according to the display tracking information output in step S33. An image 102 is displayed on the display unit 21.

  FIG. 6 shows an example of a display image displayed on the display unit 21. In the example of FIG. 6, a display image 101 displayed at the start of the tracking process and a display image 102 displayed during the tracking process are shown.

  In the lower left portion of the display image 101, a cursor P indicating a tracking target position is displayed on a person object (hereinafter simply referred to as a person) 111 designated by the user as a tracking target.

  On the other hand, in the upper left, lower left, upper right, and lower right of the display image 102, a tree object (hereinafter simply referred to as a tree) 121, a spherical object (hereinafter simply referred to as a sphere) 122, a person 111, And dog objects (hereinafter simply referred to as dogs) 123 are displayed. That is, in the display image 102, the person 111 designated as the tracking target by the user is moved and displayed in the upper right part, and the cursor P indicating the tracking target position is separated from the person 111 and is displayed on the display image 102. It is displayed in the upper left.

  Further, in the display image 102, points Q1 to Q5 each indicating a candidate position are candidate positions calculated by being read from a register built in the position calculation unit 82, the center position in the screen, and The screen is divided into four equal parts and displayed on the center positions of the upper left, upper right, lower left, and lower right divided screens. In addition, in the display image 102, characters “candidate 1” to “candidate 5” are displayed together with the points Q1 to Q5 in order to make the user recognize that the points Q1 to Q5 indicate candidate positions. ing. The display of these characters can be hidden.

  That is, at the start of the tracking process, as shown in the display image 101, the position of the cursor P on the person 111 designated as the tracking target by the user is set as the tracking target position, and the process is started. During the tracking process after a lapse of time, as shown in the display image 102, the position of the tracking target deviates from the person 111 designated as the tracking target by some disturbance such as occlusion. As a result, the position indicated by the cursor P between the tree 121 and the dog 123 has become the tracking target position.

  At this time, since the point Q3 indicating the candidate position is just displayed on the tracking target 111, the user selects the candidate position indicated by the point Q3 and instructs the change. Corresponding to this, the position calculation unit 82 uses the candidate position indicated by the point Q3, that is, the position of the center of the upper right divided screen read from the register as the position of the tracking target. The setting unit 83 is set. Thereby, from the next frame, tracking is resumed with the object including the position indicated by the point Q3, that is, the person 111 as the tracking target, and the display unit 21 uses the tracking result obtained by tracking. The position of the tracked target is indicated by a cursor P.

  As described above, by calculating and displaying the candidate positions that are candidates for the tracking target in advance, even if the position of the tracking target deviates from the tracking target set at the start of processing, the user is determined to be the tracking target. When the candidate position is indicated on the object to be desired, the user can easily select the candidate position of the display image displayed on the display unit 21 and change it without performing fine adjustments. The tracking target can be reset.

  That is, if a reliable candidate position can be calculated as the tracking target position, the candidate position is necessarily displayed on the object desired by the user as the tracking target. Next, a more reliable method for calculating candidate positions will be described.

  FIG. 7 shows another configuration example of the object tracking unit 52 of FIG. The object tracking unit 52 of FIG. 7 is common to the object tracking unit 52 of FIG. 3 in that it includes a tracking processing unit 71 and a tracking processing control unit 72, but an image feature amount calculation unit 131 is added. The point is different.

  That is, in the example of FIG. 7, the candidate position is calculated by the image feature amount calculation unit 131, so that the position calculation unit 82 determines the position of the tracking target of the next frame based on the tracking result of the tracking result storage unit 81. The display tracking information is generated using the calculated tracking target position and the candidate position calculated by the image feature amount calculation unit 131, and the display tracking information is output to the overall system control unit 53.

  The image feature amount calculation unit 131 calculates an image feature amount using the input image, calculates a candidate position based on the image feature amount, and outputs the position information to the position calculation unit 82.

  Specifically, for example, as an image feature, region division is performed by color, and a region having a large area to some extent is extracted with a representative color in the image. Then, the center of gravity of each extracted region is set as the target candidate position. It should be noted that, when performing area division, there is a possibility that the number of areas may be too large, so it is necessary to consider setting a limit on the number of candidates.

  At this time, further, feature quantities such as details and magnitude of movement are also extracted, and by adding these conditions, candidate positions that are likely to be selected by the user, that is, more reliable candidate positions for tracking Can be calculated.

  Next, the position calculation process executed by the tracking process control unit 72 of FIG. 7 will be described with reference to the flowchart of FIG.

  In step S <b> 51, the position calculation unit 82 calculates the position of the tracking target using the tracking result stored in the tracking result storage unit 81 as it is. In step S52, the image feature amount calculation unit 131 obtains an image feature amount using the input image, and calculates a candidate position based on the image feature amount. That is, the candidate position is calculated from the region extracted based on the image feature amount, and the position information is output to the position calculation unit 82.

  In step S53, the position calculation unit 82 generates display tracking information using the position of the tracking target calculated in step S51 and the candidate position calculated by the image feature amount calculation unit 131 in step S52. Output to the system control unit 53. The display tracking information output in step S53 is output to the display image generation unit 54 via the overall system control unit 53.

  Accordingly, in the above-described step S9 of FIG. 4, when the candidate position selection display is instructed by the user, the display generated according to the display tracking information output in step S53 by the display image generation unit 54 An image is displayed on the display unit 21.

  As described above, since the candidate position is calculated based on the feature amount of the input image, the candidate position can be selected by the user more than in the case of the fixed position of the screen, that is, the candidate position that is reliable as the tracking target. Can be displayed.

  FIG. 9 shows still another configuration example of the object tracking unit 52 of FIG. 9 is common to the object tracking unit 52 in FIG. 3 in that the object tracking unit 52 includes a tracking processing unit 71 and a tracking processing control unit 72, but the tracking unit 71 in FIG. The tracking processing units 71-1 to 71-n (n> 1) are different.

  That is, each of the n tracking processing units 71-1 to 71-n is based on the setting information generated by the tracking processing control unit 72, and each of the types of tracking methods described above with reference to FIG. Tracking is performed using different tracking methods (for example, tracking method A, tracking method B,... For simplicity), and the tracking result is output to tracking processing control unit 72. Details of these tracking processing units 71 will be described later with reference to FIG. 34 or FIG.

  Since the tracking result storage unit 81 stores n tracking results from the tracking processing units 71-1 to 71-n, the position calculation unit 82 stores the tracking result storage unit 81 in the tracking result storage unit 81. From one of the tracking results, the position of the tracking target of the next frame is obtained using one tracking result, the candidate position is calculated using the other tracking results, and display tracking information is generated and displayed. The tracking information is output to the overall system control unit 53. Further, the position calculation unit 82 supplies candidate position information selected by the user and instructed to be changed to the target position setting unit 83 based on the generated display tracking information and user operation information.

  The target position setting unit 83 sets the position indicated by the user operation information, or the candidate position or the position of the tracking target from the position calculation unit 82 as the position of the tracking target of the next frame, and the setting information is set for each tracking process. The data are output to the units 71-1 to 71-n, respectively.

  If no change is instructed, the position calculation unit 82 supplies the calculated position information of the tracking target to the target position setting unit 83 for the corresponding one tracking processing unit, and the calculated candidate The position information is supplied to the target position setting unit 83 as for the corresponding other tracking processing unit. That is, in the target position setting unit 83, the tracking result of each tracking processing unit 71 is set as the tracking target position of the next frame as it is.

  Next, the position calculation process executed by the tracking process control unit 72 of FIG. 9 will be described with reference to the flowchart of FIG.

  In step S71, the position calculation unit 82 uses one tracking result (that is, the tracking result corresponding to the candidate position designated by the user) from the tracking results stored in the tracking result storage unit 81 to track. Calculate the position of the object. Note that, at the first time, the tracking target 71 calculates the position of the tracking target by using the tracking result by the tracking processing unit 71 that performs the tracking processing in the tracking method preset in the tracking device 12.

  In step S <b> 72, the position calculation unit 82 calculates a candidate position using the remaining tracking results from the tracking results stored in the tracking result storage unit 81.

  In step S73, the position calculation unit 82 generates display tracking information using the position of the tracking target calculated in step S71 and the candidate position calculated in step S72, and outputs the display tracking information to the overall system control unit 53. . The display tracking information output in step S <b> 73 is output to the display image generation unit 54 via the overall system control unit 53.

  Thereby, when the candidate position selection display is instructed by the user in step S9 of FIG. 4 described above, the display generated according to the display tracking information output in step S73 by the display image generation unit 54. An image is displayed on the display unit 21.

  FIG. 11 shows an example of a display image displayed on the display unit 21. In the example of FIG. 11, as in the case of the example of FIG. 6, for example, a display image 101 displayed at the start of the tracking process and a display image 151 displayed during the tracking process are shown.

  For example, as shown in the display image 101 in FIG. 11, when the user indicates the position indicated by the cursor P on the person 111 as the tracking target at the start of the tracking process, the target position setting unit 83 sets the position of the cursor P. The tracking target position is set, and the setting information is supplied to the tracking processing units 71-1 to 71-n that perform the tracking processing in each of the tracking methods A to E, respectively. In response to this, the tracking processing units 71-1 to 71-n perform tracking by the tracking methods A to E with the position indicated by the cursor P as the tracking target position.

  In the case of the example in FIG. 11, the position of the tracking target is calculated from the tracking result of the tracking processing unit 71-1 that performs the tracking process in the tracking method A, and the other tracking processing units 71-2 to 71-5 A candidate position is calculated from the tracking result.

  In the display image 151 after the elapse of a predetermined time from the start of processing, as in the case of the display image 102, the person 111 designated as the tracking target by the user has moved to the upper right part, and the upper left part, the lower left part, In the lower right part, a tree 121, a ball 122, and a dog 123 are displayed.

  In the display image 151, the cursor P indicating the position of the tracking target calculated from the tracking result of the tracking method A (tracking processing unit 71-1) is a position deviated from the person 111 designated as the tracking target by the user. , Displayed between the tree 121 and the dog 123. A point Q2 indicating a candidate position calculated from the tracking result of the tracking method B (tracking processing unit 71-2) is displayed at a position on the person 111 that the user has designated as a tracking target. A point Q3 indicating a candidate position calculated from the tracking result of the tracking method C (tracking processing unit 71-3) is a position between the tree 121 and the ball 122 that is a position deviated from the person 111 designated as the tracking target by the user. It is displayed at the position.

  A point Q4 indicating a candidate position calculated from the tracking result of the tracking method D (tracking processing unit 71-4) is displayed at a position on the dog 123, which is a position deviated from the person 111 designated as the tracking target by the user. ing. A point Q5 indicating a candidate position calculated from the tracking result of the tracking method E (tracking processing unit 71-5) is displayed at a position on the sphere 122, which is a position deviated from the person 111 designated as the tracking target by the user. ing.

  That is, during the tracking process after a predetermined time has elapsed since the start of the tracking process, tracking by the tracking method A is performed, for example, as shown by the position of the cursor P in the display image 151, for example, deformation or occlusion. For this reason, the user has deviated from the person 111 designated as the tracking target.

  Similarly, in tracking by other tracking methods C to E, as indicated by the positions of points Q3 to Q5 in the display image 151, the person 111 designated as the tracking target by the user due to, for example, deformation or occlusion, for example. It has come off.

  At this time, the tracking process using the tracking method B is correctly performed, and a point indicating a candidate position calculated from the tracking result using the tracking method B is displayed on the person 111 instructed as a tracking target by the user. Q2 is displayed. Thereby, the user can select the candidate position indicated by the point Q2 and instruct the change.

  Then, the position calculation unit 82 determines the candidate position indicated by the point Q2, that is, the candidate position calculated from the tracking result of the tracking method B, as the tracking target position of the tracking methods A to E according to the user's instruction. As described above, the target position setting unit 83 sets the target position. Thereby, from the next frame, each tracking using the tracking methods A to E is started again with the object including the position of the point Q2, that is, the person 111 as the tracking target, and the display unit 21 The position of the tracking target calculated using the tracking result of the tracking method B is indicated by the cursor P.

  In the display image 151 in the example of FIG. 11, together with the cursor P, characters of “scheme A” indicating the tracking method A that tracks the position indicated by the cursor P are displayed, and along with the points Q2 to Q5, Although the characters “method B” to “method E” indicating the tracking methods B to E tracking the positions indicated by the points Q2 to Q5 are displayed, these characters can be hidden. It is.

  As described above, for example, when tracking is performed by method A, even if the tracking target is removed due to some disturbance, the tracking result obtained by another method is indicated as a candidate position. Yes.

  That is, when tracking is performed by a plurality of tracking methods having different tracking result tendencies, even if tracking of one tracking method cannot be performed, there is a high possibility that accurate tracking is achieved by another tracking method. Therefore, by displaying the tracking results of a plurality of tracking methods having different tracking result tendencies as candidate positions, it is highly likely that the candidate positions are displayed on the object that the user wants to be tracked.

  Accordingly, the user can easily reset the tracking target without performing fine adjustments by selecting and changing the candidate position of the display image displayed on the display unit 21.

  Here, in the object tracking unit 52 of FIG. 9 described above, when there is an instruction to change the candidate position from the user, all the n tracking methods are tracked with the candidate position selected by the user as the tracking target, respectively. However, if the tracking result of each tracking method is controlled completely independently, after a long time has passed, the tracking target is actually deformed or occluded. For example, there is a possibility that the position of the tracking target is already out of the tracking target designated by the user in the tracking by the respective tracking methods. That is, no candidate position may be displayed on the tracking target desired by the user.

  Therefore, next, the tracking target of the next frame is based on the tracking result constrained by the predetermined constraint condition by giving a predetermined constraint condition to the tracking result instead of performing tracking independently for each tracking method. An example of updating the position of will be described.

  FIG. 12 shows another configuration example of the object tracking unit 52 of FIG. In the example of FIG. 12, the tracking processing unit 71 is common, but the detailed configuration of the tracking processing control unit 72 is different. That is, in addition to the tracking result storage unit 81, the position calculation unit 82, and the position target setting unit 83, a tracking result update unit 161 is added to the tracking processing control unit 72 in FIG.

  The tracking result update unit 161 sets one of the plurality of tracking results stored in the tracking result storage unit 81 as the tracking result of the basic tracking method, and tracks the tracking of the basic tracking method when a predetermined constraint condition is satisfied. The position of the result of the tracking is updated with a part of the other tracking method or with the position of the result.

  As the predetermined constraint condition in the tracking result update unit 161, for example, the passage of time or the magnitude of the difference between the tracking result of the basic tracking method and all other tracking results is used.

  Based on the user operation information, the position calculation unit 82 obtains the position of the tracking target of the next frame from one of the tracking results in the tracking result storage unit 81 and uses the other tracking results. The candidate position is calculated, display tracking information is generated, and the display tracking information is output to the overall system control unit 53. That is, the position calculation unit 82 in FIG. 12 calculates the candidate position using the tracking result in which the position of at least a part of the tracking result of another tracking method is updated.

  Next, the position calculation process executed by the tracking process control unit 72 of FIG. 12 will be described with reference to the flowchart of FIG. In addition, since the process of subsequent steps S92 thru | or S94 is repeated in order to perform the process similar to step S71 thru | or S73 of FIG. 10, the description is abbreviate | omitted suitably.

  By the tracking process in step S2 of FIG. 4, the tracking result storage unit 81 stores the tracking results by the tracking processing units 71-1 to 71-n. In step S91, the tracking result update unit 161 executes a tracking result update process. This tracking result update process is shown in the flowchart of FIG.

  In step S111, the tracking result update unit 161 performs a time counting operation with a built-in timer, and determines whether a predetermined time has elapsed. When it is determined in step S111 that the predetermined time has elapsed, in step S112, the tracking result update unit 161 stores the position of the tracking result of at least some other tracking methods stored in the tracking result storage unit 81. Is updated based on the tracking result of the basic tracking method. That is, the position of the tracking result of at least some other tracking methods is updated with the position of the tracking result of the basic tracking method.

  If it is determined in step S111 that the predetermined time has not elapsed, the process of step S112 is skipped, and the process returns to step S91 of FIG. 13 and proceeds to step S92.

  In step S92, the position calculation unit 82 tracks one tracking result (that is, the tracking result corresponding to the candidate position designated by the user) from the tracking results stored in the tracking result storage unit 81. 71 is used to calculate the position of the tracking target.

  In step S <b> 93, the position calculation unit 82 calculates a candidate position using the remaining tracking results from the tracking results stored in the tracking result storage unit 81. In step S91, when the predetermined time has elapsed, the position of the tracking result of at least some other tracking methods is updated with the position of the tracking result of the basic tracking method, so the frame at the time of update In this case, the candidate position calculated from the updated tracking result indicates the same position as the candidate position calculated from the tracking result of the basic tracking method.

  In step S94, the position calculation unit 82 generates display tracking information using the tracking target position calculated in step S92 and the candidate position calculated in step S93, and outputs the display tracking information to the overall system control unit 53. . The display tracking information output in step S94 is output to the display image generation unit 54 via the overall system control unit 53.

  The tracking result update process of FIG. 14 described above will be described in detail with reference to FIG.

  In the example of FIG. 15, the tracking results by the two tracking methods (tracking method A and tracking method B) at time T are shown in one dimension for convenience of explanation. That is, the horizontal axis represents the passage of time t, and the vertical axis represents the position x. Here, an example will be described in which the tracking method A is the basic tracking method, and the tracking result of another tracking method B is updated every time T.

  First, tracking by the tracking method A and the tracking method B is started at the tracking target position designated by the user. The tracking method A indicated by a solid line and the tracking method indicated by a dotted line with the passage of time. Each position of the tracking result of B may be separated because tracking is performed using a different type of tracking method.

  Therefore, the position of the tracking result by the tracking method B is updated with the position of the tracking result by the tracking method A at the time T when the time T has elapsed from the start of tracking. Similarly, at the time 2T and the time 3T, the position of the tracking result by the tracking method B is updated with the position of the tracking result by the tracking method A.

  As described above, if the user does not attempt to change the tracking target, the tracking result (trajectory) that moves away with time is restrained to the tracking result of a certain tracking method. By matching, it is possible to always obtain a reliable candidate position.

  In the example of FIG. 15, an example in which the tracking method A is fixed to the basic tracking method is shown, but for example, as shown in the example of FIG. 16, the basic tracking method is changed to another tracking in the middle. It is also possible to switch to the method.

  That is, in the example of FIG. 16, an example in which the basic tracking method is switched from the tracking method A to the tracking method B immediately after time 2T is shown.

  Thereby, at time T and time 2T, the position of the tracking result by the tracking method B indicated by the dotted line is updated with the position of the tracking result by the tracking method A indicated by the solid line, but the basic tracking method is the tracking method. At time 3T after switching to B, the position of the tracking result by the tracking method A indicated by the solid line is updated with the position of the tracking result by the tracking method B indicated by the dotted line.

  In the example of FIGS. 15 and 16, an example in which the update interval is constant is shown, but it is not constant and can be made variable.

  Further, an example of the tracking result update process in step S91 in FIG. 13 will be described with reference to the flowchart in FIG. That is, the process of FIG. 17 is another example of the tracking result update process of FIG.

  By the tracking process in step S2 of FIG. 4, the tracking result storage unit 81 stores the tracking results by the tracking processing units 71-1 to 71-n. In step S131, the tracking result update unit 161 obtains the distance between the tracking result position of the basic tracking method and the tracking result position of another tracking method, and determines whether the distance is equal to or greater than a predetermined threshold.

  When it is determined in step S131 that the distance from the position of the tracking result of the basic tracking method is equal to or greater than a predetermined threshold, the tracking result update unit 161 is stored in the tracking result storage unit 81 in step S132. The position of the tracking result of some other tracking method is updated based on the tracking result of the basic tracking method.

  If it is determined in step S131 that the distance from the tracking result of the basic tracking method is not equal to or greater than the predetermined threshold, the process in step S132 is skipped, and the process returns to step S91 in FIG.

  The tracking result update process of FIG. 17 described above will be described in detail with reference to FIG.

  In the example of FIG. 18, the tracking results by the three tracking methods (tracking methods A to C) at time T are shown in one dimension for convenience of explanation. That is, the horizontal axis represents the passage of time t, and the vertical axis represents the position x. Here, an example will be described in which the tracking method A is the basic tracking method, and the other tracking methods B and C are updated when the distance of the tracking result from the tracking result with the other tracking method is greatly separated.

  First, tracking by tracking methods A to C is started at the tracking target position designated by the user, and as time passes, tracking method A indicated by a solid line, tracking method B indicated by a dotted line, and Each position of the tracking result of the tracking method C indicated by the alternate long and short dash line is separated.

  Therefore, the positions of the tracking results of the tracking methods A to C at time t are respectively xa (t), xb (t), and xc (t), and the positions of the other tracking methods B and C are compared with the tracking method A. The average D of the distance to the position is obtained.

  When the average D of the distances between the positions of the other tracking methods B and C and the position of the tracking method A is equal to or greater than a predetermined threshold (Dth) as shown in FIG. 19 (that is, time T) Thus, the position of the tracking result of the tracking methods B and C is updated with the position of the tracking result of the tracking method A. Accordingly, in the example of FIG. 19, at the time T when the average distance D becomes equal to or greater than the predetermined threshold (Dth), the average distance D once becomes 0 (that is, is reset), and the distance again The average D is added over time from 0.

  This determination formula can be expressed by the following formula (1).

  Also in this case, as in the example of FIG. 16, the basic tracking method can be switched halfway. In addition, the variance of the distance between the tracking result positions of the other tracking methods B and C and the tracking result position of the tracking method A is calculated and updated when the variance becomes large. You can also.

  In the above description, the example of using time with reference to FIGS. 15 and 16 and the example of using the magnitude of the difference in tracking results with reference to FIG. 18 have been described as conditions for constraining the tracking result. Either one or both of the constraint conditions may be used.

  Also, for example, when the tracking method A as the basic tracking method is performed in the tracking processing unit 71-1, the tracking processing unit 71-2 and the tracking processing unit 71-3 perform tracking by the same tracking method B. As shown in FIG. 20, the tracking processing unit 71-2 is updated with the tracking result of the tracking method A every 2T time at the first timing (for example, time T, 3T, 5T,...). The tracking processing unit 71-3 is updated with the tracking result of the tracking method A every 2T time at a second timing different from the first timing (for example, times 2T, 4T, 6T,...). It can also be made.

  In the example of FIG. 20, the tracking results by the two tracking methods (tracking methods A and B) at time T are shown in one dimension for convenience of explanation. That is, the horizontal axis represents the passage of time t, and the vertical axis represents the position x. Here, with tracking method A as the basic tracking method, two different update methods of tracking method B (hereinafter referred to as tracking method B-1 and tracking method B-2) at each time 2T, An example of updating at a different timing will be described.

  First, the tracking method A and the tracking method B-1 are both started at the tracking target position designated by the user, and at the time T when the tracking method A has started tracking, the tracking method A Tracking by the tracking method B-2 is also started from the position of the tracking result.

  Then, at the time 2T when the time 2T has elapsed after the tracking methods A and B-1 are started, the position of the tracking result by the tracking method B-1 indicated by the dotted line is tracked by the tracking method A indicated by the solid line. Update with the position of the result. At this time, the position of the tracking result by the tracking method B-2 indicated by the one-dot chain line is not updated, and the position of the tracking result by the tracking method B-2 that is not updated is calculated as a candidate position.

  Furthermore, at time 3T when the time 3T (time 2T since the tracking by the tracking method B-2 was updated) has elapsed since the tracking of the tracking method A was started, the tracking result by the tracking method B-2 indicated by a one-dot chain line Is updated with the position of the tracking result by the tracking method A indicated by the solid line. At this time, the position of the tracking result by the tracking method B-1 indicated by the dotted line is not updated, and the position of the tracking result by the tracking method B-1 that is not updated is calculated as a candidate position.

  Here, for example, immediately after the tracking method B-2 is updated at the time 2T, if a desired tracking result cannot be obtained by the tracking method A which is the basic tracking method, then the tracking method A at the time 3T. The tracking method B-1 updated with the tracking result is matched with the basic tracking method in which a desired tracking result cannot be obtained. On the other hand, the tracking method B-2 that is not updated at this time is updated with the tracking result of the tracking method A before the desired tracking result cannot be obtained, and there is a high possibility that the tracking method B-2 has been correctly tracked. Therefore, by calculating the tracking result of this tracking method B-2 as a candidate position, a highly reliable candidate position can be displayed.

  As described above, if the update timings are all the same, other tracking methods may be all adapted to the basic tracking method when a desired tracking result cannot be obtained immediately after the update, for example. You can get around it.

  Thereby, a more reliable candidate position can be obtained. In order to obtain the above effects, the tracking method used in the plurality of tracking processing units 71-1 to 71-n is not limited, and a plurality of tracking methods having different tracking result tendencies may be prepared. is necessary.

  For example, as a tracking method, a luminance waveform block matching method that performs block matching based on a luminance waveform, a color waveform block matching method that performs block matching based on a color waveform, a point tracking method with transfer described in Patent Document 1, motion A region center-of-gravity tracking method, a method of performing extrapolation for a certain period of time based on past movement, or the like can be given.

  The color waveform block matching method performs the same processing as the luminance waveform block matching method except that the information used is a color instead of a luminance. In the luminance waveform block matching method, if there is a change in luminance in the tracking target, there is a high possibility that it will be excluded from the target, but in the color waveform block matching method, the luminance information is eliminated by using the color information excluding the luminance component. Even if there is a change, there is a high possibility that it has been tracked correctly.

  The point tracking method with transfer will be described in detail later with reference to FIG. 38, but a tracking point transfer candidate is obtained in advance in the previous frame. For example, when no movement is found at the tracking point by block matching, the tracking point By changing to the transfer candidate, the tracking point is temporarily invisible, such as when the tracking target rotates, occlusion occurs, or a scene change occurs.

  The motion region center-of-gravity tracking method will be described in detail later with reference to FIG. 34. However, motion detection (for example, block matching) is performed at a certain sampling interval in the fixed region, and motion similar to the motion occupying the majority in the region is detected. The area to be shown is defined as a tracking target area, and the center of gravity of the area is tracked as the tracking target position.

  The method of performing extrapolation for a certain period of time with the past motion predicts the target motion based on the past motion of a certain tracking method among the tracking methods described above. For example, the tracking result tracked by the luminance waveform block matching method or the like may show a locus as shown in FIG. 21 when the tracking target is moved to the foreground by occlusion.

  In the example of FIG. 21, the locus of the tracking result from time t-5 to time t + 4 is shown. Tracking position x (t-5) to tracking position x (t + 4) represent the tracking results from time t-5 to time t + 4, respectively, and the arrows between the tracking positions indicate movements between the times. Represents. As shown in the tracking position x (t-5) to the tracking position x (t), the gentle movement is continuous from the time t-5 to the time t. The movement from the tracking position x (t) to the tracking position x (t + 1) has become discontinuous due to a cause such as the tracking target moving.

  Therefore, in such a case, as shown in FIG. 22, based on the past movement history shown by the solid line between time t-5 and time t, extrapolation movement shown by the alternate long and short dash line, extrapolation The extrapolated motion determined based on the past motion history from the determined timing (in this case, after the time t in this case) is the actually determined motion indicated by the dotted line (FIG. 21). In place of), the system continues to substitute for a certain time.

  As the extrapolation timing determined at this time, for example, after the user designates the tracking target, the extrapolation timing may be set at regular intervals (for example, every 120 frames), or by looking at the past movement history, Is discontinuous (for example, when it differs greatly from the average motion of several frames in the past).

  Further, the extrapolation motion determined at this time includes a motion several frames before the extrapolation timing, an average motion of past and several frames, and the like. Note that the previous movement may be possible even if it is not several frames before the extrapolation timing, but the previous movement may be a movement at the boundary of the object and may not be appropriate.

  Although the case of tracking using the luminance waveform block matching method has been described, the tracking method is not limited, and any tracking method may be used.

  As described above, tracking is performed using multiple tracking methods with different trends in tracking results, and the tracking results are set as candidate positions, so reliable candidates corresponding to various disturbances can be selected. Can be displayed. As a result, the user can easily reset the tracking target without making fine adjustments by selecting and changing the candidate position of the display image displayed on the display unit 21.

  In the above description, the plurality of tracking processing units 71-1 to 71-n are made to perform tracking using different tracking methods. However, for example, the tracking processing method is the same, and each tracking processing unit 71- It is also possible to cause 1 to 71-n to perform tracking processing at different tracking target positions as the default positions. Note that all tracking methods may be the same, or at least one or more may be different.

  For example, as shown in the display image 101 of FIG. 23, when the user indicates the position indicated by the cursor P on the person 111 as the tracking target at the start of the tracking process, the target position setting unit 83 sets the position of the cursor P as , Different positions in the vicinity of the cursor P that are included in the same object as the cursor P are set as the tracking target positions of the respective tracking processing units 71-1 to 71-n, and the setting information is set to the corresponding tracking. The data is supplied to the processing units 71-1 to 71-n.

  In the example of FIG. 23, an example in the case of n = 5 will be described.

  For example, the position of the cursor P on the person 111 is set as the position of the tracking target of the tracking processing unit 71-1, and the position of the point Q2 located near the upper part of the cursor P on the person 111 is the tracking processing unit 71-. 2 is set as the tracking target position, and the position of the point Q3 located near the right side of the cursor P on the person 111 is set as the tracking target position of the tracking processing unit 71-3. The position of the point Q4 located near the lower part is set as the position to be tracked by the tracking processing unit 71-4, and the position of the point Q5 located near the left side of the cursor P on the person 111 is the tracking processing unit 71-5. Is set as the tracking target position.

  In response to this, the tracking processing units 71-1 to 71-5 perform tracking with the positions indicated by the cursor P and the points Q2 to Q5 as tracking target positions, respectively.

  In the case of the example in FIG. 23, the position of the tracking target is calculated from the tracking result of the tracking processing unit 71-1 that performs the tracking process for the position of the cursor P, and the other tracking processing units 71-2 to 71-5 A candidate position is calculated from the tracking result.

  After a predetermined time has elapsed from the start of processing, in the display image 181, as in the case of the display image 151, the person 111 instructed as the tracking target by the user has moved to the upper right part, and the upper left part, the lower left part, A tree 121, a sphere 122, a person 111, and a dog 123 are displayed in the upper right part and the lower right part, respectively.

  The cursor P indicating the position of the tracking target on the display image 181 calculated from the tracking result (of the tracking processing unit 71-1) that tracks the position of the cursor P in the display image 101 for a predetermined time is the tracking target by the user. Is displayed at a position between the tree 121 and the dog 123. A point Q2 indicating a candidate position on the display image 181 calculated from the tracking result of the tracking of the position of the point Q2 in the display image 101 (tracking processing unit 71-2) for a predetermined time is a person designated as a tracking target by the user 111 is displayed at a position on the screen. A point Q3 indicating a candidate position on the display image 181 calculated from the tracking result of the tracking of the position of the point Q3 in the display image 101 for a predetermined time (tracking processing unit 71-3) is a person designated as a tracking target by the user. 111 is displayed at a position on the dog 123.

  The point Q4 indicating the candidate position on the display image 181 calculated from the tracking result of the tracking of the position of the point Q4 in the display image 101 for a predetermined time (tracking processing unit 71-4) is the person designated as the tracking target by the user It deviates from 111 and is displayed at a position on the boundary of the sphere 122. The point Q5 indicating the candidate position on the display image 181 calculated from the tracking result of the tracking of the position of the point Q5 in the display image 101 for a predetermined time (tracking processing unit 71-5) is the person designated by the user as the tracking target. 111 and is displayed at a position between the tree 121 and the sphere 122.

  That is, during the tracking process after a predetermined time has elapsed since the start of the tracking process, the tracking of the position of the cursor P is, for example, as shown in the position of the cursor P in the display image 181, for example, Due to occlusion or the like, the user has deviated from the person 111 designated as the tracking target.

  Similarly, the tracking of the positions of the other points Q3 to Q5 is also performed by the user as a tracking target due to, for example, deformation or occlusion, as indicated by the positions of the points Q3 to Q5 in the display image 181. The person 111 that has been instructed has deviated.

  At this time, the tracking process for the position of the point Q2 is performed substantially correctly, and the point Q2 indicating the candidate position calculated from the tracking result for the position of the point Q2 is displayed on the tracking target 111 in the display image 181. Since it is displayed, the user selects the candidate position indicated by the point Q2 in the display image 181 and instructs the change.

  In response to this, the position calculation unit 82 outputs the candidate position indicated by the point Q2, that is, the candidate position calculated from the tracking result of the tracking method B, to the target position setting unit 83 as the tracking target position. To do. The target position setting unit 83 sets the candidate position indicated by the point Q2 as the position of the tracking target of the tracking processing unit 71-1, and the vicinity of the upper, right, lower, and left sides of the point Q2 on the person 111. A position (not shown) located at is set as a tracking target position of the tracking processing units 71-2 to 72-5.

  Thereby, from the next frame, tracking is started by the tracking processing unit 71-1 with the object including the position of the point Q2, that is, the person 111 as a tracking target, and the tracking unit uses the tracking result. The position of the tracking target calculated in this way is indicated by a cursor P.

  The display images 101 and 181 in the example of FIG. 23 show the positional relationship between the position indicated by the cursor P and the points Q2 to Q5 and the position of the cursor P together with the cursor P and the points Q2 to Q5. The characters "center", "upper", "right", "lower", and "left" are displayed, but these characters can be hidden.

  As described above, a plurality of different positions in the vicinity including the position of the tracking target instructed by the user are set as the tracking target positions, and a plurality of tracking is performed. Reliable candidates corresponding to various disturbances can be displayed. As a result, the user can easily reset the tracking target without making fine adjustments by selecting and changing the candidate position of the display image displayed on the display unit 21.

  Next, a display example of a plurality of candidate positions obtained as described above and an operation method thereof will be described in detail. As shown in the determination process of the position calculation unit 82 described above with reference to steps S4 to S6 in FIG. 4, when the user determines that a desired tracking result is not obtained, The remote controller 55 as shown in FIG. 24 is used to instruct candidate position selection display, and by selecting and determining the corresponding candidate position, the position of the tracking target is changed to a desired candidate position. An instruction can be input.

  FIG. 24 shows a configuration example of the remote controller 55 of FIG. In the example of FIG. 24, the remote controller 55 includes nine candidate selection buttons 221-1 to 221-9, function selection buttons 222-1 to 222-4, and a determination button 223 in order from the top. .

  The function selection buttons 222-1 to 222-4 are buttons for instructing the tracking device 12 with predetermined functions. For example, the function selection button 222-4 is a button for instructing the candidate position selection display. When the function selection button 222-4 is instructed by the user, the display image is displayed in step S9 in FIG. 4 described above. The generation unit 54 generates a display image showing candidate positions together with the position of the tracking target, that is, a candidate list image 241 that can list candidate positions shown in FIG. 25 and displays the candidate list image 241 on the display unit 21.

  The candidate list image 241 in FIG. 25 is input with the tree 121 in the upper left, the sphere 122 in the lower left, the person 111 in the upper right, and the dog 123 in the lower right, as in the display image 102 in FIG. In order to indicate the position of the tracking target and a plurality of candidate positions on the image, a small image (icon) such as a cursor P and a point R is superimposed with characters indicating candidate names.

  The candidate position located between the tree 121 and the sphere 122 is superimposed with the character “candidate 1” and the point R, and the candidate position located between the tree 121 and the dog 123 is “candidate 2”. The character and the cursor P are superimposed, the character “candidate 3” and the point R are superimposed on the candidate position located on the person 111, and the candidate position located on the boundary of the sphere 122 is “ The character “candidate 4” and the point R are superimposed on each other, and the character “candidate 5” and the point R are superimposed on the candidate position located on the dog 123.

  That is, in the example of FIG. 25, for the purpose of distinguishing the candidate position being selected by the user from other candidate positions, the candidate position being selected includes a cross cursor P and other candidates at the selected candidate position along with characters indicating the candidate names. At the position, the mark shape is changed so that the point R is displayed together with the characters indicating the candidate names. For example, in addition to changing the shape of the mark, for example, the size or color of the mark can be changed to enable discrimination from the currently selected candidate position.

  In the case of the example of FIG. 25, since the candidate list image 241 is displayed immediately (that is, when there is no instruction for selection by the user yet), it is assumed that the tracking target position is selected. A cursor P is displayed at the position of the “candidate 2” to be tracked, and points R are respectively displayed at the other candidate positions (positions “candidate 1” and “candidate 3” to “candidate 5”). It is displayed.

  Returning to FIG. 24, the candidate selection buttons 221-1 to 221-9 are buttons corresponding to the candidate positions displayed in the candidate list image 241 on a one-to-one basis. For example, the candidate selection buttons 221-1 to 221- Reference numeral 5 corresponds to each candidate position indicated by the characters “candidate 1” to “candidate 5”.

  Therefore, when the user presses the candidate selection button 221-3, the cursor P is displayed at the candidate position indicated by the characters “candidate 3”, and the candidate position indicated by the characters “candidate 3” is selected. At this time, the same point R as the other candidate positions is displayed at the candidate position indicated by the characters “candidate 2”. Similarly, when the user presses other candidate selection buttons 221-1, 221-2, 221-4, and 221-5, the cursor P is displayed at the corresponding candidate position, and the cursor P is displayed so far. A point R is displayed at the candidate position.

  In the case of the example in FIG. 24, the candidate selection buttons 221-6 to 221-9 do not have corresponding candidate positions, so that the instruction to the tracking device 12 is not transmitted even if pressed.

  The decision button 223 is a button for deciding a candidate position selected by pressing the candidate selection buttons 221-1 to 221-9 as a tracking target.

  Therefore, for example, when the cursor P is displayed at the candidate position indicated by the characters “candidate 3”, that is, when the candidate position indicated by the characters “candidate 3” is selected, the user can When the determination button 223 of the controller 55 is pressed, in the tracking device 12, the candidate position indicated by the characters “candidate 3” is set as the position to be tracked.

  Thereby, for example, the candidate position indicated by the characters “candidate 3” or the candidate area including the position is set as the position to be tracked, so the object configured to include the candidate position or candidate area Is tracked as a tracking target.

  As for the selection of the candidate position, the case where the candidate selection buttons 221-1 to 221-9 corresponding to the one-to-one correspondence with the candidate positions has been described, but the candidate selection buttons 221-1 to 221-9 are not provided. For example, the function selection button 222-3 in FIG. 24 is configured as a button for selecting a candidate, and the function selection button 222-3 is selected as shown in FIG. 26 each time the function selection button 222-3 is pressed. The candidate positions can be switched in order.

  Also, the remote controller 55 can be configured with both of these candidate selection buttons 221-1 to 221-9 and the function selection button 222-3, or the candidate selection buttons 221-1 to 221-9 or functions It is also possible to configure the remote controller 55 with only one of the selection buttons 222-3.

  In the example of FIG. 26, an example of transition of candidate list images 251-1 to 251-5 by selecting five candidate positions is shown.

  First, the candidate list image 251-1 is displayed on the display unit 21 as a display image. In the candidate list image 251-1, as described above with reference to FIG. 25, the cursor P indicating selection is displayed at the candidate position indicated by the characters “candidate 1”, and a point R is displayed at the other candidate positions. It is displayed.

  For example, when the candidate list image 251-1 in which the candidate position indicated by the characters “candidate 1” is being selected is displayed, if the function selection button 222-3 is pressed once by the user, the display unit 21, the point R is displayed at the candidate position indicated by the characters “candidate 1” and the cursor P is displayed at the candidate position indicated by the characters “candidate 2”, as indicated by the arrows. A candidate list image 251-2 in which the candidate position indicated by the characters “candidate 2” is being selected is displayed as a display image.

  If the user presses the function selection button 222-3 once while the candidate list image 251-2 is displayed, the display unit 21 displays the characters “candidate 2” as indicated by an arrow. A point R is displayed at the indicated candidate position, and a cursor P is displayed at the candidate position indicated by the characters “candidate 3”, that is, the candidate list indicated by the candidate position indicated by the characters “candidate 3” is being selected. An image 251-3 is displayed as a display image. If the user presses the function selection button 222-3 once while the candidate list image 251-3 is displayed, the display unit 21 displays the characters “candidate 3” as indicated by the arrow. The point R is displayed at the indicated candidate position, and the cursor P is displayed at the candidate position indicated by the characters “candidate 4”. That is, the candidate list indicated by the candidate position indicated by the characters “candidate 4” is being selected. An image 251-4 is displayed as a display image.

  If the user presses the function selection button 222-3 once while the candidate list image 251-4 is displayed, the display unit 21 displays the characters “candidate 4” as indicated by the arrow. A point R is displayed at the indicated candidate position, and a cursor P is displayed at the candidate position indicated by the characters “candidate 5”, that is, the candidate list indicated by the candidate position indicated by the characters “candidate 5” is being selected. An image 251-5 is displayed as a display image. If the function selection button 222-3 is pressed once by the user while the candidate list image 251-5 is displayed, the display unit 21 displays characters “candidate 5” as indicated by an arrow. The point R is displayed at the indicated candidate position, and the cursor P is displayed at the candidate position indicated by the character “candidate 1”. That is, the candidate position indicated by the character “candidate 1” is selected again. Candidate list image 251-1 is displayed as a display image.

  Therefore, when the candidate list image 251-1 in which the candidate position indicated by the characters “candidate 1” is being selected is displayed, for example, when the person 111 is desired to be tracked, the user selects the function selection button 222. -3 should be pressed twice.

  As a result, the candidate list image 251-3 in which the candidate position indicated by the characters “candidate 3” is being selected is displayed as a display image. If the user presses the enter button 223 of the remote controller 55, the tracking device 12 sets the candidate position indicated by the characters “candidate 3” as the tracking target position.

  That is, the user simply presses the function selection button 222-3 of the remote controller 55 to select a desired candidate position, and then presses the decision button 223 of the remote controller 55, and the characters “candidate 3” are displayed. The person 111 that is an object including the candidate position or candidate area shown can be tracked as a tracking target.

  Next, another configuration example and operation of the display image generation unit 54 in FIG. 2 will be described. FIG. 27 is a detailed configuration example of the display image generation unit 54 that generates a zoom image.

  27 includes an enlarged signal processing unit 301 and a tracking result selection candidate display control unit 302.

  The enlarged signal processing unit 301 generates a zoom image using the input image according to the user operation information from the overall system control unit 53 and the display tracking information obtained by the tracking processing of the object tracking unit 52. The zoomed image is output to the tracking result selection candidate display unit 302.

  The tracking result selection candidate display unit 302 uses the zoom image from the enlarged signal processing unit 301 to generate a display image using the input image as necessary, and displays the generated display image on the display unit 21. Let For example, the tracking result selection candidate display unit 302 uses the input image as necessary for the zoom image, and displays the user operation information from the overall system control unit 53 and the tracking processing of the object tracking unit 52. A display image is generated by superimposing a reduced image (that is, the candidate list image reduced with reference to FIG. 25) indicating the position of the tracking target and the candidate position generated according to the tracking information. The displayed image is displayed on the display unit 21.

  Next, the operation of the display image generation unit 54 in FIG. 27 will be described. FIG. 28 is a flowchart for explaining the details of the display image generation processing of the display image generation unit 54 of FIG. This display image generation process is another example of the display image generation process in step S9 of FIG.

  In step S301, the enlarged signal processing unit 301 uses the input image to display a zoom image according to user operation information from the overall system control unit 53 and display tracking information obtained by the tracking processing of the object tracking unit 52. Generate.

  For example, when the candidate position selection display is not instructed by the user, the user indicates the position indicated by the cursor P on the person 111 as the tracking target at the start of the tracking process, as shown in the display image 101 of FIG. In this case, the position of the cursor P is set as the tracking target position by the target position setting unit 83, and the position calculation unit 82 displays the tracking target position and candidate position information as display tracking information. It is transmitted to the image generation unit 54.

  Therefore, the enlarged signal processing unit 301 generates a zoom image 321 centered on the position of the tracking target (that is, the person 111 designated as the tracking target by the user), and displays the generated zoom image 321 as a tracking result selection candidate display. The data is output to the unit 302.

  This zoom image generation processing can be performed using the class classification adaptation processing previously proposed by the present applicant. For example, Japanese Patent Laid-Open No. 2002-196737 discloses a process for converting a 525i signal into a 1080i signal using a coefficient obtained by learning in advance. This process is substantially the same as the process of enlarging the image 9/4 times in both the vertical direction and the horizontal direction. However, since the display unit 21 has a fixed number of pixels, the enlarged signal processing unit 301 converts a 525i signal into a 1080i signal and then centers the tracking point, for example, when creating a 9 / 4-fold image. A zoom image can be generated by selecting a predetermined number of pixels (a number of pixels corresponding to the display unit 21).

  Based on this principle, a zoom image with an arbitrary magnification can be generated.

  In step S <b> 302, the tracking result selection candidate display unit 302 generates a display image using the zoom image from the enlarged signal processing unit 301, and causes the display unit 21 to display the generated display image. That is, the tracking result selection candidate display unit 302 uses the input image as necessary for the zoom image, and displays the user operation information from the overall system control unit 53 and the tracking processing of the object tracking unit 52. A display image is generated by superimposing a reduced list of candidate list images indicating candidate positions and the positions of tracking targets generated according to the tracking information.

  In this case, since the candidate position selection display is not instructed by the user, as shown in FIG. 29, a zoom image 321 generated around the position of the person 111 to be tracked is displayed as a display image. Displayed on the unit 21.

  On the other hand, when the candidate position selection display is instructed by the user, for example, a zoom image generated centering on the selected candidate position is generated, and as shown in FIG. 21 can also be displayed.

  For example, with reference to the candidate list image 241 in FIG. 25, in the candidate list image 241 in FIG. 25, candidate positions indicated by the characters “candidate 1” (that is, candidates located between the tree 121 and the sphere 122). When (position) is selected, the enlarged signal processing unit 301 generates a zoom image 351-1 centered on a candidate position located between the tree 121 and the sphere 122, and the display unit 21 displays As shown in FIG. 30, the generated zoom image 351-1 is displayed as a display image.

  Here, for example, as in the case of the example of FIG. 26, the function selection button 222-3 provided in the remote controller 55 by the user and having a function of switching the candidate position selected each time the button is pressed is used. Suppose that

  That is, when the zoom button 351-1 of FIG. 30 is displayed and the function selection button 222-3 is pressed once by the user, it is indicated by the characters “candidate 1” in the candidate list image 241 of FIG. 25. For example, the selection is switched to a candidate position indicated by the characters “candidate 2” (that is, a candidate position located between the tree 121 and the dog 123). A zoom image 351-2 centered on a candidate position located between the dogs 123 is generated, and the generated zoom image 351-2 is displayed as a display image on the display unit 21 as indicated by an arrow. The

  When the zoom image 351-2 of FIG. 30 is displayed and the user presses the function selection button 222-3 once, the candidate indicated by the characters “candidate 2” in the candidate list image 241 of FIG. 25. For example, the selection is switched from the position to the candidate position indicated by the characters “candidate 3” (that is, the candidate position located on the person 111). In the enlarged signal processing unit 301, the candidate position located on the person 111 is selected. A zoom image 351-3 centered is generated, and the generated zoom image 351-3 is displayed as a display image on the display unit 21 as indicated by an arrow.

  When the zoom image 351-3 in FIG. 30 is displayed and the function selection button 222-3 is pressed once by the user, candidates indicated by the characters “candidate 3” in the candidate list image 241 in FIG. For example, the selection is switched from the position to the candidate position indicated by the characters “candidate 4” (that is, the candidate position located on the boundary of the sphere 122), and the enlarged signal processing unit 301 is positioned on the boundary of the sphere 122. A zoom image 351-4 centered on the candidate position to be generated is generated, and the generated zoom image 351-4 is displayed as a display image on the display unit 21 as indicated by an arrow.

  When the zoom image 351-4 in FIG. 30 is displayed and the function selection button 222-3 is pressed once by the user, candidates indicated by the characters “candidate 4” in the candidate list image 241 in FIG. For example, the selection is switched from the position to the candidate position indicated by the characters “candidate 5” (that is, the candidate position located on the dog 123), and the enlarged signal processing unit 301 selects the candidate position located on the dog 123. A zoom image 351-5 centered is generated, and the generated zoom image 351-5 is displayed as a display image on the display unit 21 as indicated by an arrow.

  When the zoom image 351-5 in FIG. 30 is displayed and the user presses the function selection button 222-3 once, the candidate indicated by the characters “candidate 5” in the candidate list image 241 in FIG. For example, the selection is switched from the position to a candidate position indicated by the characters “candidate 1” (that is, a candidate position located between the tree 121 and the sphere 122). A zoom image 351-1 centering on a candidate position located between the two is generated, and the generated zoom image 351-1 is displayed as a display image on the display unit 21 as indicated by an arrow.

  Note that it is conceivable that the candidate position being selected can be difficult to understand with only the zoom image as described above. Therefore, as shown in FIG. 31, each zoom image in FIG. The described candidate list image can be reduced and superimposed so that a zoom image centered on the selected candidate position and a candidate list image on which the cursor P is superimposed on the selected candidate position can be displayed simultaneously. .

  For example, with reference to the candidate list image 241 in FIG. 25, in the candidate list image 241 in FIG. 25, candidate positions indicated by the characters “candidate 1” (that is, candidates located between the tree 121 and the sphere 122). Position) is selected, the display unit 21 displays a candidate position located between the tree 121 and the sphere 122 in the zoom image 351-1 centered on the candidate position located between the tree 121 and the sphere 122. The candidate list image 251-1 in FIG. 26 in which the cursor P is displayed is reduced, and the superimposed display image 361-1 is displayed.

  Thereby, the user can recognize that the zoom image 351-1 is enlarged around the position of the cursor P on the candidate list image 251-1.

  Here, for example, as in the case of the example of FIG. 26, the function selection button 222-3 provided in the remote controller 55 by the user and having a function of switching the candidate position selected each time the button is pressed is used. Suppose that

  That is, when the user presses the function selection button 222-3 once while the display image 361-1 in FIG. 31 is displayed, it is indicated by the characters “candidate 1” in the candidate list image 241 in FIG. For example, the selection is switched to the candidate position indicated by the characters “candidate 2” (that is, the candidate position located between the tree 121 and the dog 123). Correspondingly, as indicated by an arrow, the display unit 21 displays a zoom image 351-2 centered on a candidate position located between the tree 121 and the dog 123 between the tree 121 and the dog 123. The candidate list image 251-2 in FIG. 26 in which the cursor P is displayed at the candidate position is reduced and the superimposed display image 361-2 is displayed.

  When the function selection button 222-3 is pressed once by the user while the display image 361-2 of FIG. 31 is displayed, candidates indicated by the characters “candidate 2” in the candidate list image 241 of FIG. For example, the selection is switched from the position to the candidate position indicated by the characters “candidate 3” (that is, the candidate position located on the person 111). Correspondingly, as indicated by an arrow, the display unit 21 has a zoom image 351-3 centered on a candidate position located on the person 111 and a cursor P at the candidate position located on the person 111. The candidate list image 251-3 displayed in FIG. 26 is reduced and a superimposed display image 361-3 is displayed.

  When the function selection button 222-3 is pressed once by the user while the display image 361-3 of FIG. 31 is displayed, candidates indicated by the characters “candidate 3” in the candidate list image 241 of FIG. For example, the selection is switched from the position to the candidate position indicated by the characters “candidate 4” (that is, the candidate position located on the boundary of the sphere 122). Correspondingly, as indicated by an arrow, the display unit 21 displays a candidate position located on the boundary of the sphere 122 in the zoom image 351-4 centered on the candidate position located on the boundary of the sphere 122. The candidate list image 251-4 in FIG. 26 in which the cursor P is displayed is reduced, and a superimposed display image 361-4 is displayed.

  If the user presses the function selection button 222-3 once while the zoom image 361-4 in FIG. 31 is displayed, the candidates indicated by the characters “candidate 4” in the candidate list image 241 in FIG. For example, the selection is switched from the position to the candidate position indicated by the characters “candidate 5” (that is, the candidate position located on the dog 123). Correspondingly, as indicated by the arrow, the display unit 21 has a zoom image 351-5 centered on the candidate position located on the dog 123, and a cursor P at the candidate position located on the dog 123. The candidate list image 251-5 displayed in FIG. 26 is reduced, and a superimposed display image 361-5 is displayed.

  When the function selection button 222-3 is pressed once by the user while the zoom image 361-5 of FIG. 31 is displayed, candidates indicated by the characters “candidate 5” in the candidate list image 241 of FIG. For example, the selection is switched from the position to the candidate position indicated by the characters “candidate 1” (that is, the candidate position located between the tree 121 and the sphere 122). Correspondingly, as indicated by an arrow, the display unit 21 displays a zoom image 351-1 centered on a candidate position located between the tree 121 and the sphere 122 between the tree 121 and the sphere 122. The candidate list image 251-1 shown in FIG. 26 in which the cursor P is displayed at the candidate position is reduced, and the superimposed display image 361-1 is displayed again.

  In the example of FIG. 31, the case has been described in which the candidate list image 251-1 is reduced and superimposed on the zoom image 351-1. For example, the candidate list image 251-1 is displayed large and the zoom image 351 is displayed. -1 can be reduced and superimposed display can be performed.

  Further, as shown in FIG. 32, the zoom image of FIG. 30 centering on each candidate position can be displayed simultaneously.

  FIG. 32 shows an example of the case where there are four candidate positions, that is, the case where the candidate list image 241 shown in FIG. 25 includes four candidate positions indicated by “candidate 2” to “candidate 5”.

  In the example of FIG. 32, a zoom image generated around the candidate position indicated by the characters “candidate 2” in the candidate list image 241 of FIG. 25 (that is, the candidate position located between the tree 121 and the dog 123). 351-2, a zoom image 351-3 generated around the candidate position indicated by the characters “candidate 3” in the candidate list image 241 of FIG. 25 (that is, the candidate position located on the person 111), FIG. The zoom image 351-4 generated around the candidate position indicated by the characters “candidate 4” in the candidate list image 241 (that is, the candidate position located on the boundary of the sphere 122), and the candidate list image of FIG. It is constituted by a zoom image 351-5 generated around the candidate position indicated by the characters “Candidate 5” in 241 (that is, the candidate position located on the dog 123). Multiple zoom image 371-1 to 371-4 is shown.

  For example, when the candidate selection button 221-2 corresponding to the character “candidate 2” on the remote controller 55 is pressed by the user, for example, a frame 381 is superimposed on the display unit 21. A multiple zoom image 371-1 focused on the zoom image 351-2 generated around the candidate position indicated by the characters “candidate 2” is displayed as a display image.

  When the user presses the candidate selection button 221-3 corresponding to the character “candidate 3” on the remote controller 55, for example, a frame 381 is superimposed on the display unit 21, so that “candidate” A multiple zoom image 371-2 focused on the zoom image 351-3 generated around the candidate position indicated by the characters “3” is displayed as a display image.

  When the user presses the candidate selection button 221-4 corresponding to the character “candidate 4” on the remote controller 55, for example, a frame 381 is superimposed on the display unit 21, so that “candidate” A multiple zoom image 371-3 focused on the zoom image 351-4 generated around the candidate position indicated by the characters “4” is displayed as a display image.

  Similarly, when the user presses the candidate selection button 221-5 corresponding to the character “candidate 5” on the remote controller 55, for example, a frame 381 is superimposed on the display unit 21. A plurality of zoom images 371-4 focused on a zoom image 351-5 generated around the candidate position indicated by the characters “candidate 5” are displayed as a display image.

  As described above, by operating the remote controller 55 and switching the zoom image focused by the frame 381, the user can confirm the candidate position that he / she selects.

  Furthermore, as shown in FIG. 33, the multiple zoom images of FIG. 32 can be reduced and displayed superimposed on the candidate list image of FIG.

  For example, in the candidate list image 241 in FIG. 25, when a candidate position indicated by the characters “candidate 2” (that is, a candidate position located between the tree 121 and the dog 123) is selected, the display unit 21 displays The candidate list image 251-1 shown in FIG. 26 in which the cursor P is displayed at the candidate position located between the tree 121 and the sphere 122 is positioned between the tree 121 and the sphere 122 by being superimposed on the candidate list image 251-1 in FIG. A display image 391-1 on which the zoom image 371-1 in FIG. 32 focused on the zoom image 351-2 generated with the candidate position as the center is reduced and superimposed is displayed.

  Here, for example, as in the case of the example of FIG. 26, the function selection button 222-3 provided in the remote controller 55 by the user and having a function of switching the candidate position selected each time the button is pressed is used. Suppose that

  When the function selection button 222-3 is pressed once by the user while the display image 391-1 shown in FIG. 33 is displayed, candidates indicated by the characters “candidate 2” in the candidate list image 241 shown in FIG. For example, the selection is switched from the position to the candidate position indicated by the characters “candidate 3” (that is, the candidate position located on the person 111). Correspondingly, as indicated by the arrow, the display unit 21 is displayed on the candidate list image 251-3 in FIG. 26 where the cursor P is displayed at the candidate position located on the person 111. A zoomed image 351-3 centered on the candidate position to be focused is displayed on the display image 391-2 on which the plurality of zoomed images 371-2 in FIG. 32 are reduced and superimposed.

  When the function selection button 222-3 is pressed once by the user while the display image 361-2 of FIG. 33 is displayed, candidates indicated by the characters “candidate 3” in the candidate list image 241 of FIG. For example, the selection is switched from the position to the candidate position indicated by the characters “candidate 4” (that is, the candidate position located on the boundary of the sphere 122). Correspondingly, as indicated by the arrow, the display unit 21 displays the cursor P on the candidate position located on the boundary of the sphere 122 in the candidate list image 251-4 of FIG. A display image 391-3 in which the multiple zoom images 371-3 in FIG. 32 focused on the zoom image 351-4 centered on the candidate position located on the boundary is reduced and superimposed is displayed.

  When the zoom image 391-3 of FIG. 33 is displayed and the function selection button 222-3 is pressed once by the user, candidates indicated by the characters “candidate 4” in the candidate list image 241 of FIG. For example, the selection is switched from the position to the candidate position indicated by the characters “candidate 5” (that is, the candidate position located on the dog 123). Correspondingly, as indicated by an arrow, the display unit 21 is positioned on the dog 123 in the candidate list image 251-5 of FIG. 26 in which the cursor P is displayed at the candidate position positioned on the dog 123. A display image 391-4 on which a plurality of zoom images 371-4 of FIG. 32 focused on the zoom image 351-5 centered on the candidate position to be reduced is superimposed is displayed.

  When the function selection button 222-3 is pressed once by the user while the zoom image 391-4 of FIG. 33 is displayed, it is indicated by the characters “candidate 5” in the candidate list image 241 of FIG. For example, the selection is switched to the candidate position indicated by the characters “candidate 2” (that is, the candidate position located between the tree 121 and the dog 123). Correspondingly, as indicated by an arrow, the display unit 21 displays a tree in the candidate list image 251-2 in FIG. 26 in which the cursor P is displayed at a candidate position located between the tree 121 and the dog 123. 32, the zoom image 351-2 shown in FIG. 32 focused on the zoom image 351-2 centered on the candidate position located between 121 and the dog 123 is reduced and displayed again on the display image 391-4.

  30, 31, and 33, an example in which operation is performed using the function selection button 222-3 will be described, and in FIG. 32, operation is performed using candidate selection buttons 221-1 to 221-9. Although the example has been described, the display in FIGS. 30, 31, and 33 can be operated using the candidate selection buttons 221-1 to 221-9, and the display in FIG. 32 is also the function selection button 222-. 3 can be operated.

  As described above, the user can easily select a desired candidate position by clearly displaying the candidate position.

  In the above description, as an example of displaying the tracking target candidate position, the function button 222-4 of the remote controller 55 in FIG. 24 is pressed by the user as the timing for displaying the tracking target candidate position. As described above, for example, as the tracking is started, the tracking target candidate position is always displayed (that is, all tracking results in the case of the object tracking unit 52 in FIG. 9), or the user is prompted to select a candidate. In addition, the candidate positions to be tracked can be displayed every predetermined time (for example, 10 seconds).

  Further, the tracking device 12 can also display the tracking target candidate position at a timing estimated to be not the tracking result desired by the user.

  This estimation is executed by the overall system control unit 53 in FIG. 2 as described below. First, for example, in the object tracking unit 52 of FIG. 9, when the basic tracking method is performed by block matching, if the reliability value of the motion vector detected by the tracking processing unit 71-1 that tracks by the block matching method is low, When the determination is made (for example, when the determination in step S1124 in FIG. 43 described later is No), the overall system control unit 53 estimates that the tracking result is not the tracking result desired by the user, and causes the display image generation unit 54 to The candidate position of the tracking target can be displayed by controlling.

  Further, the tracking processing unit 71 is configured with a scene change detection unit 1053 that detects a scene change, which will be described later with reference to FIG. 38, and when the scene change is detected by the scene change detection unit 1053, the entire system control is performed. For example, the unit 53 can estimate that the tracking result is not the tracking result desired by the user, and can control the display image generation unit 54 to display the tracking target candidate position.

  Further, in the case of the object tracking unit 52 of FIG. 9, as described above with reference to FIG. 18, it is determined that the average distance between the tracking results of the basic tracking method and the other tracking methods is large, or the total tracking result When it is determined that the tracking results of the plurality of tracking methods are greatly different, for example, the overall system control unit 53 estimates that the tracking result is not the tracking result desired by the user. Thus, the display image generation unit 54 can be controlled to display the tracking target candidate positions.

  In the case where the object tracking unit 52 in FIG. 9 causes the tracking processing units 71-1 to 71-n to perform tracking processing at a plurality of different tracking target positions, as described above with reference to FIG. When it is determined that the tracking results are significantly different, the overall system control unit 53 estimates that the tracking result is not the tracking result desired by the user, for example, controls the display image generation unit 54, and performs tracking Candidate positions can be displayed.

  As a result, the user can immediately recognize that the tracking is out of the desired tracking target. Then, the user can immediately correct the tracking target with an easy operation of selecting a candidate position.

  Next, a detailed configuration example and operation of the tracking processing unit 71 in FIG. 3 will be described. FIG. 34 is a block diagram illustrating a functional configuration example of the tracking processing unit 71 based on the motion region center-of-gravity tracking method. In this example, the tracking processing unit 71 includes a motion vector detecting unit 501, a frequency distribution calculating unit 502, a sample point extracting unit 503, a centroid calculating unit 504, and a tracking point updating unit 505.

  An input image from the input terminal 51 is input to the motion vector detection unit 501 and the sample point extraction unit 503. The motion vector detection unit 501 detects a motion vector in an area centered on the tracking point in the input image. The frequency distribution calculation unit 502 uses the motion vector detected by the motion vector detection unit 501 to calculate the frequency distribution of the motion vector in the region.

  Based on the motion vector frequency distribution, the sample point extraction unit 503 extracts a sample point that shows a movement similar to a movement that occupies a large number in an area centered on the tracking point in the input image, and extracts the sample point on the tracking target. The point. The centroid calculation unit 504 calculates the centroid of the sample point based on whether or not the point in the region is the sample point to be tracked extracted by the sample point extraction unit 503.

  The tracking point update unit 505 updates the tracking point by adding the maximum frequency motion to the center of gravity calculated by the center of gravity calculating unit 504, and uses the updated tracking point information as a tracking result as a tracking processing control unit. 72.

  Next, the operation of the tracking processing unit 71 in FIG. 34 will be described. FIG. 35 is a flowchart illustrating details of the tracking process executed by the tracking processing unit 71 in step S3 of FIG.

  In step S501, the motion vector detection unit 501 waits for input of an image of the next frame, and in step S502, detects a motion vector in an area centered on the tracking point in the input image.

  That is, by capturing the next frame (next frame) temporally from the frame including the tracking point (previous frame) in the process of step S501, two consecutive frames of images are obtained. .

  The motion vector detection unit 501 uses the tracking target position set by the target position setting unit 83 in step S7 or S8 in FIG. 4 (for example, the position on the object 522 of the person specified as the tracking target by the user) as the tracking point. 36, as shown in FIG. 36, for each sampling point of the sampling interval (Sx, Sy) in the region 521 centered on the tracking point P in the input image 511 of the previous frame input before in time. The motion vector is detected by estimating the corresponding sample point of the subsequent frame. The size of the area 521 is m * Sx × n * Sy (* represents multiplication), where m and n are the number of samples.

  In step S <b> 503, the frequency distribution calculation unit 502 calculates the frequency distribution of the motion vector in the region 521 using the motion vector detected by the motion vector detection unit 501.

  For example, if the motion candidates in the region 521 are Vx (horizontal motion: −16 ≦ Vx ≦ 16) and Vy (vertical motion: −16 ≦ Vy ≦ 16), 33 × 33 = 1089 box, that is, a motion vector Boxes for coordinates corresponding to possible values are prepared, and when a motion vector is generated, 1 is added to the coordinates corresponding to the motion vector. For example, when (Vx, Vy) = (2, 2) at a certain sample point, 1 is added to the box of (2, 2). By performing this for all the sample points in the area 521, the motion vector frequency distribution in the area 521 is calculated.

  In step S504, the sample point extraction unit 503 is similar to the motion occupying a large number in the region 521 centered on the tracking point P in the input image based on the frequency distribution of the motion vector calculated by the frequency distribution calculation unit 502. A sample point indicating the movement to be extracted is extracted and used as a point on the tracking target.

  That is, as shown in FIG. 37 by enlarging the area 521 in FIG. 36, the tracking point P, which is the position of the tracking target set so that the human object 522 in the input image 511 of the previous frame is set as the tracking target, is set. Since the human object 522 has a large proportion in the central area 521, a large number of motion vectors (thick arrows) detected from the sample points on the human object 522 are occupied.

  Therefore, the sample point extraction unit 503 extracts sample points that show a movement similar to a movement that occupies a large number in the area 521 centered on the tracking point P, and sets it as a point on the tracking target.

  In step S505, the center-of-gravity calculation unit 504 determines the center of gravity of the sample point Sa (x, y) based on whether or not the point in the region 521 is the sample point to be tracked extracted by the sample point extraction unit 503. G (x, y) is calculated. This calculation formula is expressed by the following formula (2).

  Here, flag (i, j) (1 ≦ i ≦ m, 1 ≦ j ≦ n) is a flag indicating whether or not the sample point is a tracking target, and is 1 when it is a sample point. If it is not a sample point, it is 0.

  In step S <b> 506, the tracking point update unit 505 updates the tracking point by adding the movement with the maximum frequency to the center of gravity G (x, y) calculated by the center of gravity calculating unit 504.

  The updated tracking point information is output as a tracking result to the tracking processing control unit 72, the tracking processing ends, the processing returns to step S2 in FIG. 4, and then the tracking result storage in step S3. Based on the tracking result stored in the unit 81, a position calculation process by the tracking process control unit 72 is executed.

  Next, another detailed configuration example and operation of the tracking processing unit 71 in FIG. 3 will be described. FIG. 38 is a block diagram illustrating a functional configuration example of the tracking processing unit 71 that performs the tracking processing by the point tracking method with transfer. In this example, the tracking processing unit 71 includes a template matching unit 1051, a motion estimation unit 1052, a scene change detection unit 1053, a background motion estimation unit 1054, a region estimation related processing unit 1055, a transfer candidate holding unit 1056, and a tracking point determination unit 1057. , A template holding unit 1058, and a control unit 1059.

  The template matching unit 1051 performs a matching process between the input image and the template image held in the template holding unit 1058. The motion estimation unit 1052 estimates the motion of the input image, and obtains the motion vector obtained as a result of the estimation and the accuracy of the motion vector, the scene change detection unit 1053, the background motion estimation unit 1054, the region estimation related processing unit 1055, And output to the tracking point determination unit 1057. The scene change detection unit 1053 detects a scene change based on the accuracy supplied from the motion estimation unit 1052.

  The background motion estimation unit 1054 executes processing for estimating the background motion based on the motion vector and the accuracy supplied from the motion estimation unit 1052, and supplies the estimation result to the region estimation related processing unit 1055. The region estimation related processing unit 1055 is based on the motion vector and accuracy supplied from the motion estimation unit 1052, the background motion supplied from the background motion estimation unit 1054, and the tracking point information supplied from the tracking point determination unit 1057. Perform region estimation processing. Further, the region estimation related processing unit 1055 generates a transfer candidate based on the input information, supplies the transfer candidate to the transfer candidate holding unit 1056, and holds it. Further, the region estimation related processing unit 1055 creates a template based on the input image, supplies the template to the template holding unit 1058, and holds it.

  The tracking point determination unit 1057 determines a tracking point based on the motion vector and accuracy supplied from the motion estimation unit 1052 and the transfer candidate supplied from the transfer candidate holding unit 1056, and information on the determined tracking point is obtained. It outputs to the area estimation related processing unit 1055.

  The control unit 1059 controls each part of the template matching unit 1051 to the template holding unit 1058 based on the setting information from the tracking processing control unit 72 (that is, the position information of the tracking target) to track the set tracking target. In addition, a tracking result such as information on the position of the tracking point obtained by tracking on the screen is output to the tracking processing control unit 72.

  Next, the operation of the tracking processing unit 71 will be described. FIG. 39 is a flowchart for explaining the details of the tracking process executed by the tracking processing unit 71 in step S2 of FIG.

  As shown in FIG. 39, the tracking processing unit 71 basically executes normal processing and exception processing. That is, normal processing is performed in step S1051. The details of this normal process will be described later with reference to FIG. 43, and the process of tracking the tracking point based on the position information of the tracking target set by the tracking process control unit 72 is executed by this process.

  When the tracking point cannot be changed in the normal processing in step S1051, exception processing is executed in step S1052. The details of the exception processing will be described later with reference to the flowchart of FIG. 58. When the tracking point becomes invisible from the image by the exception processing, the return processing to the normal processing is executed by template matching. If it is determined that the tracking process cannot be continued due to the exception process (cannot return to the normal process), the process is terminated. If it is determined that recovery is possible, the process returns to step S1051. In this way, the normal process in step S1051 and the exception process in step S1052 are repeatedly executed sequentially for each frame.

  In the tracking processing unit 71 in FIG. 38, the normal processing and the exception processing cause tracking to occur as shown in FIGS. 40 to 42, such that the tracking target rotates, an occlusion occurs, a scene change occurs, and so on. Even when a point is temporarily invisible, tracking is possible.

  That is, for example, as shown in FIG. 40, a person's face 1104 as a tracking target (object) is displayed in the frame n−1, and this person's face 1104 has a right eye 1102 and a left eye 1103. ing. It is assumed that the user designates, for example, the right eye 1102 (exactly one pixel therein) as the tracking point 1101. In the example of FIG. 40, in the next frame n, the person moves to the left in the figure, and in the next frame n + 1, the person's face 1104 rotates in the clockwise direction. As a result, the right eye 1102 that has been visible until now is not displayed, and tracking cannot be performed with the conventional method. Therefore, in the normal processing in step S1051 described above, the left eye 1103 on the face 1104 as the same object as the right eye 1102 is selected, and the tracking point is switched (set) to the left eye 1103. This enables tracking.

  In the display example of FIG. 41, the ball 1121 moves from the left side in the drawing of the face 1104 in the frame n−1, and the ball 1121 just covers the face 1104 in the next frame n. In this state, the face 1104 including the right eye 1102 designated as the tracking point 1101 is not displayed. When such an occlusion occurs, the face 1104 as the object is not displayed, so there is no transfer point to be tracked instead of the track point 1101, and thereafter it becomes difficult to track the track point. However, in the present invention, the right eye 1102 as the tracking point 1101 is stored in advance as an image of the frame n-1 (actually the previous frame in time) as a template, and the ball 1121 moves further to the right. When the right eye 1102 designated as the tracking point 1101 appears again in frame n + 1, it is confirmed that the right eye 1102 as the tracking point 1101 is displayed again by the exception processing in step S1052 described above, and the right eye 1102 is tracked again. The point 1101 is tracked.

  In the example of FIG. 42, the face 1104 is displayed in the frame n−1, but in the next frame n, the automobile 1111 covers and covers the whole including the human face. That is, in this case, a scene change has occurred. In the present invention, even if the scene change occurs and the tracking point 1101 does not exist from the image, if the automobile 1111 moves and the right eye 1102 is displayed again in the frame n + 1, the exception processing in step S1052 It is confirmed based on the template that the right eye 1102 as the tracking point 1101 has appeared again, and the right eye 1102 can be tracked again as the tracking point 1101.

  Next, the details of the normal processing in step S1051 in FIG. 39 will be described with reference to the flowchart in FIG. In step S1121, the tracking point determination unit 1057 executes normal processing initialization processing. The details will be described later with reference to FIG. 44, but by this processing, the user is designated to track based on the setting information from the tracking target control unit 72 set in step S7 or S8 of FIG. An area estimation range based on the tracking point is designated. This area estimation range is the same object as the tracking point specified by the user (for example, when the tracking point is a human eye, a human face as a rigid body that moves like the eye, a human body, or the like) ) Is a range to be referred to when estimating the range of points belonging to. A transfer point is selected from points in the region estimation range.

  Next, in step S1122, the control unit 1059 controls each unit to wait for input of an image of the next frame. In step S1123, the motion estimation unit 1052 estimates the motion of the tracking point. That is, based on the setting information from the tracking target control unit 72, a frame (following frame) that is temporally later than the frame (tracking frame) including the tracking point designated by the user is captured in the process of step S1122. Since two consecutive frames of images have been obtained, in step S1123, the movement of the tracking point is estimated by estimating the position of the tracking point of the subsequent frame corresponding to the tracking point of the previous frame.

  Note that “preceding in time” means the processing order (input order). Normally, images of each frame are input in the order of imaging. In this case, the frame captured earlier in time becomes the previous frame, but the frame captured later in time is processed (input) first. ), The frame imaged later in time becomes the previous frame.

  In step S1124, the motion estimation unit 1052 determines whether the tracking point can be estimated as a result of the process in step S1123. Whether or not the tracking point can be estimated is determined, for example, by comparing the accuracy value of a motion vector (described later) generated and output by the motion estimation unit 1052 with a preset threshold value. Specifically, it is possible to estimate if the accuracy of the motion vector is equal to or greater than a threshold, and it is determined that estimation is impossible if the accuracy is smaller than the threshold. In other words, the possibility here is determined relatively strictly, and if the estimation is not actually impossible but the accuracy is low, it is determined as impossible. As a result, more reliable tracking processing can be performed.

  In step S1124, it is determined that the motion estimation result at the tracking point and the motion estimation result at a point in the vicinity of the tracking point can be estimated if they match the motions that occupy the majority, and if they do not match, it is determined that the estimation is impossible. It is also possible to do so.

  When it is determined that the movement of the tracking point can be estimated (the probability that the tracking point is correctly set on the corresponding point on the same object (if the right eye 1102 is designated as the tracking point 1101, the right eye 1102 If the probability of being correctly tracked is relatively high), the process proceeds to step S1125, and the tracking point determination unit 1057 shifts the tracking point by the estimated motion (motion vector) obtained in the process of step S1123. In other words, this determines the tracking position in the subsequent frame after tracking the tracking point of the previous frame. The tracking position information determined in step S1125 is output to the tracking processing control unit 72 as a tracking result.

  After step S1125, region estimation related processing is executed in step S1126. The details of this area estimation related process will be described later with reference to FIG. 47, and this process updates the area estimation range specified in the initialization process of the normal process in step S1121. Furthermore, if the tracking point is not displayed because the target object rotates, for example, the candidate as a switching point (transfer candidate) as a point to be switched to is the state (can still be tracked). In the state), it is extracted (created) in advance. Further, when the transfer to the transfer candidate cannot be performed, the tracking is temporarily interrupted, but a template is created in advance in order to confirm that the tracking is possible again (the tracking point appears again).

  After the region estimation related processing in step S1126 is completed, the processing returns to step S1121 again, and the subsequent processing is repeatedly executed.

  That is, as long as the initialization process of the normal process is performed based on the setting information from the tracking target control unit 72 set in step S7 or S8 in FIG. 4 and the movement of the tracking point specified by the user can be estimated. The processes in steps S1121 to S1126 are repeatedly executed for each frame, and tracking is performed.

  On the other hand, when it is determined in step S1124 that the movement of the tracking point cannot be estimated (impossible), that is, as described above, for example, the accuracy of the motion vector is equal to or less than the threshold value. The process proceeds to step S1127. In step S1127, the tracking point determination unit 1057 has the transfer candidate generated in the region estimation-related processing in step S1126 held in the transfer candidate holding unit 1056, and therefore, the transfer candidate closest to the original tracking point is among them. Select one. The tracking point determination unit 1057 determines whether or not a transfer candidate has been selected in step S1128. If a transfer candidate has been selected, the tracking point determination unit 1057 proceeds to step S1129 and sets the tracking point as the transfer candidate selected in the process of step S1127. Change (change). That is, the transfer candidate point is set as a new tracking point. Thereafter, the process returns to step S1123, and the process of estimating the movement of the tracking point selected from the transfer candidates is executed.

  In step S1124, it is determined again whether or not the movement of the newly set tracking point can be estimated. If it can be estimated, a process of shifting the tracking point by the estimated movement is performed in step S1125, and step S1126. In, the area estimation related process is executed. Thereafter, the processing returns to step S1121 again, and the subsequent processing is repeatedly executed.

  If it is determined in step S1124 that the newly set tracking point cannot be estimated, the process returns to step S1127, and the next transfer candidate closest to the original tracking point is selected from the transfer candidates. In step S1129, the transfer candidate is set as a new tracking point. The process after step S1123 is repeated for the new tracking point.

  Even if all the prepared transfer candidates are used as new tracking points, if the movement of the tracking point cannot be estimated, it is determined in step S1128 that the transfer candidate cannot be selected, and this normal process is performed. Is terminated. Then, the process proceeds to the exception process in step S1052 of FIG.

  Next, details of the initialization process of the normal process in step S1121 of FIG. 43 will be described with reference to the flowchart of FIG.

  In step S1141, the control unit 1059 determines whether the current process is a process for returning from the exception process. That is, it is determined whether or not the processing returns to the normal processing in step S1051 after the exception processing in step S1052 is completed. In the process of the first frame, since the exception process in step S1052 has not been executed yet, it is determined that the process has not returned from the exception process, and the process proceeds to step S1142. In step S1142, the tracking point determination unit 1057 executes a process of setting the tracking point to the position of the tracking point instruction. The tracking point determination unit 1057 supplies the set tracking point information to the region estimation related processing unit 1055.

  In step S1143, the region estimation related processing unit 1055 sets a region estimation range based on the position of the tracking point set in the processing of step S1142. This region estimation range is a reference range when estimating a point on the same rigid body as the tracking point, and more specifically, tracking so that the same rigid body portion as the tracking point occupies most of the region estimation range in advance. By setting the position and size of the estimated area range to follow the same rigid body part as the point, the part that shows the most movement in the estimated area range can be estimated as the same rigid body part as the tracking point It is for doing so. In step S 1143, as a default value, for example, a predetermined fixed range centered on the tracking point is set as the region estimation range.

  Thereafter, the processing proceeds to step S1122 in FIG.

  On the other hand, if it is determined in step S1141 that the current process is a process for returning from the exception process in step S1052, the process proceeds to step S1144, and the tracking point determination unit 1057 is described later with reference to FIG. By the exception processing, the tracking point and the area estimation range are set based on the position matching the template. For example, a point on the current frame that matches the tracking point on the template is set as the tracking point, and a certain range set in advance from the point is set as the region estimation range. Thereafter, the process proceeds to step S1122 of FIG.

  The above process will be described with reference to FIG. That is, in step S1142 of FIG. 44, for example, as shown in FIG. 45, when the human eye 102 of the frame n−1 is designated as the tracking point 1101, a predetermined region including the tracking point 1101 is specified in step S1143. Is designated as the area estimation range 1133. In step S1124, it is determined whether or not sample points within the region estimation range 1133 can be estimated in the next frame. In the case of the example in FIG. 45, in the frame n + 1 next to the frame n, the left half region 1134 in the drawing including the left eye 1102 is hidden by the ball 1121 in the region estimation range 1133. 1101 motion cannot be estimated in the next frame n + 1. Therefore, in such a case, one of the points in the area designation range 1133 (in the face 1104 as a rigid body including the right eye 1102) prepared in advance as a transfer candidate in the previous frame n−1 is selected. One point (for example, the left eye 1103 included in the face 1104 (exactly one pixel therein)) is selected, and that point is set as the tracking point in the frame n + 1.

  The region estimation related processing unit 1055 has a configuration as shown in FIG. 46 in order to execute the region estimation related processing in step S1126 of FIG. That is, the motion estimation unit 1052 receives the motion vector and the accuracy, the background motion estimation unit 1054 receives the background motion, and the tracking point determination unit 1057 tracks the tracking point. Position information is input. In addition to the motion vector and the accuracy supplied from the motion estimation unit 1052, the transfer candidate extraction unit 1162 is supplied with the output of the region estimation unit 1161. In addition to the input image, the template creation unit 1163 receives the output of the region estimation unit 1161.

  The region estimation unit 1161 estimates a rigid region including the tracking point based on the input, and outputs the estimation result to the transfer candidate extraction unit 1162 and the template creation unit 1163. The transfer candidate extraction unit 1162 extracts transfer candidates based on the input, and supplies the extracted transfer candidates to the transfer candidate holding unit 1056. The template creation unit 1163 creates a template based on the input, and supplies the created template to the template holding unit 1058.

  FIG. 47 shows details of the region estimation related processing (the processing in step S1126 of FIG. 43) executed by the region estimation related processing unit 1055. First, in step S1161, the region estimation unit 1161 executes region estimation processing. The details will be described later with reference to FIG. 48. By this processing, the points of the region on the image that are estimated to belong to the same object (the rigid body that moves in synchronization with the tracking point) to which the tracking point belongs. Are extracted as points of the region estimation range.

  In step S1162, the transfer candidate extraction unit 1162 executes transfer candidate extraction processing. The details of the process will be described later with reference to FIG. 53, but transfer candidate points are extracted from the points of the range estimated as the area estimation range by the area estimation unit 1161 and held in the transfer candidate holding unit 1056.

  In step S1163, the template creation unit 1163 executes template creation processing. Details thereof will be described later with reference to FIG. 54, and a template is created by this processing.

  Next, the details of the region estimation processing in step S1161 in FIG. 47 will be described with reference to the flowchart in FIG.

  First, in step S1181, the region estimation unit 1161 determines a sample point as a candidate point of a point estimated to belong to the same target as the tracking point.

  For example, as shown in FIG. 49, the sample points are predetermined in a horizontal direction and a vertical direction with reference to a fixed reference point 1201 among pixels in the entire screen of a frame indicated by a white square in the drawing. Pixels at positions separated by the number of pixels can be used as sample points (represented by black squares in the figure). In the example of FIG. 49, the upper left pixel of each frame is set as the reference point 1201 (the reference point 1201 is indicated by a cross in the figure), and is separated by 5 in the horizontal direction and 5 in the vertical direction. The pixel at the position is taken as the sample point. That is, in the case of this example, pixels at positions dispersed throughout the entire screen are taken as sample points. In this example, the reference point is a point at the same position fixed in each frame n, n + 1.

  Note that the reference point 1201 can be dynamically changed so as to be a point at a different position for each frame n, n + 1.

  In the example of FIG. 49, the interval between the sample points is a fixed value in each frame, but the interval between the sample points for each frame is, for example, 5 pixels in frame n and 8 in frame n + 1. It can also be variable with the pixel. As an interval reference at this time, an area of a region estimated to belong to the same object as the tracking point can be used. Specifically, if the area of the region estimation range is narrowed, the interval is also shortened.

  Alternatively, the interval between sample points can be made variable in one frame. The distance from the tracking point can be used as a reference for the interval at this time. That is, the sample point closer to the tracking point has a smaller interval, and the farther from the tracking point, the larger the interval.

  When the sample points are determined as described above, in step S1182, the region estimation unit 1161 then determines the region estimation range (steps S1143 and S1144 in FIG. 44 or steps S1206 and S1208 in FIG. 50 described later). The process of estimating the movement of the sample points within (determined in the process of (1)) is executed. That is, the region estimation unit 1161 extracts the corresponding point of the next frame corresponding to the sample point within the region estimation range based on the motion vector supplied from the motion estimation unit 1052.

  In step S1183, the region estimation unit 1161 executes processing for excluding points based on motion vectors whose accuracy is lower than a preset threshold among the sample points estimated in step S1182. The accuracy of the motion vector necessary for this processing is supplied from the motion estimation unit 1052. Thereby, only the points estimated based on the motion vector with high accuracy are extracted from the sample points within the region estimation range.

  In step S1184, the region estimation unit 1161 extracts the full screen motion in the motion estimation result within the region estimation range. The full screen motion means a motion that takes the area corresponding to the same motion and maximizes the area. Specifically, a motion histogram is generated by assigning a weight proportional to the sample point interval at each sample point to the motion of each sample point, and one motion (one motion vector) having the maximum weighting frequency is generated. Extracted as full screen motion. When generating a histogram, for example, a representative value of motion can be prepared with pixel accuracy, and a motion having a value with one pixel accuracy can be added to the histogram.

  In step S1185, the region estimation unit 1161 extracts sample points within the region estimation range having full screen motion as the result of region estimation. In this case, the sample points having the full screen motion include not only the sample points having the same motion as the full screen motion but also the difference in motion from the full screen motion is equal to or less than a predetermined threshold value set in advance. The sample point can also be a sample point having full screen motion here.

  In this way, among the sample points within the region estimation range determined by the processing of steps S1143, S1144, S1206, and S1208, the sample points that have full screen motion are estimated to belong to the same target as the tracking points. Is finally extracted (generated).

  Next, in step S1186, the region estimation unit 1161 executes a region estimation range update process. Thereafter, the processing proceeds to step S122 in FIG.

  FIG. 50 shows details of the region estimation range update processing in step S1186 of FIG. In step S1201, the region estimation unit 1161 calculates the center of gravity of the region. This region means a region composed of sample points extracted in the process of step S1185 in FIG. 48 (region composed of points estimated to belong to the same target as the tracking point). That is, one motion vector (full screen motion) corresponds to this area. For example, as shown in FIG. 51A, among sample points indicated by white squares in the figure, sample points having full-screen motion in the processing of step S1185 in FIG. In FIG. 51A, sample points indicated by black squares are extracted, and a region constituted by the sample points is extracted (estimated) as a region 1222. Then, the center of gravity 1224 of the region 1222 is further calculated. Specifically, the sample point centroid obtained by assigning the weight of the sample point interval to each sample point is obtained as the centroid of the region. This process has the meaning of obtaining the position of the region in the current frame.

  Next, in step S202, the region estimation unit 1161 executes processing for shifting the center of gravity of the region by full screen motion. This process has the meaning of causing the area estimation range 1221 to follow the movement of the position of the area and to move to the estimated position in the next frame. As shown in FIG. 51B, when the tracking point 1223 in the current frame appears as the tracking point 1233 in the next frame based on the motion vector 1238, the full-screen motion vector 1230 substantially corresponds to the tracking point motion vector 1238. Therefore, a point 1234 on the same frame (next frame) as the tracking point 1233 is obtained by shifting the center of gravity 1224 in the current frame based on the motion vector 1230 (full screen motion). If the area estimation range 1231 is set around the point 1234, the area estimation range 1221 is moved to the estimated position in the next frame by following the movement of the position of the area 1222.

  In step S1203, the region estimation unit 1161 determines the size of the next region estimation range based on the region estimation result. Specifically, the sum of squares of sample point intervals (intervals of points indicated by black squares in the region 1222 in FIG. 51A) regarding all sample points estimated as the region is regarded as the area of the region 1222, and this The size of the region estimation range 1231 in the next frame is determined so that the size is slightly larger than the area. In other words, the size of the region estimation range 1231 increases as the number of sample points in the region 1222 increases, and decreases as it decreases. In this way, not only the enlargement / reduction of the area 1222 can be followed, but also the entire screen area within the area estimation range 1221 can be prevented from becoming a peripheral area to be tracked.

  When the full screen motion extracted in step S1184 in FIG. 48 matches the background motion, the background and the tracking target cannot be distinguished by the motion. Therefore, the background motion estimation unit 1054 always performs background motion estimation processing. In step S1204, the region estimation unit 1161 extracts the background motion supplied from the background motion estimation unit 1054 and the processing of step S1184 in FIG. It is determined whether or not the full screen motion matches. If the full screen motion matches the background motion, in step S1205, the region estimation unit 1161 limits the size of the next region estimation range so that the current region estimation range is maximized. Thereby, it is suppressed that the background is erroneously recognized as the tracking target and the size of the area estimation range is enlarged.

  If it is determined in step S1204 that the full screen motion and the background motion do not match, the processing in step S1205 is not necessary and is skipped.

  Next, in step S1206, the region estimation unit 1161 determines the size of the next region estimation range around the shifted region centroid. As a result, the area estimation range is determined such that the center of gravity coincides with the already obtained area centroid after the shift, and the size thereof is proportional to the area width.

  In the example of FIG. 51B, the area estimation range 1231 is determined to have a size corresponding to the area 1222 with the shifted center of gravity 1234 based on the motion vector (full screen movement) 1230 as the center.

  It is necessary to ensure (ensure) that the region having the full screen motion within the region estimation range 1231 is the region of the tracking target (for example, the face 1104 in FIG. 45). Therefore, in step S1207, the region estimation unit 1161 determines whether or not the tracking point is included in the next region estimation range. If the tracking point is not included in step S1207, the region estimation unit 1161 determines whether or not the tracking point is included in step S1208. A process for shifting the region estimation range is executed. If the tracking point is included in the next region estimation range, the processing in step S1208 is not necessary and is skipped.

  As a specific method of shifting in this case, a method of minimizing the moving distance, a minimum distance at which the tracking point is included along a vector from the center of gravity of the region estimation range before the shift to the tracking point A way to move only is considered.

  In order to emphasize the robustness of tracking, a method of not performing a shift to include a tracking point in the region is also conceivable.

  In the example of FIG. 51C, since the region estimation range 1231 does not include the tracking point 1233, the region estimation range 1241 is shifted to a position indicated as the region estimation range 1241 (a position including the tracking point 1233 at the upper left).

  51A to 51C show the case where the shift process of step S1208 is necessary, but FIGS. 52A to 52C show the case where the shift process of step S1208 is not necessary (the tracking point is the next area estimation range in step S1207). Example).

  As shown in FIGS. 52A to 52C, when all the sample points in the area estimation range 1221 are area points, the shift process in step S1208 in FIG. 50 is not necessary.

  51A to 51C and FIGS. 52A to 52C show an example in which the region estimation range is a rectangle, the region estimation range may be a circle.

  As described above, the position estimation range position and size for the next frame are determined to include the tracking point by the region estimation range update process of FIG. 50 (step S1186 in FIG. 48).

  In the update process of the area estimation range in FIG. 50, the area estimation range is a rectangular (or circular) fixed shape, but may be a variable shape.

  Next, the transfer candidate extraction process in step S1162 of FIG. 47 will be described with reference to the flowchart of FIG.

  In step S <b> 1261, the transfer candidate extraction unit 1162 holds, as transfer candidates, the point shift results of the estimated motion corresponding to each of the points estimated as the full screen motion region. That is, instead of using the points obtained as region estimation results as they are, a process for extracting the results shifted based on the respective motion estimation results is performed for use in the next frame. The changed transfer candidates are supplied to and held in the transfer candidate holding unit 56.

  This process will be described as follows with reference to FIG. That is, in the example of FIG. 45, the tracking point 1101 exists in the frames n−1 and n, but in the frame n + 1, the tracking point 1101 does not exist because it is hidden by the ball 1121 flying from the left side in the drawing. Therefore, in frame n + 1, it is necessary to transfer the tracking point to another point on the face 1104 as the tracking target (for example, the left eye 1103 (actually closer to the right eye 1102)). Therefore, a transfer candidate is prepared in advance in a frame before transfer is actually required.

  Specifically, in the case of the example in FIG. 45, the motion estimation result in the region estimation range 1133 from frame n to frame n + 1 needs to be changed in the region estimation range 1133, so that there is a high probability that it cannot be estimated correctly. Is expected. That is, in the example of FIG. 45, the transfer occurs because the tracking point and a part of the same object are hidden. As a result, in the region estimation range 1133 in the frame n, regarding the portion 1134 where the object is hidden in the frame n + 1 (the shaded portion in FIG. 45) 1134, the motion is not estimated correctly, and it is estimated that the accuracy of the motion is low. It is estimated that the accuracy is not low and a motion estimation result is meaningless.

  In such a case, there is a high possibility that the region estimation is erroneous due to a decrease in motion estimation results that can be used in region estimation or a mixture of erroneous motion estimation results. On the other hand, such a possibility is generally greater in the region estimation between the previous frame n−1 and the frame n in comparison with the estimation between the frame n and the frame n + 1. Expected to be lower.

  Therefore, in order to reduce the risk, the region estimation result is not used as it is, but the region estimation result obtained in the previous frame n-1 (or a frame earlier in time) is used as the movement destination in the next frame. It is desirable to use as a transfer candidate for improving the performance.

  However, the region estimation result can be used as it is.

  FIG. 54 shows details of the template creation processing in step S1163 of FIG. In step S1281, the template creation unit 1163 determines a small region corresponding to each of the points estimated as the region (region of full screen motion). In the example of FIG. 55, the small region 1322 is determined corresponding to the point 1321 of the region.

  In step S1282, the template creation unit 1163 sets the area of the sum of the small areas determined in the process of step S1281 as the template range. In the example of FIG. 55, the sum area of the small areas 1322 is a template range 1331.

  In step S1283, the template creation unit 1163 creates a template from the template range information and image information set in step S1282, and supplies the template to the template holding unit 1058 for holding. Specifically, pixel data in the template range 1331 is used as a template.

  56, the small region 1341 corresponding to the point 1321 of the region has a larger area than the small region 1322 in FIG. As a result, the template range 1351 of the sum area of the small areas 1341 is also wider than the template range 1331 of FIG.

  It is conceivable that the size of the small region is proportional to the interval between the sample points, but the proportionality constant at that time can be determined so that the area is the square of the sample point interval, or larger or smaller than that. It is also possible.

  For example, a fixed size or shape range centered on the tracking point may be used as the template range without using the region estimation result.

  FIG. 57 shows the positional relationship between the template and the area estimation range. The template range 1403 includes a tracking point 1405. The upper left point of the circumscribed rectangle 1401 circumscribing the template range 1403 is a template reference point 1404. The vector 1406 from the template reference point 1404 to the tracking point 1405 and the vector 1407 from the template reference point 1404 to the upper left reference point 1408 in the region estimation range 1402 are used as the template range 1403 information. The template is composed of pixels included in the template range 1403. The vectors 1406 and 1407 are used for returning to normal processing when the same image as the template is detected.

  In the above processing, unlike the case of the transfer candidate, the example in which both the range and the pixel correspond to the current frame is used as the template. However, as in the case of the transfer candidate, the destination in the next frame is set as the template. Can also be used.

  As described above, a template made up of pixel data including a tracking point is created in advance during normal processing in the same manner as a transfer candidate.

  Details of the exception processing in step S1052 performed following the normal processing in step S1051 of FIG. 39 described above will be described with reference to the flowchart of FIG. As described above, this process is performed when it is determined in step S1124 in FIG. 43 that it is impossible to estimate the movement of the tracking point, and further, in step S1128, it is determined that the transfer candidate for changing the tracking point cannot be selected. Will be executed.

  In step S1401, the control unit 1059 executes exception processing initialization processing. The details of this processing are shown in the flowchart of FIG.

  In step S1421, the control unit 1059 causes a scene change when tracking of the tracking point cannot be performed (when the movement of the tracking point cannot be estimated and a transfer candidate for changing the tracking point cannot be selected). It is determined whether it has been. The scene change detection unit 1053 constantly monitors whether there is a scene chain based on the estimation result of the motion estimation unit 1052, and the control unit 1059 performs step S1421 based on the detection result of the scene change detection unit 1053. Execute the judgment. Specific processing of the scene change detection unit 1053 will be described later with reference to FIGS. 71 and 72.

  If a scene change has occurred, it is presumed that the reason why tracking is not possible is that a scene change has occurred, and in step S1422, the control unit 1059 sets the mode to scene change. On the other hand, if it is determined in step S1421 that no scene change has occurred, the control unit 1059 sets the mode to another mode in step S1423.

  After the process of step S1422 or step S1423, in step S1424, the template matching unit 1051 executes a process of selecting the oldest template in terms of time. Specifically, as shown in FIG. 60, for example, when an exception process is executed when moving from frame n to frame n + 1, a frame n−m + 1 is generated for frame n and is stored in the template holding unit 1058. A template generated with respect to the frame mn + 1, which is the oldest template in time, is selected from the held m-frame templates.

  In this way, the template just before the transition to exception processing (the template generated for frame n in the case of FIG. 60) is not used, but the template slightly before in time is selected by the occlusion to be tracked or the like. This is because when the transition to exception processing occurs, the tracking target is already considerably hidden immediately before the transition, and it is highly likely that the tracking target cannot be captured sufficiently large in the template at that time. Therefore, reliable tracking is possible by selecting a template in a frame slightly before in time.

  Next, in step S1425, the template matching unit 1051 executes processing for setting a template search range. For example, the template search range is set such that the position of the tracking point immediately before the transition to the exception process is the center of the template search range.

  That is, as shown in FIG. 61, when the right eye 1102 of the subject's face 1104 is designated as the tracking point 1101 in the frame n, the ball 1121 flies from the left direction in the figure, and the tracking point 1101 in the frame n + 1. Suppose that the face 1104 including is hidden and the tracking point 1101 appears again in the frame n + 2. In this case, a region centered on the tracking point 1101 (included in the template range 1411) is set as the template search range 1412.

  In step S1426, the template matching unit 1051 resets the number of elapsed frames and the number of scene changes after shifting to exception processing to zero. The number of frames and the number of scene changes are used in continuation determination processing (steps S1461, S1463, S1465, and S1467 in FIG. 63) in step S1405 in FIG.

  After the exception process initialization process is completed as described above, in step S1402 of FIG. 58, the control unit 1059 executes a process of waiting for the next frame. In step S1403, the template matching unit 1051 performs a template matching process within the template search range. In step S1404, the template matching unit 1051 determines whether it is possible to return to normal processing.

  Specifically, the absolute value sum of the difference between the template several frames before (a pixel in the template range 1411 in FIG. 61) and the matching target pixel in the template search range is calculated by the template matching process. More specifically, the sum of absolute values of differences between the pixels in the predetermined block in the template range 1411 and the predetermined block in the template search range is calculated. The position of the block is sequentially moved within the template range 1411, and the sum of absolute values of the differences of each block is added to obtain a value at the position of the template. Then, the position where the sum of absolute values of the differences becomes the smallest when the template is sequentially moved within the template search range and its value are searched. In step S1404, the absolute sum of the minimum differences is compared with a predetermined threshold value set in advance. If the sum of absolute values of the differences is less than or equal to the threshold value, an image including the tracking point (included in the template) has appeared again, so it is determined that the normal processing can be restored, and the processing Returns to the normal processing of step S1051 of FIG.

  Then, as described above, in step S1141 of FIG. 44, it is determined that the process returns from the exception process. In step S1144, the position where the difference absolute value sum is minimum is set as the template-matched position. The tracking point and area estimation range are set based on the positional relationship between the template position and the tracking point area estimation range held corresponding to the template. That is, as described above with reference to FIG. 57, the area estimation range 1402 is set based on the vectors 1406 and 1407 with the tracking point 1405 as a reference.

  However, in the region estimation process in step S1161 of FIG. 47, when a method that does not use the region estimation range is used, the region estimation range is not set.

  In step S1404 in FIG. 58, it is determined whether or not it is possible to return to normal processing by comparing the value obtained by dividing the minimum sum of absolute differences by the activity of the template with a threshold value. Also good. As the activity in this case, the value calculated in step S1603 in FIG. 65 by the activity calculation unit 1602 in FIG. 64 described later can be used.

  Alternatively, it is possible to return to the normal processing by comparing a value obtained by dividing the current minimum absolute difference sum by the minimum absolute difference sum one frame before with a predetermined threshold. It may be determined whether or not. In this case, it is not necessary to calculate the activity. That is, in step S1404, the correlation between the template and the template search range is calculated, and determination is performed based on the comparison between the correlation value and the threshold value.

  If it is determined in step S1404 that it is not possible to return to the normal process, the process proceeds to step S1405, and the continuation determination process is executed. The details of the continuation determination process will be described later with reference to the flowchart of FIG. 63, whereby it is determined whether or not the exception process can be continued.

  In step S1406, the control unit 1059 sets whether exception processing (tracking of tracking points in exception processing) can be continued based on the result of the continuation determination processing (steps S1466 and S1468 in FIG. 63 described later). (Based on the flag that was set). If the exception process can be continued, the process returns to step S1402, and the subsequent processes are repeatedly executed. That is, the process of waiting until the tracking point appears again is repeatedly executed.

  On the other hand, if it is determined in step S1406 that exception processing cannot be continued (whether it is determined in step S1465 in FIG. 63 described later that the number of frames that have elapsed after the tracking point disappears is equal to or greater than the threshold value THfr. Or, if it is determined in step S1467 that the number of scene changes is equal to or greater than the threshold value THsc), the exception process is no longer possible and the tracking process is terminated. Instead of ending the tracking process, it may be possible to return to the normal process again using the tracking point that has been held. The exception handling in this case is shown in FIG. 62 are the same as steps S1401 to S1405 in FIG. 58, and thus the description thereof is omitted.

  That is, when it is determined whether or not exception processing can be continued by the continuation determination processing in step S1445, then in step S1446, the control unit 1059 performs exception processing (tracking of tracking points in exception processing). Is determined based on the result of the continuation determination process (based on a flag set in steps S1466 and S1468 in FIG. 63 described later). If the exception process can be continued, the process returns to step S1442 and the subsequent processes are repeatedly executed. That is, the process of waiting until the tracking point appears again is repeatedly executed.

  On the other hand, if it is determined in step S1446 that the exception processing cannot be continued (whether it is determined in step S1465 in FIG. 63 described later that the number of elapsed frames after the tracking point disappears is equal to or greater than the threshold value THfr. Or, when it is determined in step S1467 that the number of scene changes is equal to or greater than the threshold THsc), the exception processing is no longer possible and the processing returns to the normal processing in step S1051 of FIG.

  In this case, as described above, in step S1141 in FIG. 44, it is determined that the process returns from the exception process, and in step S1144, based on the position of the tracking point immediately before the transition to the exception process held. The tracking point and the area estimation range are set.

  FIG. 63 shows details of the continuation determination process in step S1405 (or step S1445 in FIG. 62) in FIG. In step S1461, the control unit 1059 executes a process of adding 1 to the number of elapsed frames as a variable. The number of elapsed frames is previously reset to 0 in the exception process initialization process (step S1426 in FIG. 59) in step S1401 in FIG.

  Next, in step S1462, the control unit 1059 determines whether or not there is a scene change. Whether or not there is a scene change can be determined based on the detection result, since the scene change detection unit 1053 always executes the detection process. If there is a scene change, the process advances to step S1463, and the control unit 1059 adds 1 to the number of scene changes as a variable. The number of scene changes is also reset to 0 in the initialization process of step S1426 in FIG. If no scene change has occurred during the transition from the normal process to the exception process, the process of step S1463 is skipped.

  Next, in step S1464, the control unit 1059 determines whether or not the currently set mode is a scene change. This mode is set in steps S1422 and S1423 of FIG. If the currently set mode is a scene change, the process advances to step S1467, and the control unit 1059 determines whether or not the number of scene changes is smaller than a preset threshold value THsc. If the number of scene changes is smaller than the threshold THsc, the process proceeds to step S1466, and the control unit 1059 sets a flag that allows continuation. If the number of scene changes is equal to or greater than the threshold THsc, the process proceeds to step S1468 and cannot be continued. Set the flag.

  On the other hand, when it is determined in step S1464 that the mode is not a scene change (when it is determined that the mode is other), the process proceeds to step S1465, and the control unit 1059 determines whether or not the number of elapsed frames is smaller than the threshold value THfr. Determine whether. The number of elapsed frames is also reset to 0 in advance in step S1426 of the exception process initialization process of FIG. If it is determined that the number of elapsed frames is smaller than the threshold value THfr, a continuation flag is set in step S1466, and if it is determined that the number of elapsed frames is greater than or equal to the threshold value THfr, the process continues in step S1468. A disabled flag is set.

  As described above, when the number of scene changes at the time of template matching processing is equal to or greater than the threshold value THsc, or when the number of elapsed frames is equal to or greater than the threshold value THfr, further exception processing is impossible.

  When the mode is other, a condition that the number of scene changes is 0 may be added to determine whether or not continuation is possible.

  In the above, it is assumed that the frame of the image is used as a processing unit and all frames are used. However, processing is not performed in units of fields or using all frames or fields, but is extracted by thinning out at a predetermined interval. It is also possible to use a modified frame or field.

  Further, in the above, the estimated movement destination of the point in the area is used as the transfer candidate. However, in this case, even if the whole screen motion is (0, 0), each point in the area is , (-1, 1), (1, 0), etc., the movement is shifted by the amount of each movement. Instead of using the destination point as a transfer candidate as it is, it is also possible to set the closest point among sample points obtained in advance as a transfer candidate. Of course, in order to reduce the processing load, each point may be shifted by the amount corresponding to the full screen movement.

  Further, instead of using the estimated destination of the point in the area as the transfer candidate, it is possible to use the point in the area as it is.

  Next, a configuration example of the motion estimation unit 1052 in FIG. 38 will be described with reference to FIG. In this embodiment, the input image is supplied to the evaluation value calculation unit 1601, the activity calculation unit 1602, and the motion vector detection unit 1606. The evaluation value calculation unit 1601 calculates an evaluation value related to the degree of coincidence between the two objects associated by the motion vector, and supplies the evaluation value to the normalization processing unit 1604. The activity calculation unit 1602 calculates the activity of the input image and supplies it to the threshold value determination unit 1603 and the normalization processing unit 1604. The motion vector detection unit 1606 detects a motion vector from the input image and supplies it to the evaluation value calculation unit 1601 and the integration processing unit 1605.

  The normalization processing unit 1604 normalizes the evaluation value supplied from the evaluation value calculation unit 1601 based on the activity supplied from the activity calculation unit 1602, and supplies the obtained value to the integration processing unit 1605. The threshold determination unit 1603 compares the activity supplied from the activity calculation unit 1602 with a predetermined threshold, and supplies the determination result to the integration processing unit 1605. The integration processing unit 1605 calculates the accuracy of the motion vector based on the normalization information supplied from the normalization processing unit 1604 and the determination result supplied from the threshold determination unit 1603, and the obtained accuracy is detected by the motion vector detection. The motion vector supplied from the unit 1606 is output.

  Next, the motion estimation process of the motion estimation unit 1052 will be described with reference to the flowchart in FIG. Although the motion vector is obtained for a point, the accuracy is calculated using image data of a small block in the vicinity of two points associated by the motion vector, for example, centering on the point. In step S1601, the motion vector detection unit 11606 detects a motion vector from the input image. For this detection, for example, a block matching method or a gradient method is used. The detected motion vector is supplied to the evaluation value calculation unit 1601 and the integration processing unit 1605.

  In step S1602, the evaluation value calculation unit 1601 calculates an evaluation value. Specifically, for example, the sum of absolute differences of pixel values of two blocks centered on two points associated with motion vectors is calculated. That is, the motion vector V (vx, vy) detected by the motion vector detection unit 1606 in step S1601, the point P (Xp, Yp) on the image Fi of the previous frame based on the motion vector, and the later time The relationship of the point Q (Xq, Yq) on the image Fj of the frame is expressed by the following equation.

  The evaluation value calculation unit 1601 calculates an evaluation value Eval (P, Q, i, j) for a block centered on the point P and a block centered on the point Q based on the following equation.

  Each block is a square having 2L + 1 pixels on one side. The summation ΣΣ in the above equation is performed between corresponding pixels when x is from −L to L and y is from −L to L. Therefore, for example, when L = 2, nine differences are obtained, and the sum of the absolute values is calculated. The evaluation value indicates that the two blocks match well as the value approaches zero.

  The evaluation value calculation unit 1601 supplies the generated evaluation value to the normalization processing unit 1604.

  In step S1603, the activity calculation unit 1602 calculates an activity from the input image. The activity is a feature amount representing the complexity of the image, and as shown in FIG. 66, the difference between the pixel of interest Y (x, y) and the adjacent 8 pixels Y (x + i, y + j) for each pixel. The average value of the sum of absolute values is calculated based on the following formula as activity Activity (x, y) at the target pixel position.

  In the example of FIG. 66, among 3 × 3 pixels, the value of the pixel of interest Y (x, y) located at the center is 110, and the values of eight pixels adjacent to it are 80, 70, and 75, respectively. , 100, 100, 100, 80, 80, the activity Activity (x, y) is expressed by the following equation.

  Activity (x, y) = {| 80-110 | + | 70-110 | + | 75-110 | + | 100-110 | + | 100-110 | + | 80-110 | + | 80−110 |} /8=24.375.

  Similar processing is performed for all pixels in the frame.

  In order to calculate the motion vector accuracy in units of blocks, the sum of the activities of all the pixels in the block expressed by the following equation is defined as the activity (block activity) Blockactivity (i, j) of the block.

  In addition, the activity may be a variance value, a dynamic range, or the like.

  In step S1604, the threshold determination unit 1603 compares the block activity calculated by the activity calculation unit 1602 with a predetermined threshold set in advance. Then, a flag indicating whether or not the input block activity is larger than the threshold value is output to the integration processing unit 1605.

  Specifically, as a result of the experiment, the block activity and the evaluation value have the relationship shown in FIG. 67 using the motion vector as a parameter. In FIG. 67, the horizontal axis represents block activity Blockactivity (i, j), and the vertical axis represents the evaluation value Eval. When the motion is correctly detected (when the correct motion vector is given), the block activity and the evaluation value are distributed in a region R1 below the curve 1621 in the figure. On the other hand, if an incorrect motion (incorrect motion vector) is given, the block activity and the evaluation value are distributed in the region R2 on the left side of the diagram from the curve 1622 (the region above the curve 1622). There is almost no distribution in the region other than R2 and the region other than the region R1 below the curve 1621). Curves 1621 and 1622 intersect at point P. The value of the block activity at this point P is set as the threshold value THa. The threshold value THa means that if the value of the block activity is smaller than that, the corresponding motion vector may be incorrect (this will be described in detail later). The threshold determination unit 1603 outputs a flag indicating whether or not the value of the block activity input from the activity calculation unit 1602 is greater than the threshold THa to the integration processing block 1605.

  In step S1605, the normalization processing unit 1604 executes normalization processing. Specifically, the normalization processing unit 1604 calculates the motion vector accuracy VC according to the following equation.

  However, if the value of the motion vector accuracy VC is less than 0, the value is replaced with 0. Of the motion vector accuracy VC, the value obtained by dividing the evaluation value by the block activity is a straight line 1623 where the position on the graph of FIG. 67 defined by the value is the slope connecting the origin O and the point P is 1. Therefore, it indicates whether the region is in the lower region or the upper region in the figure. That is, the slope of the straight line 1623 is 1, and if the value obtained by dividing the evaluation value by the block activity is greater than 1, the points corresponding to the value are points distributed in the area above the straight line 1623. Means that. The motion vector accuracy VC obtained by subtracting this value from 1 means that the smaller the value, the higher the possibility that the corresponding points are distributed in the region R2.

  On the other hand, if the value obtained by dividing the evaluation value by the block activity is smaller than 1, it means that the points corresponding to the value are distributed in the lower area of the straight line 1623 in the figure. The motion vector accuracy VC at that time means that the larger the value (closer to 0), the corresponding points are distributed in the region R1. The normalization processing unit 1604 outputs the motion vector accuracy VC obtained by the calculation in this way to the integration processing unit 1605.

  In step S1606, the integration processing unit 1605 executes integration processing. Details of this integration processing are shown in the flowchart of FIG.

  In step S1631, the integration processing unit 1605 determines whether the block activity is equal to or less than the threshold value THa. This determination is performed based on the flag supplied from the threshold determination unit 1603. If the block activity is less than or equal to the threshold THa, the integration processing unit 1605 sets the value of the motion vector accuracy VC calculated by the normalization processing unit 1604 to 0 in step S1632. If it is determined in step S1631 that the activity value is greater than the threshold value THa, the processing in step S1632 is skipped, and the value of the motion vector accuracy VC generated by the normalization processing unit 1604 is output as it is together with the motion vector. Is done.

  This is because even if the value of the motion vector accuracy VC calculated by the normalization processing unit 1604 is positive, if the block activity value is smaller than the threshold value THa, a correct motion vector may not be obtained. Because there is. That is, as shown in FIG. 67, between the origin O and the point P, the curve 1622 protrudes from the curve 1621 to the lower side in the drawing (lower than the straight line 1623). In a region R3 in which the value of the block activity is smaller than the threshold Tha and is surrounded by the curves 1621 and 1622, the value obtained by dividing the evaluation value by the block activity is distributed in both the regions R1 and R2. There is a high possibility that a correct motion vector is not obtained.

  Therefore, in such a distribution state, processing is performed assuming that the accuracy of the motion vector is low. Therefore, in step S1632, the motion vector accuracy VC is set to 0 when the value is positive even if the value is positive. In this way, when the value of the motion vector accuracy VC is positive, it is possible to reliably represent that the correct motion vector is obtained. In addition, the larger the value of the motion vector accuracy VC, the higher the probability that a correct motion vector is obtained (the probability that the distribution is included in the region R1 increases).

  This coincides with an empirical rule that, in general, it is difficult to detect a motion vector with high reliability in a region where the luminance change is small (region where the activity is small).

  FIG. 69 shows a configuration example of the background motion estimation unit 1054 of FIG. In this configuration example, the background motion estimation unit 1054 includes a frequency distribution calculation unit 1651 and a background motion determination unit 1652.

  The frequency distribution calculation unit 1651 calculates a frequency distribution of motion vectors. However, this frequency is weighted so that a probable motion is weighted by using the motion vector accuracy VC supplied from the motion estimation unit 1052. Based on the frequency distribution calculated by the frequency distribution calculation unit 1651, the background motion determination unit 1652 performs a process of determining a motion with the maximum frequency as a background motion, and outputs the background motion to the region estimation related processing unit 1055.

  The background motion estimation process of the background motion estimation unit 54 will be described with reference to FIG.

  In step S1651, the frequency distribution calculation unit 1651 calculates a motion frequency distribution. Specifically, the frequency distribution calculating unit 1651 has 1089 (= 16 ×) assuming that the x coordinate and the y coordinate of the motion vector as a background motion candidate are each expressed in a range of ± 16 pixels from the reference point. 2 + 1) × (16 × 2 + 1)), that is, a box for coordinates corresponding to possible values of a motion vector, and when a motion vector is generated, 1 is added to the coordinates corresponding to the motion vector. In this way, the motion vector frequency distribution can be calculated.

  However, when one motion vector is generated, if 1 is added, if the frequency of occurrence of a motion vector with low accuracy is high, a motion vector with low certainty may be determined as the background motion. . Therefore, when a motion vector is generated, the frequency distribution calculation unit 1651 does not add a value 1 to a box (coordinates) corresponding to the motion vector, but a value obtained by multiplying the value 1 by the motion vector accuracy VC (= Motion vector accuracy VC). The value of the motion vector accuracy VC is normalized as a value between 0 and 1, and the closer the value is to 1, the higher the accuracy. Therefore, the frequency distribution obtained in this way is a frequency distribution obtained by weighting motion vectors based on their accuracy. This reduces the risk that a motion with low accuracy is determined as the background motion.

  Next, in step S1652, the frequency distribution calculation unit 1651 determines whether or not the process of calculating the motion frequency distribution has been completed for all blocks. If there is a block that has not yet been processed, the process returns to step S1651, and the process of step S1651 is executed for the next block.

  As described above, the motion frequency distribution calculation process is performed on the entire screen, and if it is determined in step S1652 that the processing of all blocks has been completed, the process proceeds to step S1653, and the background motion determination unit 1652 A process for searching for the maximum value of the distribution is executed. That is, the background motion determination unit 1652 selects a frequency having the maximum frequency from the frequencies calculated by the frequency distribution calculation unit 1651, and determines a motion vector corresponding to the frequency as a motion vector of the background motion. The motion vector of the background motion is supplied to the region estimation related processing unit 1055, and is used for, for example, a determination process of whether or not the full screen motion and the background motion match in step S1204 of FIG.

  FIG. 71 shows a detailed configuration example of the scene change detection unit 1053 of FIG. In this example, a scene change detection unit 1053 is configured by a motion vector accuracy average calculation unit 1671 and a threshold determination unit 1672.

  The motion vector accuracy average calculation unit 1671 calculates the average value of all the screens of the motion vector accuracy VC supplied from the motion estimation unit 1052, and outputs the average value to the threshold determination unit 1672. The threshold value determination unit 1672 compares the average value supplied from the motion vector accuracy average calculation unit 1671 with a predetermined threshold value, and determines whether or not the scene change is based on the comparison result. The result is output to the control unit 1059.

  Next, the operation of the scene change detection unit 1053 will be described with reference to the flowchart of FIG. In step S1681, the motion vector accuracy average calculation unit 1671 calculates the sum of vector accuracy. Specifically, the motion vector accuracy average calculation unit 1671 performs a process of adding the value of the motion vector accuracy VC calculated for each block output from the integration processing unit 1605 of the motion estimation unit 1052.

  In step S1682, the motion vector accuracy average calculation unit 1671 determines whether or not the processing for calculating the sum of the vector accuracy VC has been completed for all the blocks. If the processing has not yet been completed, the processing of step S1681 is repeated. . By repeating this process, the sum of motion vector accuracy VC of each block for one screen is calculated. If it is determined in step S1682 that the calculation of the sum of motion vector accuracy VCs for one entire screen has been completed, the process advances to step S1683, and the motion vector accuracy average calculation unit 1671 calculates the average value of the vector accuracy VC. Execute. Specifically, a value obtained by dividing the sum of the vector accuracy VC for one screen calculated in the process of step S1681 by the added number of blocks is calculated as an average value.

  In step S1684, the threshold value determination unit 1672 compares the average value of the motion vector accuracy VC calculated by the motion vector accuracy average calculation unit 1671 in the process of step S1683 with a preset threshold value, and whether or not the threshold value is smaller than the threshold value. Determine whether. In general, when a scene change occurs between two frames having different times in a moving image, there is no corresponding image, so even if a motion vector is calculated, the motion vector is not certain.

  Therefore, if the average value of the vector accuracy VC is smaller than the threshold value, the threshold value judgment unit 1672 turns on the scene change flag in step S1865. If not smaller than the threshold value (if it is equal to or larger than the threshold value), in step S1586, the scene is determined. Turn off the change flag. When the scene change flag is on, it indicates that there is a scene change, and when the scene change flag is off, it indicates that there is no scene change.

  This scene change flag is supplied to the control unit 1059 and is used to determine whether or not there is a scene change in step S1421 in FIG.

  As described above, by configuring the tracking processing unit 71 in FIG. 3, an object to be tracked is rotated (FIG. 40), occlusion occurs (FIG. 41), or a scene change point causes an object tracking point. Even when the image is temporarily not displayed (FIG. 42), the moving object (tracking point) in the image can be accurately tracked.

  The position information of the tracking point of the object tracked in this way is output to the tracking processing control unit 72 as a tracking result by the tracking processing unit 71 of FIG. The tracking processing control unit 72 executes position calculation processing by the tracking processing control unit 72 based on the tracking result stored in the tracking result storage unit 81 in step S3 of FIG.

  As described above, the candidate position that is the target of the tracking process is calculated from a fixed point on the display screen, a point on the image based on the image feature amount, or a tracking result obtained by a plurality of tracking processes. Candidate positions with high reliability can be displayed on the display unit 21.

  As a result, the tracking method being used cannot be accurately tracked due to the occurrence of occlusion, or the desired tracking is not being performed due to the occurrence of a relatively long occlusion, etc. Even so, the user can easily reset the desired tracking target simply by selecting and instructing the candidate position displayed on the display unit 21.

  Accordingly, in the monitoring system of FIG. 1, the imaging device 11 is a very inexpensive camera that does not have an optical zoom function. Even if the intruder B is excluded from the tracking target, Since the user A who is a monitor can quickly correct the position of the tracking target without stopping or temporarily stopping the reproduction, the intruder B can be tracked with a simple operation. The zoomed image 32 can be seen.

  As a result, it is possible to provide a monitoring system that is cheaper and more secure than the prior art.

  FIG. 73 shows a configuration example when the present invention is applied to an animal appreciation system. In this animal appreciation system, the imaging device 11 and the tracking device 12 connected to the imaging device 11 and using the tracking device 12 having the display unit 21 are used to view an image captured by the imaging device 11 and displayed on the display unit 21. The monkey 2001 moving around a predetermined area of the zoo is carefully watched by the user C who is a viewer.

  The imaging device 11 captures a predetermined area of the zoo and inputs the image 2002 to the tracking device 12. That is, an image 2002 in which a monkey 2001 moving around in a predetermined area is captured is input to the tracking device 12.

  The tracking device 12 uses the input image 2002 to track the monkey 2001 as a tracking target in response to an instruction from the user C, and generates, for example, a zoomed image 2003 based on the tracking result. This is displayed on the unit 21. Since the monkey 2001 moves around, it is difficult to perform accurate tracking for a long time. When the monkey 2001 is deviated from the tracking target, the tracking device 12 is instructed to display the candidate position of the tracking target.

  In the tracking device 12, as described above, a candidate position to be subjected to the tracking process from a fixed point on the display screen, a point on the image based on the image feature amount, or a tracking result obtained by a plurality of tracking processes. Therefore, the candidate position with high reliability can be displayed on the display unit 21. Accordingly, the user can easily reset the desired monkey 2001 as a tracking target simply by selecting and instructing the candidate position displayed on the display unit 21.

  Thus, in the tracking device 12, even when the tracking target is removed, the correction can be made immediately, so that the user C can enjoy watching the monkey 2001 without missing a valuable opportunity. .

  It is possible to record the video captured by the imaging device in advance, and then perform tracking zoom, but the atmosphere that can only be experienced in real time (real world) will be lost, so the fun is There is a risk of drastic decrease.

  That is, the present invention is particularly effective for a system in which a user operation result is reflected in a tracking result in real time.

  The present invention can be applied not only to a monitoring system and an animal appreciation system but also to a television receiver and various image processing apparatuses.

  In the above description, the processing unit of an image is a frame, but the present invention can also be applied to a case where a field is a processing unit.

  It does not matter whether the above-described series of processing is realized by hardware or software. When the above-described series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. It is installed on a general-purpose personal computer or the like from a recording medium such as a network or a removable medium.

  In addition, the steps of executing the series of processes described above in this specification are performed in parallel or individually even if they are not necessarily processed in time series, as well as processes performed in time series in the order described. The processing to be performed is also included.

It is a figure which shows the structural example of the monitoring system to which this invention is applied. It is a block diagram which shows the structural example of the tracking apparatus of FIG. It is a block diagram which shows the structural example of the object tracking part of FIG. It is a flowchart explaining the process of a tracking apparatus. It is a flowchart explaining the position calculation process of step S3 of FIG. It is a figure which shows the example of a display image. It is a block diagram which shows the other structural example of the object tracking part of FIG. It is a flowchart explaining the other example of a position calculation process. It is a block diagram which shows the further another structural example of the object tracking part of FIG. It is a flowchart explaining the further another example of a position calculation process. It is a figure which shows the example of a display image. 10 shows another configuration example of the object tracking unit in FIG. 9. It is a flowchart explaining the other example of a position calculation process. It is a flowchart explaining a tracking result update process. It is a figure explaining the tracking result update process of FIG. It is a figure explaining the tracking result update process of FIG. It is a flowchart explaining a tracking result update process. It is a figure explaining the tracking result update process of FIG. It is a figure explaining the tracking result update process of FIG. It is a figure explaining the other example of a tracking result update process. It is a figure explaining the system which extrapolates for a fixed time with a past motion. It is a figure explaining the system which extrapolates for a fixed time with a past motion. It is a figure which shows the example of a display image. It is a figure which shows the structural example of the remote controller of FIG. It is a figure which shows the example of a display image. It is a figure which shows the example of a transition of a display image. It is a figure which shows the other structural example of the display image generation part of FIG. It is a flowchart explaining a display image generation process. It is a figure which shows the example of a display image. It is a figure which shows the example of a display image. It is a figure which shows the example of a transition of a display image. It is a figure which shows the example of a transition of a display image. It is a figure which shows the example of a display image. It is a block diagram which shows the structural example of the tracking process part of FIG. It is a flowchart explaining a tracking process. It is a figure explaining the area | region which detects a motion vector. It is a figure explaining the frequency of a motion vector. It is a block diagram which shows the structural example of the tracking process part of FIG. It is a flowchart explaining a tracking process. It is a figure explaining tracking when a tracking object rotates. It is a figure explaining the tracking when an occlusion occurs. It is a figure explaining the tracking when a scene change occurs. It is a flowchart explaining a normal process. It is a flowchart explaining the initialization process of a normal process. It is a figure explaining a transfer candidate extraction process. It is a block diagram which shows the structural example of an area | region estimation related process part. It is a flowchart explaining an area | region estimation related process. It is a flowchart explaining area | region estimation processing. It is a figure explaining the process which determines a sample point. It is a flowchart explaining the update process of an area estimation range. It is a figure explaining the update of a region estimation range. It is a figure explaining the update of a region estimation range. It is a flowchart explaining a transfer candidate extraction process. It is a flowchart explaining a template creation process. It is a figure explaining template creation. It is a figure explaining template creation. It is a figure explaining the positional relationship of a template and a tracking point. It is a flowchart explaining exception processing. It is a flowchart explaining the initialization process of an exception process. It is a figure explaining selection of a template. It is a figure explaining the setting of a search range. It is a flowchart explaining the other example of exception processing. It is a flowchart explaining a continuation determination process. It is a block diagram which shows the structural example of a motion estimation part. It is a flowchart explaining a motion estimation process. It is a figure explaining calculation of activity. It is a figure explaining the relationship between an evaluation value and activity. It is a flowchart explaining an integration process. It is a block diagram which shows the structural example of a background motion estimation part. It is a flowchart explaining a background motion estimation process. It is a block diagram which shows the structural example of a scene change detection part. It is a flowchart explaining a scene change detection process. It is a figure which shows the structural example of the animal appreciation system to which this invention is applied.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 11 Imaging device, 12 Tracking apparatus, 21 Display part, 52 Object tracking part, 53 Whole system control part, 54 Display image generation part, 55 Remote controller, 71,71-1 thru | or 71-n Tracking process part, 72 Tracking process control Unit, 81 tracking result storage unit, 82 position calculation unit, 83 target position setting unit, 131 image feature amount calculation unit, 161 tracking result update unit, 301 enlarged signal processing unit, 302 tracking result selection candidate display unit

Claims (21)

In an image processing apparatus for displaying a moving object,
In response to a user operation, tracking means for tracking a moving object on the image as a tracking target;
Candidate calculation means for calculating a candidate position as the tracking target candidate by the tracking means;
Display control means for controlling the display of the candidate position calculated by the candidate calculation means;
An image processing apparatus comprising: a target setting unit configured to set the displayed candidate position as the tracking target in a frame next to the tracking unit in response to a user operation. The candidate calculation means includes
The image processing apparatus according to claim 1, wherein a predetermined position in a screen stored in advance is read to calculate the candidate position. The candidate calculation means includes
The image processing apparatus according to claim 1, wherein the candidate position is calculated based on a feature amount of the image. The candidate calculation means includes
The image processing apparatus according to claim 1, wherein the candidate position is calculated based on tracking results obtained by a plurality of tracking units. The image processing apparatus according to claim 4, wherein the plurality of tracking units perform tracking using a plurality of different types of tracking methods. The target setting means includes
The image processing apparatus according to claim 5, wherein the candidate position to be displayed is set as the tracking target in a next frame of the plurality of tracking units in response to a user operation. The image processing apparatus according to claim 4, wherein the plurality of tracking units perform tracking using a plurality of different neighboring positions on the object as tracking targets. The target setting means includes
8. A plurality of different neighboring positions including the candidate position are set as the tracking target in the next frame of the plurality of tracking means based on the displayed candidate position in response to a user operation. An image processing apparatus according to 1. 5. The updating device according to claim 4, further comprising: an updating unit configured to update a tracking result of a part or all of the plurality of tracking units based on a tracking result of one tracking unit among the plurality of tracking units. Image processing apparatus. The updating unit is configured to track a part or all of the plurality of tracking units based on a tracking result by one of the plurality of tracking units every time a predetermined time elapses. The image processing apparatus according to claim 9, wherein the result is updated. The updating means is based on a result of tracking by one of the plurality of tracking means at a first timing when a predetermined time has elapsed, and is based on a part of the plurality of tracking means. Update tracking results,
Different from the first timing, at every second timing when the predetermined time has elapsed, the tracking result by one tracking means of the plurality of tracking means is another one of the plurality of tracking means. The image processing apparatus according to claim 9, wherein the tracking result by the tracking unit of the unit is updated. The updating unit is configured to perform a part or all of the plurality of tracking units based on a tracking result of one tracking unit among the plurality of tracking units when the tracking results of the plurality of tracking units are greatly different. The image processing apparatus according to claim 9, wherein the tracking result by the tracking unit is updated. The image processing apparatus according to claim 1, wherein the display control unit controls a list display of the candidate positions that are displayed on the image while distinguishing candidate positions being selected by a user operation from other candidate positions. The display control means superimposes a first small image on the selected candidate position on the image, and differs from the first small image on the other candidate position on the image. The image processing apparatus according to claim 13, wherein a list display of the candidate positions is controlled by superimposing two small images. The display control means includes
An image generating means for generating a zoom image centered on the candidate position;
The image processing apparatus according to claim 1, wherein display of a zoom image centered on the candidate position generated by the image generation unit is controlled. The display control means includes
The image processing apparatus according to claim 15, wherein display of a plurality of zoom images centered on the plurality of candidate positions generated by the image generation unit is controlled. The display control means includes
In the zoom image centered on the candidate position generated by the image generation means, the candidate position being selected by the user's operation is displayed as a list of candidate positions displayed on the image in distinction from other candidate positions. The image processing apparatus according to claim 15, wherein the display is controlled. The display control means includes
The candidate position currently selected by the user's operation is distinguished from other candidate positions on the list display of the candidate position shown on the image, and the zoom centered on the candidate position generated by the image generation means The image processing apparatus according to claim 15, wherein display on which an image is superimposed is controlled. In an image processing method of an image processing apparatus for displaying a moving object,
In response to the user's operation, the candidate position as the tracking target candidate of the tracking means for tracking the moving object on the image as the tracking target is calculated,
Controlling the display of the calculated candidate position;
An image processing method including a step of setting the displayed candidate position as the tracking target in the next frame of the tracking unit in response to a user operation. A program for causing a computer to perform processing for displaying a moving object,
In response to the user's operation, the candidate position as the tracking target candidate of the tracking means for tracking the moving object on the image as the tracking target is calculated,
Controlling the display of the calculated candidate position;
A program comprising the step of setting the displayed candidate position as the tracking target in the next frame of the tracking means in response to a user operation.   A recording medium on which the program according to claim 20 is recorded.

Images