JP2023161440A - Video processing device, control method for the same, and program - Google Patents

Abstract

To continue tracking a tracking target with high accuracy even when the tracking target is hidden by an overlap with another object.SOLUTION: An imaging processing device is for controlling a viewing direction with respect to a second imaging apparatus capable of making the viewing direction changeable on the basis of a video from a first imaging apparatus capturing an overhead video of the entire court where a sporting event is held. The imaging processing device has a detection unit and a control unit. The detection unit is for detecting an object in a video received from the first imaging apparatus. The control unit is for estimating a position of an object to be tracked after its movement on the basis of a movement vector of the object to be tracked among objects detected by the detection unit and controlling the second imaging apparatus so that the estimated position is located within the angle of view. The control unit has a determination unit and a setting unit. The determination unit is for determining an object that blocks the object to be tracked on the basis of the positional relationship of objects detected from the previous frame when it is determined that the object to be tracked is not included among objects detected in the current frame of a video received from the first imaging apparatus. The setting unit is for setting the object determined by the determination unit as an object to be tracked instead of the object to be tracked.SELECTED DRAWING: Figure 1

Description

The present invention relates to a video processing device, a control method thereof, and a program.

Automatic shooting of sports videos has been attracting attention in recent years from the perspective of saving labor. Automatic shooting is a technology that automatically controls cameras that pan, tilt, and zoom to track the details of a play, in addition to the camera that provides an overview of the entire sports scene. There are various ways to determine the angle of view depending on the sport in question, but in contact sports such as basketball and soccer, the angle of view is often determined based on the position of the ball.

In order to automatically keep track of a tracked target, recognition technology that uses overhead video as input is often used. However, on the other hand, in such sports scenes, the ball may not be visible from an overhead view due to overlapping players. In such a scene, the camera operator determines the angle of view by estimating the ball position. However, with the recognition technology used in automatic shooting, in such scenes where the ball cannot be captured from an overhead image, the coordinate position of the ball becomes "unknown" and the information necessary for automatically determining the angle of view is missing.

In Patent Document 1, automatic tracking is performed by performing an evaluation that takes into account overlap based on a combined area of each recognized target object and a difference area with past information. In Patent Document 2, a suspicious person candidate is determined based on changes in characteristics before and after the overlap of objects occurs, and a location where the overlap occurs is output.

Patent No. 6787075 JP2020-88687A

However, when trying to automatically shoot a sports video using the above-mentioned conventional technology, the following problems exist. First, in Patent Document 1, if there is a large difference in the size of the recognition target, changes in the larger target will greatly affect the evaluation value of the combination area, so relatively small objects such as balls are tracked. Sometimes it is not possible to track accurately. Further, in Patent Document 2, since changes in features before and after the occurrence of overlap are evaluated, the position of the object cannot be estimated while it is hidden. Therefore, this technology is not suitable for real-time distribution of sports videos.

The present invention has been made in view of these problems, and even if a plurality of objects of different sizes are being recognized and the tracked target is hidden due to overlapping, etc., the position of the tracked target can be estimated. can do.

In order to solve this problem, for example, the video processing device of the present invention has the following configuration. That is,
An image processing device that controls the viewing direction of a second imaging device whose viewing direction can be changed based on an image from a first imaging device that captures a bird's-eye view of the entire court where sports are played, the video processing device comprising:
detection means for detecting an object in the video received from the first imaging device;
Estimating the position of the tracked object after movement from the movement vector of the tracked object among the objects detected by the detection means, and controlling the second imaging device so that the estimated position is within the angle of view. and a control means for
The control means includes:
If it is determined that the tracking target object is not included in the objects detected from the current frame of the video received from the first imaging device, detecting an object blocking the tracking target object from the previous frame. determination means for determining based on the positional relationship of the objects;
and setting means for setting the object determined by the determination means as a tracking target instead of the tracking target object.

According to the present invention, even if the tracked target is hidden due to overlapping with another object, it is possible to continue tracking the tracked target with high accuracy.

FIG. 1 is a diagram showing an example of an image processing device according to a first embodiment. FIG. 1 is a block diagram showing the configuration of an image processing device according to a first embodiment. 5 is a flowchart showing a processing procedure in the first embodiment. (a) Processing procedure for the entire image processing device (b) Processing procedure when the tracking target cannot be detected FIG. 3 is a diagram illustrating an example of an overhead video according to the first embodiment. FIG. 3 is a diagram showing results of object detection processing according to the first embodiment. FIG. 3 is a diagram showing output result information of object detection processing according to the first embodiment. FIG. 3 is a diagram for explaining coordinate update processing when a tracking target has not been detected according to the first embodiment. 7 is a flowchart showing a processing procedure in the second embodiment. (a) Processing procedure for the entire image processing device (b) Processing procedure when the tracking target is detected FIG. 7 is a diagram for explaining coordinate update processing when a tracking target has been detected and another class exists within a predetermined range of the tracking target according to the second embodiment. FIG. 7 is a diagram for explaining coordinate update processing when a tracking target has been detected and no other class exists within a predetermined range of the tracking target according to the second embodiment. 7 is a flowchart showing a processing procedure according to a third embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

[First embodiment]
In the first embodiment, a system will be described in which an overhead video of a sport captured by an overhead camera is input, a set tracking target is recognized from the video, and the tracking target is automatically tracked using a pan/tilt camera. Note that in this embodiment, "class" represents a ball and a player, and it is assumed that a player can assign a class to each player. Note that, for ease of understanding, this embodiment will be explained using basketball as an example.

As shown in FIG. 1(a), the system in the first embodiment includes a video processing device 100 and cameras 100 and 200. Video processing device 100 is communicably connected to cameras 100 and 200. The connection type may be wired or wireless. The camera 101 has a viewing angle for capturing an overhead image of the entire court, and its viewing direction is fixed. On the other hand, although the camera 102 has a narrower viewing angle than the camera 101, it has a pan head 102a so that the viewing direction (pan angle, tilt angle) can be changed. There are no particular restrictions on the installation positions of the cameras 101 and 102, but in the embodiment, as shown in the figure, the camera 101 is installed on an extension of the center line of the court, and the camera 102 is installed in advance from one end line of the court. It is assumed that the Note that the cameras 101 and 102 in the embodiment shoot and transmit moving images at, for example, 30 frames/second.

FIG. 1(b) is a block configuration diagram of the video processing device 100 in the first embodiment. The video processing device 100 is realized by an information processing device, typically a general-purpose personal computer, and an application program running thereon. Therefore, the image processing device 100 has a CPU 111 that controls the entire device, a ROM 112 that stores BIOS and a boot program, an OS (operating system) executed by the CPU 111, a video processing application program in the embodiment, and a workstation. It has a RAM 113 used as an area. Furthermore, the video processing device 100 includes an HDD (hard disk drive) as an external storage device for holding an OS and video processing application programs, and for storing video obtained by shooting as a moving image file. The video processing device 100 also includes an interface 115 for communicating with the cameras 101 and 102, an operation unit (including a keyboard, a mouse, etc.) 116 that accepts operations from the user, and a It has a display unit 117 for displaying various setting menus and the like while the video processing application program in the embodiment is being executed. In order to simplify the explanation, it is assumed that the cameras 101 and 102 are each network cameras, and the interface 115 is also an interface for communicating with the network. As described above, the cameras 101, 102 and the video processing device 100 only need to be able to communicate with each other, so there is no particular restriction on the type of interface.

In the above configuration, when the video processing device 100 is powered on, the CPU 111 executes the boot program in the ROM 112, loads the OS from the HDD 114 to the RAM 113, and transfers control to the OS. A user interface using the operation unit 116 and the display unit 117 is constructed between the video processing device 100 and the user. When the user operates the operation unit 116 to input an instruction to execute a video application, the CPU 111 loads the video processing application program from the HDD 114 onto the RAM 113, and executes the application using the control section of the OS. As a result, the video processing device 100 realizes the functions described with reference to FIG. 1(a).

FIG. 2 shows a functional block diagram of the video processing device 100 when the CPU 111 executes the video application program described above. Each processing unit should be understood as a module executed by the CPU 111.

The input unit 201 receives overhead video data captured by the camera 101 and outputs the video data to the object detection unit 202. The object detection unit 202 receives video data from the input unit 201 and detects an object in the video. Then, the object detection unit 202 outputs information indicating which object is located at which position in the video. This is done by using an object detection model generated in advance from training data. In this embodiment, a neural network, particularly a convolutional neural network, is used for the object detection process, but other models may be used instead of the convolutional neural network. Furthermore, other conventional machine learning techniques other than the neural network may be used. The object detection unit 202 outputs the above detection result (information indicating the position of the object and the type of the object) to the tracking target vector calculation unit 204 and the tracking target determination unit 205.

The tracking target input unit 203 is configured with a user interface, and allows the user to input the class of the tracking target from among the classes defined in the object detection model to the image processing apparatus 100. In this embodiment, it is assumed that the "ball" class is set as the tracking target.

The tracking target vector calculation unit 204 calculates a movement vector for the class (ball in the embodiment) specified by the tracking target input unit 203 from the detection result input from the object detection unit 202, and uses the calculation result as the tracking target. It is output to the determining unit 205.

A tracking target determination unit 205 determines a tracking target based on the results of the object detection unit 202 and the tracking target vector calculation unit 204 and the class specified by the tracking target input unit 203, and tracks information about the determined tracking target. It is output to the target coordinate updating unit 206.

The tracking target coordinate updating unit 206 updates the detection coordinates of the tracking target determined by the tracking target determining unit 205, generates parameters for pan/tilt drive control, and outputs the parameters to the pan/tilt drive control unit 207.

The pan/tilt drive control unit 207 generates pan angle and tilt angle commands based on the parameters input from the tracked target coordinate update unit 206, sends the generated commands to the camera 102, and determines whether the tracked target exists within the angle of view. Have the video taken.

The input unit 208 receives moving images from the camera 102 as video data, and outputs the video data to the recording unit 209. The recording unit 209 records the video data input from the input unit 208 on the recording medium 210 (HDD 114) as a moving image data file. Note that instead of recording the video from the camera 102, it may be distributed via a network.

Hereinafter, the operation of the information processing apparatus 100 in this embodiment configured as described above will be explained along the flowchart illustrated in FIG. 3(a). Note that, in the following description, since receiving and recording video from the camera 102 is not the main focus of the present invention, the following description will focus on remote control of the camera 102.

In S301, the input unit 201 obtains an overhead video from the camera 101. FIG. 4 is an example of an overhead image. Players 401, 402, 403, 404 and a ball 405 are captured in the video. The illustration shows that a player 401 is holding a ball 405. In the embodiment, the camera 102 is required to be controlled to an angle of view that focuses on the player 401 and the ball 405. Note that although the number of players is four in the illustration, it should be understood that this is for ease of understanding.

In S302, the tracking target input unit 203 obtains the class of the tracking target object via the operation unit 116.

In S303, the object detection unit 202 outputs the bird's-eye view image acquired in S301 to the object detection model to execute object detection processing. FIG. 5 shows an example of the result of object detection processing performed by the object detection unit 202 on the image shown in FIG. The object detection unit 202 recognizes the player and the ball as objects, and outputs information indicating the position of the area occupied by each object and the class (player or ball). The position of the area occupied by the player and the ball is the coordinate that specifies the rectangular frame circumscribing the area. In FIG. 5, the detection results of the object detection unit 202 are indicated by frames 501, 502, 503, 504, and 505.

FIG. 6 is an example of output result information of object detection processing by the object detection unit 202. Frame number 601 holds the frame number on which object detection processing was performed. Class 602 holds the class of the detected object. Image coordinates 603 hold the coordinates of the detected object in the overhead video. The position and size of the detection frame can be determined by the coordinates (Xl1a, Yl1a) of the upper left corner of the frame and the coordinates (Xr1a, Yr1a) of the lower right corner of the frame. The tracking target 604 in this embodiment is of the "Ball" class, and it is required that the image coordinates of this class can always be estimated. In the following explanation, when the expressions ``coordinates of the ball'' or ``coordinates of the player'' are simply used, the meaning of the coordinates shall refer to the coordinates of the center position of the detection frame of the corresponding object.

In S304, the tracking target determination unit 205 determines whether the tracking target 604 (ball) can be detected from the current frame based on the result of the object detection process from the object detection unit 202. If the tracking target determining unit 205 determines that the tracking target 604 (ball) has been detected from the current frame, the process branches to S305, and if it determines that the tracking target 604 (ball) has not been detected, the process branches to S306.

In S305, the tracking target determining unit 205 determines the coordinates of the tracking target 604 (ball) as the target whose coordinate information is to be updated, and notifies the tracking target coordinate updating unit 206 of the determined coordinates. Upon receiving this notification, the tracking target coordinate updating unit 206 updates the coordinate information of the tracking target with the coordinates of the detection frame 505 of the "Ball" class obtained from the current frame.

When the process advances to S306, the tracking target determining unit 205 performs coordinate update processing when the tracking target 604 (ball) has not been detected.

The details of this process will be explained with reference to the flowchart of FIG. 3(b).

In S307, the tracking target determination unit 205 acquires coordinate information and movement vector information of the tracking target 604 (ball) up to the previous frame from the tracking target vector calculation unit 204. Then, in S308, the tracking target determining unit 205 determines whether another class existed within a predetermined range from the coordinates of the tracking target 604 (ball) in the previous frame. If it is determined that the class exists, the process branches to S309, and if it is determined that the class does not exist, the process branches to S310.

In S309, the tracking object determining unit 205 determines the object whose coordinate information is to be updated based on the coordinates of the tracking object 604 (ball) up to the previous frame and the detection results of other classes. This coordinate information determination method will be explained with reference to FIG. The illustration shows a situation in which a tracked object 604 (ball) overlaps a "Player2" class 701, which is not a tracked object, and the "Ball" class cannot be detected. The coordinates 702 of the tracked object 604 (ball) up to the previous frame are retained. Therefore, the tracking target determination unit 205 determines which of the positions of classes that are not tracking targets obtained from the current frame are within a preset distance range from the position of the tracking target 604 (ball) in the previous frame. Determine the closest class as the temporary tracking target. In the case of FIG. 7, the tracking target determining unit 205 estimates that the tracking target 604 (ball) overlaps (is obstructed by) the "Player2" class, and determines the "Player2" class as the temporary tracking target. . Then, the tracking target determining unit 205 notifies the tracking target coordinate updating unit 206 of the coordinates of the detection frame 701 of the current frame of the "Player2" class as the coordinate information of the tracking target, and causes the tracking target coordinate updating unit 206 to update the coordinate information.

On the other hand, when the process proceeds to S310, the tracking target determining unit 205 determines the coordinates based on the coordinates of the tracking target 604 (ball) up to the previous frame and the movement vector information of the tracking target 604 (ball) up to the previous frame. Update information. The case where the tracked object 604 (ball) cannot be detected and cannot be estimated from the information up to the previous frame is typically when the tracked object 604 (ball) has moved out of the recognition target area (court). be. Therefore, the tracking target determining unit 205 starts from the coordinates of the tracking target up to the previous frame, and sets the current tracking target at the coordinate position one frame later (1/30 second in the embodiment) along the movement vector. It is estimated that the target exists, and the estimated coordinates are notified to the tracking target coordinate updating unit 206 to be updated.

The updated coordinates when the processes of S305 and S306 (FIG. 3(b)) are completed are coordinates obtained based on the current frame. It has also been explained that the object to be tracked at this time is usually a ball, but it temporarily becomes a player.

The tracking target determining unit 205 of the embodiment estimates the position of the tracking target in the next frame. Therefore, in S311, the tracking target determining unit 205 sets information for specifying the tracking target in the tracking target vector calculating unit 204, and causes the tracking target vector to calculate the movement vector of the tracking target in the current frame.

For example, when the process transitions from S305 to S311, the tracking target determining unit 205 sets the “Ball” class to the tracking target vector calculating unit 204. In response to this setting, the tracking target vector calculation unit 204 calculates a movement vector from the difference between the coordinates of the "Ball" class in the current frame and the previous frame. For example, let the center coordinates of the "Ball" class and detection frame in the current frame t be (X _t , Y _t ), and let the center coordinates of the "Ball" class detection frame in the previous frame t-1 be (X _{t -1} , Y _t-1 ), the movement vector ΔX is obtained by the following equation (1).
ΔX=(X _t - X _t-1 , Y _t - Y _t-1 )...(1)

Further, when the process transitions from S309 to S311, the tracking target determining unit 205 sets the “Player2” class, which has temporarily become the tracking target, in the tracking target vector calculating unit 204. In response to this setting, the tracking target vector calculation unit 204 calculates a movement vector from the difference between the coordinates of the "Player2" class in the current frame and the previous frame. Although the class is different, the movement vector is the same as in (1) above.

Further, when the process transitions to S311 via S310, the tracking target determining unit 205 sets the “Ball” class to the tracking target vector calculating unit 204. In this case, the "Ball" class does not exist in the current frame. Therefore, the tracking target vector calculation unit 204 calculates the coordinates of the "Ball" class of the previous frame and the two previous frames using equation (1). The calculated movement vector is precisely the movement vector of the "Ball" class one frame before, but since they are temporally close, in this embodiment, this is the movement vector of the "Ball" class in the current frame. Adopted as a movement vector.

In S312, the tracking target determination unit 205 outputs the movement vector ΔX calculated by the tracking target vector calculation unit 204 as described above to the pan/tilt drive control unit 207. The pan/tilt drive control unit 207 adds the movement vector ΔX input from the tracking target determining unit 205 to the updated coordinates by the tracking target coordinate updating unit 206, thereby determining the coordinates of the tracking target after movement in the next frame. presume. Then, the pan/tilt drive control unit 207 issues pan angle and tilt angle commands so that the line of sight of the camera 102 is directed toward the estimated coordinates after movement (the center of the angle of view). It is generated and transmitted to the camera 102.

As described above, according to the present embodiment, even if the tracked object is blocked by another object, the position of the tracked object can be estimated based on the coordinate information and movement vector up to the previous frame of the tracked object. It is possible to capture natural moving images targeting the tracked object.

[Second embodiment]
A second embodiment will be described. Note that in the second embodiment, descriptions of the same components as those in the first embodiment will be omitted. In the first embodiment, when the tracking target has been detected, the image coordinates of the tracking target can be set to the center of the angle of view. However, in sports scenes, it may be desirable to always set the player involved in the ball as the tracking target. In this embodiment, the tracking target is determined based on the positional relationship with other classes other than the detected tracking target and the movement vector information of the tracking target up to the previous frame. By doing this, you can always set the player involved in the ball as the tracking target.

The operation of the video processing device 100 in the second embodiment will be described below along the flowchart illustrated in FIG. 8(a).

The processes of S801 to S804, S806, S811, and S812 in FIG. 8(a) are the same as the processes of S301 to S304, S306, S311, and S312 of FIG. 3 of the first embodiment, so the description thereof will be omitted.

In S805, the tracking target determination unit 205 performs coordinate update processing when the tracking target 604 (ball) has been detected.

Details of the process of S805 will be explained with reference to FIG. 8(b).

In S807, if the tracking target 604 (ball) has been detected, the tracking target determination unit 205 determines whether another class exists within a predetermined range from the tracking target 604 (ball). Then, if the tracking target determination unit 205 determines that another class exists within a predetermined range from the tracking target 604, the process proceeds to S808, and if it determines that there is no other class, the process proceeds to S809.

In S808, the tracking target determining unit 205 determines the target whose coordinates are to be updated based on the image coordinates of the tracking target 604 (ball). This process of determining the coordinate update target will be explained with reference to FIG. 9. In FIG. 9, if the position of the detection frame 901 of the "Ball" class of the tracking target 604 (ball) and the detection frame 902 of the "Player1" class, which is not the tracking target, are within a predetermined range, the "Player1" class is tracked. It can be determined that the object 604 (ball) is held. Therefore, the tracking target determination unit 205 determines the "Player1" class as the tracking target. Then, the tracking target coordinate updating unit 206 updates the coordinate information of the tracking target with the image coordinates of the detection frame 902 of the "Player1" class.

On the other hand, when the process proceeds to S809, the tracking target determining unit 205 acquires movement vector information of the tracking target 604 (ball) up to the previous frame from the tracking target vector calculating unit 204. Then, in S810, the tracking target determining unit 205 selects a target whose coordinate information is to be updated based on the current coordinates of the tracking target 604 (ball) and the movement vector information of the tracking target 604 (ball) up to the previous frame. decide. This process will be explained with reference to FIG. FIG. 10 shows an example in which there is no detection frame of a class other than the tracking target within a predetermined range of the detection frame 1001 of the “Ball” class of the tracking target 604 (ball). This is exactly the case when the tracked object 604 (ball) is moving temporarily, such as when passing or shooting. In this case, it is estimated that the player will have possession of the ball after several frames. The tracking target determining unit 205 selects "Player 2" which exists at the closest position on the extension line of the movement vector 1002 up to the previous frame, starting from the current coordinates of the detection frame 1001 of the "Ball" class of the tracking target 604 (ball). Determine the class to be tracked. Then, the tracking target coordinate updating unit 206 updates the coordinate information of the tracking target with the image coordinates of the detection frame 1003 of the "Player2" class.

As described above, according to the present embodiment, even if it is desirable to set a player involved in the ball as a tracking target, the positional relationship with other classes other than the detected tracking target and the movement of the tracking target to the previous frame can be determined. Tracking targets can be determined based on vector information.

[Third embodiment]
Next, a third embodiment of the present invention will be described. Note that descriptions of the same components as those in the first and second embodiments will be omitted. In the first and second embodiments, when it is determined that the tracking target has been detected through the object detection process, the tracking target presence processing is performed to update the coordinates of the tracking target. However, in object detection processing, not limited to convolutional neural networks, there is a possibility that false detections may occur, and tracking accuracy may decrease depending on the detection model and the environment in which the image is taken. In the third embodiment, it is determined whether the detection result of the tracking target is reliable, and if it is determined to be reliable, the coordinate update process is performed when the tracking target is detected. By doing so, it is possible to suppress a decrease in tracking accuracy due to erroneous detection in object detection processing.

The operation of the video processing device 100 in this embodiment will be described below along the flowchart illustrated in FIG. 11. Note that the processes in S1101 to S1106, S1111, and S1112 are the same as S801 to S806, S811, and S812 in FIG. 3 of the second embodiment, so their description will be omitted.

In S1113, if the tracking target 604 (ball) has been detected, the tracking target determination unit 205 determines whether the tracking target 604 (ball) has been detected for at least two consecutive frames. If the tracking target determining unit 205 determines that a tracking target has been detected for two or more consecutive frames, the process proceeds to S1105, and if it determines that a tracking target has not been detected, the process proceeds to S1106. In other words, if the "Ball" class has been detected at t-1, one frame before the current one, and the "Ball" class has been detected at the current frame t, the tracking target determining unit 205 selects the frame t-1. 1, the result of the object detection unit 202 that the tracking target 604 (ball) has been detected is determined to be reliable. On the other hand, if the "Ball" class is detected in the current frame t, but the tracking target 604 (ball) is not detected in the previous frame t-1, the result of the object detection unit 202 for the current frame has a low reliability. is determined to be low.

As described above, according to the present embodiment, it is possible to suppress the influence of uncertain erroneous detection in object detection processing. Note that in the third embodiment, it is determined whether the tracking target has been detected for two or more consecutive frames, but the number of consecutive frames may be changed as appropriate depending on the frame rate of the camera. .

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible.

For example, in S310 of the first embodiment, the image coordinates may be updated using the center or center of gravity of the detection result of a class other than the tracking target 604 (ball).

Further, in S311 of the first embodiment, a movement vector may be calculated using changes in coordinate information from two or more frames ago.

Further, in S810 of the second embodiment, the centers of multiple classes existing on the extension line of the vector 1002 from the coordinates of the detection frame 1001 of the tracking target 604 (ball) to the previous frame and the position of the center of gravity are determined as the tracking target. You may.

Further, in S1113 of the third embodiment, it may be determined whether the detection result of the tracking target is reliable using the reliability score of the output result of the object detection process.

(Other examples)
The present invention provides a system or device with a program that implements one or more of the functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The disclosure herein includes the following object tracking device, method, and program.
(Item 1)
An image processing device that controls the viewing direction of a second imaging device whose viewing direction can be changed based on an image from a first imaging device that captures a bird's-eye view of the entire court where sports are played, the video processing device comprising:
detection means for detecting an object in the video received from the first imaging device;
Estimating the position of the tracked object after movement from the movement vector of the tracked object among the objects detected by the detection means, and controlling the second imaging device so that the estimated position is within the angle of view. and a control means for
The control means includes:
If it is determined that the tracking target object is not included in the objects detected from the current frame of the video received from the first imaging device, detecting an object blocking the tracking target object from the previous frame. determination means for determining based on the positional relationship of the objects;
and setting means for setting the object determined by the determination means as a tracking target instead of the tracking target object.
(Item 2)
The determining means is configured to select an object, which is within a preset distance range and has the shortest distance from the position of the tracked object detected from the previous frame, among the objects detected from the current frame, by blocking the tracked object. The image processing device according to item 1, characterized in that the image processing device determines that the object is present.
(Item 3)
If there is no object within a preset distance range from the position of the tracked object detected from the previous frame among the objects detected from the current frame, the determination means detects the position of the tracked object in the previous frame. The video processing device according to item 1 or 2, characterized in that the tracking target object is determined to be located at a position of the current frame estimated from a movement vector.
(Item 4)
The control means determines whether the object to be tracked is present in the current frame of the video received from the first imaging device and whether another object is present within a preset distance from the object to be tracked. further comprising a second determining means for determining whether or not;
The setting means determines that the determination result of the second determination means is that the tracking target object exists in the current frame of the video received from the first imaging device, and that the tracking target object is set in advance from the tracking target object. When indicating that there is no other object within the distance range, another object located on the extension of the movement vector of the tracked object obtained from the current frame and the previous frame is substituted for the tracked object. The video processing device according to any one of items 1 to 3, wherein the video processing device is set as a tracking target.
(Item 5)
The control means causes the determining means to determine that the tracked object has been detected from the current frame when the tracked object is detected consecutively in a preset number of frames up to the current frame. The video processing device according to any one of items 1 to 4, characterized in that the video processing device performs the following.
(Item 6)
6. The video processing device according to any one of items 1 to 5, wherein the detection means detects a player and a ball as objects, and detects the ball as an object to be tracked.
(Item 7)
1. A control method for an image processing device that controls a viewing direction of a second imaging device whose viewing direction can be changed based on an image from a first imaging device that captures an overhead image of the entire court on which a sport is played, the method comprising: ,
a detection step of detecting an object in the video received from the first imaging device;
Estimating the position of the tracked object after movement from the movement vector of the tracked object among the objects detected in the detection step, and controlling the second imaging device so that the estimated position is within the angle of view. and a control process to
The control step includes:
If it is determined that the tracking target object is not included in the objects detected from the current frame of the video received from the first imaging device, detecting an object blocking the tracking target object from the previous frame. a determination step of determining based on the positional relationship of the objects;
A method for controlling an image processing device, comprising: a setting step of setting the object determined in the determination step as a tracking target instead of the tracking target object.
(Item 8)
A program that, when read and executed by a computer, causes the computer to function as each means included in the device according to any one of items 1 to 6.

The invention is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the sections are attached to disclose the scope of the invention.

DESCRIPTION OF SYMBOLS 100... Video processing device, 101, 102... Camera, 201, 208... Input unit, 202... Object detection unit, 203... Tracking target input unit, 204... Tracking target vector calculation unit, 205... Tracking target determining unit, 206... Tracking Target coordinate update unit, 207...Pan/tilt drive control unit

Claims (8)

An image processing device that controls the viewing direction of a second imaging device whose viewing direction can be changed based on an image from a first imaging device that captures a bird's-eye view of the entire court where sports are played, the video processing device comprising:
detection means for detecting an object in the video received from the first imaging device;
Estimating the position of the tracked object after movement from the movement vector of the tracked object among the objects detected by the detection means, and controlling the second imaging device so that the estimated position is within the angle of view. and a control means for
The control means includes:
If it is determined that the tracking target object is not included in the objects detected from the current frame of the video received from the first imaging device, detecting an object blocking the tracking target object from the previous frame. determination means for determining based on the positional relationship of the objects;
and setting means for setting the object determined by the determination means as a tracking target instead of the tracking target object. The determining means is configured to select an object, which is within a preset distance range and has the shortest distance from the position of the tracked object detected from the previous frame, among the objects detected from the current frame, by blocking the tracked object. The image processing device according to claim 1, wherein the image processing device determines that the object is present. If there is no object within a preset distance range from the position of the tracked object detected from the previous frame among the objects detected from the current frame, the determination means detects the position of the tracked object in the previous frame. The video processing device according to claim 1, wherein the tracking target object is determined to be located at a position of the current frame estimated from a movement vector. The control means determines whether the object to be tracked is present in the current frame of the video received from the first imaging device and whether another object is present within a preset distance from the object to be tracked. further comprising a second determining means for determining whether or not;
The setting means determines that the determination result of the second determination means is that the tracking target object exists in the current frame of the video received from the first imaging device, and that the tracking target object is set in advance from the tracking target object. When indicating that there is no other object within the distance range, another object located on the extension of the movement vector of the tracked object obtained from the current frame and the previous frame is substituted for the tracked object. The video processing device according to claim 1, wherein the video processing device is set as a tracking target. The control means causes the determining means to determine that the tracked object has been detected from the current frame when the tracked object is detected consecutively in a preset number of frames up to the current frame. The video processing device according to claim 1, wherein the video processing device performs the following. The video processing device according to claim 1, wherein the detection means detects a player and a ball as objects, and detects the ball as an object to be tracked. 1. A control method for an image processing device that controls a viewing direction of a second imaging device whose viewing direction can be changed based on an image from a first imaging device that captures an overhead image of the entire court on which a sport is played, the method comprising: ,
a detection step of detecting an object in the video received from the first imaging device;
Estimating the position of the tracked object after movement from the movement vector of the tracked object among the objects detected in the detection step, and controlling the second imaging device so that the estimated position is within the angle of view. and a control process to
The control step includes:
If it is determined that the tracking target object is not included in the objects detected from the current frame of the video received from the first imaging device, detecting an object blocking the tracking target object from the previous frame. a determination step of determining based on the positional relationship of the objects;
A method for controlling an image processing device, comprising: a setting step of setting the object determined in the determination step as a tracking target instead of the tracking target object. A program for causing the computer to execute each step of the method according to claim 7 by being read and executed by a computer.

Images