JP6853528B2 - Video processing programs, video processing methods, and video processing equipment - Google Patents

Description

The embodiment relates to a video processing program, a video processing method, and a video processing apparatus.

A video processing device, a video processing method, and a video processing program for tracking a player or a ball by analyzing a sports game video are known. Such a video processing device predicts the position of the ball in the current frame based on the position information of a plurality of frames in the past, and searches the peripheral region thereof. However, with such a method, it is difficult to deal with the case where the ball is not found in the area.

Japanese Unexamined Patent Publication No. 2004-46647 International Publication No. 2015/190071

Provided are a video processing device, a video processing method, and a video processing program capable of detecting and tracking a ball or a player by image recognition from a sports game video and mapping the movement locus on a two-dimensional field.

The image processing program of the embodiment causes a computer to detect a first object and a plurality of second objects different from the first object from the image in order to process an image obtained by shooting a sport, and the detected first object, And the position of a plurality of second objects is estimated for each frame in the video, and when the first object cannot be detected in a certain frame, it is located in the vicinity of the first object in the frame before the frame. It is considered that the second object holds the first object. The plurality of second objects include third and fourth objects. If the third object and the fourth object are close to each other while the third object is being tracked by assuming that it holds the first object, it is considered that the fourth object also holds the first object. It is tracked, and after the lapse of the first time in the video, the moving speed of the third object and the moving speed of the fourth object are compared, and the tracking of the third and fourth objects having the faster moving speed is continued, and the moving speed is increased. The slower tracking is stopped.

The block diagram of the image processing apparatus which concerns on 1st Embodiment. The figure which shows the program provided in the image processing apparatus which concerns on 1st Embodiment. The functional block diagram of the processor provided in the video processing apparatus which concerns on 1st Embodiment. The figure which shows an example of the field in the sports of the analysis target of the image processing apparatus which concerns on 1st Embodiment. The flowchart of the main program which concerns on 1st Embodiment. The flowchart of the analysis program which concerns on 1st Embodiment. The flowchart of the field detection program which concerns on 1st Embodiment. The figure which shows the detection result of the field in the field detection program which concerns on 1st Embodiment. The figure which shows the correspondence between the display area of the image shown in FIG. 7 and a two-dimensional field. The flowchart of the object detection program which concerns on 1st Embodiment. The figure which shows the detection result of the object in the object detection program which concerns on 1st Embodiment. FIG. 10 is a diagram showing a display area of an image shown in FIG. 10 and a correspondence between an object and a two-dimensional field. The flowchart of the object position estimation program which concerns on 1st Embodiment. The figure explaining the setting method of the ball holder in the object position estimation program which concerns on 1st Embodiment. The figure explaining the setting method of the ball holder in the object position estimation program which concerns on 1st Embodiment. The flowchart explaining the setting method of the ball holder in the object position estimation program which concerns on 2nd Embodiment. The figure explaining the setting method of the ball holder in the object position estimation program which concerns on 2nd Embodiment. The figure explaining the setting method of the ball holder in the object position estimation program which concerns on 2nd Embodiment. The figure explaining the setting method of the ball holder in the object position estimation program which concerns on 2nd Embodiment. The figure explaining the setting method of the ball holder in the object position estimation program which concerns on 2nd Embodiment. The flowchart of the play type discrimination program which concerns on 3rd Embodiment. The figure explaining the play type discrimination method in the play type discrimination program which concerns on 3rd Embodiment. The figure explaining the play type discrimination method in the play type discrimination program which concerns on 3rd Embodiment. The figure explaining the play type discrimination method in the play type discrimination program which concerns on 3rd Embodiment. The figure which shows the discrimination result of the play type in the play type discrimination program which concerns on 3rd Embodiment. The flowchart of the scene division program which concerns on 4th Embodiment. The figure explaining the scene division method in the scene division program which concerns on 4th Embodiment. The figure which shows the division result of the scene in the scene division program which concerns on 4th Embodiment. The figure which shows the execution result of the reproduction program which concerns on 5th Embodiment. The figure explaining the optional function of video display in the reproduction program which concerns on 5th Embodiment. The figure explaining the optional function of the field display in the reproduction program which concerns on 5th Embodiment. The figure explaining the image processing apparatus which concerns on a modification.

Hereinafter, embodiments will be described with reference to the drawings. The referenced drawings are schematic. In the following description, elements having the same function and configuration are designated by a common reference numeral. The alphabet after the numbers that make up the reference code is used to distinguish between elements that are referenced by a reference code that contains the same number and have a similar structure. If it is not necessary to distinguish between the elements represented by the reference code containing the same number, these elements are referred to by the reference code containing only the number.

[1] First Embodiment The video processing apparatus according to the first embodiment will be described below.

[1-1] Configuration [1-1-1] Device Configuration First, the overall configuration of the video processing device will be described with reference to FIG. FIG. 1 shows a block diagram of the video processing apparatus. As shown in FIG. 1, the video processing device 1 includes a processor (CPU) 10, a random access memory (RAM) 11, a read-only memory (ROM) 12, a storage unit 13, and a display 14.

The CPU 10 controls the operation of the entire video processing device 1. For example, the CPU 10 executes a program used for video processing described later in response to an operation by a user or an instruction from an external host device (not shown). The CPU 10 also manages the memory space of the storage unit 13.

The RAM 11 is a memory used as a work area of the CPU 10. As the RAM 11, a semiconductor memory such as a DRAM is used.

The ROM 12 is a non-volatile memory in which control programs, control data, and the like are stored in advance. The ROM 12 holds, for example, a BIOS (Basic Input / Output System).

The storage unit 13 stores a video file used for analysis by the user and data generated by video processing. Further, the storage unit 13 holds various programs used for video processing and data related to the programs. The program held in the storage unit 13 is read out when the video processing device 1 executes the program, and is expanded in the RAM 11. As the storage unit 13, for example, an SSD (Solid State Drive) or a hard disk drive is used. The storage unit 13 may store data as long as it can store data, and other storage media may be used as the storage unit 13.

The display 14 displays a GUI (Graphical User Interface) or the like corresponding to various programs by the CPU 10. For example, the display 14 is used to reproduce the result of video processing.

The configuration of the video processing device 1 described above is merely an example, and other configurations may be used. For example, the video processing device 1 does not have to include the storage unit 13 and the display 14. In this case, the storage unit 13 and the display 14 are externally connected to the video processing device 1, respectively.

Further, in the above description, the case where the program used for the video processing is stored in the storage unit 13 has been described as an example, but the present invention is not limited to this. For example, the program executed by the video processing device 1 may be stored in a server on a network (not shown). In this case, when the video processing device 1 executes the video processing, various programs are distributed from the server on the network to the video processing device 1. Then, the video processing device 1 that has received the various programs expands these programs in the RAM 11 and executes the video processing.

[1-1-2] Program Configuration Next, the configuration of the program used for video processing will be described with reference to FIG. FIG. 2 shows various programs developed in the RAM 11 during video processing. As shown in FIG. 2, when the video processing is executed, the main program 20, the analysis program 21, the reproduction program 27, and the display data 28 are expanded in the RAM 11.

The main program 20 is software for selectively executing the analysis processing of the video file and the reproduction processing of the analysis data obtained by the analysis processing in the video processing device 1. In the following description, the video file is, for example, a game video used for broadcasting sports, etc., and includes video data and audio data. The format of the video file corresponding to this video processing is not particularly limited.

The analysis program 21 is a program that analyzes a video file by executing various programs. The analysis program 21 includes a field detection program 22, an object detection program 23, an object position estimation program 24, a play type determination program 25, and a scene division program 26. The configuration of the analysis program 21 is not limited to this. For example, the analysis program 21 may be composed of one program, and may not be divided for each program.

By executing the analysis program 21, the CPU 10 has, for example, a field detection unit 30, a first object detection unit 31A, a second object detection unit 31B, a team type determination unit 32, and an object position estimation unit 33, as shown in FIG. It functions as a movement direction estimation unit 34, a movement speed estimation unit 35, a play type determination unit 36, a specific sound detection unit 37, and a scene division unit 38. In this example, the name is changed for each process of the CPU 10, but the CPU 10 can execute a plurality of processes at the same time.

The field detection unit 30 is realized by the CPU 10 executing the field detection program 22, and detects the field area displayed in the moving image. The object detection unit 31 and the team type determination unit 32 are realized by the CPU 10 executing the object detection program 23. The first object detection unit 31A detects a person displayed in the moving image. The second object detection unit 31B detects the ball displayed in the moving image. The team type determination unit 32 determines the type of the person displayed in the moving image. The object position estimation unit 33 is realized by executing the object position estimation program 24, and estimates the movement loci of the player and the ball from the detection results of the field, the person, and the ball. The movement direction estimation unit 34, the movement speed estimation unit 35, and the play type determination unit 36 are realized by executing the play type determination program 25. The moving direction estimation unit 34 estimates the moving direction of the person and the ball from the detection results of the field, the person and the ball. The movement speed estimation unit 35 estimates the movement speed of the person and the ball from the detection results of the field, the person, and the ball. The play type determination unit 36 determines the type of play at each time point of the video file from the detection results of the field, the person, and the ball. The specific sound detection unit 37 and the scene division unit 38 are realized by the CPU 10 executing the scene division program 26. The specific sound detection unit 37 detects a specific sound such as a whistle sound of a referee from the voice data. The scene division unit 38 divides the scene of the video file based on the timing when the specific sound detection unit 37 detects the specific sound.

The playback program 27 is a program that executes playback processing of the analyzed video file. For the reproduction process, for example, the analyzed video file and the associated analysis data are used. The case where the video file and the analysis data are used for the reproduction process is given as an example, but the present invention is not limited to this. For example, by including the moving image data and the audio data of the video file in the analysis data itself, it is possible to execute the reproduction process only with the analysis data.

The display data 28 includes image files associated with various programs. For example, the CPU 10 displays the corresponding image file in the display data 28 on the display 14 when various programs are executed.

<Example of field>
Examples of the sport analyzed by using the video processing device 1 described above include rugby. In the following description, a case where a rugby game video is analyzed by the video processing device 1 will be used as an example. The field configuration of the rugby used in this example is shown in FIG.

As shown in FIG. 4, the field FLD of rugby includes goal lines 40A and 40B, halfway lines 41, 10-meter lines 42A and 42B, 22-meter lines 43A and 43B, touch lines 44A and 44B, and 5-meter lines 45A and 45B. , And 15 meter lines 46A and 46B are provided. Goal posts 47A and 47B are installed on the goal lines 40A and 40B, respectively.

The size of the entire field FLD is set within 70 meters between the touch lines 44A and 44B and within 100 meters between the goal lines 40A and 40B. Of the lines in the field FLD, the distance between the halfway line 41 and the 10-meter line 42 is constant regardless of the match venue, and the distance between the halfway line 41 and the 22-meter line 43 does not depend on the match venue. It is constant. Similarly, the distance between the touch line 44 and the 5-meter line 45 is constant regardless of the match venue, and the distance between the touch line 44 and the 15-meter line 46 is constant regardless of the match venue.

[1-2] Operation [1-2-1] Main program 20
Next, the operation of the video processing device 1 based on the main program 20 will be described with reference to FIG. FIG. 5 shows an operation flowchart of the video processing device 1 by the main program 20. As shown in FIG. 5, the CPU 10 first requests the user to select an operation (step S10). At this time, the user can select, for example, either "analysis processing" or "regeneration processing".

When the video processing device 1 receives the analysis processing command from the user, the CPU 10 requests the user to select a video file to execute the analysis processing. Then, when the video processing device 1 accepts the selection of the video file by the user (step S11), the CPU 10 executes the analysis program 21 to analyze the video file (step S12). The details of the analysis method of this video file will be described later. When the analysis of the video file is completed, the CPU 10 associates the obtained analysis data with the video file and saves it in the storage unit 13 (step S13), and ends the process by the main program 20.

On the other hand, when the video processing device 1 receives a playback processing command from the user, the CPU 10 requests the user to select analysis data for executing the playback processing. Then, when the video processing device 1 accepts the user's selection of analysis data (step S14), the CPU 10 executes the playback program 27 and displays the analysis result of the video file (step S15). A specific example of the method of displaying the analysis result will be described later. When the display of the analysis data is completed, the CPU 10 ends the process by the main program 20.

The operation by the main program 20 described above is an example, and is not limited to this. For example, when the user selects a video file to be analyzed, the CPU 10 may execute the analysis program 21 and subsequently execute the playback program 27. In this way, the video processing device 1 can continuously display the analysis result of the video file simply by selecting the video file to be analyzed.

[1-2-2] Analysis program 21
Next, the operation of the video processing apparatus 1 based on the analysis program 21 will be described. When the analysis program 21 is executed, the CPU 10 executes the field detection program 22, the object detection program 23, and the object position estimation program 24, and functions as the field detection unit 30, the object detection unit 31, and the object position estimation unit 33. To do. The overall operation flowchart of the video processing apparatus 1 in this analysis processing is shown in FIG. The details of each step will be described later.

As shown in FIG. 6, the field detection unit 30 first detects the field area from the match video (step S20). Then, the field detection unit 30 calculates the coordinate information seen from the entire field corresponding to the detected field area and transmits it to the object detection unit 31 and the object position estimation unit 33 (not shown). Further, the object detection unit 31 detects the player and the ball from the game video (step S21). At this time, the object detection unit 31 determines the coordinates of the player and the ball on the field from the coordinate information from the field detection unit 30 and the positions of the player and the ball on the image. Then, the object position estimation unit 33 receives the coordinate information of the player and the ball determined by the object detection unit 31 and tracks the player and the ball, respectively (step S22). The operations in steps S20 to S22 are continuously executed in this analysis process.

As the game progresses, the object detection unit 31 may lose sight of the ball due to the influence of the ball being hidden behind the player (step S23). At this time, the object position estimation unit 33 detects that the object detection unit 31 has lost sight of the ball. Then, the object position estimation unit 33 tracks, for example, a player who is closest to the position where the object detection unit 31 has lost sight of the ball as a ball holder (step S24).

After that, when the object detection unit 31 rediscovers the ball in the vicinity of the ball holder (step S25), the object position estimation unit 33 detects that the ball has been rediscovered. Then, the object position estimation unit 33 estimates the position of the ball from the movement locus of the ball holder (step S26). Specifically, the movement trajectory of the ball during the period when the ball is lost is regarded as the same as, for example, the ball holder.

In this way, the image processing device 1 can continuously grasp the position of the ball during the match even if there is a period during which the ball is not detected in the match video. Hereinafter, an example of detailed operations of the image processing apparatus 1 by the field detection unit 30, the object detection unit 31, the team type determination unit 32, and the object position estimation unit 33 for realizing this analysis processing will be described in order.

<Field detection unit 30>
First, the detailed operation of the video processing apparatus 1 based on the field detection program 22 will be described with reference to FIGS. 7 and 8. FIG. 7 shows an operation flowchart of the image processing device 1 by the field detection program 22, and FIG. 8 shows an example of the field image to be analyzed by the field detection program 22.

As shown in FIG. 7, the field detection unit 30 first selects a frame for executing the field detection process from the moving image data (step S30). The frame selected at this time corresponds to, for example, the first frame of the selected moving image data.

Next, the field detection unit 30 extracts a field area from the video VTR of the frame (step S31). The field area is extracted from the video VTR, for example, based on the color of the field. The color information of this field may be preset by the user, or a color occupying a wide range in the video VTR by the field detection unit 30 may be regarded as the color of the field. Further, the field detection unit 30 may consider a region having a large green area as a field.

Next, the field detection unit 30 detects a line from the video VTR (step S32). Specifically, for example, the inside of the field area extracted in step S31 is masked with the color of the field. As a result, the foreground in the field area is extracted, and the field detection unit 30 can detect the line in the field area from the foreground. In the example shown in FIG. 8, the field detection unit 30 detects the lines L1 and L2 extending in the horizontal direction and the lines L3 and L4 extending in the vertical direction. In the following description, the regions where the line L1 is in contact with the outer peripheral portion of the video VTR are referred to as end portions E1 and E2, and the regions where the line L2 is in contact with the outer peripheral portion of the video VTR are referred to as end portions E3 and E4.

Next, the field detection unit 30 executes the coordinate calculation process of the area displayed on the video VTR (step S33). The coordinate calculation process is a process of associating which coordinates of the entire field the field area displayed on the video VTR corresponds to. For example, when lines L1 to L4 are detected as shown in FIG. 8, the field detection unit 30 first determines which line in the field the lines L1 to L4 correspond to.

Specifically, the field detection unit 30 determines the type of the detected line by checking, for example, the interval between the detected lines and the line type of the line. In the case of the example shown in FIG. 8, the field detection unit 30 determines that the lines L1 and L2 are the touch lines 44A and 44B, respectively, because the lines L1 and L2 extending in the lateral direction are solid lines, for example. Further, in the field detection unit 30, the line L3 is a 22-meter line 43A and the line L4 is 10 because the line L3 extending in the vertical direction is a solid line and the line L4 is a broken line and the distance between the line L3 and the line L4. It is determined that the line is the metric line 42A.

Then, the field detection unit 30 calculates which coordinates of the entire field the area displayed on the video VTR corresponds to from the information of the discriminated lines L1 to L4 and the information of the display area of the video VTR. When the coordinate calculation process is executed on the video VTR shown in FIG. 8, the information shown in FIG. 9 can be obtained. As shown in FIG. 9, in the two-dimensional field FLD, a region connecting the ends E1 to E4 of the video VTR in order is shown, and the outside of the video region is masked with diagonal lines. Here, the two-dimensional field FLD indicates an image of a field in which the entire field is displayed. Then, the field detection unit 30 stores the coordinate data of the video VTR associated in this way in the storage unit 13.

Next, the field detection unit 30 confirms whether or not the frame that has been processed so far is the last frame (step S34). If the frame in which the process is executed does not correspond to the last frame (step S34, NO), the field detection unit 30 returns to step 20, selects a subsequent frame, and executes the same process. On the other hand, when the frame in which the processing is executed corresponds to the last frame (step S34, YES), the field detection unit 30 ends the execution of the field detection program 22.

The line determination method in the above description is merely an example, and is not limited thereto. For example, the touch line 44 may be discriminated by recognizing a flag provided in the field FLD, or the goal lines 40A and 40B may be discriminated by recognizing the goal posts 47A and 47B. Is also good.

Further, in the above description, the case where the field detection process is executed for each frame has been described as an example, but the present invention is not limited to this. For example, the field detection unit 30 may discriminate between the field and the line by checking the difference in the image from the subsequent frame in the processing after the field is detected.

<Object detection unit 31 and team type determination unit 32>
Next, the detailed operation of the image processing apparatus 1 based on the object detection program 23 will be described with reference to FIGS. 10 and 11. FIG. 10 shows an operation flowchart of the image processing device 1 by the object detection program 23, and FIG. 11 shows an example of a game image to be analyzed by the object detection program 23.

As shown in FIG. 10, the object detection unit 31 first selects a frame for executing the object detection process from the moving image data (step S40). The frame selected at this time corresponds to, for example, the first frame of the selected video VTR.

Next, the object detection unit 31 extracts an object on the field from the video VTR of the frame (step S41). The image processing in step S41 is the same as the image processing in step S32 described with reference to FIG. 7, and the object is detected by extracting the foreground in the field area.

Next, the object detection unit 31 determines whether the object extracted as the foreground in the field area is a person or a ball (step S42). Specifically, as shown in FIG. 11, the first object detection unit 31A determines the players P1 to P8, and the second object detection unit 31B determines the ball BL. Further, when the first object detection unit 31A confirms a situation in which a plurality of athletes are in contact with each other, the first object detection unit 31A recognizes the mass of the plurality of athletes as dense HD. In the case of rugby, this dense HD is compatible with scrums, racks, malls, and the like.

Then, the team type determination unit 32 determines the team type from the extracted player color information (step S43). Specifically, the team type determination unit 32 groups persons having color information having similar characteristics, for example, in a rectangular area in which players P1 to P8 are detected. In the example shown in FIG. 11, since players P1 to P4 wear black uniforms and players P5 to P8 wear white uniforms, the team type determination unit 32 classifies players P1 to P4 into team A. , Players P5 to P8 are classified into Team B.

Next, the object detection unit 31 calculates the coordinates of the object detected on the field (step S34). Specifically, based on the coordinate information of the field area obtained by the field detection program 22, the first object detection unit 31A calculates the coordinates of the athletes P1 to P8 and the dense HD on the two-dimensional field FLD, and the second The object detection unit 31B calculates the coordinates of the ball BL on the two-dimensional field FLD. When the coordinate calculation process is executed on the video VTR shown in FIG. 11, the information shown in FIG. 12 can be obtained. As shown in FIG. 12, in the two-dimensional field FLD, the players P1 to P4 are indicated by black circles, the athletes P5 to P8 are indicated by white circles, the dense HD is indicated by a broken line circle, and the ball BL is indicated by an ellipse. Has been done. Then, the object detection unit 31 stores the information of the players P1 to P8, the dense HD, and the ball BL associated in this way in the storage unit 13.

Next, the object detection unit 31 confirms whether or not the frame that has been processed so far is the last frame (step S45). If the frame in which the process is executed does not correspond to the last frame (step S45, NO), the process returns to step 40, and the object detection unit 31 selects the following frame and executes the same process. On the other hand, when the frame in which the processing is executed corresponds to the last frame (step S45, YES), the object detection unit 31 ends the execution of the object detection program 23.

In step S41 described above, the second object detection unit 31B searches for and detects the ball based on, for example, the size, shape, and color information of the ball registered in advance. The information of this ball may be input by the user, or the user may select some information such as a manufacturer and a model number so that the preset data can be referred to.

Further, in the above description, the case where the first object detection unit 31A recognizes a mass of a plurality of athletes as dense HD has been described as an example, but the number of people regarded as dense by the first object detection unit 31A is limited to two or more. It can be set to any number of people.

Further, in steps S31 and S32 described above, as the person detected by the first object detection unit 31A, a referee, a linesmen, a reserve player, or the like may be detected in addition to the player. In this case, the first object detection unit 31A can determine the referee and the linesmen from, for example, that the referee and the linesmen have different color information from the players of the A team and the B team, and the number of objects belonging to the group. You can. Further, the first object detection unit 31A can determine the reserve player by, for example, considering a person outside the touch line 44 in the field area as a player who is not playing at that time.

Further, in the above description, the case where the object detection process is executed for each frame has been described as an example, but the present invention is not limited to this. For example, the object detection unit 31 may discriminate between the player and the ball by checking the difference in the image from the subsequent frame in the processing after detecting the object.

<Object position estimation unit 33>
Next, the detailed operation of the image processing apparatus 1 based on the object position estimation program 24 will be described with reference to FIG. FIG. 13 shows an operation flowchart of the video processing device 1 by the object position estimation program 24.

As shown in FIG. 13, the object position estimation unit 33 first selects a frame for executing the object position estimation process from the moving image data (step S50). Next, the object position estimation unit 33 tracks by referring to the coordinate information of the player and the ball calculated by the object detection program 23 (step S51).

Next, the object position estimation unit 33 confirms whether or not a ball has been detected in the video VTR of the frame (step S52).

When the ball is not detected in step S52 (step S52, NO), the object position estimation unit 33 confirms whether or not the ball holder is set (step S53). When the ball holder is not set (step S53, NO), the object position estimation unit 33 considers the player closest to the place where the ball is lost as the ball holder and gives the ball holder information (step S54). .. Then, when the ball holder is not set in step S53 (step S53, YES), the object position estimation unit 33 shifts to step S56 after the processing of step S54.

When the ball is detected in step S52 (step S52, YES), the object position estimation unit 33 cancels the ball holder setting if it is set, and tracks the player as the ball holder. Is stopped (step S55). That is, the ball holder information given to the player is released when the ball is rediscovered. Then, the object position estimation unit 33 shifts to step S56 after the processing of step S55. In step S55, the object position estimation unit 33 confirms whether or not the frame that has been processed so far is the last frame (step S55). ).

If the frame in which the process is executed does not correspond to the last frame (step S56, NO), the process returns to step S50, and the object detection unit 31 selects the following frame and executes the same process.

When the frame in which the process is executed corresponds to the last frame (step S56, YES), the object position estimation unit 33 estimates the position of the ball in the moving image data from the information of the ball holder (step S57). Then, the object position estimation unit 33 stores the estimated information on the movement locus of the player and the ball in the storage unit 13, and ends the execution of the object position estimation program 24.

Specific examples of the situation in which the ball holder is set by the object position estimation program described above will be described with reference to FIGS. 14 and 15.

FIG. 14 is an example in the case where there is no dense HD in the vicinity of the ball BL before being lost. (1) of FIG. 14 shows players P1 and P2, and shows a situation in which player P1 holds the ball BL. From this situation, it is assumed that the ball BL held by the player P1 disappears from the image. Then, as shown in (2) of FIG. 14, the object position estimation unit 33 adds the ball holder information to the player P1 who was in the immediate vicinity of the lost ball BL, and tracks the player P1 as the ball holder.

FIG. 15 is an example of a case where there is a dense HD in the vicinity of the ball BL before it is lost. In FIG. 15 (1), players P1 to P3 of team A, players P4 to P6 of team B, and dense HD are shown, and a situation in which player P1 holds the ball BL is shown. From this situation, it is assumed that the ball BL held by the player P1 disappears from the image. Then, as shown in (2) of FIG. 15, the object position estimation unit 33 adds the ball holder information to the dense HD near the lost ball BL, and tracks the dense HD as the ball holder.

The target to which the ball holder information is given in this way may be changed depending on the presence or absence of dense HD in the vicinity of the lost ball. The range specified by "near" in this example can be set to any numerical value.

[1-3] Effect of the first embodiment In a game image in sports such as rugby, the ball may be hidden behind a player or a crowd (scrum, maul, rack, etc.) formed near the ball and cannot be seen.

Therefore, the video processing device 1 according to the first embodiment sets and tracks the ball holder when the ball is lost in the analysis processing of the sports game video. Specifically, when the object detection unit 31 constantly detects the ball and the player and the object detection unit 31 cannot detect the ball, the player closest to the position where the object position estimation unit 33 last detected the ball. The ball holder information is given to the ball and the ball is tracked. Then, the object position estimation unit 33 maintains the set ball holder information until the object detection unit 31 re-detects the ball.

As a result, the object position estimation unit 33 can interpolate the trajectory of the ball that the object detection unit 31 could not detect. In other words, the object position estimation unit 33 can continuously grasp the movement locus of the ball by tracking the ball holder when the ball is lost. That is, the video processing device 1 according to the first embodiment can improve the tracking accuracy of the ball in the analysis processing of the sports game video.

Further, the video processing device 1 according to the first embodiment can narrow the area for searching for a lost ball by setting a ball holder. Specifically, the ball that disappears from the image is likely to be held by a nearby player (ball holder), and is likely to be rediscovered from the vicinity of the ball holder. That is, the image processing device 1 improves the ball detection accuracy and improves the ball detection accuracy by the object detection unit 31A referring to the ball holder information and searching for the ball from the vicinity of the player to which the ball holder information is given. The detection speed can be increased.

[2] Second Embodiment Next, the video processing device 1 according to the second embodiment will be described. The video processing device 1 according to the second embodiment sets a plurality of ball holders in the video processing device 1 described in the first embodiment.

[2-1] Operation First, in the operation of the image processing device 1 based on the object position estimation program 24 with reference to FIG. 16, a case where a plurality of ball holders are set will be described. FIG. 16 shows a flowchart of an operation in which the video processing apparatus 1 additionally processes while the loop of steps S50 to S56 in FIG. 13 is repeated.

The object position estimation unit 33 executes the operation shown in FIG. 16 when the player passes near the ball holder. Specifically, the object position estimation unit 33 additionally adds the ball holder information to the player who has passed the vicinity of the ball holder (step S60). That is, when the operation of step S60 is executed, there is a situation in which a plurality of ball holders exist.

When the lost ball is rediscovered as the object position estimation program 24 progresses (step S61), the object position estimation unit 33 corrects the ball holder information during the period when a plurality of ball holders exist (step S62). ). Specifically, the object position estimation unit 33 erases the history of a player other than the ball holder who is presumed to have actually held the ball during the period as a ball holder. Then, the object position estimation unit 33 continues the object position estimation process described with reference to FIG. A specific example of a situation in which a plurality of ball holders are set in this way is shown in FIG.

FIG. 17 is an example of a case where one ball holder intersects with another player. (1) of FIG. 17 shows players P1 and P2, and shows a situation in which player P1 is given ball holder information. From this situation, player P2 passes near player P1. Then, as shown in (2) of FIG. 17, the object position estimation unit 33 also assigns the ball holder information to the player P2 who intersects with the player P1 who is the ball holder, and both the player P1 and the player P2. Track the player as the ball holder. Then, when the ball BL is re-detected from the vicinity of the player P2 in the following situation, the object position estimation unit 33 estimates that the ball holder after (2) is the player P2. That is, since the object position estimation unit 33 can consider that the ball holder during the period (1) to (2) is the player P1 and the ball holder after (2) is the player P2, the trajectory of the ball BL is shown in FIG. It can be inferred that the locus is as shown by the broken line shown in (3).

As described above, the object position estimation unit 33 can extract the correct ball holder information from the plurality of ball holder information. The video processing device 1 may apply the methods shown in FIGS. 18 to 20 as a method of narrowing down the candidates for the ball holders from a plurality of ball holders.

The method shown in FIG. 18 is for estimating the correct ball holder from the velocity information of a plurality of ball holders. (1) of FIG. 18 shows players P1 and P2, and shows a situation in which ball holder information is given to both players P1 and P2. From this situation, the moving speed estimation unit 35 calculates the speeds of the athlete P1 and the athlete P2, and the object position estimation unit 33 compares the speeds of the athlete P1 and the athlete P2. Then, when the object position estimation unit 33 detects, for example, that the player P1 has a faster movement speed than the player P2, the object position estimation unit 33 stops tracking the player P2 as a ball holder as shown in (2) of FIG. To do. As a method for the object position estimation unit 33 to compare the speeds, the movement speed estimation unit 35 may compare by calculating the specific speeds of the athlete P1 and the athlete P2, but the comparison is not limited to this. For example, even if the moving speed estimation unit 35 notifies the object position estimation unit 33 of the relative speed relationship between the player P1 and the player P2, the object position estimation unit 33 narrows down the candidates for the ball holder. good. Further, the timing at which the object position estimation unit 33 compares the moving speeds of the plurality of ball holders is set to be after a predetermined time has elapsed after the occurrence of the plurality of ball holders. The number of frames that specify this predetermined time can be changed as appropriate. Further, the object position estimation unit 33 may limit the candidates for the ball holders when the difference in the moving speeds of the plurality of ball holders exceeds a certain threshold value.

The method shown in FIG. 19 is to infer the correct ball holder from the subsequent play of the plurality of ball holders. FIG. 19 (1) shows players P1 and P2 of team A and players P3 and P4 of team B, and shows a situation in which player P1 and player P2 are given ball holder information. From this situation, when the player P1 of the A team and the player P3 of the B team come into contact with each other to form a dense HD, the object position estimation unit 33 is a ball holder as shown in (2) of FIG. The dense HD formed by the player P1 is regarded as the ball holder, and the tracking of the player P2 as the ball holder is stopped.

The method shown in FIG. 20 is for estimating the correct ball holder when a plurality of ball holders are generated starting from the density. (1) of FIG. 20 shows the players P1 and P2 of the A team, the players P3 and P4 of the B team, and the dense HD1, and shows the situation where the ball holder information is given to the players P1 and the dense HD1. Has been done. From this situation, when the player P1 of the A team and the player P3 of the B team come into contact with each other to form a dense HD, the object position estimation unit 33 is a ball holder as shown in (2) of FIG. The dense HD2 newly formed by the player P1 is regarded as a ball holder, and the tracking of the dense HD1 as a ball holder is stopped.

[2-2] Effect of Second Embodiment In the video processing device 1 according to the second embodiment, the object position estimation unit 33 temporarily assigns ball holder information to a player who has passed near the ball holder. Track multiple ball holders. Then, when the ball is rediscovered, the ball holder information in the period set by the plurality of ball holders is corrected.

As a result, the video processing device 1 according to the second embodiment can improve the tracking accuracy of the ball as compared with the first embodiment, and can improve the accuracy of re-detecting the lost ball by the object detection unit 31. .. Then, the video processing device 1 according to the second embodiment can grasp a more accurate trajectory of the ball during the game.

[3] Third Embodiment Next, the video processing device 1 according to the third embodiment will be described. The video processing device 1 according to the third embodiment further executes the play type determination program 25 when the analysis program 21 is executed in the video processing device 1 described in the first embodiment.

[3-1] Operation First, the operation of the video processing device 1 based on the play type determination program 25 will be described with reference to FIG. FIG. 21 shows an operation flowchart of the video processing device 1 by the play type determination program 25.

As shown in FIG. 21, the play type determination unit 36 first selects a frame for executing the play type determination process from the moving image data (step S70).

Next, the play type determination unit 36 confirms the type of the object identified by the object detection program 23 (step S71). Specifically, among the detected objects, the player and the ball are confirmed. If the ball is not detected, the ball holder information is acquired from the analysis result of the object position estimation program 24.

Next, the play type determination unit 36 confirms the movement speed of the object calculated by the movement direction estimation unit 34 (step S72), and further confirms the movement direction of the object calculated by the movement speed estimation unit 35 (step S73). Then, the play type determination unit 36 determines the play type at that time from the position information of the object detected by the object detection unit 31 and the moving direction and moving speed of the confirmed object (step S74). Here, examples of the type of play determined by the play type determination unit 36 include "run", "pass", and "kick".

The criterion for "run" is set to the conditions shown in FIG. 22, for example. FIG. 22 shows players P1 and P2, and shows a situation in which player P1 holds the ball BL. And player P1 and player P2 are running in the direction of the enemy team. In this situation, the play type determination unit 36 recognizes that the player P1 and the ball BL are moving at substantially the same speed and in substantially the same direction, and determines that the play type is “run”.

The criterion for "pass" is set to the conditions shown in FIG. 23, for example. FIG. 23 shows players P1 and P2, showing a situation in which player P1 is passing the ball BL toward player P2. In this situation, the play type determination unit 36 recognizes that the ball BL away from the player P1 is moving behind the offside line 50, and determines that the play type is "pass".

The criterion for "kick" is set to the conditions shown in FIG. 24, for example. FIG. 24 shows players P1 and P2, and shows a situation in which player P1 is kicking the ball BL toward the front of player P2 who has run into it. In this situation, the play type determination unit 36 recognizes that the ball BL away from the player P1 is moving ahead of the offside line 50, and determines that the play type is “kick”.

As described above, the play type determination unit 36 determines the play type at that time, and stores the determined play type information in the storage unit 13. The information obtained by executing the play type determination program 25 is recorded as shown in FIG. 25, for example. FIG. 25 shows an example of a table showing the analysis results by the play type determination program 25.

In the analysis result of the play type determination program 25 shown in FIG. 25, the number of frames at the time when the play type changes is recorded. For example, when the number of frames of the moving image data is 60, the play type is “kick”. And when the number of frames is "420", the play type is changed to "run". In such a case, the period from "60" to "420" in the number of frames indicates that the play type was "kick". The same applies to the play type in the following frame. The method of recording the analysis result of the play type determination program 25 is not limited to this, and for example, the corresponding play type may be recorded in all frames.

Then, the object detection unit 31 confirms whether or not the frame that has been processed so far is the last frame (step S75). If the frame in which the process is executed does not correspond to the last frame (step S75, NO), the process returns to step 70, and the play type determination unit 36 selects the following frame and executes the same process. On the other hand, when the frame in which the processing is executed corresponds to the last frame (step S75, YES), the play type determination unit 36 ends the execution of the play type determination program 25.

It is also possible to use information on the moving speed as a method for discriminating between the "pass" and the "kick" described above. For example, since the speed of the "kick" is expected to be faster than the speed of the "pass", a predetermined threshold value is set for the moving speed of the ball BL away from the player. Then, for example, when the speed of the ball is higher than the threshold value, it is determined as "kick", and when it is equal to or less than the threshold value, it is determined as "pass". As a result, the play type determination unit 36 can improve the determination accuracy of the "pass" and the "kick".

[3-2] Effect of Third Embodiment The video processing device 1 according to the third embodiment plays based on the moving direction and moving speed of the ball calculated by the moving direction estimating unit 34 and the moving speed estimating unit 35. The type determination unit 36 automatically determines the type of play at each time point in the match video. As a result, the video processing device 1 according to the third embodiment can reduce the load on the user to analyze the game video.

[4] Fourth Embodiment Next, the video processing device 1 according to the fourth embodiment will be described. The video processing device 1 according to the fourth embodiment further executes the scene division program 26 when the analysis program 21 is executed in the video processing device 1 described in the first embodiment.

[4-1] Operation First, the operation of the video processing device 1 based on the scene division program 26 will be described with reference to FIG. FIG. 26 shows an operation flowchart of the video processing device 1 by the scene division program 26.

As shown in FIG. 26, the specific sound detection unit 37 first analyzes the audio data in the video file (step S80). Specifically, the specific sound detection unit 37 confirms, for example, the waveform of the voice from the start to the end of the voice data. Then, the specific sound detection unit 37 confirms whether or not the specific sound is detected in the analyzed voice data (step S81). Examples of this specific sound include a whistle sound of a referee used during a game, and the specific sound detection unit 37 detects a sound having a specific frequency corresponding to the whistle sound.

When the specific sound is detected (step S81, YES), the specific sound detection unit 37 notifies the scene division unit 38 of the number of frames of the moving image data corresponding to the time when the specific sound is detected. Then, the scene division unit 38 that receives the notification divides the scene of the video file with reference to the frame (step S82). Here, the "scene" indicates a break of one play, and specifically, a period of the image separated by the whistle sound.

When the specific sound is not detected in step S81 (step S81, NO), or after the scene division process in step S82 is completed, the specific sound detection unit 37 confirms whether or not all the sound analysis is completed (step). S83). If the sound analysis is not completed (step S83, NO), the process returns to step S80, and the same process is repeated by the specific sound detection unit 37 and the scene division unit 38. On the other hand, when the sound analysis is completed (step S83, YES), the specific sound detection unit 37 ends the execution of the scene division program 26.

A specific example of the scene division process described above is shown in FIG. 27. In FIG. 27, the progress state of the video file and the time axis are displayed.

As shown in FIG. 27, the video of the match starts at time t0, and the whistle sound of the start of the match is detected by the specific sound detection unit 37 at time t1. Then, the scene division unit 38 divides the scene of the video file at time t1. Similarly, thereafter, the scene dividing unit 38 divides the video file at the times t2 and t3 when the specific sound detecting unit 37 detects the specific sound. In the example shown in FIG. 27, the information between the time t3 and the time t4 is omitted, and the specific sound detection unit 37 detects the whistle at the end of the game at the time t4. Then, the video ends at time t5. By such processing, the video file is divided into scenes 1 to n (n is an integer of 5 or more) as shown in FIG. 27.

Then, the user can add tag information to the divided scenes after the analysis program 21 ends. FIG. 28 shows an example in which tag information is added to the divided scenes. As shown in FIG. 28, tag information is added for each scene. As the tag information, for example, information such as the start and end of the game and information on the play type such as scrum, lineout, and try are used.

The specific sound information detected by the specific sound detection unit 37 may use a specific frequency directly specified by the user. Further, as the information of the specific sound, the preset frequency information may be used based on the information such as the whistle manufacturer and model number selected by the user.

[4-2] Effect of Fourth Embodiment In the video processing device 1 according to the fourth embodiment, the scene division unit 38 of the game video is based on the timing when the specific sound detection unit 37 detects the specific sound from the game video. Divide the scene automatically. As a result, the video processing device 1 according to the third embodiment can easily structure the division of play in the game video, and can reduce the load on the user to analyze the game video.

[5] Fifth Embodiment Next, the video processing apparatus 1 according to the fifth embodiment will be described. The video processing device 1 according to the fifth embodiment is an execution example of the reproduction program 27 that displays the analysis result in the video processing device 1 described in the first to fourth embodiments.

[5-1] Operation First, the operation of the video processing device 1 based on the reproduction program 27 will be described with reference to FIG. 29. FIG. 29 shows an example of a screen displayed on the display 14 by the reproduction program 27.

When the playback program 27 is executed as shown in FIG. 29, for example, a video display area 60, a video control unit 61, a field display area 62, a tag selection area 63, and a thumbnail area 64 are displayed on the display 14.

The video display area 60 is an area for displaying the analyzed moving image data. In this area 60, the CPU 10 can overlay display the detection result of the object by the analysis program 21 based on the setting by the user. As the overlay display setting, for example, the size, shape, color, and the like of the frame indicating the position of the detected player are specified for each type of object.

The image control unit 61 displays a button for controlling the image to be reproduced in the image display area 60. When the CPU 10 detects that the play button, pause button, stop button, and progress bar in the video control unit 61 have been operated by the user, the CPU 10 applies an operation corresponding to each button to the video displayed in the area 60. To do.

The field display area 62 is an area for displaying the analysis result of the analysis program 21. Specifically, a two-dimensional field is displayed in the area 62, and icons are displayed at the coordinates where the players, the crowd, and the ball are located at the time displayed in the area 60. As a method of displaying players, crowds, and balls in this area 62, the size, shape, color, and the like of the icon can be specified for each type of object, as in the area 60. Further, the CPU 10 may indicate the field area displayed in the area 60 in the area 62. In this case, the CPU 10 masks and displays the outside of the image area as shown in FIG. 9, for example.

The tag selection area 63 is an area for displaying tag information given by the user after the analysis program 21 is executed. When the CPU 10 detects that the tag information displayed in the area 63 has been selected by the user, the CPU 10 causes the thumbnail area 64 to display thumbnails of the scenes corresponding to the selected tag information. The CPU 10 may overlay the start time of the scene on the plurality of thumbnails displayed in the thumbnail area 64. Then, when the CPU 10 detects that one of the thumbnails in the area 64 is selected by the user, the CPU 10 displays the image of the scene corresponding to the thumbnail in the area 60.

The CPU 10 can overlay the lines and grids on the video display area 60 as shown in FIG. 30 based on the settings made by the user. FIG. 30 is an example in which lines and grids are overlaid on the image shown in FIG. In this example, the CPU 10 overlays different lines on the 10-meter line 42A, the 22-meter line 43A, the 5-meter line 45 and 45B, and the 15-meter line 46A and 46B, and grids GD in the vertical and horizontal directions of the field, respectively. Is displayed as an overlay. Further, the CPU 10 can change the line type, thickness, color, etc. of the line to be overlaid for each line type, and similarly change the line type, thickness, color, display interval, etc. of the grid. It is possible.

Further, the CPU 10 can display the loci of the player and the ball on the field display area 62 as shown in FIG. 31 based on the setting by the user. FIG. 31 is an example in which the movement locus TR of the player and the ball BL is added to the example shown in FIG. 12 and displayed in the field display area 62. As shown in FIG. 31, the CPU 10 can display the locus TRs of the players P1 to P8 and the ball BL on the field display area 62. Further, the CPU 10 can change the line type, thickness, color, etc. of the locus TR for each type of object. In this example, the locus TR of the players P1 to P8 is indicated by a solid arrow, and the locus TR corresponding to the ball BL is indicated by a broken line arrow. Further, the CPU 10 can set the number of past frames to be displayed from the currently displayed frame for the movement locus TR in an arbitrary period.

Although the players included in the dense HD are omitted in FIG. 31, the CPU 10 can display the players included in the dense HD without omitting them. In this case, the CPU 10 displays, for example, a black circle and a white circle indicating a player in a broken line circle indicating dense HD.

Further, in the above description, the case where the analysis result is displayed in the two-dimensional field has been described as an example, but the present invention is not limited to this. For example, the entire field may be represented by a 3D perspective view to display similar information.

[5-2] Effect of Fifth Embodiment The video processing device 1 according to the fifth embodiment can display the match video and the analysis result of the match video in the first to fourth embodiments together. Specifically, the video processing device 1 can simultaneously display the game video and the two-dimensional field in which information such as players and balls at the time corresponding to the game video is mapped. Further, the image processing device 1 can overlay the displayed game image with a grid or the like, and can further display the movement trajectory for the player and the ball mapped on the two-dimensional field.

As a result, the video processing device 1 according to the fifth embodiment can help the user analyze the play factors from the game video, so that the analysis time of the game video that contributes to the strengthening of the user's team is significantly shortened. Can be done.

Further, in the video processing device 1 according to the fifth embodiment, the user can select and play back a desired scene from the thumbnail images of the scenes displayed in the list. As a result, the video processing device 1 according to the fifth embodiment can easily find the scene that the user wants to refer to, so that the efficiency of analyzing the game video by the user can be improved.

[6] Modifications, etc. The video processing program of the above embodiment uses a computer to process a video of a sport, from the video to a first object <BL, FIG. 11>, and a plurality of second objects different from the first object. It is possible to detect the object <P1-P8, FIG. 11> and estimate the positions of the detected first object and a plurality of second objects for each frame in the video, and detect the first object in a certain frame. If not, it is considered that the second object located in the vicinity of the first object holds the first object in the frame before the frame <ball holder, FIG. 14>.

This makes it possible to provide a video processing program capable of continuously tracking the ball detected from the sports game video.

In the above embodiment, the case where the video processing is executed by the video processing device 1 has been described as an example, but the present invention is not limited to this. For example, the user may execute the video processing by using the cloud service as shown in FIG. 32. FIG. 32 shows a communication network NW that connects a computer PC owned by a user, a server SV used for a cloud service, and the computer PC and the server SV. In this case, the program used for the video processing described in the above embodiment is managed on the server SV side. Then, the user can access the server SV via the communication network NW and upload the file of the game video to be analyzed to the server SV side to execute the process described in the above embodiment. The usage method of the cloud service is not limited to this, and various usage methods can be considered. For example, in video processing, the program to be executed may be shared between the user's computer PC and the server SV side for processing.

Further, in the above embodiment, the case where the video processing of the video processing device 1 is targeted for all from the start to the end of the video file has been described as an example, but the present invention is not limited to this. For example, the video processing apparatus 1 may execute the analysis process only during the period based on an arbitrary period set by the user.

Further, in the above embodiment, the case where various video processes are executed by software has been described as an example, but the present invention is not limited to this. For example, some processes may be realized by dedicated hardware, or all processes may be realized by dedicated hardware. For example, in video processing, it is possible to use a circuit that functions as a field detection unit 30 and a circuit that functions as an object detection unit 31. The same applies to other processes. These hardwares may be built in the video processing device 1 or may be used by an external connection.

Further, in the above embodiment, the flowchart used for explaining the various programs is an example, and the present invention is not limited to this. For example, the order of the processes in each flowchart can be changed as much as possible. For example, the process of step S57 described with reference to FIG. 13 may be executed between steps S55 and S56.

Further, in the above embodiment, the case where the ball holder is set immediately after the ball cannot be detected has been described as an example, but the present invention is not limited to this. For example, when the period during which the ball cannot be detected from the moving image reaches a predetermined number of frames or more, the ball holder may be set. Further, when a predetermined number of frames are detected before and after the time when the ball cannot be detected, the position of the ball in the current frame can be linearly interpolated from the position where the ball is detected before and after that. ..

Further, in the above embodiment, various programs can be executed in parallel as much as possible. Further, when it is necessary for various programs to acquire common information, the information can be shared between the programs. Further, the field detection method and the object detection method described in the above embodiment are merely examples, and are not limited thereto. Different methods may be used as long as the field, player, and ball can be detected.

Further, in the above embodiment, the "color information" indicates, for example, a color histogram. That is, the team type determination unit 32 can group by detecting a specific waveform from the detected color histogram of each player, for example.

Although the embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and the equivalent scope thereof.

1 ... Video processing device, 10 ... CPU, 11 ... RAM, 12 ... ROM, 13 ... Storage unit, 14 ... Display, 20 ... Main program, 21 ... Analysis program, 22 ... Field detection program, 23 ... Object detection program, 24 ... Object position estimation program, 25 ... Play type determination program, 26 ... Scene division program, 27 ... Playback program, 28 ... Display data, 30 ... Field detection unit, 31 ... Object detection unit, 32 ... Team type determination unit, 33 ... Object position estimation unit, 34 ... movement direction estimation unit, 35 ... movement speed estimation unit, 36 ... play type determination unit, 37 ... specific sound detection unit, 38 ... scene division unit.

Claims (14)

To computer to process footage of sports,
The first object is detected from the image,
A plurality of second objects different from the first object are detected from the video.
Estimating the position of the detected first object and the positions of the plurality of detected second objects is repeated for each frame in the video.
If it can not detect the first object in a certain frame, the previous frame of the frame, all with the second object located in the vicinity of the first object is holding the first object,
The plurality of second objects include third and fourth objects.
When the third object and the fourth object are close to each other when the third object is being tracked assuming that the third object is holding the first object, the fourth object also holds the first object. The moving speed of the third object is compared with the moving speed of the fourth object after the lapse of the first time in the video, and the moving speed of the third and fourth objects is faster. A video processing program that keeps track of one and stops tracking the slower one. To computer to process footage of sports,
The first object is detected from the image,
A plurality of second objects different from the first object are detected from the video.
Estimating the position of the detected first object and the positions of the plurality of detected second objects is repeated for each frame in the video.
If it can not detect the first object in a certain frame, the previous frame of the frame, all with the second object located in the vicinity of the first object is holding the first object,
The plurality of second objects include third and fourth objects.
When the third object and the fourth object come into contact with each other after the third object is considered to hold the first object, the density including the third object and the fourth object becomes said. A video processing program that considers the first object to be held. A claim that causes the second object, which is deemed to hold the first object, to be tracked as holding the first object in a subsequent frame until the next detection of the first object. The video processing program according to claim 1 or 2. In previous frame can not be detected with the first object, wherein the plurality of nearest second object to the first object of the second object is considered to be holding the first object, to claim 1 The video processing program according to any one of claims 3. Claims 1 to claim that the frame for searching for the second object located in the vicinity of the first object when the first object cannot be detected is a frame immediately before the first object cannot be detected. Item 4. The video processing program according to any one of items 4. The video processing program according to any one of claims 1 to 5 , which is grouped based on the color information of the plurality of second objects. The field area displayed in the image is detected, and the field area is detected.
The correspondence between the coordinates of the field area and the entire field shown in two dimensions is calculated.
The coordinates of the first object and the plurality of second objects in the entire field are calculated.
The video processing program according to any one of claims 1 to 6 , wherein an image of the entire field, a position of the first object, and a position of the plurality of second objects are displayed on a display unit. The method according to any one of claims 1 to 7 , wherein the type of play in the sport is determined based on the movement locus of the first object, the second object, and the second object holding the first object. Video processing program. The video processing program according to claim 8 , wherein a play type is determined based on the moving speed and moving direction of the first object. On the display unit of each of the movement locus of the previous SL the second object to the first object in a line of different features, the image processing program according to any one of claims 1 to 9. The first sound is detected in the sound in the video,
The video processing program according to any one of claims 1 to 10 , wherein the scene of the video is divided based on the timing at which the first sound is detected. Tag information is added to each of the divided scenes.
An index based on the tag information is displayed on the display unit, and the index is displayed on the display unit.
The video processing program according to claim 11 , wherein when the signal for selecting the index is received, the information of the scene corresponding to the selected index is displayed in a list. In the video of sports, the first object is detected from the video.
A plurality of second objects different from the first object are detected from the video,
Estimating the position of the detected first object and the positions of the plurality of detected second objects is repeated for each frame in the video.
If it can not detect the first object in a certain frame, the previous frame of the frame, all with the second object located in the vicinity of the first object is holding the first object,
When the third object included in the plurality of second objects is being tracked assuming that the first object is held, the third object and the second object included in the plurality of second objects are included. When the four objects are close to each other, the fourth object is also regarded as holding the first object and tracked, and after the lapse of the first time in the video, the moving speed of the third object and the movement speed of the fourth object An image processing method for comparing with a moving speed, continuing tracking of the third and fourth objects having a faster moving speed, and stopping tracking of the slower moving speed. In the video of sports, the first object detection unit that detects the first object from the video,
A second object detection unit that detects a plurality of second objects different from the first object from the video,
It is provided with a position estimation unit that repeats estimating the position of the detected first object and the positions of the plurality of detected second objects for each frame in the video.
In the position estimation unit, when the first object detection unit cannot detect the first object in a certain frame, the second object located in the vicinity of the first object in the frame before the frame is the second object. everyone and the holding the first object,
The position estimation unit considers that the third object included in the plurality of second objects holds the first object and tracks the third object, and the third object and the plurality of second objects. When the fourth object included in the object is in close proximity, the fourth object is also regarded as holding the first object and tracked, and the moving speed of the third object after the lapse of the first time in the video. An image processing device that compares the moving speed of the fourth object with the moving speed of the fourth object, continues tracking of the third and fourth objects having a faster moving speed, and stops tracking of the slower moving speed of the third and fourth objects.

Images