JP7285045B2 - Image synthesizing device, image synthesizing method and program - Google Patents

Description

The present invention relates to an image synthesizing device, an image synthesizing method, and a program.

In broadcast program images, etc., computer graphics images (hereinafter referred to as CG images) that describe the names, descriptions, and other related matters of objects such as people and buildings that appear in the images are displayed on or near the objects. A composite image placed in

At present, in these synthesizing techniques, the layout position for arranging the CG image is determined in advance based on the object, and the computer graphics image is laid out at that position to generate a synthesized video.

However, depending on the position of the object on the image, the CG image may not fit within the screen if the CG image is placed at a predetermined position. As a result, in such cases, synthesis of CG images has been given up.

Also, when there are many objects, the number of CG images to be synthesized increases accordingly. In such a case, if the CG images were arranged at predetermined positions as in the conventional art, the CG images often overlapped with each other.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an image synthesizing device, an image synthesizing method, and a program for adaptively controlling the superimposed position of a CG image according to the position of an object on a video image.

One aspect of the present invention is an image synthesizing device that synthesizes a computer graphic image related to a target object appearing in a video with the video, comprising: an image recognition unit that recognizes the target object from the video; A position detection unit for detecting a position on the image of the target object recognized by the image recognition unit; and a placement of the computer graphic image on the image using the position on the image of the target object detected by the position detection unit. The image synthesizing device includes an image layout unit that adaptively determines a position, and an image synthesizing unit that generates an image in which the computer graphic image is superimposed on the layout position determined by the image layout unit.

One aspect of the present invention is an image synthesis method for synthesizing a computer graphic image related to a target object appearing in a video with the video, wherein the target object is image-recognized from the video, and the recognized target object is recognized. and adaptively determining the placement position of the computer graphic image on the video using the detected position of the target object on the video, wherein the computer graphic image is placed by the image placement unit An image synthesizing method for generating an image superimposed on a determined arrangement position.

One aspect of the present invention is a computer program for synthesizing a computer graphic image related to a target object appearing in a video with the video, comprising: image recognition processing for recognizing the target object from the video; Position detection processing for detecting a position on the video of the target object recognized by the image recognition processing, and placement of the computer graphic image on the video using the position on the video of the target object detected by the position detection processing. A program for causing a computer to execute an image arrangement process for adaptively determining a position and an image composition process for generating an image in which the computer graphic image is superimposed on the arrangement position determined by the image arrangement unit.

Since the present invention adaptively controls the superimposed position of the CG image according to the position of the object on the video, it is possible to generate a video in which the CG image is superimposed at an appropriate position.

FIG. 1 is a block diagram of an image synthesizing device 1 according to the first embodiment. FIG. 2 is a diagram showing an example of recognition of the position and size of the target object on the screen by the target object position/size detection unit 12. In FIG. FIG. 3 is a diagram showing an example of a CG image stored in the CG image storage unit 13. As shown in FIG. FIG. 4 is a diagram for explaining an example of controlling the size of a CG image. FIG. 5 is a diagram for explaining an example of how the CG image placement unit 15 determines the placement position of the CG image. FIG. 6 is a block diagram of the image synthesizing device 1 configured by the computer system 2. As shown in FIG. FIG. 7 is an operation flowchart of the image synthesizing device 1. FIG. FIG. 8 is a diagram for explaining a specific operation of the image synthesizing device 1. FIG. FIG. 9 is a diagram for explaining a specific operation of the image synthesizing device 1. FIG. FIG. 10 is a diagram for explaining specific operations of the image synthesizing apparatus 1. FIG. FIG. 11 is a diagram for explaining specific operations of the image synthesizing apparatus 1. FIG. FIG. 12 is a diagram for explaining a modification of the first embodiment. FIG. 13 is a block diagram of the image synthesizing device 1 of the second embodiment. FIG. 14 is a diagram showing an image of a marathon runner or the like taken as a target object from the front. FIG. 15 is a diagram showing an image of a marathon runner or the like taken as a target object from the front. FIG. 16 is a block diagram of the image synthesizing device 1 of the third embodiment. FIG. 17 is a diagram for explaining a modification of the second embodiment. FIG. 18 is a diagram for explaining a modification of the second embodiment.

<First embodiment>
An image synthesizing device and program according to a first embodiment of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram of an image synthesizing device 1 according to this embodiment. The image synthesizer 1 includes an image recognition section 11, a target object position/size detection section 12, a CG image storage section 13, a CG image selection section 14, a CG image arrangement section 15, and an image synthesis section 16. Prepare.

The image recognition unit 11 performs image recognition of an object (hereinafter referred to as a target object) related to a CG image or to be displayed, from an input video. Image recognition of the target object is performed based on the feature amount of the target object (for example, if the target object is a specific person, the feature amount of the face) or the like. As image recognition methods for recognizing such a target object, there are image recognition techniques using pattern matching and machine learning using techniques such as deep learning. Image recognition of the target object is preferably performed at predetermined time intervals (for example, every 1 second or 2 seconds).

The designation of the target object to be recognized by the image recognition unit 11 is performed by designating the target object for which the user wishes to display the CG image to the image recognition unit 1 . The target object can be specified by either a method of specifying an individual individual (e.g., a method of specifying a specific person) or a method of not specifying an individual individual (e.g., a method of specifying a person appearing in a video). But it's okay. Also, the number of target objects to be specified may be singular or plural. Then, the recognition result of the image-recognized target object (for example, attribute information of the target object, information specifying the target object on the image, etc.) is output to the target object position/size detection unit 12 . Furthermore, in the case of image recognition of a plurality of target objects, if priority is given to the target objects, the target object position/size detector 12 assigns priority to the information of the recognized target objects. output to In the following description, an example in which the target object is mainly a specific person will be described, but the target object is not limited to a person, and may be an object such as a figurine, a building, or the like.

The target object position/size detection unit 12 receives the recognition result of the image recognition unit 11 and detects the position and size of the target object on the video screen. Recognition of the position and size of the target object on the screen by the target object position/size detection unit 12 is shown in FIG. 2, for example. The example in FIG. 2 is an example in which a person is the target object. With respect to the person recognized by the image recognition unit 11, the target object position/size detection unit 12 sets the distance of the straight line connecting the top of the recognized person's head to the midpoint of the neck as L, as shown in FIG. A region of interest having a size of L×2L is set at a position downward from . Also, the midpoint of the neck of the recognized person is set as the center point. Then, the size of the recognized person is set as the area of the attention area, and the center point is set as the position coordinates of the recognized person on the screen. This example is just an example, and other methods may be used. For example, an area having a size of L×L surrounding the face may be set as the attention area. Furthermore, the face itself of the person who is the target object may be set as the attention area. Then, the target object position/size detection unit 12 outputs the attention area, center point, and attribute information of the target object to the CG image selection unit 14 .

As shown in FIG. 3, the CG image storage unit 13 stores the CG image of the target object superimposed on the video. The stored CG images are CG images having the same contents but having different sizes according to the size of the attention area of the target object for each attribute of the target object. As for the size of the CG image, basically, the larger the size of the attention area of the target object, the larger the CG image. If the size of the CG image is made proportional to the size of the target object, the CG image may become unnaturally large or small. Therefore, as shown in FIG. 4, when the size of the target object exceeds or falls below a predetermined size, the size of the CG image may be controlled so as not to increase or decrease any further.

The CG image selection unit 14 receives the attention area of the target object and the attribute information of the target object from the target object position/size detection unit 12, matches the attribute information of the target object, and selects an image corresponding to the attention area of the target object. A CG image of the size is selected from the CG image storage unit 13 . Regarding the selection of the CG image by the CG image selection unit 14, if the priority is set for the target object, it is selected from the CG image storage unit 13 in the order of the priority.

In this example, a plurality of CG images are stored in the CG image storage unit 13, and the CG image selection unit 14 selects a CG image from the CG image storage unit 13 according to the size and attribute information of the target object. but not limited to this. For example, the CG image selection unit 14 may automatically generate a CG image according to the attribute information of the target object and the size of the detected target object. In the case of automatically generating a CG image, as described above, if the size of the CG image is proportional to the size of the target object, the CG image becomes unnaturally large or small. It may get lost. Therefore, as shown in FIG. 4, when the size of the target object exceeds or falls below a predetermined size, the size of the CG image may be controlled so as not to increase or decrease any further.

The CG image placement unit 15 places the CG image selected by the CG image selection unit 14 at an appropriate position on the video screen. A plurality of CG image video image placement positions on the screen are set in advance, and the placement positions are given an order of priority. Then, according to the order of priority, an attempt is made to arrange the CG image, and the CG image is arranged at the possible arrangement position with the highest priority.

An example of how the CG image placement unit 15 determines the placement position of the CG image will be described with reference to FIG. First, the layout position of the CG image is set on a circle centered on the center point of the target object. In the example of FIG. 5, positions of 0 degrees, 90 degrees, 180 degrees, and 270 degrees are the layout positions of the CG image. The priority of the arrangement position is set to the arrangement position a of 180 degrees, the arrangement position b of 0 degrees, the arrangement position c of 270 degrees, and the arrangement position d of 90 degrees in the highest order. The priority order of the arrangement positions described above is an example, and the present invention is not limited to this. For example, the priority of the placement positions may be placement position b, placement position a, placement position c, and placement position d in the highest order.

The CG image placement unit 15 first tries to place the CG image at the placement position a, which has the highest priority. At this time, if all the CG images arranged at the arrangement position a can be displayed within the screen and, as will be described later, do not overlap the attention area of the target object with the higher priority, the CG image arrangement The position is determined as placement position a. On the other hand, if the CG image placed at the placement position a protrudes from the screen and cannot be displayed entirely, or if it overlaps with the attention area of the target object with the higher priority, the placement position a It is determined that the CG image cannot be arranged. Then, an attempt is made to arrange the CG image at the arrangement position b, which has the next highest priority. Such determination is performed until the arrangement position is determined. If the placement position cannot be determined even after all the placement positions have been verified, it is decided not to determine the placement of the CG image and not to superimpose the CG image on the video.

The determination of the placement position of the CG image placement unit 15 described above is an example, and the method is not limited to this, and other methods may be used.

The image synthesizing unit 16 superimposes the CG image selected by the CG image selecting unit 14 on the arrangement position on the image determined by the CG image arranging unit 15 to generate an image synthesized with the CG image.

Specifically, the image synthesizing device 1 described above can be realized by a computer system 2 (information processing device) having a processor that performs various kinds of arithmetic processing. FIG. 6 is a block diagram of the image synthesizing device 1 configured by the computer system 2. As shown in FIG.

The image synthesizing apparatus 1 includes hardware such as a processor 21, a memory (ROM or RAM) 22, a storage device (hard disk, semiconductor disk, etc.) 23, an input device (keyboard, mouse, touch panel, etc.) 24, a display device 25, a communication device 26, and the like. It can be configured by a general-purpose computer 2 having hardware resources.

Programs stored in the storage device 23 are loaded into the memory 22 and executed by the processor 21 so that the computer 2 performs image recognition processing, target object position/size detection processing, CG image selection processing, CG Image arrangement processing and image composition processing are realized. Image recognition processing corresponds to the image recognition unit 11, target object position/size detection processing corresponds to the target object position/size detection unit 12, CG image selection processing corresponds to the CG image selection unit 14, The CG image arrangement processing corresponds to the CG image arrangement section 15, and the image composition processing corresponds to the image composition section 16. FIG.

Next, the operation of the image synthesizing device 1 described above will be described.

FIG. 7 is an operation flowchart of the image synthesizing device 1. FIG. In the following explanation, it is assumed that the arrangement positions of CG images and their priorities are those shown in FIG.

First, a case where the target object is a single specific person X and the specific person X moves on the image will be described with reference to the operation flowchart of FIG. 7 and specific examples of FIGS.

First, the number of target objects is set (Step 100). In this example, the target object is only the specific person X, and since it is singular, P=1.

The image recognition unit 11 recognizes the image of the specific person X whose CG image is to be displayed (Step 101). Then, the recognition result of the image-recognized target object (for example, attribute information of the target object, information specifying the target object on the image, etc.) is output to the target object position/size detection unit 12 . When the target objects are given priority, the information of the recognized target object is given priority and output to the target object position/size detection unit 12 .

The target object position/size detection unit 12 detects the position and size of the specific person X recognized by the image recognition unit 11 on the screen. That is, the attention area and center point of the specific person X are calculated (Step 102).

The CG image selection unit 14 sets the priority M of the target object to 1 (Step 103). The CG image selection unit 14 acquires the attention area and attribute information of the specific person X with the priority M=1 (Step 104). The CG image selection unit 14 selects a CG image corresponding to the size of the attention area of the specific person X and the attribute information (Step 105).

The CG image placement unit 15 acquires the center point of the specific person X from the target object position/size detection unit 12 and the CG image selected from the CG image selection unit 14 (Step 106). Then, the number of priorities of the layout positions of the CG images is set to Q (Step 107), the priority is set to N, and N=1 (Step 108).

The CG image placement unit 15 attempts to place the CG image at the placement position with priority 1 (Step 109). FIG. 8 shows the position of the specific person X on the video at the time t on the video. is shown. In addition, the priority order of the arrangement position is set to the arrangement position a of 180 degrees, the arrangement position b of 0 degrees, the arrangement position c of 270 degrees, and the arrangement position d of 90 degrees. Therefore, the CG image placement unit 15 attempts to place the CG image at the placement position a with the priority of 1. FIG.

The CG image placement unit 15 determines whether a CG image can be placed at the placement position with the priority of 1 (Step 110). In the example of FIG. 8, when the CG image is placed at the placement position a, the entire CG image can be displayed and does not overlap the attention area of another target object, so the CG image can be placed at the placement position a. I judge.

When the CG image placement unit 15 determines that the placement is possible, the image synthesizing unit 16 generates an image in which the CG image is superimposed on the placement position (Step 111). The example in the lower diagram of FIG. 8 shows an image in which a CG image is superimposed at the arrangement position a.

The CG image selection unit 14 determines whether or not the target object to be processed remains (Step 112). That is, it is determined whether M=P. In this example, the target object is a single specific person X, so M=P=1. Therefore, the process ends.

Next, the case where the specific person X moves at time t+1 will be described with reference to the operation flowchart of FIG. 7 and the specific example of FIG.

First, the number of target objects is set (Step 100). In this example, since the specific person X is singular, P=1.

The image recognition unit 11 recognizes the image of the specific person X whose CG image is to be displayed (Step 101). Then, the recognition result of the image-recognized target object (for example, attribute information of the target object, information specifying the target object on the image, etc.) is output to the target object position/size detection unit 12 . When the target objects are given priority, the information of the recognized target object is given priority and output to the target object position/size detection unit 12 .

The target object position/size detection unit 12 detects the position and size of the specific person X recognized by the image recognition unit 11 on the screen. That is, the attention area and center point of the specific person X are calculated (Step 102).

The CG image selection unit 14 sets the priority M of the target object to 1 (Step 103). The CG image selection unit 14 acquires the attention area and attribute information of the specific person X with the priority M=1 (Step 104). The CG image selection unit 14 selects a CG image corresponding to the size of the attention area of the specific person X and the attribute information (Step 105). What should be noted here is the size of the selected CG image. The size of the attention area of the specific person X at time t is larger than the size of the attention area of the specific person X at time t+1. Therefore, it should be noted that the CG image selected for the specific person X at time t+1 is larger than the CG image selected for the specific person X at time t.

The CG image placement unit 15 acquires the attention area and center point of the specific person X from the target object position/size detection unit 12 and the CG image selected from the CG image selection unit 14 (Step 106). Then, the number of priorities of the layout positions of the CG images is set to Q (Step 107), the priority is set to N, and N=1 (Step 108). In this example, since the priority of the layout position of the CG image is 1 to 4, the number of priority is 4 and Q=4.

The CG image placement unit 15 attempts to place the CG image at the placement position with priority 1 (Step 109). FIG. 9 shows the position of the specific person X on the video at time t+1 on the video. is shown. In addition, the priority order of the arrangement position is set to the arrangement position a of 180 degrees, the arrangement position b of 0 degrees, the arrangement position c of 270 degrees, and the arrangement position d of 90 degrees. Therefore, the CG image placement unit 15 attempts to place the CG image at the placement position a with the priority of 1. FIG.

The CG image placement unit 15 determines whether a CG image can be placed at the placement position a with the priority of 1 (Step 110). In the example of FIG. 9, when the CG image is arranged at the arrangement position a, the CG image cannot be entirely displayed within the screen, so it is determined that the CG image cannot be arranged at the arrangement position a.

If it is determined that the layout is not possible, the CG image layout section 15 determines whether there is a layout position where layout is possible (Step 113). That is, whether N=Q. In the example of FIG. 9, there are remaining positions b, c, and d that can be arranged. Therefore, the CG image placement unit 15 adds 1 to N (Step 114), and attempts to place the CG image at the placement position with priority 2 (Step 109). In the example of FIG. 9, when the CG image is placed at the placement position b, the CG image can be placed at the placement position b because the entire CG image can be displayed and does not overlap the attention area of another target object. I judge. On the other hand, if N=Q, go to step 112 .

When the CG image placement unit 15 determines that the placement is possible, the image synthesizing unit 16 generates an image in which the CG image is superimposed on the placement position (Step 111). The example in the lower diagram of FIG. 9 shows an image in which a CG image is superimposed at the arrangement position b.

The CG image selection unit 14 determines whether or not the target object to be processed remains (Step 112). That is, it is determined whether M=P. In this example, the target object is a single specific person X, so M=P=1. Therefore, the process ends.

Next, a case where the target objects are a plurality of specific persons X and Y will be described using the operation flowchart of FIG. 7 and specific examples of FIGS. 10 and 11. FIG.

First, the number of target objects is set (Step 100). In this example, since there are a plurality of specific persons X and Y, P=2. Also, the priority of the specific person X is set to 1, and the priority of the specific person Y is set to 2.

The image recognition unit 11 recognizes the specific person X and the specific person Y, whose CG images are to be displayed, from the video (Step 101). Then, the recognition result of the image-recognized target object (for example, attribute information of the target object, information specifying the target object on the image, etc.) is output to the target object position/size detection unit 12 . The information on the recognized specific person X is given priority 1, and the information on the recognized specific person Y is given priority 2 and output to the target object position/size detection unit 12 .

The target object position/size detection unit 12 detects the positions and sizes of the specific person X and the specific person Y recognized by the image recognition unit 11 on the screen. That is, the attention areas and center points of the specific person X and the specific person Y are calculated (Step 102).

The CG image selection unit 14 sets the priority M of the target object to 1 (Step 103). In the example of FIG. 10, the priority of the specific person X is 1, and the priority of the specific person Y is 2.

The CG image selection unit 14 acquires the attention area and attribute information of the specific person X with the priority M=1 (Step 104). The CG image selection unit 14 selects a CG image corresponding to the size of the attention area of the specific person X and the attribute information (Step 105).

The CG image placement unit 15 acquires the attention area and center point of the specific person X from the target object position/size detection unit 12 and the CG image selected from the CG image selection unit 14 (Step 106). Then, the number of priority levels of the layout position of the CG image is set to Q (Step 107), the priority level is set to N, and N=1 (Step 108). In this example, since the priority of the layout position of the CG image is 1 to 4, the number of priority is 4 and Q=4.

The CG image placement unit 15 attempts to place the CG image at the placement position with priority 1 (Step 109). FIG. 10 shows the position on the video of the specific person X at the time t on the video. is shown. In addition, the priority order of the arrangement position is set to the arrangement position a of 180 degrees, the arrangement position b of 0 degrees, the arrangement position c of 270 degrees, and the arrangement position d of 90 degrees. Therefore, the CG image placement unit 15 attempts to place the CG image at the placement position a with the priority of 1. FIG.

The CG image placement unit 15 determines whether a CG image can be placed at the placement position with the priority of 1 (Step 110). In the example of FIG. 10, when the CG image is placed at the placement position a, the entire CG image can be displayed and does not overlap the attention area of another target object with a higher priority. It is determined that it can be placed in a.

When the CG image placement unit 15 determines that the placement is possible, the image synthesizing unit 16 generates an image in which the CG image is superimposed on the placement position (Step 111). The example in the lower diagram of FIG. 10 shows an image in which a CG image of a specific person X is superimposed at the arrangement position a.

The CG image selection unit 14 determines whether or not the target object to be processed remains (Step 112). That is, it is determined whether M=P. In this example, M=1 and P=2, so M=P is not true. Therefore, 1 is added to M (Step 115), and the process returns to Step 104.

The CG image selection unit 14 acquires the attention area and attribute information of the specific person Y with the priority M=2 (Step 104). The CG image selection unit 14 selects a CG image corresponding to the size and attribute information of the attention area of the specific person Y (Step 105). What should be noted here is the size of the selected CG image. As shown in FIG. 11, the size of the attention area of the specific person X at time t is smaller than the size of the attention area of the specific person Y at time t. Therefore, it should be noted that the CG image selected for the specific person Y is smaller than the CG image selected for the specific person X.

The CG image placement unit 15 acquires the attention area and center point of the specific person Y from the target object position/size detection unit 12 and the CG image selected from the CG image selection unit 14 (Step 106). Then, the number of priorities of the layout positions of the CG images is set to Q (Step 107), and N=1 (Step 108).

The CG image placement unit 15 attempts to place the CG image at the placement position with priority 1 (Step 109). FIG. 11 shows the position of the specific person Y on the video, and shows the arrangement positions a to d of four CG images on a circle with the center point of the specific person Y as the center. is. In addition, the priority order of the arrangement position is set to the arrangement position a of 180 degrees, the arrangement position b of 0 degrees, the arrangement position c of 270 degrees, and the arrangement position d of 90 degrees. Therefore, the CG image placement unit 15 attempts to place the CG image at the placement position a with the priority of 1. FIG.

The CG image placement unit 15 determines whether a CG image can be placed at the placement position a with the priority of 1 (Step 110). In the example of FIG. 11, when the CG image is placed at the placement position a, the CG image overlaps the attention area of the specific person X with the higher priority, so it is determined that the CG image cannot be placed at the placement position a. .

If it is determined that the layout is not possible, the CG image layout section 15 determines whether there is a layout position where layout is possible (Step 113). That is, whether N=Q. In the example of FIG. 11, there are remaining positions b, c, and d that can be arranged. Therefore, the CG image placement unit 15 adds 1 to N (Step 114), and attempts to place the CG image at the placement position with priority 2 (Step 109). In the example of FIG. 11, when the CG image is arranged at the arrangement position b, the CG image cannot be displayed on the screen, so it is determined that the CG image cannot be arranged at the arrangement position b.

If it is determined that the layout is not possible, the CG image layout section 15 determines whether there is a layout position where layout is possible (Step 113). That is, whether N=Q. In the example of FIG. 11, there are still positions c and d that can be arranged. Therefore, the CG image placement unit 15 adds 1 to N (Step 114) and attempts to place the CG image at the placement position with the priority of 3 (Step 109). In the example of FIG. 11, when the CG image is placed at the placement position c, the entire CG image can be displayed, or since it does not overlap the attention area of the target object of the specific person X, the CG image is placed at the placement position c. It is determined that placement is possible. On the other hand, if N=Q, go to step 112 .

When the CG image placement unit 15 determines that the placement is possible, the image synthesizing unit 16 generates an image in which the CG image is superimposed on the placement position (Step 111). The example in the lower diagram of FIG. 11 shows an image in which a CG image of a specific person Y is superimposed at the arrangement position c.

The CG image selection unit 14 determines whether or not the target object to be processed remains (Step 112). That is, it is determined whether M=P. In this example, since the target objects are two specific persons X and Y, M=P=2. Therefore, the process ends.

In the first embodiment, the position and size of the CG image to be superimposed are adaptively controlled according to the position and size of the target object on the video. It is possible to generate an image that is superimposed with

The display of the CG image may be continued while the target object is being displayed (appeared) on the video, or it may not be displayed after a predetermined period of time has passed since the start of the display of the CG image. If the target object displaying the CG image once disappears from the video and is displayed (appears) again, control may be performed so that the CG image is not displayed.

<Modification of First Embodiment>
In the first embodiment, an example has been described in which the CG image selection unit 14 selects a CG image stored in advance and synthesizes the selected CG image with a video.

However, when many CG images are arranged, or depending on the arrangement positions of the CG images, it may be difficult to understand which target object the CG image belongs to.

Therefore, in the modified example of the first embodiment, an example in which a CG image to be superimposed on a video image is modified according to the arrangement position will be described.

As a representative method of deformation, there is a method of providing balloons and arrows indicating which target object the CG image to be superimposed is the CG image. FIG. 12 shows an example in which a CG image is deformed by balloons. In the example of FIG. 12, each of the CG images a to d is an image of a balloon, and shows an example in which the origin of the balloon is set to the center point of the target object.

These transformations are made possible by, after the CG image placement unit 15 has determined the placement position of the CG image, transforming the balloon of the selected CG image from the determined placement position toward the center point of the target object. .

<Second Embodiment>
In competitions such as relay road races and marathons, many people appear in the video, and the rankings fluctuate greatly. In such a case, if all athletes participating in the competition are set as target objects and CG images are arranged for all athletes, the CG images make it difficult to see the original video. Therefore, it is conceivable to limit the number of CG images to be laid out by giving priority to the target objects, such as from the top.

However, simply recognizing the target object does not give the ranking.

Therefore, in the second embodiment, a method of identifying the positional relationship of the recognized target objects on the video and setting the priority of the target objects based on the positional relationship will be described.

FIG. 13 is a block diagram of the image synthesizing device 1 of the second embodiment.

The image synthesizing device 1 of the second embodiment includes a positional relationship identifying section 30 in addition to the image synthesizing device 1 of the first embodiment.

The positional relationship identifying section 30 identifies the positional relationship on the image of the target object identified by the image identifying section 11 . One way to identify the positional relationship is to use the size of the target object on the image. In a two-dimensional image, objects in the foreground appear larger, and objects in the distance appear smaller. This is also the case when shooting a competition. For example, in a marathon or a long-distance relay race, when the athletes are photographed from the front, the athletes running in front appear larger on the image, and the athletes behind them appear smaller on the image. The same is true when the player is photographed from behind. The positional relationship identification unit 30 uses such a property to identify the positional relationship of each target object from the size of the target object identified by the image identification unit 11 on the image.

Specifically, the positional relationship identification unit 30 focuses on the size (for example, area) of the region of interest detected by the target object position/size detection unit 12 as an index of the size of the identified target object on the image. do.

For example, FIG. 14 shows a case where the target object is a marathoner or the like, and the image is taken from the front of the athlete. In the example of FIG. 14, player A's attention area is the largest, and player C's attention area is the smallest. Therefore, it can be identified that the ranking of the players is Athlete A, Athlete B, and Athlete C. On the other hand, FIG. 15 shows the case of an image of a player photographed from behind. In the example of FIG. 15 as well, the attention area of player A is the largest, and the attention area of person C is the smallest. However, since the images are shot of the players from behind, it can be identified that the players are ranked as player C, player B, and player A.

Such a photographing direction may be obtained from the camera that takes the photograph, or it may be possible to identify whether the face is the front or the rear by image recognition, and to distinguish whether the image is photographed from the front or from the rear. .

By focusing on the size of the identified target object on the image in this way, the positional relationship identifying section 30 identifies the positional relationship on the image of the target object identified by the image identifying section 1 . Then, the positional relationship identification unit 30 gives priority to the target object whose positional relationship is identified to the center point, the size of the attention area, and the attribute information of the target object detected by the target object position/size detection unit 12. and output to the CG image selection unit 14 . Note that the attention area described above is only an example, and other methods may be used. For example, an area having a size of L×L surrounding the face may be set as the attention area. Furthermore, the face itself of the person who is the target object may be set as the attention area.

In the above example, the positional relationship is identified by focusing on the attention area of the identified target object, but the present invention is not limited to this. Attention may be paid to the length such as L that determines the size of the attention area. Furthermore, for example, it may be the size of the player in the image of the target object itself. Another method is to use a reference point on the image. An example of a reference point is a vanishing point on an image. To specify the positional relationship, first, the image identification unit 1 acquires the position information of the identified player on the video. The distance between the vanishing point on the video and the position information of each player on the video may be calculated, and the longer the distance, the closer to the front of the video it may be estimated.

The CG image placement unit 15 places the CG images in the order of priority of the target objects identified by the positional relationship identification unit 30, as in the first embodiment described above. At this time, the CG image placement unit 15 can limit the number of CG images to be placed by setting the target objects to which the CG images are to be placed to, for example, the target objects ranked first to third from the top. can.

<Modification of Second Embodiment>
A modification of the second embodiment will be described. In the second embodiment, a method of setting the priority of target objects according to the positional relationship of the target objects (players) has been described. In the modification of the second embodiment, it is the same until the priority of the target object (player) is selected according to the positional relationship of the target object (player), but the priority of the target object (player) is used. Next, an example of changing the priority of the CG image placement position by the CG image placement unit 15 will be described.

FIG. 17 is a diagram for explaining a modification of the second embodiment. In the example of FIG. 17, the priority of the placement position on the screen of the video of the CG image by the CG image placement unit 15 is shown. , and the priority order of placement positions where the running order of the target object (athlete) is 4 or lower.

The priority of the placement positions of the target object (athlete) running order 1 to 3 is as follows. It is set to the arrangement position b. On the other hand, the priority of the placement positions for which the target object (athlete) has a running order of 4th or lower is 0-degree placement position b, 270-degree placement position c, 90-degree placement position d, and 180-degree placement position d. It is set to the arrangement position a.

The reason why the priority of the placement position of the CG image on the screen is changed according to the running order of the target object (athlete) is that the higher the running order of the target object (athlete), that is, the closer the target object (athlete) is to the front of the video. The CG image of the target object (athlete) is displayed below the target object (athlete) (for example, the placement position a), and the target object (athlete) with the lower running order is displayed behind the image. The closer the target object (player) is, the more the player's CG image is displayed above the target object (player) (for example, the placement position b).

FIG. 18 is a diagram showing an example of displaying CG images of six players from player A to player F according to the placement order shown in FIG. In FIG. 17, the first to third place runners are assigned the priority of the first to third running order, and the CG image of the leading player A is displayed at the highest priority position a. The CG image of player B in second place is displayed at position c, which has the second highest priority, and the CG image of player C in third place is displayed at position a, which has the highest priority. . On the other hand, the 4th place and lower runners are assigned the priority of the 4th place and lower running order. The CG image of player E ranked 6th is displayed at the highest priority placement position b, and the CG image of player F ranked sixth is displayed at the highest priority placement position b.

The priority order of the arrangement positions described above is an example, and the present invention is not limited to this. You may change the priority of an arrangement position suitably.

By doing so, when a large number of target objects (athletes) are shown in the video, the higher the target object (athlete) in the running order, that is, the more forward the target object (athlete) is, the more the target object (athlete) is projected. The CG image of the object (athlete) is displayed below (chest position), and the lower the running order of the target object (athlete), that is, the more backward the target object (athlete) is, the more the target object (athlete) is displayed. A CG image is displayed above, and natural image expression can be performed.

<Third Embodiment>
Depending on the content, it is likely that the size of the target object that appears does not change in many cases. For example, in an image captured by a fixed-point camera installed on a rooftop, an object such as a building that appears in the screen changes as the camera pans and tilts, but the size of the object itself does not change in many cases. When arranging a CG image in such a video, it is thought that the position of the target object should be taken into consideration.

Therefore, the third embodiment is an example in which the size of the target object is not taken into consideration, but only the position of the target object is taken into consideration when arranging it on the screen of the CG image.

FIG. 16 is a block diagram of the image synthesizing device 1 of the third embodiment.

The third embodiment differs from the first embodiment in that a target object position detector 40 is provided instead of the target object position/size detector 12 . The target object position detection unit 40 receives the recognition result of the image recognition unit 11 and detects the position of the target object on the video screen. That is, the function of detecting the size of the target object is removed from the target object position/size detection unit 12 .

Furthermore, the CG image selection unit 14 selects a CG image of one size regardless of the size of the target object. As described above, a CG image may be generated in real time instead of being selected from CG images prepared in advance.

Other operations are the same as those of the first embodiment, so description thereof is omitted.

Although the present invention has been described above with reference to preferred embodiments, it is not necessary to include all the configurations of the embodiments, and not only can they be combined as appropriate, but the present invention is not necessarily limited to the above embodiments. However, it can be modified and implemented in various ways within the scope of its technical ideas.

1 image synthesis device 2 computer 11 image recognition unit 12 target object position/size detection unit 13 CG image storage unit 14 CG image selection unit 15 CG image placement unit 16 image synthesis unit 21 processor 22 memory 23 storage device 24 input device 25 display Device 26 Communication device 30 Positional relationship identification unit 40 Target object position detection unit

Claims (9)

An image compositing device for compositing a computer graphic image associated with a target object appearing in a video into the video, comprising:
an image recognition unit capable of image recognition of a plurality of target objects from the video;
a position detection unit that detects the position on the image of each target object recognized by the image recognition unit;
a size detection unit that detects the size of each target object recognized by the image recognition unit on the image;
a positional relationship identification unit that identifies the positional relationship between target objects using the size of each target object on the image detected by the size detection unit and determines the priority of each target object;
Using the position of each target object on the video detected by the position detection unit and the priority of each target object determined by the positional relationship identification unit, the arrangement position of the computer graphic image on the video is determined. an image placement unit that adaptively determines;
an image synthesizing unit that generates an image in which the computer graphic image is superimposed at the layout position determined by the image layout unit ;
has
The image placement unit predetermines a plurality of candidate placement positions, which are candidates for the placement position of the computer graphic image and have different priorities according to the priority of the target object,
The image arrangement unit arranges the computer graphic image at the candidate arrangement position of each recognized target object, and all of the computer graphic images are displayed on the video, and the target object with the high priority and the computer graphic image are arranged. determining the candidate layout position with the highest priority among the candidate layout positions that do not overlap with the graphic image as the layout position of the computer graphic image;
Image synthesizer. the priority of the target object is the running order of the target object;
The image placement unit predetermines a plurality of candidate placement positions, which are candidates for the placement position of the computer graphic image and are given different priorities according to the running order of the target object,
The image arrangement unit arranges the computer graphic image at the candidate arrangement position of each recognized target object, and all of the computer graphic images are displayed on the video, and the target object with the highest running order and the computer determining a candidate layout position with the highest priority among layout positions that do not overlap with the graphic image as the layout position of the computer graphic image;
2. The image synthesizing device according to claim 1. an image selection unit that selects a computer graphic image corresponding to the size of the target object on the image detected by the size detection unit;
3. The image synthesizing apparatus according to claim 1, wherein said image arranging section arranges the computer graphic image selected by said image selecting section at said candidate arrangement position. A memory storing computer graphic images of a plurality of sizes including at least a computer graphic image of a first size, which is the smallest size, and a computer graphic image of a second size, which is the largest size. has a part
The image selection unit selects the computer graphics image of the first size when the size of the detected target object on the video is equal to or less than a predetermined size, and selecting a computer graphic image of said second size if the size is greater than or equal to a predetermined size;
4. The image synthesizing device according to claim 3. The size detection unit detects a size of a region of interest including at least part of the target object recognized by the image recognition unit as a size of the target object on the image.
5. The image synthesizing device according to any one of claims 1 to 4 . 6. The image synthesizing device according to claim 5, wherein the predetermined area of the target object is the attention area. 7. The image synthesizing apparatus according to claim 1, wherein said image placement unit places said computer graphic image only on target objects having predetermined higher priority among the priority of each target object. . An image compositing method for compositing a computer graphic image associated with a target object appearing in a video into the video, comprising:
a plurality of candidate placement positions, which are candidates for the placement position of the computer graphic image and have different priorities according to the priority of the target object, are determined in advance;
Image recognition of a plurality of target objects from the video,
Detecting the position of each recognized target object on the image,
detecting the size of each recognized target object on the image,
using the size of each detected target object on the image to identify the positional relationship between the target objects and determine the priority of each target object;
arranging the computer graphic image at the candidate arrangement position of each recognized target object by using the detected position of each target object on the video and the determined priority of each target object; All of the computer graphic image is displayed on the video, and the highest priority candidate layout position among the candidate layout positions where the target object with the high priority and the computer graphic image do not overlap is used as the layout of the computer graphic image. position and
generating a video in which the computer graphic image is superimposed on the video at the determined arrangement position;
Image composition method. 1. A computer program for synthesizing a computer graphic image associated with a target object appearing in a video into said video, comprising:
an image recognition process capable of recognizing a plurality of target objects from the video;
a position detection process for detecting the position on the image of each target object recognized by the image recognition process;
a size detection process for detecting the size of each target object recognized by the image recognition process on the image;
a positional relationship identification process for identifying the positional relationship between the target objects using the size of each target object on the image detected by the size detection process, and determining the priority of each target object;
Using the position of each target object on the video detected by the position detection process and the priority of each target object identified by the positional relationship identification process, the arrangement position of the computer graphic image on the video is determined. adaptively determined image arrangement processing;
causing a computer to perform an image synthesizing process for generating an image in which the computer graphic image is superimposed at the arrangement position determined by the image arrangement process ;
The computer stores in advance a plurality of candidate placement positions, which are candidates for the placement position of the computer graphic image and have different priorities according to the priority of the target object,
The image arrangement process arranges the computer graphic image at the candidate arrangement position of each recognized target object, all of the computer graphic images are displayed on the video, and the target objects with high priority and the computer determining the candidate layout position with the highest priority among the candidate layout positions that do not overlap with the graphic image as the layout position of the computer graphic image;
program.

Images