JP2022094907A - Generation device, generation method, and program - Google Patents

Abstract

To generate a virtual viewpoint image capable of addressing a problem that a position of a subject is different from a desired position.SOLUTION: An image generation device 120 includes: a subject shape generation section 405 for obtaining shape data representing a shape of a subject based on a plurality of photographed images obtained by photographing the subject in a photographing area by a plurality of photographing devices; a subject shape center-of-gravity calculation section 406 for identifying a subject position; a subject position setting section 408 for identifying a referential position representing a reference for the position of the subject; and a virtual viewpoint image generation section 412 for generating a virtual viewpoint image corresponding to a deviation between the subject position and the referential position based on the shape data.SELECTED DRAWING: Figure 4

Description

The present disclosure relates to a technique for generating a virtual viewpoint image.

In recent years, a plurality of shooting devices are arranged around a shooting area to shoot, and an image (virtual viewpoint) viewed from a designated viewpoint (virtual viewpoint) is used by using a plurality of shot images acquired from each shooting device. The technique of generating an image) is attracting attention. According to this technique, it is possible to generate an image in which a sports competition such as soccer or rugby, a concert, a dance, or the like is viewed from an arbitrary viewpoint, so that the user can be given a high sense of presence.

Patent Document 1 discloses a technique for generating and displaying an arbitrary virtual camera image using an image in which a plurality of cameras are arranged so as to surround the subject and the subject is photographed. According to Patent Document 1, in the viewing content using the technique of virtual camera image generation, it is possible to view the dance, acting, etc. of the photographed performer (subject) from various angles.

Japanese Unexamined Patent Publication No. 2008-15756

However, for example, in dance or acting, the position of the performer who is the subject may be different from the desired position at the time of shooting. As a result, even in the virtual viewpoint image generated based on the captured image, the position of the performer is different from the desired position. In addition, when the position of the subject is deviated during shooting, the user does not notice that the position of the subject is different from the desired position on the virtual viewpoint image depending on the position of the specified virtual viewpoint. There can also be.

This disclosure has been made in view of the above issues. The purpose is to generate a virtual viewpoint image that can correspond to the position of the subject being different from the desired position.

The generation device according to the present disclosure includes acquisition means for acquiring shape data representing the shape of the subject based on a plurality of captured images obtained by photographing the subject in the photographing area by a plurality of photographing devices, and a subject in the photographing area. Based on the first specific means for specifying the subject position, which is the position of the subject, the second specific means for specifying the reference position as the reference for the position of the subject, and the shape data acquired by the acquisition means. It is characterized by having a generation means for generating a virtual viewpoint image according to a deviation between a subject position specified by the first specific means and a reference position specified by the second specific means.

According to the present disclosure, it is possible to generate a virtual viewpoint image that can correspond to a position of a subject different from a desired position.

It is a figure for demonstrating the structure of an image processing system. It is a figure which shows an example of the installation of a plurality of photographing apparatus. It is a figure for demonstrating the hardware configuration of an image generator. It is a figure for demonstrating the functional structure of the image generation apparatus in 1st Embodiment. It is a flowchart for demonstrating the process performed by the image generation apparatus in 1st Embodiment. It is a figure which shows an example of arrangement of the shape data of a subject in 1st Embodiment. It is a figure which shows an example of the virtual viewpoint image generated in 1st Embodiment. It is a figure for demonstrating the functional structure of the image generation apparatus in 2nd Embodiment. It is a flowchart for demonstrating the process performed by the image generation apparatus in 2nd Embodiment. It is a figure which shows an example of arrangement of the shape data of a subject in 2nd Embodiment. It is a figure for demonstrating the functional structure of the image generation apparatus in 3rd Embodiment. It is a flowchart for demonstrating the process performed by the image generation apparatus in 3rd Embodiment. It is a figure which shows an example of arrangement of the shape data of a subject in 3rd Embodiment. It is a figure for demonstrating the functional structure of the image generation apparatus in 4th Embodiment. It is a figure which shows an example of the installation of the virtual camera in 4th Embodiment. It is a figure which shows an example of the subject to be photographed, and the generated virtual viewpoint image. It is a figure which shows an example of the arrangement of the shape data of a subject outside the photographing area in 1st Embodiment.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The components described in the following embodiments are examples of the embodiments, and the present disclosure is not limited to them.

(First Embodiment)
FIG. 1 is a diagram showing an image processing system 100 according to the present embodiment. The image processing system 100 includes a plurality of photographing devices 110, an image generation device 120, and a terminal device 130. Each photographing device 110 and the image generating device 120 are connected via a communication cable such as a LAN (Local Area Network) cable. In the present embodiment, the communication cable is a LAN cable, but the communication cable is not limited to the embodiment.

The photographing device 110 is, for example, a digital camera capable of photographing an image (still image and moving image). Each shooting device 110 is installed so as to surround a shooting area in a shooting studio or the like, and shoots an image (video). In the present embodiment, a case where a plurality of performers such as a dance scene are photographed as a subject will be described as an example. The captured image is transmitted from the photographing device 110 to the image generation device 120. FIG. 2 is a diagram showing an installation example of the photographing apparatus 110. In the present embodiment, it is assumed that the plurality of photographing devices 110 are installed so as to photograph all or a part of the photographing studio. That is, the image processing system 100 of the present embodiment includes a plurality of photographing devices 110 for photographing a subject from a plurality of directions.

The image generation device 120 stores the captured images obtained by the photographing device 110, and when the virtual viewpoint information and the reproduction time information are input by the user operation in the terminal device 130, the captured image and the virtual viewpoint are obtained. Based on this, a virtual viewpoint image is generated. Here, the virtual viewpoint information includes information indicating the position of the virtual viewpoint (virtual viewpoint) in the virtual space constructed from the captured image and the line-of-sight direction from the virtual viewpoint. The information included in the virtual viewpoint information is not limited to this. For example, information regarding the width (angle of view) of the field of view of the virtual viewpoint may be included. Further, the configuration may include information indicating at least one of the position of the virtual viewpoint, the line-of-sight direction from the virtual viewpoint, and the angle of view of the virtual viewpoint. The reproduction time information is time information from the recording start time of the captured image. The user can generate a virtual viewpoint image of a scene corresponding to the specified reproduction time in the recorded captured image by, for example, operating the terminal device 130 described later to specify the reproduction time.

The image generation device 120 is, for example, a server device and has a database function and an image processing function. In the database, an image of a scene in which a subject does not exist in advance, such as before the start of actual shooting, is stored as a background image via the shooting device 110. Further, in a scene in which a subject exists, the image generation device 120 performs image processing on a region (hereinafter, also referred to as a foreground image) corresponding to a specific object such as a person such as a performer and a tool used by the performer in the captured image. Separate and keep as a foreground image. The specific object may be an object such as a prop whose image pattern is predetermined.

The virtual viewpoint image corresponding to the virtual viewpoint information shall be generated from the background image managed by the database and the specific object image. As a virtual viewpoint image generation method, for example, model-based rendering (MBR) is used. The MBR is a method of generating a virtual viewpoint image using a three-dimensional shape generated based on a plurality of captured images of a subject captured from a plurality of directions. Specifically, the 3D shape (model) of the target scene obtained by a 3D shape restoration method such as the visual volume crossing method or Multi-View-Stereo (MVS) is used to view the scene from a virtual viewpoint. It is a technique to generate as an image. As the method of generating the virtual viewpoint image, a rendering method other than the MBR may be used. The generated virtual viewpoint image is transmitted to the terminal device 130 via a LAN cable or the like.

The terminal device 130 is, for example, a PC (Personal Computer) or a tablet. The controller 131 is, for example, a mouse, a keyboard, a 6-axis controller, and a touch panel, and the user operates the controller 131 to display a still image or a moving image on the screen. The terminal device 130 displays, for example, a virtual viewpoint image received from the image generation device 120 on the display unit 132. The terminal device 130 further receives an instruction to move the playback time and the virtual viewpoint (instruction regarding the amount of movement and the direction of movement) in response to a user operation to the connected controller 131, and indicates instruction information according to the received instruction. The transmission signal is transmitted to the image generator 120.

FIG. 3 is a diagram showing a hardware configuration of the image generation device 120.

The image generation device 120 includes a CPU 301, a ROM 302, a RAM 303, an HDD 304, a display unit 305, an input unit 306, and a communication unit 307. The CPU 301 reads the control program stored in the ROM 302 and executes various processes. The RAM 303 is used as a temporary storage area for the main memory, work area, etc. of the CPU 301. The HDD 304 stores various data, various programs, and the like. The display unit 305 displays various information. The input unit 306 has a keyboard and a mouse, and accepts various operations by the user. The communication unit 307 performs communication processing with an external device such as the photographing device 110 via the network. The network includes Ethernet (registered trademark). Further, as another example, the communication unit 307 may wirelessly communicate with an external device.

In the example shown in FIG. 3, the HDD 304, the display unit 305, and the input unit 306 are included in the image generation device 120, but the present invention is not limited to this. For example, at least one of the HDD 304, the display unit 305, and the input unit 306 may be connected to the outside of the image generation device 120 as another device. Further, the functions and processes of the image generation device 120, which will be described later, are realized by the CPU 301 reading a program stored in the ROM 302 or the HDD 304 and executing this program. Further, the hardware configuration of the terminal device 130 is the same as the hardware configuration of the image generation device 120.

FIG. 4 is a diagram showing a functional configuration of the image generation device 120. Here, the processing performed by the image generation device 120 in the present embodiment will be described. In the present embodiment, it is assumed that a dance scene is photographed and a virtual viewpoint image is generated based on the photographed image. However, the standing position of the subject during the dance may be different from the desired standing position. FIG. 16A shows a subject in a virtual space generated by shooting a shooting area surrounded by a line with a plurality of cameras, and a virtual camera corresponding to a virtual viewpoint (hereinafter referred to as a virtual camera). It is a conceptual diagram of. Specify a virtual camera that shoots the subject from any angle, such as the virtual camera 2101 that shoots the subject from the front and watches it, or the virtual camera 2102 that shoots the subject from the side and watches it. It is possible. FIG. 16C is a screen display of a virtual viewpoint image corresponding to the virtual camera 2102. Here, it is assumed that the performer wants to perform in a standing position so as to be in a straight line when taken from the virtual camera 2102. However, when the image is actually taken, the standing position may shift, resulting in a virtual viewpoint image in which the standing position of the subject is not aligned on a straight line as shown in FIG. 16 (c).

It is also conceivable that the performer can confirm the standing position by looking at the virtual viewpoint image. At this time, depending on the designated virtual camera, the deviation of the standing position may not be easily recognized. FIG. 16B is a screen display of a virtual viewpoint image corresponding to the virtual camera 2101. In the virtual viewpoint image corresponding to the virtual camera 2101, the standing positions of the subjects appear to be aligned. However, in reality, the standing position of the subject is displaced on the virtual viewpoint image taken from the virtual camera 2102. The image generator 120 in the present embodiment aims to solve the above problem.

The image generation device 120 includes a photographed image input 401, a foreground background separation unit 402, a photographed image data storage unit 403, a camera parameter holding unit 404, a subject shape generation unit 405, a subject shape center of gravity calculation unit 406, a subject shape moving unit 407, and a subject shape moving unit 407. It has a subject position determining unit 408. Further, the image generation device 120 includes a user input unit 409, a virtual viewpoint information setting unit 410, a coloring information calculation unit 411, a virtual viewpoint image generation unit 412, and an image output unit 413. Hereinafter, each processing unit will be described.

The captured image input unit 401 converts the transmission signal input from the photographing device 110 via the LAN cable into captured image data, and outputs the transmitted signal to the foreground background separating unit 402. The foreground background separation unit 402 uses as background image data a captured image data storage unit that captures a scene in which the subject does not exist in advance, such as before the subject's performance starts, among the captured images input from the captured image input unit 401. Output to 403. Further, the subject is extracted from the image taken during the performance of the subject and output as the foreground image data to the photographed image data storage unit 403.

The captured image data storage unit 403 is a database, and among the captured image data input from the foreground background separation unit 402, an image previously captured in the absence of a subject is stored in the HDD 304 as background image data. Further, the captured image data storage unit 403 stores the difference data between the background image data and the captured image data in which the subject is present in the HDD 304 as the foreground image data. Further, the captured image data storage unit 403 outputs the foreground image data to the subject shape generation unit 405. Further, the background image data and the foreground image data designated by the coloring information calculation unit 411 are output to the coloring information calculation unit 411.

The camera parameter holding unit 404 holds camera setting information such as the shooting position information of the plurality of shooting devices 110, the focal length of the lens of the shooting device 110, and the shutter speed of the shooting device 110 as camera parameter information. It is assumed that the plurality of photographing devices 110 are installed at predetermined positions and the camera parameter information is acquired in advance. Further, the camera parameter holding unit 404 outputs the camera parameter information to the subject shape generation unit 405 and the coloring information calculation unit 411.

The subject shape generation unit 405 generates shape data representing the shape of the subject by using the foreground image data and the camera parameter information. It is assumed that the subject shape generation unit 405 generates shape data of the subject by using, for example, a three-dimensional shape restoration method such as a visual volume crossing method. Further, the shape data is output to the subject shape center of gravity calculation unit 406 and the coloring information calculation unit 411.

The subject shape center of gravity calculation unit 406 specifies the position of the subject in the photographing area. Specifically, the subject shape center of gravity calculation unit 406 uses the shape data input from the subject shape generation unit 405 to specify the center of gravity of the subject shape as the subject position. At this time, the subject shape center of gravity calculation unit 406 calculates the position of the center of gravity when the subject is viewed from a predetermined viewpoint position. The subject shape center of gravity calculation unit 406 calculates, for example, the position of the center of gravity at the viewpoint of looking down on the subject from directly above as the position of the subject. The subject shape center of gravity calculation unit 406 outputs the subject center of gravity information including the position of the subject center of gravity to the subject shape moving unit 407.

The subject shape moving unit 407 is a position for arranging the shape data of the subject based on the subject center of gravity information input from the subject shape center of gravity calculation unit 406 and the subject moving destination position information input from the subject position setting unit 408 described later. To decide. Here, the subject movement destination position information is information indicating a reference position (hereinafter, also referred to as a reference position) in which the subject should be placed. The subject shape moving unit 407 determines the arrangement of the shape data according to the deviation between the reference position and the position of the subject.

When the subject movement destination position information is the grid point information at a predetermined interval (for example, 3 meter interval) set on the floor surface, the subject shape moving unit 407 sets the position of the grid point as a reference position and the position of the grid point. , Arrange the shape data of the subject so that the position of the center of gravity of the subject matches. The shape data of the subject may be regenerated based on the subject movement destination position information. Further, when the deviation between the reference position and the position of the subject is equal to or greater than a predetermined threshold value, the subject shape moving unit 407 changes the position of the shape data so that the deviation becomes smaller than the predetermined threshold value, and the changed position. The shape data may be arranged in. The subject shape moving unit 407 outputs the moved shape data to the virtual viewpoint image generation unit 412.

The subject position setting unit 408 outputs the subject movement destination position information in the three-dimensional space preset by the user to the subject shape moving unit 407. For example, it is assumed that grid point information at intervals of 3 meters corresponding to the floor surface is output so that a plurality of subjects are arranged in a predetermined section on a predetermined straight line. The subject movement destination position information is not limited to the grid points, and may be, for example, a straight line or a curved line.

The user input unit 409 converts the transmission signal input from the terminal device 130 via the LAN cable into user input data. When the user input data is the reproduction time information and the virtual viewpoint information, the reproduction time information and the virtual viewpoint information are output to the virtual viewpoint information setting unit 410.

The virtual viewpoint information setting unit 410 updates the current position, direction, and reproduction time in the virtual space based on the reproduction time information and the virtual viewpoint information input from the user input unit 409.

After that, the reproduction time information and the virtual viewpoint information are output to the subject shape generation unit 405, the coloring information calculation unit 411, and the virtual viewpoint image generation unit 412. The origin of the virtual space shall be set in advance such as the center of the stadium.

The coloring information calculation unit 411 inputs the foreground image data and the background image data based on the reproduction time information and the virtual viewpoint information input from the virtual viewpoint information setting unit 410 from the captured image data storage unit 403. Further, the camera parameters are input from the camera parameter holding unit 404, and the shape data is input from the subject shape generation unit 405. Next, the subject shape viewed from the virtual viewpoint position is rendered (colored) with the color information of the image data taken by the actual camera at the corresponding time, and the coloring information of the subject shape is retained. For example, if the subject based on the shape data is visible from the virtual viewpoint and the actual camera position information is within a predetermined range from the position of the virtual viewpoint, the foreground image data of the actual camera is used as the color of the shape. do. Further, the coloring information calculation unit 411 outputs coloring information to the virtual viewpoint image generation unit 412.

The virtual viewpoint image generation unit 412 inputs the foreground image data and the background image data based on the reproduction time information and the virtual viewpoint information input from the virtual viewpoint information setting unit 408 from the captured image data storage unit 403. Further, the camera parameters are input from the camera parameter holding unit 404, and the moving shape data is input from the subject shape moving unit 407. After that, the background image data is subjected to projection transformation and image processing so that it can be seen as a background from the virtual viewpoint position, and is used as the background of the virtual viewpoint image. Next, for the moving subject shape viewed from the virtual viewpoint position, coloring information based on the image data taken by the photographing device at the corresponding time is input from the coloring information calculation unit 411 and rendered (coloring processing) with the color information. To generate a virtual viewpoint image. Finally, the virtual viewpoint image generated by the virtual viewpoint image generation unit 412 is output to the image output unit 413. The image output unit 413 converts the image data input from the virtual viewpoint image generation unit 412 into a transmission signal that can be transmitted to the terminal device 130, and outputs the image data to the terminal device 130.

Next, the operation of the image generation device 120 will be described. FIG. 5 is a flowchart showing the operation of the image generation device 120 according to the first embodiment. The following processing is performed by the CPU 301 reading and executing the program stored in the ROM 302 or the HDD 304. When the terminal device 130 inputs the virtual viewpoint information and the reproduction time, the process is started.

The virtual viewpoint information setting unit 410 determines whether or not the virtual viewpoint information and the reproduction time information have been input via the user input unit 409 (S501). If the virtual viewpoint information and the reproduction time information are not input (No in S501), the system waits. On the other hand, when the virtual viewpoint information and the reproduction time information are input (Yes in S501), the virtual viewpoint information and the reproduction time information are output to the subject shape generation unit 405, the coloring information calculation unit 411, and the virtual viewpoint image generation unit 412. ..

Next, the subject shape generation unit 405 includes the foreground image data input from the captured image data storage unit 403 based on the reproduction time information input from the virtual viewpoint information setting unit 410, and the camera input from the camera parameter holding unit 404. Read the parameter information (S502). Subsequently, the subject shape generation unit 405 estimates the three-dimensional shape of the subject (S503). For example, it is assumed that the shape data of the subject is generated by using a three-dimensional shape restoration method such as the visual volume crossing method. Here, the shape data is composed of a plurality of point groups, and each point includes position information.

Next, the coloring information calculation unit 411 inputs the reproduction time information input from the virtual viewpoint information setting unit 410, the foreground image data and the background image data based on the virtual viewpoint information from the captured image data storage unit 403. Further, the camera parameters are input from the camera parameter holding unit 404, and the shape data is input from the subject shape generation unit 405. Next, with respect to the subject shape viewed from the virtual viewpoint position, rendering (coloring processing) is performed using the color information of the image data taken by the actual camera at the corresponding time, and the coloring information of the subject shape is retained (S504).

Next, the subject shape center of gravity calculation unit 406 specifies the position of the center of gravity of the subject shape input from the subject shape generation unit 405 when viewed from a predetermined viewpoint position as the subject position (S505). In this embodiment, the position of the center of gravity when viewed from directly above is used as the subject shape center of gravity information. FIG. 6A is a conceptual diagram of the shape data of the subject viewed from directly above in the virtual space and the position of the center of gravity of the shape data. In the present embodiment, since a plurality of subjects are rearranged on a predetermined grid point, the center position of the front-back direction axis of the subject, the center position of the left-right direction axis, and the like are calculated from the viewpoint of looking down on the subject from directly above. And calculate the position of the center of gravity. The position of the center of gravity is indicated by a black dot of the subject, and the position of the center of gravity is arranged on a straight line or a grid point.

Returning to the description of FIG. 5, the subject shape moving unit 407 determines whether or not there is subject moving destination position information in the three-dimensional space from the subject position setting unit 408 (S506). When there is subject movement destination position information in the three-dimensional space from the subject position setting unit 408 (Yes in S506), the subject movement destination position information is input (S507). On the other hand, if there is no subject movement destination position information (No in S506), the process proceeds to S509. FIG. 6B is a conceptual diagram of the grid point positions on the floor surface showing the reference position of the subject based on the subject movement destination position information. For example, it is assumed that grid point information at intervals of 3 meters corresponding to the floor surface is output so that a plurality of subjects are arranged in a predetermined section on a predetermined straight line.

Returning to the description of FIG. 5, next, the subject shape moving unit 407 has shape data based on the subject center of gravity information input from the subject shape center of gravity calculation unit 406 and the subject moving destination position information input from the subject position setting unit 408. (S508). FIG. 6C is a conceptual diagram of the subject shape after movement. The subject position was changed so that the grid point position based on the subject movement destination position information and the center of gravity position of the subject shape match from the viewpoint looking down from directly above the subject shape, and the shape data was moved to the changed subject position. It is shown that. As a result, the position of the moving subject shape is moved so as to be within a certain distance. The subject position may be changed so that the reference position and the subject position become smaller than a predetermined threshold value, and the shape data may be arranged.

Returning to the description of FIG. 5, the virtual viewpoint image generation unit 412 uses the image data taken by the photographing device 110 at the designated time with respect to the shape data viewed from the virtual viewpoint position based on the moved shape data. And render (coloring process) to generate a virtual viewpoint image (S509). That is, the color information of the subject determined and held before the subject is moved is rendered after the subject is moved, so that the subject is displayed at the moved position. The virtual viewpoint image generation unit 412 outputs the virtual viewpoint image generated by the virtual viewpoint image generation unit 412 to the image output unit 413 (S510). The process described above may be performed based on the recorded and accumulated image data, or may be performed in real time in parallel with the image taken by the photographing apparatus 110.

In the above description of FIG. 5, it is explained that the color information of the subject determined and held before the subject is moved is rendered after the subject is moved, but the present invention is not limited to this. For example, the target subject is moved to a designated position, and the shooting positions of all the shooting devices 110 and positions other than the target subject are also relatively moved so as to maintain the positional relationship with the target subject to be targeted. Render only the subject. A virtual viewpoint image may be generated by sequentially performing this for each subject and synthesizing them. That is, instead of determining the color information before moving the subject, the color information may be calculated and rendered after the subject is moved.

FIG. 7 is a diagram showing a virtual viewpoint image generated by performing the process shown in FIG. 5 when the subject shown in FIG. 16A is photographed. FIG. 7A shows the position of the shape data of the subject in the three-dimensional space corresponding to the shooting area. According to FIG. 7A, while a plurality of subjects perform a dance or the like, they are moved to a predetermined grid point position, which is a standing position different from that at the time of the performance, and at a standing position at a certain distance. , The same operation is taken with a virtual camera. Whether the subject is photographed from the position of the virtual camera 901 or the subject is photographed from the position of the virtual camera 902, the standing position is different from the standing position where the person actually performed a dance or the like.

FIG. 7B is a display example of a virtual viewpoint image corresponding to the virtual camera 901 displayed on the terminal device 130. Even if you shoot from the front of the subject, you can always keep a certain position and perform. Further, FIG. 7C is a display example of a virtual viewpoint image corresponding to the virtual camera 902. Even if the subject is photographed from the side, it can be rearranged in a straight line and displayed, so that it is possible to always perform at the same position.

As described above, according to the embodiment of the first embodiment, when a plurality of subjects are photographed by a plurality of cameras and a virtual viewpoint image is photographed and displayed, it is possible to generate a virtual viewpoint image in which the subject is rearranged at a desired position. .. Compared to the time when the performance was shot, it is possible to view the performance with the same subject in the video of the virtual camera. In the above-mentioned example, the reference position is represented by a grid point, but the reference position may be represented by a predetermined straight line or curve. In this case, if the position of the subject deviates from the line, the shape data is arranged so that the position of the subject matches the position of an arbitrary point on the line (for example, the point on the line closest to the current position of the subject). .. Further, the reference position may be a point (coordinates) at an arbitrary position. At this time, if there are a plurality of subjects, a point at a reference position may be set for each subject.

Further, the reference position may be specified based on the subject positions of a plurality of subjects. For example, when three subjects are photographed, it is assumed that the standing position is set so that the three subjects are positioned in a straight line. At this time, a straight line connecting the subject positions of two of the three people is calculated, and the subject position of the remaining one person is changed with the point on the calculated straight line as the reference position. By doing so, a virtual viewpoint image in which the shape data is arranged so that the three subjects are positioned on a straight line is generated. It is also applicable when the number of subjects is other than three.

Further, the reference position is not limited to arranging the subject in the three-dimensional space position corresponding to the shooting area as shown in FIG. 7A, and may be set to any three-dimensional space position outside the shooting area. .. FIG. 17 shows an example in which the shape data of the subject is arranged at a three-dimensional spatial position outside the photographing area. In the example shown in FIG. 17, the reference position for moving the shape data of the target subject is set to a position outside the photographing area. According to FIG. 17, even if a plurality of subjects actually shoot a performance such as dance in the shooting area, the position of the virtual camera 901 is such that the performance such as dance is performed in a wider area than the shooting area. Is filmed.

(Second embodiment)
In this embodiment, the position of the subject is changed based on a predetermined feature of the subject. FIG. 8 is a diagram showing a functional configuration of the image generation device 1100 according to the second embodiment. The image generation device 1100 has a subject feature generation unit 1101 instead of the subject shape center of gravity calculation unit 406 of the image generation device 120 according to the first embodiment shown in FIG. The hardware configuration of the image generation device 1100 is the same as that of the image generation device 120 according to the first embodiment. Further, the same components as those of the image generation device 120 are designated by the same reference numerals, and the description thereof will be omitted.

The subject feature generation unit 1101 calculates the predetermined feature recognition of the subject and its position from the shape data input from the subject shape generation unit 405 and the coloring information corresponding to the shape data. For example, when the faces of a plurality of subjects are featured and rearranged on a straight line or on a predetermined grid point on the floor surface, the face of the subject is recognized using the shape data and the coloring information, and the position of the face is specified. do. As a result, in the subsequent processing, the shape data is arranged so that the position of the face is on a predetermined straight line or a grid point from the viewpoint of looking down on the position of the face from directly above. That is, the subject feature generation unit 1101 specifies a predetermined portion in the shape of the subject as the subject position. The subject feature generation unit 1101 outputs the subject feature position information to the subject shape moving unit 407.

FIG. 9 is a flowchart showing image processing by the image processing apparatus 1100 according to the second embodiment. Since S501 to S504 are the same as the description of FIG. 5, the description will be omitted. Further, since the description in S506 to S510 is the same as the description in FIG. 5, the description will be omitted.

In S1201, the subject shape generation unit 405 specifies the position of a predetermined portion as the subject position. At this time, the subject shape generation unit 405 identifies a position having a predetermined feature in the shape data of the subject by using image analysis such as face recognition.

FIG. 10 is an example in which the face recognition of the subject is analyzed from the shape data and the coloring information, and the position of the face is specified and used as the subject position. As shown in FIG. 10A, the subject feature generation unit 1101 specifies the position of the face of the subject. Further, the subject shape moving unit 407 arranges the shape data based on the subject position specified by the subject feature generation unit 1101. As a result, as shown in FIG. 10B, the shape data is arranged so that the position of the face coincides with the grid points when viewed from above.

For feature recognition (for example, face recognition) and position specification of the subject, features (faces) are extracted from a plurality of captured images obtained by the photographing device 110, and the feature positions (three-dimensional) are further based on camera parameter information. The position in space) shall be calculated. However, the present invention is not limited to this, and for example, a virtual camera may be set at a predetermined position to generate a virtual viewpoint image, and the generated virtual viewpoint image may be used for feature recognition and position specification.

As described above, the image generation device 1100 of the present embodiment specifies the position of a predetermined part of the subject, arranges the shape data according to the deviation between the specified position and the reference position, and generates a virtual viewpoint image. .. The predetermined portion is not limited to the face, but may be hands, feet, shoes, or the like. As a result, for example, when CM (commercial) photography of shoes is performed, the characteristics of the shoes are identified, and the shape data is such that the position of the shoes and the reference position match or the deviation of the positions becomes smaller than a predetermined threshold value. Can be placed. By doing so, it is possible to generate a virtual viewpoint image in which the position of the shoe is a desired position.

(Third embodiment)
The third embodiment is an example of changing the position of the subject based on the position of the center of gravity of the subject in the time axis direction. FIG. 11 is a diagram showing a functional configuration of the image generation device 1500 according to the third embodiment. The image generation device 1500 has a subject shape average center of gravity calculation unit 1601 instead of the subject shape center of gravity calculation unit 406 of the image generation device 120 according to the first embodiment shown in FIG. The hardware configuration of the image generation device 1500 in this embodiment is the same as that in the above-described embodiment. Further, the same reference numerals are given to the same functional configurations, and the description thereof will be omitted.

The subject shape average center of gravity calculation unit 1501 calculates the position of the center of gravity of the subject in each of the moving image frames of the captured image obtained by the photographing device 110. For example, when a dance scene of 10 minutes is shot, the position of the center of gravity at the viewpoint of looking down on the subject from directly above is specified in each frame, and the average position of the center of gravity of each moving image frame is calculated. The subject shape average center of gravity calculation unit 1501 sets the calculated average position of the center of gravity as the subject position, and outputs the information of the average position to the subject shape moving unit 407.

FIG. 12 is a flowchart showing image processing by the image processing apparatus 1500 according to the third embodiment. Of the processes shown in FIG. 16, the same processes as those shown in FIG. 5 will be omitted.

After the processing of S504, the subject shape average center of gravity calculation unit 1501 calculates the position of the center of gravity when viewed from a predetermined viewpoint position of the subject shapes of the plurality of shooting frames input from the subject shape generation unit 405. Further, the subject shape average center of gravity calculation unit 1501 calculates the average position of the center of gravity based on the center of gravity of the subject in each moving image frame, and outputs the information of the average position to the subject shape moving unit 407 (S1601).

FIG. 13 is a diagram showing the average position of the center of gravity of the subject. For example, in a series of dance scenes, it is assumed that the subject moves in the direction of the arrow shown in FIG. 13 (a). Here, the position of the center of gravity of the subject is calculated by the frame at the time before the movement during the performance of the subject and the frame after the movement during the performance. Further, the average position of the center of gravity of each frame is calculated. In FIG. 13A, the point 1301 is specified as the average position of the center of gravity.

FIG. 13B is a diagram when the average position of the center of gravity is moved based on the reference position. The shape data is arranged so that the average center of gravity position of the subject matches the grid point position based on the subject movement destination position information from the viewpoint looking down from directly above the subject. As a result, the position of the subject is arranged at a fixed position on average during a series of shooting, and moves around the position.

As described above, according to the present embodiment, by calculating the average position of the center of gravity of the subject with respect to the shooting time and rearranging the subject on a predetermined high viewpoint, the subject can be rearranged without discomfort even in a series of movements of the scene. .. In this embodiment, the average center of gravity position is calculated, but the present invention is not limited to this, and the subject center of gravity position of any frame for shooting may be used as the basic position. For example, the position of the center of gravity of the subject of the frame at the start of shooting can be rearranged according to the grid point position of the floor surface, or the position of the center of gravity of the subject of the frame immediately before the end of shooting may be used. .. Further, in the present embodiment, the center of gravity of the subject is specified and the average position of the center of gravity is set as the subject position, but the present invention is not limited to this. For example, the position of a predetermined portion described in the second embodiment may be specified, and the average position thereof may be used as the subject position.

(Fourth Embodiment)
In the fourth embodiment, a method of moving the subject relative to the virtual camera based on the positional relationship before and after the movement of the subject, taking a picture, and synthesizing the image with the virtual camera image before the movement will be described. FIG. 14 has a subject position difference calculation unit 1901, a plurality of chaos viewpoint information setting units 1902, and a plurality of virtual viewpoint image generation units 1903. The hardware configuration of the image generation device 1100 is the same as that of the above-described embodiment. Further, the same reference numerals are given to the same configurations, and the description thereof will be omitted.

The subject position difference calculation unit 1901 inputs the subject center of gravity information from the subject shape center of gravity calculation unit 405. Further, information on the reference position of the subject is input from the subject position setting unit 408, and the difference between the reference position and the position of the center of gravity of the subject is calculated.

The multiple virtual viewpoint setting unit 1902 updates the current position, direction, and reproduction time in the virtual space based on the reproduction time information and the virtual viewpoint information input from the user input unit 409, and obtains the reproduction time information and the virtual viewpoint information. Is output to the plurality of virtual viewpoint image generation units 1903. Further, when the multiple virtual viewpoint setting unit 1902 inputs the positional relationship between the position before the movement of the subject and the position after the movement of the subject as the subject difference information from the subject position difference calculation unit 1901, the difference virtual viewpoint based on the subject difference information. Generate information.

The plurality of virtual viewpoint image generation units 1903 input foreground image data and background image data based on the reproduction time information and virtual viewpoint information input from the plurality of virtual viewpoint information setting units 1902 from the captured image data storage unit 403. Further, the camera parameters are input from the camera parameter holding unit 404, and the shape data after movement is input from the subject shape moving unit 407. After that, the background image data is subjected to projection transformation and image processing so that it can be seen as a background from the virtual viewpoint position, and is used as the background of the virtual viewpoint image. Next, the moving subject shape viewed from the virtual viewpoint position is rendered (colored) with the color information based on the image data taken by the actual camera at the corresponding time to generate a virtual viewpoint image. Further, when the difference virtual viewpoint information obtained by the subject position difference calculation unit 1901 calculating the subject difference information is input, the virtual viewpoint image is generated at the designated position of the difference virtual viewpoint information, and the virtual viewpoint image is based on the user input. Synthesize. Finally, the virtual viewpoint image is output to the image output unit 413.

FIG. 15 is a conceptual diagram of a virtual camera position at a moving subject position in a virtual space. The position of the center (0, 0) of the floor surface is set as the origin, and the subject position information after movement is also set to the position of the center (0, 0) of the floor surface. The unit is meters. Assuming that the position of the subject before movement is (0, 2), when the subject is placed at the reference position (0, 0), the position (x, y) of the virtual camera 2001 is moved by (0, 2). It is moved to the position (x, y + 2). Then, the subject is photographed from the moved virtual camera 2001 to generate a virtual viewpoint image. Further, the generated virtual viewpoint image is combined with the virtual viewpoint image corresponding to the virtual camera 2001 that captures a subject other than the subject to be moved.

As described above, according to the present embodiment, the virtual camera is moved with respect to the subject to be moved to generate a virtual viewpoint image, and the virtual viewpoint image is combined with the virtual viewpoint image corresponding to the virtual camera for shooting a subject other than the subject to be moved. This makes it possible to move the position of the subject only by synthesizing the virtual viewpoint image.

(Other embodiments)
In the above-described embodiment, an example of generating a virtual viewpoint image by rearranging the shape data of the subject according to the deviation between the subject position and the reference position has been described. However, the present invention is not limited to this, and a configuration may be used in which a virtual viewpoint image including information that can identify that the subject position and the reference position are deviated is generated. For example, the shape data may be arranged at the specified subject position, and information indicating the reference position (for example, a grid point or a straight line) may be displayed. Further, the shape data may be arranged at the subject position and may be displayed so that the subject deviated from the reference position can be identified. This is, for example, surrounding the shape data of the subject deviating from the reference position with a line, changing the color or brightness of the subject to emphasize it, and the like. By these methods, the user who sees the virtual viewpoint image can recognize that the subject position and the reference position are deviated from each other. For example, when the performer wants to confirm the standing position by looking at the generated virtual viewpoint image, the deviation of the performer's position can be easily identified regardless of the position of the virtual viewpoint.

The present disclosure supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

120 Image generator 405 Subject shape generation unit 406 Subject shape center of gravity calculation unit 408 Subject position setting unit 412 Virtual viewpoint image generation unit

Claims (17)

Acquisition means for acquiring shape data representing the shape of the subject based on a plurality of captured images obtained by photographing the subject in the photographing area by a plurality of photographing devices.
The first specifying means for specifying the subject position, which is the position of the subject in the shooting area,
A second specifying means for specifying a reference position that serves as a reference for the position of the subject, and
Based on the shape data acquired by the acquisition means, a virtual viewpoint image corresponding to the deviation between the subject position specified by the first specific means and the reference position specified by the second specific means is generated. A generator characterized by having a generator. The first aspect of the present invention is characterized in that the generation means generates a virtual viewpoint image by arranging the shape data based on the reference position according to the deviation being equal to or more than a predetermined threshold value. Generator. The generation means is
The subject position specified by the first specific means is changed so that the deviation becomes smaller than a predetermined threshold value.
The generation device according to claim 1, wherein a virtual viewpoint image is generated by arranging the shape data based on the changed subject position. The generation means is
Based on the deviation, the subject position is changed so that the subject position specified by the first specific means and the reference position match.
The generation device according to claim 1, wherein a virtual viewpoint image is generated by arranging the shape data based on the changed subject position. The generator according to any one of claims 1 to 4, wherein the reference position is specified based on grid points arranged at predetermined intervals. The generator according to any one of claims 1 to 5, wherein the reference position is a position outside the photographing area. The first specifying means identifies a plurality of subject positions corresponding to a plurality of subjects in the shooting area.
The second specifying means according to any one of claims 1 to 6, wherein the second specifying means specifies the reference position based on a plurality of subject positions specified by the first specifying means. Generator. The generation device according to claim 7, wherein the second specifying means specifies the reference position so that the plurality of subject positions are at predetermined intervals. The generator according to claim 6 or 7, wherein the second specifying means specifies the reference position so that the plurality of subject positions are positioned on a predetermined straight line. The first specifying means according to any one of claims 1 to 9, wherein the first specifying means specifies the subject position based on the shape of the subject represented by the shape data acquired by the acquiring means. The generator described. The generation device according to claim 10, wherein the first specifying means specifies the position of the center of gravity in the shape of the subject represented by the shape data as the subject position. The generation device according to claim 10, wherein the first specifying means specifies the position of a predetermined portion in the shape of the subject represented by the shape data as the subject position. The generation device according to any one of claims 1 to 12, wherein the generation means generates a virtual viewpoint image including information that can identify the deviation. The generation means obtains a virtual viewpoint image including information representing a subject position specified by the first specific means and a reference position specified by the second specific means as information that can identify the deviation. The generator according to claim 13, wherein the generator is generated. The generation means generates a virtual viewpoint image including information that can identify a subject whose position deviates from the reference position is specified as the subject position by the first specific means as information that can identify the deviation. 13. The generator according to claim 13 or 14. An acquisition step of acquiring shape data representing the shape of the subject based on a plurality of captured images obtained by photographing the subject in the photographing area by a plurality of photographing devices.
The first specific step of specifying the subject position, which is the position of the subject in the shooting area, and
A second specific step of specifying a reference position that serves as a reference for the position of the subject, and
Based on the shape data acquired in the acquisition step, a generation that generates a virtual viewpoint image according to the deviation between the position specified in the first specific step and the reference position specified in the second specific step. A production method characterized by having a process. A program for operating a computer as the generator according to any one of claims 1 to 15.

Images