JP2022050305A - Processing system, control method therefor, and program - Google Patents

Abstract

To provide a technique for supporting an object (performer) when capturing images used for generating virtual viewpoint images.SOLUTION: A processing system for generating a virtual viewpoint image representing a view from a virtual viewpoint is provided, the system being configured to present an indication for determining the orientation of an object in a space including the object to be shot by multiple image capturing means.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a processing system and its control method and program.

Recently, attention has been paid to a technique of installing virtual viewpoint contents at different positions, performing synchronous shooting with a plurality of cameras, and generating virtual viewpoint contents using the multi-viewpoint images obtained by the shooting. According to such a technique, for example, a highlight scene of soccer or basketball can be viewed from various angles, so that a user can be given a high sense of presence as compared with a normal image.

Patent Document 1 discloses a technique in which a plurality of cameras are arranged so as to surround a subject, and an image of an arbitrary viewpoint is generated using images of the subject taken by the plurality of cameras.

Japanese Unexamined Patent Publication No. 2008-015756

Traditionally, there are so-called tally (tally lamps, tally lights). A tally is a red lamp attached to a TV camera or the like, which lights up when the device is in use and normally works with a switcher or the like. The performer (subject) knows that he or she is being photographed, so it can be used as a trigger for dialogue or action.

However, in the system that generates the virtual viewpoint content described above, there is no actual camera (real camera) at the position of the virtual viewpoint. Therefore, for example, when shooting in a studio, the performer does not know where to look, which may limit the production.

The present disclosure has been made in view of the above problems, and is an object of the present invention to provide a technique for supporting a subject (performer) in taking an image used for generating a virtual viewpoint image.

In order to solve this problem, for example, the processing system of the present disclosure has the following configuration. That is,
It is a processing system for generating a virtual viewpoint image that represents the appearance from a virtual viewpoint.
Multiple shooting methods for shooting the space including the subject,
In the space, the presenting means for making a presentation for determining the direction of the subject is provided.

According to the present disclosure, it is possible to provide a technique for supporting a subject (performer) in shooting an image used for generating a virtual viewpoint image.

The whole block diagram of the image processing system in 1st Embodiment. The figure which shows the structure in the space surrounding a subject and a part thereof in 1st Embodiment. Explanatory drawing for explaining a marker display. The flowchart which shows the marker display processing in 1st Embodiment. Explanatory drawing for demonstrating marker display processing in 2nd Embodiment. The figure for demonstrating the structure for marker display in 3rd Embodiment. The figure for demonstrating the structure which presents a marker in 4th Embodiment. The figure for demonstrating the method of presenting a marker in 5th Embodiment. The figure which shows the hardware configuration of each apparatus constituting the system of an embodiment. The figure which shows the hardware composition of the apparatus in the 1st modification of 1st Embodiment. The flowchart which shows the processing procedure in 1st modification of 1st Embodiment. The figure which shows the structure in the space surrounding a subject in the 2nd modification of 1st Embodiment, and a part thereof. The figure which shows the structure in the space surrounding a subject and a part thereof in 3rd and 4th modification of 1st Embodiment. The flowchart which shows the processing procedure in 5th modification of 1st Embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are given the same reference numbers, and duplicate explanations are omitted.

[First Embodiment]
A system for taking pictures by installing a plurality of cameras in facilities such as a stadium, a concert hall, and a studio will be described with reference to FIG.

FIG. 1 is an overall configuration diagram of an image processing system 100 to which the first embodiment is applied. The system 100 includes a photographing device 1, a real space information holding device 2, an image processing device 3, a user terminal 4, and a marker output device 5, which are connected via a network. The network may be wired or wireless.

In the above configuration, the real space information holding device 2, the image processing device 3, the user terminal 4, and the marker output device 5 are an information processing device represented by a personal computer (PC) and an application program executed on the information processing device. Can be realized by. FIG. 9 is a hardware block configuration diagram of the information processing apparatus 200. The information processing device 200 has a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, a display device 215, an operation unit 216, and a communication interface (I / F) 217, and has a structure in which these are connected to the system bus 218. When the power of this device is turned on, the CPU 211 executes the boot program of the ROM 212, loads the OS (operating system) from the auxiliary storage device 214 (for example, the hard disk) into the RAM 213, and executes the OS. As a result, the information processing apparatus 200 can input various instructions from the user via the operation unit 26 and execute the corresponding processing. Further, the CPU 211 can execute the application program stored in advance in the auxiliary storage device 214 under the control of the OS. For example, depending on the type of the application program, the information processing device 200 functions as a real space information holding device 2, an image processing device 3, a user terminal 4, or a marker output device 5. For example, when the information processing device 200 functions as the image processing device 3, the communication I / F 217 includes a network (to communicate with the user terminal 4 and the marker output device 5), a photographing device 1, and a real space information holding device 2. It will be connected so that it can communicate with. Further, when the information processing device 200 functions as the marker output device 5, the communication I / F 217 is connected to a network (to communicate with the user terminal 4 and the marker output device) and a projector. Become. When the information processing device 200 functions as the real space information holding device 2, the auxiliary storage device 214 may function as a file server after storing the real space information (details will be described later).

The photographing device 1 is composed of a plurality of cameras installed so as to surround the competition field, the studio, etc., and they transmit images obtained by synchronously photographing each other to the image processing device 3.

The real space information holding device 2 holds information about a predetermined range of space including the subject (performer). Specifically, it shows the 3D model information of an object (background object) reflected as a background in a virtual viewpoint image such as a field of a stadium, spectator seats, or equipment of a studio, and a range in which a virtual viewpoint can be set. Three-dimensional spatial information, as well as the installation position, shooting direction, focal length, etc. of each of the photographing devices 1. Since the information held in the real space information holding device 2 is referred to when the image processing device 3 generates a virtual viewpoint image, it may be provided in the image processing device 3.

The image processing device 3 has a virtual viewpoint image generation unit 301, a virtual camera path calculation unit 302, a virtual camera path information holding unit 303, a marker coordinate calculation unit 304, a display image generation unit 305, and a communication unit 306.

The virtual viewpoint image generation unit 301 generates a three-dimensional model of the foreground object based on the images of the plurality of viewpoints acquired from the photographing apparatus 1. Then, the virtual viewpoint image generation unit 301 applies a texture matching the virtual viewpoint acquired from the virtual camera path calculation unit 302 to the generated foreground 3D model and the background 3D model obtained from the real space information holding device 2. A virtual viewpoint image is generated by mapping and rendering. In the process of this generation, the virtual viewpoint image generation unit 301 calculates the coordinates of the foreground object and the background object reflected in the virtual viewpoint image to be generated, and executes texture mapping and rendering only for the coordinates. The virtual viewpoint image generation unit 301 passes these coordinate values to the virtual camera path information holding unit 303, which will be described later, as the subject foreground coordinates and the real space background coordinates.

The virtual camera path calculation unit 302 calculates temporally continuous virtual camera path parameters based on the contents of instructions given by the user to the virtual camera path instruction unit 403 of the user terminal 4. The virtual camera parameters are at least the position, orientation (gaze direction) and angle of view (focal length) of the virtual camera, and are frames attached to the multi-viewpoint image so that the parameters at which moment in the shooting scene can be specified. It is associated with a number or timecode. At the time of this calculation, the real space information obtained from the real space information holding unit 2 is referred to, and the virtual camera path is set within the range in which the virtual viewpoint can be set.

The virtual camera information holding unit 303 accumulates the subject foreground coordinates and the real space background coordinates received from the virtual viewpoint image generation unit 301, and the virtual camera path parameters calculated by the virtual camera path calculation unit 302.

The marker coordinate calculation unit 304 displays a marker in the real space based on the virtual camera parameters related to the virtual camera stored in the virtual camera information holding unit 303 and the three-dimensional space information (background information) held by the real space information holding unit 2. Information is generated for the coordinate parameters of the information.

The display image generation unit 305 generates a display image to be displayed on the image display unit 402 of the user terminal 4. The display image generated here is a virtual viewpoint image generated by the virtual viewpoint image generation unit 301. The virtual viewpoint image is an image showing the appearance from the virtual viewpoint.

The communication unit 306 is exchanged between the image processing device 3, the user terminal 4, and the marker output device 5 via a network (not shown) or the like. The communication unit 306 transmits / receives image, voice, text data, instruction information such as a virtual camera path instruction sent from the user side when generating a virtual viewpoint image, between the image processing device 3 and the user terminal 4. .. The communication unit 306 transmits / receives display parameters such as coordinate information necessary for marker display, marker shape, color, size, and display information between the image processing device 3 and the marker output device 5.

The user terminal 4 has a communication unit 401, an image display unit 402, a virtual camera path instruction unit 403, a user information transmission unit 404, and a marker information transmission unit 405.

The communication unit 401 transmits and receives various information to and from the device communication unit 306 described above as described above. The image display unit 402 receives and displays the image generated by the display image generation unit 305 described above. The virtual camera path instruction unit 403 receives an instruction for the virtual camera path from the user and passes it to the virtual camera path calculation unit 302 via the communication unit 401 and the communication unit 306. Here, the user does not necessarily have to specify all of the virtual camera parameters exactly for the entire time he wants to watch. For example, based on various viewpoints such as wanting to watch a virtual viewpoint image focusing on a specific player or performer, always watching a certain area around the ball, or watching a place where a more remarkable event is occurring. It is also possible to enter instructions.

The user information transmission unit 404 attaches user information including a user ID to the information transmitted from the communication unit 401 to the communication unit 306. The marker information transmission unit 405 attaches marker type parameter information such as a marker type (including shape and color) to the information transmitted from the communication unit 401 to the communication unit 306.

The marker output device 5 includes a communication unit 501, a control data generation unit 502, a marker control unit 503, and a display data output unit 504. The marker output device 5 displays (or presents) a recognizable marker for a person as a subject. The marker to be displayed is output for the position related to the virtual camera parameter. That is, the position of the marker is controlled by the marker output device 5 in conjunction with the virtual camera parameter.

As described above, the communication unit 501 communicates with the marker coordinate calculation unit 304 and the user terminal 4 via the device communication unit 306 of the image processing device 3 and the network or the like.

The control data generation unit 502 integrates the data output from the virtual viewpoint image generation unit 301 and the marker coordinate calculation unit 304 and the data output from the virtual camera path instruction unit 403 and the marker information transmission unit 405, and determines the type of marker and the data. Control data such as display update time and position coordinates are generated and output to the marker control unit 503.

The marker control unit 503 converts the marker into display data (screen coordinate system) based on display parameters such as the display position, size, shape, and display content of the marker, and outputs the data to the display data output unit 504.

The display data output unit 504 is composed of a display device such as a projector, and the display data is output based on the output data of the marker control unit 503.

FIG. 2A is a schematic diagram of a real space surrounding a subject in the image processing system according to the present embodiment. The image processing system in the present embodiment has a plurality of cameras (shooting device 1) so as to surround a predetermined range of real space including a subject. Further, it has a projector (display data output unit 504) in order to display a marker recognizable by the subject (human) at a position based on virtual camera path information.

The illustrated image processing system is an example of the case where the present configuration is implemented in a circular studio, and the cameras 21a to 21f, the projectors 22a to 22f, and the screens 23a to 23f are arranged so as to surround the subject 24. The image processing device 3 generates a virtual viewpoint image from the plurality of viewpoint images acquired by the cameras 21a to 21f, and calculates the coordinates on the screen on the extension line connecting the subject 24 and the virtual viewpoint as the marker coordinates. Then, the marker output device 5 receives the marker coordinates and displays the marker on the corresponding screen. Specifically, the marker display is projected onto the screen by the projectors 22a to 22f, and the projector 22a projects onto the front screen area 23e. Similarly, the projector 22b displays a marker in the screen area 23d, the projector 22c displays a marker in the screen area 23f, and the projector 22d displays a marker in the screen area 23b. Further, the projector 22e displays a marker in the screen area 23a, and the projector 22f displays a marker in the screen area 23c. The projectors 22a to 22f are connected by a network, and the display data output unit 504 is configured to transmit display data to the projector to be displayed. The screen is also configured on the floor surface and the top surface, and the projector 22b and the projector 22e project onto the floor surface screen 25. Further, the projectors 27a and 27b project onto the top screen 26.

FIG. 2B is a partial cross-sectional view of the circular studio of the present embodiment shown in FIG. 2A. As described above, the marker display is configured to be projected onto the screen by the projector, the projector 22 projects onto the screens 23 and 25, and the projector 27 projects a user-configured type of marker onto the screen 26. ..

FIG. 3A is a diagram in which a part of the configuration is extracted from the image processing system shown in FIG. 2A. The virtual viewpoint 31 indicates a virtual camera position when a certain virtual viewpoint image is generated. Further, the marker 32 is projected onto the screen area 23a by the projector 22e. Now, since the subject 24 cannot actually see the position of the virtual camera (virtual viewpoint 31), when trying to match the line of sight with the virtual camera, the projected marker 32 is seen. .. The projection position of the marker 32 is controlled by the marker control unit 503 so that the projection position is on the extension connecting the face (eyes) of the subject 24 and the virtual viewpoint 31. Further, the distance from the face of the subject to the virtual viewpoint 31 is expressed by the size of the circle of the marker 32b. In this way, where the virtual viewpoint 31 is located between the marker projected on the screen and the face of the subject can be expressed by the display form (here, the size) of the marker. For example, when the circle is enlarged, it can be realized by notifying the subject in advance that the virtual camera is located near the subject and when the circle is decreased, the virtual camera is located near the marker.

3 (b) to 3 (e) are views showing the positional relationship between the subject, the virtual camera, and the marker, and are views of FIG. 3 (a) viewed from the side. In FIGS. 3B to 3E, the white circle 33 indicates a virtual viewpoint, and the black circle 34 indicates a marker projected on the screen. As shown in this figure, the face of the subject, the virtual viewpoint, and the marker are configured to be aligned on a straight line. Since the screen is also configured on the floor surface and the top surface, as shown in FIGS. 3 (b) to 3 (e), the subject 24 is virtual even if the virtual camera is at an angle from above or below the subject. You can look at the camera.

FIG. 4 is a flowchart showing the processing of the control data generation unit 502 of the marker output device 5.

In S100, the control data generation unit 502 starts the generation process. In S101, the control data generation unit 502 performs a marker display setting file generation process based on the data output from the marker information transmission unit 405. This configuration file contains information that defines the marker type and marker display parameters (color, shape, etc.).

In S102, the control data generation unit 502 counts the number of subjects from the three-dimensional model of the subject in the virtual viewpoint image output from the virtual viewpoint image generation unit 301, and assigns a subject ID.

In S103, the control data generation unit 502 calculates the coordinates of the face of the subject in the world coordinate system from the three-dimensional model information of the subject. If there are multiple subjects, the same processing is performed for the number of subjects. Further, the face can be specified from the three-dimensional model of the subject by using a general image processing technique from the shape and feature points of the model.

In S104, the control data generation unit 502 calculates the marker coordinates to be displayed based on the setting file, the subject position (face coordinates), and the virtual camera coordinates. The marker coordinates are calculated in the world coordinate system, and the marker coordinates that intersect the screen on the extension line of the face coordinates and the virtual camera coordinates are calculated. As a result, the screen to be projected and the projector to be projected are determined.

In S105, the control data generation unit 502 generates display parameter information from the setting file and outputs the display parameter information together with the marker coordinates to the marker control unit 503.

As a result, the marker control unit 503 controls the display data output unit 504 to display the set type of marker at the set position.

Since the marker to be displayed is a symbol indicating the direction in which the virtual viewpoint exists for the subject, it may be an icon or the like representing the virtual viewpoint camera. In this case, the icon may be displayed in a size corresponding to the distance between the subject and the virtual viewpoint in the real space. As a result, the subject can grasp not only the direction in which the virtual viewpoint exists but also the distance to the virtual viewpoint.

Further, when there are a plurality of subjects, the marker coordinates are calculated for each subject and displayed with different settings (color and shape), so that the line of sight can be aligned with the same position. At this time, it can be realized by notifying each subject of the marker setting in advance. Further, if a plurality of subjects always look at the same position, the number of markers may be one.

[First Modification Example of First Embodiment]
In the above embodiment, the image processing system has been described as an independent device in which the real space information holding device 2, the image processing device 3, the user terminal 4, and the marker output device 5 are independent devices, but when the processing capacity is sufficient. May realize these with one information processing device and an application program. The configuration of the image processing system in this case is shown in FIG. The image processing system includes an information processing device 300, a plurality of cameras 350a and 350b as a photographing device 1, and a plurality of projectors 360a and 360b as a display data output unit 504. The information processing device 300 includes a CPU 301, a ROM 302, a RAM 303, an auxiliary storage device 304, a display device 305, an operation unit 306, a camera I / F 307, and a projector I / F. Here, the camera I / F307 and the projector I // F308 may be combined into one I / F. It is assumed that the auxiliary storage device 304 is capable of real space information including the OS. Further, the configuration related to shooting and projection surrounding the subject is the same as in FIG. 2A.

When the power of the apparatus is turned on, the CPU 301 loads the OS stored in the auxiliary storage device 304 into the RAM 303 according to the boot program stored in the ROM 302, and executes the OS. As a result, this device functions as a device that performs processing according to instructions from the user. Further, the CPU 301 loads the image processing program from the auxiliary storage device 304 into the RAM 303 and executes the image processing program, whereby the present device performs the virtual viewpoint image generation unit 301, the virtual camera path calculation unit 302, and the virtual camera information holding unit in FIG. Functions as 303, marker coordinate calculation unit 304, display image generation unit 305, image display unit 402, virtual camera path instruction unit 403, user information transmission unit 404, marker information transmission unit 405, control data generation unit 502, marker control unit 503. Will be done.

The processing procedure when the CPU 301 executes the image processing program will be described with reference to the flowchart of FIG. For the sake of simplicity, it is assumed that the information about the marker has already been set.

In S401, the CPU 301 inputs information related to the virtual viewpoint via the operation unit 306. This information includes the coordinates of the virtual viewpoint in the real space, the line-of-sight direction from the virtual viewpoint, and the angle of view.

In S402, the CPU 301 receives captured image data from the cameras 350a, 350b, ... Via the camera I / F307. Then, in S403, the CPU 301 refers to the real space information (particularly information such as the installation position and direction of the camera) stored in the auxiliary storage device 304 to generate a virtual viewpoint image. Then, in S404, the CPU 301 displays the generated virtual viewpoint image on the display device 305.

In S405, the CPU 301 calculates the intersection position of the line connecting the coordinates of the face of the subject and the coordinates of the virtual viewpoint and the screen surface in the real space as the marker display position. Then, in S406, the CPU 301 causes the projector whose projection target is the calculated marker display position to display the marker at the marker position via the projector I / F 308.

After that, the CPU 301 determines in S407 whether or not the user has instructed to end the application, and if not, returns the process to S401 and repeats the above process.

In this modification, the real space information holding device 2, the image processing device 3, the user terminal 4, and the marker output device 5 are integrated into the information processing device of FIG. 3, but some of them are independent devices. It doesn't matter.

Also, for example, when generating a virtual viewpoint image of a performer's movement in a movie, it is easier for the performer to perform if there is a sense of presence, so each projector displays not only the marker but also the surrounding scenery surrounding the performer. You may let me.

The marker presentation position may be determined regardless of the virtual viewpoint. In particular, the position of the marker may be any position as long as it directs the line of sight of the subject, and may be specified by the user, for example.
[Second variant of the first embodiment]
In the above embodiment, the marker is displayed by projecting the marker onto the screen from the projector, but the configuration of the display data output unit 504 is not limited to this. As shown in FIG. 12, a plurality of liquid crystal displays may be installed so as to be spread on the surfaces of the screens 23, 25, 26, and 28. In this case, in S104, the control data generation unit 502 determines the display 28a corresponding to the marker coordinates calculated in the world coordinate system, and determines the display coordinates on the display screen based on the position and orientation of the display 28a calibrated in advance. calculate. The calibration of the display can be performed by calculating the three-dimensional coordinates of the marker displayed on the display by triangulation using the cameras 21a to 21f and associating them with the coordinates of the marker on the display. Further, in S105, the marker control unit 503 controls the display data output unit 504 to display the marker 32 of the type set in the display coordinates on the corresponding display 28a. The screen color may be displayed inconspicuously on the screen other than the marker display coordinates on the display and the screen on the other display on which the marker is not projected, or the surrounding scenery surrounding the performer may be displayed.

[Third variant of the first embodiment]
In the first embodiment, the direction in which the virtual viewpoint exists is displayed by using a point marker, but a three-dimensional model of the virtual camera may be projected on the screen to further indicate the posture of the virtual camera. At this time, the three-dimensional model of the virtual camera is stored in the auxiliary storage device 214 in advance, and is expanded in the RAM 213 when the application program is executed. The type of marker is a virtual camera 3D model. FIG. 13A shows how a three-dimensional model of a virtual camera is projected. In S405, the CPU 301 virtually projects the three-dimensional model of the virtual camera from the coordinates 24f of the face of the subject in the real space toward the screen 23a, and calculates the three-dimensional coordinates of the image 32c. Then, by back-projecting the three-dimensional coordinates of the image 32c toward the projector 22d, an image of the marker to be displayed by the projector 22d is generated. The virtual camera three-dimensional model may have any shape, texture, and transmittance, and may be a three-dimensional arrow model instead of the camera.

As a result, the subject can recognize the posture of the virtual viewpoint in addition to the direction in which the virtual viewpoint exists.

[Fourth variant of the first embodiment]
In the above embodiment, the direction and posture in which the virtual viewpoint exists are displayed by using a point or a three-dimensional model, but a virtual viewpoint image may be displayed to show the angle of view of the virtual viewpoint. At this time, the display image generation unit 305 transmits the virtual viewpoint image generated immediately before to the projector or the display. The type of marker is a virtual viewpoint image. FIG. 13B shows how a virtual viewpoint image is projected. In S104, the control data generation unit 502 calculates the marker coordinates on the screen or the display, and performs a projective conversion on the virtual viewpoint image so that the marker coordinates are at the center and the virtual viewpoint image can be seen directly from the subject. .. The projective transformation uses an affine transformation or a homography transformation. In S105, the marker control unit 503 controls the display data output unit 504 to project the projected virtual viewpoint image 32d onto the display coordinates on the corresponding screen or display.

As a result, the subject can recognize the direction in which the virtual viewpoint exists, the angle of view at the virtual viewpoint, and the image quality from the virtual viewpoint by looking at the virtual viewpoint image. If necessary, the user may change the display setting of the marker to flip the virtual viewpoint image left and right, and display the virtual viewpoint image so that the subject itself can be seen in a mirror image.

[Fifth variant of the first embodiment]
When the number of virtual viewpoints is very large, such as 10 or more, and there are a plurality of virtual viewpoints moving at high speed, displaying the marker as it is may look annoying to the subject. In this modification, virtual viewpoints that are close to each other are displayed together, and markers of virtual viewpoints that move at high speed are displayed so as to be difficult to see.

The virtual camera information holding unit 303 transmits a plurality of virtual camera paths in the same time code to the marker coordinate calculation unit 304. FIG. 14 shows a flow of changing the marker display in the marker coordinate calculation unit 304. Acquire a plurality of coordinates and velocities as virtual camera information in S201. In S202, the distance threshold value (radius 2 m) and the quantity threshold value (= 10) described in the marker display setting are acquired, and there are 10 or more of the received virtual cameras within 2 m. To determine. If No, the process proceeds to S204. In the case of Yes, the center of gravity of a set of virtual cameras within 2 m from each other is determined as a representative point in S203, and the marker is displayed only for the representative point. At this time, the color or marker type may be changed to indicate that the point is a representative point, or the number of elements of the set may be displayed as text. In S204, 2 to 10 m / s based on the minimum speed threshold value (2 m / s), the maximum speed threshold value (10 m / s), and the maximum transmission amount (90%) described in the marker display setting. Determined by linearly changing the transmission amount of the speed virtual camera. The transmittance of 0 to 2 m / s is 0%, and the transmittance of 10 m / s or more is 90%. The method for determining the amount of transmission is not limited to this. Further, not limited to transparency, the marker display may be changed so that the faster the speed of the virtual camera, the harder it is to see by changing the brightness, saturation, and the like.

As a result, the subject can concentrate on acting even when the number of virtual viewpoints is large.

Further, as an effect of this modification, when there is a delay from the shooting of the subject to the display of the marker, the displacement of the position due to the delay of the virtual camera that moves quickly is made difficult to be visually recognized by the transparency of the marker. A virtual camera that does not move quickly has relatively little displacement due to delay, and can be easily viewed without being transmitted.

[Second Embodiment]
In the first embodiment and its modification described above, the position of the marker visually recognized by the subject is configured so that the line of sight (line of sight) matches the position of the virtual camera among the virtual camera parameters. .. The content expressed by the marker is not limited to the above.

In the second embodiment, although the marker to be displayed is related to the virtual camera parameter, it indicates the position where the subject wants to align the line of sight (the position where the subject's line of sight is desired to be directed) instead of aligning the line of sight with the virtual camera. With such a configuration, it is possible to generate a scene of the line of sight of the subject toward the marker as an image captured by a virtual camera.

FIG. 5A is a diagram for explaining the present embodiment. As described above, the marker 51 in the figure indicates a position where the subject wants to align his / her line of sight (a position where he / she wants to align his / her line of sight with the subject). The type of marker of this embodiment is different from that of the first embodiment. Therefore, the marker type is switched by the marker type parameter information transmitted by the marker information transmission unit 405 of the user terminal 4.

Reference numerals 52a and 52b in the figure are virtual cameras, and although the subject 24 is not actually visible, it is possible to generate an image obtained by capturing an image in which the line of sight is aligned with the marker 51 by the virtual camera.

FIG. 5B is a diagram schematically showing the positional relationship between the virtual camera and the marker. The positional relationship between the virtual camera and the marker in the present embodiment is defined in advance in the marker information transmission unit 405 as a setting file. In the setting file, for example, the virtual camera path 55 and the corresponding marker 53 are defined as an angle 57 formed by an extension line with the face of the subject. In this way, it is possible to create a free-viewpoint image by using an image of a subject whose line of sight is directed at a certain target as an image from various angles.

[Third Embodiment]
In the first embodiment and the modification, and the second embodiment, the marker is configured to be visible to the subject by projecting it on the screen with a projector. However, when generating a free-viewpoint image in a temporary studio or a shooting location, it may not always be possible to configure the screen. Therefore, in the present embodiment, an indicator is prepared on the mounting member on which the camera is mounted, and the position of the virtual camera is recognized by the subject by the display method of the indicator.

The third embodiment will be described with reference to FIG. The cameras 61a to 61c are mounted on the mounting members 62a to 62c arranged so as to surround the subject in the real space. Indicators are provided on these mounting members, and the lighting position, blinking speed, display color, etc. in the height direction are controlled by the marker control unit 503. Reference numeral 64 in the figure indicates the position of the virtual camera and cannot be directly seen by the subject. However, the subject can recognize the approximate position of the virtual camera by displaying the indicator. For example, when the indicators 63a and 63b are lit now, it can be recognized that the position of the virtual camera is between the actual cameras 61a and 61b and is at the height of the approximate indicator. Further, when the indicator 63a blinks, it can be recognized that the position of the virtual camera is on the side closer to the actual camera 61a.

In this way, by configuring the indicator on the mounting member of the actual camera, the approximate virtual camera position can be configured so that the subject can recognize it.

When the third embodiment is adopted, the marker output device 5 in FIG. 1 controls the display of the indicator.

[Fourth Embodiment]
In each of the above-described embodiments (including modified examples), the marker presentation is represented by a marker that displays the position of the virtual camera two-dimensionally in the real space. In this embodiment, a configuration is described in which the position itself of the virtual camera is three-dimensionally presented to the subject so as to be visible.

As the marker in the fourth embodiment, for example, a flying device capable of self-sustaining flight having a hovering function such as a drone is used. As the marker of the present embodiment, any marker may be used as long as it has a power, can fly, and has a configuration in which the coordinate values can be controlled. That is, the marker of the present embodiment receives the virtual camera position coordinates output by the marker control unit 503 as the world coordinate values in the real space and flies to the position. There may be a plurality of markers.

As shown in FIG. 7, for example, the marker has a configuration in which the flying object 71 has a mark (in the figure, an arrow display 72 and a side marking 73) so that the line-of-sight direction of the virtual camera can be understood.

With this configuration, the subject can recognize the position of the virtual camera. When there are a plurality of markers and a free viewpoint image is generated, these markers may be included in the angle of view of the virtual camera. However, since the position of the virtual camera and the shape of the marker are known in advance, it is possible to erase the marker and generate a free viewpoint image as an image. At this time, the coordinates of the marker are projected on each of the cameras 21 and the projected coordinates on the camera image are excluded from the model generation and rendering processing to prevent the image of the marker from affecting the model generation and rendering. In addition, the target area at the position may be erased by smoothing it with the same texture as the surroundings and performing a composite process, or by displaying a marker in the same color as the background in advance.

[Variation example of the fourth embodiment]
Although the drone is used in the above-mentioned fourth embodiment, the position and posture of the virtual camera may be displayed by using another device that can display three-dimensionally. For example, there are aerial displays that emit air by exciting plasma, anaglyphs, polarized shutter type 3D displays, and eyeglass-type or contact lens-type devices such as holo lenses that can display AR. In each case, the virtual camera is displayed three-dimensionally on the subject by projecting a three-dimensional model of the marker on the coordinates of the left and right eyes of the subject in the virtual space and generating an image to be displayed on each device. Can be done. Further, when there are a plurality of virtual cameras, the marker displayed for each device may be controlled to make it easier to recognize a specific virtual camera for each subject.

As a result, the subject can visually recognize the position and posture of the virtual camera three-dimensionally while avoiding the danger of contacting the drone.

[Fifth Embodiment]
In the embodiments described above, the marker indicates information related to the line of sight of the subject. When the subject plays in real space, it may not always be necessary for the line of sight to match something. In that case, when the subject stands at a position in the real space and the performance is performed, the subject may want to recognize the standing position. In the fifth embodiment, the display of the marker is projected on the floor surface. The subject can recognize the standing position linked to the virtual camera based on the content displayed on the floor surface.

The configuration of this embodiment is shown in FIG. The projector 83 is attached to a free platform 84, and the projection position can be controlled by a pan angle, a tilt angle, and a zoom magnification. Further, as the display content 86, a standing position, a moving direction, and the like are displayed. Further, the display position and display contents of the marker are controlled by the virtual camera position and the marker position by the setting file defined by the marker information transmission unit 405. With such a configuration, it is possible to generate, for example, a free-viewpoint image in which a virtual camera captures a movement from one standing position to another.

In the embodiments described so far, the subject has been configured to visually recognize the position of the marker at the time of shooting, but the configuration is not necessarily limited. For example, by superimposing the marker position on the archive video as described above, the display is not limited to the configuration used only at the time of shooting, and the viewer can also see the relationship between the virtual camera and the line of sight. good.

In order to realize the above, it can be realized by preparing multiple streams for each virtual viewpoint. Furthermore, by preparing an image that gives a bird's-eye view of the real space and making the position of the marker interactable, the user can switch the camera position and refer to the free viewpoint image.

In addition, images taken by multiple cameras are retained as archived images, and when the operator later generates free-viewpoint images, the face position (line of sight) of the subject is detected and the position on the extension line in the virtual space is reached. Markers may be displayed. With such a configuration, the operability of the operator is improved when determining the angle of view of the free viewpoint image.

Further, since the markers in the embodiments described so far are naturally reflected in the images taken by each camera, they may be an obstacle in the generated free-viewpoint image. However, since the display position, color, shape, etc. of the marker are known in the virtual space, it can be erased by smoothing the target area of the marker with the same texture as the surroundings and performing a composite process. Further, it can be easily solved as described above by displaying the marker in the same color as the background in advance.

(Other examples)
The present disclosure supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiment, and various modifications and modifications can be made without departing from the spirit and scope of the invention.

1 ... Multi-viewpoint image holding unit, 2 ... Subject information holding unit, 3 ... Image processing device, 301 ... Virtual viewpoint image generation unit, 302 ... Virtual camera path calculation unit, 303 ... Virtual camera information holding unit, 304 ... Marker coordinate calculation Unit, 305 ... Display image generation unit, 306 ... Device communication unit, 4 ... User terminal, 401 ... Terminal communication unit, 402 ... Image display unit, 403 ... Virtual camera path indicator unit, 404 ... User information transmission unit, 405 ... Marker Information transmission unit, 5 ... Marker output device, 501 ... Device communication unit, 502 ... Control data generation unit, 503 ... Marker control unit, 504 ... Display data output unit

Claims (19)

It is a processing system for generating a virtual viewpoint image that represents the appearance from a virtual viewpoint.
Multiple shooting methods for shooting the space including the subject,
A processing system comprising a presentation means for presenting to determine the orientation of the subject in the space. Specific means for specifying the position of the virtual viewpoint and
The processing system according to claim 1, wherein the presenting means makes a presentation for determining the direction of the subject in the space based on the position of the virtual viewpoint specified by the specific means. .. The plurality of photographing means and a screen for projection are arranged so as to surround the subject.
The processing system according to claim 1 or 2, wherein the presentation means projects information for determining the orientation of the subject on the screen. The third aspect of the present invention is characterized in that the presenting means projects information for determining the orientation of the subject at a position of an intersection of an extension line connecting the subject and the virtual viewpoint and the screen. Processing system. The presenting means projects information for determining the orientation of the subject at the position of the intersection of the screen and the line deviated in a predetermined direction with respect to the extension line connecting the subject and the virtual viewpoint. The processing system according to claim 3. The presenting means according to any one of claims 1 to 5, wherein the presenting means presents information for determining the orientation of the subject in a size corresponding to the distance between the subject and the virtual viewpoint. Processing system. The plurality of photographing means are attached to a mounting member having an indicator that lights a position set with respect to a position in the height direction, and are arranged so as to surround the subject.
The processing system according to claim 1 or 2, wherein the presentation means lights a position at which the subject's line of sight is directed. The processing system according to claim 1 or 2, wherein the presentation means is a means for locating the flight device at a position where the line of sight of the subject is directed. The processing system according to claim 8, wherein the flying device is provided with a mark indicating a line-of-sight direction of a virtual viewpoint. The presenting means is a means for controlling the pan and tilt angles of a projector mounted on a mounting member capable of controlling the pan angle and the tilt angle, and the subject is placed on a floor corresponding to a position where the subject's line of sight is directed. The processing system according to claim 1 or 2, wherein the processing system is characterized by projecting information for determining the orientation of the. The processing system according to any one of claims 1 to 10, wherein the presenting means presents information for determining the orientation of the subject at the position of the virtual viewpoint. Claims 1 to 10 are characterized in that the presenting means presents information for determining the direction of the subject at a position different from the position of the virtual viewpoint and at a position in a direction in which the subject is directed. The processing system according to any one of the above. The processing system according to any one of claims 1 to 12, wherein the orientation of the subject is the orientation of the face of the person who is the subject. The processing system according to any one of claims 1 to 12, wherein the orientation of the subject is the orientation of the line of sight of the person who is the subject. A user terminal having a designation means for designating the virtual viewpoint and a display means for displaying the virtual viewpoint image, and
When the virtual viewpoint image is generated, the image processing device and
The processing system according to any one of claims 1 to 14, further comprising a presentation device having the presentation means. The processing system according to any one of claims 1 to 15, wherein the presentation means presents a virtual viewpoint image. The processing system according to any one of claims 1 to 16, wherein the presenting means controls a color or a shape according to the number or speed of the virtual viewpoints. It is a control method of a processing system for generating a virtual viewpoint image that represents the appearance from a virtual viewpoint.
A step of photographing a space including a subject by a plurality of photographing means, a presenting step of presenting for determining the orientation of the subject in the space, and a presentation step.
A control method characterized by having. A program for causing a computer to perform each step of the method according to claim 18.

Images