JP2023018253A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To generate a virtual viewpoint image so that an object is represented in a desired display mode.SOLUTION: A virtual viewpoint image generation apparatus 400 comprises: a display information setting unit 402 for setting a display mode of an object in a virtual viewpoint image generated based on imaged images obtained by imaging the object with a plurality of imaging apparatuses 100 based on at least the position of the object and the gesture of the object; and a virtual viewpoint image generation unit 404 for generating the virtual viewpoint image so that the object is represented in the set display mode.SELECTED DRAWING: Figure 4

Description

The present disclosure relates to technology for generating a virtual viewpoint image.

2. Description of the Related Art There is a method of generating a virtual viewpoint image from an arbitrary viewpoint by using a plurality of captured images obtained by imaging an object from a plurality of viewpoints by installing a plurality of imaging devices at different positions.

Patent Literature 1 discloses a method of arranging a plurality of cameras so as to surround an object and generating a virtual viewpoint image corresponding to an arbitrary viewpoint using an image obtained by imaging the object.

JP 2008-15756 A

Using the method of Patent Literature 1, for example, it is conceivable to generate a virtual viewpoint image in which the performer in the studio is viewed from a virtual viewpoint based on a captured image obtained by capturing the studio. In this case, since there are objects other than the performer, such as staff and cameramen, in the studio, the objects other than the performer may appear in the virtual viewpoint image. If an object different from the performer appears in the virtual viewpoint image, the user may not be able to concentrate on the performer and view the virtual viewpoint image. On the other hand, depending on the performance, it may be desirable to show an object other than the performer, such as a staff member or a cameraman, in the virtual viewpoint image.

The technology of the present disclosure has been made in view of this problem, and aims to generate a virtual viewpoint image such that an object is represented in a desired display mode.

The image processing apparatus according to the present disclosure determines the display mode of the object in the virtual viewpoint image generated based on the captured images obtained by imaging the object with a plurality of imaging devices, at least from the position of the object and the gesture of the object. and a generating means for generating the virtual viewpoint image so that the object is displayed in the set display mode.

The technology of the present disclosure can generate a virtual viewpoint image such that an object is represented in a desired display mode.

1 is a configuration diagram of an image processing system; FIG. FIG. 2 is a diagram for explaining the arrangement of imaging devices in a studio; FIG. 2 is a hardware configuration diagram of a virtual viewpoint image generation device; FIG. 2 is a block diagram for explaining the functional configuration of a virtual viewpoint image generation device; 4 is a flowchart for explaining rendering processing; 4 is a flowchart for explaining a process of setting display information of an object; FIG. 5 is a diagram showing an example of a method for setting display information of an object; FIG. 5 is a diagram showing an example of a method for setting display information of an object; FIG. 5 is a diagram showing an example of a method for setting display information of an object; FIG. 5 is a diagram showing an example of a method for setting display information of an object; 4A and 4B are diagrams for explaining a method of displaying an object that straddles the boundary of the rendering area; FIG.

Hereinafter, details of the technology of the present disclosure will be described based on embodiments with reference to the accompanying drawings.

<Embodiment 1>
FIG. 1 is a diagram illustrating an example of an image processing system that generates a virtual viewpoint image. By installing multiple imaging devices at different positions, images are taken in synchronization from multiple viewpoints, and the multiple images obtained by the imaging are used to view from a virtual viewpoint that does not depend on the viewpoint from the actual imaging device. There is a method of generating an image (virtual viewpoint image). This virtual viewpoint image is generated by consolidating images obtained by imaging with a plurality of imaging devices into an image processing device such as a server, and executing processing such as rendering based on the virtual viewpoint in the image processing device. Realized. Further, by displaying the virtual viewpoint image on a display unit of a terminal or the like, the viewer can view the virtual viewpoint image. Note that the virtual viewpoint image may be a moving image or a still image. In the following embodiments, the virtual viewpoint image will be described as a moving image.

In the description of the present embodiment, the virtual viewpoint may be replaced with a virtual camera (virtual camera). At this time, the position of the virtual viewpoint corresponds to the position of the virtual camera, and the line-of-sight direction from the virtual viewpoint corresponds to the orientation direction of the virtual camera. Also, the virtual viewpoint image corresponds to an image within the angle of view of the virtual camera. The position and orientation of the virtual camera can be specified by the operator of the virtual camera. Therefore, it is possible to generate an image from an arbitrary viewpoint.

The image processing system of this embodiment includes a plurality of imaging devices 100 , foreground extraction devices 150 , 3D model generation devices 200 , storage devices 300 , virtual viewpoint image generation devices 400 , and virtual viewpoint path generation devices 500 . It also has a virtual viewpoint image display device 600 and a three-dimensional model selection device 700 .

The imaging device 100 is a device such as a digital video camera that captures images (moving images). A plurality of image capturing apparatuses 100 are arranged, and all the image capturing apparatuses 100 perform image capturing at the same time, for example, by receiving GPS or an external synchronization signal. The imaging device 100 may be a device that captures sound and other sensor information in addition to images.

FIG. 2 is a diagram for explaining the arrangement of the imaging device 100. As shown in FIG. The studio 10 is a facility in which the imaging device 100 performs imaging in order to obtain a captured image for generating a virtual viewpoint image. In this embodiment, the image processing system including the imaging device 100 is constructed in a studio, but the location for constructing the image processing system is not limited to the studio, and may be a stadium, an arena, or the like.

It is assumed that the studio 10 has a performer 80 and a kuroko (kuromo) 51 as objects. A performer 80 is an object rendered in the virtual viewpoint image. An object such as the performer 80 is not limited to a person, and may be an animal or an immobile object. Kuroko 51 is a person such as a cameraman or staff in the studio. The mole 51 will be described as an object that is set to be displayed or not to be displayed in the virtual viewpoint image. Objects such as the mole 51 are not limited to humans, and may be animals or non-moving objects.

A plurality of imaging devices 100 according to the present embodiment are installed in the studio 10 and arranged so as to mainly capture images of the rendering area 20 . FIG. 2 shows an example in which 14 imaging devices 100 are installed, but the number of imaging devices 100 is not limited to 14 units.

The rendering area 20 is a space within the studio 10 in which a virtual viewpoint image can be generated, and is also referred to as a drawing space. In the case of FIG. 2, the performer 80 present in the rendering area 20 can be displayed within the virtual viewpoint image. On the other hand, moles 51 outside the rendering area 20 are not displayed in the virtual viewpoint image in the case of FIG.

The tracking area 30 is an area where the image processing system can track the performer 80 and the black child 51, which are objects existing in the studio 10, and obtain their respective positions. Object positions outside the tracking area 30 are not tracked.

In this embodiment, the sizes of the studio 10, the tracking area 30, and the rendering area 20 are illustrated as studio 10>tracking area 30>rendering area 20 in FIG. Alternatively, the size of studio 10, tracking area 30, and rendering area 20 may be the same.

Returning to FIG. 1, the description of the system configuration is continued. The foreground extraction device 150 is a device that acquires a captured image obtained by imaging by the imaging device 100 and a time code that is time information when the captured image was captured, and performs image processing on the acquired captured image. be. The foreground extraction device 150 executes a process of extracting an area representing an object from the acquired captured image as a foreground area. Specifically, a silhouette image is generated in which a foreground area and a non-foreground area included in the captured image are indicated in binary. It also generates a foreground texture image. A foreground silhouette image and a texture image are sometimes collectively referred to as foreground data. Foreground refers to the image area in which the object exists in the captured image. A user or the like can specify in advance what kind of object should be extracted as the foreground.

As a method for extracting the foreground from the captured image, for example, there is a method using background difference information. In this method, a captured image obtained by capturing a studio in which no object exists is stored in advance as a background image. Then, an area in which the difference value between the pixel values of the captured image and the background image is larger than a threshold value is determined as the foreground. Note that the method for extracting the foreground is not limited to the method using background difference information. Alternatively, a method using parallax, a method using feature amount, a method using machine learning, or the like may be used. The generated foreground data is output to the 3D model generation device 200 .

In this embodiment, the foreground extraction device 150 and the imaging device 100 are described as being different devices, but the foreground extraction device 150 and the imaging device 100 may be an integrated device. Alternatively, each functional block may be provided in another device.

The three-dimensional model generation device 200 is a device implemented by a computer such as a PC, workstation, or server. The 3D model generation device 200 acquires the foreground data of the time code to be processed and the camera parameters of the imaging device 100 from the foreground extraction device 150, and obtains the 3D model representing the 3D shape of the object inside the rendering area 20. to generate Three-dimensional model data is also called three-dimensional shape data.

The camera parameters include parameters representing the three-dimensional position, orientation (orientation of the imaging device), focal length, and principal point (center on the captured image) of each imaging device 100 on the world coordinates. The camera parameters are measured and calculated by taking an image of a calibration pattern for calibrating the position of the imaging device 100 in advance, and performing coordinate conversion between the three-dimensional coordinates defined on the basis of the calibration pattern and the two-dimensional camera coordinates. It is assumed that there is

The 3D model generation device 200 will be described as generating a 3D model using VisualHull, which is a method for generating a 3D model of an object. VisualHull obtains a cone extending in three-dimensional space so as to pass each point on the contour of the foreground area from the optical principal point position of the imaging device, and extracts the common portion of the cone area corresponding to each imaging device in three dimensions. It is a method of modeling. Examples of calculation algorithms include VIM (Volume intersection method) and SCM (Space carving method). In SCM, a three-dimensional space is filled with tiny cubes, and each individual cube in the three-dimensional space is back-projected onto the plane of each imaging device 100 . Then, a three-dimensional model is generated by leaving cubes back-projected inside the foreground silhouettes of all imaging devices 100 as the foreground and deleting other cubes. Individual cubes in three-dimensional space are generally called voxels, and the foreground in VisualHull is called a silhouette.

The 3D shape data, which is the data of the generated 3D model, is output to the storage device 300 as a point cloud format, which is a set of voxels in a 3D space, for example. In this embodiment, it is assumed that the 3D model is output as a point group format, but the format of the 3D model is not limited, and the 3D model can also be output in a different format such as a mesh format. may be A three-dimensional model is output in association with the given identifier and the positional information.

The position information may be position information acquired using a wireless tag such as RFID or GPS attached to the performer 80 or the black child 51, or may be coordinate information of a three-dimensional position on the world coordinates in the tracking area 30. Positional information associated with the three-dimensional model is referred to as positional information of the three-dimensional model. Since the positional information of the 3D model (object) is tracked in this way, the 3D model generation device 200 can assign the same identifier to the 3D model of the same object.

Then, the three-dimensional model generating device 200 generates time codes corresponding to the plurality of captured images used to generate the three-dimensional model, silhouette images and texture images (foreground data) corresponding to each of the plurality of imaging devices 100, to output Furthermore, the three-dimensional model to which the identifier is assigned (three-dimensional shape data) and positional information (positional information of the three-dimensional model) associated with the three-dimensional model and the identifier are output to the storage device 300 .

The storage device 300 is a device such as a computer such as a PC, workstation, or server. Stores the 3D model, which is point group format data generated by the 3D model generation device 200, the identifier of each 3D model, the position information of the 3D model, the time code, and the foreground data generated by the foreground extraction device 150. have a function.

In this embodiment, the storage unit 301 (see FIG. 4) of the storage device 300 stores the data output from the three-dimensional model generation device 200. FIG. Alternatively, the 3D model generation device 200 may directly output data to the virtual viewpoint image generation device 400 . In this case, the image processing system may be configured without the storage device 300 .

The virtual viewpoint path generation device 500 is a device realized by a computer such as a PC, workstation, server, or the like. The virtual viewpoint path generation device 500 is a device that generates virtual viewpoint path information, which is virtual camera information for generating a virtual viewpoint image. The virtual viewpoint path information includes data such as the position, orientation, and angle of view of the virtual camera for each frame, the movement speed of the virtual camera, and the playback speed. The virtual viewpoint path generation device 500 outputs the generated virtual viewpoint path information to the virtual viewpoint image generation device 400 .

The three-dimensional model selection device 700 is a computer such as a PC, workstation, or server, and is a device having a UI. Based on the user's instruction, the display information (also referred to as display form information) of the object in the tracking area 30 is set. Display information will be described later.

The virtual viewpoint image generation device 400 is a device implemented by a computer such as a PC, workstation, or server. The three-dimensional model, the silhouette image, and the texture image (foreground data), which are point cloud format data stored in the storage device 300, the virtual viewpoint path information generated by the virtual viewpoint path generation device 500, and the specified time code. get. Then, the virtual viewpoint image generation device 400 generates a virtual viewpoint image by executing rendering processing or the like on the three-dimensional model in the point group format inside the rendering area 20 based on the acquired data. Details will be described later.

The virtual viewpoint image display device 600 is implemented by a liquid crystal monitor, a projector, or the like, and is a monitor that displays a virtual viewpoint image.

The 3D model generation device 200, the foreground extraction device 150, the storage device 300, the virtual viewpoint image generation device 400, the virtual viewpoint path generation device 500, and the 3D model selection device 700 are connected via a LAN network. Although the network topology of this embodiment is daisy chain connection, the network topology is not limited to daisy chain connection. Alternatively, a star, bus, or mesh type connection via a relay device such as a Hub may be used. Moreover, the connection method is not limited to the LAN network, but may be via another wired connection, wireless LAN, public radio, or the Internet. Also, SDI (Serial Digital Interface) or DVI (Digital Visual Interface), which is an interface for image transmission, may be used.

[Hardware configuration]
FIG. 3 is a diagram showing the hardware configuration of the virtual viewpoint image generation device 400. As shown in FIG. The hardware configuration of the foreground extraction device 150, the 3D model generation device 200, the virtual viewpoint path generation device 500, and the 3D model selection device 700 is also the same as the configuration of the virtual viewpoint image generation device 400 described below. .

The virtual viewpoint image generation device 400 has a CPU 311 , a ROM 312 , a RAM 313 , an auxiliary storage device 314 , a display section 315 , an operation section 316 , a communication I/F 317 and a bus 318 .

The CPU 311 controls the entire virtual viewpoint image generation device 400 using computer programs and data stored in the ROM 312 and the RAM 313, thereby realizing each function of the virtual viewpoint image generation device 400 shown in FIG. . Note that the virtual viewpoint image generation device 400 may have one or a plurality of dedicated hardware different from the CPU 311 , and at least part of the processing by the CPU 311 may be executed by the dedicated hardware. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors).

ROM 312 stores programs that do not require modification. The RAM 313 temporarily stores programs and data supplied from the auxiliary storage device 314, data supplied from the outside via the communication I/F 317, and the like. The auxiliary storage device 314 is composed of, for example, a hard disk drive or the like, and stores various data such as image data and audio data.

The display unit 315 is composed of, for example, a liquid crystal display, an LED, and the like, and displays a GUI (Graphical User Interface) for the user to operate the virtual viewpoint image generation device 400 and the like. An operation unit 316 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, and the like, and inputs various instructions to the CPU 311 in response to user operations. The CPU 311 operates as a display control unit that controls the display unit 315 and as an operation control unit that controls the operation unit 316 . In this embodiment, the display unit 315 and the operation unit 316 are described as existing inside the virtual viewpoint image generation device 400 . may exist as a separate device of

A communication I/F 317 is used for communication with a device external to the virtual viewpoint image generation device 400 . For example, when the virtual viewpoint image generation device 400 is connected to an external device by wire, a communication cable is connected to the communication I/F 317 . If the virtual viewpoint image generation device 400 has a function of wirelessly communicating with an external device, the communication I/F 317 has an antenna. A bus 318 connects each unit of the virtual viewpoint image generation device 400 and transmits information.

[Function configuration]
FIG. 4 is a block diagram showing the functional configuration. Functions of the storage device 300 and the virtual viewpoint image generation device 400 will be described with reference to FIG.

The virtual viewpoint image generation device 400 has a tracking information acquisition unit 401 , a display information setting unit 402 , a display information management unit 403 and a virtual viewpoint image generation unit 404 .

A tracking information acquisition unit 401 acquires position information of a three-dimensional model stored in the storage unit 301 of the storage device 300 .

A display information setting unit 402 sets display information based on the positional information of the three-dimensional model. A method of setting display information will be described later. The display mode of the object in the virtual viewpoint image that can be set in this embodiment is normal rendering display or transparent display. Note that the display mode that can be set is not limited to normal rendering display or transparent display. In addition, avatar display, semi-transparent display, mosaic display, and silhouette display may be included instead of the object. Also, annotation display may be settable for objects.

The display information management unit 403 registers display information of each three-dimensional model (object). Specifically, by storing the display information of the three-dimensional model (object) set by the display information setting unit 402 or the display information of the three-dimensional model set by the user in association with the three-dimensional model, each Each display information is associated with an object and managed. Also, the display information management unit 403 manages the time code when the display information is changed. The display information management unit 403 may manage the change history of display information.

The virtual viewpoint image generation unit 404 generates a virtual viewpoint image based on the position of the virtual viewpoint received from the virtual viewpoint path generation device 500 and the angle of view corresponding to the virtual viewpoint. Specifically, in the display mode of the display information associated with the objects managed by the display information management unit 403, each object is displayed inside the rendering area 20 on the virtual viewpoint image. Work on the 3D model of the object. Details will be described using a flowchart.

Each of the functional units described above is realized by executing a predetermined program by the CPU 311 of the virtual viewpoint image generation device 400, but is not limited to this. In addition, for example, hardware such as GPU (Graphics Processing Unit) or FPGA (Field Programmable Gate Array) for speeding up calculation may be used. Each functional unit may be implemented by cooperation of software and hardware such as a dedicated IC, or a part or all of the functions may be implemented by hardware alone.

The storage device 300 has a storage unit 301 . The storage unit 301 has a function of associating and storing the three-dimensional model output by the three-dimensional model generation device 200 and the positional information of the three-dimensional model. The storage unit 301 is implemented by, for example, a volatile memory, hard disk, or SSD.

Note that the function of each unit in FIG. 4 is described as being implemented by each device described above, but the device that implements the function of each unit is not limited. For example, the 3D model generation device 200 may have all of the functions described above. Alternatively, the 3D model generation device 200 or the virtual viewpoint image generation device 400 may have the storage unit 301 . 400 may be connected.

[Flow of Virtual Viewpoint Image Generation Processing]
FIG. 5 is a flowchart for explaining an example of processing for generating a virtual viewpoint image. A series of processes shown in the flowchart of FIG. 5 are performed by the CPU 311 of the virtual viewpoint image generation device 400 developing the program code stored in the ROM 312 in the RAM 313 and executing the program code. Also, some or all of the functions of the steps in FIG. 5 may be realized by hardware such as ASIC and electronic circuits. Note that the symbol "S" in the description of each process means a step in the flowchart.

The virtual viewpoint image generation unit 404 executes the generation of the virtual viewpoint image for one frame by receiving a virtual viewpoint image generation instruction for each frame from the virtual viewpoint path generation device 500 . S501 to S504 are processing for generating a virtual viewpoint image for one frame. The virtual viewpoint path generation device 500 designates the time code in chronological order to generate and output time-series virtual viewpoint images, thereby displaying the virtual viewpoint images of the moving images on the virtual viewpoint image display device 600 . Note that the time code may be specified not only in chronological order, but also in reverse playback or in the past.

In S<b>501 , the virtual viewpoint image generation unit 404 acquires the virtual camera information (the position, orientation, and angle of view of the virtual camera) and the time code in the generation target frame from the virtual viewpoint path generation device 500 . For example, the virtual viewpoint image generation unit 404 acquires the virtual camera information of the frame for which the virtual viewpoint image is to be generated by acquiring the virtual viewpoint path information from the virtual viewpoint path generation device 500 .

In S<b>502 , the virtual viewpoint image generation unit 404 acquires data corresponding to the frame to be generated from the storage device 300 based on the virtual camera information acquired in S<b>501 . That is, the three-dimensional model in the point cloud format, the silhouette image, and the texture image of the object included in the angle of view of the virtual camera in the generation target frame are acquired.

In S<b>503 , the virtual viewpoint image generation unit 404 stores the display information corresponding to the time code of the frame to be generated, among the display information associated with the object included in the angle of view of the virtual camera, in the display information management unit 403 . Get from

In S504, the virtual viewpoint image generating unit 404 generates a virtual view of the generation target frame so that each object included in the field angle of the virtual camera of the generation target frame is represented in the display mode indicated by the display information acquired in S503. Generate a viewpoint image. Specifically, processing corresponding to the display information associated with each three-dimensional model is performed on the three-dimensional model acquired in S502.

In this embodiment, the virtual viewpoint image generation unit 404 performs normal rendering processing or transparency processing based on display information. If the display information of the object is normal rendering display, the virtual viewpoint image generating unit 404 executes normal rendering processing on the 3D model of the object. The normal rendering process is a method of coloring according to a captured image, and is a process of coloring by projecting a texture based on a texture image onto a three-dimensional model in a point group format.

When the display information of an object is transparent display, the virtual viewpoint image generation unit 404 executes transparent processing on the three-dimensional model of the object. Transparency processing is processing that does not project textures onto a point cloud format three-dimensional model of an object. When transparency processing is performed, the object on the virtual viewpoint image can be represented so that the three-dimensional model corresponding to the object is not displayed in the virtual viewpoint image.

The virtual viewpoint image generation unit 404 converts the processing result of S504 into an image format and outputs the image format to the virtual viewpoint image display device 600 . The image format to be output is DVI, but any format is acceptable. In addition to DVI, there are methods such as SDI and streaming output of moving image files on a network. It should be noted that although the image is output to the virtual viewpoint image display device 600 in this embodiment, it may be output to the storage device 300 .

In S505, the virtual viewpoint image generation unit 404 determines whether to generate the virtual viewpoint image of the next frame. If there is a next frame, the process returns to S501 to repeat the generation processing of the virtual viewpoint image for one frame.

[Display information setting process]
FIG. 6 is a flowchart for explaining display information setting processing executed by the virtual viewpoint image generation device 400 . A series of processes shown in the flowchart of FIG. 6 are performed by the CPU 311 of the virtual viewpoint image generation device 400 developing the program code stored in the ROM 312 in the RAM 313 and executing the program code. Also, some or all of the functions of the steps in FIG. 6 may be realized by hardware such as ASIC and electronic circuits.

The display information setting process shown in the flowchart of FIG. 6 will be described as being performed in parallel with the virtual viewpoint image generation process shown in the flowchart of FIG. For example, when generating a virtual viewpoint image for one frame, a three-dimensional model of the frame to be generated is acquired in the flowchart of FIG. On the other hand, in the process of the flowchart of FIG. 6, the position information of the 3D model in the frame to be generated is acquired as will be described later. Alternatively, the process of the flowchart of FIG. 6 may be performed before the virtual viewpoint image generation process shown in the flowchart of FIG.

In S<b>601 , the tracking information acquisition unit 401 retrieves each identifier for identifying the three-dimensional model (object) set by the three-dimensional model generation device 200 and the position information of the three-dimensional model associated with the identifier from the storage device 300 . get.

In S602, the display information setting unit 402 sets display information for each object based on the position information of the three-dimensional model of each object acquired in S601. A method of setting display information in this step will be described later.

In S603, when the managed display information is changed as a result of the process of S602, the display information management unit 403 updates the display information associated with the object to the changed display information. Specifically, the display information is managed in association with the object by associating it with the three-dimensional model of the object, so the display information associated with the three-dimensional model is updated. The display information management unit 403 also records and manages a time code as time information indicating the timing of changing the display information. Note that the processing of S603 is processed so as to be completed before the virtual viewpoint image generation unit 404 acquires the display information in S503 of the flowchart of FIG.

If the user has set the display information via the 3D model selection device 700, the user's setting may be given priority, or the setting executed by the display information setting unit 402 in this flowchart may be given priority. good too. For example, it is possible to set in advance which of the user's setting and the setting by the display information setting unit 402 has priority, and display information may be set according to the setting.

In S604, the display information setting unit 402 determines whether or not there is an instruction to generate a virtual viewpoint image of the next frame. If there is a next frame, the process returns to S601 to continue setting the display information of the object in the next frame.

FIG. 7 is a diagram for explaining the display information setting process in S602. An example of a method for setting display information by the display information setting unit 402 will be described with reference to FIG. In this embodiment, the position of the plane is represented by coordinates (X, Y). Also, the description will be made assuming that the height is represented by coordinates (Z) and the position in the three-dimensional space is represented by coordinates (X, Y, Z).

In the tracking area 30 of FIG. 7, the diagonal is a line connecting a point A with coordinates (X, Y)=(0, 0) and a point D with coordinates (X, Y)=(100, 50). is a rectangle with Also, the tracking area 30 is assumed to be a rectangular parallelepiped area (space) whose height is coordinate (Z)=(30).

In the rendering area 20 of FIG. 7, the diagonal is a line connecting a point E with coordinates (X, Y)=(10, 10) and a point H with coordinates (X, Y)=(90, 40). is a rectangle with The rendering area 20 is assumed to be a rectangular parallelepiped area (space) whose height is coordinate (Z)=(20).

A first set plane 720, which will be described later, is a partial area of the boundary plane between the inside and outside of the rendering area 20, and is a point X at coordinates (20, 10, 10) and a point X at coordinates (30, 10, 0). It is a rectangular area whose diagonal is a line connecting the point Y of . Of the boundary surfaces between the inside and outside of the rendering area 20, the boundary surface other than the area of the first setting surface 720 is called a second setting surface. The size of each area shown in FIG. 7 is an example, and the size of each area is not limited to the size shown in FIG. Also, although the first setting surface 720 is rectangular, it is not limited to being rectangular. Triangles, polygons, circles, or arbitrary faces are acceptable.

In this embodiment, the display information setting unit 402 determines the boundary surface when the object enters the rendering area 20 based on the position information of the three-dimensional model (object), and displays the image according to the position of the boundary surface. Set the mode to the display information.

When an object enters the rendering area 20 via the first setting plane 720, the display mode is set so that the display information associated with the object is displayed in the virtual viewpoint image by normal rendering processing. That is, the normal rendering display is set as the display information of the object.

When an object enters the rendering area 20 via a second setting plane (boundary plane other than the first setting plane 720), display information corresponding to the object is displayed in the virtual viewpoint image by transparency processing. set. That is, transparent display is set as the display information of the object.

For example, in FIG. 7, mole 701 enters rendering area 20 from tracking area 30 through first set plane 720 . Therefore, the display information setting unit 402 sets display information associated with the mole 701 so that normal rendering processing is performed. On the other hand, the mole 702 enters the rendering area 20 from the tracking area 30 through the second setting plane (boundary plane other than the first setting plane 720). Therefore, the display information setting unit 402 sets display information associated with the mole 702 so that transparency processing is performed.

The size and position of the first setting surface 720 and the second setting surface are not limited to the positions and sizes shown in FIG. The first setting plane and the second setting plane may be set in an arbitrary area of the boundary plane between the rendering area 20 and the tracking area 30, or a plurality of first setting planes may be set. Also, the first setting plane 720 does not have to be a part of the boundary plane between the inside and outside of the rendering area 20 . For example, a first set plane may be defined within the rendering area 20 .

A method for determining whether an object has entered the rendering area 20 via the first setting plane 720 is, for example, only when the entire three-dimensional model of the object passes through the first setting plane 720. It is determined that the rendering area 20 has been entered via 720 . Alternatively, when a part of the three-dimensional model passes through the first setting plane 720 , it may be determined that it has entered the rendering area 20 through the first setting plane 720 . For example, when a preset percentage (for example, 50% or more) of the three-dimensional model passes through the first setting plane 720, it may be determined that the rendering area 20 has entered the rendering area 20 via the first setting plane 720.

Alternatively, transparent display may be set as the display information of the object passing through the first setting plane 720, and normal rendering display may be set as the display information of the object passing through the second setting plane.

Further, when a certain object passes through the first setting plane 720, the currently set display mode may be changed to another display mode as the display information of the object. In this case, the display information setting unit 402 does not have to change the display information of an object when the object passes through the second setting surface. For example, if normal rendering display is set in the display information of an object before passing through the first setting plane 720, the display mode is changed to transparent display after passing through the first setting plane 720, and the display information is displayed. May be set.

Further, the display information setting unit 402 may set the display information of the object that has left the rendering area 20 based on the boundary surface that the object passed through when leaving the tracking area 30 . In that case, the first setting plane and the second setting plane when leaving the rendering area 20 and the first setting plane and the second setting plane when entering the rendering area 20 are different positions and It can be size.

FIG. 8 is a diagram for setting display information associated with a newly appearing object without going through the boundary surface of the rendering area 20. In FIG.

It is assumed that transparent display is set for the display information of the mole 802 . In addition, it is assumed that Kuroko 802 has an accessory 801 such as a plastic bottle in his hand. In this case, a three-dimensional model may be generated for the small item 801 as part of the mole 802 , and the small item 801 is displayed as a part of the mole 802 in a transparent manner in the same manner as the mole 802 .

When Kuroko 802 drops the small item 801 on the floor, the small item 801 is recognized as a three-dimensional model of a new object. A small object 801 that has fallen to the floor is represented as a small object 801a. An identifier is assigned to the three-dimensional model of the small article 801a, and display information is set and associated with the three-dimensional model. In this way, a new object may appear in the rendering area 20 without intervening a boundary surface, and a three-dimensional model may be generated. In this case, the display information associated with the 3D model (object) from which the 3D model is separated is taken over, and the display information of the new 3D model is set. In the example of FIG. 8, the transparent display set as the display information of the mole 802 is set as the display information associated with the three-dimensional model of the small item 801a.

On the other hand, it is assumed that Kuroko 802 puts the small article 801 on the table 800 which is another object. An accessory 801 placed on the table 800 is referred to as an accessory 801b. In this case, the accessory 801b may be a part of the table 800, and a three-dimensional model may be generated integrally with the table 800. FIG. When an object is separated and becomes a part of another object to generate a three-dimensional model, the display information may be set so as to take over the display information associated with the three-dimensional model to which it is combined.

Assuming that normal rendering display is set as the display information of the table 800, the accessory 801b coupled to the table 800 will be rendered as normal rendering display. Furthermore, when the small article 801b falls from the table 800, the small article 801b separated from the table 800 is represented as the small article 801c. Therefore, normal rendering display is set in the display information of the small article 801c.

Note that weighting may be set for each three-dimensional model. Then, when a plurality of 3D models whose display modes are set in the display information are combined, the display mode of which object is to be inherited is determined by weighting, and the display information of the object is set. good.

Note that the method of setting the display information in S602 is not limited to the method using the setting area described with reference to FIG. Another method of setting display information based on object position information will now be described.

FIG. 9 is a diagram for explaining a method of setting the display information of the three-dimensional model in the display information registration area provided in the tracking area 30. As shown in FIG. In the display information setting method shown in FIG. 9, a transparent model registration area 910 and a drawing model registration area 920 are provided in the tracking area 30 .

When an object exists in the transparent model registration area 910 for a preset period of time, the display information setting unit 402 sets the display information of the object to transparent display. On the other hand, when an object exists in the drawing model registration area 920 for a preset period, the display information setting unit 402 sets the display information of the object to normal rendering display.

In FIG. 9, since the accessory 901 is placed in the transparent model registration area 910, transparent display is set in the display information. Therefore, even if the small article 901 is placed in the rendering area 20 , the three-dimensional model of the small article 901 undergoes transparency processing by the virtual viewpoint image generation unit 404 . Since the accessory 902 is placed in the drawing model registration area 920, normal rendering display is set in the display information. Therefore, when the small article 902 is placed in the rendering area 20 , the three-dimensional model of the small article 902 is normally rendered by the virtual viewpoint image generation unit 404 .

Although FIG. 9 shows an example in which the transparent model registration area 910 and the drawing model registration area 920 are defined outside the rendering area 20 , they may be defined within the rendering area 20 . In addition, other three-dimensional models such as people and animals are similarly processed instead of small objects.

Also, even if the small object 901 that has undergone transparent processing because the display information is set to transparent display is placed on the table 904 that has already been set to normal rendering display, the display information of the small object 901 does not need to be changed. , the transparency process may be continued. Similarly, even if a mole 903 that has already been set to transparent display holds the small object 902 that has undergone normal rendering processing because the normal rendering display has been set in the display information, the display information of the small object 902 does not need to be changed. Rendering may continue.

In addition, although FIG. 9 illustrates an example in which only one transmissive model registration area 910 and drawing model registration area 920 are provided, they may be provided in a plurality of places.

In addition, when an object exists in a predetermined registration area for a predetermined period of time, the display information setting unit 402 associates an object with the object so as to change the currently set display mode to another display mode. display information may be set. For example, if an object for which transparent display is set in certain display information exists in the transparent model registration area 910 for a predetermined period of time, the display mode may be changed to normal rendering display and the display information may be set. .

Next, as a method of setting display information in S602, a method other than the method based on the position information of the object described with reference to FIGS. 6 and 9 will be described.

FIG. 10 is a diagram for explaining a method of detecting a predetermined gesture from a 3D model and setting display information based on the gesture. Kuroko 1001 in FIG. 10 is waving his hand to the side. When the display information setting unit 402 detects a gesture of opening the hand and waving it sideways, the display information setting unit 402 sets normal rendering display as the display information of the object that performed the gesture. For this reason, the mole 1001 in FIG. 1 has been subjected to transparency processing, but is displayed after being changed to normal rendering processing. On the other hand, Kuroko 1002 is shaking his hand while holding it. When the display information setting unit 402 detects a gesture of holding a hand and shaking it vertically, the display information setting unit 402 sets transparent display as the display information of the object that performed the gesture. For this reason, the mole 1002 in FIG. 10 has been normally rendered, but has been changed to transparent processing and represented on the virtual viewpoint image.

Note that the gesture pattern for setting display information is not limited to the above pattern, and a different gesture may be used. Other gestures, such as jumping, shaking the head, rotating, etc., may also be used, and multiple gestures may be defined. A gesture is not limited to an action, and may be a predetermined pose such as a raised hand. Also, the display mode may be determined by a combination of a plurality of gestures.

As the display information setting method executed by the display information setting unit 402 in S602, the method of using the setting surface of FIG. 7, the method of inheriting the display information of the separation source of FIG. 8, the method of providing the registration area of FIG. 9, and the method of FIG. described a gesture-based method for In S602, the display information setting unit 402 may set the display information by selecting one of the display information setting methods described with reference to FIGS. May be set.

As described above, according to the present embodiment, display information for each object can be set, so that a virtual viewpoint image is generated so that each object is represented in a desired display mode in accordance with the rendering. can do.

If an object whose display information has been changed exists within the angle of view of the virtual camera, the object suddenly appears or disappears when the virtual viewpoint image is displayed as a moving image. Therefore, if the display information of the object within the angle of view of the virtual camera is changed, the viewer may feel uncomfortable. In order to suppress the viewer's discomfort, the virtual viewpoint image generation unit 404 may change the angle of view of the virtual camera to generate the virtual viewpoint image so that the object whose display information is changed is not reflected.

For example, the virtual viewpoint image generation unit 404 notifies the virtual viewpoint path generation device 500 that there is an object whose display information is to be changed. Upon receiving the notification, the virtual viewpoint path generation device 500 changes the position and direction of the virtual camera so that the object whose display information is to be changed is not captured, and sends the changed virtual camera information to the virtual viewpoint image generation unit. 404. In addition, until the virtual viewpoint path generation device 500 changes to a virtual camera with an angle of view that does not capture the object whose display information is to be changed, the virtual viewpoint image generation unit 404 prevents the display mode from being changed from the previous frame. A virtual viewpoint image generation process may be executed.

Also, depending on the frame, if an object exists on the boundary between the inside and the outside of the rendering area 20, the display information to be set for the object existing on the boundary may be set without using the above-described method.

FIG. 11 is a diagram showing objects on the boundary of the rendering area 20. FIG. It is assumed that normal rendering display is set for the display information associated with the mole 1101 . Also, the mole 1101 is about to enter the rendering area 20 from outside the rendering area 20, so it is assumed to exist on the boundary of the rendering area 20. FIG. In this case, when the virtual viewpoint image generation unit 404 performs normal rendering processing on the three-dimensional model of the mole 1101, only the portion included in the rendering area 20 will be rendered. For this reason, when a virtual viewpoint image is generated by performing normal rendering processing on a three-dimensional model of an object at the boundary between the outside and the inside of the rendering area 20, the generated virtual viewpoint image may give the viewer a sense of discomfort. .

In order to suppress the viewer's discomfort, the display information setting unit 402 sets transparent display as the display information of the mole 1101 until the entire three-dimensional model of the mole 1101 enters the rendering area 20 . Then, the display information setting unit 402 may set the display information so that the display mode of the mole 1101 is changed to the normal rendering display after the entire three-dimensional model of the mole 1101 enters the rendering area 20. .

On the other hand, if a mole in the rendering area 20 is displayed by normal rendering and the hand or toe of the mole is out of the rendering area 20, the mole is preferably displayed by normal rendering as it is. For this reason, the displayed information may be changed depending on whether a portion (a certain percentage, a specific portion, etc.) of the three-dimensional model is inside or outside the rendering area 20 .

For example, depending on the volume ratio of the three-dimensional model within the rendering area 20, the display information of the object may be switched between normal rendering display and transparent display. For example, if a portion of a 3D model for which normal rendering display is set goes outside the rendering area 20, the volume of the portion outside the rendering area 20 is 30% of the entire 3D model of that object. If it exceeds, switch to transparent display and set the display information. On the other hand, when part of the 3D model for which transparent display is set enters the rendering area 20, if the volume of the part entering the rendering area 20 exceeds 50% of the entire 3D model of the object, it is normal Display information may be changed to rendering display.

<Other embodiments>
The present disclosure provides a program that implements one or more functions of the above-described embodiments to a system or device via a network or storage medium, and one or more processors in a computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

100 imaging device 400 virtual viewpoint image generation device 402 display information setting unit 404 virtual viewpoint image generation unit

Claims (18)

setting means for setting, based on at least one of a position of the object and a gesture of the object, a display mode of the object in a virtual viewpoint image generated based on captured images obtained by imaging the object with a plurality of imaging devices; ,
generating means for generating the virtual viewpoint image so that the object is represented in the set display mode;
An image processing device comprising: 2. The image processing apparatus according to claim 1, wherein, when an object passes through a predetermined area, the setting means sets a display mode corresponding to the predetermined area to the object that has passed through the predetermined area. . wherein, when the object passes through the predetermined area, the setting means changes the display mode of the object that has passed through the predetermined area to a display mode different from the currently set display mode. Item 1. The image processing apparatus according to item 1. The generating means generates the virtual viewpoint image such that an object existing in a rendering space included in the space in which the plurality of imaging devices are arranged is represented in a display mode set for the object. The image processing apparatus according to any one of claims 1 to 3, wherein: 5. The image processing apparatus according to claim 4, further comprising acquisition means for acquiring a position of an object existing in a space having the same size as said rendering space or a space larger than said rendering space. The setting means
When an object enters the drawing space through a predetermined area set as a boundary of the drawing space, a first display mode is set as the display mode of the object that has passed through the predetermined area, and the object enters the drawing space. When the drawing space is entered by passing through an area other than the predetermined area of the boundary, a second display mode different from the first display mode is used as the display mode of the object that has passed through the other area. The image processing apparatus according to claim 4 or 5, characterized by: setting The setting means
when only a part of an object is included in the drawing space, the display mode of the object is set based on the ratio of the part to the whole of the object whose part is included in the drawing space. 7. The image processing apparatus according to any one of claims 4 to 6. The setting means
wherein, when only a part of an object is included in the drawing space, a display mode of the object is set so that the object whose part is included in the drawing space is transparently displayed. Item 8. The image processing device according to any one of Item 7. The setting means
When an object exists in a predetermined area for a predetermined period of time, the display mode of the object existing in the predetermined area is changed to a display mode different from the currently set display mode. The image processing apparatus according to any one of claims 1 to 8. The setting means
setting a first display mode as a display mode of the object existing in the first area when the object exists in the first area for a first period of time;
When an object exists in a second area different from the first area for a second period, the display mode of the object existing in the second area is the second display mode different from the first display mode. The image processing apparatus according to any one of claims 1 to 8, wherein the display mode of is set. The generating means is
Acquiring three-dimensional shape data of an object generated based on captured images obtained by imaging by the plurality of imaging devices to generate the virtual viewpoint image;
The setting means
For a second object for which the three-dimensional shape data has been generated integrally with the first object, setting a display mode as part of the first object,
When the three-dimensional shape data of the second object is generated by separating from the first object, the display mode of the separated second object is the same as the display mode of the first object. 11. The image processing apparatus according to any one of claims 1 to 10, wherein is set. The setting means
12. The method according to any one of claims 1 to 11, wherein when an object performs a predetermined gesture, a display mode corresponding to the predetermined gesture is set for the object that performed the predetermined gesture. Image processing device. The display mode set for the object by the setting means includes at least a display mode in which the object is displayed by projecting and drawing the texture of the captured image, and a display mode in which the object is displayed in a transparent manner. The image processing apparatus according to any one of claims 1 to 12. The setting means
14. The image processing apparatus according to any one of claims 1 to 13, wherein when a plurality of objects exist, a display mode is set for each of the plurality of objects. 15. The image processing apparatus according to any one of claims 1 to 14, further comprising managing means for managing the set display mode in association with the three-dimensional shape data of the object. The generating means is
generating the virtual viewpoint image corresponding to the virtual viewpoint;
When there is an object whose display mode is to be changed within an angle of view corresponding to the virtual viewpoint, the virtual viewpoint image is generated by changing the virtual viewpoint to a virtual viewpoint whose angle of view does not include the object whose display mode is to be changed. The image processing apparatus according to any one of claims 1 to 15, characterized by: a setting step of setting, based on at least one of a position of the object and a gesture of the object, a display mode of the object in a virtual viewpoint image generated based on captured images obtained by imaging the object with a plurality of imaging devices; ,
a generation step of generating the virtual viewpoint image such that the object is represented in the set display mode;
An image processing method characterized by comprising: A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 16.

Images