Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/66—Transforming electric information into light information
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N7/00—Television systems
  • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Definitions

• one or more aspects of the present disclosure aim to make it easy to reflect real-time conditions in free viewpoint video.
• FIG. 1 is a block diagram illustrating a schematic configuration of an information processing system according to first to sixth embodiments.
• 1 is a block diagram illustrating a schematic configuration of an information processing device according to a first embodiment.
• FIG. 2 is a block diagram illustrating a schematic configuration of a PC.
• FIG. 11 is a block diagram illustrating a schematic configuration of an information processing device according to a second embodiment.
• 13A and 13B are schematic diagrams for explaining 3D data.
• FIG. 13 is a block diagram illustrating a schematic configuration of an information processing device according to third and fourth embodiments.
• FIG. 13 is a block diagram illustrating a schematic configuration of an information processing device according to a fifth embodiment.
• FIG. 23 is a block diagram illustrating a schematic configuration of an information processing device according to a sixth embodiment.
• FIG. 1 is a block diagram illustrating a schematic configuration of an information processing system 100 according to the first embodiment.
• the information processing system 100 includes an imaging device 110, a user device 120, and an information processing device 130.
• the imaging device 110 and the information processing device 130 are connected to a network 101 such as the Internet, and the user-side device 120 and the information processing device 130 are directly connected.
• the connection between the user-side device 120 and the information processing device 130 may be wired or wireless.
• the above connection relationship is merely an example, and the imaging device 110 and the information processing device 130 may be directly connected to each other.
• the user device 120 may be connected to the network 101.
• the imaging device 110 captures video in real time.
• the video captured by the imaging device 110 is also called real-time video.
• the real-time video is sent to the information processing device 130 via the network 101.
• the user side device 120 has at least a display as a display unit that displays the output image from the information processing device 130, and an output unit that can output a signal that can identify the position and line of sight of the virtual user in the virtual space provided by the information processing device 130.
• the user device 120 may also include an input unit that accepts input of instructions from a user. The instructions input to the input unit may be sent to the information processing device 130 from an output unit.
• the information processing device 130 functions as a video output device that outputs video to the user device 120.
• FIG. 2 is a block diagram illustrating a schematic configuration of information processing device 130 according to the first embodiment.
• the information processing device 130 includes a position identification unit 131, a gaze direction identification unit 132, a 3D data storage unit 133, a free viewpoint video generation unit 134, a switching determination unit 135, a real-time video acquisition unit 136, a switching processing unit 137, and a video output unit 138.
• the position identification unit 131 receives a signal from the user-side device 120 via a connection unit (not shown), for example, and identifies the position of the virtual user moving in the virtual space via the user-side device 120. Position information indicating the identified position is provided to the free viewpoint video generation unit 134 and the switching determination unit 135.
• the user can move the virtual user corresponding to the user within the virtual space via the user-side device 120, and the position identification unit 131 can identify the position to which the virtual user has moved based on a signal from the user-side device 120.
• the user-side device 120 is an HMD
• the user can input various instructions to the user-side device 120 and move the virtual user corresponding to the user within the virtual space in accordance with the instructions.
• the user is able to change the gaze direction within a predetermined space via the user-side device 120, and the gaze direction identification unit 132 identifies the gaze direction changed by the user in response to an instruction from the user.
• the user device 120 is an HMD
• the direction of the user's line of sight can be identified based on signals from various sensors provided in the HMD.
• the 3D data storage unit 133 stores 3D data, which is data in 3D of a virtual space provided to the user via the user-side device 120.
• the 3D data only needs to include data necessary to generate a free viewpoint image, which is an image in 2D of a virtual space, from a position and line of sight direction within that virtual space.
• the 3D data only needs to be generated in advance using known technologies such as CAD (Computer Aided Design), LiDAR (Light Detection and Ranging), photogrammetry, NeRF (Neural Radiance Fields), 3D Gaussian Splatting, etc.
• the free viewpoint video generation unit 134 is a first video generation unit that generates a first video, which is a video of a virtual space at a position identified by the position identification unit 131 and in a line of sight direction identified by the line of sight direction identification unit 132.
• the free viewpoint video generation unit 134 refers to the 3D data stored in the 3D data storage unit 133, and generates, as a first image, a free viewpoint video that is a 2D image of the virtual space in the line of sight direction indicated by the line of sight direction information from the line of sight direction identification unit 132, at the position indicated by the position information from the position identification unit 131.
• the generated free viewpoint video is provided to the switching processing unit 137.
• the switching determination unit 135 is a determination unit that determines whether to switch between the free viewpoint video from the free viewpoint video generation unit 134 and the real-time video captured by the imaging device 110. For example, the switching determination unit 135 determines whether or not a predetermined condition is satisfied when a free viewpoint video is being output.
• the real-time video acquisition unit 136 acquires real-time video from the imaging device 110 via a communication unit (not shown).
• the acquired real-time video is provided to the switching processing unit 137.
• the switching processing unit 137 functions as a video processing unit that receives a notification from the switching determination unit 135 and switches between the free viewpoint video and the real-time video.
• the switched video is applied to the video output unit 138 as an output video. In other words, when a predetermined condition is satisfied, the switching processing unit switches the video to be output from the free viewpoint video, which is the first video, to the real-time video, which is the second video.
• the video output unit 138 outputs the output video from the switching processing unit 137 to the user device 120, for example, via a communication unit (not shown).
• the information processing device 130 described above can be realized by a computer such as the PC 10 shown in FIG.
• the PC 10 includes a storage 11 such as a hard disk drive (HDD) or a solid state drive (SSD), a memory 12, a processor 13 such as a central processing unit (CPU), a communication interface (I/F) 14 such as a network interface card (NIC), and a connection I/F 15 such as a universal serial bus (USB) connector.
• a storage 11 such as a hard disk drive (HDD) or a solid state drive (SSD)
• a memory 12 such as a central processing unit (CPU), a communication interface (I/F) 14 such as a network interface card (NIC), and a connection I/F 15 such as a universal serial bus (USB) connector.
• a storage 11 such as a hard disk drive (HDD) or a solid state drive (SSD)
• a memory 12 such as a central processing unit (CPU), a communication interface (I/F) 14 such
• the 3D data storage unit 133 can be realized by the storage 11.
• the position identification unit 131, the gaze direction identification unit 132, the free viewpoint video generation unit 134, the switching determination unit 135, the real-time video acquisition unit 136, the switching processing unit 137 and the video output unit 138 can be realized by loading a program stored in the storage 11 into the memory 12 and having the processor 13 execute the program.
• a communication unit (not shown) can be realized by the communication I/F 14 .
• a connection section (not shown) can be realized by the connection I/F 15 .
• the above programs may be downloaded to the storage 11 from a recording medium (not shown) via a reader/writer (not shown) or from the network 101 via the communication I/F 14, and then loaded onto the memory 12 and executed by the processor 13.
• the programs may be directly loaded onto the memory 12 from a recording medium via a reader/writer or from the network 101 via the communication I/F 14, and executed by the processor 13.
• the program may be provided by a program product such as a recording medium.
• real-time video is output only when the virtual user enters a predetermined range in the space in which the virtual user moves, so that the real-time situation can be easily reflected in the free viewpoint video.
• switching between the free viewpoint video and the real-time video is performed depending on the position of the virtual user, but the first embodiment is not limited to this example.
• switching may be performed when the user inputs a predetermined instruction to the user-side device 120.
• an information processing system 200 according to the second embodiment includes an imaging device 210, a user device 120, and an information processing device 230.
• the user device 120 of the information processing system 200 according to the second embodiment is similar to the user device 120 of the information processing system 100 according to the first embodiment.
• the imaging device 210 captures real-time video, which is real-time video.
• the imaging device 210 is configured to capture video in a 360° direction so that a 2D video from the viewpoint of the virtual user can be generated.
• the real-time video is sent to the information processing device 230 via the network 101.
• FIG. 4 is a block diagram showing a schematic configuration of an information processing device 230 according to the second embodiment.
• the information processing device 230 includes a position identification unit 131, a gaze direction identification unit 132, a 3D data memory unit 233, a free viewpoint video generation unit 234, a switching determination unit 135, a real-time video acquisition unit 136, a switching processing unit 237, a video output unit 138, and a real-time viewpoint video generation unit 239.
• the position identification unit 131, the gaze direction identification unit 132, the switching determination unit 135, the real-time video acquisition unit 136 and the video output unit 138 of the information processing device 230 in embodiment 2 are similar to the position identification unit 131, the gaze direction identification unit 132, the switching determination unit 135, the real-time video acquisition unit 136 and the video output unit 138 of the information processing device 130 in embodiment 1.
• the line of sight direction information generated by line of sight direction identification section 132 is also provided to real-time viewpoint video generation section 239 .
• the real-time video acquired by the real-time video acquisition unit 136 is provided to a real-time viewpoint video generation unit 239 .
• the 3D data storage unit 233 stores 3D data, which is data in 3D of the virtual space provided to the user via the user-side device 120.
• 3D data is data required to generate a free viewpoint image, which is a 2D image of the virtual space 103 of that space 102, as shown in FIG. 5(B).
• the virtual space 103 is a space that virtually represents the space 102 in which the imaging device 210 is installed.
• the free viewpoint video generating unit 234 references the 3D data stored in the 3D data storage unit 233 and generates a 2D video of the virtual space in the line of sight indicated by the line of sight direction information from the line of sight direction identifying unit 132 at the position indicated by the position information from the position identifying unit 131.
• the 2D video generated here is a video of the virtual space 103 which is a virtual representation of the space 102 in which the imaging device 210 is placed.
• the free viewpoint video generation unit 234 includes a switching range image, which is an image indicating the switching range, in the 2D video.
• the switching range image may be any image that allows the user to recognize that the video will switch when the virtual user enters that position.
• the switching range image is a circular line superimposed at a position corresponding to the ground.
• a cylindrical semi-transparent image corresponding to the switching range may be superimposed as the switching range image.
• the free viewpoint video generation unit 234 provides the 2D video or the video with the switching range image superimposed on the 2D video as a free viewpoint video to the switching processing unit 237.
• FIG. 6 is a block diagram showing a schematic configuration of an information processing device 330 according to the third embodiment.
• the information processing device 330 includes a position identification unit 131, a gaze direction identification unit 132, a 3D data memory unit 333, a free viewpoint video generation unit 334, a switching determination unit 135, a real-time video acquisition unit 136, a switching processing unit 337, a video output unit 138, and a real-time viewpoint video generation unit 239.
• the free viewpoint video generation unit 334 references the 3D data stored in the 3D data storage unit 333, and generates a 2D video of the virtual space in the line of sight indicated by the line of sight direction information from the line of sight direction identification unit 132 at the position indicated by the position information from the position identification unit 131.
• the free viewpoint video generation unit 334 then provides the 2D video generated here to the switching processing unit 337 as a free viewpoint video.
• the switching processing unit 337 switches between the free viewpoint video and the real-time viewpoint video.
• the switching processing unit 337 performs an immersion maintaining process for maintaining the immersion in the video when switching from the free viewpoint video to the real-time viewpoint video.
• the immersion maintaining process is, for example, a process that prevents switching from a free viewpoint video to a real-time viewpoint video in one frame.
• Whiteout is a method of displaying a white image between the image before and after the switch for a predetermined period of time. It is desirable for the predetermined period of time to be relatively short. This is expected to have the effect of suggesting a change in the world view.
• the video after switching moves from the right to the left of the video before switching. This allows the video to switch gradually from one direction, maintaining a sense of immersion.
• the user's sense of immersion in the video can be maintained when the video switches.
• the switching processing unit 337 performs immersion maintenance processing when switching from the free viewpoint video to the real-time viewpoint video, but the third embodiment is not limited to this example.
• the switching processing unit 337 may also perform immersion maintenance processing when switching from the real-time viewpoint video to the free viewpoint video.
• the information processing system 400 according to the fourth embodiment includes an imaging device 210 , a user device 120 , and an information processing device 430 .
• the user device 120 of the information processing system 400 according to the fourth embodiment is similar to the user device 120 of the information processing system 100 according to the first embodiment.
• the imaging device 210 of the information processing system 400 according to the fourth embodiment is similar to the imaging device 210 of the information processing system 200 according to the second embodiment.
• the information processing device 430 in the fourth embodiment includes a position identification unit 131, a gaze direction identification unit 132, a 3D data storage unit 333, a free viewpoint video generation unit 334, a switching determination unit 135, a real-time video acquisition unit 136, a switching processing unit 437, a video output unit 138, and a real-time viewpoint video generation unit 239.
• the position identification unit 131, the gaze direction identification unit 132, the switching determination unit 135, the real-time video acquisition unit 136, and the video output unit 138 of the information processing device 430 in embodiment 4 are similar to the position identification unit 131, the gaze direction identification unit 132, the switching determination unit 135, the real-time video acquisition unit 136, and the video output unit 138 of the information processing device 130 in embodiment 1.
• the line of sight direction information generated by line of sight direction identification section 132 is also provided to real-time viewpoint video generation section 239 .
• the real-time video acquired by the real-time video acquisition unit 136 is provided to a real-time viewpoint video generation unit 239 .
• the real-time viewpoint video generation unit 239 of the information processing device 430 in the fourth embodiment is similar to the real-time viewpoint video generation unit 239 of the information processing device 230 in the second embodiment. Furthermore, the 3D data storage unit 333 and the free viewpoint video generation unit 334 of the information processing device 430 in embodiment 4 are similar to the 3D data storage unit 333 and the free viewpoint video generation unit 334 of the information processing device 330 in embodiment 3.
• the switching processing unit 437 performs immersion maintenance processing when switching from the free viewpoint video to the real-time viewpoint video, but the third embodiment is not limited to such an example.
• the switching processing unit 437 may also perform immersion maintenance processing when switching from the real-time viewpoint video to the free viewpoint video.
• FIG. 7 is a block diagram showing a schematic configuration of an information processing device 530 according to the fifth embodiment.
• the information processing device 530 includes a position identification unit 131, a gaze direction identification unit 132, a 3D data memory unit 333, a free viewpoint video generation unit 334, a switching determination unit 135, a real-time video acquisition unit 136, a switching processing unit 537, a video output unit 138, and a real-time viewpoint video generation unit 239.
• the real-time video acquisition unit 136 receives real-time video, which is a 360° video, and the real-time viewpoint video generation unit 239 generates real-time viewpoint video by extracting video in the user's line of sight from the real-time video, but embodiments 2 to 5 are not limited to such examples.
• the gaze direction identification unit 132 may send gaze direction information to the imaging device 210 via a communication unit (not shown), and the imaging device 210 may capture a real-time viewpoint video corresponding to the gaze direction.
• the switching determination unit 635 determines to switch the video to be output to the user side device 120 from the superimposed free viewpoint video to the free viewpoint video. The above determination is notified to the object superimposition unit 640 .
• the object superimposition unit 640 extracts, as a dynamic object, an object in the real-time image that shows a difference between the image at a certain time and the image after a certain time has passed.
• the method of extracting an object by difference may be a known technique such as "foreground/background region segmentation algorithm based on a mixed normal distribution (Efficient Adaptive Density Estimation par Image Pixel for the Task of Background Subtraction)" or "algorithm combining a statistical background estimation method and region segmentation based on Bayesian estimation in pixel units (Visual Tracking of Human Visitors under Variable-Lighting Conditions for a Responsive Audio Art Installation)".
• the object superimposition section 640 stops extracting the dynamic object and stops superimposing the dynamic object.
• the video switched as described above is provided to the video output unit 138 as an output video.
• the switching range may be superimposed on the free viewpoint video as shown in the second embodiment.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Computer Graphics (AREA)
• Computer Hardware Design (AREA)
• General Engineering & Computer Science (AREA)
• Software Systems (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Processing Or Creating Images (AREA)

Priority Applications (2)

Applications Claiming Priority (1)

Publications (1)

Family

ID=96386896

Family Applications (1)

Country Status (2)

Citations (4)

Family Cites Families (13)

• 2024
  • 2024-01-12 WO PCT/JP2024/000625 patent/WO2025150187A1/ja active Pending
  • 2024-01-12 JP JP2024541785A patent/JPWO2025150187A1/ja active Pending

Patent Citations (4)

Also Published As

Similar Documents

Legal Events