WO2022024780A1 - 情報処理装置、情報処理方法、映像配信方法及び情報処理システム - Google Patents

情報処理装置、情報処理方法、映像配信方法及び情報処理システム Download PDF

Info

Publication number
WO2022024780A1
WO2022024780A1 PCT/JP2021/026587 JP2021026587W WO2022024780A1 WO 2022024780 A1 WO2022024780 A1 WO 2022024780A1 JP 2021026587 W JP2021026587 W JP 2021026587W WO 2022024780 A1 WO2022024780 A1 WO 2022024780A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
subject
dimensional
information processing
video
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/026587
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
徹也 福安
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Priority to JP2022540167A priority Critical patent/JPWO2022024780A1/ja
Priority to US18/014,249 priority patent/US12260491B2/en
Priority to CN202180049906.0A priority patent/CN116075860B/zh
Priority to EP21851014.7A priority patent/EP4178199A4/en
Publication of WO2022024780A1 publication Critical patent/WO2022024780A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—Three-dimensional [3D] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—Three-dimensional [3D] image rendering
    • G06T15/04—Texture mapping
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/243—Image signal generators using stereoscopic image cameras using three or more two-dimensional [2D] image sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/8146—Monomedia components thereof involving graphical data, e.g. 3D object, 2D graphics

Definitions

  • This disclosure relates to an information processing device, an information processing method, a video distribution method, and an information processing system.
  • Volumetric technology also called volumetric capture technology
  • volumetric capture technology reconstructs the three-dimensional shape of the subject (object) inside it and redraws it from a free viewpoint using multiple cameras arranged so as to surround the subject.
  • This volumetric technology by arranging the camera so as to capture the back and the top of the head, it is possible to generate an image (volumetric image) in which the viewer can see the subject from all directions.
  • an information processing device an information processing method, a video distribution method, and an information processing system that make it possible to generate an image to which a wide range of effects are added from a three-dimensional object generated by volumetric technology or the like. ..
  • the information processing apparatus includes a three-dimensional model of the subject and a two-dimensional image generated by using a plurality of captured images obtained by imaging the subject. Based on this, the first generation unit is provided to generate an image in which the subject generated from the three-dimensional model and the two-dimensional image coexist.
  • RGB virtual viewpoint image
  • One Embodiment 1-1 Functional configuration of information processing system 1-2. Flow of processing performed by the information processing system 1-3.
  • Volumetric technology is a technology that reconstructs the three-dimensional shape of the subject (object) inside it using multiple cameras arranged so as to surround the subject, and redraws it from a free viewpoint. .. By arranging the camera so that it also captures the back and top of the head, the viewer can see the subject from all directions. Since various processes such as capture, modeling, and rendering in such volumetric technology have enormous calculation costs and take a long time to process, they are usually assumed to be processed offline. However, if various processes in the volumetric technology can be executed online in real time, it will be possible to immediately generate a volumetric image from the captured 3D object and distribute it to the user.
  • the volumetric image may be, for example, an image generated by using a 3D object generated by a volumetric technique.
  • Real-time execution of various processes in volumetric technology can be realized by utilizing, for example, HPC (High Performance Computing) technology used for large-scale processing in supercomputers and data centers.
  • HPC High Performance Computing
  • a volumetric video generated by superimposing a 3D object generated by the volumetric technology on a background object created in advance is viewed. It will be.
  • simply superimposing a background object on a 3D object does not always make it possible to produce a variety of video contents such as live music, watching sports, lectures, and lectures.
  • a volumetric video with a limited resource such as one personal computer (hereinafter referred to as a PC)
  • the processing cannot catch up with the captured video data and the real-time property is impaired. There is a risk that it will end up.
  • the volumetric technology incorporates additional processing such as superimposition of other 3D objects, image quality improvement, effects, etc., the overall processing load increases and it becomes difficult to secure real time.
  • a PC equipped with Linux can perform low-latency distributed processing (hereinafter referred to as GPU distributed processing) by distributing processing to multiple GPUs (Graphics Processing Units), but a library of necessary processing.
  • GPU distributed processing low-latency distributed processing
  • the development efficiency is poor.
  • various processes in the volumetric technology are distributed to a plurality of computers to enable rapid generation of volumetric images. For example, it enables the generation of volumetric video in which real time is ensured.
  • the degree of freedom in the system development environment is improved, so that it is possible to construct a system in which a decrease in development efficiency is suppressed.
  • FIG. 1 is a block diagram showing a schematic configuration of the information processing system of the present embodiment.
  • the information processing system 10 includes a data acquisition unit 11, a 3D model generation unit 12, a rendering unit 13, a transmission unit 14, a reception unit 15, and a display unit 16.
  • the display unit 16 does not have to be included in the information processing system 10.
  • the data acquisition unit 11 includes image data for generating a three-dimensional model of the subject 90, which is an object to be imaged (hereinafter referred to as an actual camera image.
  • the image data in the present description includes image data such as a moving image. Get).
  • a plurality of real cameras 70a, 70b, 70c, 70d, 70e, ... (Hereinafter, real cameras 70a, 70b, 70c, 70d, 70e, 7) Arranged so as to surround the subject 90 are summarized.
  • a plurality of viewpoint images captured by the real camera 70) are acquired as real camera images.
  • the plurality of viewpoint images are preferably images taken in synchronization by a plurality of real cameras 70.
  • the data acquisition unit 11 may acquire a plurality of real camera images from different viewpoints by, for example, moving one real camera 70 and taking an image of the subject 90 from a plurality of viewpoints.
  • the present invention is not limited to this, and the data acquisition unit 11 may acquire one real camera image of the subject 90.
  • the 3D model generation unit 12 described later may generate a three-dimensional model of the subject 90 from one real camera image by using, for example, machine learning.
  • the data acquisition unit 11 may perform calibration based on the actual camera image and acquire the internal parameters and the external parameters of each actual camera 70. Further, the data acquisition unit 11 may acquire a plurality of depth information indicating distances from a plurality of viewpoints to the subject 90, for example.
  • the 3D model generation unit 12 generates a 3D model having 3D information of the subject based on an actual camera image for generating a 3D model of the subject 90.
  • the 3D model generation unit 12 uses, for example, a so-called Visual Hull, and cuts the three-dimensional shape of the subject based on images from a plurality of viewpoints (for example, silhouette images from a plurality of viewpoints) to obtain a three-dimensional model of the subject. May be generated.
  • the 3D model generation unit 12 can further transform the three-dimensional model generated by using Visual Hull with high accuracy by using a plurality of depth information indicating the distances from the viewpoints of a plurality of points to the subject.
  • the 3D model generation unit 12 may generate a three-dimensional model of the subject 90 from one real camera image of the subject 90.
  • the 3D model generated by the 3D model generation unit 12 can be said to be a moving image of the 3D model in that it is generated in chronological order in frame units. Further, since the three-dimensional model is generated by using the real camera image acquired by the real camera 70, it can also be called a live-action three-dimensional model.
  • the shape information representing the surface shape of the subject 90 may be expressed in the form of three-dimensional shape mesh data called a polygon mesh, which is expressed by the connection between the vertices (Vertex) and the vertices. ..
  • the three-dimensional shape mesh data has, for example, three-dimensional coordinates of the vertices of the mesh and index information indicating which vertices are combined to form a triangular mesh.
  • the method of expressing the three-dimensional model is not limited to this, and may be described by a so-called point cloud expression method expressed by the position information of points.
  • textures also called texture images.
  • textures also called texture images.
  • a View Independent texture method in which the color is constant when viewed from any direction
  • a View Dependent texture method in which the color changes depending on the viewing direction.
  • either or both of these may be adopted, and other texture methods may be adopted.
  • the rendering unit 13 projects, for example, the 3D shape mesh data of the 3D model from the drawing camera viewpoint (corresponding to the virtual viewpoint described later), and creates a texture representing the color and pattern of the mesh with respect to the projected mesh shape.
  • texture mapping By performing texture mapping to paste, a volumetric image of a 3D model is generated. Since the viewpoint at this time is a free viewpoint set arbitrarily regardless of the camera position at the time of imaging, the viewpoint is also referred to as a virtual viewpoint in the present embodiment.
  • the viewing viewpoint data may be input to the rendering unit 13 from the user terminal after the user's viewing point (Region of Interest) is detected by the user's display device (hereinafter, also referred to as a user terminal).
  • the rendering unit 13 may adopt, for example, a billboard rendering method in which the object is rendered so that the vertical posture of the object with respect to the viewing viewpoint is maintained.
  • the billboard rendering method may be applied to objects that are of low interest to the viewer, and other rendering methods may be adopted for other objects.
  • the rendering unit 13 appropriately executes various processes such as adding shadows, improving image quality, and effects on the generated volumetric video, and finally generates video content to be delivered to the user.
  • the transmitting unit 14 transmits (distributes) a video stream of video content output from the rendering unit 13 to one or more user terminals including a receiving unit 15 and a display unit 16 via a predetermined network.
  • the predetermined network includes the Internet, LAN (Local Area Network) (including Wi-Fi, etc.), WAN (Wide Area Network), mobile communication network (LTE (Long Term Evolution), 4th generation mobile communication system, 5th. It may be various networks such as (including generation mobile communication systems and the like).
  • the receiving unit 15 is provided in the user terminal described above, and receives the video content transmitted (distributed) from the transmitting unit 14 via a predetermined network.
  • the display unit 16 is provided on the user terminal described above, and displays the video content received by the reception unit 15 to the user.
  • the user terminal may be an electronic device capable of viewing video content, such as a head-mounted display, a spatial display, a television, a PC, a smartphone, a mobile phone, or a tablet terminal. Further, the display unit 16 may be a 2D monitor or a 3D monitor.
  • the information processing system 10 of FIG. 1 shows a series of flows from a data acquisition unit 11 that acquires a captured image, which is a material for generating content, to a display unit 16 that controls a user terminal to be viewed by a user.
  • a data acquisition unit 11 that acquires a captured image, which is a material for generating content
  • a display unit 16 that controls a user terminal to be viewed by a user.
  • this does not mean that all functional blocks are required for the embodiment of the present embodiment, and the present embodiment can be implemented for each functional block or a combination of a plurality of functional blocks.
  • the rendering unit 13 may be arranged on the user terminal side including the display unit 16. ..
  • the transmission data is transmitted to the server in which the 3D model generation unit 12 is arranged.
  • a coding unit for compression may be provided, and a decoding unit for decoding the compressed data may be provided in the server in which the rendering unit 13 is arranged.
  • the same implementer may implement all of them, or different implementers may implement each functional block or each process (step) described later.
  • the business operator X generates 3D contents through the data acquisition unit 11 and the 3D model generation unit 12.
  • the 3D content may be distributed through the platform of the operator Y, and a plurality of implementers may jointly implement the information processing device of the operator Z so as to perform rendering and display control of the 3D content. ..
  • each of the above-mentioned functional blocks can be implemented on the cloud.
  • the rendering unit 13 may be implemented on the user terminal side or the server side. In that case, information is exchanged between the user terminal and the server.
  • the data acquisition unit 11, the 3D model generation unit 12, the rendering unit 13, the transmission unit 14, the reception unit 15, and the display unit 16 are collectively described as an information processing system 10.
  • the information information system 10 of the present specification is referred to as an information system if two or more functional blocks are involved.
  • the data acquisition unit 11 does not include the reception unit 15 and the display unit 16.
  • Two of the 3D model generation unit 12, the rendering unit 13, and the transmission unit 14 can be collectively referred to as an information processing system 10.
  • FIG. 3 is a flowchart showing an example of the flow of processing performed by the information processing system.
  • the data acquisition unit 11 acquires an actual camera image of a subject taken by a plurality of actual cameras 70 (step S11).
  • the 3D model generation unit 12 generates a 3D model having 3D information (3D shape mesh data and texture) of the subject based on the actual camera image acquired in step S11 (step S12).
  • the rendering unit 13 renders the 3D model generated in step S12 based on the 3D shape mesh data and the texture to generate video content to be presented to the user (step S13).
  • the transmission unit 14 transmits (delivers) the video content generated in step S13 to the user terminal (step S14).
  • the receiving unit 15 of the user terminal receives the video content transmitted from the transmitting unit 14 (step S15). Then, the display unit 16 of the user terminal displays the video content received in step S15 to the user (step S16). After that, the information processing system 10 ends this operation.
  • FIG. 4 is a hardware block diagram showing an example of the hardware configuration of the information processing system.
  • the CPU 21, ROM 22, and RAM 23 are connected to each other via the bus 24.
  • the input / output interface 25 is also connected to the bus 24.
  • An input unit 26, an output unit 27, a storage unit 28, a communication unit 29, and a drive 20 are connected to the input / output interface 25.
  • the input unit 26 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, and the like.
  • the output unit 27 includes, for example, a display, a speaker, an output terminal, and the like.
  • the storage unit 28 includes, for example, a hard disk, a RAM disk, a non-volatile memory, and the like.
  • the communication unit 29 includes, for example, a network interface and the like.
  • the drive 20 drives a removable medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
  • the CPU 21 loads the program stored in the storage unit 28 into the RAM 23 via the input / output interface 25 and the bus 24, and executes the above-mentioned series. Process.
  • the RAM 23 also appropriately stores data and the like necessary for the CPU 21 to execute various processes.
  • the program executed by the computer can be recorded and applied to removable media such as package media, for example.
  • the program can be installed in the storage unit 28 via the input / output interface by mounting the removable media in the drive.
  • the program can also be provided via wired or wireless transmission media such as local area networks, the Internet, and digital satellite broadcasts.
  • the program can be received by the communication unit 29 and installed in the storage unit 28.
  • FIG. 5 is a block diagram showing a more detailed configuration example of the rendering unit 13 according to the present embodiment.
  • the rendering unit 13 includes an initial virtual viewpoint image generation unit 131, a transmission unit 132, a reception unit 133, and a final virtual viewpoint image generation unit 134.
  • the rendering unit 13 may include a plurality of final virtual viewpoint video generation units 134.
  • the 3D model (3D shape mesh data and texture) generated by the 3D model generation unit 12 is input to the rendering unit 13.
  • virtual viewpoint information information about a virtual viewpoint when rendering a three-dimensional model
  • This virtual viewpoint information includes information indicating the position of the virtual viewpoint (hereinafter referred to as virtual viewpoint position) and information of a three-dimensional rotation matrix with respect to the reference position of the virtual viewpoint (hereinafter referred to as virtual viewpoint rotation information). obtain.
  • the rendering unit 13 includes a real camera image acquired by any one or more of the real cameras 70 (hereinafter, N viewpoints (N is an integer of 0 or more)) and the real cameras 70 of each of the N viewpoints.
  • the viewpoint position hereinafter referred to as the actual camera viewpoint position
  • the information indicating the three-dimensional rotation matrix with respect to the reference position of the viewpoint position of the actual camera 70 of each of the N viewpoints hereinafter referred to as the actual camera viewpoint rotation information
  • the actual camera viewpoint position and the actual camera viewpoint rotation information may be, for example, information included in the internal parameters and the external parameters obtained by the calibration by the data acquisition unit 11, and in the following description, these may be collectively the actual camera. It is called viewpoint information.
  • the initial virtual viewpoint image generation unit 131 generates a virtual viewpoint image by rendering a three-dimensional model from a virtual viewpoint based on the input three-dimensional shape mesh data, texture, and virtual viewpoint information. Then, the initial virtual viewpoint image generation unit 131 uses the generated virtual viewpoint image to generate an intermediate rendered image to be transmitted to the final virtual viewpoint image generation unit 134.
  • the intermediate rendered image may be an image obtained by tying a virtual viewpoint image (RGB) and a virtual viewpoint image (Dept) into one image, and may be an auxiliary image, an actual camera image, or the like. The image may be an image further included in the above one image. The details of the intermediate rendered video will be described later.
  • the initial virtual viewpoint image generation unit 131 may correspond to, for example, one example of the second generation unit in the claims.
  • the transmission unit 132 and the reception unit 133 are configured to transmit the intermediate rendered video generated by the initial virtual viewpoint video generation unit 131 to one or a plurality of final virtual viewpoint video generation units 134.
  • the propagation of the intermediate rendered image via the transmission unit 132 and the reception unit 133 may be via a predetermined network such as a LAN, WAN, the Internet, or a mobile communication network, or HDMI (registered trademark) (High-Definition Multimedia). It may be via a predetermined interface such as Interface) or USB (Universal Serial Bus).
  • a predetermined network such as a LAN, WAN, the Internet, or a mobile communication network, or HDMI (registered trademark) (High-Definition Multimedia). It may be via a predetermined interface such as Interface) or USB (Universal Serial Bus).
  • the present invention is not limited to these, and various communication means may be used.
  • the final virtual viewpoint video generation unit 134 In the final virtual viewpoint video generation unit 134, the intermediate rendered video input from the initial virtual viewpoint video generation unit 131 via the transmission unit 132 and the reception unit 133 is shared by one or more final virtual viewpoint video generation units 134. Based on the actual camera viewpoint information, the initial virtual viewpoint video generation unit 131 performs unexecuted processing to finally generate video content to be presented to the user. For example, the final virtual viewpoint image generation unit 134 performs processing such as superimposing a background object or another 3D object on the volumetric image generated from the three-dimensional model, and improving the image quality of the volumetric image. Further, the final virtual viewpoint image generation unit 134 may also perform arrangement of the actual camera image 33 on the generated volumetric image, effect processing, and the like. The final virtual viewpoint image generation unit 134 may correspond to, for example, one example of the first generation unit in the claims.
  • FIG. 6 is a diagram showing an example of an intermediate rendered image according to the present embodiment.
  • the virtual viewpoint video (RGB) 31 the virtual viewpoint video (depth) 32
  • the actual camera video 33 the auxiliary video 34 constitute one image data. It has a structure that is combined (referred to as tyling in this description).
  • the actual camera image 33 and the auxiliary image 34 may correspond to, for example, one example of the captured image in the claims.
  • the actual camera image 33 and the auxiliary image 34 in the present embodiment and the plurality of captured images in the range of the request are not limited to the actual camera image acquired by the actual camera 70, and may be a movie, a music video, or a promotional video thereof. Regardless of whether the video is taken online or created offline, such as video that has been broadcast on TV, video that has been distributed online, or video that is shared at a video conference. , Various video contents and the like. Further, the intermediate rendered image 30 may correspond to, for example, one example of a packing image in the claims.
  • FIG. 7 is a diagram showing an example of a virtual viewpoint image (RGB) according to the present embodiment.
  • the virtual viewpoint image (RGB) 31 is a current volume generated by rendering, for example, a 3D model (3D shape mesh data and texture) from a virtual viewpoint based on virtual viewpoint information. It may be a metric image.
  • the virtual viewpoint image (RGB) 31 can hold the texture information when the three-dimensional model is viewed from the virtual viewpoint.
  • This virtual viewpoint image (RGB) 31 may correspond to, for example, one example of the first texture image in the claims.
  • FIG. 8 is a diagram showing an example of a virtual viewpoint image (depth) according to the present embodiment.
  • the virtual viewpoint image (depth) 32 may be image data showing the depth (depth) of each pixel in the virtual viewpoint image (RGB) 31 from the virtual viewpoint, and may be three-dimensional from the virtual viewpoint position. It may be a depth image generated by calculating the distance (depth information) to each point in the shape mesh data.
  • the virtual viewpoint image (depth) 32 can hold the depth information from the virtual viewpoint to the 3D model when the 3D model is viewed from the virtual viewpoint.
  • FIG. 9 is a diagram showing an example of an actual camera image according to the present embodiment.
  • the actual camera image 33 may be, for example, an actual camera image taken by any one selected of the actual cameras 70.
  • the selection of any one unit may be random, may be selected by the content creator, or may be selected by the content viewer (user).
  • FIG. 10 is a diagram showing an example of an auxiliary video according to the present embodiment.
  • the auxiliary image 34 is, for example, image data used for improving the image quality of the volumetric image finally provided to the user, for example, the subject from the same viewpoint as the virtual viewpoint.
  • the image may be different from the virtual viewpoint image (RGB) 31 in which 90 is reflected.
  • the auxiliary image 34 may be, for example, an actual camera image taken by any one selected of the actual cameras 70.
  • the selection of any one unit may be random, may be selected by the content creator, or may be selected by the content viewer (user), as in the case of the actual camera image 33. ..
  • the auxiliary image 34 may be omitted.
  • the auxiliary image 34 may correspond to, for example, one example of the second texture image in the claims.
  • the initial virtual viewpoint video generation unit 131 and one or a plurality of final virtual viewpoint video generation units 134 have virtual viewpoint information (virtual) in order to enable processing based on the same virtual viewpoint. Viewpoint position and virtual viewpoint rotation information) are shared.
  • the initial virtual viewpoint video generation unit 131 and / or the plurality of final virtual viewpoint video generation units 134 can perform processing based on the same virtual viewpoint, so that each processing in the initial virtual viewpoint video generation unit 131 and / or Each process in the final virtual viewpoint video generation unit 134 is distributed to a plurality of computers to be executed, or a plurality of final virtual viewpoint video generation units 134 are distributed to a plurality of computers and each of them is provided to a user. It is possible to generate various volumetric images (corresponding to video contents).
  • FIG. 11 is a block diagram showing a more detailed configuration example of the initial virtual viewpoint image generation unit according to the present embodiment.
  • the initial virtual viewpoint video generation unit 131 includes a virtual viewpoint video (RGB) generation unit 1312, an auxiliary video generation unit 1313, a virtual viewpoint video (depth) generation unit 1314, and an intermediate rendering video generation unit. It is equipped with 1315.
  • RGB virtual viewpoint video
  • depth virtual viewpoint video
  • the three-dimensional shape mesh data, the texture, the virtual viewpoint information, and the actual camera viewpoint information are input to the virtual viewpoint image (RGB) generation unit 1312.
  • Three-dimensional shape mesh data, texture, virtual viewpoint information, actual camera viewpoint information, and actual camera image (N viewpoint) are input to the auxiliary image generation unit 1313.
  • the actual camera image (N viewpoint) is also input to the intermediate rendering image generation unit 1315.
  • the virtual viewpoint image (RGB) generation unit 1312 projects the three-dimensional shape mesh data from the virtual viewpoint as described above as the operation of the rendering unit 13, and the color and pattern of the mesh are projected with respect to the projected mesh shape. Perform texture mapping to paste the texture that represents.
  • the virtual viewpoint video (RGB) 31 generated thereby is input to the virtual viewpoint video (depth) generation unit 1314 and the intermediate rendering video generation unit 1315, respectively.
  • the auxiliary image generation unit 1313 is an auxiliary image 34 used for image quality improvement implemented by the final virtual viewpoint image generation unit 134, which will be described later.
  • As the auxiliary image 34 an actual camera image acquired by any one or more of the actual cameras 70 may be used.
  • the virtual viewpoint image (depth) generation unit 1314 is the depth information of each point (corresponding to a pixel) on the three-dimensional model determined when the virtual viewpoint image (RGB) generation unit 1312 generates the virtual viewpoint image (RGB) 31. From, a virtual viewpoint image (depth) 32 which is a depth image is generated. The generated virtual viewpoint image (depth) 32 is input to the intermediate rendering image generation unit 1315.
  • the virtual viewpoint video (RGB) generation unit 1312 determines the depth information by an absolute value (mm or the like)
  • the virtual viewpoint video (depth) generation unit 1314 quantizes the depth information of each pixel to virtualize the depth information.
  • the viewpoint image (depth) 32 may be generated.
  • the bit depth of other image data such as the virtual viewpoint image (RGB) 31 is 8 bits
  • the virtual viewpoint image (depth) generation unit 1314 sets the depth information (mm or the like) of each pixel from “0” to “0”. It may be quantized into the depth information of 256 gradations of "255". As a result, it is not necessary to set the bit depth of the intermediate rendered video to a deeper depth, and it is possible to suppress an increase in the amount of data to be transmitted.
  • the intermediate rendered video generation unit 1315 has a virtual viewpoint video (RGB) 31 input from the virtual viewpoint video (RGB) generation unit 1312 and a virtual viewpoint video (depth) 32 input from the virtual viewpoint video (depth) generation unit 1314.
  • RGB virtual viewpoint video
  • depth virtual viewpoint video
  • the directly input real camera image (N viewpoint) 33 and the auxiliary image 34 input from the auxiliary image generation unit 1313 in a predetermined arrangement, one intermediate rendered image 30 (FIG. 6). See).
  • the generated intermediate rendered video 30 is output to the transmitting unit 132 in the rendering unit 13.
  • FIG. 12 is a block diagram showing a more detailed configuration example of the final virtual viewpoint image generation unit according to the present embodiment.
  • the final virtual viewpoint image generation unit 134 includes an object generation unit 1342, a camera image update unit 1343, a virtual viewpoint image generation unit 1344, a shadow generation unit 1345, an image quality improvement unit 1346, and an effect. It is provided with a processing unit 1347.
  • Objects (contents) superimposed on the 3D model and virtual viewpoint information are input to the object generation unit 1342.
  • An intermediate rendered image 30 and virtual viewpoint information are input to the camera image update unit 1343.
  • the intermediate rendered image 30 is also input to the virtual viewpoint image generation unit 1344, the shadow generation unit 1345, and the image quality improvement unit 1346.
  • the virtual viewpoint information may be input to the virtual viewpoint image generation unit 1344, the shadow generation unit 1345, the image quality improvement unit 1346, and the effect processing unit 1347.
  • the object generation unit 1342 generates a three-dimensional object, a background object, or the like (hereinafter referred to as an additional object) superimposed on the volumetric image provided to the user by using the input object (content).
  • the generated additional object is input to the virtual viewpoint video generation unit 1344.
  • the camera image update unit 1343 extracts the actual camera image 33 from the intermediate rendered image 30.
  • the actual camera image 33 incorporated in the intermediate rendered image 30 may be synchronized with the virtual viewpoint image (RGB), that is, the actual camera image at the same time.
  • the actual camera image 33 may be a two-dimensional image showing the subject 90 corresponding to the same time as the subject 90 used to generate the three-dimensional model.
  • the extracted real camera image 33 is input to the virtual viewpoint image generation unit 1344.
  • the virtual viewpoint video generation unit 1344 extracts the virtual viewpoint video (RGB) 31 and the virtual viewpoint video (depth) 32 from the intermediate rendered video 30, and the extracted virtual viewpoint video (RGB) 31 and virtual viewpoint video (depth) 32. Is used to reconstruct the 3D model. Further, the virtual viewpoint image generation unit 1344 superimposes the additional object input from the object generation unit 1342 on the reconstructed three-dimensional model, if necessary. That is, the virtual viewpoint video generation unit 1344 arranges the three-dimensional model and the additional object in the same virtual space. At that time, the virtual viewpoint image generation unit 1344 adjusts the positional relationship between the three-dimensional model and the additional object based on, for example, the depth information of the additional object based on the virtual viewpoint and the virtual viewpoint image (depth) 32. You may.
  • the virtual viewpoint image generation unit 1344 renders the reconstructed 3D model (and additional objects) using the virtual viewpoint based on the virtual viewpoint information (virtual viewpoint position and virtual viewpoint rotation information). Generate a volumetric image of the dimensional model (and additional objects).
  • the virtual viewpoint video generation unit 1344 generates video content by superimposing the actual camera image 33 input from the camera image update unit 1343 on a predetermined area of the generated volumetric video.
  • the video content generated thereby is, for example, a video in which a subject generated from the three-dimensional model and a subject based on the actual camera image 33 coexist, and can correspond to one example of the video in the claims.
  • the present invention is not limited to this, and the virtual viewpoint image generation unit 1344 arranges the real camera image 33 on a plane in the virtual space in which the 3D model (and additional object) is arranged, and arranges the 3D model (and additional object). ) And the actual camera image 33 may be rendered using a virtual viewpoint to directly generate the image content.
  • the synthetic model of the three-dimensional model (and the additional object) generated at that time and the actual camera image 33 can correspond to one example of the synthetic three-dimensional model in the claims.
  • the video content generated in this way is input to the shadow generation unit 1345.
  • the shadow generation unit 1345 casts a shadow on the object included in the volumetric video in the input video content. This makes it possible to enhance the reality of volumetric images. However, when the shadow is not added, the shadow generation unit 1345 may be omitted or invalidated.
  • the shadow generation unit 1345 is three-dimensional based on the virtual viewpoint image (depth) 32 included in the intermediate rendered image 30, the depth information of the additional object generated when the object generation unit 1342 generates the additional object, and the like. Add shadows to images of models and additional objects.
  • the video content in which the object is shaded is input to the image quality improvement unit 1346.
  • the light source information regarding the position and type (hue, etc.) of the light source used when generating the shadow is also input to the shadow generation unit 1345.
  • the light source information may be included in the virtual viewpoint information, or may be separately input to the shadow generation unit 1345.
  • the image quality improvement unit 1346 executes a process for improving the image quality of the input video content.
  • the image quality improvement unit 1346 performs volumetric processing such as blurring processing, alpha blending processing, noise reduction processing, and the like to blur the contours of a three-dimensional model and an image of an additional object included in the volumetric video in the video content. Suppresses time-series fluctuations that occur in images.
  • the image quality improving unit 1346 may use the auxiliary image 34 included in the intermediate rendered image 30 for the above processing.
  • the image quality improvement unit 1346 removes the color of the green background at the time of rendering so that the green background does not remain on the outline of the 3D model.
  • the image before the removal is referred to as an auxiliary image 34.
  • the image quality improvement unit 1346 is included in the volumetric image 40 by calculating the difference between the auxiliary image 34 and the volumetric image 40 generated by the virtual viewpoint image generation unit 1344.
  • the contour 41 of the image of the three-dimensional model (and the additional object) may be extracted, and based on the extracted contour 41, blurring processing, alpha blending processing, noise reduction processing, and the like may be performed on the volumetric image.
  • the image quality improving unit 1346 uses the volumetric image 40 and / or the auxiliary image 34 in order to substantially match the image of the three-dimensional model included in the volumetric image 40 with the image of the subject 90 included in the auxiliary image 34.
  • the scale, tilt, color, etc. of may be adjusted.
  • the effect processing unit 1347 can execute various effect processing on the video content. For example, the effect processing unit 1347 applies an effect such as an aura to the image of a 3D model or an additional object, applies an effect such as a mosaic to the actual camera image 33, or scatters petals as a background. Perform effect processing. As a result, the range of production can be further expanded. However, when the effect is not applied, the effect processing unit 1347 may be omitted or invalidated. Then, the effect processing unit 1347 outputs the video content to which the effect is applied to the transmission unit 14 as the final video content to be distributed to the user. As a result, the final video content transmitted from the transmission unit 14 is received by the reception unit of the user terminal and displayed on the display unit 16 of the user terminal.
  • an effect such as an aura to the image of a 3D model or an additional object
  • an effect such as a mosaic to the actual camera image 33, or scatters petals as a background.
  • Perform effect processing As a result, the range of production can be further expanded.
  • FIG. 14 is a diagram showing an example of video content delivered to a user in the present embodiment.
  • the actual camera image 33 (at the same time) synchronized with the volumetric image is superimposed on the volumetric image including the image 52 of the three-dimensional model.
  • the auxiliary video 34 synchronized with the volumetric video is also superimposed.
  • An effect 51 is applied to the auxiliary image 34 so that the projected subject 90 emits an aura.
  • FIG. 15 is a diagram showing another example of the volumetric video delivered to the user in the present embodiment.
  • the image 53 of the background object is superimposed on the image 52 of the three-dimensional model. That is, the volumetric image is generated with the background object arranged in the virtual space in which the three-dimensional model is arranged.
  • the actual camera image 33 (at the same time) synchronized with the volumetric image is superimposed on the volumetric image.
  • the image superimposed on the volumetric image is not limited to the actual camera image 33 (including the actual camera image by the auxiliary image 34), and various separately prepared images such as promotional videos, animations, and movie scenes are prepared separately. It may be image data.
  • the present embodiment it is possible to apply various effects to the volumetric video and the image data superimposed on the volumetric video. As a result, it becomes possible to further expand the range of production in the video content provided to the user.
  • the present embodiment it is possible to superimpose other three-dimensional objects, background objects, etc. on the volumetric video and display the presentation. As a result, it becomes possible to further expand the range of production in the video content provided to the user.
  • image data and additional objects superimposed on the volumetric video may be configured to be switched automatically or at any timing based on the user operation.
  • FIG. 16 is a system configuration diagram showing a specific example of the information processing system according to the present embodiment.
  • the information processing system 10 includes a cloud server 83, a post server 85, a distribution server 87, and a user terminal 89.
  • the user terminal 89 does not have to be included in the information processing system 10.
  • the actual camera image 81 for generating the volumetric image is acquired, for example, by shooting the subject 90 with one or more actual cameras 70 (see FIG. 2) installed in the volumetric studio 80.
  • the audio mixer 82 may generate audio data based on the sound signal recorded together with the actual camera image 81.
  • the volumetric studio 80 may be indoors or outdoors, and may be an environment in which the subject 90 can be photographed from one direction or multiple directions.
  • the actual camera image 81 acquired by the actual camera 70 and the audio data generated by the audio mixer 82 are input to the cloud server 83 arranged on the predetermined network.
  • the cloud server 83 for example, the data acquisition unit 11, the 3D model generation unit 12, the initial virtual viewpoint image generation unit 131, and the transmission unit 132 in the rendering unit 13 in the configuration shown in FIG. 1 are arranged.
  • the cloud server 83 generates a three-dimensional model of the subject 90 by using the 3D model generation unit 12 from the actual camera image 81 acquired by the data acquisition unit 11. Then, the cloud server 83 generates the virtual viewpoint video (RGB) 31 and the virtual viewpoint video (depth) 32 from the three-dimensional model in the initial virtual viewpoint video generation unit 131 of the rendering unit 13, and the generated virtual viewpoint video ( An intermediate rendered image 30 is generated from the RGB) 31 and the virtual viewpoint image (depth) 32, and the input real camera image 81 (corresponding to the actual camera image 33 and the auxiliary image 34).
  • RGB virtual viewpoint video
  • depth virtual viewpoint video
  • An intermediate rendered image 30 is generated from the RGB) 31 and the virtual viewpoint image (depth) 32, and the input real camera image 81 (corresponding to the actual camera image 33 and the auxiliary image 34).
  • the cloud server 83 is merely an example, and various information processing devices arranged on the network, such as a fog server and an edge server, may be used.
  • the generated intermediate rendered video 30 is transmitted from the transmitting unit 132 of the rendering unit 13 to the post server 85 via a predetermined network.
  • the post server 85 for example, the reception unit 133 in the rendering unit 13 and the final virtual viewpoint image generation unit 134 are arranged.
  • the post server 85 extracts the virtual viewpoint image (RGB) 31, the virtual viewpoint image (depth), the actual camera image 33, and the auxiliary image 34 from the intermediate rendered image 30, and based on these, the image finally presented to the user. Generate content 50.
  • the specific operation may be the same as that of the final virtual viewpoint image generation unit 134 described above.
  • the video content 50 generated in this way is converted into a distribution stream 88 corresponding to each user terminal 89 in the distribution server 87 in which the transmission unit 14 is arranged, and then is converted from the distribution server 87 via a predetermined network. It is delivered to the user terminal 89.
  • the user terminal 89 receives the distributed distribution stream 88 at the receiving unit 15, and restores the video content from the distribution stream 88. Then, the user terminal 89 displays the restored video content 50 on the display unit 16 toward the user.
  • the three-dimensional data is converted into the intermediate rendered image 30 which is the two-dimensional image data in the process executed by the rendering unit 13. Therefore, the data for generating the volumetric video can be transferred by HDMI (registered trademark) capture or codec compression, so that it is possible to reduce the traffic at the time of transmission and shorten the data transmission time. It becomes. As a result, it is possible to improve the real-time property in the distribution of video content including volumetric video.
  • HDMI registered trademark
  • processing related to the generation of the volumetric video for example, generation of the virtual viewpoint video, shadow generation, image quality improvement, Effect processing, etc.
  • processing related to the generation of the volumetric video can be distributed and executed on multiple computers. This makes it possible to reduce the load on each computer.
  • the processes related to the generation of the volumetric video can be freely assigned to the computers that are good at each process, so that the load on each computer can be further reduced.
  • the generation of the virtual viewpoint image (RGB) 31 and the virtual viewpoint image (depth) 32 is executed by a computer incorporating Linux (registered trademark) capable of GPU distributed processing, and shadow addition and image quality are subsequently described.
  • Improvements and effect processing can be sorted by executing them on a computer equipped with an OS (Operating System) that can use a library that can easily realize them. As a result, it is possible to shorten the overall processing time by efficiently executing each processing, and it is possible to improve the real-time performance in the distribution of video content including volumetric video.
  • OS Operating System
  • the program for realizing each part (each process) described in the above-described embodiment may be executed in any device.
  • the device may have the necessary functional blocks so that the necessary information can be obtained.
  • each step of one flowchart may be executed by one device, or may be shared and executed by a plurality of devices.
  • one device may execute the plurality of processes, or the plurality of devices may share and execute the plurality of processes.
  • a plurality of processes included in one step can be executed as processes of a plurality of steps.
  • the processes described as a plurality of steps can be collectively executed as one step.
  • a computer-executed program may allow the processing of the steps of writing the program to be executed in chronological order in the order described herein, in parallel, or in calls. It may be executed individually at the required timing such as when it is broken. That is, as long as there is no contradiction, the processes of each step may be executed in an order different from the above-mentioned order. Further, the processing of the step for describing this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.
  • the present technology can also have the following configurations. (1) Based on the 3D model of the subject generated by using a plurality of captured images obtained by imaging the subject and the 2D image, the subject generated from the 3D model and the 2D image exist at the same time.
  • An information processing device including a first generation unit that generates a moving image.
  • the two-dimensional image is a two-dimensional image using at least one captured image of the plurality of captured images used to generate a three-dimensional model of the subject.
  • Device Based on the 3D model of the subject generated by using a plurality of captured images obtained by imaging the subject and the 2D image, the subject generated from the 3D model and the 2D image exist at the same time.
  • An information processing device including a first generation unit that generates a moving image.
  • the two-dimensional image is a two-
  • the first generation unit generates the image based on the three-dimensional model of the subject and the two-dimensional image showing the subject corresponding to the same time as the subject used for generating the three-dimensional model.
  • the information processing apparatus according to any one of (1) to (3) above, comprising a second generation unit for generating a packing image in which a depth image obtained by converting depth information into a two-dimensional image is packed in one frame.
  • the packing image further includes at least one captured image among the plurality of captured images.
  • the texture image and the captured image included in the packing image are images at the same time.
  • the three-dimensional model of the subject is converted into the two-dimensional texture information based on the virtual viewpoint set in the virtual space in which the three-dimensional model is arranged.
  • the information processing apparatus according to any one of (1) to (9), further comprising a shadow generation unit that casts a shadow on the area of the subject included in the video.
  • the information processing apparatus according to any one of (1) to (10), further comprising an image quality improving unit for improving the image quality of the image using at least one captured image among the plurality of captured images.
  • the information processing apparatus according to any one of (1) to (11), further comprising an effect processing unit for applying effect processing to the video.
  • the information processing apparatus according to any one of (1) to (12), further comprising a transmission unit that transmits the video generated by the first generation unit to one or more user terminals via a predetermined network. ..
  • the first generation unit arranges a three-dimensional model of the subject generated by using a plurality of captured images obtained by imaging the subject in a three-dimensional space, and forms at least one of the plurality of captured images.
  • the above-mentioned one of (1) to (13) which generates a synthetic three-dimensional model including the three-dimensional model and the two-dimensional image by arranging the based two-dimensional image in the three-dimensional space.
  • Information processing device (15) The information processing according to (14) above, wherein the first generation unit generates the video by rendering the synthetic three-dimensional model based on a virtual viewpoint set in the virtual space in which the three-dimensional model is arranged. Device.
  • An information processing method that involves generating images in which are present at the same time.
  • a three-dimensional model of the subject is generated using a plurality of captured images obtained by imaging the subject. Based on the three-dimensional model of the subject and the two-dimensional image, an image in which the subject generated from the three-dimensional model and the two-dimensional image coexist is generated.
  • a video distribution method including distributing the video to a user terminal via a predetermined network.
  • An image pickup device that generates a plurality of captured images of the subject by imaging the subject, and an image pickup device.
  • Information that generates an image in which the subject generated from the three-dimensional model and the two-dimensional image coexist based on the three-dimensional model of the subject generated by using the plurality of captured images and the two-dimensional image.
  • a user terminal that displays the video generated by the information processing device to the user, and a user terminal.
  • Information processing system equipped with.
  • Information processing system 11 Data acquisition unit 12 3D model generation unit 13 Rendering unit 14 Transmitter unit 15 Receiver unit 16 Display unit 20 Drive 21 CPU 22 ROM 23 RAM 24 Bus 25 Input / output interface 26 Input section 27 Output section 28 Storage section 29 Communication section 30 Intermediate rendering video 31 Virtual viewpoint video (RGB) 32 Virtual viewpoint video (depth) 33, 81 Real camera video 34 Auxiliary video 40 Volumetric video 41 Contour 50, 50A, 50B Video content 51 Effect 52 3D model image 53 Background object image 70, 70a, 70b, 70c, 70d, 70e, ...
  • Real camera 80 Volumetric Studio 82 Audio Mixer 83 Cloud Server 85 Post Server 87 Distribution Server 88 Distribution Stream 89 User Terminal 90 Subject 131 Initial Virtual Viewpoint Video Generator 132 Transmission Unit 133 Receiver 134 Final Virtual Viewpoint Video Generation Unit 1312 Virtual Viewpoint Video ( RGB) Generation unit 1313 Auxiliary image generation unit 1314 Virtual viewpoint image (depth) generation unit 1315 Intermediate rendering image generation unit 1342 Object generation unit 1343 Camera image update unit 1344 Virtual viewpoint image generation unit 1345 Shadow generation unit 1346 Image quality improvement unit 1347 Effect processing Department

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Graphics (AREA)
  • General Physics & Mathematics (AREA)
  • Geometry (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Processing Or Creating Images (AREA)
  • Image Generation (AREA)
PCT/JP2021/026587 2020-07-30 2021-07-15 情報処理装置、情報処理方法、映像配信方法及び情報処理システム Ceased WO2022024780A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP2022540167A JPWO2022024780A1 (cg-RX-API-DMAC7.html) 2020-07-30 2021-07-15
US18/014,249 US12260491B2 (en) 2020-07-30 2021-07-15 Information processing device, information processing method, video distribution method, and information processing system
CN202180049906.0A CN116075860B (zh) 2020-07-30 2021-07-15 信息处理装置、信息处理方法、视频分发方法和信息处理系统
EP21851014.7A EP4178199A4 (en) 2020-07-30 2021-07-15 Information processing device, information processing method, video distribution method, and information processing system

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
JP2020-129414 2020-07-30
JP2020129414 2020-07-30
JP2020133615 2020-08-06
JP2020-133615 2020-08-06

Publications (1)

Publication Number Publication Date
WO2022024780A1 true WO2022024780A1 (ja) 2022-02-03

Family

ID=80036386

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/026587 Ceased WO2022024780A1 (ja) 2020-07-30 2021-07-15 情報処理装置、情報処理方法、映像配信方法及び情報処理システム

Country Status (5)

Country Link
US (1) US12260491B2 (cg-RX-API-DMAC7.html)
EP (1) EP4178199A4 (cg-RX-API-DMAC7.html)
JP (1) JPWO2022024780A1 (cg-RX-API-DMAC7.html)
CN (1) CN116075860B (cg-RX-API-DMAC7.html)
WO (1) WO2022024780A1 (cg-RX-API-DMAC7.html)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023189580A1 (ja) * 2022-03-31 2023-10-05 ソニーグループ株式会社 画像処理装置及び画像処理システム
JP2024035671A (ja) * 2022-09-02 2024-03-14 ジオビンゴ合同会社 情報処理方法、プログラム、情報処理装置

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12675962B2 (en) * 2021-03-11 2026-07-07 Quintar, Inc. Alignment of 3D graphics extending beyond frame in augmented reality system with remote presentation
US12028507B2 (en) * 2021-03-11 2024-07-02 Quintar, Inc. Augmented reality system with remote presentation including 3D graphics extending beyond frame
US12423915B2 (en) * 2021-04-30 2025-09-23 Nippon Telegraph And Telephone Corporation Distribution control system, distribution control apparatus, distribution control method, and program
CN113891057A (zh) * 2021-11-18 2022-01-04 北京字节跳动网络技术有限公司 视频的处理方法、装置、电子设备和存储介质
US12651456B1 (en) 2025-07-09 2026-06-09 MXV Inc. System for generating a real-time object-focused video

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017082076A1 (ja) * 2015-11-11 2017-05-18 ソニー株式会社 符号化装置および符号化方法、復号装置および復号方法
WO2017191701A1 (ja) * 2016-05-02 2017-11-09 株式会社ソニー・インタラクティブエンタテインメント 表示制御装置、表示制御方法及びプログラム
JP2018049591A (ja) * 2016-09-14 2018-03-29 キヤノン株式会社 画像処理装置、画像生成方法及びプログラム
WO2019021375A1 (ja) 2017-07-25 2019-01-31 富士通株式会社 映像生成プログラム、映像生成方法および映像生成装置
JP2019106617A (ja) * 2017-12-12 2019-06-27 キヤノン株式会社 情報処理装置、情報処理方法及びプログラム

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2465791A (en) * 2008-11-28 2010-06-02 Sony Corp Rendering shadows in augmented reality scenes
US9407904B2 (en) * 2013-05-01 2016-08-02 Legend3D, Inc. Method for creating 3D virtual reality from 2D images
JP2014067372A (ja) * 2012-09-27 2014-04-17 Kddi Corp 運動視差再現プログラムおよび装置ならびにその三次元モデル生成プログラムおよび装置
EP3565259A4 (en) * 2016-12-28 2019-11-06 Panasonic Intellectual Property Corporation of America DISTRIBUTION METHOD FOR THREE-DIMENSIONAL MODEL, RECEIVING METHOD FOR THREE-DIMENSIONAL MODEL, DISTRIBUTION DEVICE FOR THREE-DIMENSIONAL MODEL AND RECEIVING DEVICE FOR THREE-DIMENSIONAL MODEL
JP7200678B2 (ja) * 2017-02-20 2023-01-10 ソニーグループ株式会社 画像処理装置および画像処理方法
US11514613B2 (en) * 2017-03-16 2022-11-29 Samsung Electronics Co., Ltd. Point cloud and mesh compression using image/video codecs
US20210134049A1 (en) * 2017-08-08 2021-05-06 Sony Corporation Image processing apparatus and method
US10887574B2 (en) * 2018-07-31 2021-01-05 Intel Corporation Selective packing of patches for immersive video
JP7418101B2 (ja) * 2019-07-26 2024-01-19 キヤノン株式会社 情報処理装置、情報処理方法、及びプログラム

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017082076A1 (ja) * 2015-11-11 2017-05-18 ソニー株式会社 符号化装置および符号化方法、復号装置および復号方法
WO2017191701A1 (ja) * 2016-05-02 2017-11-09 株式会社ソニー・インタラクティブエンタテインメント 表示制御装置、表示制御方法及びプログラム
JP2018049591A (ja) * 2016-09-14 2018-03-29 キヤノン株式会社 画像処理装置、画像生成方法及びプログラム
WO2019021375A1 (ja) 2017-07-25 2019-01-31 富士通株式会社 映像生成プログラム、映像生成方法および映像生成装置
JP2019106617A (ja) * 2017-12-12 2019-06-27 キヤノン株式会社 情報処理装置、情報処理方法及びプログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4178199A4

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023189580A1 (ja) * 2022-03-31 2023-10-05 ソニーグループ株式会社 画像処理装置及び画像処理システム
JP2024035671A (ja) * 2022-09-02 2024-03-14 ジオビンゴ合同会社 情報処理方法、プログラム、情報処理装置
JP7663242B2 (ja) 2022-09-02 2025-04-16 ジオビンゴ合同会社 情報処理方法、プログラム、情報処理装置

Also Published As

Publication number Publication date
CN116075860A (zh) 2023-05-05
EP4178199A4 (en) 2023-11-08
EP4178199A1 (en) 2023-05-10
JPWO2022024780A1 (cg-RX-API-DMAC7.html) 2022-02-03
US12260491B2 (en) 2025-03-25
US20230260199A1 (en) 2023-08-17
CN116075860B (zh) 2026-02-03

Similar Documents

Publication Publication Date Title
US12260491B2 (en) Information processing device, information processing method, video distribution method, and information processing system
US20240296626A1 (en) Method, apparatus, electronic device and storage medium for reconstructing 3d images
US11019259B2 (en) Real-time generation method for 360-degree VR panoramic graphic image and video
US12034787B2 (en) Hybrid streaming
CA2948903C (en) Method, system and apparatus for generation and playback of virtual reality multimedia
US10049495B2 (en) System and method for providing virtual reality content
US11769299B1 (en) Systems and methods for capturing, transporting, and reproducing three-dimensional simulations as interactive volumetric displays
CN108366293A (zh) Vr视频转码方法及装置
WO2022191070A1 (ja) 3dオブジェクトのストリーミング方法、装置、及びプログラム
JP7505481B2 (ja) 画像処理装置および画像処理方法
KR102417959B1 (ko) 3차원 입체 콘텐츠 제공 장치 및 그 방법
Nomoto et al. Dynamic multi-projection mapping based on parallel intensity control
TW202141418A (zh) 處理拆分渲染中的遮蔽之方法及裝置
CN117596373A (zh) 基于动态数字人形象进行信息展示的方法及电子设备
CN104601950B (zh) 一种视频监控方法
CN116485966A (zh) 视频画面渲染方法、装置、设备和介质
US20260045023A1 (en) Information processing system, operation method of information processing system, and program
JP5200141B2 (ja) 映像提示システム、映像提示方法、プログラム及び記録媒体
Xu et al. SplatBus: A Gaussian Splatting Viewer Framework via GPU Interprocess Communication
Scheer et al. A client-server architecture for real-time view-dependent streaming of free-viewpoint video
JP4929848B2 (ja) 映像データの伝送システムとその方法、送信処理装置とその方法
WO2021200261A1 (ja) 情報処理装置、生成方法、およびレンダリング方法
CN119991526A (zh) 面向城市级数字孪生的协同渲染方法、设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21851014

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022540167

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 2021851014

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

WWG Wipo information: grant in national office

Ref document number: 18014249

Country of ref document: US