WO2021140956A1 - 情報処理装置および方法 - Google Patents

情報処理装置および方法 Download PDF

Info

Publication number
WO2021140956A1
WO2021140956A1 PCT/JP2020/048726 JP2020048726W WO2021140956A1 WO 2021140956 A1 WO2021140956 A1 WO 2021140956A1 JP 2020048726 W JP2020048726 W JP 2020048726W WO 2021140956 A1 WO2021140956 A1 WO 2021140956A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
comparison
display
objects
unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/048726
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
由佳 木山
遼平 高橋
石川　毅
平林　光浩
木村　淳
久野　浩
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Priority to US17/790,680 priority Critical patent/US20230043591A1/en
Priority to JP2021570018A priority patent/JPWO2021140956A1/ja
Publication of WO2021140956A1 publication Critical patent/WO2021140956A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—Three-dimensional [3D] animation
    • G06T13/40—Three-dimensional [3D] animation of characters, e.g. humans, animals or virtual beings
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23412—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H04N21/23614—Multiplexing of additional data and video streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/8146—Monomedia components thereof involving graphical data, e.g. 3D object, 2D graphics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/85—Assembly of content; Generation of multimedia applications
    • H04N21/854—Content authoring
    • H04N21/85406—Content authoring involving a specific file format, e.g. MP4 format
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/61—Scene description
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/028—Multiple view windows (top-side-front-sagittal-orthogonal)

Definitions

  • the present disclosure relates to information processing devices and methods, and more particularly to information processing devices and methods that make it easier to compare 3D objects.
  • 3D content also referred to as 3D content
  • 3D space also referred to as 3D space
  • 3D content for example, a 6DoF content that represents a three-dimensional object in a three-dimensional space and can freely set the line-of-sight direction and the viewpoint position during reproduction has been proposed.
  • Non-Patent Document 1 As a method of delivering 6DoF content, for example, a method of composing a 3D space with a plurality of 3D objects and transmitting it as a plurality of object streams has been proposed. Then, at that time, for example, it was proposed to use a description method called a scene description (see, for example, Non-Patent Document 1).
  • this description method describes a method in which two 3D objects (3D objects) included in each of a plurality of 6DoF contents are arranged side by side or overlapped and played back at the same timing, and both objects are displayed in a comparable manner.
  • 3D objects 3D objects
  • This disclosure was made in view of such a situation, and makes it easier to compare 3D objects.
  • the information processing device on one aspect of the present technology is an image processing device including a comparison information generation unit that generates comparison information, which is information for displaying a plurality of 3D objects of 6DoF contents in a comparable manner.
  • the information processing method of one aspect of the present technology is an image processing method that generates comparative information, which is information for displaying a plurality of 3D objects of 6DoF contents in a comparable manner.
  • comparison information which is information for displaying a plurality of 3D objects of 6DoF contents in a comparable manner is generated.
  • Non-Patent Document 1 (above)
  • Non-Patent Document 2 ISO / IEC 14496-12: 2015, Information technology. Coding of audio-visual objects. Part 12, ISO base media file format
  • Non-Patent Document 3 ISO / IEC 23009-1: 2014, Information technology, Dynamic adaptive streaming over HTTP (DASH), Part 1, Media presentation description and segment formats
  • Non-Patent Document 4 Khronos glTF2.0, Khronos glTF2.0, https://github.com/KhronosGroup/glTF/tree/master/specification/2.0
  • Non-Patent Document 2 terms used in the MPEG-DASH standard described in Non-Patent Document 3, and "camera” object and “animation” described in Non-Patent Document 4.
  • Structures / terms used in objects, etc. are within the scope of this disclosure and meet the support requirements of the claims, even if they are not directly defined herein.
  • technical terms such as Parsing, Syntax, and Semantics are also within the scope of the present disclosure even if they are not directly defined in the present specification. It shall meet the support requirements of the claims.
  • 6DoF content can be looked around in all directions in a three-dimensional space (also called 3D space) (the line-of-sight direction can be set freely), and can be moved in the space (viewpoint position can be set freely). ) Content.
  • the video included in the 6DoF content that can freely set the viewpoint position and line-of-sight direction as described above is also called a 6DoF video.
  • 6DoF content it is possible to focus on a certain 3D object (3D object) included in the content and change the viewpoint position and the line-of-sight direction so that the 3D object is viewed from the surroundings.
  • a 3D object contained in a certain 6DoF content and a 3D object contained in another 6DoF content are displayed side by side or overlapped, and are played back at the same timing, and the movements of both objects are moved from the surroundings. It is possible to realize a viewing experience such as seeing and comparing. For example, in sports content, it is possible to have a viewing experience in which the movements of two famous athletes (pitching form, batting form, etc.) are compared from the surroundings.
  • Non-Patent Document 1 a description method called a scene description.
  • this description method could not describe the information on how to display the 3D objects in a comparable manner as described above. Therefore, when displaying such multiple objects in a comparable manner, select two comparable contents, match the playback timing of those contents, and make adjustments so that the appearance of each content is the same. It was necessary to perform complicated work for playback control by the user or the like. In particular, in the case of 6DoF content, there is a possibility that more complicated work may be required for adjustment for comparison due to the high degree of freedom that the content can be viewed from any position.
  • Multivideo https://dotapps.jp/products/com-tattin-multivideo-0002
  • two videos are selected from the existing content and overlapped. It can be played side by side at the same time.
  • the content to be compared in this application is a two-dimensional image, and it is not possible to compare 3D objects of 6DoF content.
  • it is necessary for the user to perform complicated work such as selecting the content to be compared and specifying the reproduction timing, the area and size of each video, and the like.
  • Non-Patent Document 1 to Non-Patent Document 4 described above disclosed a method for describing a method for displaying 3D objects in a comparable manner.
  • comparison information which is information for displaying multiple 3D objects of 6DoF content in a comparable manner.
  • FIG. 1 is a block diagram showing an example of a configuration of a distribution system, which is an aspect of an information processing system to which the present technology is applied.
  • the distribution system 100 shown in FIG. 1 is a system that distributes 6DoF contents.
  • the distribution system 100 includes a generation device 101, a server 102, and a client device 103.
  • the generation device 101, the server 102, and the client device 103 are communicably connected to each other via the network 104.
  • the distribution system 100 can have an arbitrary number of each device. That is, the generation device 101, the server 102, and the client device 103 may each have a plurality of units.
  • the generation device 101 performs processing related to the generation of 6DoF content.
  • the generator 101 uses media data such as 6DoF video, scene description (SceneDescription) which is the metadata thereof, and MPD (ISO / IEC 23009-1) of DASH (Dynamic Adaptive Streaming over HTTP, ISO / IEC 23009-1). MediaPresentationDescription) etc. can be generated.
  • the generation device 101 may supply (upload) the generated data to the server 102 via the network 104.
  • Server 102 performs processing related to distribution of 6DoF content.
  • the server 102 may acquire the data of the 6DoF content described above supplied by the generation device 101.
  • the server 102 can manage the acquired data and provide a service for delivering 6DoF content.
  • the server 102 may distribute 6DoF content data (MPD, scene description, media data, etc.) to the client device 103 or the like via the network 104 in response to a request from the client device 103 or the like.
  • 6DoF content data MPD, scene description, media data, etc.
  • the client device 103 performs processing related to playback of 6DoF content.
  • the client device 103 may request the server 102 to provide 6DoF content via the network 104.
  • the client device 103 can acquire 6DoF content data (MPD, scene description, media data, etc.) distributed from the server 102 in response to the request via the network 104.
  • the client device 103 can reproduce the acquired 6DoF content.
  • the client device 103 can perform rendering to generate an image in a desired viewpoint position and line-of-sight direction, and display the image on a monitor.
  • the network 104 is a communication network composed of an arbitrary communication medium.
  • the communication performed via the network 104 may be wired communication, wireless communication, or both. That is, the network 104 may be a communication network for wired communication, a communication network for wireless communication, or a communication network composed of both of them. Further, the network 104 may be composed of a single communication network or may be composed of a plurality of communication networks.
  • the Internet may be included in this network 104.
  • the public telephone line network may be included in this network 104.
  • a wide area communication network for wireless mobiles such as a so-called 3G line or 4G line may be included in the network 104.
  • WAN Wide Area Network
  • LAN Local Area Network
  • the network 104 may include a wireless communication network that performs communication conforming to the Bluetooth (registered trademark) standard.
  • a communication path for short-range wireless communication such as NFC (Near Field Communication) may be included in the network 104.
  • the communication path for infrared communication may be included in the network 104.
  • the network 104 may include a communication network for wired communication conforming to standards such as HDMI (registered trademark) (High-Definition Multimedia Interface) and USB (registered trademark) (Universal Serial Bus). In this way, the network 104 may include a communication network or a communication path of any communication standard.
  • HDMI registered trademark
  • USB registered trademark
  • the client device 103 can arrange and overlap 3D objects of a plurality of 6DoF contents to reproduce and display them in a comparable manner. Then, the generation device 101 generates and signals comparison information which is information for performing a comparable display of such a 3D object. The client device 103 performs the above-mentioned reproduction / display based on the comparison information. As a result, the user of the client device 103 can more easily compare 3D objects without requiring complicated work.
  • FIG. 2 is a block diagram showing a main configuration example of the generation device 101. It should be noted that FIG. 2 shows the main things such as the processing unit and the data flow, and not all of them are shown in FIG. That is, in the generation device 101, there may be a processing unit that is not shown as a block in FIG. 2, or there may be a processing or data flow that is not shown as an arrow or the like in FIG. As shown in FIG. 2, the generation device 101 has a control unit 111 and a generation processing unit 112.
  • the control unit 111 performs processing related to the control of the generation processing unit 112.
  • the generation processing unit 112 is controlled by the control unit 111 to perform processing related to data generation of 6DoF contents.
  • the generation processing unit 112 includes a data input unit 121, a preprocessing unit 122, an encoding unit 123, a comparison information generation unit 124, a file generation unit 125, an MPD generation unit 126, a storage unit 127, and It has an upload unit 128.
  • the data input unit 121 performs processing related to media data input.
  • the data input unit 121 can receive media data and scene descriptions input from the outside (for example, another device) and supply them to the preprocessing unit 122.
  • the data input unit 121 may generate media data or a scene description of 6DoF content.
  • the data input unit 121 may have a camera or the like, image a subject with the camera or the like, generate media data from the captured image, and further generate a scene description corresponding to the media data.
  • the pre-processing unit 122 performs processing related to pre-processing for media data.
  • the preprocessing unit 122 can acquire the media data and the scene description supplied from the data input unit 121.
  • the pre-processing unit 122 can appropriately perform image processing or the like on the acquired media data, and can appropriately edit the scene description.
  • the data input unit 121 may supply the media data to the preprocessing unit 122, and the preprocessing unit 122 may generate a scene description corresponding to the media data.
  • the preprocessing unit 122 can supply the media data and the scene description to the coding unit 123 and the comparison information generation unit 124.
  • the coding unit 123 performs processing related to coding. For example, the coding unit 123 can acquire the media data and the scene description supplied from the preprocessing unit 122. In addition, the coding unit 123 can encode the media data and generate the coded data. The coding method is arbitrary. Further, the coding unit 123 can supply the coded data and the scene description of the generated media data to the file generation unit 125.
  • the comparison information generation unit 124 performs processing related to generation of comparison information, which is information for reproducing and displaying 3D objects in a comparable manner.
  • the comparison information generation unit 124 can acquire the media data and the scene description supplied from the preprocessing unit 122. Further, the comparison information generation unit 124 can generate comparison information based on the data. Further, the comparison information generation unit 124 can supply the generated comparison information to the file generation unit 125.
  • the file generation unit 125 performs processing related to the generation of a file that summarizes the data of 6DoF contents. For example, the file generation unit 125 can acquire the coded data and the scene description of the media data supplied from the coding unit 123. Further, the file generation unit 125 can acquire the comparison information supplied from the comparison information generation unit 124. Further, the file generation unit 125 can file the data, that is, generate a file containing the data. At that time, the file generation unit 125 may include the comparison information in the scene description. That is, the file generation unit 125 may generate a file including the encoded data of the media data and the scene description including the comparison information. Further, the file generation unit 125 can supply the generated file to the MPD generation unit 126 and the storage unit 127.
  • the MPD generation unit 126 performs processing related to MPD generation. For example, the MPD generation unit 126 can acquire the file supplied from the file generation unit 125. Further, the MPD generation unit 126 corresponds to the file (media data, scene description, comparison information, etc. stored in the file) based on the media data, scene description, comparison information, etc. stored in the file. MPD can be generated. For example, the MPD generation unit 126 can generate an MPD including comparative information. Further, the MPD generation unit 126 can supply the generated MPD to the storage unit 127.
  • the storage unit 127 performs processing related to storage of 6DoF contents. For example, the storage unit 127 can acquire the file supplied from the file generation unit 125. In addition, the storage unit 127 can store the acquired file. Further, the storage unit 127 can acquire the MPD supplied from the MPD generation unit 126. In addition, the storage unit 127 can store the acquired MPD. Further, the storage unit 127 can supply the stored file or MPD to the upload unit 128 at a predetermined timing or in response to a predetermined event, request, or the like.
  • the upload unit 128 performs processing related to uploading 6DoF content.
  • the upload unit 128 can acquire a file or MPD supplied from the storage unit 127.
  • the upload unit 128 can communicate with the server 102 via the network 104 and transmit (upload) the acquired file or MPD to the server 102.
  • the comparison information generation unit 124 generates comparison information.
  • the file generation unit 125 can file the comparison information.
  • the MPD generation unit 126 can generate an MPD that reflects the comparison information.
  • the upload unit 128 can upload a file including the comparison information and an MPD reflecting the comparison information. That is, comparison information can be signaled. Therefore, the user of the client device 103 can make the comparison of 3D objects easier.
  • each of these processing units has an arbitrary configuration.
  • each processing unit may be configured by a logic circuit that realizes the above-mentioned processing.
  • each processing unit has, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and the above-mentioned processing is realized by executing a program using them. You may do so.
  • each processing unit may have both configurations, and a part of the above-mentioned processing may be realized by a logic circuit, and the other may be realized by executing a program.
  • the configurations of the respective processing units may be independent of each other. For example, some processing units realize a part of the above-mentioned processing by a logic circuit, and some other processing units execute a program.
  • the above-mentioned processing may be realized by the other processing unit by both the logic circuit and the execution of the program.
  • FIG. 3 is a block diagram showing a main configuration example of the client device 103. It should be noted that FIG. 3 shows the main things such as the processing unit and the data flow, and not all of them are shown in FIG. That is, in the client device 103, there may be a processing unit that is not shown as a block in FIG. 3, or there may be a processing or data flow that is not shown as an arrow or the like in FIG. As shown in FIG. 3, the client device 103 has a control unit 151 and a reproduction processing unit 152. The control unit 151 performs processing related to the control of the reproduction processing unit 152. The playback processing unit 152 is controlled by the control unit 151 to perform processing related to playback of 6DoF content.
  • the reproduction processing unit 152 includes an MPD processing unit 161, a data acquisition control unit 162, a comparison information acquisition unit 163, a display control unit 164, a coded data acquisition unit 165, a decoding unit 166, and a buffer 167. It has a display information generation unit 168 and a display unit 169.
  • MPD processing unit 161 performs processing related to MPD.
  • the MPD processing unit 161 can acquire an MPD corresponding to a desired 6DoF content specified by a user, an application, or the like.
  • the MPD processing unit 161 communicates with the server 102 via the network 104, requests the server 102 for the MPD corresponding to the desired 6DoF content, and acquires the MPD supplied in response to the request. Can be done. Further, the MPD processing unit 161 can parse the acquired MPD and supply the result to the data acquisition control unit 162.
  • the data acquisition control unit 162 performs processing related to control of 6DoF content acquisition.
  • the data acquisition control unit 162 can acquire the parse result of the MPD supplied from the MPD processing unit 161. Further, the data acquisition control unit 162 can control the acquisition of comparison information and scene description and the acquisition of encoded data of media data based on the parsing result (that is, the content of MPD).
  • the data acquisition control unit 162 can control the comparison information acquisition unit 163 to control which comparison information and scene description are to be acquired. Further, the data acquisition control unit 162 can acquire the comparison information acquired by the comparison information acquisition unit 163.
  • the data acquisition control unit 162 controls the coded data acquisition unit 165 based on the MPD parsing result (that is, the content of the MPD) and the comparison information, and determines which 6DoF content coded data is to be acquired. Can be controlled.
  • the comparison information acquisition unit 163 performs processing related to acquisition of comparison information.
  • the comparison information acquisition unit 163 can acquire the comparison information corresponding to the MPD acquired by the MPD processing unit 161 under the control of the data acquisition control unit 162.
  • the comparison information acquisition unit 163 communicates with the server 102 via the network 104, requests the server 102 for the desired comparison information specified by the data acquisition control unit 162, and is supplied in response to the request. Comparison information can be obtained.
  • the comparison information acquisition unit 163 can also acquire the scene description corresponding to the comparison information in the same manner. For example, when the comparison information is included in the scene description, the comparison information acquisition unit 163 can acquire the scene description including the comparison information in the same manner. Further, the comparison information acquisition unit 163 can supply the acquired comparison information (or scene description) to the data acquisition control unit 162 and the display control unit 164.
  • the display control unit 164 performs processing related to controlling the display of 6DoF content. For example, the display control unit 164 can acquire the comparison information (or the scene description) supplied from the comparison information acquisition unit 163. In addition, the display control unit 164 generates display control information that controls the playback / display of 6DoF content so that the 3D object can be reproduced and displayed in a comparable manner based on the comparison information (or scene description). , It can be supplied to buffer 167.
  • the coded data acquisition unit 165 performs processing related to the acquisition of coded data.
  • the coded data acquisition unit 165 is a medium corresponding to the MPD acquired by the MPD processing unit 161 and the comparison information (or scene description) acquired by the comparison information acquisition unit 163 under the control of the data acquisition control unit 162.
  • Data encoding Data can be acquired. That is, the coded data acquisition unit 165 can acquire the coded data of the media data of the 3D object to be compared, for example.
  • the coded data acquisition unit 165 communicates with the server 102 via the network 104, requests the data of the desired 6DoF content specified by the data acquisition control unit 162 from the server 102, and responds to the request. The coded data supplied in the above can be obtained. Further, the coded data acquisition unit 165 can supply the acquired coded data to the decoding unit 166.
  • the decoding unit 166 performs processing related to decoding the coded data. For example, the decoding unit 166 can acquire the coded data of the media data supplied from the coded data acquisition unit 165. Further, the decoding unit 166 decodes the coded data and generates (restores) the media data. It should be noted that this decoding method is arbitrary and may correspond to the coding method of the coding unit 123. Further, the decoding unit 166 can supply the generated (restored) media data to the buffer 167.
  • Buffer 167 performs processing related to data retention.
  • the buffer 167 can acquire and hold the media data supplied from the decoding unit 166.
  • the buffer 167 can acquire and hold the display control information supplied from the display control unit 164.
  • the buffer 167 can supply the retained information to the display information generation unit 168 at a predetermined timing or in response to a predetermined event, request, or the like.
  • the display information generation unit 168 performs processing related to the generation of display information such as image data to be displayed and its metadata. For example, the display information generation unit 168 can read and acquire media data, display control information, and the like held in the buffer 167. Further, the display information generation unit 168 can generate display information based on the data read from the buffer 167. Further, the display information generation unit 168 can supply the generated display information to the display unit 169.
  • the display unit 169 has a monitor, and performs processing related to display using the monitor. For example, the display unit 169 can acquire the display information supplied from the display information generation unit 168. In addition, the display unit 169 can display the display information on the monitor.
  • the comparison information acquisition unit 163 acquires the comparison information, and the display control unit 164 generates the display control information based on the comparison information. That is, the display control unit 164 controls the display of the 6DoF content so that a plurality of 3D objects of the 6DoF content are displayed in a comparable manner.
  • the display information generation unit 168 generates display information for displaying a plurality of 3D objects of 6DoF contents in a comparable manner based on the display control information, and displays the display information on the display unit 169.
  • the display unit 169 can display a plurality of 3D objects of 6DoF contents in a comparable manner. In this way, the client device 103 can more easily reproduce and display a plurality of 3D objects in a comparable manner based on the signaled comparison information. That is, the user can more easily compare 3D objects.
  • Each of these processing units (MPD processing unit 161 to display unit 169) of the client device 103 has an arbitrary configuration.
  • each processing unit may be configured by a logic circuit that realizes the above-mentioned processing.
  • each processing unit may have, for example, a CPU, ROM, RAM, etc., and execute a program using them to realize the above-mentioned processing.
  • each processing unit may have both configurations, and a part of the above-mentioned processing may be realized by a logic circuit, and the other may be realized by executing a program.
  • the configurations of the respective processing units may be independent of each other. For example, some processing units realize a part of the above-mentioned processing by a logic circuit, and some other processing units execute a program.
  • the above-mentioned processing may be realized by the other processing unit by both the logic circuit and the execution of the program.
  • the generation device 101 generates, for example, data as shown in FIG. 4 as 6DoF content data.
  • FIG. 4 is a diagram showing an example of distribution data distributed as 6DoF content.
  • the distribution data is 18 3D objects (OBJ1-1 to OBJ1-3, OBJ2-1 to OBJ2-3, OBJ3-1 to OBJ3-3, OBJ4-1 to OBJ4-3, OBJ5. -1 to OBJ5-3, OBJ6-1 to OBJ6-3) media data is included.
  • 3D objects are different 3D objects from each other.
  • the distribution data includes 6 scene descriptions (SD1 to SD6).
  • SD1 is a scene description corresponding to the media data of three 3D objects (OBJ1-1, OBJ1-2, OBJ1-3).
  • SD2 is a scene description corresponding to the media data of three 3D objects (OBJ2-1, OBJ2-2, OBJ2-3).
  • SD3 is a scene description corresponding to the media data of three 3D objects (OBJ3-1, OBJ3-2, OBJ3-3).
  • SD4 is a scene description corresponding to the media data of three 3D objects (OBJ4-1, OBJ4-2, OBJ4-3).
  • SD5 is a scene description corresponding to the media data of three 3D objects (OBJ5-1, OBJ5-2, OBJ5-3).
  • SD6 is a scene description corresponding to the media data of three 3D objects (OBJ6-1, OBJ6-2, OBJ6-3).
  • the distribution data includes comparison information for each scene description.
  • Compare_data_ext1 is the comparison information corresponding to SD1.
  • Compare_data_ext2 is the comparison information corresponding to SD2.
  • Compare_data_ext3 is the comparison information corresponding to SD3.
  • Compare_data_ext4 is the comparison information corresponding to SD4.
  • Compare_data_ext5 is the comparison information corresponding to SD5.
  • Compare_data_ext6 is the comparison information corresponding to SD6.
  • the distribution data includes the MPD corresponding to these data. That is, the MPD includes information for accessing each scene description (SD1 access information, SD2 access information, SD3 access information, SD4 access information, SD5 access information, and SD6 access information).
  • SD1 access information SD2 access information
  • SD3 access information SD4 access information
  • SD5 access information SD6 access information
  • MPD is information for accessing the media data of each 3D object (OBJ1-1 access information, OBJ1-2 access information, OBJ1-3 access information, OBJ2-1 access information, OBJ2-2 access information, OBJ2- 3 Access information, OBJ3-1 access information, OBJ3-2 access information, OBJ3-3 access information, OBJ4-1 access information, OBJ4-2 access information, OBJ4-3 access information, OBJ5-1 access information, OBJ5-2 access Information, OBJ5-3 access information, OBJ6-1 access information, OBJ6-2 access information, and OBJ6-3 access information).
  • This comparison information may include comparison identification information which is information about the 3D object to be compared.
  • the comparison information includes media data of a comparable 3D object and identification information (compare id) of a group of scene descriptions (also referred to as a comparable group) corresponding to the 3D object as the comparison identification information. That is, "compare id" is information indicating a comparable group.
  • the generation device 101 If there is no such comparison identification information, it is necessary for the user or the like to perform complicated work such as specifying which of the 18 3D objects in FIG. 4 can be compared. As described above, the generation device 101 generates and signals this comparative identification information, so that the client device 103 can compare which 3D object can be compared with which 3D object. It becomes possible to easily identify based on the identification information.
  • this comparative identification information can also be described in the MPD.
  • the comparative identification information corresponding to the scene description may be described in association with the access information to each scene description.
  • the comparison information may include comparison display information which is information regarding the display of comparable 3D objects. That is, the comparison display information is information on how to reproduce and display the 3D object when the 3D object is reproduced and displayed in a comparable manner.
  • the configuration of the distribution data is arbitrary and is not limited to the example of FIG.
  • the number of 3D objects and scene descriptions is arbitrary.
  • the number of 3D objects corresponding to each scene description is arbitrary.
  • the comparison information is generated for each scene description, the number is not limited to the example of FIG.
  • comparable 3D objects are also optional. That is, the comparable group composed of the distribution data is not limited to the example of FIG. Further, the value of compare id is also arbitrary and is not limited to x, y, and z in FIG.
  • the comparable display of 3D objects is arbitrary.
  • a plurality of 6DoF images including a 3D object to be compared may be displayed side by side. That is, in this case, the comparison information includes information for performing such a display.
  • the 6DoF video 201 including the object 201A which is the 3D object to be compared and the 6DoF video 202 including the object 202A which is also the 3D object to be compared are displayed side by side.
  • the state of the bat swing by the object 201A and the state of the bat swing by the object 202A can be compared.
  • one 3D object placed in one scene description (6DoF scene description) can be compared with a 3D object placed in another scene description (6DoF scene description).
  • Each 6DoF image is displayed side by side with its viewpoint position, line-of-sight direction, angle of view, etc. adjusted appropriately.
  • viewpoint position information indicating the viewpoint position in each 6DoF scene description for viewing comparable 3D objects from the same angle to the same size
  • Line-of-sight direction information indicating the line-of-sight direction at the viewpoint position
  • angle of view information indicating the display area
  • start time information indicating the start time of the 3D object comparison target operation
  • end time indicating the start time of the 3D object comparison target operation
  • FIG. 6 shows an example of signaling of comparative information when such a display is performed.
  • This example is described in glTF2.0 (see Non-Patent Document 4), which is a format for arranging 3D still image contents, animation data, etc. in 6DoF space. Due to its extensibility and the fact that various tools are already supported, glTF2.0 is expected to be adopted as a scene description of MPEG-I, so that 3D video content and 3D Audio content can also be described. Consideration of standard extension in Japan has begun.
  • an extension "CompareDataExtension” for signaling comparison information is defined in a glTF object called "scene”, and in that extension, comparison identification information and comparison display information are signaled.
  • “Compare_sub_info” is information indicating the subcategory of "compare_id”. For example, as this "compare_sub_info”, it is possible to signal arbitrary information regarding the comparison target such as date and time, player name, ball type (for example, straight, curve, fork, etc.), ball speed, rotation speed, and the like.
  • the usage of this "compare_sub_info” information is arbitrary. For example, this "compare_sub_info” information may be used for sorting and searching content. Further, for example, “compare_sub_info.type” may signal the category information, and “compare_sub_info.data” may signal the actual value in the category.
  • Object_id is information indicating the 3D object to be compared. That is, "object_id” is identification information indicating which 3D object is the comparison target among the 3D objects corresponding to this scene description. In other words, this "object_id” is signaled to associate the scene description with the 3D object to be compared.
  • Viewpoint_position is information indicating the initial viewpoint position for viewing this 3D object (also referred to as initial viewpoint position information, for example, a three-dimensional array) for comparison.
  • view_orientation is information indicating an initial line-of-sight method at this initial viewpoint position (also referred to as initial line-of-sight method information, for example, a three-dimensional array) for comparison.
  • view_window is information indicating the vertical and horizontal angles of view (also referred to as angle of view information, for example, a two-dimensional array) for cutting out a 6DoF image including a 3D object for comparison.
  • “window_arrangement_type” is information (also referred to as arrangement method information) indicating an arrangement method (for example, whether to arrange horizontally or vertically) of 6DoF images at the time of comparative display.
  • "compareStartTime” is information indicating the time when the comparison is started (also referred to as the comparison start time, for example, the elapsed time from the beginning of the content).
  • “compareEndTime” is information indicating the time when the comparison ends (also referred to as the comparison end time, for example, the elapsed time from the beginning of the content).
  • “compare_time_type” is information about the playback speed of each content to be compared.
  • the content is played back at a fast-forward playback speed (that is, a playback speed faster than the normal speed) so as to match the comparison end times of the two. )
  • slow playback that is, playback at a playback speed slower than the normal speed
  • "compare_time_type” is information indicating the type of such playback speed (that is, information indicating what type of playback speed the content is played back).
  • window_arrangement_type and “compare_time_type” should have the same information for those that make the same comparison, so they are signaled to have the same value for those with the same "compare_id”.
  • ComparisonData Extension is an array that can have comparison information for each 3D object, assuming that one 6DoF video contains multiple comparable 3D objects. That is, the comparison information may be configured for each 3D object.
  • the above “CompareDataExtension” can be signaled in, for example, a scene description. That is, the comparison information can be included in the scene description.
  • the comparative identification information may be signaled to the MPD. That is, the comparative identification information can be included in the MPD.
  • the client device 103 can know which data can be compared before acquiring the 6DoF scene description including the comparison information. Therefore, the client device 103 can, for example, create list information of the comparison target from the comparison identification information of the MPD and present it to the user as a UI (User Interface).
  • UI User Interface
  • FIG. 7 is a diagram showing an example of MPD in that case.
  • the comparative identification information "compare_id" and “compare_sub_info” are signaled to the supplemental property (supplementalProperty) of the adaptation set (AdaptationSet) including the 3D object to be compared.
  • supplementalProperty the supplemental property of the adaptation set (AdaptationSet) including the 3D object to be compared.
  • the position for signaling the comparative identification information is arbitrary and is not limited to the example of FIG.
  • the generation device 101 can generate and signal such comparison information by executing the file generation process.
  • An example of the flow of the file generation process in this case will be described with reference to the flowchart of FIG.
  • the data input unit 121 of the generation device 101 acquires the media data and the scene description of the 6DoF content to be distributed in step S101.
  • step S102 the preprocessing unit 122 appropriately preprocesses the media data and the scene description.
  • step S103 the comparison information generation unit 124 generates comparison information based on media data, a scene description, settings by the user, and the like. At that time, the comparison information generation unit 124 generates comparison information for each scene description.
  • step S104 the coding unit 123 encodes the media data and generates the coded data.
  • step S105 the file generation unit 125 generates a file including the comparison information generated in step S103, the scene description, the coded data of the media data in step S104, and the like.
  • step S106 the storage unit 127 stores the file generated in step S105.
  • step S107 the upload unit 128 reads the file stored in the storage unit 127 at a predetermined timing or in response to a predetermined event, request, or the like, and uploads the file to the server 102.
  • the generation device 101 can signal the comparison information.
  • the client device 103 can grasp the comparable contents based on the comparison information.
  • the client device 103 can display the 3D object to be compared based on the comparison information so that the user can easily compare the objects. Therefore, the user of the client device 103 can more easily compare the 3D objects. For example, the user can more easily compare the behavior of the 3D object to be compared from all perspectives.
  • the client device 103 can reproduce and display a plurality of 3D objects of 6DoF contents in a comparable manner.
  • An example of the flow of the reproduction process in this case will be described with reference to the flowchart of FIG.
  • the MPD processing unit 161 of the client device 103 accesses the server 102 in step S121 and acquires the MPD of the desired 6DoF content.
  • the MPD processing unit 161 parses the acquired MPD.
  • step S122 the comparison information acquisition unit 163 acquires the comparison identification information signaled to the MPD based on the parsing result.
  • the display unit 169 presents the user with list information of 3D objects that can be compared based on the comparison identification information.
  • the display control unit 164 identifies 3D objects belonging to the same comparable group with each other based on "compare_id”, generates display control information for displaying the list, and supplies the display control information to the buffer 167. ..
  • the display information generation unit 168 acquires the display control information via the buffer 167 and generates display information including a list of comparable 3D objects.
  • the display unit 169 presents a list of comparable 3D objects to the user by displaying the display information on the monitor.
  • the user inputs a selection of 3D objects to compare based on the presented list of comparable 3D objects.
  • the input unit (not shown) of the client device 103 accepts an input operation by the user or the like. That is, the selection input of the 3D object to be compared by the user or the like is accepted.
  • step S124 the coded data acquisition unit 165 acquires the coded data corresponding to the 3D object to be compared selected by the user, that is, the coded data of the media data of the 3D object.
  • the comparison information acquisition unit 163 acquires the comparison display information corresponding to the 3D object to be compared.
  • the comparison information acquisition unit 163 acquires the comparison display information corresponding to the 3D object to be compared selected by the user based on "compare_sub_info", "object_id", and the like.
  • the comparison information acquisition unit 163 acquires the scene description including the comparison display information corresponding to the 3D object to be compared selected by the user, and the scene. Get comparison display information from the description.
  • step S126 the decoding unit 166 decodes the coded data acquired in step S124 and generates (restores) the media data of the 3D object to be compared.
  • step S127 the display information generation unit 168 and the like reproduce and display the 3D object and the scene description (6DoF scene description) to be compared based on the comparison display information and the like acquired in step S125.
  • the display control unit 164 reproduces and displays a 3D object or a scene description (6DoF scene description) to be compared in a comparable manner based on the comparison display information or the like.
  • the display control unit 164 can specify the initial viewpoint position from "viewpoint_position”. Further, the display control unit 164 can specify the initial line-of-sight direction from "view_orientation”. Further, the display control unit 164 can specify the vertical and horizontal angles of view for cutting the 6DoF image from the "view_window”. That is, the display control unit 164 can set each 6DoF video to be compared based on this information.
  • the display control unit 164 can specify the arrangement method of the 6DoF video from "window_arrangement_type”. Further, the display control unit 164 can specify the comparison start time from “compareStartTime”, specify the comparison end time from “compareEndTime”, and specify the playback speed from "compare_time_type”. That is, the display control unit 164 can set how to display each 6DoF video to be compared based on the information.
  • the display information generation unit 168 generates display information as shown in FIG. 5, for example, according to the display control, and the display unit 169 displays the display information on the monitor. By doing so, multiple 3D objects are played and displayed in a comparable manner.
  • step S127 When the process of step S127 is completed, the reproduction process is completed.
  • the client device 103 can grasp the comparable contents based on the signaled comparison information. Further, the client device 103 can reproduce and display a plurality of 3D objects in a comparable manner based on the comparison information. At that time, the client device 103 can display the 3D object to be compared based on the comparison information so that the user can easily compare the objects. Therefore, the user of the client device 103 can more easily compare the 3D objects. For example, the user can more easily compare the behavior of the 3D object to be compared from all perspectives.
  • FIG. 6 is an example of a scene description, and the description of the scene description is not limited to the example of FIG.
  • a scene description other than glTF2.0 for example, a scene description described in Non-Patent Document 1 may be extended to signal similar information.
  • the comparison of 6DoF video including the 3D object to be compared is not limited to two, and the comparison of two or more can be realized with the same signal.
  • "view_window” as an array, it is possible to signal the angle of view information in the case of three comparisons and the angle of view information in the case of four comparisons. The same applies to the following examples.
  • the comparable display method of the 3D object is arbitrary and is not limited to the example of FIG.
  • one of the 6DoF contents to be compared utilizes both the 6DoF scene description and the 3D object, the other utilizes only the 3D object, and in one scene description the 3D object and the other. It may be displayed by superimposing it on the 3D object of. That is, for example, as shown in FIG. 10, another 3D object is superimposed and displayed on the 6DoF content corresponding to one of the plurality of 3D objects to be compared so that the 3D objects are superimposed and displayed. You may. That is, in this case, the comparison information includes information for performing such a display.
  • the object 202A is arranged and displayed so as to be superimposed on the 6DoF video 201 including the object 201A in a state suitable for comparison with the object 201A. That is, one 3D object and the background are superimposed on the other 3D object.
  • this display for example, the state of the bat swing by the object 201A and the state of the bat swing by the object 202A can be superimposed and compared.
  • the viewpoint position information in each 6DoF scene description and the position information (line of sight) for indicating the center position of the 3D object are shown.
  • Comparison display information such as angle of freedom information indicating the display area, start time information and end time information of the operation to be compared with the 3D object, and information indicating how to use the time information is required.
  • FIG. 11 shows an example of signaling of comparative information when such a display is performed. This example is also described in glTF 2.0, as in FIG. The description of the same parts as in the example of FIG. 6 will be omitted. In the case of this example, "object_centre_position”, “priority”, “overlay_scene_transparency”, and “overlay_object_type” are signaled as comparison display information.
  • Object_centre_position is information (for example, a three-dimensional array) indicating the center point of a 3D object (the center point of a bounding box that is a cube that stores a 3D object). This information can be used not only to superimpose two 3D objects, but also to derive the line-of-sight direction. That is, this information includes line-of-sight direction information.
  • Priority is information about priority. For example, this information may indicate the content on which the overlay is based (ie, content that utilizes not only 3D objects but also scene descriptions). That is, the client device 103 may determine which content is to be based on the priority indicated in this "priority".
  • priority may indicate the priority directly or indirectly.
  • priority may indicate what to use as a priority.
  • "priority” signals that the date and time information when the content was created is based on the oldest content
  • "compare_sub_info” signals the date and time information when the content was created. May be good.
  • the client device 103 grasps that the date and time information in which the content was created is based on the oldest content based on the "priority", and the date and time information signaled as "compare_sub_info". You can select the content to base on.
  • “Overlay_scene_transparency” signals the transparency of the 6DoF video that contains the 3D object overlaid. By setting this transparency to 100%, it is possible to superimpose only 3D objects without superimposing scene descriptions.
  • the transmittance may be less than 100%. That is, a plurality of 3D objects and a background may be superimposed. That is, one of the base scene descriptions (including the 3D object to be compared) may be superimposed on the other scene description (including the 3D object to be compared) to be displayed.
  • the object 202A is superimposed on the 6DoF image 201, but the 6DoF image 202 may be superimposed on the 6DoF image 201 with a predetermined transparency. By doing so, it is possible to superimpose the backgrounds of a plurality of 6DoF images.
  • “Overlay_object_type” signals how to display 3D objects to be overlaid.
  • this "overlay_object_type” specifies the display method such as skeleton, difference information, bone, and line. For example, make the transparency of the superimposed 3D object larger than 0%, set the pixel value of the superimposed part as the difference value of both 3D objects, display the superimposed 3D object as a line, or display the superimposed 3D object as a line.
  • a display method such as displaying only the outer shape (outer frame) can be specified by this information.
  • the display control unit 164 can specify the center point of each 3D object to be compared from "object_centre_position". Therefore, the display control unit 164 can superimpose each 3D object by arranging each 3D object so that the center points are aligned (same position). The display control unit 164 can also specify the initial line-of-sight direction from this "object_centre_position".
  • the display control unit 164 can determine which content is to be based on the "priority”. That is, the display control unit 164 can specify the base 6DoF video based on the "priority".
  • the display control unit 164 can specify the transparency of the 6DoF image including the 3D object superimposed on the "overlay_scene_transparency”. That is, the display control unit 164 can set the transparency of the 6DoF image including the 3D object to be superimposed on the transparency to the transparency specified by "overlay_scene_transparency".
  • the display control unit 164 can specify the display method of the 3D object to be superimposed from "overlay_object_type”. That is, the display control unit 164 can set the display method of the 3D objects to be superimposed to the type specified by "overlay_object_type".
  • the display information generation unit 168 generates the display information as shown in FIG. 10, for example, according to the display control as described above, and the display unit 169 displays the display information on the monitor. By doing so, multiple 3D objects are played and displayed in a comparable manner.
  • the "CompareDataExtension” can be signaled, for example, in the scene description. That is, the comparison information can be included in the scene description.
  • ⁇ Comparison example> In such a 6DoF image displayed in a comparable manner, it is desirable that the 3D objects to be compared stay at substantially the same position and operate. For example, when a 3D object such as swimming or sprinting moves in a 6DoF scene description, the viewpoint position, line-of-sight direction, etc. are set according to the movement so that the 3D object to be compared stays at approximately the same position in the 6DoF video. You can change it.
  • the above-mentioned signal indicates the initial value of information for comparison. If the initial values are matched, even if the position of the 3D object changes with the passage of time, the comparison display can be performed in the same way.
  • one of the 6DoF contents to be compared uses both the 6DoF scene description and the 3D object, the other uses only the 3D object, and in one scene description, with that 3D object. It may be displayed side by side with the other 3D object. That is, for example, as shown in FIG. 12, the other 3D objects are superimposed and displayed on the 6DoF content corresponding to one of the plurality of 3D objects to be compared so that the 3D objects are displayed side by side. May be good. That is, in this case, the comparison information includes information for performing such a display.
  • the object 202A is arranged so as to be displayed side by side in the 6DoF video 201 including the object 201A in a state suitable for comparison with the object 201A. That is, one 3D object and the background are superimposed on the other 3D object.
  • this display for example, in the 6DoF video 201, the state of the bat swing by the object 201A and the state of the bat swing by the object 202A can be compared side by side.
  • the viewpoint position information in each 6DoF scene description and the position information (line of sight) for indicating the center position of the 3D object are shown.
  • FIG. 13 shows an example of signaling of comparative information when such a display is performed. This example is also described in glTF 2.0, like the examples in FIGS. 6 and 10. The description of the same parts as those in the examples of FIGS. 6 and 10 will be omitted. In the case of this example, "other_object_position" is signaled as the comparison display information.
  • “Other_object_position” is position information indicating the location of the center point of the 3D object to be superimposed. This position information may be signaled as an array according to the number of 3D objects to be arranged. That is, when comparing N 3D objects, (N-1) position information may be signaled as an array. For example, in the case of FIG. 12, since it is a comparison of two 3D objects, one position information is signaled.
  • These comparative display information is used, for example, in the reproduction process executed by the client device 103.
  • the display control unit 164 can specify the location of the center point of the 3D object superimposed from "other_object_position". Therefore, the display control unit 164 can set the position of the center point of each 3D object to a position where the 3D objects do not overlap each other based on this information.
  • the display information generation unit 168 generates the display information as shown in FIG. 12, for example, according to the display control as described above, and the display unit 169 displays the display information on the monitor. By doing so, multiple 3D objects are played and displayed in a comparable manner.
  • the "CompareDataExtension” can be signaled, for example, in the scene description. That is, the comparison information can be included in the scene description.
  • the client device 103 views one 6DoF image from two different viewpoints when displaying it. It is necessary to provide a display that looks like it is. If this process is not performed, two 3D objects are placed in one 6DoF image, so the 3D objects may be hidden behind the other depending on the angle, or the size of the two 3D objects may increase due to perspective.
  • the display may be different.
  • Second Embodiment> ⁇ Signaling of comparison information corresponding to multiple scene descriptions>
  • a method of signaling comparative information corresponding to a plurality of scene descriptions will be described.
  • the configuration of the distribution system 100 and each device (for example, the generation device 101 and the client device 103) constituting the distribution system 100 is the same as in the case of the first embodiment described above. That is, unless otherwise specified, the description of the configuration of the distribution system 100 and each device made in the first embodiment can be applied to the second embodiment as well.
  • FIG. 14 is a diagram showing an example of distribution data distributed as 6DoF content.
  • the comparison information (comparison identification information and comparison display information) that was distributed and signaled for each scene description in the example of FIG. 4 is listed and put together into one, and each scene disk. Placed at the same level as the description. That is, a Compare List, which is a list of comparison information, is generated and signaled separately from the scene description (SD1 to SD6). That is, this CompareList includes comparison information (Compare_data1 to Compare_data6) corresponding to each scene description. Such a Compare List makes it easier to access comparison information. When an MPD is generated, the MPD may include information for accessing this CompareList (CompareList access information). This access information makes it easier to access the comparison information.
  • This comparison information may include comparison control information which is information for listing comparison information about a plurality of 3D objects.
  • FIG. 15 shows an example of signaling of comparative information when such a display is performed. This example is written in binary data. The description of the parts similar to the parameters described in the first embodiment will be omitted.
  • the comparison information may include "number_of_compare”, “number_of_compare_data”, “Scene_description_file_name”, etc. as the comparison control information.
  • “Number_of_compare” is information indicating the number of “compare_id” to be signaled.
  • “number_of_compare_data” is information indicating the number of 3D objects to be compared grouped by this "compare_id”.
  • "Scene_description_file_name” is information indicating a 6DoF image associated with each 3D object. For example, the number of "compare_id” is signaled by "number_of_compare” in order to summarize the comparison information for each "compare_id”.
  • the MPD may be signaled with access information to the comparison information (Compare List).
  • the client device 103 can acquire the compare_list at the beginning of the process. That is, the comparison information (Compare List) can be easily acquired. Then, the client device 103 can also create list information of the comparison target based on the comparison information (Compare List) and present it to the user as a UI (User Interface).
  • FIG. 16 is a diagram showing an example of MPD in that case.
  • the information indicating that the comparison information is listed that is, it is a CompareList
  • AdaptationSet adaptation set
  • the position where the information indicating that the Compare List is signaled is arbitrary, and is not limited to the example of FIG.
  • step S103 the comparison information generation unit 124 lists and summarizes the comparison information corresponding to each of the plurality of scene descriptions, that is, the comparison information corresponding to the plurality of scene descriptions. To generate. Other processing is performed in the same manner as in the case of FIG.
  • the MPD processing unit 161 of the client device 103 accesses the server 102 in step S141 and acquires the MPD of the desired 6DoF content.
  • the MPD processing unit 161 parses the acquired MPD.
  • step S142 the comparison information acquisition unit 163 acquires a file including the compareList signaled to the MPD based on the parsing result. That is, the comparison information acquisition unit 163 acquires the compareList shown in the MPD.
  • step S143 the comparison information acquisition unit 163 acquires the comparison identification information from the compareList acquired in step S142.
  • the display unit 169 presents the user with list information of 3D objects that can be compared based on the comparison identification information.
  • the display control unit 164 identifies 3D objects belonging to the same comparable group with each other based on "compare_id”, generates display control information for displaying the list, and supplies the display control information to the buffer 167. ..
  • the display information generation unit 168 acquires the display control information via the buffer 167 and generates display information including a list of comparable 3D objects.
  • the display unit 169 presents a list of comparable 3D objects to the user by displaying the display information on the monitor.
  • the user inputs a selection of 3D objects to compare based on the presented list of comparable 3D objects.
  • the input unit (not shown) of the client device 103 accepts an input operation by the user or the like. That is, the selection input of the 3D object to be compared by the user or the like is accepted.
  • step S145 the coded data acquisition unit 165 acquires the coded data corresponding to the 3D object to be compared selected by the user, that is, the coded data of the media data of the 3D object.
  • the comparison information acquisition unit 163 acquires the scene description corresponding to the 3D object.
  • the comparison information acquisition unit 163 acquires the comparison display information corresponding to the 3D object to be compared. For example, the comparison information acquisition unit 163 acquires the comparison display information corresponding to the 3D object to be compared selected by the user based on "compare_sub_info", "object_id", and the like.
  • step S146 the decoding unit 166 decodes the coded data acquired in step S145 and generates (restores) the media data of the 3D object to be compared.
  • step S147 the display information generation unit 168 and the like reproduce and display the 3D object and the scene description (6DoF scene description) to be compared based on the comparison display information and the like acquired in step S145.
  • the display control unit 164 reproduces and displays a 3D object or a scene description (6DoF scene description) to be compared in a comparable manner based on the comparison display information or the like.
  • the display control unit 164 controls the display based on each parameter of the comparative display information.
  • the display information generation unit 168 generates display information capable of comparing a plurality of 3D objects according to the display control, and the display unit 169 displays the display information on the monitor. By doing so, multiple 3D objects are played and displayed in a comparable manner.
  • step S147 When the process of step S147 is completed, the reproduction process is completed.
  • the client device 103 can more easily reproduce and display a plurality of 3D objects in a comparable manner based on the signaled comparison information. Therefore, the user can more easily compare 3D objects.
  • compare_sub_info may be used to signal the number of times it has been distributed (displayed) or the number of times it has been liked. For example, this information may be updated each time an update is made to realize sort display in the order of popular contents.
  • comparison information corresponding to each scene description is collected as one as described above, the comparison information is dispersed for each scene description as in the case of the first embodiment. , Such updating of comparison information can be realized more easily.
  • priority it may be indicated that the number of impressions (number of distributions) and the like are used as the priority.
  • scene descriptions for comparison scene descriptions for comparison between 3D objects OBJ2 and OBJ4 (ComareSD1-2), scene descriptions for comparison between 3D objects OBJ2 and OBJ9 (ComareSD1-3), and 3D objects
  • Scene description for comparison between OBJ2 and OBJ11 ComareSD1-4
  • scene description for comparison between 3D object OBJ4 and OBJ9 ComareSD2-3
  • scene description for comparison between 3D objects OBJ4 and OBJ11 ComareSD2-4
  • a scene description ComareSD3-1) for comparison between 3D objects OBJ9 and OBJ11 is included.
  • comparison information comparison information (Compare_data_ext1-2) corresponding to the scene description for comparison (ComareSD1-2), comparison information (Compare_data_ext1-3) corresponding to the scene description for comparison (ComareSD1-3), Comparison information (Compare_data_ext1-4) corresponding to the comparison scene description (ComareSD1-4), comparison information (Compare_data_ext1-5) corresponding to the comparison scene description (ComareSD1-5), comparison scene description Comparison information (Compare_data_ext2-3) corresponding to (ComareSD2-3), comparison information (Compare_data_ext2-4) corresponding to the scene description for comparison (ComareSD2-4), and scene description for comparison (ComareSD3-4). ) Corresponds to the comparison information (Compare_data_ext3-4).
  • the distribution data includes the MPD corresponding to these data. That is, the MPD contains information for accessing the scene description for comparison.
  • FIG. 19 shows an example of signaling of comparative information when such a display is performed. This example is also described in glTF 2.0, as in FIG. The description of the same parts as those of the above-described examples will be omitted with reference to FIG. 6 and the like.
  • This example is a signal example when realizing a case where two 3D objects are arranged side by side on one background and compared as in the example of FIG.
  • the "scene” described in the scene description for comparison has three “nodes”. Two of these "nodes” are information for arranging a 3D object in association with a "scene”, and have "translation”, “rotation”, “scale”, etc. as the arrangement information. These placement information is signaled so that the placement is suitable for comparison.
  • the remaining one "node” is a "camera” and has information corresponding to the viewpoint position information and the angle of view information. Up to this point, the existing glTF 2.0 functions have been used.
  • comparison_id which is comparative identification information
  • compare_time_type which indicates the temporal reproduction method of the two 3D objects at the time of comparison
  • step S103 the comparison information generation unit 124 generates a scene description for comparison, and generates comparison information corresponding to the scene description.
  • the comparison information generation unit 124 generates a scene description for comparison including comparison information corresponding to a plurality of 3D objects to be compared.
  • Other processing is performed in the same manner as in the case of FIG.
  • the MPD processing unit 161 of the client device 103 accesses the server 102 in step S161 and acquires the MPD of the desired 6DoF content.
  • the MPD processing unit 161 parses the acquired MPD.
  • step S162 the comparison information acquisition unit 163 acquires the information of the comparison scene description (compareSD) signaled to the MPD based on the parsing result.
  • the display unit 169 presents the comparison scene description (compareSD) list information to the user based on the comparison scene description (compareSD) information.
  • the display control unit 164 identifies scene descriptions (compareSD) for comparison belonging to the same comparable group with each other based on the "compare_id" shown in the MPD, and generates display control information for displaying the list.
  • the display control information is supplied to the buffer 167.
  • the display information generation unit 168 acquires the display control information via the buffer 167, and generates display information including a list of scene descriptions (compareSD) for comparison.
  • the display unit 169 presents a list of scene descriptions (compareSD) for comparison to the user by displaying the display information on the monitor.
  • the user inputs the selection of the scene description (compareSD) for comparison based on the presented list of scene descriptions (compareSD) for comparison.
  • the scene description (compareSD) for comparison the 3D object to be compared can be selected.
  • the input unit (not shown) of the client device 103 accepts an input operation by the user or the like. That is, the user or the like accepts the selection input of the 3D object to be compared.
  • step S164 the comparison information acquisition unit 163 acquires the comparison scene description (compareSD) selected by the user, and acquires the comparison information from the scene description (compareSD) for comparison.
  • step S165 the coded data acquisition unit 165 acquires the coded data corresponding to the 3D object to be compared selected by the user, that is, the coded data of the media data of the 3D object.
  • the comparison information acquisition unit 163 acquires the scene description corresponding to the 3D object. Then, the comparison information acquisition unit 163 acquires the comparison display information from the scene description.
  • step S166 the decoding unit 166 decodes the coded data acquired in step S165 and generates (restores) the media data of the 3D object to be compared.
  • step S167 the display information generation unit 168 and the like reproduce and display the 3D object and the scene description (6DoF scene description) to be compared based on the comparison information and the like acquired in step S164 and step S165.
  • the display control unit 164 reproduces and displays a 3D object or a scene description (6DoF scene description) to be compared in a comparable manner based on the comparison display information or the like.
  • the display control unit 164 controls the display based on each parameter of the comparative display information.
  • the display information generation unit 168 generates display information capable of comparing a plurality of 3D objects according to the display control, and the display unit 169 displays the display information on the monitor. By doing so, multiple 3D objects are played and displayed in a comparable manner.
  • step S167 When the process of step S167 is completed, the reproduction process is completed.
  • the client device 103 can more easily reproduce and display a plurality of 3D objects in a comparable manner based on the signaled comparison information. Therefore, the user can more easily compare 3D objects.
  • comparison information (Compare_data_ext_x) corresponding to the comparison scene description (ComareSD_for_ext_x
  • comparison information (Compare_data_ext_y) corresponding to the comparison scene description (ComareSD_for_ext_x)
  • comparison scene description (ComareSD_for_ext_z)
  • Generate comparison information (Compare_data_ext_z) corresponding to.
  • the 3D object to be linked is not specified at this point, and will be linked after being specified by the user.
  • FIG. 23 shows an example of signaling of comparative information when such a display is performed. This example is also described in glTF 2.0, as in FIG. The description of the same parts as those of the above-described examples will be omitted with reference to FIG. 6 and the like.
  • the "scene” described in the scene description for comparison has three “nodes”. Two of these "nodes” are information for arranging a 3D object in association with a "scene”, and have "translation”, “rotation”, “scale”, etc. as the arrangement information. These placement information is signaled so that the placement is suitable for comparison.
  • the remaining one "node” is a "camera” and has information corresponding to the viewpoint position information and the angle of view information. Up to this point, the existing glTF 2.0 functions have been used.
  • comparison_id which is comparative identification information
  • compare_time_type which indicates the temporal reproduction method of the two 3D objects at the time of comparison
  • object_overlay_type which indicates the shape of the overlapping objects
  • step S103 the comparison information generation unit 124 creates only one scene description for comparison for each comparison group (compare_id). At this point, the 3D object is not linked to the scene description for comparison, and the link is made during playback. For example, the comparison information generation unit 124 generates a scene description for comparison including comparison information corresponding to a group of scene descriptions including a comparable 3D object. Other processing is performed in the same manner as in the case of FIG.
  • the MPD processing unit 161 of the client device 103 accesses the server 102 in step S181 and acquires the MPD of the desired 6DoF content.
  • the MPD processing unit 161 parses the acquired MPD.
  • step S182 the comparison information acquisition unit 163 acquires the file information of the comparison scene description (compareSD) signaled to the MPD based on the parsing result.
  • step S183 the comparison information acquisition unit 163 acquires comparison information from the comparison scene description (compareSD).
  • step S184 the display control unit 164 acquires information on comparable objects from the comparison information.
  • step S185 the display unit 169 and the like display the list information of the comparable objects on the monitor and present it to the user.
  • the display control unit 164 identifies comparable 3D objects belonging to the same comparable group to each other based on the "compare_id" shown in the comparison scene description (compareSD), and displays a list thereof.
  • Information is generated and the display control information is supplied to the buffer 167.
  • the display information generation unit 168 acquires the display control information via the buffer 167, and generates display information including a list of scene descriptions (compareSD) for comparison.
  • the display unit 169 presents a list of comparable 3D objects to the user by displaying the display information on the monitor.
  • the user inputs the selection of the 3D object to be compared based on the presented list. That is, the user selects the 3D object to be compared from the 3D objects that can be compared.
  • the input unit (not shown) of the client device 103 accepts an input operation by the user or the like. That is, the user or the like accepts the selection input of the 3D object to be compared.
  • step S186 the coded data acquisition unit 165 acquires the coded data corresponding to the 3D object to be compared selected by the user, that is, the coded data of the media data of the 3D object.
  • step S187 the decoding unit 166 decodes the coded data acquired in step S186 and generates (restores) the media data of the 3D object to be compared.
  • step S188 the display information generation unit 168 and the like associate the media data of the 3D object to be compared with the scene description (compareSD) for comparison based on the comparison information and the like, and display the image appropriately. Play back at the right time.
  • the display information generation unit 168 generates display information capable of comparing a plurality of 3D objects according to the display control, and the display unit 169 displays the display information on the monitor. By doing so, multiple 3D objects are played and displayed in a comparable manner.
  • step S188 When the process of step S188 is completed, the reproduction process is completed.
  • the client device 103 can more easily reproduce and display a plurality of 3D objects in a comparable manner based on the signaled comparison information. Therefore, the user can more easily compare 3D objects.
  • the background used here may be newly prepared, or the same background as the scene description (6DoF scene description) including the base 3D object to be compared may be used. good.
  • the comparison information may include comparative viewing method information, which is information that specifies a viewing method for the content in which the 3D object is displayed in a comparable manner. In the present embodiment, a method of signaling this comparative viewing method information will be described.
  • the content author indicates whether or not comparison is recommended as the intention of the content author, and when it is further recommended, a comparison scenario (for example, a comparison target is selected from the list display for the user). Signal information indicating that (the client selects the comparison content according to the priority).
  • the list is displayed, and the user selects the comparison target from the list and displays the comparison.
  • the content is viewed from the beginning (normal viewing) and comparative display is performed during the playback will be described.
  • Comparison is not recommended 2.
  • the comparison display is based on the priority.
  • the comparison display may be such that the user can select from the list display. Therefore, such comparative viewing method information is signaled.
  • the comparison display list shall be displayed at the "alert timing for comparison" in FIG. 26.
  • the comparison display is performed according to the comparison target time of the content being viewed.
  • a viewing experience such as returning to normal viewing is provided.
  • FIG. 27 shows an example of signaling of comparative information when such a display is performed. This example is also described in glTF 2.0 as in FIG. The description of the same parts as those of the above-described examples will be omitted with reference to FIG.
  • FIG. 27 the same signal method as in the example of FIG. 11 (as shown in FIG. 10, 6DoF corresponding to one of the plurality of 3D objects to be compared so that the 3D objects are displayed in an overlapping manner).
  • An example of signaling the above-mentioned comparative viewing method information in the case where another 3D object is superimposed and displayed on the content is shown.
  • "compare_method" is signaled.
  • “Compare_method” is comparative viewing method information, and is information that specifies a viewing method for content in which 3D objects are displayed in a comparable manner.
  • the value of "compare_method” and the viewing method specified by “compare_method” are arbitrary. An example is shown below.
  • FIG. 28 is a diagram showing an example of MPD in that case.
  • "compare_method” is signaled to the supplemental property (supplementalProperty) of the adaptation set (AdaptationSet) including the scene description of the viewing content as the comparative viewing method information.
  • supplementalProperty the supplemental property of the adaptation set
  • AdaptationSet adaptation set
  • the position where the comparative viewing method information is signaled is arbitrary, and is not limited to the example of FIG. 28.
  • the comparison information generation unit 124 can generate the above-mentioned comparative viewing method information as the comparison information. Other processing is performed in the same manner as in other cases such as FIG. By doing so, the generation device 101 can indicate whether or not the comparison is recommended as the intention of the content author, and can signal information indicating a comparison scenario when the comparison is recommended. By doing so, it is possible to provide the playback side (client device 103, etc.) with a comparison scenario when there is a comparison target during normal viewing.
  • the MPD processing unit 161 of the client device 103 accesses the server 102 in step S201 and acquires the MPD of the desired 6DoF content.
  • the MPD processing unit 161 parses the acquired MPD.
  • step S202 the comparison information acquisition unit 163 acquires the scene description signaled to the MPD based on the parsing result, and reproduces the normal viewing.
  • the comparison information acquisition unit 163 acquires the comparison viewing method information corresponding to the content. For example, when the scene description includes the comparative viewing method information, the comparison information acquisition unit 163 acquires the comparative viewing method information from the scene description.
  • step S203 the display control unit 164 controls the viewing method based on the comparative viewing method information. For example, the display control unit 164 determines whether or not the value of "compare_method" is "0". If it is determined that the value of "compare_method” is "0", the process proceeds to step S205.
  • step S205 the display control unit 164 controls the display so as to continue normal viewing.
  • the display information generation unit 168 generates display information according to the control thereof, and the display unit 169 displays the display information on the monitor. That is, in this case, normal viewing is continued, and playback / display capable of comparing a plurality of 3D objects as described in the first embodiment or the like is not performed.
  • step S205 When the process of step S205 is completed, the reproduction process is completed. If it is determined in step S204 that the value of "compare_method" is not "0", the process proceeds to FIG.
  • step S211 of FIG. 30 the display control unit 164 determines whether or not the value of "compare_method” is "1". If it is determined that the value of "compare_method” is "1", the process proceeds to step S212.
  • step S212 the comparison information acquisition unit 163 obtains the information of the scene description including the comparable 3D object from the MPD, and acquires the scene description.
  • step S213 the data acquisition control unit 162 acquires priority information from each scene description acquired in step S212.
  • step S214 the coded data acquisition unit 165 sets the 3D object having the highest priority as the 3D object to be compared based on the priority information, and acquires the coded data of the 3D object to be compared.
  • step S215 the decoding unit 166 decodes the acquired coded data.
  • the comparison information acquisition unit 163 acquires the comparison display information. For example, when the scene description includes the comparison display information, the comparison display information is acquired from the scene description.
  • the display control unit 164 reproduces the 3D object to be compared and the scene description (6DoF scene description) based on the comparison display information.
  • the display information generation unit 168 generates display information according to the control thereof, and the display unit 169 displays the display information on the monitor. That is, in this case, when the comparison target time is reached, the reproduction / display in which a plurality of 3D objects can be compared is performed as described in the first embodiment and the like.
  • step S217 the display control unit 164 controls the display so as to continue the normal viewing.
  • the display information generation unit 168 generates display information according to the control thereof, and the display unit 169 displays the display information on the monitor.
  • step S217 the process returns to FIG. 29, and the reproduction process is completed. That is, normal viewing is continued until the end of the content.
  • step S211 If it is determined in step S211 that the value of "compare_method" is not "1", the process proceeds to step S221.
  • step S221 the comparison information acquisition unit 163 obtains the comparison information information corresponding to the scene description including the comparable 3D object from the MPD, and acquires the comparison information. For example, when the scene description includes comparison information, the comparison information acquisition unit 163 acquires the scene description including the comparable 3D object, and acquires the comparison information from the scene description.
  • step S222 the display control unit 164 identifies a comparable 3D object based on the comparison information, and displays a list of the comparable 3D objects.
  • the display information generation unit 168 generates display information according to the control thereof, and the display unit 169 displays the display information on the monitor. That is, a list of comparable 3D objects is presented to the user and the like.
  • the user or the like selects a 3D object to be compared based on the presentation, and inputs the selection.
  • the input unit (not shown) of the client device 103 accepts the input, that is, the selection of the 3D object to be compared by the user or the like.
  • step S223 the coded data acquisition unit 165 acquires the coded data of the 3D object to be compared selected by the user or the like.
  • step S224 the decoding unit 166 decodes the acquired encoded data.
  • the comparison information acquisition unit 163 acquires the comparison display information. For example, when the scene description includes the comparison display information, the comparison display information is acquired from the scene description.
  • the display control unit 164 reproduces the 3D object to be compared and the scene description (6DoF scene description) based on the comparison display information.
  • the display information generation unit 168 generates display information according to the control thereof, and the display unit 169 displays the display information on the monitor. That is, in this case, when the comparison target time is reached, the reproduction / display in which a plurality of 3D objects can be compared is performed as described in the first embodiment and the like.
  • step S226 the display control unit 164 controls the display so as to continue the normal viewing.
  • the display information generation unit 168 generates display information according to the control thereof, and the display unit 169 displays the display information on the monitor.
  • step S226 the process returns to FIG. 29, and the reproduction process is completed. That is, normal viewing is continued until the end of the content.
  • the client device 103 can more easily reproduce and display a plurality of 3D objects in a comparable manner based on the signaled comparison information. Further, the client device 103 can realize a wider variety of viewing methods based on the comparative viewing method information. For example, the client device 103 can grasp a comparison scenario when there is a comparison target during normal viewing and provide it to the user. Therefore, the user can more easily compare 3D objects.
  • FIG. 31 shows a configuration example of ISOBMFF showing the reference relationship.
  • thumbnail display information information such as the first viewpoint position, line-of-sight direction, and angle of view information of the thumbnail and display rule information (information that signals a change in position information with time)
  • ItemProperty information that signals a change in position information with time
  • the information related to thumbnails may be any information as long as it is related to thumbnails.
  • the above-mentioned thumbnail display information included in ISOBMFF, information indicating that the comparison display information included in the comparison information is diverted, and the like may be included.
  • FIG. 32 shows an example of signaling information regarding thumbnails in such a display. This example is described by binary data as in the case of FIG. The description of the parts similar to the parameters described with reference to FIG. 15 will be omitted.
  • "thumbnail_type”, "use_object_data_for_thumbnail_view_flag”, and "use_compare_view_for_thumbnail_view_flag" are signaled as information about thumbnails.
  • “Thumbnail_type” is information indicating which type of thumbnail is used among the stored thumbnails. For example, in the case of FIG. 31, "vthm”, which is a 3D object of a low-resolution moving image, and "3dst”, which is a 3D object of a still image, are stored ("iinf" BOX). Therefore, one of these is signaled to "thumbnail_type".
  • the type of thumbnail is arbitrary and is not limited to the example of FIG.
  • “Use_object_data_for_thumbnail_view_flag” is flag information indicating whether to divert the thumbnail display information stored in the 3D object of the thumbnail when displaying the thumbnail. If this "use_object_data_for_thumbnail_view_flag” is "true”, the thumbnail display information stored in the 3D object will be used. On the other hand, if this "use_object_data_for_thumbnail_view_flag” is "false", the thumbnail display information is not diverted, and "use_compare_view_for_thumbnail_view_flag” is signaled.
  • "Use_compare_view_for_thumbnail_view_flag” is flag information indicating whether to divert the information for comparison display.
  • "use_compare_view_for_thumbnail_view_flag” is "true”
  • the above-mentioned comparison display information “viewpoint_position”, “view_orientation”, “view_window”, etc. are diverted.
  • "use_compare_view_for_thumbnail_view_flag” is "false”
  • this information will be signaled separately.
  • the client device 103 By signaling information about thumbnails such as these, it is possible to provide the client device 103 with information such as specifying a thumbnail when displaying a list and diverting the display method of the thumbnail included in the thumbnail 3D object. it can. As a result, the client device 103 can easily display thumbnails of 3D objects and the like corresponding to each scene description and the like based on the information about the thumbnails when displaying the list of the scene descriptions and the like.
  • the above-mentioned comparison display information can be dynamically changed in the time direction, for example.
  • a signal method for dynamically changing the comparative display information in this way will be described.
  • FIG. 33 shows an example of signaling information that dynamically changes the comparative display information. This example is also described by binary data as in FIGS. 15 and 32. This example is an extension of the compare_list so that it can store information that changes with time when the compare_list is signaled as described in the second embodiment. In this example, "num_of_change_data”, “time_scale”, and “applicable_time” are signaled as information that changes compare_list over time.
  • “Num_of_change_data” is information indicating the number of time-changing data
  • “time_scale” is time scale information used for time information.
  • "applicable_time” is information indicating the time when the data is applied. Then, "viewpoint_position”, “view_orientation”, and “view_window” are signaled as the time comparison display information indicated by the "applicable_time”. That is, these comparative display information is applied at the time indicated by “applicable_time” (the time on the time scale indicated by “time_scale”). This process is looped "num_of_change_data” times.
  • the client device 103 can dynamically change the comparison display information.
  • the storage method of timed metadata in ISOBMFF may be used.
  • the signal method for dynamically changing the comparative display information is not limited to the above example. Other methods will be described based on the example of signaling of the first embodiment.
  • FIGS. 34 and 35 are diagrams showing an example of a concept when signaling time-changing comparative display information using the mechanism of glTF2.0.
  • the comparison identification information and the static data (static) of the comparison display information are signaled by defining an extension in the "scene” as in the case of the first embodiment (FIG. 34).
  • the initial value information of the dynamic data (dynamic) of the comparison information is signaled using the "node” having the "camera” of glTF2.0.
  • the "channel” belonging to the “animation” in FIG. 35 contains information (channel.target.node) of the target “node” and information (channel.target.path) indicating what kind of change is used. It is possible to specify the type of linking with “node” and how to move it.
  • “Sampler” has reference information to "accessor” which has access information to the file that stores animation data.
  • FIG. An example of signaling such dynamically changing comparative display information is shown in FIG. This example is also described in glTF 2.0 as in FIG.
  • the initial information of the dynamic comparison display information is that "Obj1-2" has “camera” as a child node, and “translation” and “rotation” of "node” associated with “camera” object. It is shown. This information is used as an initial value in the viewpoint position and the line-of-sight direction.
  • the angle of view information is signaled by specifying "perspective.yfov” and “perspective.aspectRatio" of "camera”.
  • the client device 103 By performing such signaling, it is possible to provide dynamically changing comparative display information to the playback side (client device 103, etc.). As a result, the client device 103 can dynamically change the comparison display information.
  • FIG. 37 is a block diagram showing a configuration example of computer hardware that executes the above-mentioned series of processes programmatically.
  • the CPU Central Processing Unit
  • ROM Read Only Memory
  • RAM Random Access Memory
  • the input / output interface 910 is also connected to the bus 904.
  • An input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected to the input / output interface 910.
  • the input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, and the like.
  • the output unit 912 includes, for example, a display, a speaker, an output terminal, and the like.
  • the storage unit 913 is composed of, for example, a hard disk, a RAM disk, a non-volatile memory, or the like.
  • the communication unit 914 includes, for example, a network interface.
  • the drive 915 drives a removable medium 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
  • the CPU 901 loads the program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executes the above-described series. Is processed.
  • the RAM 903 also appropriately stores data and the like necessary for the CPU 901 to execute various processes.
  • the program executed by the computer can be recorded and applied to the removable media 921 as a package media or the like, for example.
  • the program can be installed in the storage unit 913 via the input / output interface 910 by mounting the removable media 921 in the drive 915.
  • This program can also be provided via wired or wireless transmission media such as local area networks, the Internet, and digital satellite broadcasting. In that case, the program can be received by the communication unit 914 and installed in the storage unit 913.
  • this program can be installed in advance in ROM 902 or storage unit 913.
  • control information related to the present technology described in each of the above embodiments may be transmitted from the coding side to the decoding side.
  • control information for example, enabled_flag
  • control information for example, present_flag
  • control information indicating an object to which the present technology is applied (or an object to which the present technology is not applied) may be transmitted.
  • control information may be transmitted that specifies the block size (upper limit, / lower limit, or both) to which the present technology is applied (or permission or prohibition of application), a frame, a component, a layer, or the like.
  • This technique can be applied to any image coding / decoding method. That is, as long as it does not contradict the above-mentioned technology, the specifications of various processes related to image coding / decoding such as conversion (inverse transformation), quantization (inverse quantization), coding (decoding), and prediction are arbitrary. It is not limited to the example. In addition, some of these processes may be omitted as long as they do not contradict the present technology described above.
  • this technology can be applied to a multi-viewpoint image coding / decoding system that encodes / decodes a multi-viewpoint image including images of a plurality of viewpoints (views).
  • the present technology may be applied to the coding / decoding of each viewpoint (view).
  • this technology is applied to a hierarchical image coding (scalable coding) / decoding system that encodes / decodes a hierarchical image that is layered (layered) so as to have a scalability function for a predetermined parameter. can do.
  • the present technology may be applied in the coding / decoding of each layer.
  • the generation device 101 and the client device 103 have been described as application examples of the present technology, but the present technology can be applied to any configuration.
  • this technology is a transmitter or receiver (for example, a television receiver or mobile phone) for satellite broadcasting, cable broadcasting such as cable TV, distribution on the Internet, and distribution to terminals by cellular communication, or It can be applied to various electronic devices such as devices (for example, hard disk recorders and cameras) that record images on media such as optical disks, magnetic disks, and flash memories, and reproduce images from these storage media.
  • devices for example, hard disk recorders and cameras
  • a processor as a system LSI (Large Scale Integration) or the like (for example, a video processor), a module using a plurality of processors (for example, a video module), a unit using a plurality of modules (for example, a video unit)
  • a processor as a system LSI (Large Scale Integration) or the like
  • a module using a plurality of processors for example, a video module
  • a unit using a plurality of modules for example, a video unit
  • it can be implemented as a configuration of a part of the device, such as a set (for example, a video set) in which other functions are added to the unit.
  • this technology can be applied to a network system composed of a plurality of devices.
  • the present technology may be implemented as cloud computing that is shared and jointly processed by a plurality of devices via a network.
  • this technology is implemented in a cloud service that provides services related to images (moving images) to arbitrary terminals such as computers, AV (AudioVisual) devices, portable information processing terminals, and IoT (Internet of Things) devices. You may try to do it.
  • the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a device in which a plurality of modules are housed in one housing are both systems. ..
  • Systems, devices, processing units, etc. to which this technology is applied can be used in any field such as transportation, medical care, crime prevention, agriculture, livestock industry, mining, beauty, factories, home appliances, weather, nature monitoring, etc. .. Moreover, the use is arbitrary.
  • this technology can be applied to systems and devices used for providing ornamental contents and the like.
  • the present technology can be applied to systems and devices used for traffic such as traffic condition supervision and automatic driving control.
  • the present technology can be applied to systems and devices used for security purposes.
  • the present technology can be applied to a system or device used for automatic control of a machine or the like.
  • the present technology can be applied to systems and devices used for agriculture and livestock industry.
  • the present technology can also be applied to systems and devices for monitoring natural conditions such as volcanoes, forests and oceans, and wildlife. Further, for example, the present technology can be applied to systems and devices used for sports.
  • the "flag” is information for identifying a plurality of states, and is not only information used for identifying two states of true (1) or false (0), but also three or more states. It also contains information that can identify the state. Therefore, the value that this "flag” can take may be, for example, 2 values of 1/0 or 3 or more values. That is, the number of bits constituting this "flag” is arbitrary, and may be 1 bit or a plurality of bits.
  • the identification information (including the flag) is assumed to include not only the identification information in the bitstream but also the difference information of the identification information with respect to a certain reference information in the bitstream. In, the "flag” and “identification information” include not only the information but also the difference information with respect to the reference information.
  • various information (metadata, etc.) related to the coded data may be transmitted or recorded in any form as long as it is associated with the coded data.
  • the term "associate" means, for example, to make the other data available (linkable) when processing one data. That is, the data associated with each other may be combined as one data or may be individual data.
  • the information associated with the coded data (image) may be transmitted on a transmission path different from the coded data (image).
  • the information associated with the coded data (image) may be recorded on a recording medium (or another recording area of the same recording medium) different from the coded data (image). Good.
  • this "association" may be a part of the data, not the entire data. For example, an image and information corresponding to the image may be associated with each other in an arbitrary unit such as a plurality of frames, one frame, or a part within the frame.
  • the embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.
  • the configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units).
  • the configurations described above as a plurality of devices (or processing units) may be collectively configured as one device (or processing unit).
  • a configuration other than the above may be added to the configuration of each device (or each processing unit).
  • a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit). ..
  • the above-mentioned program may be executed in any device.
  • the device may have necessary functions (functional blocks, etc.) so that necessary information can be obtained.
  • each step of one flowchart may be executed by one device, or may be shared and executed by a plurality of devices.
  • the plurality of processes may be executed by one device, or may be shared and executed by a plurality of devices.
  • a plurality of processes included in one step can be executed as processes of a plurality of steps.
  • the processes described as a plurality of steps can be collectively executed as one step.
  • the processing of the steps for writing the program may be executed in chronological order in the order described in the present specification, and the calls may be made in parallel or in parallel. It may be executed individually at the required timing such as when it is broken. That is, as long as there is no contradiction, the processing of each step may be executed in an order different from the above-mentioned order. Further, the processing of the step for writing this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.
  • a plurality of technologies related to this technology can be independently implemented independently as long as there is no contradiction.
  • any plurality of the present technologies can be used in combination.
  • some or all of the techniques described in any of the embodiments may be combined with some or all of the techniques described in other embodiments. It is also possible to carry out a part or all of any of the above-mentioned techniques in combination with other techniques not described above.
  • the present technology can also have the following configurations.
  • An information processing device including a comparison information generator that generates comparison information, which is information for displaying a plurality of 3D objects of 6DoF contents in a comparable manner.
  • the comparison information includes comparative identification information which is information about the 3D object to be compared.
  • the comparison information includes comparative display information which is information related to the display of the 3D object.
  • the comparison information includes comparison control information which is information for listing the comparison information regarding a plurality of the 3D objects.
  • the information processing device includes comparative viewing method information which is information on a viewing method of the 6DoF content.
  • the information processing apparatus wherein the comparison information is configured for each of the 3D objects.
  • the comparison information includes information for displaying the 6DoF contents corresponding to each of the plurality of 3D objects to be compared side by side.
  • the other 3D objects are superimposed on the 6DoF content corresponding to one of the plurality of 3D objects to be compared so that the 3D objects are superimposed and displayed.
  • the information processing apparatus which includes information for causing the information to be processed.
  • the comparison information is for superimposing another 3D object on the 6DoF content corresponding to one of the plurality of 3D objects to be compared so that the 3D objects are displayed side by side.
  • the information processing apparatus (1) The information processing apparatus according to (1). (10) The information processing device according to (1), wherein the comparison information generation unit generates a scene description including the comparison information. (11) The information processing device according to (1), wherein the comparison information generation unit generates the comparison information corresponding to a plurality of scene descriptions. (12) The information processing apparatus according to (1), wherein the comparison information generation unit generates a scene description for comparison including the comparison information corresponding to the plurality of 3D objects to be compared. (13) The information processing apparatus according to (1), wherein the comparison information generation unit generates a comparison scene description including the comparison information corresponding to a group of scene descriptions including the comparable 3D object. (14) The information processing apparatus according to (1), further comprising an MPD generation unit that generates an MPD (Media Presentation Description) including the comparison information. (15) An information processing method for generating comparative information, which is information for displaying a plurality of 3D objects of 6DoF contents in a comparable manner.
  • 100 distribution system 101 generator, 102 server, 103 client device, 111 control unit, 112 generation processing unit, 121 data input unit, 122 preprocessing unit, 123 coding unit, 124 comparison information generation unit, 125 file generation unit, 126 MPD generation unit, 127 storage unit, 128 upload unit, 151 control unit, 152 playback processing unit, 161 MPD processing unit, 162 data acquisition control unit, 163 comparison information acquisition unit, 164 display control unit, 165 coded data acquisition unit. , 166 decoding unit, 167 buffer, 168 display information generation unit, 169 display unit

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Graphics (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
PCT/JP2020/048726 2020-01-08 2020-12-25 情報処理装置および方法 Ceased WO2021140956A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US17/790,680 US20230043591A1 (en) 2020-01-08 2020-12-25 Information processing apparatus and method
JP2021570018A JPWO2021140956A1 (https=) 2020-01-08 2020-12-25

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020-001412 2020-01-08
JP2020001412 2020-01-08

Publications (1)

Publication Number Publication Date
WO2021140956A1 true WO2021140956A1 (ja) 2021-07-15

Family

ID=76788453

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/048726 Ceased WO2021140956A1 (ja) 2020-01-08 2020-12-25 情報処理装置および方法

Country Status (3)

Country Link
US (1) US20230043591A1 (https=)
JP (1) JPWO2021140956A1 (https=)
WO (1) WO2021140956A1 (https=)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016524262A (ja) * 2013-07-10 2016-08-12 リアル ビュー イメージング リミテッド ３次元ユーザインタフェース
JP2016177037A (ja) * 2015-03-19 2016-10-06 株式会社ニコン 観察装置、観察方法、及びプログラム
JP2019030492A (ja) * 2017-08-08 2019-02-28 コニカミノルタ株式会社 Ｘ線画像処理装置及びｘ線画像処理方法

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011185650A (ja) * 2010-03-05 2011-09-22 Omron Corp モデル作成装置およびモデル作成プログラム
US20190096135A1 (en) * 2017-09-26 2019-03-28 Aquifi, Inc. Systems and methods for visual inspection based on augmented reality
US10311646B1 (en) * 2018-02-26 2019-06-04 Capital One Services, Llc Dynamic configuration of an augmented reality overlay

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016524262A (ja) * 2013-07-10 2016-08-12 リアル ビュー イメージング リミテッド ３次元ユーザインタフェース
JP2016177037A (ja) * 2015-03-19 2016-10-06 株式会社ニコン 観察装置、観察方法、及びプログラム
JP2019030492A (ja) * 2017-08-08 2019-02-28 コニカミノルタ株式会社 Ｘ線画像処理装置及びｘ線画像処理方法

Also Published As

Publication number Publication date
US20230043591A1 (en) 2023-02-09
JPWO2021140956A1 (https=) 2021-07-15

Similar Documents

Publication Publication Date Title
JP7399224B2 (ja) メディアコンテンツを送信するための方法、装置及びコンピュータプログラム
KR102258448B1 (ko) 핫스팟 및 roi 관련 메타데이터를 이용한 360도 비디오를 송수신하는 방법 및 그 장치
KR102559862B1 (ko) 미디어 콘텐츠 전송을 위한 방법, 디바이스, 및 컴퓨터 프로그램
WO2019202207A1 (en) Processing video patches for three-dimensional content
JP7726209B2 (ja) 情報処理装置および方法
KR20190008901A (ko) 가상 현실 미디어 콘텐트의 스트리밍을 개선하는 방법, 디바이스, 및 컴퓨터 프로그램
US10931930B2 (en) Methods and apparatus for immersive media content overlays
CN112219403B (zh) 沉浸式媒体的渲染视角度量
CN107851425B (zh) 信息处理设备、信息处理方法和程序
US20250380037A1 (en) Information processing apparatus, information processing method, reproduction processing apparatus, and reproduction processing method
JP7035401B2 (ja) 画像処理装置およびファイル生成装置
GB2506911A (en) Streaming data corresponding to divided image portions (tiles) via a description file including spatial and URL data
CN114930869B (zh) 用于视频编码和视频解码的方法、装置和计算机程序产品
US20220303641A1 (en) Information processing device and information processing method
KR101944601B1 (ko) 기간들에 걸쳐 오브젝트들을 식별하기 위한 방법 및 이에 대응하는 디바이스
KR102243666B1 (ko) 360도 비디오를 전송하는 방법, 360도 비디오를 수신하는 방법, 360도 비디오 전송 장치, 360도 비디오 수신 장치
WO2021140956A1 (ja) 情報処理装置および方法
TW201939964A (zh) 資訊處理裝置及方法
WO2023058258A1 (ja) 画像処理システム、並びに、画像処理装置および方法
VRT et al. First Version of Playout Clients

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20912605

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021570018

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20912605

Country of ref document: EP

Kind code of ref document: A1