US20220239994A1 - Information processing apparatus, information processing method, reproduction processing apparatus, and reproduction processing method - Google Patents
Information processing apparatus, information processing method, reproduction processing apparatus, and reproduction processing method Download PDFInfo
- Publication number
- US20220239994A1 US20220239994A1 US17/617,014 US202017617014A US2022239994A1 US 20220239994 A1 US20220239994 A1 US 20220239994A1 US 202017617014 A US202017617014 A US 202017617014A US 2022239994 A1 US2022239994 A1 US 2022239994A1
- Authority
- US
- United States
- Prior art keywords
- file
- content
- content configuration
- information
- selection information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000010365 information processing Effects 0.000 title claims abstract description 38
- 238000003672 processing method Methods 0.000 title claims abstract description 12
- 238000007781 pre-processing Methods 0.000 claims abstract description 70
- 238000009877 rendering Methods 0.000 claims description 10
- 230000004048 modification Effects 0.000 description 69
- 238000012986 modification Methods 0.000 description 69
- 238000010586 diagram Methods 0.000 description 46
- 238000000034 method Methods 0.000 description 18
- 230000005540 biological transmission Effects 0.000 description 16
- 230000006978 adaptation Effects 0.000 description 13
- 238000004458 analytical method Methods 0.000 description 8
- 238000005259 measurement Methods 0.000 description 6
- 230000003044 adaptive effect Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 230000001276 controlling effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001151 other effect Effects 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/84—Generation or processing of descriptive data, e.g. content descriptors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/8146—Monomedia components thereof involving graphical data, e.g. 3D object, 2D graphics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/23412—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/23439—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements for generating different versions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
- H04N21/2362—Generation or processing of Service Information [SI]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
- H04N21/4345—Extraction or processing of SI, e.g. extracting service information from an MPEG stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
- H04N21/4347—Demultiplexing of several video streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/44209—Monitoring of downstream path of the transmission network originating from a server, e.g. bandwidth variations of a wireless network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
- H04N21/462—Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
- H04N21/4621—Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/845—Structuring of content, e.g. decomposing content into time segments
- H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/85406—Content authoring involving a specific file format, e.g. MP4 format
Definitions
- the present invention relates to an information processing apparatus, an information processing method, a reproduction processing apparatus, and a reproduction processing method.
- a two-dimensional video which is used to distribute a movie or the like, is mainly distributed.
- the two-dimensional video may be referred to as two-dimensional (2D) content.
- a 360-degree video that can be viewed in all directions is also distributed on a video distribution site on the web. Being viewable in all directions indicates that a line-of-sight direction can be freely selected.
- the 360-degree video is called a 3 degrees of freedom (3DoF) video or 3DoF content. It is basic in both the 2D content and the 3DoF content that a two-dimensionally encoded video is distributed from a distribution server and displayed to a client.
- the 3DoF+ content is content that can be viewed in all directions similarly to the 3DoF content and furthermore allows a slight shift of a viewpoint position. It is assumed that the shift of the viewpoint position in the 3DoF+ content is allowed in a range in which a user can move a head in a sitting state. In the 3DoF+ content, the shift of the viewpoint position is implemented by using one or a plurality of two-dimensionally encoded videos.
- the 6DoF video is a video that can be viewed in all directions in a three-dimensional space and can be viewed by walking around in the three-dimensional space displayed. Walking around in the three-dimensional space means that a viewpoint position can be freely selected.
- the three-dimensional space may be referred to as 3D space.
- the 6DoF content is three-dimensional content in which a three-dimensional space is represented by one or a plurality of three-dimensional model data.
- the three-dimensional model data may be referred to as 3D model data, and the three-dimensional content may also be referred to as 3D content.
- the 6DoF content is transmitted as a plurality of object streams by configuring a three-dimensional space with a plurality of three-dimensional model data.
- configuration information about the three-dimensional space which is called a scene description
- An example thereof is a moving picture experts group (MPEG)-4 scene description.
- the scene description which is a representation method, is a method in which a scene is represented by a graph having a tree hierarchical structure, which is called a scene graph, while the scene graph is represented in a binary format.
- the 6DoF content is a video material representing a three-dimensional space with three-dimensional model data at each time.
- schemes for representing the 6DoF content include the following three schemes.
- One scheme is a representation scheme referred to as an object-based representation scheme in the present invention.
- 6DoF content has a content configuration in which three-dimensional model data for each three-dimensional object such as a person or a thing, which is an individual target object to be displayed in a video, is arranged in a three-dimensional space to represent the entire three-dimensional space.
- the object-based representation scheme is characterized in that a client who performs reproduction of 6DoF content simultaneously processes the largest number of three-dimensional model data between the three schemes.
- definition can be changed in displaying each three-dimensional object such as an individual person or thing. Therefore, it can be said that the object-based representation scheme is a configuration method in which a client has a high degree of freedom in reproduction processing between the three methods.
- 6DoF content has a content configuration in which an entire target three-dimensional space is represented as one three-dimensional model data without separating each three-dimensional object such as a person or a thing as three-dimensional model data.
- the space-based representation scheme is characterized in that a client processes one three-dimensional model data at the time of reproduction, requiring the lowest processing capability between the three schemes.
- a client has an extremely low degree of freedom in reproduction processing, because the definition of the entire 6DoF content is fixed.
- the other one is a combination of a space-based representation scheme and an object-based representation scheme.
- this representation scheme will be referred to as a mixed-type representation scheme.
- 6DoF content has a content configuration in which a specific three-dimensional object is represented by independent three-dimensional model data, and a three-dimensional space excluding the specific three-dimensional object is represented by one three-dimensional model data.
- a client uses a plurality of three-dimensional model data in reproduction processing, but the number of three-dimensional model data is smaller than that used in the object-based representation scheme.
- a client in the mixed-type representation scheme, a client is required to have a higher processing capability than that in the space-based representation scheme, but may have a lower processing capability than that in the object-based representation scheme.
- a client has a higher degree of freedom in reproduction processing than that in the space-based representation scheme and a lower degree of freedom in reproduction processing than that in the object-based representation scheme.
- 6DoF content has a different content configuration.
- the client selects a content configuration of a representation scheme in which the degree of freedom is as high as possible, such that the viewing experience of the user can be further expanded.
- Non Patent Literature 1 “ISO/IEC 14496-11:2015”, Information technology. Coding of audio-visual objects. Part 11: Sene description and application engine, 2015-11
- a client selects the content configuration after performing various analyses.
- analyses include analysis of the entire scene description, analysis of AdaptationSet in media presentation description (MPD), and the like.
- analyses include analysis of what is not to be actually used, and thus can be said to be less efficient for a client device in selecting a content configuration.
- the present disclosure provides an information processing apparatus, an information processing method, a reproduction processing apparatus, and a reproduction processing method that enable a client device to efficiently select a content configuration.
- a preprocessing unit generates content configuration selection information, with respect to one or a plurality of contents, for determining whether or not each of the contents is reproducible, each of the contents having a content configuration including one or more three-dimensional objects and space arrangement information therefor to represent a virtual space.
- a file generation unit generates a file including data about the virtual space and the content configuration selection information.
- FIG. 1 is a diagram illustrating a configuration of 6DoF content.
- FIG. 2 is a system configuration diagram of an example of a distribution system.
- FIG. 3 is a block diagram of a file generation device.
- FIG. 4 is a diagram for describing a method of storing content configuration selection information according to a first embodiment.
- FIG. 5 is a diagram illustrating an example of syntax for an extended switch node in the first embodiment.
- FIG. 6 is a block diagram of a client device.
- FIG. 7 is a flowchart of file generation processing by the file generation device according to the first embodiment.
- FIG. 8 is a flowchart of reproduction processing executed by the client device according to the first embodiment.
- FIG. 9 is a diagram illustrating an ISOBMFF file for scene description in Modification (1) of the first embodiment.
- FIG. 10 is a diagram illustrating an example of syntax for content configuration selection information stored in SampleEntry in Modification (1) of the first embodiment.
- FIG. 11 is a diagram illustrating an example of a group of content configuration selection information.
- FIG. 12 is a diagram illustrating a Matroska Media Container format.
- FIG. 13 is a diagram for describing a method of storing content configuration selection information according to Modification (4) of the first embodiment.
- FIG. 14 is a diagram illustrating an example of syntax for a RequiedPerformance node.
- FIG. 15 is a diagram illustrating an example of describing 6DoFContentStruct Descriptor.
- FIG. 16 is a diagram illustrating semantics for CSC in Modification (5) of the first embodiment.
- FIG. 17 is a diagram for describing a method of storing content configuration selection information according to a second embodiment.
- FIG. 18 is a diagram illustrating an example of syntax for an extended switch node in the second embodiment.
- FIG. 19 is a diagram illustrating an example of the syntax for content configuration selection information stored in SampleEntry in Modification (1) of a third embodiment.
- FIG. 20 is a diagram illustrating semantics for CSC in Modification (2) of the third embodiment.
- FIG. 21 is a diagram for describing a method of using configuration information stored in an AdaptationSet for three-dimensional model data.
- FIG. 22 is a hardware configuration diagram of a computer.
- Non Patent Literature 1 (as described above)
- Non Patent Literature 2 “ISO/IEC 14496-12:2015”, Information technology. Coding of audio-visual objects. Part 12: ISO base media file format, 2015-12
- Non Patent Literature 3 “ISO/IEC 23009-1:2014”, Information technology. Dynamic adaptive streaming over HTTP(DASH), Part 1: Media presentation description and segment formats, 2014-5
- the disclosures of the above-described non patent literatures are also incorporated into the present specification by reference. That is, the disclosures of the above-described non patent literatures also serve as a basis for determination as to support requirements.
- the structures/terms used for the scene description described in Non Patent Literature 1, the file structure described in Non Patent Literature 2, and the terms used for the MPEG-DASH standard described in Non Patent Literature 3 are not directly described in the detailed description of the invention, they are considered to fall within the scope of the disclosure of the present technology and satisfy the support requirements of the claims.
- technical terms such as parsing, syntax, and semantics are not directly described in the detailed description of the invention, they are also considered to fall within the scope of the disclosure of the present technology and satisfy the support requirements of the claims.
- FIG. 1 is a diagram illustrating a configuration of 6DoF content.
- a client may determine whether or not a client device can sufficiently exhibit its own reproduction capability, based on the following three indexes.
- a first index is an index for determining whether or not the scene description file and the three-dimensional model data file can be each individually decoded.
- a second index is an index for determining whether or not the scene description file and the three-dimensional model data file can be decoded together.
- a third index is an index for determining whether or not data rendering can be performed after decoding processing.
- the rendering refers to arrangement and display in a three-dimensional space. In a case where the client device determines whether or not reproduction processing can be performed with respect to each content configuration on the basis of such indexes, the use of the following information may be considered.
- First information is a @mimeType attribute and a @codecs attribute stored in an AdaptationSet representing the scene description among AdaptationSets included in the MPD file.
- the client device determines whether or not the scene description can be decoded. More specifically, it can be determined whether or not the client device supports the file format of the scene description using the @mimeType attribute. In addition, it can be determined whether or not the client device is compatible with a codec encoding the scene description using the @codecs attribute. Accordingly, it is ascertained which format the scene description has been created in between a MPEG-4 scene description format and a GL transmission format (glTF) 2.0 format, and it is ascertained whether or not the client device can reproduce the scene description.
- glTF GL transmission format
- the second information is a sceneProfileLevelIndication filed stored in a case where the scene description is represented in an ISO base media file format (ISOBMFF). Based on this information, the client device determines whether or not data rendering can be performed after the scene description is decoded. This information includes information for determining a reproduction processing capability of the client device used to reconstruct a three-dimensional space from the scene graph (hierarchical structure) represented by the scene description (data on the scene graph).
- ISO base media file format ISO base media file format
- the sceneProfileLevelIndication filed includes a maximum number of points for each scene
- the sceneProfileLevelIndication filed includes a maximum number of vertices of a face, a maximum number of faces, and a maximum number of vertices for each scene. That is, based on this information, it is ascertained what degree of reproduction processing capability is required for the entire scene.
- Third information is the number of external three-dimensional model data files for configuring the scene, which is obtained from the scene graph represented by the scene description file. Based on this information, the client device determines whether or not the scene description file and the three-dimensional model data file can be decoded. For example, the client device determines that reproduction is available when the number of its own decoders for three-dimensional model data is larger than the number of external three-dimensional model data files for configuring the scene. In this case, the larger the number of decoders used, the higher the reproduction processing capability required of the client device.
- Fourth information is a @mimeType attribute and a @codecs attribute stored in an AdaptationSet representing each three-dimensional model data among AdaptationSets included in the MPD file.
- the @mimeType attribute includes, for example, information about a file format in which 3D model data is stored.
- the @codecs attribute includes information about what codec the 3D model data is encoded by and information about a profile or a level of the codec. Based on this information, the client device determines whether or not each three-dimensional model data can be decoded. More specifically, it can be determined whether or not the client device supports the file format of each three-dimensional model data using the @mimeType attribute. In addition, it can be determined whether or not the client device is compatible with the codec encoding each three-dimensional model data using the @codecs attribute.
- the client device can determine whether or not rendering of each three-dimensional model data can be performed.
- the @codecs attribute includes a maximum number of points
- the @codecs attribute includes a maximum number of vertices of a face, a maximum number of faces, and a maximum number of vertices.
- Fifth information is a @bandwidth attribute stored in a Representation included in the MPD file for each three-dimensional model data. Based on this information, the client device determines whether or not each three-dimensional model data can be decoded. For example, by using this information, the client device can determine whether the bit rate is a bit rate at which the three-dimensional model data can be reproduced alone or a bit rate at which the three-dimensional model data can be reproduced in the entire scene.
- the first, fourth, and fifth information are used as the first index
- the third, fourth, and fifth information are used as the second index
- the second and fourth information are used as the third index.
- a content creator desires to provide a user with content that can be reproduced in a degree of freedom as high as possible, thereby enhancing the value of the content. Meanwhile, it is preferable that the number of reproducible client devices is large. Thus, the content creator may consider preparing a plurality of content configurations to distribute 6DoF content.
- the content configurations of the object-based, space-based, and mixed-type representation schemes will be referred to as an object-based content configuration, a space-based content configuration, and a mixed-type content configuration, respectively.
- the client device selects the mixed-type content configuration for reproduction if its reproduction processing capability is high, but selects the space-based content configuration for reproduction if its reproduction processing capability is low.
- the scene description is created to include two content configurations.
- the client device has analyzed the entire scene description and analyzed the information described in AdaptationSets of MPD for three-dimensional model data for configuring a scene using the first to fifth information. This processing is not efficient because a content configuration that is not to be actually used in each scene is analyzed.
- a client device is not provided with information for determining which one of contents having different content configurations from each other can be reproduced. For that reason, it has been difficult for the client device to determine whether or not reproduction processing is available unless decoding and rendering are actually performed. Therefore, a system enabling a client device to efficiently select a content configuration will be described.
- FIG. 2 is a system configuration diagram of an example of a distribution system.
- the distribution system 100 includes a file generation device 1 that is an information processing apparatus, a client device 2 that is a reproduction processing apparatus, and a Web server 3 .
- the file generation device 1 , the client device 2 , and the Web server 3 are connected to a network 4 . Then, the file generation device 1 , the client device 2 , and the Web server 3 can communicate with each other via the network 4 .
- the distribution system 100 may include a plurality of file generation devices 1 and a plurality of client devices 2 .
- the file generation device 1 generates 6DoF content.
- the file generation device 1 uploads the generated 6DoF content to the Web server 3 .
- the Web server 3 provides the 6DoF content to the client device 2
- the distribution system 100 can adopt another configuration.
- the file generation device 1 may include the functions of the Web server 3 to store the generated 6DoF content therein by itself, and provide the stored 6DoF content to the client device 2 .
- the Web server 3 retains the 6DoF content uploaded from the file generation device 1 . Then, the Web server 3 provides the 6DoF content designated by a request from the client device 2 .
- the client device 2 transmits a request to the Web server 3 to transmit the 6DoF content thereto. Then, the client device 2 acquires the 6DoF content designated by the request for transmission from the Web server 3 . Then, the client device 2 decodes the 6DoF content to generate a video, such that the video is displayed on a display device such as a monitor.
- the 6DoF content represents a three-dimensional space with one or more three-dimensional objects.
- the three-dimensional object is represented using a coordinate system in a bounding box normalized by a local coordinate system of the 6DoF content, and compressed and encoded into a bitstream.
- Scene description is used to arrange the bit stream in the three-dimensional space.
- a scene displaying each three-dimensional object at each time is represented by a graph in a tree hierarchical structure, which is called a scene graph, and the scene graph is represented in a binary format or in a text format.
- the scene graph is space display control information, and information related to the display of the three-dimensional object is configured by defining a node as a constituent unit and hierarchically combining a plurality of nodes.
- the nodes include a node for information about coordinate transformation from one coordinate system to another coordinate system, a node for information about a position and a size of a three-dimensional object, and a node for information about access to a three-dimensional object and audio data.
- the 6DoF content includes scene description data that is space display control information and media data of a plurality of three-dimensional objects (that is represented in accordance with, for example, mesh data and texture data of the three-dimensional objects).
- the 6DoF content may include audio data.
- the media data about the three-dimensional objects is also applicable in another format, such as point cloud.
- the scene description file is based on MPEG-4 Scene Description (ISO/IEC 14496-11).
- MPEG-4 Scene Description data is obtained by binarizing the scene graph in a format called a binary format for scenes (BIFS).
- BIFS binary format for scenes
- the scene graph can be transformed into the BIFS using a predetermined algorithm.
- a scene can be regulated at each time by storing a scene description in ISOBMFF, thereby making it possible to represent a three-dimensional object whose position and size change.
- FIG. 3 is a block diagram of the file generation device.
- the file generation device 1 which is an information processing apparatus, includes a generation processing unit 10 and a control unit 11 .
- the control unit 11 executes processing for controlling the generation processing unit 10 .
- the control unit 11 collectively controls operation timings of the respective parts of the generation processing unit 10 and the like.
- the generation processing unit 10 includes a data input unit 101 , a preprocessing unit 102 , an encoding unit 103 , a file generation unit 104 , and a transmission unit 105 .
- the data input unit 101 receives an input of original information for generating a three-dimensional object, meta information, and the like.
- the data input unit 101 outputs the acquired original information to the preprocessing unit 102 .
- the data input unit 101 receives an input of data.
- the data received by the data input unit 101 includes 3D objects and metadata such as information about arrangement of the 3D objects.
- the data input unit 101 outputs the acquired data to the preprocessing unit 102 .
- the preprocessing unit 102 receives the input of the data including the 3D objects and the metadata such as information about arrangement of the 3D objects from the data input unit 101 . Then, the preprocessing unit 102 determines a bit stream configuration on the basis of the acquired data, and generates a scene graph using the metadata of each 3D object and information about access to the bit stream.
- the metadata includes control information such as what codec is used for compression.
- the preprocessing unit 102 generates content configuration selection information for each content configuration, the content configuration selection information including any information of the above-described first to fifth information for one or more content configurations.
- the content configuration selection information provides an index of a reproduction processing capability required for reproducing a scene of each content configuration.
- the preprocessing unit 102 stores the content configuration selection information for each content configuration in the scene description. Accordingly, the client device 2 can select a content configuration capable of reproduction processing by using the content configuration selection information.
- the storage of the content configuration selection information according to the present embodiment will be described in detail.
- FIG. 4 is a diagram for describing a method of storing content configuration selection information according to the first embodiment.
- the preprocessing unit 102 aligns child nodes under the switch node in the scene description for each content configuration.
- a content configuration 301 is a content configuration in a mixed-type representation scheme
- a content configuration 302 is a content configuration in a space-based representation scheme.
- the preprocessing unit 102 extends the switch node to store information to be used for determining whether or not decoding and rendering can be performed with respect to an entire scene in each content configuration as content configuration selection information.
- FIG. 5 is a diagram illustrating an example of syntax for the extended switch node in the first embodiment.
- the preprocessing unit 102 indicates a plurality of content configurations in a choice field of the switch node. Further, the preprocessing unit 102 newly adds a Points field, a VertivesParFace field, a Faces field, an Indices field, a Num3DmodeData field, a 3DmodeIDataMimeType field, a 3DmodeDataCodec field, and a Bitrate field indicating content configuration selection information for each content configuration. Then, the preprocessing unit 102 stores values for each content configuration by storing values in the newly added fields in a content configuration order indicated in the choice field.
- Points is the number of points of a point cloud.
- VertivesParFace is the number of vertices of a face of a mesh. Faces is the number of faces of a mesh.
- Indices is the number of vertices of a mesh.
- Num3DmodelData is the number of externally referred-to three-dimensional model data.
- the Num3DmodelData corresponds to the third information.
- 3DmodelDataMimeType is a MimeType of externally referred-to three-dimensional model data.
- 3DmodelDataCodec is a codec of externally referred-to three-dimensional model data.
- the Num3DmodelData and 3DmodelDataCodec correspond to the fourth information.
- Bitrate is a bit rate including externally referred-to three-dimensional model data. The Bitrate corresponds to the fifth information.
- the preprocessing unit 102 outputs the three-dimensional object and the generated scene graph to the encoding unit 103 .
- the preprocessing unit 102 also outputs the metadata to the file generation unit 104 .
- the encoding unit 103 receives inputs of the three-dimensional object and the scene graph from the preprocessing unit 102 . Then, the encoding unit 103 encodes the three-dimensional object to generate a bit stream. Also, the encoding unit 103 encodes the acquired scene graph to generate a scene description. Thereafter, the encoding unit 103 outputs the generated bit stream and scene description to the file generation unit 104 .
- the file generation unit 104 receives inputs of the bit stream and the scene description from the encoding unit 103 .
- the file generation unit 104 also receives an input of the metadata from the preprocessing unit 102 . Then, the file generation unit 104 stores the acquired bit stream in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the bit stream.
- the file generation unit 104 also stores data about the scene description in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the scene description.
- the file generation unit 104 generates a media presentation description (MPD) file on the basis of the data acquired from the preprocessing unit 102 .
- MPD media presentation description
- a media type and meta information of 6DoF content such as information about a segment file for a moving image or a sound are stored.
- the transmission unit 105 acquires the ISOBMFF files of the bit stream and the scene description and the MPD file from the file generation unit 104 , and transmits and uploads the acquired files to the Web server 3 .
- FIG. 6 is a block diagram of the client device.
- the client device 2 includes a reproduction processing unit 20 and a control unit 21 .
- the control unit 21 controls operation of each part of the reproduction processing unit 20 .
- the control unit 21 collectively controls operation timings of the respective parts of the reproduction processing unit 20 .
- the reproduction processing unit 20 includes a file acquisition unit 201 , a measurement unit 202 , a file processing unit 203 , a decoding processing unit 204 , a display control unit 205 , a display information generation unit 206 , and a display unit 207 .
- the file acquisition unit 201 acquires an MPD file corresponding to 6DoF content to be reproduced from the Web server 3 . Then, the file acquisition unit 201 acquires information about a scene description of the 6DoF content to be reproduced on the basis of the MPD file.
- the file acquisition unit 201 accesses the Web server 3 and acquires an ISOBMFF file in which the scene description of 6DoF content to be displayed is stored. Then, the file acquisition unit 201 outputs the ISOBMFF file, in which the scene description is stored, to the file processing unit 203 .
- the file acquisition unit 201 acquires information about a bit stream selected by the file processing unit 203 from the file processing unit 203 . Then, the file acquisition unit 201 accesses the Web server 3 and acquires a segment file for the selected bit stream. Thereafter, the file acquisition unit 201 outputs the acquired segment file for the bit stream to the file processing unit 203 .
- the measurement unit 202 measures a transmission band of a transmission path between the client device 2 and the Web server. Then, the measurement unit 202 outputs a result of measuring the transmission band to the file processing unit 203 .
- the file processing unit 203 receives an input of the MPD file corresponding to the 6DoF content to be reproduced from the file acquisition unit 201 . Then, the file processing unit 203 parses the acquired MPD file to acquire information about a scene description of the 6DoF content to be reproduced. The file processing unit 203 also recognizes a plurality of data to be used for adaptive distribution. For example, if the adaptive distribution is for switching a bit rate, information about a segment file for a bit stream corresponding to each bit rate is acquired. In this case, the file processing unit 203 outputs the information about the scene description of the 6DoF content to be reproduced to the file acquisition unit 201 .
- the file processing unit 203 receives an input of the ISOBMFF file, in which the scene description is stored, from the file acquisition unit 201 .
- the file processing unit 203 parses the acquired ISOBMFF file.
- the file processing unit 203 acquires a switch node of the scene description.
- the file processing unit 203 acquires content configuration selection information from the switch node.
- the file processing unit 203 selects a content configuration to be used according to a reproduction processing capability of the client device 2 from the acquired content configuration selection information.
- the file processing unit 203 acquires child nodes in a scene graph corresponding to the selected content configuration.
- the file processing unit 203 acquires coordinate transformation information, arrangement information of the three-dimensional object, and access information in the scene of the selected content configuration.
- the file processing unit 203 receives an input of the result of measuring the transmission band from the measurement unit 202 . Then, the file processing unit 203 selects a segment file for a bit stream to be reproduced on the basis of a result of parsing the scene description, information indicating the transmission band acquired from the measurement unit 202 , and the like. Then, the file processing unit 203 outputs information about the selected segment file for the bit stream to the file acquisition unit 201 . In this case, the adaptive distribution according to the bit rate is implemented by changing the segment file for the bit stream to be selected according to the transmission band.
- the file processing unit 203 receives an input of the selected segment file for the bit stream from the file acquisition unit 201 . Then, the file processing unit 203 extracts bit stream data from the acquired segment file for the bit stream and outputs the extracted data to the decoding processing unit 204 .
- the decoding processing unit 204 receives an input of the bit stream data from the file processing unit 203 . Then, the decoding processing unit 204 decodes the acquired bit stream data. Thereafter, the decoding processing unit 204 outputs the decoded bit stream data to the display information generation unit 206 .
- the display control unit 205 receives an input of information about a viewpoint position and a line-of-sight direction of an operator from an input device, which is not illustrated. Then, the display control unit 205 outputs the acquired information about the viewpoint position and the viewpoint direction to the display information generation unit 206 .
- the display control unit 205 also receives, from the file processing unit 203 , an input of information about what kind of three-dimensional object there is.
- the operator can input designation information indicating a three-dimensional object of interest using the input device, instead of the information about the viewpoint position and the line-of-sight.
- the display control unit 205 acquires the designation information indicating the three-dimensional object of interest designated by the operator.
- the display control unit 205 outputs the information about the viewpoint position and the viewpoint direction to the display information generation unit 206 to track the three-dimensional object designated as the designation information over time. Accordingly, an image for tracking the three-dimensional object designated, for example, by the operator can be displayed.
- the display control unit 205 generates information for specifying the designated three-dimensional object in the 6DoF content.
- the display information generation unit 206 receives the scene description, the decoded bit stream data, and the acquired information about the viewpoint position and viewpoint direction to generate display information.
- the display information generation unit 206 will be described in detail below.
- the input of the bit stream data is received from the decoding processing unit 204 .
- the display information generation unit 206 arranges the three-dimensional object, which is the acquired bit stream data, in the three-dimensional space on the basis of the scene description. Further, the display information generation unit 206 receives the input of the information about the viewpoint position and the line-of-sight direction of the operator from the display control unit 205 . Then, the display information generation unit 206 performs rendering of the three-dimensional object arranged in the three-dimensional space in accordance with the viewpoint position and the line-of-sight direction to generate an image for display. Thereafter, the display information generation unit 206 supplies the generated image for display to the display unit 207 .
- the display unit 207 includes a display device such as a monitor.
- the display unit 207 receives an input of the image for display generated by the display information generation unit 206 . Then, the display unit 207 displays the acquired image for display on the display device.
- FIG. 7 is a flowchart of file generation processing by the file generation device according to the first embodiment.
- the preprocessing unit 102 generates content configuration selection information (Step S 1 ).
- the preprocessing unit 102 generates the content configuration selection information to be stored in a switch node of a scene description graph.
- the preprocessing unit 102 generates data of the scene graph in which the content configuration selection information is stored in the switch node (Step S 2 ).
- the preprocessing unit 102 outputs the data of the scene graph, in which the content configuration selection information is stored in the switch node, to the encoding unit 103 .
- the encoding unit 103 encodes the data about a three-dimensional object to generate a bit stream of the three-dimensional object. In addition, the encoding unit 103 encodes the acquired scene graph to generate a scene description (Step S 3 ).
- the file generation unit 104 stores the acquired bit stream in an ISOBMFF file for each segment to generate a segment file for the bit stream.
- the file generation unit 104 stores data about the scene description in an ISOBMFF file for each segment to generate a segment file for the scene description (Step S 4 ).
- the transmission unit 105 outputs the segment files generated by the file generation unit 104 to the Web server 3 (Step S 5 ).
- FIG. 8 is a flowchart of reproduction processing executed by the client device according to the first embodiment.
- the file acquisition unit 201 acquires an MPD file corresponding to 6DoF content to be reproduced from the Web server 3 (Step S 11 ).
- the file acquisition unit 201 outputs the acquired MPD file to the file processing unit 203 .
- the file processing unit 203 executes analysis processing by parsing the MPD file input from the file acquisition unit 201 (Step S 12 ).
- the file processing unit 203 specifies a scene description of the 6DoF content to be reproduced on the basis of an analysis result.
- the file acquisition unit 201 acquires the scene description specified by the file processing unit 203 .
- the file processing unit 203 parses the scene description acquired by the file acquisition unit 201 to acquire a switch node.
- the file processing unit 203 acquires content configuration selection information from the switch node (Step S 13 ).
- the file processing unit 203 selects a content configuration according to a reproduction processing capability of the client device 2 using the acquired content configuration selection information (Step S 14 ).
- the file processing unit 203 parses a child node corresponding to the selected content configuration. Thereafter, the file processing unit 203 acquires a segment file for a bit stream corresponding to the 6DoF content to be reproduced on the basis of a parsing result (Step S 15 ). The decoding processing unit 204 decodes the segment file for the bit stream. Thereafter, the decoding processing unit 204 outputs bit stream data to the display information generation unit 206 .
- the display control unit 205 outputs input information about a viewpoint position and a line-of-sight direction to the display information generation unit 206 .
- the display information generation unit 206 executes viewing processing by rendering the three-dimensional object and adding position information using the information about the viewpoint position and the line-of-sight direction acquired from the display control unit 205 to generate an image for display and displaying the generated image for display on the display unit 207 (Step S 16 ).
- the file generation device stores content configuration selection information in the switch node, which is a root node of the scene description, and provides the content configuration selection information to the client device.
- the client device acquires the switch node, which is the root node of the scene description, to obtain the content configuration selection information, and selects a content configuration using the acquired content configuration selection information.
- the client device can acquire the content configuration selection information once the switch node is acquired, and can acquire the information for selecting a content configuration without having to analyze other nodes for the scene description. Therefore, it is possible to efficiently select a content configuration.
- a content creator can prepare a plurality of content configurations so that 6DoF content can be distributed to correspond to client devices having different reproduction capabilities. Then, the client device can efficiently select a content configuration according to its own reproduction processing capability.
- a file generation device 1 according to the present modification is different from that according to the first embodiment in that content configuration selection information is stored in SampleEntry of an ISOBMFF file for scene description.
- a preprocessing unit 102 of the file generation device 1 according to the present modification generates content configuration selection information as meta. Then, the preprocessing unit 102 transmits metadata including the content configuration selection information to a file generation unit 104 .
- the file generation unit 104 receives inputs of the bit stream and the scene description from the encoding unit 103 .
- the file generation unit 104 also receives an input of the metadata including the content configuration selection information from the preprocessing unit 102 .
- the file generation unit 104 stores the acquired bit stream in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the bit stream, similarly to that in the first embodiment.
- the file generation unit 104 acquires the content configuration selection information from the metadata. Then, the file generation unit 104 stores the content configuration selection information in the SampleEntry of the ISOBMFF file for scene description.
- FIG. 9 is a diagram illustrating an ISOBMFF file for scene description in Modification (1) of the first embodiment. Then, the file generation unit 104 stores the content configuration selection information for each content configuration in the 6DoFContentStructBox.
- the file generation unit 104 stores content configuration selection information indicated by syntax illustrated in FIG. 10 in the 6DoFContentStructBox.
- FIG. 10 is a diagram illustrating an example of the syntax for the content configuration selection information stored in the SampleEntry in Modification (1) of the first embodiment.
- the file configuration selection information in this case is the same information as the newly added fields in the first embodiment as illustrated in FIG. 5 .
- the file generation unit 104 stores the data about the scene description in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the scene description. Then, the file generation unit 104 outputs the ISOBMFF file for scene description including file configuration information to the transmission unit 105 .
- a file processing unit 203 of a client device 2 receives an input of the ISOBMFF file for scene description from a file acquisition unit 201 . Then, Initialization Segment of the ISOBMFF file for scene description is acquired. Next, the file processing unit 203 acquires the content configuration selection information from the 6DoFContentStructBox in the acquired Initialization Segment. Then, the file processing unit 203 selects a content configuration to be used from the scene description using the acquired content configuration selection information.
- the client device can acquire content configuration selection information before analyzing the scene description itself, thereby selecting a content configuration without analyzing the scene description itself. In this way, the client device according to the present embodiment does not need to analyze the scene description itself to select a content configuration.
- the configuration of Modification (1) is valid in a case where the reproduction processing capability required for reproduction in the content configuration does not change every time period. In this case, it is not necessary to extend the scene description itself.
- a file generation device 1 according to the present modification is different from that according to the first embodiment in that a value of each content configuration selection information is determined for grouping, and it is indicated according to the grouping what group each content configuration belongs to, thereby indicating content configuration selection information of each content configuration.
- FIG. 11 is a diagram illustrating an example of a group of content configuration selection information.
- groups to which 01 to 03 are given as RequiedPerformanceID, which is an identification number, are set. Then, for each group, it is indicated what value is set for each content configuration selection information.
- a preprocessing unit 102 of the file generation device 1 has information about the groups of content configuration selection information illustrated in FIG. 11 . Then, the preprocessing unit 102 stores, in a switch node, information indicated by RequiedPerformanceID as to which group each content configuration corresponds to.
- the file processing unit 203 of the client device 2 analyzes the switch node of the scene description to acquire RequiedPerformanceID of a group to which each content configuration belongs. Then, the file processing unit 203 selects a content configuration by determining a reproduction processing capability required for reproduction in each content configuration from content configuration selection information allocated to a group to which each content configuration belongs.
- the file generation device notifies content configuration selection information for each content configuration using the groups of content configuration selection information. Accordingly, it is not necessary to generate detailed content configuration selection information for each content configuration, making the processing for generating the content configuration selection information simple.
- This scheme is also applicable to Modification (1) of the first embodiment.
- a file generation device 1 according to the present modification is different from that according to the first embodiment in that Matroska Media Container (http://www.matroska.org/) is used, instead of ISOBMFF, as a file format at the time of transmitting three-dimensional model data.
- Matroska Media Container http://www.matroska.org/
- FIG. 12 is a diagram illustrating a Matroska Media Container format.
- a file generation unit 104 of the file generation device 1 according to the present modification stores a 6DoFContentStruct element having content configuration selection information in a Track Entry element.
- the file generation unit 104 sets Element Type as binary, and stores SelectContentStructMetadata( ), which is illustrated in FIG. 10 , as binary data for extensible binary meta language (EBML) data.
- EBML extensible binary meta language
- a file processing unit 203 of a client device 2 acquires Initialization Segment of a Matroska Media Container file including scene description. Then, the file processing unit 203 acquires the content configuration selection information from the 6DoFContentStruct element included in the Initialization Segment to select a content configuration.
- container configuration selection information is generated for each container configuration in the present modification, the present modification is not limited thereto, and for example, Element Type can be stored as Integer to indicate RequiedPerformanceID as in Modification (2).
- a file generation device 1 according to the present embodiment is different from that according to the first embodiment in that content configuration selection information is stored for scene description of each content configuration.
- FIG. 13 is a diagram for describing a method of storing content configuration selection information according to Modification (4) of the first embodiment.
- a file generation unit 104 of the file generation device 1 newly defines a RequiedPerformance node 311 under a group node for a content configuration 301 , and newly defines a RequiedPerformance node 312 under a group node for a content configuration 302 . Then, the file generation unit 104 stores content configuration selection information in the RequiedPerformance node 311 for the content configuration 301 . Also, the file generation unit 104 stores content configuration selection information in the RequiedPerformance node 312 for the content configuration 302 .
- FIG. 14 is a diagram illustrating an example of syntax for the RequiedPerformance node.
- the file generation unit 104 defines a RequiedPerformance node as a node having content configuration selection information for one content configuration using the syntax illustrated in FIG. 14 .
- information serving as indexes for determining whether or not the content configuration is capable of reproduction is registered in the RequiedPerformance node.
- Points also represents the number of points of a point cloud.
- VertivesParFace represents the number of vertices of a face of a mesh. Faces represents the number of faces of a mesh. Indices represents the number of vertices of a mesh.
- Num3DmodelData represents the number of externally referred-to three-dimensional model data.
- 3DmodelDataMimeType represents a MimeType of externally referred-to three-dimensional model data.
- 3DmodelDataCodec represents a codec of externally referred-to three-dimensional model data.
- Bitrate represents a bit rate including externally referred-to three-dimensional model data.
- a file processing unit 203 of a client device 2 acquires a RequiedPerformance node for each content configuration. At this stage, the file processing unit 203 does not acquire another child node under this group node. Next, the file processing unit 203 acquires content configuration selection information for each content configuration from each RequiedPerformance node. Then, the file processing unit 203 selects a content configuration. Thereafter, the file processing unit 203 acquires and parses the group node and its subsequent nodes of the selected content configuration.
- the client device can select a content configuration by acquiring and analyzing up to the RequiedPerformance node immediately under the group node for each content configuration. Therefore, the processing can be reduced as compared with that in a case where the entire scene description is analyzed.
- the content configuration selection information can be provided to the client device without changing the existing node.
- a node storing the content configuration selection information is generated as a child node of the group node, but it may be a child node of another node as long as the another node is a root node for the content configuration.
- container configuration selection information is generated for each container configuration in the present modification, the present modification is not limited thereto, and for example, the RequiedPerformance node may be configured to retain the RequiedPerformanceID of Modification (2).
- a file generation device 1 according to the present modification is different from that according to the first embodiment in that content configuration selection information is stored in an AdaptationSet of an MPD file indicating information about access to scene description.
- a preprocessing unit 102 of the file generation device 1 according to the present modification generates content configuration selection information. Then, the preprocessing unit 102 transmits metadata including the content configuration selection information to a file generation unit 104 .
- the file generation unit 104 receives inputs of the bit stream and the scene description from the encoding unit 103 .
- the file generation unit 104 also receives an input of the metadata including the content configuration selection information from the preprocessing unit 102 .
- the file generation unit 104 stores the acquired bit stream in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the bit stream.
- the file generation unit 104 also stores data about the scene description in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the scene description.
- the file generation unit 104 generates an MPD file on the basis of the data acquired from the preprocessing unit 102 . At this time, the file generation unit 104 acquires the content configuration selection information included in the metadata. Then, the file generation unit 104 defines a 6DoFContentStruct descriptor illustrated in FIG. 15 in an AdaptationSet 320 of the MPD file for scene description as illustrated in FIG. 4 .
- FIG. 15 is a diagram illustrating an example of describing 6DoFContentStruct Descriptor. Further, in accordance with the acquired content configuration selection information, the file generation unit 104 stores a CSC element for each content configuration in the 6DoFContentStruct Descriptor, and registers the content configuration selection information as an attribute of the CSC element.
- FIG. 16 is a diagram illustrating semantics for CSC in Modification (5) of the first embodiment.
- the CSC describes a definition of an information element indicating a capability of a content configuration.
- the CSC includes one or more elements up to 255 elements.
- @Use indicates attribute information as to whether each element to be used is optional or mandatory.
- CSC@points represents the number of points of a point cloud.
- CSC@VertivesParFace represents the number of vertices of a face of a mesh.
- CSC@Faces represents the number of faces of a mesh.
- CSC@Indices represents the number of vertices of a mesh.
- CSC@Num3DmodelData represents the number of externally referred-to three-dimensional model data.
- CSC@3DmodelDataMimeType represents a MimeType of externally referred-to three-dimensional model data.
- CSC@3DmodelDataCodec represents a codec of externally referred-to three-dimensional model data.
- CSC@Bitrate represents a bit rate including externally referred-to three-dimensional model data.
- a file processing unit 203 of a client device 2 acquires the content configuration selection information by parsing the MPD file. That is, the file processing unit 203 can acquire the content configuration selection information at the time of acquiring the MPD file, and can determine whether or not there is reproducible content in the scene description.
- the client device can select a content configuration capable of reproduction without acquiring the scene description. Accordingly, it is possible to efficiently select a content configuration.
- the content configuration selection information is stored in the MPD file, it can be said to be valid in a case where the reproduction processing capability required for reproducing the content configuration does not change every time period.
- container configuration selection information is generated for each container configuration in the present modification, the present modification is not limited thereto, and for example, the attribute of the CSC element may be configured to store information indicating RequiedPerformanceID as in Modification (2).
- a file generation device 1 according to the present embodiment is different from that according to the first embodiment in that a scene description for each content configuration is configured in a separate file, and content configuration selection information is stored in an AdaptationSet for each scene description.
- the file generation device 1 according to the present embodiment is illustrated in the block diagram of FIG. 3 as well.
- a client device 2 according to the present embodiment is illustrated in the block diagram of FIG. 6 as well.
- description of operation of each unit that is identical to that in the first embodiment may be omitted.
- FIG. 17 is a diagram for describing a method of storing content configuration selection information according to the second embodiment.
- a preprocessing unit 102 of the file generation device 1 according to the present modification generates scene descriptions 331 and 332 for respective content configurations as illustrated in FIG. 17 .
- the scene description 331 is a scene description for content configuration #1.
- the scene description 332 is a scene description for content configuration #2.
- the preprocessing unit 102 generates content configuration selection information for each of the content configurations #1 and #2, and outputs the generated content configuration selection information to a file generation unit 104 .
- the file generation unit 104 acquires the scene descriptions 331 and 332 from an encoding unit 103 . In addition, the file generation unit 104 acquires the content configuration selection information for each of the content configurations #1 and #2 from the preprocessing unit 102 .
- the file generation unit 104 stores the scene descriptions 331 and 332 in ISOBMFF files.
- the file generation unit 104 stores information for each content configuration in an AdaptationSet of an MPD file for each of the scene descriptions 331 and 332 using 6DoFContentStruct Descriptor illustrated in FIG. 15 .
- a file processing unit 203 of a client device 2 acquires the MPD file from a file acquisition unit 201 . Then, the file processing unit 203 acquires the content configuration selection information included in each AdaptationSet of the MPD file for each of the scene descriptions 331 and 332 . Then, the file processing unit 203 selects a content configuration to be used by using the acquired content configuration selection information. Thereafter, the file processing unit 203 acquires the scene description of the selected content configuration from a Web server 3 via the file acquisition unit 201 .
- the client device can select a content configuration at the time of acquiring the MPD file.
- the schemes according to the first embodiment and the modifications thereof since a scene description including a content configuration that is not to be used is acquired, data that is not to be used is acquired.
- the client device according to the present embodiment does not need to acquire data about a scene description for an unnecessary content configuration, the client device according to the present embodiment can efficiently select a content configuration as compared with those according to the first embodiment and the modifications thereof.
- the content configuration selection information is stored in the MPD file, it can be said to be valid in a case where the reproduction processing capability required for reproducing the content configuration does not change every time period.
- a file generation device 1 according to the present modification is different from that according to the second embodiment in that content configuration selection information is stored in SampleEntry of an ISOBMFF file for scene description.
- a preprocessing unit 102 of the file generation device 1 according to the present modification generates content configuration selection information. Then, the preprocessing unit 102 transmits metadata including the content configuration selection information to a file generation unit 104 .
- the file generation unit 104 receives inputs of the bit stream and the scene description from the encoding unit 103 . In addition, the file generation unit 104 receives an input of the metadata including the content configuration selection information from the preprocessing unit 102 .
- the file generation unit 104 stores the acquired bit stream in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the bit stream.
- the file generation unit 104 also stores data about the scene description in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the scene description.
- the file generation unit 104 stores the content configuration selection information included in the metadata in the SampleEntry of the ISOBMFF file for scene description.
- the file generation unit 104 can store the content configuration selection information according to the storage method illustrated in FIGS. 9 and 10 .
- a file processing unit 203 of a client device 2 acquires the ISOBMFF file for scene description from a file acquisition unit 201 .
- the file processing unit 203 acquires Initialization Segment of the acquired ISOBMFF file for scene description.
- the file processing unit 203 acquires the content configuration selection information from 6DoFContentStructBox included in the Initialization Segment of the ISOBMFF file for scene description.
- the file processing unit 203 selects a content configuration to be used by using the acquired content configuration selection information.
- the file processing unit 203 acquires the scene description of the selected content configuration from a Web server 3 via the file acquisition unit 201 .
- the client device acquires the content configuration selection information from the 6DoFContentStructBox included in the Initialization Segment of the ISOBMFF file for scene description. Thereafter, the client device only needs to acquire data for a scene description of a content configuration that is to be used, and does not need to acquire data for a scene description of the other content configuration that is not to be used. Therefore, it is possible to efficiently select a content configuration.
- Matroska Media Container can be used instead of ISOBMFF.
- the file generation unit 104 can store the content configuration selection information and provide the content configuration selection information to the client device in the same manner as in Modification (3) of the first embodiment.
- a maximum processing capability, a bit rate, a definition, and the like can be changed by adaptation. Then, the reproduction processing capability required of the client changes due to the adaptation in bit rate or the like for each three-dimensional model data in one content configuration. For example, when a stream having the largest variation in the number of faces of a mesh as three-dimensional model data in a certain content configuration is selected, the client is required to have a high reproduction processing capability, but when a stream having the smallest variation in the number of faces of a mesh as three-dimensional model data in a certain content configuration is selected, the reproduction processing capability required of the client can be reduced.
- a content creator can increase the number of clients capable of reproduction by providing the content adapted in the content configuration.
- information for the client to determine whether the content adapted in the content configuration can be reproduced by the client is not provided. Therefore, a system enabling a client to efficiently select a content configuration in a case where there is adaptation in the content configuration will be described.
- a file generation device 1 according to the present embodiment is different from that according to the first embodiment in that, together with information indicating a maximum reproduction processing capability required for reproduction, information indicating a minimally required reproduction processing capability is generated and stored, and provided to a client device 2 .
- the file generation device 1 according to the present embodiment is illustrated in the block diagram of FIG. 3 as well.
- a client device 2 according to the present embodiment is illustrated in the block diagram of FIG. 6 as well.
- description of operation of each unit that is identical to that in the first embodiment may be omitted.
- a preprocessing unit 102 of the file generation device 1 generates content configuration selection information for each content configuration.
- the preprocessing unit 102 extends a switch node of the scene description as illustrated in FIG. 18 to store the content configuration selection information for each content configuration.
- FIG. 18 is a diagram illustrating an example of syntax for the extended switch node in the second embodiment.
- MaxPoints is a maximum number of points of a point cloud.
- MinPoints is a minimum number of points of a point cloud.
- MaxVertivesParFace is a maximum number of vertices of a face of a mesh.
- MinVertivesParFace is a minimum number of vertices of a face of a mesh.
- MaxFaces is a maximum number of faces of a mesh.
- MinFaces is a minimum number of faces of a mesh.
- MaxIndices is a maximum number of vertices of a mesh.
- MinIndices is a minimum number of vertices of a mesh.
- MaxNum3DmodelData is a maximum number of externally referred-to three-dimensional model data.
- MinNum3DmodelData is a minimum number of externally referred-to three-dimensional model data.
- Max3DmodelDataCodec is a maximum value of a codec of externally referred-to three-dimensional model data.
- Min3DmodelDataCodec is a minimum value of a codec of externally referred-to three-dimensional model data.
- MaxBitrate is a maximum bit rate including externally referred-to three-dimensional model data.
- MinBitrate is a minimum bit rate including externally referred-to three-dimensional model data.
- content configuration information indicating a maximum value is information indicating a reproduction processing capability with which the content can be reliably reproduced.
- content configuration information indicating a minimum value is information indicating a reproduction processing capability with which the content can be reproduced when adaptation is executed, and can be said to be information indicating a reproduction processing capability with which a part of the content can be reproduced.
- the preprocessing unit 102 outputs a scene graph including the switch node represented by syntax illustrated in FIG. 18 to an encoding unit 103 .
- a file processing unit 203 of the client device 2 acquires the switch node from the acquired scene description. Then, the file processing unit 203 acquires system configuration selection information for each system configuration from the switch node. Then, the file processing unit 203 selects a system configuration to be used by using the acquired system configuration selection information. In this case, even if the client device 2 does not satisfy a reproduction processing capability required based on content configuration selection information indicating a maximum value in a certain system configuration, the file processing unit 203 can select the content configuration as long as a reproduction processing capability required based on content configuration selection information indicating a minimum value is satisfied.
- the file processing unit 203 parses a group node and subsequent nodes of the selected system configuration for scene description.
- the client device since a maximum reproduction processing capability is required for a content configuration regardless of adaptation, a content configuration capable of reproduction if adaptation is performed therein is also not selected.
- the client device can select a content configuration in which reproduction is available when adaptation is performed for reproduction, even though it is difficult to perform reproduction in the selected content configuration when the highest reproduction processing capability is required.
- the content creator can also consider adaptation in a content configuration, so that 6DoF content can be distributed to correspond to client devices having different reproduction capabilities for reproduction. That is, the content creator can increase the number of client devices capable of reproduction with respect to one content configuration.
- the client device can consider adaptation in the content configuration to select a content configuration capable of reproduction according to its own reproduction processing capability.
- a file generation device 1 according to the present modification is different from that according to the third embodiment in that content configuration selection information is stored in SampleEntry of an ISOBMFF file for scene description.
- a file generation unit 104 receives an input of the content configuration selection information from a preprocessing unit 102 .
- the file generation unit 104 newly adds 6DoFContentStructBox to the SampleEntry of the ISOBMFF file for scene description.
- the file generation unit 104 stores content configuration selection information indicated by syntax illustrated in FIG. 19 in the 6DoFContentStructBox.
- FIG. 19 is a diagram illustrating an example of the syntax for the content configuration selection information stored in the SampleEntry in Modification (1) of the third embodiment.
- the file generation unit 104 outputs the ISOBMFF file for scene description including file configuration information to a transmission unit 105 .
- a file processing unit 203 of a client device 2 receives an input of the ISOBMFF file for scene description from a file acquisition unit 201 . Then, Initialization Segment of the ISOBMFF file for scene description is acquired. Next, the file processing unit 203 acquires the content configuration selection information from the 6DoFContentStructBox in the acquired Initialization Segment. Then, the file processing unit 203 selects a content configuration to be used from the scene description using the acquired content configuration selection information.
- the file processing unit 203 can select the content configuration as long as a reproduction processing capability required based on content configuration selection information indicating a minimum value is satisfied.
- the client device can also select a content configuration in which reproduction is available when adaptation is performed for reproduction. Accordingly, the content creator can also consider adaptation in a content configuration, so that 6DoF content can be distributed to correspond to client devices having different reproduction capabilities for reproduction.
- a file generation device 1 according to the present modification is different from that according to the second embodiment in that content configuration selection information is stored in an AdaptationSet of an MPD file indicating information about access to scene description.
- a preprocessing unit 102 of the file generation device 1 according to the present modification generates content configuration selection information. Then, the preprocessing unit 102 transmits metadata including the content configuration selection information to a file generation unit 104 .
- the file generation unit 104 receives inputs of the bit stream and the scene description from the encoding unit 103 .
- the file generation unit 104 also receives an input of the metadata including the content configuration selection information from the preprocessing unit 102 .
- the file generation unit 104 stores the acquired bit stream in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the bit stream.
- the file generation unit 104 also stores data about the scene description in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the scene description.
- the file generation unit 104 generates an MPD file on the basis of the data acquired from the preprocessing unit 102 .
- the file generation unit 104 defines a 6DoFContentStruct descriptor, as illustrated in FIG. 20 , in an AdaptationSet for scene description of the MPD file.
- the file generation unit 104 stores a CSC element for each content configuration in the 6DoFContentStruct Descriptor, and registers the content configuration selection information as an attribute of the CSC element.
- FIG. 20 is a diagram illustrating semantics for CSC in Modification (2) of the third embodiment. As illustrated in FIG. 20 , the CSC describes a definition of an information element indicating a capability of a content configuration.
- the file generation unit 104 outputs the ISOBMFF file for scene description including file configuration information to a transmission unit 105 .
- a file processing unit 203 of a client device 2 acquires the content configuration selection information by parsing the MPD file. Then, the file processing unit 203 selects a content configuration to be used from the scene description using the acquired content configuration selection information. In this case, even if the client device 2 does not satisfy a reproduction processing capability required based on content configuration selection information indicating a maximum value in a certain system configuration, the file processing unit 203 can select the content configuration as long as a reproduction processing capability required based on content configuration selection information indicating a minimum value is satisfied.
- the client device can also select a content configuration in which reproduction is available when adaptation is performed for reproduction. Accordingly, the content creator can also consider adaptation in a content configuration, so that 6DoF content can be distributed to correspond to client devices having different reproduction capabilities for reproduction.
- container configuration selection information is generated for each container configuration in the second embodiment and Modifications (1) and (2) thereof, they are not limited thereto, and for example, RequiedPerformanceID can be used to represent a maximum value and a minimum value.
- a file generation device 1 according to the present embodiment is different from those according to the first to third embodiments in that each configuration information for three-dimensional model data is stored in a Representation included in an AdaptationSet for three-dimensional model data in MPD.
- the file generation device 1 according to the present embodiment is illustrated in the block diagram of FIG. 3 as well.
- a client device 2 according to the present embodiment is illustrated in the block diagram of FIG. 6 as well.
- description of operation of each unit that is identical to that in the first embodiment may be omitted.
- a file generation unit 104 of the file generation device 1 acquires content configuration selection information for each content configuration from a preprocessing unit 102 . Then, when generating an MPD file, the file generation unit 104 stores the configuration information for three-dimensional model data in each Representation of the AdaptationSet for three-dimensional model data.
- the file generation unit 104 stores, for example, the number of points, in the case of a point cloud, in a Representation@numPoint attribute as the configuration information for three-dimensional model data.
- the file generation unit 104 stores, in the case of a mesh, the number of vertices of a face in a Representation@vpf attribute, the number of faces in a Representation@numFase attribute, and the number of vertices in a Representation@numIndices attribute.
- the file processing unit 203 of the client device 2 can select a content configuration capable of reproduction processing independently from the selection based on the content configuration selection information.
- the file generation unit 104 may store the configuration information in another element such as AdaptationSet or Preselection.
- the file generation unit 104 may store a minimum value and a maximum value in the AdaptationSet for three-dimensional model data of the MPD. For example, in the case of a point cloud, the file generation unit 104 stores a maximum value and a minimum value of the number of points in AdaptationSet@MaxNumPoint and AdaptationSet@MinNumPoint attributes, respectively.
- the file generation unit 104 stores a maximum value and a minimum value of the number of vertices of a face in AdaptationSet@MaxVpf and AdaptationSet@MinVpf attributes, respectively, stores a maximum value and a minimum value of the number of faces in AdaptationSet@MaxNumFace and AdaptationSet@MinFace attributes, respectively, and stores a maximum value and a minimum value of the number of vertices AdaptationSet@MaxNumIndices and AdaptationSet@MinIndices attributes, respectively.
- the configuration according to the present embodiment can be used to determine a reproduction processing capability required for reproduction in a case where three-dimensional model data is transmitted using an MPD file without using a scene description.
- a case where there is an AdaptationSet 340 for three-dimensional model data about a point cloud in an MPD file as in FIG. 21 will be described.
- FIG. 21 is a diagram for describing a method of using configuration information stored in an AdaptationSet for three-dimensional model data.
- the file generation unit 104 stores information about a maximum value and a minimum value of the number of points in the AdaptationSet 340 for three-dimensional model data. Furthermore, the file generation unit 104 stores the numbers of points in Representations 341 to 343 included in the AdaptationSet 340 .
- a file processing unit 203 of a client device 2 can determine whether or not there is a Representation capable of reproduction processing at a level of the AdaptationSet 340 . In a case where there is a Representation capable of reproduction processing, the file processing unit 203 can select the Representation capable of reproduction processing from among the Representations 341 to 343 for reproduction.
- the file generation unit 104 may store the configuration information for three-dimensional model data in another place.
- the file generation unit 104 newly defines 3DmModelDataMetadataBox in SampleEntry of an ISOBMFF file for three-dimensional model data.
- the file generation unit 104 may store the number of points in the newly defined 3DmModelDataMetadataBox, and in a case where the three-dimensional model data is a mesh, the file generation unit 104 may store the number of vertices of a face, the number of faces, and the number of vertices in the newly defined 3DmModelDataMetadataBox.
- the file generation unit 104 can use Matroska Media Container as a file format instead of ISOBMFF.
- the file generation unit 104 newly stores a 3DmodelDataMetadata element for the configuration information in a Track Entry element.
- the file generation unit 104 sets Element Type as binary.
- EBML data in a case where the D model data is a point cloud, the file generation unit 104 stores the number of points as binary data, and in a case where the D model data is a mesh, the file generation unit 104 stores the number of vertices of a face, the number of faces, and the number of vertices as binary data.
- the above-described embodiments and modifications thereof can also be applied in a case where it is determined whether or not one content configuration is a content configuration capable of reproduction processing.
- the content configuration selection information can be used. For example, a content creator may distribute content in a configuration using one three-dimensional model data and a spherical video. In this case, by adding information indicating a spherical video, which is not three-dimensional model data, as the content configuration selection information, the client device can determine whether or not the content has a content configuration capable of reproduction using the content configuration selection information.
- FIG. 22 is a hardware configuration diagram of a computer.
- the file generation device 1 and the client device 2 can be implemented by a computer 90 illustrated in FIG. 22 .
- a processor 91 a memory 92 , a network interface 93 , a non-volatile storage 94 , an input/output interface 95 , and a display interface 86 are connected to each other via a bus.
- external devices such as an input device, an output device, a storage device, and a drive are connected to the input/output interface 95 .
- the input device is, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, or the like.
- the output device is, for example, a speaker, an output terminal, or the like.
- the storage device is, for example, a hard disk, a random access memory (RAM) disk, or the like.
- the drive drives a removable medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
- a display 98 which is a display device, is connected to a display interface 96 .
- the network interface 93 is connected to an external network.
- the file generation device 1 and the client device 2 are connected to each other via the network interface 93 .
- the file generation device 1 and the client device 2 are connected to the Web server 3 via the network interface 93 .
- the non-volatile storage 94 is a built-in auxiliary storage device such as a hard disk or a solid state drive (SSD).
- the processor 91 loads a program stored, for example, in the non-volatile storage 94 into the memory 92 via the bus to execute the program, thereby performing the above-described series of processing.
- the memory 92 also appropriately stores data and the like required for the processor 91 to execute various types of processing.
- the program executed by the processor 91 can be recorded in the removable medium, for example, as a package medium or the like for application.
- the program can be installed in the non-volatile storage 94 via the input/output interface 95 by mounting the removable medium on the drive, which is an external device 97 .
- this program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.
- the program can be received through the network interface 93 and installed in the non-volatile storage 94 .
- this program can be installed in the non-volatile storage 94 in advance.
- An information processing apparatus comprising:
- a preprocessing unit that generates content configuration selection information, with respect to one or a plurality of contents, for determining whether or not each of the contents is reproducible, each of the contents having a content configuration including one or more three-dimensional objects and space arrangement information therefor to represent a virtual space;
- a file generation unit that generates a file including data about the virtual space and the content configuration selection information.
- the information processing apparatus wherein the preprocessing unit sets the content configuration selection information to include availability identification information for determining whether or not decoding and rendering are executable entirely with respect to the virtual space in a reproduction processing apparatus reproducing the contents.
- the information processing apparatus according to (1) or (2), wherein the preprocessing unit stores the content configuration selection information in a scene description.
- the information processing apparatus wherein the preprocessing unit stores the content configuration selection information in a switch node of the scene description.
- the information processing apparatus stores the content configuration selection information for each content configuration in the scene description.
- the preprocessing unit generates the content configuration selection information as metadata
- the file generation unit generates a content file storing the content configuration selection information.
- the information processing apparatus according to (6), wherein the file generation unit generates the content file as an ISO base media file format (ISOBMFF) file, and stores the content configuration selection information in 6DoFContentStructBox of SampleEntry of the content file.
- ISO base media file format ISO base media file format
- the information processing apparatus wherein the preprocessing unit has the content configuration selection information for each group in which the content configuration is determined in advance, and sets the content configuration selection information of the group to which each of the contents belongs as the content configuration selection information of each of the contents.
- the preprocessing unit generates the content configuration selection information as metadata
- the file generation unit generates a metadata file storing the content configuration selection information.
- the information processing apparatus wherein the file generation unit generates the metadata file as a media presentation description (MPD) file, and stores the content configuration selection information in AdaptationSet of the MPD file.
- MPD media presentation description
- the preprocessing unit generates different scene descriptions for the respective content configurations
- the file generation unit stores the content configuration selection information in AdaptationSet of an MPD file or 6DoFContentStructBox of SampleEntry of an ISOBMFF file for each of the scene descriptions.
- the information processing apparatus according to any one of (1) to (11), wherein the preprocessing unit sets information indicating a reproduction processing capability with which the content is reproducible as the content configuration selection information.
- the information processing apparatus wherein the preprocessing unit sets the content configuration selection information to include information indicating a reproduction processing capability with which a part of the content is reproducible.
- a reproduction processing apparatus comprising:
- a file acquisition unit that acquires a file including content configuration selection information, with respect to one or a plurality of contents, for determining whether or not each of the contents is reproducible, and data about the contents, each of the contents having a content configuration including one or more three-dimensional objects and space arrangement information therefor to represent a virtual space;
- a file processing unit that acquires the content configuration selection information from the file acquired by the file acquisition unit, determines whether or not each of the contents is reproducible on the basis of the acquired content configuration selection information, and selects the content to be reproduced;
- a reproduction unit that reproduces the content selected by the file processing unit.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computer Graphics (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Information Transfer Between Computers (AREA)
Abstract
Description
- The present invention relates to an information processing apparatus, an information processing method, a reproduction processing apparatus, and a reproduction processing method.
- In the current video distribution, a two-dimensional video, which is used to distribute a movie or the like, is mainly distributed. Hereinafter, the two-dimensional video may be referred to as two-dimensional (2D) content. Further, a 360-degree video that can be viewed in all directions is also distributed on a video distribution site on the web. Being viewable in all directions indicates that a line-of-sight direction can be freely selected. The 360-degree video is called a 3 degrees of freedom (3DoF) video or 3DoF content. It is basic in both the 2D content and the 3DoF content that a two-dimensionally encoded video is distributed from a distribution server and displayed to a client.
- There is also content called 3DoF+ content. The 3DoF+ content is content that can be viewed in all directions similarly to the 3DoF content and furthermore allows a slight shift of a viewpoint position. It is assumed that the shift of the viewpoint position in the 3DoF+ content is allowed in a range in which a user can move a head in a sitting state. In the 3DoF+ content, the shift of the viewpoint position is implemented by using one or a plurality of two-dimensionally encoded videos.
- Furthermore, it has been proposed to distribute a 6DoF video, which is called 6DoF content, as a video with a higher degree of freedom. The 6DoF video is a video that can be viewed in all directions in a three-dimensional space and can be viewed by walking around in the three-dimensional space displayed. Walking around in the three-dimensional space means that a viewpoint position can be freely selected. Hereinafter, the three-dimensional space may be referred to as 3D space.
- The 6DoF content is three-dimensional content in which a three-dimensional space is represented by one or a plurality of three-dimensional model data. The three-dimensional model data may be referred to as 3D model data, and the three-dimensional content may also be referred to as 3D content.
- As an example of a method of distributing 6DoF content, the 6DoF content is transmitted as a plurality of object streams by configuring a three-dimensional space with a plurality of three-dimensional model data. At that time, configuration information about the three-dimensional space, which is called a scene description, may be used. An example thereof is a moving picture experts group (MPEG)-4 scene description. The scene description, which is a representation method, is a method in which a scene is represented by a graph having a tree hierarchical structure, which is called a scene graph, while the scene graph is represented in a binary format.
- The 6DoF content is a video material representing a three-dimensional space with three-dimensional model data at each time. Examples of schemes for representing the 6DoF content include the following three schemes.
- One scheme is a representation scheme referred to as an object-based representation scheme in the present invention. In the object-based representation scheme, 6DoF content has a content configuration in which three-dimensional model data for each three-dimensional object such as a person or a thing, which is an individual target object to be displayed in a video, is arranged in a three-dimensional space to represent the entire three-dimensional space. The object-based representation scheme is characterized in that a client who performs reproduction of 6DoF content simultaneously processes the largest number of three-dimensional model data between the three schemes. On the other hand, in the object-based representation scheme, definition can be changed in displaying each three-dimensional object such as an individual person or thing. Therefore, it can be said that the object-based representation scheme is a configuration method in which a client has a high degree of freedom in reproduction processing between the three methods.
- Another one is a representation scheme referred to as a space-based representation scheme in the present invention. In the space-based representation scheme, 6DoF content has a content configuration in which an entire target three-dimensional space is represented as one three-dimensional model data without separating each three-dimensional object such as a person or a thing as three-dimensional model data. The space-based representation scheme is characterized in that a client processes one three-dimensional model data at the time of reproduction, requiring the lowest processing capability between the three schemes. On the other hand, it can be said that a client has an extremely low degree of freedom in reproduction processing, because the definition of the entire 6DoF content is fixed.
- The other one is a combination of a space-based representation scheme and an object-based representation scheme. Hereinafter, this representation scheme will be referred to as a mixed-type representation scheme. In the mixed-type representation scheme, 6DoF content has a content configuration in which a specific three-dimensional object is represented by independent three-dimensional model data, and a three-dimensional space excluding the specific three-dimensional object is represented by one three-dimensional model data. In the mixed-type representation scheme, a client uses a plurality of three-dimensional model data in reproduction processing, but the number of three-dimensional model data is smaller than that used in the object-based representation scheme. That is, in the mixed-type representation scheme, a client is required to have a higher processing capability than that in the space-based representation scheme, but may have a lower processing capability than that in the object-based representation scheme. In addition, it can similarly be said that a client has a higher degree of freedom in reproduction processing than that in the space-based representation scheme and a lower degree of freedom in reproduction processing than that in the object-based representation scheme.
- As described above, in each of the representation schemes, 6DoF content has a different content configuration. At that point, in a case where a scene description is created to include several 6DoF contents in different representation schemes, it is preferable that the client selects a content configuration of a representation scheme in which the degree of freedom is as high as possible, such that the viewing experience of the user can be further expanded.
- Non Patent Literature 1: “ISO/IEC 14496-11:2015”, Information technology. Coding of audio-visual objects. Part 11: Sene description and application engine, 2015-11
- However, in a case where an appropriate content configuration is selected from a scene description, a client selects the content configuration after performing various analyses. Examples of the analyses include analysis of the entire scene description, analysis of AdaptationSet in media presentation description (MPD), and the like. Such analyses include analysis of what is not to be actually used, and thus can be said to be less efficient for a client device in selecting a content configuration.
- Therefore, the present disclosure provides an information processing apparatus, an information processing method, a reproduction processing apparatus, and a reproduction processing method that enable a client device to efficiently select a content configuration.
- According to the present disclosure, a preprocessing unit generates content configuration selection information, with respect to one or a plurality of contents, for determining whether or not each of the contents is reproducible, each of the contents having a content configuration including one or more three-dimensional objects and space arrangement information therefor to represent a virtual space. A file generation unit generates a file including data about the virtual space and the content configuration selection information.
-
FIG. 1 is a diagram illustrating a configuration of 6DoF content. -
FIG. 2 is a system configuration diagram of an example of a distribution system. -
FIG. 3 is a block diagram of a file generation device. -
FIG. 4 is a diagram for describing a method of storing content configuration selection information according to a first embodiment. -
FIG. 5 is a diagram illustrating an example of syntax for an extended switch node in the first embodiment. -
FIG. 6 is a block diagram of a client device. -
FIG. 7 is a flowchart of file generation processing by the file generation device according to the first embodiment. -
FIG. 8 is a flowchart of reproduction processing executed by the client device according to the first embodiment. -
FIG. 9 is a diagram illustrating an ISOBMFF file for scene description in Modification (1) of the first embodiment. -
FIG. 10 is a diagram illustrating an example of syntax for content configuration selection information stored in SampleEntry in Modification (1) of the first embodiment. -
FIG. 11 is a diagram illustrating an example of a group of content configuration selection information. -
FIG. 12 is a diagram illustrating a Matroska Media Container format. -
FIG. 13 is a diagram for describing a method of storing content configuration selection information according to Modification (4) of the first embodiment. -
FIG. 14 is a diagram illustrating an example of syntax for a RequiedPerformance node. -
FIG. 15 is a diagram illustrating an example of describing 6DoFContentStruct Descriptor. -
FIG. 16 is a diagram illustrating semantics for CSC in Modification (5) of the first embodiment. -
FIG. 17 is a diagram for describing a method of storing content configuration selection information according to a second embodiment. -
FIG. 18 is a diagram illustrating an example of syntax for an extended switch node in the second embodiment. -
FIG. 19 is a diagram illustrating an example of the syntax for content configuration selection information stored in SampleEntry in Modification (1) of a third embodiment. -
FIG. 20 is a diagram illustrating semantics for CSC in Modification (2) of the third embodiment. -
FIG. 21 is a diagram for describing a method of using configuration information stored in an AdaptationSet for three-dimensional model data. -
FIG. 22 is a hardware configuration diagram of a computer. - Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following respective embodiments, the same parts are denoted by the same reference numerals, and description thereof will not be repeated. In addition, the scope of the present technology disclosed herein is not limited to the embodiments, and the contents of the following non patent literatures disclosed at the time of filing the application are also incorporated herein.
- Non Patent Literature 1: (as described above)
- Non Patent Literature 2: “ISO/IEC 14496-12:2015”, Information technology. Coding of audio-visual objects. Part 12: ISO base media file format, 2015-12
- Non Patent Literature 3: “ISO/IEC 23009-1:2014”, Information technology. Dynamic adaptive streaming over HTTP(DASH), Part 1: Media presentation description and segment formats, 2014-5
- That is, the disclosures of the above-described non patent literatures are also incorporated into the present specification by reference. That is, the disclosures of the above-described non patent literatures also serve as a basis for determination as to support requirements. For example, even though the structures/terms used for the scene description described in
Non Patent Literature 1, the file structure described inNon Patent Literature 2, and the terms used for the MPEG-DASH standard described inNon Patent Literature 3 are not directly described in the detailed description of the invention, they are considered to fall within the scope of the disclosure of the present technology and satisfy the support requirements of the claims. Similarly, for example, even though technical terms such as parsing, syntax, and semantics are not directly described in the detailed description of the invention, they are also considered to fall within the scope of the disclosure of the present technology and satisfy the support requirements of the claims. - In addition, the present disclosure will be described according to the following item order.
- 1. First Embodiment
- 1.1 Modification (1) of First Embodiment
- 1.2 Modification (2) of First Embodiment
- 1.3 Modification (3) of First Embodiment
- 1.4 Modification (4) of First Embodiment
- 1.5 Modification (5) of First Embodiment
- 2. Second Embodiment
- 2.1 Modification (1) of Second Embodiment
- 3. Third Embodiment
- 3.1 Modification (1) of Third Embodiment
- 3.2 Modification (2) of Third Embodiment
- 4. Fourth Embodiment
- When distributing 6DoF contents each having a content configuration in an object-based, space-based, or mixed-type representation scheme, for example, a scene description file, an MPD file, and a 3D model data file are configured as illustrated in
FIG. 1 to be distributed.FIG. 1 is a diagram illustrating a configuration of 6DoF content. In such current content configurations, a client may determine whether or not a client device can sufficiently exhibit its own reproduction capability, based on the following three indexes. - A first index is an index for determining whether or not the scene description file and the three-dimensional model data file can be each individually decoded. A second index is an index for determining whether or not the scene description file and the three-dimensional model data file can be decoded together. A third index is an index for determining whether or not data rendering can be performed after decoding processing. The rendering refers to arrangement and display in a three-dimensional space. In a case where the client device determines whether or not reproduction processing can be performed with respect to each content configuration on the basis of such indexes, the use of the following information may be considered.
- First information is a @mimeType attribute and a @codecs attribute stored in an AdaptationSet representing the scene description among AdaptationSets included in the MPD file. Based on this information, the client device determines whether or not the scene description can be decoded. More specifically, it can be determined whether or not the client device supports the file format of the scene description using the @mimeType attribute. In addition, it can be determined whether or not the client device is compatible with a codec encoding the scene description using the @codecs attribute. Accordingly, it is ascertained which format the scene description has been created in between a MPEG-4 scene description format and a GL transmission format (glTF) 2.0 format, and it is ascertained whether or not the client device can reproduce the scene description.
- The second information is a sceneProfileLevelIndication filed stored in a case where the scene description is represented in an ISO base media file format (ISOBMFF). Based on this information, the client device determines whether or not data rendering can be performed after the scene description is decoded. This information includes information for determining a reproduction processing capability of the client device used to reconstruct a three-dimensional space from the scene graph (hierarchical structure) represented by the scene description (data on the scene graph). For example, in the case of a point cloud, the sceneProfileLevelIndication filed includes a maximum number of points for each scene, and in the case of a mesh, the sceneProfileLevelIndication filed includes a maximum number of vertices of a face, a maximum number of faces, and a maximum number of vertices for each scene. That is, based on this information, it is ascertained what degree of reproduction processing capability is required for the entire scene.
- Third information is the number of external three-dimensional model data files for configuring the scene, which is obtained from the scene graph represented by the scene description file. Based on this information, the client device determines whether or not the scene description file and the three-dimensional model data file can be decoded. For example, the client device determines that reproduction is available when the number of its own decoders for three-dimensional model data is larger than the number of external three-dimensional model data files for configuring the scene. In this case, the larger the number of decoders used, the higher the reproduction processing capability required of the client device.
- Fourth information is a @mimeType attribute and a @codecs attribute stored in an AdaptationSet representing each three-dimensional model data among AdaptationSets included in the MPD file. The @mimeType attribute includes, for example, information about a file format in which 3D model data is stored. In addition, the @codecs attribute includes information about what codec the 3D model data is encoded by and information about a profile or a level of the codec. Based on this information, the client device determines whether or not each three-dimensional model data can be decoded. More specifically, it can be determined whether or not the client device supports the file format of each three-dimensional model data using the @mimeType attribute. In addition, it can be determined whether or not the client device is compatible with the codec encoding each three-dimensional model data using the @codecs attribute.
- Further, in a case where information about reproduction compatibility of three-dimensional model data is included in the @codecs attribute, the client device can determine whether or not rendering of each three-dimensional model data can be performed. In this case, for example, in a case where the three-dimensional model data is a point cloud, the @codecs attribute includes a maximum number of points, and in a case where the three-dimensional model data is a mesh, the @codecs attribute includes a maximum number of vertices of a face, a maximum number of faces, and a maximum number of vertices.
- Fifth information is a @bandwidth attribute stored in a Representation included in the MPD file for each three-dimensional model data. Based on this information, the client device determines whether or not each three-dimensional model data can be decoded. For example, by using this information, the client device can determine whether the bit rate is a bit rate at which the three-dimensional model data can be reproduced alone or a bit rate at which the three-dimensional model data can be reproduced in the entire scene.
- Among the above-described information, the first, fourth, and fifth information are used as the first index, the third, fourth, and fifth information are used as the second index, and the second and fourth information are used as the third index.
- Here, a content creator desires to provide a user with content that can be reproduced in a degree of freedom as high as possible, thereby enhancing the value of the content. Meanwhile, it is preferable that the number of reproducible client devices is large. Thus, the content creator may consider preparing a plurality of content configurations to distribute 6DoF content. Hereinafter, the content configurations of the object-based, space-based, and mixed-type representation schemes will be referred to as an object-based content configuration, a space-based content configuration, and a mixed-type content configuration, respectively.
- For example, in a case where the content creator prepares a mixed-type content configuration and a space-based content configuration, the client device selects the mixed-type content configuration for reproduction if its reproduction processing capability is high, but selects the space-based content configuration for reproduction if its reproduction processing capability is low. In this case, the scene description is created to include two content configurations. In this case, conventionally, in order to select a content configuration, the client device has analyzed the entire scene description and analyzed the information described in AdaptationSets of MPD for three-dimensional model data for configuring a scene using the first to fifth information. This processing is not efficient because a content configuration that is not to be actually used in each scene is analyzed.
- However, in a 6DoF content distribution system according to the related art, a client device is not provided with information for determining which one of contents having different content configurations from each other can be reproduced. For that reason, it has been difficult for the client device to determine whether or not reproduction processing is available unless decoding and rendering are actually performed. Therefore, a system enabling a client device to efficiently select a content configuration will be described.
-
FIG. 2 is a system configuration diagram of an example of a distribution system. Thedistribution system 100 includes afile generation device 1 that is an information processing apparatus, aclient device 2 that is a reproduction processing apparatus, and aWeb server 3. Thefile generation device 1, theclient device 2, and theWeb server 3 are connected to a network 4. Then, thefile generation device 1, theclient device 2, and theWeb server 3 can communicate with each other via the network 4. Here, although it is illustrated inFIG. 1 that each kind of device is included as one device, thedistribution system 100 may include a plurality offile generation devices 1 and a plurality ofclient devices 2. - The
file generation device 1 generates 6DoF content. Thefile generation device 1 uploads the generated 6DoF content to theWeb server 3. Here, although it is described in the present embodiment that theWeb server 3 provides the 6DoF content to theclient device 2, thedistribution system 100 can adopt another configuration. For example, thefile generation device 1 may include the functions of theWeb server 3 to store the generated 6DoF content therein by itself, and provide the stored 6DoF content to theclient device 2. - The
Web server 3 retains the 6DoF content uploaded from thefile generation device 1. Then, theWeb server 3 provides the 6DoF content designated by a request from theclient device 2. - The
client device 2 transmits a request to theWeb server 3 to transmit the 6DoF content thereto. Then, theclient device 2 acquires the 6DoF content designated by the request for transmission from theWeb server 3. Then, theclient device 2 decodes the 6DoF content to generate a video, such that the video is displayed on a display device such as a monitor. - Here, the 6DoF content will be described. The 6DoF content represents a three-dimensional space with one or more three-dimensional objects. The three-dimensional object is represented using a coordinate system in a bounding box normalized by a local coordinate system of the 6DoF content, and compressed and encoded into a bitstream. Scene description is used to arrange the bit stream in the three-dimensional space.
- There are a plurality of standards for the scene description. Basically, a scene displaying each three-dimensional object at each time is represented by a graph in a tree hierarchical structure, which is called a scene graph, and the scene graph is represented in a binary format or in a text format. Here, the scene graph is space display control information, and information related to the display of the three-dimensional object is configured by defining a node as a constituent unit and hierarchically combining a plurality of nodes. The nodes include a node for information about coordinate transformation from one coordinate system to another coordinate system, a node for information about a position and a size of a three-dimensional object, and a node for information about access to a three-dimensional object and audio data.
- Note that, in the following description, it is assumed that the 6DoF content includes scene description data that is space display control information and media data of a plurality of three-dimensional objects (that is represented in accordance with, for example, mesh data and texture data of the three-dimensional objects). In addition, the 6DoF content may include audio data. The media data about the three-dimensional objects is also applicable in another format, such as point cloud. Further, in the present embodiment, the scene description file is based on MPEG-4 Scene Description (ISO/IEC 14496-11).
- MPEG-4 Scene Description data is obtained by binarizing the scene graph in a format called a binary format for scenes (BIFS). The scene graph can be transformed into the BIFS using a predetermined algorithm. Furthermore, a scene can be regulated at each time by storing a scene description in ISOBMFF, thereby making it possible to represent a three-dimensional object whose position and size change.
- Next, the
file generation device 1 will be described in detail.FIG. 3 is a block diagram of the file generation device. As illustrated inFIG. 3 , thefile generation device 1, which is an information processing apparatus, includes ageneration processing unit 10 and a control unit 11. The control unit 11 executes processing for controlling thegeneration processing unit 10. For example, the control unit 11 collectively controls operation timings of the respective parts of thegeneration processing unit 10 and the like. Thegeneration processing unit 10 includes a data input unit 101, apreprocessing unit 102, anencoding unit 103, afile generation unit 104, and atransmission unit 105. - The data input unit 101 receives an input of original information for generating a three-dimensional object, meta information, and the like. The data input unit 101 outputs the acquired original information to the
preprocessing unit 102. The data input unit 101 receives an input of data. The data received by the data input unit 101 includes 3D objects and metadata such as information about arrangement of the 3D objects. The data input unit 101 outputs the acquired data to thepreprocessing unit 102. - The
preprocessing unit 102 receives the input of the data including the 3D objects and the metadata such as information about arrangement of the 3D objects from the data input unit 101. Then, thepreprocessing unit 102 determines a bit stream configuration on the basis of the acquired data, and generates a scene graph using the metadata of each 3D object and information about access to the bit stream. The metadata includes control information such as what codec is used for compression. - In addition, the
preprocessing unit 102 generates content configuration selection information for each content configuration, the content configuration selection information including any information of the above-described first to fifth information for one or more content configurations. The content configuration selection information provides an index of a reproduction processing capability required for reproducing a scene of each content configuration. - Then, the
preprocessing unit 102 stores the content configuration selection information for each content configuration in the scene description. Accordingly, theclient device 2 can select a content configuration capable of reproduction processing by using the content configuration selection information. Hereinafter, the storage of the content configuration selection information according to the present embodiment will be described in detail. -
FIG. 4 is a diagram for describing a method of storing content configuration selection information according to the first embodiment. As illustrated inFIG. 4 , thepreprocessing unit 102 aligns child nodes under the switch node in the scene description for each content configuration. InFIG. 4 , for example, acontent configuration 301 is a content configuration in a mixed-type representation scheme, and acontent configuration 302 is a content configuration in a space-based representation scheme. Thepreprocessing unit 102 extends the switch node to store information to be used for determining whether or not decoding and rendering can be performed with respect to an entire scene in each content configuration as content configuration selection information. -
FIG. 5 is a diagram illustrating an example of syntax for the extended switch node in the first embodiment. For example, thepreprocessing unit 102 indicates a plurality of content configurations in a choice field of the switch node. Further, thepreprocessing unit 102 newly adds a Points field, a VertivesParFace field, a Faces field, an Indices field, a Num3DmodeData field, a 3DmodeIDataMimeType field, a 3DmodeDataCodec field, and a Bitrate field indicating content configuration selection information for each content configuration. Then, thepreprocessing unit 102 stores values for each content configuration by storing values in the newly added fields in a content configuration order indicated in the choice field. - Points is the number of points of a point cloud. VertivesParFace is the number of vertices of a face of a mesh. Faces is the number of faces of a mesh. Indices is the number of vertices of a mesh. The Points, VertivesParFace, Faces, and Indices correspond to the second information. Num3DmodelData is the number of externally referred-to three-dimensional model data. The Num3DmodelData corresponds to the third information. 3DmodelDataMimeType is a MimeType of externally referred-to three-dimensional model data. 3DmodelDataCodec is a codec of externally referred-to three-dimensional model data. The Num3DmodelData and 3DmodelDataCodec correspond to the fourth information. Bitrate is a bit rate including externally referred-to three-dimensional model data. The Bitrate corresponds to the fifth information.
- The
preprocessing unit 102 outputs the three-dimensional object and the generated scene graph to theencoding unit 103. Thepreprocessing unit 102 also outputs the metadata to thefile generation unit 104. - The
encoding unit 103 receives inputs of the three-dimensional object and the scene graph from thepreprocessing unit 102. Then, theencoding unit 103 encodes the three-dimensional object to generate a bit stream. Also, theencoding unit 103 encodes the acquired scene graph to generate a scene description. Thereafter, theencoding unit 103 outputs the generated bit stream and scene description to thefile generation unit 104. - The
file generation unit 104 receives inputs of the bit stream and the scene description from theencoding unit 103. Thefile generation unit 104 also receives an input of the metadata from thepreprocessing unit 102. Then, thefile generation unit 104 stores the acquired bit stream in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the bit stream. Thefile generation unit 104 also stores data about the scene description in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the scene description. - In addition, the
file generation unit 104 generates a media presentation description (MPD) file on the basis of the data acquired from thepreprocessing unit 102. In the MPD file, a media type and meta information of 6DoF content such as information about a segment file for a moving image or a sound are stored. - The
transmission unit 105 acquires the ISOBMFF files of the bit stream and the scene description and the MPD file from thefile generation unit 104, and transmits and uploads the acquired files to theWeb server 3. -
FIG. 6 is a block diagram of the client device. As illustrated inFIG. 6 , theclient device 2 includes a reproduction processing unit 20 and acontrol unit 21. Thecontrol unit 21 controls operation of each part of the reproduction processing unit 20. For example, thecontrol unit 21 collectively controls operation timings of the respective parts of the reproduction processing unit 20. The reproduction processing unit 20 includes afile acquisition unit 201, a measurement unit 202, afile processing unit 203, a decoding processing unit 204, adisplay control unit 205, a displayinformation generation unit 206, and adisplay unit 207. - The
file acquisition unit 201 acquires an MPD file corresponding to 6DoF content to be reproduced from theWeb server 3. Then, thefile acquisition unit 201 acquires information about a scene description of the 6DoF content to be reproduced on the basis of the MPD file. - The
file acquisition unit 201 accesses theWeb server 3 and acquires an ISOBMFF file in which the scene description of 6DoF content to be displayed is stored. Then, thefile acquisition unit 201 outputs the ISOBMFF file, in which the scene description is stored, to thefile processing unit 203. - Further, the
file acquisition unit 201 acquires information about a bit stream selected by thefile processing unit 203 from thefile processing unit 203. Then, thefile acquisition unit 201 accesses theWeb server 3 and acquires a segment file for the selected bit stream. Thereafter, thefile acquisition unit 201 outputs the acquired segment file for the bit stream to thefile processing unit 203. - The measurement unit 202 measures a transmission band of a transmission path between the
client device 2 and the Web server. Then, the measurement unit 202 outputs a result of measuring the transmission band to thefile processing unit 203. - The
file processing unit 203 receives an input of the MPD file corresponding to the 6DoF content to be reproduced from thefile acquisition unit 201. Then, thefile processing unit 203 parses the acquired MPD file to acquire information about a scene description of the 6DoF content to be reproduced. Thefile processing unit 203 also recognizes a plurality of data to be used for adaptive distribution. For example, if the adaptive distribution is for switching a bit rate, information about a segment file for a bit stream corresponding to each bit rate is acquired. In this case, thefile processing unit 203 outputs the information about the scene description of the 6DoF content to be reproduced to thefile acquisition unit 201. - The
file processing unit 203 receives an input of the ISOBMFF file, in which the scene description is stored, from thefile acquisition unit 201. Thefile processing unit 203 parses the acquired ISOBMFF file. Then, thefile processing unit 203 acquires a switch node of the scene description. Thereafter, thefile processing unit 203 acquires content configuration selection information from the switch node. Next, thefile processing unit 203 selects a content configuration to be used according to a reproduction processing capability of theclient device 2 from the acquired content configuration selection information. Thereafter, thefile processing unit 203 acquires child nodes in a scene graph corresponding to the selected content configuration. Then, thefile processing unit 203 acquires coordinate transformation information, arrangement information of the three-dimensional object, and access information in the scene of the selected content configuration. - In addition, the
file processing unit 203 receives an input of the result of measuring the transmission band from the measurement unit 202. Then, thefile processing unit 203 selects a segment file for a bit stream to be reproduced on the basis of a result of parsing the scene description, information indicating the transmission band acquired from the measurement unit 202, and the like. Then, thefile processing unit 203 outputs information about the selected segment file for the bit stream to thefile acquisition unit 201. In this case, the adaptive distribution according to the bit rate is implemented by changing the segment file for the bit stream to be selected according to the transmission band. - Thereafter, the
file processing unit 203 receives an input of the selected segment file for the bit stream from thefile acquisition unit 201. Then, thefile processing unit 203 extracts bit stream data from the acquired segment file for the bit stream and outputs the extracted data to the decoding processing unit 204. - The decoding processing unit 204 receives an input of the bit stream data from the
file processing unit 203. Then, the decoding processing unit 204 decodes the acquired bit stream data. Thereafter, the decoding processing unit 204 outputs the decoded bit stream data to the displayinformation generation unit 206. - The
display control unit 205 receives an input of information about a viewpoint position and a line-of-sight direction of an operator from an input device, which is not illustrated. Then, thedisplay control unit 205 outputs the acquired information about the viewpoint position and the viewpoint direction to the displayinformation generation unit 206. - The
display control unit 205 also receives, from thefile processing unit 203, an input of information about what kind of three-dimensional object there is. Alternatively, the operator can input designation information indicating a three-dimensional object of interest using the input device, instead of the information about the viewpoint position and the line-of-sight. Thedisplay control unit 205 acquires the designation information indicating the three-dimensional object of interest designated by the operator. Then, for example, in a case where the three-dimensional object is tracked to view, thedisplay control unit 205 outputs the information about the viewpoint position and the viewpoint direction to the displayinformation generation unit 206 to track the three-dimensional object designated as the designation information over time. Accordingly, an image for tracking the three-dimensional object designated, for example, by the operator can be displayed. In addition, for example, in a case where the position of the three-dimensional object is displayed, thedisplay control unit 205 generates information for specifying the designated three-dimensional object in the 6DoF content. - The display
information generation unit 206 receives the scene description, the decoded bit stream data, and the acquired information about the viewpoint position and viewpoint direction to generate display information. The displayinformation generation unit 206 will be described in detail below. - The input of the bit stream data is received from the decoding processing unit 204. Then, the display
information generation unit 206 arranges the three-dimensional object, which is the acquired bit stream data, in the three-dimensional space on the basis of the scene description. Further, the displayinformation generation unit 206 receives the input of the information about the viewpoint position and the line-of-sight direction of the operator from thedisplay control unit 205. Then, the displayinformation generation unit 206 performs rendering of the three-dimensional object arranged in the three-dimensional space in accordance with the viewpoint position and the line-of-sight direction to generate an image for display. Thereafter, the displayinformation generation unit 206 supplies the generated image for display to thedisplay unit 207. - The
display unit 207 includes a display device such as a monitor. Thedisplay unit 207 receives an input of the image for display generated by the displayinformation generation unit 206. Then, thedisplay unit 207 displays the acquired image for display on the display device. - Next, a flow of file generation processing by the
file generation device 1 according to the first embodiment will be described in detail with reference toFIG. 7 .FIG. 7 is a flowchart of file generation processing by the file generation device according to the first embodiment. - The
preprocessing unit 102 generates content configuration selection information (Step S1). In the present embodiment, thepreprocessing unit 102 generates the content configuration selection information to be stored in a switch node of a scene description graph. - Then, the
preprocessing unit 102 generates data of the scene graph in which the content configuration selection information is stored in the switch node (Step S2). Thepreprocessing unit 102 outputs the data of the scene graph, in which the content configuration selection information is stored in the switch node, to theencoding unit 103. - Next, the
encoding unit 103 encodes the data about a three-dimensional object to generate a bit stream of the three-dimensional object. In addition, theencoding unit 103 encodes the acquired scene graph to generate a scene description (Step S3). - Next, the
file generation unit 104 stores the acquired bit stream in an ISOBMFF file for each segment to generate a segment file for the bit stream. In addition, thefile generation unit 104 stores data about the scene description in an ISOBMFF file for each segment to generate a segment file for the scene description (Step S4). - The
transmission unit 105 outputs the segment files generated by thefile generation unit 104 to the Web server 3 (Step S5). - Next, a flow of reproduction processing executed by the
client device 2 according to the present embodiment will be described with reference toFIG. 8 .FIG. 8 is a flowchart of reproduction processing executed by the client device according to the first embodiment. - The
file acquisition unit 201 acquires an MPD file corresponding to 6DoF content to be reproduced from the Web server 3 (Step S11). Thefile acquisition unit 201 outputs the acquired MPD file to thefile processing unit 203. - The
file processing unit 203 executes analysis processing by parsing the MPD file input from the file acquisition unit 201 (Step S12). Thefile processing unit 203 specifies a scene description of the 6DoF content to be reproduced on the basis of an analysis result. - Then, the
file acquisition unit 201 acquires the scene description specified by thefile processing unit 203. Next, thefile processing unit 203 parses the scene description acquired by thefile acquisition unit 201 to acquire a switch node. Next, thefile processing unit 203 acquires content configuration selection information from the switch node (Step S13). - Then, the
file processing unit 203 selects a content configuration according to a reproduction processing capability of theclient device 2 using the acquired content configuration selection information (Step S14). - Next, the
file processing unit 203 parses a child node corresponding to the selected content configuration. Thereafter, thefile processing unit 203 acquires a segment file for a bit stream corresponding to the 6DoF content to be reproduced on the basis of a parsing result (Step S15). The decoding processing unit 204 decodes the segment file for the bit stream. Thereafter, the decoding processing unit 204 outputs bit stream data to the displayinformation generation unit 206. - The
display control unit 205 outputs input information about a viewpoint position and a line-of-sight direction to the displayinformation generation unit 206. The displayinformation generation unit 206 executes viewing processing by rendering the three-dimensional object and adding position information using the information about the viewpoint position and the line-of-sight direction acquired from thedisplay control unit 205 to generate an image for display and displaying the generated image for display on the display unit 207 (Step S16). - As described above, the file generation device according to the present embodiment stores content configuration selection information in the switch node, which is a root node of the scene description, and provides the content configuration selection information to the client device. The client device acquires the switch node, which is the root node of the scene description, to obtain the content configuration selection information, and selects a content configuration using the acquired content configuration selection information.
- In a configuration in which content configuration selection information is not stored, the entire scene description has been analyzed to collect information for selecting a content configuration. In contrast, in the distribution system according to the present embodiment, the client device can acquire the content configuration selection information once the switch node is acquired, and can acquire the information for selecting a content configuration without having to analyze other nodes for the scene description. Therefore, it is possible to efficiently select a content configuration.
- A content creator can prepare a plurality of content configurations so that 6DoF content can be distributed to correspond to client devices having different reproduction capabilities. Then, the client device can efficiently select a content configuration according to its own reproduction processing capability.
- A
file generation device 1 according to the present modification is different from that according to the first embodiment in that content configuration selection information is stored in SampleEntry of an ISOBMFF file for scene description. - A
preprocessing unit 102 of thefile generation device 1 according to the present modification generates content configuration selection information as meta. Then, thepreprocessing unit 102 transmits metadata including the content configuration selection information to afile generation unit 104. - The
file generation unit 104 receives inputs of the bit stream and the scene description from theencoding unit 103. Thefile generation unit 104 also receives an input of the metadata including the content configuration selection information from thepreprocessing unit 102. - Concerning the bit stream, the
file generation unit 104 stores the acquired bit stream in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the bit stream, similarly to that in the first embodiment. - On the other hand, when data about the scene description is stored in an ISOBMFF file, the
file generation unit 104 acquires the content configuration selection information from the metadata. Then, thefile generation unit 104 stores the content configuration selection information in the SampleEntry of the ISOBMFF file for scene description. - For example, as illustrated in
FIG. 9 , thefile generation unit 104 newly adds 6DoFContentStructBox illustrated asBOX 303 to the SampleEntry of the ISOBMFF file for scene description.FIG. 9 is a diagram illustrating an ISOBMFF file for scene description in Modification (1) of the first embodiment. Then, thefile generation unit 104 stores the content configuration selection information for each content configuration in the 6DoFContentStructBox. - For example, the
file generation unit 104 stores content configuration selection information indicated by syntax illustrated inFIG. 10 in the 6DoFContentStructBox.FIG. 10 is a diagram illustrating an example of the syntax for the content configuration selection information stored in the SampleEntry in Modification (1) of the first embodiment. The file configuration selection information in this case is the same information as the newly added fields in the first embodiment as illustrated inFIG. 5 . - In this way, the
file generation unit 104 stores the data about the scene description in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the scene description. Then, thefile generation unit 104 outputs the ISOBMFF file for scene description including file configuration information to thetransmission unit 105. - A
file processing unit 203 of aclient device 2 according to the present modification receives an input of the ISOBMFF file for scene description from afile acquisition unit 201. Then, Initialization Segment of the ISOBMFF file for scene description is acquired. Next, thefile processing unit 203 acquires the content configuration selection information from the 6DoFContentStructBox in the acquired Initialization Segment. Then, thefile processing unit 203 selects a content configuration to be used from the scene description using the acquired content configuration selection information. - As described above, the client device according to the present modification can acquire content configuration selection information before analyzing the scene description itself, thereby selecting a content configuration without analyzing the scene description itself. In this way, the client device according to the present embodiment does not need to analyze the scene description itself to select a content configuration. The configuration of Modification (1) is valid in a case where the reproduction processing capability required for reproduction in the content configuration does not change every time period. In this case, it is not necessary to extend the scene description itself.
- A
file generation device 1 according to the present modification is different from that according to the first embodiment in that a value of each content configuration selection information is determined for grouping, and it is indicated according to the grouping what group each content configuration belongs to, thereby indicating content configuration selection information of each content configuration. -
FIG. 11 is a diagram illustrating an example of a group of content configuration selection information. In a table ofFIG. 11 , groups to which 01 to 03 are given as RequiedPerformanceID, which is an identification number, are set. Then, for each group, it is indicated what value is set for each content configuration selection information. - A
preprocessing unit 102 of thefile generation device 1 according to the present embodiment has information about the groups of content configuration selection information illustrated inFIG. 11 . Then, thepreprocessing unit 102 stores, in a switch node, information indicated by RequiedPerformanceID as to which group each content configuration corresponds to. - The
file processing unit 203 of theclient device 2 according to the present embodiment analyzes the switch node of the scene description to acquire RequiedPerformanceID of a group to which each content configuration belongs. Then, thefile processing unit 203 selects a content configuration by determining a reproduction processing capability required for reproduction in each content configuration from content configuration selection information allocated to a group to which each content configuration belongs. - As described above, the file generation device according to the present embodiment notifies content configuration selection information for each content configuration using the groups of content configuration selection information. Accordingly, it is not necessary to generate detailed content configuration selection information for each content configuration, making the processing for generating the content configuration selection information simple. This scheme is also applicable to Modification (1) of the first embodiment.
- A
file generation device 1 according to the present modification is different from that according to the first embodiment in that Matroska Media Container (http://www.matroska.org/) is used, instead of ISOBMFF, as a file format at the time of transmitting three-dimensional model data. -
FIG. 12 is a diagram illustrating a Matroska Media Container format. Afile generation unit 104 of thefile generation device 1 according to the present modification stores a 6DoFContentStruct element having content configuration selection information in a Track Entry element. At this time, thefile generation unit 104 sets Element Type as binary, and stores SelectContentStructMetadata( ), which is illustrated inFIG. 10 , as binary data for extensible binary meta language (EBML) data. - A
file processing unit 203 of aclient device 2 according to the present modification acquires Initialization Segment of a Matroska Media Container file including scene description. Then, thefile processing unit 203 acquires the content configuration selection information from the 6DoFContentStruct element included in the Initialization Segment to select a content configuration. - In this way, even in a case where a container format different from that in Modification (1) is used, content configuration selection information is provided, such that the client device can select a container configuration. Furthermore, although container configuration selection information is generated for each container configuration in the present modification, the present modification is not limited thereto, and for example, Element Type can be stored as Integer to indicate RequiedPerformanceID as in Modification (2).
- A
file generation device 1 according to the present embodiment is different from that according to the first embodiment in that content configuration selection information is stored for scene description of each content configuration.FIG. 13 is a diagram for describing a method of storing content configuration selection information according to Modification (4) of the first embodiment. - For example, as illustrated in
FIG. 4 , afile generation unit 104 of thefile generation device 1 according to the present modification newly defines aRequiedPerformance node 311 under a group node for acontent configuration 301, and newly defines aRequiedPerformance node 312 under a group node for acontent configuration 302. Then, thefile generation unit 104 stores content configuration selection information in theRequiedPerformance node 311 for thecontent configuration 301. Also, thefile generation unit 104 stores content configuration selection information in theRequiedPerformance node 312 for thecontent configuration 302. -
FIG. 14 is a diagram illustrating an example of syntax for the RequiedPerformance node. Specifically, thefile generation unit 104 defines a RequiedPerformance node as a node having content configuration selection information for one content configuration using the syntax illustrated inFIG. 14 . As illustrated inFIG. 14 , information serving as indexes for determining whether or not the content configuration is capable of reproduction is registered in the RequiedPerformance node. In this case, Points also represents the number of points of a point cloud. VertivesParFace represents the number of vertices of a face of a mesh. Faces represents the number of faces of a mesh. Indices represents the number of vertices of a mesh. Num3DmodelData represents the number of externally referred-to three-dimensional model data. 3DmodelDataMimeType represents a MimeType of externally referred-to three-dimensional model data. 3DmodelDataCodec represents a codec of externally referred-to three-dimensional model data. Bitrate represents a bit rate including externally referred-to three-dimensional model data. - A
file processing unit 203 of aclient device 2 according to the present modification acquires a RequiedPerformance node for each content configuration. At this stage, thefile processing unit 203 does not acquire another child node under this group node. Next, thefile processing unit 203 acquires content configuration selection information for each content configuration from each RequiedPerformance node. Then, thefile processing unit 203 selects a content configuration. Thereafter, thefile processing unit 203 acquires and parses the group node and its subsequent nodes of the selected content configuration. - As described above, the client device according to the present modification can select a content configuration by acquiring and analyzing up to the RequiedPerformance node immediately under the group node for each content configuration. Therefore, the processing can be reduced as compared with that in a case where the entire scene description is analyzed. In addition, the content configuration selection information can be provided to the client device without changing the existing node.
- Here, in the present modification, a node storing the content configuration selection information is generated as a child node of the group node, but it may be a child node of another node as long as the another node is a root node for the content configuration. Furthermore, although container configuration selection information is generated for each container configuration in the present modification, the present modification is not limited thereto, and for example, the RequiedPerformance node may be configured to retain the RequiedPerformanceID of Modification (2).
- A
file generation device 1 according to the present modification is different from that according to the first embodiment in that content configuration selection information is stored in an AdaptationSet of an MPD file indicating information about access to scene description. - A
preprocessing unit 102 of thefile generation device 1 according to the present modification generates content configuration selection information. Then, thepreprocessing unit 102 transmits metadata including the content configuration selection information to afile generation unit 104. - The
file generation unit 104 receives inputs of the bit stream and the scene description from theencoding unit 103. Thefile generation unit 104 also receives an input of the metadata including the content configuration selection information from thepreprocessing unit 102. - Then, the
file generation unit 104 stores the acquired bit stream in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the bit stream. Thefile generation unit 104 also stores data about the scene description in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the scene description. - Further, the
file generation unit 104 generates an MPD file on the basis of the data acquired from thepreprocessing unit 102. At this time, thefile generation unit 104 acquires the content configuration selection information included in the metadata. Then, thefile generation unit 104 defines a 6DoFContentStruct descriptor illustrated inFIG. 15 in anAdaptationSet 320 of the MPD file for scene description as illustrated inFIG. 4 . FIG. 15 is a diagram illustrating an example of describing 6DoFContentStruct Descriptor. Further, in accordance with the acquired content configuration selection information, thefile generation unit 104 stores a CSC element for each content configuration in the 6DoFContentStruct Descriptor, and registers the content configuration selection information as an attribute of the CSC element. -
FIG. 16 is a diagram illustrating semantics for CSC in Modification (5) of the first embodiment. As illustrated inFIG. 16 , the CSC describes a definition of an information element indicating a capability of a content configuration. The CSC includes one or more elements up to 255 elements. Here, @Use indicates attribute information as to whether each element to be used is optional or mandatory. - In a case where a plurality of CSC elements appear, the CSC elements are registered in a content configuration order described in the choice field of the switch node of the scene description. In this case, CSC@points represents the number of points of a point cloud. CSC@VertivesParFace represents the number of vertices of a face of a mesh. CSC@Faces represents the number of faces of a mesh. CSC@Indices represents the number of vertices of a mesh. CSC@Num3DmodelData represents the number of externally referred-to three-dimensional model data. CSC@3DmodelDataMimeType represents a MimeType of externally referred-to three-dimensional model data. CSC@3DmodelDataCodec represents a codec of externally referred-to three-dimensional model data. CSC@Bitrate represents a bit rate including externally referred-to three-dimensional model data.
- A
file processing unit 203 of aclient device 2 according to the present embodiment acquires the content configuration selection information by parsing the MPD file. That is, thefile processing unit 203 can acquire the content configuration selection information at the time of acquiring the MPD file, and can determine whether or not there is reproducible content in the scene description. - As described above, the client device according to the present embodiment can select a content configuration capable of reproduction without acquiring the scene description. Accordingly, it is possible to efficiently select a content configuration. However, in this case, since the content configuration selection information is stored in the MPD file, it can be said to be valid in a case where the reproduction processing capability required for reproducing the content configuration does not change every time period. Furthermore, although container configuration selection information is generated for each container configuration in the present modification, the present modification is not limited thereto, and for example, the attribute of the CSC element may be configured to store information indicating RequiedPerformanceID as in Modification (2).
- A
file generation device 1 according to the present embodiment is different from that according to the first embodiment in that a scene description for each content configuration is configured in a separate file, and content configuration selection information is stored in an AdaptationSet for each scene description. Thefile generation device 1 according to the present embodiment is illustrated in the block diagram ofFIG. 3 as well. Aclient device 2 according to the present embodiment is illustrated in the block diagram ofFIG. 6 as well. In the following description, description of operation of each unit that is identical to that in the first embodiment may be omitted. -
FIG. 17 is a diagram for describing a method of storing content configuration selection information according to the second embodiment. Apreprocessing unit 102 of thefile generation device 1 according to the present modification generates scene descriptions 331 and 332 for respective content configurations as illustrated inFIG. 17 . The scene description 331 is a scene description forcontent configuration # 1. In addition, the scene description 332 is a scene description forcontent configuration # 2. Then, thepreprocessing unit 102 generates content configuration selection information for each of thecontent configurations # 1 and #2, and outputs the generated content configuration selection information to afile generation unit 104. - The
file generation unit 104 acquires the scene descriptions 331 and 332 from anencoding unit 103. In addition, thefile generation unit 104 acquires the content configuration selection information for each of thecontent configurations # 1 and #2 from thepreprocessing unit 102. - Then, the
file generation unit 104 stores the scene descriptions 331 and 332 in ISOBMFF files. In addition, thefile generation unit 104 stores information for each content configuration in an AdaptationSet of an MPD file for each of the scene descriptions 331 and 332 using 6DoFContentStruct Descriptor illustrated inFIG. 15 . - A
file processing unit 203 of aclient device 2 according to the present embodiment acquires the MPD file from afile acquisition unit 201. Then, thefile processing unit 203 acquires the content configuration selection information included in each AdaptationSet of the MPD file for each of the scene descriptions 331 and 332. Then, thefile processing unit 203 selects a content configuration to be used by using the acquired content configuration selection information. Thereafter, thefile processing unit 203 acquires the scene description of the selected content configuration from aWeb server 3 via thefile acquisition unit 201. - As described above, the client device according to the present embodiment can select a content configuration at the time of acquiring the MPD file. Meanwhile, in the schemes according to the first embodiment and the modifications thereof, since a scene description including a content configuration that is not to be used is acquired, data that is not to be used is acquired. In this way, since the client device according to the present embodiment does not need to acquire data about a scene description for an unnecessary content configuration, the client device according to the present embodiment can efficiently select a content configuration as compared with those according to the first embodiment and the modifications thereof. However, in this case, since the content configuration selection information is stored in the MPD file, it can be said to be valid in a case where the reproduction processing capability required for reproducing the content configuration does not change every time period.
- A
file generation device 1 according to the present modification is different from that according to the second embodiment in that content configuration selection information is stored in SampleEntry of an ISOBMFF file for scene description. - A
preprocessing unit 102 of thefile generation device 1 according to the present modification generates content configuration selection information. Then, thepreprocessing unit 102 transmits metadata including the content configuration selection information to afile generation unit 104. - The
file generation unit 104 receives inputs of the bit stream and the scene description from theencoding unit 103. In addition, thefile generation unit 104 receives an input of the metadata including the content configuration selection information from thepreprocessing unit 102. - Then, the
file generation unit 104 stores the acquired bit stream in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the bit stream. Thefile generation unit 104 also stores data about the scene description in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the scene description. - When the scene description is stored in the ISOBMFF file, the
file generation unit 104 stores the content configuration selection information included in the metadata in the SampleEntry of the ISOBMFF file for scene description. In this case, thefile generation unit 104 can store the content configuration selection information according to the storage method illustrated inFIGS. 9 and 10 . - A
file processing unit 203 of aclient device 2 according to the present embodiment acquires the ISOBMFF file for scene description from afile acquisition unit 201. Next, thefile processing unit 203 acquires Initialization Segment of the acquired ISOBMFF file for scene description. Then, thefile processing unit 203 acquires the content configuration selection information from 6DoFContentStructBox included in the Initialization Segment of the ISOBMFF file for scene description. Then, thefile processing unit 203 selects a content configuration to be used by using the acquired content configuration selection information. Thereafter, thefile processing unit 203 acquires the scene description of the selected content configuration from aWeb server 3 via thefile acquisition unit 201. - As described above, the client device according to the present embodiment acquires the content configuration selection information from the 6DoFContentStructBox included in the Initialization Segment of the ISOBMFF file for scene description. Thereafter, the client device only needs to acquire data for a scene description of a content configuration that is to be used, and does not need to acquire data for a scene description of the other content configuration that is not to be used. Therefore, it is possible to efficiently select a content configuration.
- Here, as a file format at the time of transmitting three-dimensional model data, Matroska Media Container can be used instead of ISOBMFF. When the Matroska Media Container is used, the
file generation unit 104 can store the content configuration selection information and provide the content configuration selection information to the client device in the same manner as in Modification (3) of the first embodiment. - In distribution of 6DoF content, a maximum processing capability, a bit rate, a definition, and the like can be changed by adaptation. Then, the reproduction processing capability required of the client changes due to the adaptation in bit rate or the like for each three-dimensional model data in one content configuration. For example, when a stream having the largest variation in the number of faces of a mesh as three-dimensional model data in a certain content configuration is selected, the client is required to have a high reproduction processing capability, but when a stream having the smallest variation in the number of faces of a mesh as three-dimensional model data in a certain content configuration is selected, the reproduction processing capability required of the client can be reduced.
- That is, a content creator can increase the number of clients capable of reproduction by providing the content adapted in the content configuration. However, information for the client to determine whether the content adapted in the content configuration can be reproduced by the client is not provided. Therefore, a system enabling a client to efficiently select a content configuration in a case where there is adaptation in the content configuration will be described.
- A
file generation device 1 according to the present embodiment is different from that according to the first embodiment in that, together with information indicating a maximum reproduction processing capability required for reproduction, information indicating a minimally required reproduction processing capability is generated and stored, and provided to aclient device 2. Thefile generation device 1 according to the present embodiment is illustrated in the block diagram ofFIG. 3 as well. Aclient device 2 according to the present embodiment is illustrated in the block diagram ofFIG. 6 as well. In the following description, description of operation of each unit that is identical to that in the first embodiment may be omitted. - A
preprocessing unit 102 of thefile generation device 1 according to the present embodiment generates content configuration selection information for each content configuration. Next, thepreprocessing unit 102 extends a switch node of the scene description as illustrated inFIG. 18 to store the content configuration selection information for each content configuration. -
FIG. 18 is a diagram illustrating an example of syntax for the extended switch node in the second embodiment. MaxPoints is a maximum number of points of a point cloud. MinPoints is a minimum number of points of a point cloud. MaxVertivesParFace is a maximum number of vertices of a face of a mesh. MinVertivesParFace is a minimum number of vertices of a face of a mesh. MaxFaces is a maximum number of faces of a mesh. MinFaces is a minimum number of faces of a mesh. MaxIndices is a maximum number of vertices of a mesh. MinIndices is a minimum number of vertices of a mesh. MaxNum3DmodelData is a maximum number of externally referred-to three-dimensional model data. MinNum3DmodelData is a minimum number of externally referred-to three-dimensional model data. Max3DmodelDataCodec is a maximum value of a codec of externally referred-to three-dimensional model data. Min3DmodelDataCodec is a minimum value of a codec of externally referred-to three-dimensional model data. MaxBitrate is a maximum bit rate including externally referred-to three-dimensional model data. MinBitrate is a minimum bit rate including externally referred-to three-dimensional model data. - Here, content configuration information indicating a maximum value is information indicating a reproduction processing capability with which the content can be reliably reproduced. In addition, content configuration information indicating a minimum value is information indicating a reproduction processing capability with which the content can be reproduced when adaptation is executed, and can be said to be information indicating a reproduction processing capability with which a part of the content can be reproduced.
- The
preprocessing unit 102 outputs a scene graph including the switch node represented by syntax illustrated inFIG. 18 to anencoding unit 103. - A
file processing unit 203 of theclient device 2 according to the present embodiment acquires the switch node from the acquired scene description. Then, thefile processing unit 203 acquires system configuration selection information for each system configuration from the switch node. Then, thefile processing unit 203 selects a system configuration to be used by using the acquired system configuration selection information. In this case, even if theclient device 2 does not satisfy a reproduction processing capability required based on content configuration selection information indicating a maximum value in a certain system configuration, thefile processing unit 203 can select the content configuration as long as a reproduction processing capability required based on content configuration selection information indicating a minimum value is satisfied. - Thereafter, the
file processing unit 203 parses a group node and subsequent nodes of the selected system configuration for scene description. - In the first embodiment, since a maximum reproduction processing capability is required for a content configuration regardless of adaptation, a content configuration capable of reproduction if adaptation is performed therein is also not selected. In contrast, the client device according to the present embodiment can select a content configuration in which reproduction is available when adaptation is performed for reproduction, even though it is difficult to perform reproduction in the selected content configuration when the highest reproduction processing capability is required.
- Accordingly, the content creator can also consider adaptation in a content configuration, so that 6DoF content can be distributed to correspond to client devices having different reproduction capabilities for reproduction. That is, the content creator can increase the number of client devices capable of reproduction with respect to one content configuration. In addition, the client device can consider adaptation in the content configuration to select a content configuration capable of reproduction according to its own reproduction processing capability.
- A
file generation device 1 according to the present modification is different from that according to the third embodiment in that content configuration selection information is stored in SampleEntry of an ISOBMFF file for scene description. - A
file generation unit 104 receives an input of the content configuration selection information from apreprocessing unit 102. Thefile generation unit 104 newly adds 6DoFContentStructBox to the SampleEntry of the ISOBMFF file for scene description. Then, thefile generation unit 104 stores content configuration selection information indicated by syntax illustrated inFIG. 19 in the 6DoFContentStructBox.FIG. 19 is a diagram illustrating an example of the syntax for the content configuration selection information stored in the SampleEntry in Modification (1) of the third embodiment. - Thereafter, the
file generation unit 104 outputs the ISOBMFF file for scene description including file configuration information to atransmission unit 105. - A
file processing unit 203 of aclient device 2 according to the present modification receives an input of the ISOBMFF file for scene description from afile acquisition unit 201. Then, Initialization Segment of the ISOBMFF file for scene description is acquired. Next, thefile processing unit 203 acquires the content configuration selection information from the 6DoFContentStructBox in the acquired Initialization Segment. Then, thefile processing unit 203 selects a content configuration to be used from the scene description using the acquired content configuration selection information. In this case, even if theclient device 2 does not satisfy a reproduction processing capability required based on content configuration selection information indicating a maximum value in a certain system configuration, thefile processing unit 203 can select the content configuration as long as a reproduction processing capability required based on content configuration selection information indicating a minimum value is satisfied. - As described above, in the scheme in which the content configuration selection information is stored in the SampleEntry of the ISOBMFF file for scene description, the client device can also select a content configuration in which reproduction is available when adaptation is performed for reproduction. Accordingly, the content creator can also consider adaptation in a content configuration, so that 6DoF content can be distributed to correspond to client devices having different reproduction capabilities for reproduction.
- A
file generation device 1 according to the present modification is different from that according to the second embodiment in that content configuration selection information is stored in an AdaptationSet of an MPD file indicating information about access to scene description. - A
preprocessing unit 102 of thefile generation device 1 according to the present modification generates content configuration selection information. Then, thepreprocessing unit 102 transmits metadata including the content configuration selection information to afile generation unit 104. - The
file generation unit 104 receives inputs of the bit stream and the scene description from theencoding unit 103. Thefile generation unit 104 also receives an input of the metadata including the content configuration selection information from thepreprocessing unit 102. - Then, the
file generation unit 104 stores the acquired bit stream in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the bit stream. Thefile generation unit 104 also stores data about the scene description in an ISOBMFF file for each segment to be converted into a file, thereby generating a segment file for the scene description. - Further, the
file generation unit 104 generates an MPD file on the basis of the data acquired from thepreprocessing unit 102. At this time, thefile generation unit 104 defines a 6DoFContentStruct descriptor, as illustrated inFIG. 20 , in an AdaptationSet for scene description of the MPD file. Then, in accordance with the content configuration selection information, thefile generation unit 104 stores a CSC element for each content configuration in the 6DoFContentStruct Descriptor, and registers the content configuration selection information as an attribute of the CSC element.FIG. 20 is a diagram illustrating semantics for CSC in Modification (2) of the third embodiment. As illustrated inFIG. 20 , the CSC describes a definition of an information element indicating a capability of a content configuration. - Thereafter, the
file generation unit 104 outputs the ISOBMFF file for scene description including file configuration information to atransmission unit 105. - A
file processing unit 203 of aclient device 2 according to the present embodiment acquires the content configuration selection information by parsing the MPD file. Then, thefile processing unit 203 selects a content configuration to be used from the scene description using the acquired content configuration selection information. In this case, even if theclient device 2 does not satisfy a reproduction processing capability required based on content configuration selection information indicating a maximum value in a certain system configuration, thefile processing unit 203 can select the content configuration as long as a reproduction processing capability required based on content configuration selection information indicating a minimum value is satisfied. - As described above, in the scheme in which the content configuration selection information is stored in the AdaptationSet for scene description of the MPD file, the client device can also select a content configuration in which reproduction is available when adaptation is performed for reproduction. Accordingly, the content creator can also consider adaptation in a content configuration, so that 6DoF content can be distributed to correspond to client devices having different reproduction capabilities for reproduction.
- Note that, although container configuration selection information is generated for each container configuration in the second embodiment and Modifications (1) and (2) thereof, they are not limited thereto, and for example, RequiedPerformanceID can be used to represent a maximum value and a minimum value.
- A
file generation device 1 according to the present embodiment is different from those according to the first to third embodiments in that each configuration information for three-dimensional model data is stored in a Representation included in an AdaptationSet for three-dimensional model data in MPD. Thefile generation device 1 according to the present embodiment is illustrated in the block diagram ofFIG. 3 as well. Aclient device 2 according to the present embodiment is illustrated in the block diagram ofFIG. 6 as well. In the following description, description of operation of each unit that is identical to that in the first embodiment may be omitted. - A
file generation unit 104 of thefile generation device 1 according to the present embodiment acquires content configuration selection information for each content configuration from apreprocessing unit 102. Then, when generating an MPD file, thefile generation unit 104 stores the configuration information for three-dimensional model data in each Representation of the AdaptationSet for three-dimensional model data. - The
file generation unit 104 stores, for example, the number of points, in the case of a point cloud, in a Representation@numPoint attribute as the configuration information for three-dimensional model data. In addition, thefile generation unit 104 stores, in the case of a mesh, the number of vertices of a face in a Representation@vpf attribute, the number of faces in a Representation@numFase attribute, and the number of vertices in a Representation@numIndices attribute. - In the first to third embodiments and the respective examples thereof, by storing such configuration information, the
file processing unit 203 of theclient device 2 can select a content configuration capable of reproduction processing independently from the selection based on the content configuration selection information. Here, an example in which each configuration information for three-dimensional model data is stored in the Representation has been described. However, thefile generation unit 104 may store the configuration information in another element such as AdaptationSet or Preselection. - Further, in addition to the storage of the configuration information, the
file generation unit 104 may store a minimum value and a maximum value in the AdaptationSet for three-dimensional model data of the MPD. For example, in the case of a point cloud, thefile generation unit 104 stores a maximum value and a minimum value of the number of points in AdaptationSet@MaxNumPoint and AdaptationSet@MinNumPoint attributes, respectively. In addition, in the case of a mesh, thefile generation unit 104 stores a maximum value and a minimum value of the number of vertices of a face in AdaptationSet@MaxVpf and AdaptationSet@MinVpf attributes, respectively, stores a maximum value and a minimum value of the number of faces in AdaptationSet@MaxNumFace and AdaptationSet@MinFace attributes, respectively, and stores a maximum value and a minimum value of the number of vertices AdaptationSet@MaxNumIndices and AdaptationSet@MinIndices attributes, respectively. - The configuration according to the present embodiment can be used to determine a reproduction processing capability required for reproduction in a case where three-dimensional model data is transmitted using an MPD file without using a scene description. For example, a case where there is an AdaptationSet 340 for three-dimensional model data about a point cloud in an MPD file as in
FIG. 21 will be described.FIG. 21 is a diagram for describing a method of using configuration information stored in an AdaptationSet for three-dimensional model data. - For example, as illustrated in
FIG. 21 , thefile generation unit 104 stores information about a maximum value and a minimum value of the number of points in the AdaptationSet 340 for three-dimensional model data. Furthermore, thefile generation unit 104 stores the numbers of points inRepresentations 341 to 343 included in the AdaptationSet 340. - When parsing the MPD file, a
file processing unit 203 of aclient device 2 can determine whether or not there is a Representation capable of reproduction processing at a level of the AdaptationSet 340. In a case where there is a Representation capable of reproduction processing, thefile processing unit 203 can select the Representation capable of reproduction processing from among theRepresentations 341 to 343 for reproduction. - Alternatively, the
file generation unit 104 may store the configuration information for three-dimensional model data in another place. For example, thefile generation unit 104 newly defines 3DmModelDataMetadataBox in SampleEntry of an ISOBMFF file for three-dimensional model data. Then, in a case where the three-dimensional model data is a point cloud, thefile generation unit 104 may store the number of points in the newly defined 3DmModelDataMetadataBox, and in a case where the three-dimensional model data is a mesh, thefile generation unit 104 may store the number of vertices of a face, the number of faces, and the number of vertices in the newly defined 3DmModelDataMetadataBox. - In addition, the
file generation unit 104 can use Matroska Media Container as a file format instead of ISOBMFF. In this case, thefile generation unit 104 newly stores a 3DmodelDataMetadata element for the configuration information in a Track Entry element. At this time, thefile generation unit 104 sets Element Type as binary. As EBML data, in a case where the D model data is a point cloud, thefile generation unit 104 stores the number of points as binary data, and in a case where the D model data is a mesh, thefile generation unit 104 stores the number of vertices of a face, the number of faces, and the number of vertices as binary data. - In each of the above-described embodiments and modifications thereof, it has been described that there are a plurality of content configurations, but the above-described embodiments and modifications thereof can also be applied in a case where it is determined whether or not one content configuration is a content configuration capable of reproduction processing. Further, even in a case where the client device does not have a reproduction processing capability with respect to 6DoF content having specific three-dimensional model data, the content configuration selection information can be used. For example, a content creator may distribute content in a configuration using one three-dimensional model data and a spherical video. In this case, by adding information indicating a spherical video, which is not three-dimensional model data, as the content configuration selection information, the client device can determine whether or not the content has a content configuration capable of reproduction using the content configuration selection information.
-
FIG. 22 is a hardware configuration diagram of a computer. Thefile generation device 1 and theclient device 2 can be implemented by acomputer 90 illustrated inFIG. 22 . In thecomputer 90, aprocessor 91, amemory 92, anetwork interface 93, a non-volatile storage 94, an input/output interface 95, and a display interface 86 are connected to each other via a bus. - For example, external devices such as an input device, an output device, a storage device, and a drive are connected to the input/output interface 95. The input device is, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, or the like. The output device is, for example, a speaker, an output terminal, or the like. The storage device is, for example, a hard disk, a random access memory (RAM) disk, or the like. The drive drives a removable medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory. In addition, a
display 98, which is a display device, is connected to adisplay interface 96. - The
network interface 93 is connected to an external network. Thefile generation device 1 and theclient device 2 are connected to each other via thenetwork interface 93. In addition, thefile generation device 1 and theclient device 2 are connected to theWeb server 3 via thenetwork interface 93. The non-volatile storage 94 is a built-in auxiliary storage device such as a hard disk or a solid state drive (SSD). - In the
computer 90 configured as described above, theprocessor 91 loads a program stored, for example, in the non-volatile storage 94 into thememory 92 via the bus to execute the program, thereby performing the above-described series of processing. Thememory 92 also appropriately stores data and the like required for theprocessor 91 to execute various types of processing. - The program executed by the
processor 91 can be recorded in the removable medium, for example, as a package medium or the like for application. In this case, the program can be installed in the non-volatile storage 94 via the input/output interface 95 by mounting the removable medium on the drive, which is an external device 97. - Also, this program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In that case, the program can be received through the
network interface 93 and installed in the non-volatile storage 94. - Alternatively, this program can be installed in the non-volatile storage 94 in advance.
- Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present disclosure. In addition, the components of the different embodiments and modifications may be appropriately combined.
- Note that the effects described in the present specification are merely examples and are not limited, and other effects may be provided.
- Note that the present technology can also have the following configurations.
- (1)
- An information processing apparatus comprising:
- a preprocessing unit that generates content configuration selection information, with respect to one or a plurality of contents, for determining whether or not each of the contents is reproducible, each of the contents having a content configuration including one or more three-dimensional objects and space arrangement information therefor to represent a virtual space; and
- a file generation unit that generates a file including data about the virtual space and the content configuration selection information.
- (2)
- The information processing apparatus according to (1), wherein the preprocessing unit sets the content configuration selection information to include availability identification information for determining whether or not decoding and rendering are executable entirely with respect to the virtual space in a reproduction processing apparatus reproducing the contents.
- (3)
- The information processing apparatus according to (1) or (2), wherein the preprocessing unit stores the content configuration selection information in a scene description.
- (4)
- The information processing apparatus according to (3), wherein the preprocessing unit stores the content configuration selection information in a switch node of the scene description.
- (5)
- The information processing apparatus according to (3), wherein the preprocessing unit stores the content configuration selection information for each content configuration in the scene description.
- (6)
- The information processing apparatus according to (2), wherein
- the preprocessing unit generates the content configuration selection information as metadata, and
- the file generation unit generates a content file storing the content configuration selection information.
- (7)
- The information processing apparatus according to (6), wherein the file generation unit generates the content file as an ISO base media file format (ISOBMFF) file, and stores the content configuration selection information in 6DoFContentStructBox of SampleEntry of the content file.
- (8)
- The information processing apparatus according to (2), wherein the preprocessing unit has the content configuration selection information for each group in which the content configuration is determined in advance, and sets the content configuration selection information of the group to which each of the contents belongs as the content configuration selection information of each of the contents.
- (9)
- The information processing apparatus according to (2), wherein
- the preprocessing unit generates the content configuration selection information as metadata, and
- the file generation unit generates a metadata file storing the content configuration selection information.
- (10)
- The information processing apparatus according to (9), wherein the file generation unit generates the metadata file as a media presentation description (MPD) file, and stores the content configuration selection information in AdaptationSet of the MPD file.
- (11)
- The information processing apparatus according to (2), wherein
- with respect to the plurality of contents having different content configurations, the preprocessing unit generates different scene descriptions for the respective content configurations, and
- the file generation unit stores the content configuration selection information in AdaptationSet of an MPD file or 6DoFContentStructBox of SampleEntry of an ISOBMFF file for each of the scene descriptions.
- (12)
- The information processing apparatus according to any one of (1) to (11), wherein the preprocessing unit sets information indicating a reproduction processing capability with which the content is reproducible as the content configuration selection information.
- (13)
- The information processing apparatus according to (12), wherein the preprocessing unit sets the content configuration selection information to include information indicating a reproduction processing capability with which a part of the content is reproducible.
- (14)
- An information processing method for causing a computer to execute:
- generating content configuration selection information, with respect to one or a plurality of contents, for determining whether or not each of the contents is reproducible, each of the contents having a content configuration including one or more three-dimensional objects and space arrangement information therefor to represent a virtual space; and
- generating a file including data about the virtual space and the content configuration selection information.
- (15)
- A reproduction processing apparatus comprising:
- a file acquisition unit that acquires a file including content configuration selection information, with respect to one or a plurality of contents, for determining whether or not each of the contents is reproducible, and data about the contents, each of the contents having a content configuration including one or more three-dimensional objects and space arrangement information therefor to represent a virtual space;
- a file processing unit that acquires the content configuration selection information from the file acquired by the file acquisition unit, determines whether or not each of the contents is reproducible on the basis of the acquired content configuration selection information, and selects the content to be reproduced; and
- a reproduction unit that reproduces the content selected by the file processing unit.
- (16)
- A reproduction processing method for causing a computer to execute:
- acquiring a file including content configuration selection information, with respect to one or a plurality of contents, for determining whether or not each of the contents is reproducible, each of the contents having a content configuration including one or more three-dimensional objects and space arrangement information therefor to represent a virtual space;
- acquiring the content configuration selection information from the acquired file, determining whether or not each of the contents is reproducible on the basis of the acquired content configuration selection information, and selecting the content to be reproduced;
- acquiring data about the selected content; and
- reproducing the selected content using the acquired data.
- 1 FILE GENERATION DEVICE
- 2 CLIENT DEVICE
- 3 WEB SERVER
- 4 NETWORK
- 10 GENERATION PROCESSING UNIT
- 11 CONTROL UNIT
- 20 REPRODUCTION PROCESSING UNIT
- 21 CONTROL UNIT
- 101 DATA INPUT UNIT
- 102 PREPROCESSING UNIT
- 103 ENCODING UNIT
- 104 FILE GENERATION UNIT
- 105 TRANSMISSION UNIT
- 201 FILE ACQUISITION UNIT
- 202 MEASUREMENT UNIT
- 203 FILE PROCESSING UNIT
- 204 DECODING PROCESSING UNIT
- 205 DISPLAY CONTROL UNIT
- 206 DISPLAY INFORMATION GENERATION UNIT
- 207 DISPLAY UNIT
Claims (16)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/617,014 US20220239994A1 (en) | 2019-06-25 | 2020-03-31 | Information processing apparatus, information processing method, reproduction processing apparatus, and reproduction processing method |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201962866430P | 2019-06-25 | 2019-06-25 | |
PCT/JP2020/014884 WO2020261689A1 (en) | 2019-06-25 | 2020-03-31 | Information processing device, information processing method, reproduction processing device, and reproduction processing method |
US17/617,014 US20220239994A1 (en) | 2019-06-25 | 2020-03-31 | Information processing apparatus, information processing method, reproduction processing apparatus, and reproduction processing method |
Publications (1)
Publication Number | Publication Date |
---|---|
US20220239994A1 true US20220239994A1 (en) | 2022-07-28 |
Family
ID=74060549
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/617,014 Pending US20220239994A1 (en) | 2019-06-25 | 2020-03-31 | Information processing apparatus, information processing method, reproduction processing apparatus, and reproduction processing method |
Country Status (5)
Country | Link |
---|---|
US (1) | US20220239994A1 (en) |
EP (1) | EP3982638A4 (en) |
JP (1) | JPWO2020261689A1 (en) |
CN (1) | CN114026875A (en) |
WO (1) | WO2020261689A1 (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN117121495A (en) * | 2021-04-15 | 2023-11-24 | 索尼集团公司 | Information processing apparatus and method |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20090027381A1 (en) * | 2007-07-23 | 2009-01-29 | Samsung Electronics Co., Ltd. | Three-dimensional content reproduction apparatus and method of controlling the same |
US8497858B2 (en) * | 2009-02-17 | 2013-07-30 | Samsung Electronics Co., Ltd. | Graphic image processing method and apparatus |
US20150189367A1 (en) * | 2012-08-31 | 2015-07-02 | Funke Digital Tv Guide Gmbh | Electronic media content guide |
US9342864B2 (en) * | 2010-06-25 | 2016-05-17 | Microsoft Technology Licensing, Llc | Alternative semantics for zoom operations in a zoomable scene |
US11356648B2 (en) * | 2017-09-26 | 2022-06-07 | Canon Kabushiki Kaisha | Information processing apparatus, information providing apparatus, control method, and storage medium in which virtual viewpoint video is generated based on background and object data |
US20220406062A1 (en) * | 2019-09-30 | 2022-12-22 | Sony Group Corporation | Information processing apparatus, reproduction processing apparatus, and information processing method |
US11606576B2 (en) * | 2018-10-08 | 2023-03-14 | Samsung Electronics Co., Ltd. | Method and apparatus for generating media file comprising 3-dimensional video content, and method and apparatus for replaying 3-dimensional video content |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106464943B (en) * | 2014-06-30 | 2020-09-11 | 索尼公司 | Information processing apparatus and method |
JP6981257B2 (en) * | 2016-02-12 | 2021-12-15 | ソニーグループ株式会社 | Information processing equipment and information processing method |
-
2020
- 2020-03-31 WO PCT/JP2020/014884 patent/WO2020261689A1/en active Application Filing
- 2020-03-31 EP EP20831253.8A patent/EP3982638A4/en active Pending
- 2020-03-31 JP JP2021527391A patent/JPWO2020261689A1/ja active Pending
- 2020-03-31 US US17/617,014 patent/US20220239994A1/en active Pending
- 2020-03-31 CN CN202080044831.2A patent/CN114026875A/en active Pending
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20090027381A1 (en) * | 2007-07-23 | 2009-01-29 | Samsung Electronics Co., Ltd. | Three-dimensional content reproduction apparatus and method of controlling the same |
US8497858B2 (en) * | 2009-02-17 | 2013-07-30 | Samsung Electronics Co., Ltd. | Graphic image processing method and apparatus |
US9035942B2 (en) * | 2009-02-17 | 2015-05-19 | Samsung Electronics Co., Ltd. | Graphic image processing method and apparatus |
US9342864B2 (en) * | 2010-06-25 | 2016-05-17 | Microsoft Technology Licensing, Llc | Alternative semantics for zoom operations in a zoomable scene |
US20150189367A1 (en) * | 2012-08-31 | 2015-07-02 | Funke Digital Tv Guide Gmbh | Electronic media content guide |
US20150237418A1 (en) * | 2012-08-31 | 2015-08-20 | Funke Digital Tv Guide Gmbh | Electronic media server |
US11356648B2 (en) * | 2017-09-26 | 2022-06-07 | Canon Kabushiki Kaisha | Information processing apparatus, information providing apparatus, control method, and storage medium in which virtual viewpoint video is generated based on background and object data |
US11606576B2 (en) * | 2018-10-08 | 2023-03-14 | Samsung Electronics Co., Ltd. | Method and apparatus for generating media file comprising 3-dimensional video content, and method and apparatus for replaying 3-dimensional video content |
US20220406062A1 (en) * | 2019-09-30 | 2022-12-22 | Sony Group Corporation | Information processing apparatus, reproduction processing apparatus, and information processing method |
Also Published As
Publication number | Publication date |
---|---|
CN114026875A (en) | 2022-02-08 |
JPWO2020261689A1 (en) | 2020-12-30 |
EP3982638A4 (en) | 2023-03-29 |
EP3982638A1 (en) | 2022-04-13 |
WO2020261689A1 (en) | 2020-12-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2019202207A1 (en) | Processing video patches for three-dimensional content | |
US11825135B2 (en) | Information processing apparatus, information processing method, reproduction processing apparatus, and reproduction processing method | |
US10931930B2 (en) | Methods and apparatus for immersive media content overlays | |
JP7439762B2 (en) | Information processing device, information processing method, and program | |
GB2509953A (en) | Displaying a Region of Interest in a Video Stream by Providing Links Between Encapsulated Video Streams | |
JP7415936B2 (en) | Information processing device and information processing method | |
US11967153B2 (en) | Information processing apparatus, reproduction processing apparatus, and information processing method | |
KR20210019017A (en) | Content processing method and device | |
CN113302944A (en) | Information processing apparatus, information processing method, and computer program | |
US20200321015A1 (en) | Information processing device, information processing method, and program | |
KR101944601B1 (en) | Method for identifying objects across time periods and corresponding device | |
US20220239994A1 (en) | Information processing apparatus, information processing method, reproduction processing apparatus, and reproduction processing method | |
WO2019187437A1 (en) | Information processing device, information processing method, and program | |
WO2021065605A1 (en) | Information processing device and information processing method | |
WO2021002338A1 (en) | Information processing device, information processing method, reproduction processing device, and reproduction processing method | |
WO2019187434A1 (en) | Information processing device, information processing method, and program | |
US11922561B2 (en) | Methods and systems for implementing scene descriptions using derived visual tracks | |
US11974028B2 (en) | Information processing device, information processing method, reproduction processing device, and reproduction processing method | |
US20230043591A1 (en) | Information processing apparatus and method | |
US20220337800A1 (en) | Systems and methods of server-side dynamic adaptation for viewport-dependent media processing | |
VRT et al. | First Version of Playout Clients |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SONY GROUP CORPORATION, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KATSUMATA, MITSURU;TAKAHASHI, RYOHEI;HIRABAYASHI, MITSUHIRO;SIGNING DATES FROM 20211108 TO 20211112;REEL/FRAME:058315/0523 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |