EP4732543A1 - Texture mix signaling in scene description - Google Patents

Abstract

In one implementation, embodiments are provided for signaling of texture modifications for 3D scene representations. For example, one or more operators to apply to a texture image are encoded in MPEG-I Scene Description. A method for decoding one or more operators is provided which comprises parsing a scene description file comprising at least one texture information property referencing a texture, obtaining, for the at least texture information property, at least one operator signaled in the scene description file, and modifying the texture based on the at least one operator.

Description

TEXTURE MIX SIGNALING IN SCENE DESCRIPTION

This application claims the benefit of priority to European Application No. EP23305990.6, filed on 22 June 2023, which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present embodiments generally relate to texture encoding/decoding for 3D scene representations, more particularly, to signaling process modifications of a texture in a scene description file.

BACKGROUND

Three-dimensional scenes and models may be rendered in real-time to a user. Some formats such as the one proposed by Khronos / gITF (Graphics Language Transmission Format) and its extensions defined in MPEG Scene Description format or Apple / USDZ are possible ways to represent a 3D content to be rendered. Such formats support 3D model geometry, appearance, scene graph hierarchy, and animation. They are intended to be a streamlined, interoperable format for the delivery of 3D assets, while minimizing file size and runtime processing by applications. It is desirable to improve the representation of texture in such environment to offer more flexibility.

SUMMARY

The drawbacks and disadvantages of the prior art are solved and addressed by the general aspects described herein.

In one implementation, embodiments are provided for signaling of texture mix/changes/modifications for 3D scene representations. For example, embodiments allow to encode one or more operators to apply to a texture image in MPEG-I Scene Description.

According to an aspect, a method for signaling one or more operators is provided which comprises signaling in a scene description file, at least one texture information property referencing a texture and signaling in a scene description file, for the at least one texture information property, at least one operator to apply to the texture.

According to another aspect, a method for parsing or decoding one or more operators is provided which comprises parsing a scene description file comprising at least one texture information property referencing a texture, obtaining, for the at least texture information property, at least one operator signaled in the scene description file, modifying the texture based on the at least one operator.

According to another aspect, an apparatus is provided that comprises one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to perform the method mentioned above according to any one of the embodiments described herein.

One or more embodiments also provide a computer program comprising instructions which when executed by one or more processors cause the one or more processors to perform the method according to any of the embodiments described herein. One or more of the present embodiments also provide a computer readable storage medium having stored thereon instructions for processing scene description according to the methods described herein.

One or more embodiments also provide a computer readable storage medium having stored thereon a scene description generated according to the methods described above. One or more embodiments also provide a method and apparatus for transmitting or receiving the scene description generated according to the methods described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example architecture of a 3D content creation tools and modern graphics processing engine.

FIG. 2 shows an example of the syntax of a data stream encoding a 3D scene description.

FIG. 3 shows an example graph of a 3D scene description.

FIG. 4 illustrates the relations between top-level arrays either in a gITF asset or in a MPEG-I Scene Description (SD) into which the present principles may be implemented.

FIG. 5 illustrates an example of a method for encoding one or more operators in a scene description file according to an embodiment.

FIG. 6 illustrates an example of a method for decoding one or more operators from a scene description file according to an embodiment.

FIG. 7 illustrates an example of a method for decoding one or more operators from a scene description file according to another embodiment.

FIG. 8 illustrates an example of a texture resulting from a combination of two textures and a mask.

FIG. 9 illustrates an example of textures used in the combination illustrated on FIG. 8.

FIG. 10 illustrates an example of a method for parsing a Textureinfo property in a gITF file according to an embodiment.

FIG. 11 illustrates an example of a method for selecting an operator when parsing a TextureOperator property in a gITF file according to an embodiment.

FIG. 12 illustrates an example of a method for accessing a first operand of an operator when parsing a TextureOperator property in a gITF file according to an embodiment.

FIG. 13 illustrates an example of a method for accessing a second operand of an operator when parsing a TextureOperator property in a gITF file according to an embodiment. FIG. 14 illustrates an example of a method for accessing a third operand of an operator when parsing a TextureOperator property in a gITF file according to an embodiment.

FIG. 15 illustrates an example of a method for applying an addition operator to a texture according to an embodiment.

FIG. 16 illustrates an example of a method for applying a substract operator to a texture according to an embodiment.

FIG. 17 illustrates an example of a method for applying a modulate operator to a texture according to an embodiment.

FIG. 18 illustrates an example of a method for applying a replace operator to a texture according to an embodiment.

FIG. 19 illustrates an example of a method for applying an interpolate operator to a texture according to an embodiment.

FIG. 20 illustrates an example of a method for applying a blending operator to a texture according to an embodiment.

DETAILED DESCRIPTION

FIG. 1 shows an example architecture of 3D processing engine 130 which may be configured to implement the methods described herein. A device according to the architecture of FIG. 1 is linked with other devices via their bus 131 and/or via I/O interface 136.

Device 130 comprises following elements that are linked together by a data and address bus 131 : a microprocessor 132 (or CPU), which is, for example, a DSP (or Digital Signal Processor); a ROM (or Read Only Memory) 133; a RAM (or Random Access Memory) 134; a storage interface 135; an I/O interface 136 for reception of data to transmit, from an application; and a power supply (not represented in FIG. 1 ), e.g., a battery.

In accordance with an example, the power supply is external to the device. In each of mentioned memory, the word “register” used in the specification may correspond to area of small capacity (some bits) or to very large area (e.g., a whole program or large amount of received or decoded data). The ROM 133 comprises at least a program and parameters. The ROM 133 may store algorithms and instructions to perform techniques in accordance with present principles. When switched on, the CPU 132 uploads the program in the RAM and executes the corresponding instructions.

The RAM 134 comprises, in a register, the program executed by the CPU 132 and uploaded after switch-on of the device 130, input data in a register, intermediate data in different states of the method in a register, and other variables used for the execution of the method in a register.

Device 130 is linked, for example via bus 131 to a set of sensors 137 and to a set of rendering devices 138. Sensors 137 may be, for example, cameras, microphones, temperature sensors, Inertial Measurement Units, GPS, hygrometry sensors, IR or UV light sensors or wind sensors. Rendering devices 138 may be, for example, displays, speakers, vibrators, heat, fan, etc.

In accordance with examples, the device 130 is configured to implement a method according to the present principles, and belongs to a set comprising: a mobile device; a communication device; a game device; a tablet (or tablet computer); a laptop; a still picture camera; a video camera; a personal computer; a server.

In 3D applications, scene descriptions are used to combine explicit and easy-to-parse description of a scene structure and some binary representations of media content. FIG. 2 shows an example of the syntax of a data stream encoding a 3D scene description. FIG. 2 also shows an example structure 210 of a 3D scene description. The structure consists in a container which organizes the stream in independent elements of syntax. The structure may comprise a header part 220 which is a set of data common to every syntax element of the stream. For example, the header part comprises some of metadata about syntax elements, describing the nature and the role of each of them. The structure also comprises a payload comprising an element of syntax 230 and an element of syntax 240. Syntax element 230 comprises data representative of the media content items described in the nodes of the scene graph related to scene elements. Images, meshes and other raw data may have been compressed according to a compression method.

Alternatively, images, meshes and other raw data may have been directly embedded in scene description data 240 without being compressed. Element of syntax 240 is a part of the payload of the data stream and comprises data encoding the scene description as described according to the present principles.

FIG. 3 shows an example graph 310 of a 3D scene description. In this example, the scene graph may comprise a description of real objects, for example ‘plane horizontal surface’ (that can be a table or a road) and a description of 3D objects 312, for example an animation of a car. Scene description is organized as an array 310 of nodes. A node can be linked to child nodes to form a scene structure 31 1 . A node can carry a description of a real object (e.g., a semantic description) or a description of a 3D object. In the example of FIG. 3, node 301 describes a camera located in the 3D volume of the scene. Node 302 describes a car and comprises an index of a representation of the car, for example an index in an array of 3D meshes. Node 303 is a child of node 302 and comprises a description of one wheel of the car. The same way, it comprises an index to the 3D mesh of the wheel. The same 3D mesh may be used for several objects in the 3D scene as the scale, location and orientation of objects are described in the scene nodes.

FIG. 4 illustrates an example of a solution to represent texture in MPEG-I Scene Description (SD). In particular, FIG. 4 represents a gITF file structure. The entry point is the “scene” node (430) which contains “node” node(s) (435) that can be either a “camera” (410) or a “mesh” (440). The MPEG-I Scene Description (SD) format allows the storage in gITF files of texture image (490) or a link . The representation of textures (490) uses images (498) and textures samplers (490) in the gITF file. The image assets (498) define images loaded from a file or included in the gITF file (with buffer views). The texture assets (490) point to samplers allowing to define how to use the image data. For instance, the MPEG-I Scene Description (SD) format allows the definition of sampling and wrapping strategies. Moreover, an extension may apply at the gITF texture level, for instance KHR_texture_transforms, which specifies more complex transforms to apply on a texture image such as rotate or rescale the image. Then, materials assets (470) in the gITF file reference textures may be used to define how the texture is used in the rendering, like changing colors, normals, etc.

Today, this approach is limited as it does not provide flexibility in combining different texture images, or while trying to animate the texture. None of the processes mentioned above allows for the combination of different texture images using operators.

Textures mix can be performed before the creation of the gITF file. During these preprocessing steps, one computes the combination and only stores the resulting texture in the gITF file. It allows the transportation and rendering of the final scene. However, all the combination process is lost and any further changes in the rendering of the final scene are no longer allowed on the texture part. Animation of texture can be also performed with “baking” processing. The texture image for each frame of the animation can be precomputed and then sent alongside the gITF file. However, the skilled in the art will recognize that such processing quickly results in a very large amount of image data, and there is still no way to modify these combinations. It is therefore desirable to improve the signaling/encoding of the texture assets combining different texture images or animating the texture along with time.

MPEG_Texture_Mix extension in MPEG-I Scene Description

The proposed extension according to any of the embodiments disclosed hereafter is advantageously compliant with the gITF format and with the recent MPEG-I SD effort to extend gITF with MPEG-I SD extensions. The skilled in the art will recognize that the meaning and use are of the various disclosed embodiments are generic and may be coded in other formats (XML, USD, ...).

In the current MPEG-I, a combination of different texture images or animation of the texture has to be defined and computed in advance which is not convenient for adapting the texture to a parametric mesh for instance.

Therefore, one embodiment proposes to add an extension to a gITF file that provides for signaling one or more operators to be applied to a base texture. In this way, the base texture can be modified using the one or more operators when parsing the gITF file: texture = operator_n-1 ... operator^ operator₀ textureBase))) ( 1 ) where n is a number of operators to apply to the textureBase.

In an embodiment, each operator operator₀, operator^ can have from one to four operands, the first one always being the texture base, and the following ones being an output of the previous operation. All operations can clamp values to ensure that values are in a given range, for instance a range of [0..1 ].

The operation can be run on all or on a subset of channels. For an operation that does not operate on all channels, the others are left unchanged. For instance, if the current texture has four channels Red, Green, Blue and Alpha, and the operation only processes the Red, Green and Blue channels, then the output has new Red, Green and Blue values, and the same Alpha value than the first operand.

The proposed operations are designed to lead to an efficient implementation using standard shaders. For instance, they can be implemented using OpenGL existing implementations. Indeed, they are all pixel-based and their output can be interpolated: they can be implemented in a fragment shader.

Examples of operators that can be signaled are provided below. Such examples are not limiting and other operators not mentioned herein can also be signaled using the syntax of an extension as described herein according to the embodiments.

In the following, output _channeis represents the channels being modified by the operation of the operator, and output_others the other channels, that is the channels that are not being modified by the operation of the operator. An operator ADD has the following operands:

Operand 1 : A texture.

Operand 2: A texture with a same layout as operand 1 .

A result of this operator is: output channels ⁼ operand-^ + operand₂ output_others = operand^

An operator SUBTRACT has the following operands:

Operand 1 : A texture.

Operand 2: A texture with the same layout as operand 1 .

A result of this operator is: output _channels = operand^ - operand₂ output_others = operand-^

An operator MODULATE has the following operands:

Operand 1 : A texture.

Operand 2: A texture with a same layout as operand 1 .

A result of this operator is: output _{C ann}eis = operand-^ * operand₂ output_others = operand^

An operator REPLACE has the following operands:

Operand 1 : A texture.

Operand 2: A texture with a same layout as operand 1 .

A result of this operator is: output channels = operand₂ output_others = operand-^

An operator INTERPOLATE has the following operands:

Operand 1 : A texture.

Operand 2: A texture with a same layout as operand 1 .

Operand 3: A weight vector, with a value in [0,1] for each channel to process.

A result of this operator is: outputchannels = operand-^ * weight + operand₂ * (1 — weight) output_others = operand^

An operator BLEND has the following operands:

Operand 1 : A texture.

Operand 2: A texture with a same layout as operand 1 .

Operand 3: A texture with a same size as operand 1 (channels can be different). Operand 4: A string that defines how operand 3 is converted to a single-channel texture. For example the string values can be “ALPHA” or “INTENSITY”.

The fourth operand (Operand 4) can lead to the following data, depending on its string value: "ALPHA": factor = alpha operandf)

"INTENSITY": factor = (red(operandf) + green(operand₃) + blue(operandf )/3

With: alpha(): a function that returns the alpha channel of a texture red(): a function that returns the red channel of a texture green(): a function that returns the green channel of a texture blue(): a function that returns the blue channel of a texture Here a factor is defined depending on the string value of the fourth operand. The result of the operator BLEND is then given by: output _Ch_anneis = operand^ * factor + operand₂ * (1 — factor) output_others = operand^

The gITF Specification makes use of common engineering and graphics terms such as image, buffer, texture, etc. to identify and describe certain gITF constructs and their attributes, states, and behaviors. The specification defines a texture as an object that combines an image and its sampler (as disclosed in the Section 5.29 “Texture” of the gITF 2.0 specification (https://registry.khronos.Org/glTF/specs/2.0/glTF-2.0.pdf [1 ])) that allows the definition of sampling and wrapping strategies.

Table 1: The gITF Texture property

The ’’source” property of Texture references the base texture image. In the present case, it corresponds to the base texture in the equation (1 ) above. Besides, in a gITF file, one can reference a texture using a Textureinfo property. It is used in several places in materials assets, like “baseColorTexture”, “normalTexture”, or “metallicRoughnessTexture”. The gITF specification defines four properties for Textureinfo: “index”, “texCoord”, “extensions” and “extras” as disclosed in Section 5.30 “Textureinfo” of the gITF 2.0 specification ([1]) which is reproduced below:

Table 2: The gITF Textureinfo property

The ’’index” property of Textureinfo references the base texture. In our case, it corresponds to the base texture in the equation (1 ) above.

According to one embodiment, a new “MPEG_texture_mix” property is defined in the “extensions” attribute of the gITF Texture and Textureinfo properties. It is defined as follows:

Table 3: The MPEG_texture_mix property

The "operators" array contains the following TextureOperator property for each operation:

Table 4 The TextureOperator property

The "channels" property defines the channels modified by the operator:

• "RGB": all color channels: Red, Green, Blue

• "RGBA": all channels: Red, Green, Blue, Alpha • "Alpha": only the Alpha channel

The "operator" property defines the operation to use. Examples of operations that can be used have been described above.

The "operandl" property defines the first operand to use (denoted operand^ in the examples of operations described above). The "operand2" property defines the second operand to use (denoted operand^ in the examples of operations described above).

The "operands" property defines the third operand to use (denoted operand^ in the examples of operations described above). It is only required when "operator" is "BLEND". The "operandl", "operand2" and "operands" properties can have the following values:

• "PREVIOUS": this value indicates that the operand is the texture base or the output of the previous operation in the list of operators.

• "TEXTURE1": this value indicates that the operand is a texture defined by the "texturel" property.

• "TEXTURE2": this value indicates that the operand is a texture defined by the "texture2" property.

The "texturel" property can be set if "operandl", "operand2" or "operands" is "TEXTURE1". It references a texture using the standard Textureinfo property type.

The "texture2" property can be set if "operandl", "operands" or "operands" is "TEXTURE2". It references a texture using the standard Textureinfo property type.

The "clamp" property is a Boolean value telling if the output should be clamped in a given range, for example [0,1]. If false, outputs outside the range have values that depend on implementation and hardware.

The "factor" property can be set if "operator" is "BLEND". The different values:

• "ALPHA": this value indicates that Operand 3 texture alpha channel is used as a pixelbased blending factor.

• "INTENSITY": this value indicates that Operand 3 texture color channels are averaged and used as a pixel-based blending factor.

The "weight" property can be set if "operator" is "INTERPOLATE". It has as many values (in [0,1]) as there are channels to process (depends on the value of the "channels" property):

• "channels" is "RGB": "weight" has three values.

• "channels" is "RGBA": "weight" has four values.

• "channels" is "Alpha": "weight" has one value.

For the sake of clarity, the example described below only shows parts of the gITF file related to materials: it is assumed that there is a scene with a mesh (or more) that uses this material. Consequently, the following only describes the material loading procedure.

It is desired to create a texture that is the result of the blending of two textures using a third one as a mask. FIG. 8 illustrates an example of the result (on the left side) of a blending that uses the three textures shown on the right side of FIG. 8. The output is the blending of a blue texture with a yellow texture using the last texture as a mask. This blending can be implemented by a mix() function in the GLSL standard specified in [2] https://registry.khronos.org/OpenGL/specs/gl/glspec46.core.pdf). The three textures used as operands are shown on FIG. 9. The yellow and mask texture are smaller than the blue texture: during the blending, they are wrapped using a mirroring (like MIRRORED REPEAT in OpenGL texture samplers [2]- or in gITF texture samplers [1]).

According to an embodiment, to obtain this texture, the following information can be encoded in using the proposed extension in the scene description file: The material loader first parses the “images” list and load the three image files "blue.jpg", “yellow.jpg” and “mask.jpg”. Then, it parses the “textures” and "samplers" lists and assigns each texture to one of the images with wrapping properties:

• texture 0 corresponds to image in "blue.jpg" file.

• texture 1 corresponds to image in "yellow.jpg" file and has a MIRRORED REPEAT wrapping policy (both axes).

• texture 2 corresponds to image in "mask.jpg" file and has a MIRRORED REPEAT wrapping policy (both axes).

During the next stage, the material loader parses the “materials” list, and finds one with a "pbrMetallicRoughness" attribute. This attribute only defines the base color texture in the “baseColorTexture” attribute. This is a Textureinfo property: the “index” attribute defines the base texture to be used. Since “index” equals 0, it corresponds to texture 0, so the image in the “blue.jpg” file.

Then, if the material loader supports the proposed “MPEG_texture_mix” extension, it parses the “MPEG_texture_mix” property in “materials/pbrMetallicRoughness/baseColorTexture/extensions”. According to the embodiments described herein , the “MPEG_texture_mix” property comprises a list of one or more operators “operators”. In the syntax example shown above, the “operators” list contains one operation with the following properties:

• It only updates the colors since "channels" is "RGB".

• The operator is a blending since "operator" is "BLEND".

• The first operand is the base texture ("blue.jpg") since "operandl" is "PREVIOUS"

• The second operand is the yellow texture since: o "operand2" is "TEXTURE 1 " o "texturel" is a Textureinfo property with an “index” property having a value 1 , so it references texture 1 ("yellow.jpg")

• The third operand is the mask texture since: o "operand2" is "TEXTURE2" o "texture2" is a Textureinfo property with an “index” property having a value 2, so it references texture 2 ("mask.jpg")

• The fourth operand is "INTENSITY" since "factor" is a string value "INTENSITY" As a result, the output of the first MPEG_texture_mix operation is : output_Alpha = alpha texture_blue) Some embodiments described herein relate to a method for signaling, encoding or decoding one or more operators in a scene description file, such as a gITF file. The one or more operators provide for combining the texture with at least one other texture. In this way, instead of signaling a texture prepared beforehand in the scene description file, the process to obtain the final texture is encoded/signaled in the scene description file. Thus, changes to the texture or process to obtain the texture can be made at any stages of the workflow. This also allows to reduce memory footprint.

FIG. 5 illustrates an example of a method 500 for encoding/signaling one or more operators in a scene description file according to an embodiment. The operators signaled in the scene description file allows to encode a process for obtaining a mix of textures. At 501 , a texture information property referencing a texture is signaled in the scene description file. If one or more modifications are to be applied to the texture, at 502, one or more operators are signaled in the scene description file, for the texture information property. For example, the one or more operators are encoded in the scene description file according to the syntax provided above with Table 3 and Table 4.

FIG. 6 illustrates an example of a method 600 for decoding one or more operators from a scene description file according to an embodiment. At 601 , the scene description file is parsed. At least one texture information property referencing a texture is read from the scene description file. At 602, the texture information property is parsed, and one or more operators are obtained for this texture information property from in the scene description file. For example, the one or more operators are parsed according to the syntax provided above with Table 3 and Table 4. At 603, the texture is then modified by applying the parsed operators. When more than one operator are signaled for the texture information property, the operators are applied in a sequential manner to the texture. The result of a current operator applied to the texture referenced by the texture information property then replaces the texture referenced by the texture information property for the next operator to apply.

In some variants, the one or more operators are part of an attribute extension of a format of the scene description file. In this way, devices that do not support the extension do not need to parse the syntax describing the process for modifying the texture.

In some variants, the one or more operators comprise at least one of the following properties a channel property indicating one or more channels modified by the operator, an operator type, one or more operands indicating one or more input textures to be used by the operator, for an input operand, a texture information property referencing a texture, a clamp property indicating if output values are clamped to a given range, a factor property indicating a given type of blending operation, a weight property indicating weighting values for the operator. In some variants, an input operand of the operator can be a result or an output of one or more operators applied to the texture referenced by the texture information property indicated by the input operand.

FIG. 7 illustrates an example of a method 700 for decoding one or more operators from a scene description file according to another embodiment. FIG. 7 described in a more detailed manner the process for applying the operators to a texture signaled in the scene description file.

At 701 , the scene description file is parsed. At least one texture information property referencing a texture is read from the scene description file. At 702, the texture information property is parsed, and one or more operators are obtained for this texture information property from in the scene description file. For example, the one or more operators are parsed according to the syntax provided above with Table 3 and Table 4. At 703, the texture referenced by the texture information property is accessed. For each parsed operator, at 704, one or more other textures referenced by the input operands of the current operator are accessed and at 705 the texture is modified by applying the current operator. At 704, when accessing the other textures, depending on how the other texture is referenced, the other texture can be a result of an operator itself. Thus, accessing the other texture can comprise modifying the other texture.

At 706, the texture modified at 705 is output. It can be stored or rendered depending on the application.

FIG. 10-20 provides further details on the embodiments described above.

The proposed extension data is parsed any time a Textureinfo property occurs in the gITF file. Most gITF parsers already offer hook mechanisms one can register. For the embodiments described herein, the main gITF parser calls the extension code every time it finds a Textureinfo property. FIG. 10 presents an overview 1000 of the parsing of a Textureinfo according to an embodiment.

The process ingests (1001 ) Textureinfo data (top-left in the diagram), and outputs (1008) a texture (bottom-right in the diagram). It first parses the data in a default fashion (1001 ), as defined without any extension. This first process initializes (1002) the main texture.

Then, if there is no “extensions” property (false at 1003), the process is over and outputs the main texture.

If there are extensions (yes at 1003), it first applies any non-MPEG_texture_mix extensions (true at 1004 and 1005). This holds for all current extensions, but might not be true for future extensions. For these later cases, it will be up to the designers to choose the extension order. If no order is defined, it is assumed that MPEG_texture_mix always comes last.

Each extension updates the main texture: its layout and values can change, but still remains a texture (e.g. a 2D array of vectors).

If the MPEG_texture_mix extension is defined in the Textureinfo data (true at 1006) , the process parses each item in the “operators” property. Each operator updates (1007) the main texture using its attributes. The main texture modified is then output (1008).

For each TextureOperator item in the "operators" list of the MPEG_texture_mix property, the value of "operator" is used to update the main texture as described with FIG. 1 1 , according to an embodiment. The current main texture and the TextureOperator are provided as input to the process (1 101 ). The string indicated with the “operator” property of the TextureOperator is checked with respect to all defined string values ( 1102, 1 104, 1 106, 1108, 1 110, 11 12) and the corresponding operation is applied to the texture provided at input (1103, 1105, 1107, 1 109, 11 11 , 11 13). At 1 115, the modified texture is output. If no defined string values corresponds to the “operator” property of the TextureOperator, the process is over (1 114) and the input texture is not modified.

FIG. 12 illustrates an example of a method 1200 for accessing a first operand of an operator when parsing a TextureOperator property in a gITF file according to an embodiment. The first operator operand is determined using the value of "operandl " in the TextureOperator property. The current main texture and the TextureOperator are provided as input to the process (1201 ). When the string value of "operandl" is "TEXTURE1" or "TEXTURE2" (true at 1204 or 1206), the corresponding property "texturel" or "texture2" is parsed and stored in Texture 1 (1205, 1207). These properties are Textureinfo properties, as defined by the gITF standard: they can also contain extension data, including MPEG_texture_mix data.

When the string value of "operandl" is ’’PREVIOUS” (true at 1202), the input texture is copied in Texture 1 .

The process 1200 outputs (1209) the texture used as the first operator operand in Texturel.

FIG. 13 illustrates an example of a method 1300 for accessing a second operand of an operator when parsing a TextureOperator property in a gITF file according to an embodiment. This method is only required for an operator that has two or more operands. The second operator operand is determined using the value of "operand2" in the TextureOperator property. The current main texture and the TextureOperator are provided as input to the process (1301 ). When the string value of "operand2" is "TEXTURE1" or "TEXTURE2" (true at 1304 or 1306), the corresponding property "texturel" or "texture2" is parsed and stored in Texture2 (1305, 1307). These properties are Textureinfo properties, as defined by the gITF standard: they can also contain extension data, including MPEG_texture_mix data.

When the string value of "operand2" is ’’PREVIOUS” (true at 1302), the input texture is copied in Texture2.

The process 1300 outputs (1309) the texture used as the second operator operand in Texture2.

FIG. 14 illustrates an example of a method 1400 for accessing a third operand of an operator when parsing a TextureOperator property in a gITF file according to an embodiment. This method is only required for an operator with three operands. The second operator operand is determined using the value of "operand2" in the TextureOperator property. The current main texture and the TextureOperator are provided as input to the process (1401 ).

When the string value of "operands" is "TEXTURE1" or "TEXTURE2" (true at 1404 or 1406), the corresponding property "texturel" or "texture2" is parsed and stored in Texture3 (1405, 1407). These properties are Textureinfo properties, as defined by the gITF standard: they can also contain extension data, including MPEG_texture_mix data.

When the string value of "operands" is ’’PREVIOUS” (true at 1402), the input texture is copied in TextureS.

The process 1400 outputs (1409) the texture used as the third operator operand in Texture3.

FIG. 15 illustrates an example of a method 1500 for applying an addition (ADD) operator to a texture according to an embodiment.

When the "operator" property of the TextureOperator is "ADD", the following procedure is run. The current main texture Texture and the TextureOperator are provided as input to the process (1501 ). At 1502 and 1503, the first two operands textures are accessed and stored respectively in Texturel and Textu re2 (FIG. 12 and 13). Each operand texture is split (1504, 1505) according to the "channels" property value: the processed channels are stored in Texturel channels and Texture2channels and the others in Texturel others and Texture2others. Texture2others is not used in the remaining of the process. At 1506, Texturel channels and Texture2channels are summed. At 1507, the result of the operation ADD at 1506 is merged with the other channels of the first operand texture and the result is stored in Texture. If the "clamp" property is true (yes at 1508), the values of Texture are clamped (1509). At 1510, the resulting Texture is output.

FIG. 16 illustrates an example of a method 1600 for applying a substract operator to a texture according to an embodiment.

When the "operator" property of the TextureOperator \s "SUBTRACT", the following procedure is run. The current main texture Texture and the TextureOperator are provided as input to the process (1601 ). The first two operand textures are accessed (1602, 1603) and stored in Texturel and Texture2 (FIG. 12 and 13). Each operand texture is split (1604, 1605) according to the "channels" property value: the processed channels are stored in Texturel channels and Texture2channels and the others in Texture 1 others and Texture2others. Texture2others \s not used in the remaining of the process. At 1606, the SUBSTRACT operation is applied by computing Texturel channels - Texture2channels. At 1607, the result of this operation is merged with the other channels of the first operand texture and stored in Texture. If the "clamp" property is true (1608), the values of Texture are clamped (1609). The modified current texture is then output (1610) in Texture.

FIG. 17 illustrates an example of a method 1700 for applying a modulate operator to a texture according to an embodiment.

When the "operator" property of the TextureOperator is " MODULATE ", the following procedure is run. The current main texture Texture and the TextureOperator are provided as input to the process (1701 ). The first two operand textures are accessed (1702, 1703) and stored in Texturel and Textu re2 (FIG. 12 and 13). Each operand texture is split (1704, 1705) according to the "channels" property value: the processed channels are stored in Texturel channels and Texture2channels and the others in Texturel others and Texture2others. Texture2others is not used in the remaining of the process. At 1706, the MODULATE operation is applied by computing Texturel channels * Texture2channels. At 1707, the result of this operation is merged with the other channels of the first operand texture and stored in Texture. If the "clamp" property is true (1708), the values of Texture are clamped (1709). The modified current texture is then output (1710) in Texture.

FIG. 18 illustrates an example of a method 1800 for applying a replace operator to a texture according to an embodiment.

When the "operator" property of the TextureOperator is "REPLACE", the following procedure is run. The current main texture Texture and the TextureOperator are provided as input to the process (1801 ). The first two operand textures are accessed (1802, 1803) and stored in Texturel and Texture2 (FIG. 12 and 13). Each operand texture is split (1804, 1805) according to the "channels" property value: the processed channels are stored in Texturel channels and Texture2channels and the others in Texture 1 others and Textu re2others. Texture2others \s not used in the remaining of the process. Since the operation is replaced, there is no computation of a new texture, but Texture 1 channels are replaced with Texture2channels. This is done in the merging step 1807 where Texture2channels is merged with the other channels of the first operand texture (Texturel others) and stored in Texture. If the "clamp" property is true (1808), the values of Texture are clamped (1809). The modified current texture is then output (1810) in Texture.

FIG. 19 illustrates an example of a method 1900 for applying an interpolate operator to a texture according to an embodiment.

When the "operator" property of the TextureOperator is "INTERPOLATE", the following procedure is run. The current main texture Texture and the TextureOperator are provided as input to the process (1901). At 1911 , the weight value from the "weight" attribute of the TextureOperator is obtained. The first two operand textures are accessed (1902, 1903) and stored in Texturel and Textu re2 (FIG. 12 and 13). Each operand texture is split (1904, 1905) according to the "channels" property value: the processed channels are stored in Texturel channels and Texture2channels and the others in Texturel others and Texture2others. Texture2others is not used in the remaining of the process.

At 1906, the INTERPOLATE operation is applied by computing Texturel channels * weight + Texture2channels * (1 - weight).

At 1907, the result of this operation is merged with the other channels of the first operand texture and stored in Texture. If the "clamp" property is true (1908), the values of Texture are clamped (1909). The modified current texture is then output (1910) in Texture.

FIG. 20 illustrates an example of a method 2000 for applying a blending operator to a texture according to an embodiment.

When the "operator" property of the TextureOperator is "BLEND", the following procedure is run. The current main texture Texture and the TextureOperator are provided as input to the process (2001 ). The three operand textures are accessed (2002, 2003, 2011 ) and stored in Texturel, Texture2 and Texture3 (FIG. 12-14). The two first operands texture are split (2004, 2005) according to the "channels" property value: the processed channels are stored in Texturel channels and Texture2channels and the others in Texturel others and Texture2others. Texture2others is not used in the remaining of the process.

The third operand texture is processed depending on the "factor" property value (2012). If "factor" is "ALPHA", then Factor is set (2013) as the alpha channel of Texture3. If "factor" is "INTENSITY" (2014), Factor is set (2015) as the average of red, green and blue channels of Texture3. When the value of "factor" is not defined, this raises an error (2016). At 2006, the BLEND operation is applied by computing Texturel channels * Factor + Texture2channels * (1 - Factor).

At 2007, the result of this operation is merged with the other channels of the first operand texture and stored in Texture. If the "clamp" property is true (2008), the values of Texture are clamped (2009). The modified current texture is then output (2010) in Texture.

According to an embodiment, a data stream that encodes a scene description is provided. For instance, the embodiment provides modifications of the syntax of a data stream illustrated on FIG. 2. Modifications are for example provided by the syntax elements shown in Table 3 and Table 4 above. In this embodiment, the data stream comprises a scene description file that includes at least one texture information property referencing a texture, and the at least one texture information property comprises at least one operator for modifying the texture. In some variants, the at least one operator is provided with syntax for signaling at least one of a channel property indicating one or more channels modified by the operator, an operator type, one or more operands indicating one or more input textures to be used by the operator, for an input operand, a texture information property referencing a texture, a clamp property indicating if output values are clamped to a given range, a factor property indicating a given type of blending operation, or a weight property indicating weighting values for the operator.

In some variants, the at least one operator is part of an attribute extension of a format of the scene description file. In other variants, at least one input operand of the at least one operator is an output of one or more operators applied to a texture signaled in the scene description file.

According to an aspect, embodiments described herein can be advantageously used in a processing workflow. When creating 3D assets, a common approach consists in splitting the procedure in several stages. Each stage focuses on a specific task, like working on the mesh modeling, the textures, the animation, the lights, etc. These stages are usually handled by different people or teams, or by the same person but at different times. To manage this, a good solution is to store the result of each stage in a new file. Using the current gITF standard, one can already follow this approach for several tasks. It is not the case for others, like texture modeling.

Using the proposed extension MPEG_texture_mix, it becomes possible to store the texture creation process in a gITF file. Then, users in the next stage can modify how textures are created, for instance through the modification, removal or addition of new texture operations. Otherwise, in the current standard, only the final textures are available in the gITF file, with no information about how they were created. Furthermore, 3D asset software supporting this extension can propose a III to modify how texture operations are encoded in the gITF file. Indeed, the operations can be seen as a tree where each node is a particular operation. Then, the user can edit this tree, changing the node properties and modifying the tree.

According to another aspect, embodiments described herein can be advantageously used to reduce memory footprint when using gITF file. This is true for the gITF file size, but also for the usage in the CPU or GPU memory.

For instance, if several textures are the result of different operations but common sources, and if the memory footprint of sources is smaller than the sum of memory footprints of final textures, then one can save memory.

Advantageously, the disclosed extension provides high compatibility with existing implementation as a particular software, that does not support the texture mix extension, may use the base texture as defined in the standard, ignoring the operators to apply to the base texture. Consequently, an incomplete but relevant rendering is still possible.

Various numeric values are used in the present application. The specific values are for example purposes and the aspects described are not limited to these specific values.

Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and/or use of specific steps and/or actions may be modified or combined. Additionally, terms such as “first”, “second”, etc. may be used in various embodiments to modify an element, component, step, operation, etc., such as, for example, a “first decoding” and a “second decoding”. Use of such terms does not imply an ordering to the modified operations unless specifically required. So, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or in an overlapping time period with the second decoding.

The implementations and aspects described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed may also be implemented in other forms (for example, an apparatus or program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, for example, computers, cell phones, portable/personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.

Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation”, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation”, as well any other variations, appearing in various places throughout this application are not necessarily all referring to the same embodiment.

Additionally, this application may refer to “determining” various pieces of information. Determining the information may include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

Further, this application may refer to “accessing” various pieces of information. Accessing the information may include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

Additionally, this application may refer to “receiving” various pieces of information. Receiving is, as with “accessing”, intended to be a broad term. Receiving the information may include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, “receiving” is typically involved, in one way or another, during operations, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

It is to be appreciated that the use of any of the following “and/or”, and “at least one of”, for example, in the cases of “A/B”, “A and/or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and/or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as is clear to one of ordinary skill in this and related arts, for as many items as are listed.

As will be evident to one of ordinary skill in the art, implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal may be formatted to carry the bitstream of a described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.

Claims

1. A method, comprising:

- signaling in a scene description file, at least one texture information property referencing a texture,

- signaling in a scene description file, for the at least one texture information property, at least one operator to apply to the texture.

2. A method, comprising:

- parsing a scene description file comprising at least one texture information property referencing a texture,

- obtaining, for the at least texture information property, at least one operator signaled in the scene description file,

- modifying the texture based on the at least one operator.

3. The method of any one of claims 1 -2, wherein the at least one operator provides for combining the texture with at least one other texture.

4. The method of any one of claims 1 -3, wherein the at least one operator comprises at least one of:

- a channel property indicating one or more channels modified by the operator,

- an operator type,

- one or more operands indicating one or more input textures to be used by the operator,

- for an input operand, a texture information property referencing a texture,

- a clamp property indicating if output values are clamped to a given range,

- a factor property indicating a given type of blending operation,

- a weight property indicating weighting values for the operator.

5. The method of any one of claims 1 -4, wherein the at least one operator is part of an attribute extension of a format of the scene description file.

6. The method of any one of claims 1 -5, wherein at least one input operand of the at least one operator is an output of one or more operators applied to a texture signaled in the scene description file.

7. The method of any one of claims 2-6, further comprising:

- accessing the texture referenced by the at least one texture information property,

- accessing one or more other textures referenced by one or more input operand of the at least one operator,

- modifying the texture by applying the at least one operator to the texture and the one or more other textures,

- outputting the modified texture.

8. An apparatus, comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to perform the method of any of claims 1 -7.

9. A non-transitory computer readable medium comprising instructions which, when the instructions are executed by a computer, cause the computer to perform the method of any of claims 1 -7.

10. A non-transitory computer readable medium comprising a scene description file comprising at least one texture information property referencing a texture, the at least one texture information property comprising at least one operator for modifying the texture.

1 1. The non-transitory computer readable medium of claim 10, wherein the at least one operator comprises at least one of:

- a channel property indicating one or more channels modified by the operator,

- an operator type,

- one or more operands indicating one or more input textures to be used by the operator,

- for an input operand, a texture information property referencing a texture,

- a clamp property indicating if output values are clamped to a given range,

- a factor property indicating a given type of blending operation,

- a weight property indicating weighting values for the operator.

12. The non-transitory computer readable medium of any one of claims 10-1 1 , wherein the at least one operator is part of an attribute extension of a format of the scene description file.

13. The non-transitory computer readable medium of any one of claims 10-12, wherein at least one input operand of the at least one operator is an output of one or more operators applied to a texture signaled in the scene description file.