JP2022172413A - Three-dimensional expression conversion device and three-dimensional expression inverse conversion device - Google Patents

Abstract

To convert three-dimensional expression of desired characteristics into three-dimensional expression capable of being processed by an assumed processing device regardless of properties of the input three-dimensional expression.SOLUTION: A three-dimensional expression conversion device (500) generates a second three-dimensional expression representing a three-dimensional object with first three-dimensional expression representing the three-dimensional object as input. The three-dimensional expression conversion device comprises an expression conversion unit (501) that converts the first three-dimensional expression into the second three-dimensional expression based on directly or indirectly specified post-conversion characteristics.SELECTED DRAWING: Figure 1

Description

One aspect of the present invention relates to a three-dimensional object represented by a three-dimensional expression represented by a three-dimensional point cloud,
It relates to a conversion device for converting from a first three-dimensional representation to a second three-dimensional representation.

By transmitting and processing digital data (3D data) representing the whole or part of a three-dimensional object, various applications such as recording and playback of three-dimensional objects and telexistence can be realized. A three-dimensional point group (point group) is one of the representation formats (three-dimensional representation) of a three-dimensional object. A point cloud represents a three-dimensional object as a set of points. Each point that constitutes the point cloud is defined by the point's position and attribute information (for example, color). In addition, representation formats for three-dimensional objects include mesh formats, volume formats, and image formats.

In general, three-dimensional representation data has a large amount of data. As such, transmission and processing in the form of a particular three-dimensional representation requires a large amount of storage and computational resources.

V-PCC (Video-based Point Cloud Compression), which is being formulated as an international standard, is a conventional technology that reduces bandwidth and computational resources during playback by expressing and transmitting point clouds, one of the three-dimensional representations, as images. method (Non-Patent Document 1). In V-PCC, the input point cloud is clustered and divided into predetermined units (point cloud clusters). Subsequently, a two-dimensional image (patch) corresponding to the point cloud cluster is generated by projecting the point cloud cluster onto one of the prescribed projection planes. Patches corresponding to all point cloud clusters are collected and packed to generate a two-dimensional image (integrated image). Finally, the integrated image is compressed by a video coding method and transmitted. In V-PCC, two types of patches are generated from one point cloud cluster. Specifically, a geometry patch representing the three-dimensional shape of the point cloud cluster and an attribute patch representing the attribute (for example, color) of the point cloud cluster are generated. Therefore, there are two types of integrated images: a geometry integrated image that collects geometry patches and an attribute integrated image that collects attribute patches. In the following description, the term "integrated image" means both a geometry integrated image and an attribute integrated image unless otherwise specified.

Euee et al., “Video-Based Point-Cloud-Compression Standard in MPEG: From Evidence Collection to Committee Draft [Standards in a Nutshell]”, IEEE Signal Processing Magazine Volume:36, Issue: 3, pp: 118-123, May 2019

However, the prior art suffers from the problem that it is not always possible to generate 3D representation data that can be processed by a processing device designed to process 3D representations of specific properties. For example, the conventional V-PCC has a problem that the resolution of the integrated image representing the point group varies depending on the properties of the point group. As a result, a situation arises in which the resolution of the integrated image becomes high, and there is a problem that transmission and processing become difficult due to lack of computational resources.

For example, in V-PCC, when a high-density point cloud is input, there is a problem that the resolution of an integrated image to be video-encoded may become too high. In general, video decoding processing is designed for images with resolutions below a certain level, and images with resolutions higher than that resolution cannot be decoded, resulting in the problem that point clouds cannot be restored.

Also, for example, in V-PCC, if the surface area of the three-dimensional object represented by the input point cloud is large,
The total area of patches resulting from clustering is increased. As a result, the resolution of the integrated image also increases, and a problem arises that the point cloud cannot be restored.

In the above, the point cloud, which is one of the three-dimensional object representation formats, was used to explain the problem. has a similar problem.

One aspect of the present invention has been made in view of the above problems, and is capable of processing output cubic data that can be processed by an assumed subsequent processing device, regardless of the type or property of the 3D representation of the input (input point group). A three-dimensional expression conversion device that generates original expressions is realized. Also, a three-dimensional expression inverse conversion device is realized that restores the original three-dimensional expression from the three-dimensional expression converted by such a three-dimensional expression conversion device.

In order to solve the above problems, a 3D expression conversion device, a 3D expression inverse conversion device, an encoding device, a decoding device, and a processing device relating to a 3D object according to one aspect of the present invention provide the following means. do.
(First means)
Three-dimensional representation conversion means for generating a second three-dimensional representation representing the three-dimensional object from a first three-dimensional representation representing the three-dimensional object,
A representation conversion means for transforming the first three-dimensional representation into the second three-dimensional representation,
The expression conversion unit provides three-dimensional expression conversion means for performing conversion based on the post-conversion characteristics that are directly or indirectly specified.

According to this three-dimensional expression conversion means, by specifying post-conversion characteristics that can be processed by an assumed processing device, the first three-dimensional expression that cannot be processed by the processing device can be processed by the processing device. It can be transformed into a second possible three-dimensional representation.
(Second means)
Three-dimensional representation inverse transformation means for generating a third three-dimensional representation representing the three-dimensional object from a second three-dimensional representation representing the three-dimensional object as input,
a representation inverse transformation means for transforming the second three-dimensional representation into the third three-dimensional representation;
An expression conversion information interpretation means for interpreting input expression conversion information,
The representation inverse transformation means provides three-dimensional representation inverse transformation means for applying a transformation to the second three-dimensional representation based on representation transformation information.

According to this three-dimensional representation inverse conversion means, a third three-dimensional representation close to the first three-dimensional representation before conversion is obtained from the second three-dimensional representation generated by the conversion by the three-dimensional representation conversion means. can be restored. In particular, by using the expression conversion information, it is possible to determine the method of inverse conversion in consideration of the content of the conversion, so that the third three-dimensional expression that is closer to the first three-dimensional expression can be restored.
(Third means)
3D object encoding means for generating 3D data corresponding to the 3D object with a 3D object as input,
processing resolution setting means for determining resolution designation;
cluster dividing means for dividing the three-dimensional object into clusters;
patch projection means for generating patches from the clusters and generating projection information related to projection;
patch conversion means for applying conversion to the patch based on the specified resolution and generating adjustment information related to the conversion;
patch integration means for generating an integrated image by combining the patches to which the transformation is applied, and generating integration information related to the integration;
patch information encoding means for generating patch information including the projection information, the adjustment information, and the integration information;
video encoding means for applying video encoding to the integrated image to generate 2D data;
A three-dimensional object encoding means is provided, comprising a 3D data multiplexing means for multiplexing the 2D data and the patch information to form 3D data.

According to this 3D object encoding means, 3D data can be generated by converting patches so as to satisfy the specified resolution, and encoding an integrated image including the converted patches. Therefore, regardless of the properties of the input three-dimensional object, 3D data including 2D data corresponding to an integrated image having a resolution that can be decoded with desired computational resources can be generated.
(Fourth means)
Three-dimensional object decoding means for reconstructing the three-dimensional object with input of 3D data generated by encoding the three-dimensional object,
a 3D data demultiplexer that extracts patch information and 2D data from the 3D data;
patch information decoding means for deriving integration information, adjustment information, and projection information from the patch information;
video decoding means for decoding a composite image from the 2D data;
patch dividing means for dividing the integrated image into a plurality of patches based on the integrated information;
patch restoration means for converting each divided patch based on the adjustment information;
patch backprojection means for generating clusters from the patches based on the projection information;
A 3D object decoding means is provided comprising a 3D object reconstruction means for combining a plurality of clusters.

According to this three-dimensional object decoding means, a three-dimensional object can be restored by decoding 3D data obtained by converting a patch so as to satisfy a specified resolution and decoding the 3D data obtained by encoding an integrated image including the converted patch. Therefore, 3D data including 2D data corresponding to an integrated image having a resolution that can be decoded with desired computational resources can be decoded regardless of the properties of the input 3D object.
(Fifth means)
Three-dimensional object processing means for transforming and outputting an input three-dimensional object,
processing resolution setting means for determining resolution designation;
image generation means for generating an integrated image and patch information based on the resolution designation and the three-dimensional object;
Integrated image processing means for applying conversion processing to the integrated image and outputting the integrated image after conversion;
A three-dimensional object processing means is provided, comprising a three-dimensional object reproduction means for generating a three-dimensional object with the converted integrated image and the patch information as inputs.

According to this three-dimensional object processing means, the three-dimensional object can be transformed by converting the patches so as to satisfy the specified resolution and processing the integrated image composed of the converted patches. Therefore, desired processing can be applied to the three-dimensional object with desired computational resources, regardless of the properties of the input three-dimensional object.

ADVANTAGE OF THE INVENTION According to one aspect of the present invention, it is possible to realize an image generating apparatus that generates an integrated image that can be processed with target computational resources regardless of the properties of an input three-dimensional object.

Further, according to one aspect of the present invention, there is provided a three-dimensional object reproducing apparatus that reproduces a three-dimensional object based on an integrated image generated so as to be processable by target computational resources, regardless of the properties of the input point cloud. realizable.

FIG. 13 is a functional block diagram of a 3D expression conversion device and a 3D expression inverse conversion device according to Embodiment 7; 1 is a functional block diagram of an image generation device according to Embodiment 1. FIG. FIG. 4 is a diagram for explaining the relationship between a three-dimensional object, patches, and an integrated image; FIG. 4 is a diagram illustrating the relationship between patches targeting different resolutions and an integrated image; FIG. 10 is a functional block diagram of a point cloud reproduction device according to Embodiment 2; FIG. 10 is a processing flow diagram of a patch restoration unit 203 according to Embodiment 2; FIG. 10 is a functional block diagram of an image generation device according to Embodiment 3; FIG. 10 is a diagram for explaining the relationship between division of a three-dimensional object and patches according to Embodiment 3; FIG. 10 is a diagram illustrating the relationship between patches targeting different resolutions and an integrated image according to the third embodiment; FIG. 10 is a functional block diagram of a point group reproducing device according to Embodiment 4; FIG. 10 is a diagram for explaining viewpoint-dependent reproduction target selection according to Embodiment 4; FIG. 10 is a functional block diagram of a point cloud encoding device and a point cloud decoding device according to Embodiment 5; FIG. 10 is a functional block diagram of a point cloud processing device according to Embodiment 6; 4 is a processing flow diagram of a patch conversion unit 103 according to Embodiment 1. FIG.

[Embodiment 1]
An embodiment of the present invention will be described below with reference to FIGS. 2 to 4 and 14. FIG.

<Configuration of image generation device 1>
The configuration of the image generation device 100 according to this embodiment will be described based on FIG. Image generation device 100
has a cluster division unit 101, a patch projection unit 102, a patch conversion unit 103, a patch integration unit 104, and a patch information encoding unit 105.

The image generation device 100 generates and outputs an integrated image representing the point group from the input point group. The output integrated image has a resolution according to the input resolution specification. In addition, the image generation device 100 generates and outputs patch information used to reproduce the point cloud from the integrated image.

The cluster division unit 101 divides an input point cloud into a plurality of point cloud clusters and outputs the clusters. The division of the point cloud into point cloud clusters is performed by applying clustering to the point cloud. Clustering is performed so that the point cloud clusters form a continuous surface, and the normals of each point on the surface have similar directions. For example, a desired point cloud cluster can be obtained by forming an initial cluster by collecting neighboring point clouds based on the spatial positions of the points, and dividing and integrating each initial cluster based on the normal line estimation result of each point. An example of the relationship between point clouds and point cloud clusters will be described with reference to FIG. FIG. 3(a) illustrates the relationship between a point cloud corresponding to a rectangular parallelepiped object and a point cloud cluster obtained by dividing the point cloud by clustering. The cuboid is divided into point cloud clusters corresponding to 6 faces according to the normal direction. That is, it is decomposed into a point cloud cluster (FT) corresponding to the top surface, a point cloud cluster (FB) corresponding to the bottom surface, and a point cloud cluster (FS1, FS2, FS3, FS4) corresponding to each side surface.

The patch projection unit 102 generates and outputs a corresponding patch from each input point cloud cluster. The patch projection unit 102 additionally outputs projection information representing the patch generation method. A patch is generated by projecting a point group cluster onto a predetermined projection plane using a predetermined projection method. For example, one projection plane is selected from a predetermined set of projection planes, and an image obtained by orthographically projecting the point group cluster onto the selected projection plane is used as a patch. As a set of specified projection planes, for example, a coordinate system (XYZ coordinate system) set in a 3D space is used, and planes parallel to the XY, YZ, and ZX planes are used as a set of specified projection planes.

FIG. 3(b) shows patches corresponding to point cloud clusters obtained by dividing the rectangular parallelepiped shown in FIG. 3(a). When orthographic projection is performed on a projection plane perpendicular to the normal direction of each point cloud cluster, patches as shown in Fig. 3(b) are obtained. If the size of the rectangular parallelepiped is 2U in height, 1U in width, and 1U in depth, the patches (FS1, FS2, FS3, FS4) obtained from the point cloud cluster corresponding to one side have a resolution of 1U×2U. Similarly, the patches (FT, FB) corresponding to the top and bottom surfaces have a resolution of 1U x 1U. Note that when the notation of image resolution is “A×B”, it represents the resolution of width A and height B. FIG. The same notation will be used in the following description as well.

The projection information includes information identifying the projection plane and projection method used to project each patch. In the case of the above example, the projection method is orthographic projection, and identification information for the projection method can be omitted. The projection plane identification information is information for selecting the XY plane, YZ plane, or ZX plane. Other information may be used as long as the projection method and projection plane can be determined. For example, projection can be rephrased as shooting with a virtual camera. Therefore, camera parameters (focal length, focal position, shooting resolution, distortion parameter, etc.) of the virtual camera can be used as projection information.

The patch conversion unit 103 converts a plurality of input patches to the resolution indicated by the input resolution specification (
(specified resolution)), and output the resulting patch. For example, some patches are selected and reduced patches are output. The patch conversion unit 103 also outputs information identifying the conversion applied to the patch as adjustment information. For example, the adjustment information is information indicating whether or not the contraction conversion described above is applied. Details of the patch conversion processing in the patch conversion unit 103 will be described later.

The patch integration unit 104 packs input patches to form an image, thereby generating and outputting an integrated image. The patch integration unit 104 additionally outputs integration information. The integrated information is information indicating the correspondence between the position of the patch in the integrated image and the position of the point cloud cluster corresponding to the patch in the 3D space. For example, the integrated information includes the upper left coordinate and size of the area on the integrated image corresponding to the patch. Also, for example, the integrated information includes the relative positions in 3D space of the point cloud clusters corresponding to the patches.

An example of an integrated image obtained by packing the patch set shown in FIG. 3(b) is shown in FIG. 3(c). One integrated image is composed of six patches (FS1, FS2, FS3, FS4, FT, FB). In this example, the resolution of the integrated image is 4U×3U, and no transformation has been applied to any of the patches contained in the integrated image.

The patch information encoding unit 105 collectively outputs the above-described projection information, adjustment information, and integration information as patch information. Each piece of information may be compressed using lossless or lossy variable length coding as needed.

<Patch conversion processing>
Patch conversion processing in patch conversion section 103 will be described with reference to FIG. In the patch conversion process, an output patch set is generated from a plurality of patches (patch sets) input in the steps S100-S105 shown in FIG.

(S100) It is determined whether an unprocessed patch is included in the patch set, and if an unprocessed patch exists (YES in S100), S101 is executed. If there is no unprocessed patch (NO in S100), execute S105.

(S101) An unprocessed patch is selected from the patch set as a target patch. Then S102 is executed. Note that the desired operation can be realized even if the target patch is selected in any order. However, it is more preferable to select patches in descending order of resolution (larger size). This is effective in terms of processing amount because the number of times of patch conversion required to keep the resolution of the integrated image within the specified resolution can be reduced.

(S102) Pack the patch set to generate an integrated image. Here, the function provided by the patch integration unit 104 can be used for packing. Then S103 is executed.

(S103) It is determined whether the resolution of the integrated image generated in S102 is within the designated resolution. If the resolution is within the designated resolution (YES in S103), S105 is executed. Otherwise (NO in S103), S104 is executed.

(S104) Reduce the target patch and update the patch set. Then execute S100. For the reduction of the target patch, for example, a reduction in which the resolution of the target patch is halved vertically and horizontally is used. Alternatively, downscaling that halves the resolution, either vertically or horizontally, may be used.

(S105) Output the patch set and end the process.

According to the above procedure, the patch conversion unit 103 converts the input patch set based on the specified resolution and outputs the patch set. Here, the output patch set is a patch set such that the resolution of the integrated image obtained by packing is within the designated resolution by reducing the patches in S103.

Note that the patch conversion process procedure is not limited to the procedure described above with reference to FIG. 14, and another procedure that satisfies the designated resolution condition for the integrated image generated from the output patch set may be used. For example, for different combinations of patches in the patch set, an integrated image is generated for each reduced size. A patch set that satisfies the above conditions can also be output by a procedure of selecting a patch set whose resolution is equal to or less than the specified resolution and has the maximum resolution among all the integrated images that are subsequently generated.

In the above example, reduction was given as an example of conversion, but other conversions may be used. For example, patch rotation, patch division and recombination can be used as transforms. In patch division and recombination, one patch is divided into a plurality of subpatches and placed in different regions on the image. When restoring a patch, the divided subpatches are recombined to restore the patch. Also, combinations of transformations, ie, reduction and rotation, reduction and split-recombination, rotation and division-recombination, and reduction-rotation-divide-recombination, may be applied as transformations.

Next, with reference to FIG. 4, examples of patch sets and integrated images determined for different resolution specifications will be described. As a premise of this example, it is assumed that the patch set for the rectangular parallelepiped target shown in FIG.

FIG. 4(a) exemplifies the relationship between the patch set and the integrated image when the designated resolution is 2U×3U. In this case, the original patch size shown in FIG. 3(b) cannot make the integrated image smaller than the designated resolution. Therefore, by reducing the width of some patches (FS1, FS2, FS3, FS4) from the original 1U to 0.5U, the resolution of the integrated image is kept within the specified resolution.

FIG. 4(b) exemplifies the relationship between the patch set and the integrated image when the specified resolution is 4U×2U. In this case, the height of patches (FS1, FS2) corresponding to some sides is reduced (from the original 2U to 1.5U). In addition, the height and width of the patches (FT, FB) corresponding to the top and bottom surfaces are reduced (from the original 1U to 0.5U). By using the converted patch set, the resolution of the integrated image is kept within the specified resolution.

FIG. 4(c) illustrates the relationship between the patch set and the integrated image when the designated resolution is 5U×1U. In this case, the patches corresponding to the sides (FS1, FS2, FS3, FS4) are rotated and then reduced. That is, the 2U×1U patch is rotated by 90 degrees and converted into a 1U×2U patch, and further reduced in half in the vertical direction to be converted into a 0.5U×2U patch. The top and bottom patches (FT, FB) are vertically halved and converted from 1U x 1U patches to 0.5U x 1U patches. By using the converted patch set, the resolution of the integrated image is kept within the specified resolution.

<Operation of image generation device 100>
By linking the constituent functional elements included in the image generating apparatus 100 described above, the image generating process can execute an image generating process of generating and outputting an integrated image and patch information from the input point group based on the specified resolution. Image generation processing in the image generation device 100 is executed in the following procedure.

(S11) The cluster division unit 101 applies clustering to the input point cloud to generate a plurality of point cloud clusters and outputs them to the patch projection unit .

(S12) The patch projection unit 102 generates a patch set by projecting each of the input point cloud clusters, and outputs the patch set to the patch conversion unit 103. FIG. The patch projection unit 102 additionally generates projection information and outputs it to the patch information encoding unit 105 .

(S 13 ) Patch conversion section 103 converts the input patch set based on the input resolution specification, and outputs the converted patch set to patch integration section 104 . Patch conversion section 103 also generates adjustment information and outputs it to patch information encoding section 105 .

(S14) The patch integration unit 104 packs the input patch set to generate an integrated image, and outputs the integrated image as an output of the image generating apparatus 100 to the outside.

(S15) The patch information encoding unit 105 generates patch information based on the input projection information, adjustment information, and integration information, and outputs the generated patch information to the outside as an output of the image generating apparatus 100. FIG.

According to the image generation device 100 described above, it is possible to generate and output an integrated image and patch information from an input point group based on a specified resolution. In the image generation device 100, the patch integration unit 104 integrates the patch sets converted by the patch conversion unit 103 so as to satisfy the constraint of the resolution designation, thereby generating an integrated image. Therefore, by using the image generating apparatus 100, it is possible to generate and output an integrated image that expresses the point cloud at a resolution that satisfies desired memory consumption and image processing constraints regardless of the properties of the input point cloud. Also, the integrated image generated by the image generating apparatus 100 can reproduce the original point cloud by using the patch information that is also generated. An example of reproduction will be described separately as the point cloud reproduction device 200. FIG.

<Additional remarks 1 regarding Embodiment 1>
In the description of the image generation device 100 in the first embodiment above, an example in which a rectangular parallelepiped point group is input with reference to FIG. 3 has been described. can also be applied. For example, after setting a rectangular parallelepiped (bounding box) that encloses the target point cloud, a point cloud that is close to the normal direction of each surface of the rectangular parallelepiped is set as a point cloud cluster, which is similar to the method described in the first embodiment. The method can perform patch set transformation and integrated image generation.

<Additional remarks 2 regarding Embodiment 1>
In the description of the image generation device 100 in the above first embodiment, the method of generating the integrated image based on the input resolution designation has been described. In the description, it is assumed that the type of integrated image, that is, whether the integrated image is a geometry integrated image or an attribute integrated image is not distinguished, and the same method can be applied to any of them. Some applications may have different resolution requirements for geometry-integrated images and attribute-integrated images. Therefore, the image generation device 1 may be configured to input different resolution designations according to the type of integrated image targeted. Such a configuration can output a combination of integrated images that meet more flexible application requirements.

[Embodiment 2]
An embodiment of the present invention will be described below with reference to FIGS. 5 and 6. FIG.

<Structure of Point Group Reproduction Device 200>
The configuration of the point cloud reproduction device 200 according to this embodiment will be described based on FIG. Point cloud playback device 200
has a patch information decoding unit 201, a patch dividing unit 202, a patch restoring unit 203, a patch backprojecting unit 204, and a point group restoring unit 205. The point cloud reproduction device 200 restores and outputs a point cloud corresponding to the integrated image based on the input integrated image and patch information. For example, the point cloud reproduction device 200
receives the integrated image and patch information output from the image generating apparatus 100 described in the first embodiment, restores the original point cloud, and outputs it.

The patch information decoding unit 201 processes input patch information, extracts integration information, adjustment information, and projection information, and outputs them. Here, the patch information, integration information, adjustment information, and projection information are the same as those described for the image generating apparatus 100, so detailed description thereof will be omitted.

The patch division unit 202 generates and outputs patch sets based on the input integrated image and the input integrated information. The integrated information includes information corresponding to the position and size of each patch on the integrated image. A set of patches extracted from the integrated image using the information is defined as a patch set.

The patch restoration unit 203 restores the patch set based on the input patch set and the input adjustment information, and outputs the restored patch set. Note that the patch restoration unit 203 is a component corresponding to the patch conversion unit 103 included in the image generation apparatus 100. FIG. That is, it has a function of inputting the patch set converted by the patch conversion unit 103 and the adjustment information and restoring the original patch set. Details of the patch set restoration process will be described later.

The patch backprojection unit 204 generates and outputs a set of point cloud clusters corresponding to the patch set based on the input patch set and the input projection information. Note that the patch backprojection unit 204
is a component corresponding to the patch projection unit 102 included in the image generating apparatus 100. FIG. That is, the point cloud cluster projected onto the patch by the patch projection unit 102 is restored by back-projecting the patch based on the projection information. For example, a case where the projection information indicates that orthographic projection is applied with the xy plane as the projection plane will be described. Assume that the depth value z0 is recorded in the pixel at the pixel position (x0, y0) on the integrated image. In this case, the three-dimensional position of the back-projected point of the pixel is (x0, y0, z0). In the case of an attribute-integrated image, information corresponding to a depth value may not be recorded in pixels. In that case, the depth value obtained by referring to the geometry-integrated image corresponding to the attribute-integrated image is used.

The point cloud restoration unit 205 restores and outputs a point cloud based on the set of input point cloud clusters. If the point cloud clusters have a common coordinate system, the output point cloud can be generated by a simple union of the set of point cloud clusters. On the other hand, when the coordinates of the point cloud clusters are different, the output point cloud can be generated by converting each point cloud cluster into a point cloud of a common coordinate system and combining them.

<Patch restoration processing>
Patch restoration processing in the patch restoration unit 203 will be described with reference to FIG. In the patch restoration process, an output patch set is generated by converting the patch set input in the steps S201 to S204 shown in FIG.

(S201) An unprocessed patch is selected from the patch set as a target patch. Then S202 is executed.

(S202) By referring to the adjustment information, it is determined whether or not the target patch is an adjustment target patch (a patch to which conversion has been applied). If it is a patch to be adjusted (YES in S202), then S203 is executed. If the batch is not an adjustment target (NO in S202), S205 is executed.

(S203) Restore the target patch based on the adjustment information. That is, the target patch is restored by applying a process corresponding to the inverse transformation of the conversion applied to the target patch indicated by the adjustment information. For example, if the adjustment information indicates that the target patch has been subjected to shrinking transformation by half vertically and horizontally, the target patch is restored by applying enlargement transformation by doubling the height and width of the target patch. Also, for example, if the adjustment information indicates that the target patch has been rotated 90 degrees clockwise, the target patch is rotated 90 degrees counterclockwise. Restore.

(S204) It is determined whether an unprocessed patch exists in the patch set, and if an unprocessed patch exists (YES in S204), S201 is executed. If there are no unprocessed patches (NO in S204), output the current patch set and terminate the process.

According to the above procedure, the patch restoration unit 203 restores the patch set by applying conversion based on the input patch set and the input adjustment information.

<Operation of Point Group Reproduction Device 200>
By linking the constituent functional elements included in the point cloud reproduction apparatus 200 described above, it is possible to execute point cloud reproduction processing for restoring and outputting a point cloud from an input integrated image and input patch information. The point cloud reproduction process in the point cloud reproduction device 200 is executed in the following procedure.

(S21) The patch information decoding unit 201 decodes the input patch information to extract integration information, adjustment information, and projection information. The integration information is output to the patch division unit 202, the adjustment information is output to the patch restoration unit 203, and the projection information is output to the patch backprojection unit 204, respectively.

(S22) The patch dividing unit 202 derives a patch set based on the input integrated image and the input integrated information, and outputs the patch set to the patch restoring unit 203. FIG.

(S23) The patch restoration unit 203 converts the patch set based on the input patch set and the input adjustment information, and outputs the converted patch set to the patch backprojection unit 204. FIG.

(S24) The patch backprojection unit 204 generates a set of point cloud clusters based on the input patch set, and outputs the set to the point cloud reconstruction unit 205. FIG.

(S25) The point cloud reconstruction unit 205 outputs the point cloud derived based on the set of input point cloud clusters to the outside as the output of the point cloud reproduction device 200. FIG.

According to the point cloud reproducing apparatus 200 described above, a point cloud can be reproduced from patch information and an integrated image generated so as to conform to desired resolution designation. In particular, the point cloud reproduction device 200 includes a patch restoration unit 203 that converts patch sets based on adjustment information included in patch information. The patch restoration unit 203 can apply inverse transformation to the patches that have been converted so as to meet the desired resolution specification in the patch conversion unit 103 provided in the image generating apparatus 100, and restore the state before the conversion. Therefore, the quality of the reproduced point cloud can be maintained higher than when the inverse transform is not applied. For example, if a point cloud cluster is derived by back-projecting a patch that has been reduced in half vertically and horizontally, the number of points included in the point cloud cluster will be a quarter of the number of points included in the original point cloud cluster. Become. On the other hand, such a reduction in the number of points does not occur in the point cloud cluster obtained by back-projecting the patch obtained by applying the expansion of the inverse transformation to the reduced patch.

[Embodiment 3]
An embodiment of the present invention will be described below with reference to FIGS. 7 to 9. FIG.

<Configuration of image generation device 300>
The configuration of the image generation device 300 according to this embodiment will be described based on FIG. Image generation device 300
includes a primary division unit 301, a cluster division unit 101, a patch projection unit 102, a patch conversion unit 103, a patch integration unit 104, and a patch information encoding unit 105. Schematically, the image generating device 300 has a configuration obtained by adding a primary dividing unit 301 to the image generating device 100. FIG. Components common to the image generating apparatus 100 are denoted by the same reference numerals, and descriptions thereof are omitted.

The image generation device 300 generates and outputs one or more integrated images representing the point group from the input point group. Each output integrated image has a resolution according to the input resolution specification.

The primary division unit 301 determines the point group division method based on the input point group and the specified resolution, and sequentially outputs the point groups divided based on the division method. The primary division unit 301 additionally outputs information on the point group division method as primary division information.

The patch information encoding unit 105 basically has functions equivalent to those of the patch information encoding 105 described in the first embodiment. This embodiment additionally provides a function of including primary division information as an input in the output patch information.

An example of point group division in the primary division unit 301 will be described with reference to FIG. FIG. 8(a) illustrates how the cuboidal point group OBJ is divided into two sub-point groups OBJa and OBJb. The primary dividing unit 301 refers to the number of points forming the point cloud OBJ and the specified resolution, and then divides the input point cloud so as to form a point cloud that becomes an integrated image with a resolution that satisfies the specified resolution. Since there is a correlation between the number of points forming the input point cloud and the resolution of the integrated image, the number of divisions can be determined based on a description set in advance. For example, based on the observation that a point cloud of 1 million points produces a 1920×1080 integrated image. In that case, if the input point cloud is 2 million points and the specified resolution is 1920×1080, it is decided to divide the input point cloud into two sub-point clouds. When the specified resolution is 1920×540, it is decided to divide the input point cloud into four sub-point clouds. Any method can be applied to the method of dividing the point group as long as the surface area does not increase. For example, examine the centroid and distribution of the point cloud, and divide the point cloud by planes passing through the centroid. In the example of FIG. 8(a), sub-point groups are derived by dividing the point group along planes parallel to the upper and lower surfaces of the rectangular parallelepiped that pass through the center of gravity of the rectangular parallelepiped.

The primary division information includes at least division number information. In the example of FIG. 8(a), the primary division information includes the division number 2 as information.

Each divided sub-point cloud is sequentially processed by the cluster division unit 101 and the patch projection unit 102, and converted into a patch set by the method already described. FIG. 8(b) illustrates patch sets generated from the sub-point clouds OBJa and OBJb divided as shown in FIG. 8(a). A patch set containing patches FS1a, FS2a, FS3a, FS4a, FT is derived from the sub-point cloud OBJa. A patch set containing patches FS1b, FS2b, FS3b, FS4b, FB is derived from the sub-point cloud OBJb.

FIG. 9(a) shows an example of an integrated image composed of patch sets corresponding to the sub-point cloud OBJa when the resolution is specified as 3U×2U. An integrated image is constructed by directly including patches FS1a, FS2a, FS3a, FS4a, and FT included in the patch set. Since the point cloud is divided by the primary division unit 301 in consideration of the resolution designation, each patch can be stored in the integrated image as it is without applying conversion such as reduction. Therefore, it is possible to suppress the decrease in accuracy of the point cloud due to the application of the transform. Similarly, FIG. 9(b) shows an example of an integrated image composed of patch sets corresponding to the sub-point cloud OBJb when the specified resolution is 3U×2U. For comparison, FIG. 9C shows an example of an integrated image composed of patch sets corresponding to the entire point cloud OBJ when the resolution is specified as 3U×2U. Patch FS1 in FIG. 9(c) corresponds to a combination of patch FS1a in FIG. 9(a) and patch FS1b in FIG. 9(b). It can be seen that the width of FS1 is reduced to half of the width of FS1a and FS1b combined.

In order to reproduce the entire point cloud, it is necessary to process all integrated images, and the total amount of processing cannot be reduced. However, since the resolution of the integrated image to be processed at one time is suppressed to the specified resolution or less, the demand for computational resources such as frame memory required for processing can be reduced.

<Operation of Image Generation Device 300>
By linking the constituent functional elements included in the image generating apparatus 300 described above, the image generating process can execute the image generating process of generating and outputting the integrated image and patch information from the input point cloud based on the specified resolution. Image generation processing in the image generation device 100 is executed in the following procedure.

(S31) The primary division unit 301 divides the input point cloud based on the input resolution designation, and outputs the resulting point clouds to the cluster division unit 101 in order. In addition, primary division information is output to patch information encoding section 105 . Then S32 is executed.

(S32) The cluster division unit 101 applies clustering to the input point cloud to generate a plurality of point cloud clusters and outputs them to the patch projection unit . Then S33 is executed.

(S33) The patch projection unit 102 generates a patch set by projecting each of the input point cloud clusters, and outputs the patch set to the patch conversion unit 103. FIG. The patch projection unit 102 additionally generates projection information and outputs it to the patch information encoding unit 105 . Then S34 is executed.

(S34) Patch conversion section 103 converts the input patch set based on the input resolution specification, and outputs the converted patch set to patch integration section 104. FIG. Patch conversion section 103 also generates adjustment information and outputs it to patch information encoding section 105 . Then S35 is executed.

(S34) The patch integration unit 104 packs the input patch set to generate an integrated image, and outputs the integrated image as an output of the image generating apparatus 100 to the outside. If the processing of all the point groups divided in S31 has been completed, proceed to S35, otherwise proceed to S32 for the unprocessed point groups.

(S35) The patch information encoding unit 105 generates patch information based on the input primary division information, projection information, adjustment information, and integration information, and outputs the patch information to the outside as an output of the image generation device 300. FIG.

According to the image generation device 300 described above, an integrated image and patch information can be generated and output from an input point cloud based on a specified resolution. The image generation device 300 is provided with the primary division unit 301, so that an integrated image that satisfies the restriction of specifying the resolution can be generated. Therefore, by using the image generation device 300, it is possible to generate and output an integrated image that expresses the point cloud at a resolution that satisfies desired memory consumption and image processing constraints regardless of the properties of the input point cloud. In addition, the integrated image generated by the image generation device 300 can reproduce the original point cloud by using the patch information that is also generated. An example of reproduction will be described separately as a point group reproduction device 400. FIG.

<Modification 1 of Embodiment 3>
Although the configuration of the image generation device 300 shown in FIG. 7 in the third embodiment includes the patch conversion unit 103, the configuration without the patch conversion unit 103 may be used. If the primary division unit 301 guarantees that the integrated image has a resolution equal to or less than the designated resolution, the object can be achieved even if the patch change unit 103 is omitted, and the processing amount and circuit scale of the image generation device can be reduced.

Note that the configuration including the patch conversion unit 103 has the advantage that the degree of freedom of division in the primary division unit 301 increases. For example, when the number of points in the input point cloud is extremely large, if only the primary dividing unit 301 is used, it will be necessary to divide it into a large number of sub-point clouds, and overhead due to division may become a problem. Even in such a case, by using the patch conversion unit 103 together, the input point group can be represented by an integrated image with a desired resolution while suppressing the number of divisions.

<Modification 2 of Embodiment 3>
In the description of the image generation device 300 in the above third embodiment, the procedure for dividing the input point group by one dividing method determined by the primary dividing unit 301 and outputting the integrated image corresponding to each divided point group has been described. The image generation device 300 is not limited to this, and may be configured to output integrated images corresponding to a plurality of division methods. For example, when the cuboid in Fig. 8 is input, two integrated images corresponding to each sub-point cloud when the cuboid is divided into two, and one integrated image corresponding to the point cloud when the cuboid is not divided configured to output In addition, the primary division information includes the number of division methods to be output and the map information of the integrated image corresponding to each division method. This makes it possible to implement a function of selecting and processing an integrated image corresponding to an appropriate division method in consideration of computational resources on the integrated image processing side.

<Modification 3 of Embodiment 3>
Although the method of dividing the input point group has been introduced in the description of the primary division unit 301 in the third embodiment, other division methods may be applied. For example, a default set of viewpoints may be defined and the object split based on both the resolution specification and the set of viewpoints. Specifically, a sub-point group is configured including points observed from the vicinity of the viewpoints included in the viewpoint set. If the resulting sub-point cloud is too large considering the requirements of specifying the resolution, the sub-point cloud is further divided to obtain the final sub-point cloud. In addition, another sub-point cloud is constructed from points that are not included in the sub-point cloud corresponding to any viewpoint. With such a configuration, when the viewpoint at the time of observation of the point group reproduced from the integrated image is in the vicinity of the viewpoint included in the viewpoint set, all the sub-point groups are not processed, and the corresponding viewpoint is processed. The amount of processing can be reduced because only sub-point clouds need to be processed.

Note that the viewpoint set may be input to the primary dividing unit 301 after the viewpoint set is additionally input to the image generation device instead of the default viewpoint set. In this case, the user of the image generation device can generate the integrated image after setting the desired viewpoint. For example, the average amount of processing can be reduced by including viewpoints with a high probability in the input viewpoint set using the statistics of the viewpoints of the user when observing the point cloud.

[Embodiment 4]
An embodiment of the present invention will be described below with reference to FIGS. 10 and 11. FIG.

<Structure of Point Group Reproduction Device 400>
The configuration of the point cloud reproduction device 400 according to this embodiment will be described based on FIG. The point cloud reconstruction device 400 includes an integrated image selection unit 401 , a patch information decoding unit 201 , a patch division unit 202 , a patch reconstruction unit 203 , a patch backprojection unit 204 and a point cloud reconstruction unit 205 . The point cloud reproduction device 400 is roughly configured by adding an integrated image selection unit 401 to the point cloud reproduction device 200 . Constituent elements common to the point group reproduction device 200 are given the same reference numerals and explanations thereof are omitted.

The point cloud reproduction device 400 restores and outputs a point cloud corresponding to the integrated image based on the input integrated image, patch information, and viewpoint information. For example, the point cloud reproduction device 400 receives as input a plurality of integrated images and patch information output from the image generation device 400 described in Modification 3 of Embodiment 3, and restores and outputs the original point cloud.

The integrated image selection unit 401 selects and outputs an integrated image to be processed from a plurality of input integrated images based on input patch information and viewpoint information. In addition, output patch information.

An example of selection in integrated image selection section 401 will be described with reference to FIG. FIG. 11 is a diagram showing the relationship between a cuboid target OBJ and viewpoints VP1, VP2, and VP3. The cuboid OBJ and its divisions OBJa and OBJb are the same as those explained in FIG. Description will be made on the assumption that three integrated images generated by the method described in Modification 3 of Embodiment 3 are input as integrated images. That is, the input integrated image includes the three integrated images shown in FIG. The number of integrated images included in the input integrated image is included in the input patch information.

As an operation example of the integrated image selection unit 401, when the viewpoint VP1 is specified as the viewpoint information, the integrated image corresponding to OBJa, that is, the integrated image shown in FIG. 9A is selected. Since the viewpoint VP1 is located above the rectangular parallelepiped, the upper side of the rectangular parallelepiped is mainly observed from the viewpoint VP1. Therefore, by reproducing the point cloud from the integrated image corresponding to OBJa corresponding to the upper half of the rectangular parallelepiped, it is possible to reproduce the point cloud sufficient for observation from the viewpoint VP1 while suppressing the amount of restoration processing. Similarly, when the viewpoint VP2 is specified as the viewpoint information, the integrated image corresponding to OBJb, that is, the integrated image shown in FIG. 9B is selected. In addition, when the viewpoint VP3 is specified as the viewpoint information, the integrated image corresponding to the entire OBJ, that is, Fig. 9
(c) Select the merged image. Since the entire rectangular parallelepiped OBJ can be observed from the viewpoint VP3, an integrated image representing the point group of the entire rectangular parallelepiped is preferable to an integrated image representing a part of the rectangular parallelepiped. Regardless of which integrated image is selected, the resolution is suppressed to a certain level or less, so processing is possible even with limited computational resources.

In other words, the integrated image selection unit 401 selects an integrated image based on the criteria of including a large number of point groups observable from the viewpoint indicated by the viewpoint information and a small number of point groups not observable from the viewpoint.

<Operation of Point Group Reproduction Device 400>
By linking the constituent functional elements included in the point cloud reproduction device 400 described above, it is possible to execute point cloud reproduction processing for restoring and outputting a point cloud from an integrated image selected based on input viewpoint information and input patch information. . The point cloud reproduction process in the point cloud reproduction device 400 is executed in the following procedure.

(S41) The integrated image selection unit 401 outputs to the patch division unit 202 the integrated image selected based on the input patch information and the input viewpoint information. In addition, patch information is output to patch information decoding section 201 .

(S42) The patch information decoding unit 201 decodes the input patch information to extract integration information, adjustment information, and projection information. The integration information is output to the patch division unit 202, the adjustment information is output to the patch restoration unit 203, and the projection information is output to the patch backprojection unit 204, respectively.

(S43) The patch dividing unit 202 derives a patch set based on the input integrated image and the input integrated information, and outputs the patch set to the patch restoring unit 203. FIG.

(S44) The patch restoration unit 203 converts the patch set based on the input patch set and the input adjustment information, and outputs the converted patch set to the patch backprojection unit 204. FIG.

(S45) The patch backprojection unit 204 generates a set of point cloud clusters based on the input patch set, and outputs the set to the point cloud reconstruction unit 205. FIG.

(S46) The point cloud reconstruction unit 205 outputs the point cloud derived based on the set of input point cloud clusters to the outside as the output of the point cloud reconstruction unit 400. FIG.

According to the point cloud reproduction device 400 described above, a point cloud can be reproduced from the integrated image and patch information generated so as to meet the desired resolution specification. In particular, the point cloud reproduction device 400 includes an integrated image selection unit 401, which selects an integrated image according to viewpoint information. It can restore the quality of the point cloud observed from .

[Embodiment 5]
An embodiment of the present invention will be described below with reference to FIG.

<Configuration of point group encoding device 1000>
The configuration of the point group encoding device 1000 according to this embodiment will be described based on FIG. 12(a). The point cloud encoding device 1000 includes an image generation unit 100, a video encoding unit 1001, a 3D data multiplexing unit 1002, and a processing resolution setting unit 1003. The point cloud encoding device 1000 compresses and encodes the input point cloud to generate and output 3D data.

The image generator 100 is a functional block having the same functions as the image generator 100 described in the first embodiment.

A video encoding unit 1001 receives time-series images as input, compresses and encodes them, and generates and outputs corresponding 2D data. As the video encoding unit 1001, a popular video encoding method can be used. For example, AVC (Advanced Video Coding) and HEVC (High Efficiency Video Coding) schemes are applicable. Note that the output of the image generation unit 100 is an integrated image, but even an integrated image can be processed as a normal image.

A 3D data multiplexing unit 1002 outputs 3D data generated by multiplexing at least 2D data and patch information.

The processing resolution setting unit 1003 determines and outputs the processing resolution based on the processing performance assumed for the video decoding unit provided in the point cloud decoding device. For example, in many video coding schemes, profiles and levels can describe the processing performance required for video decoding. For example, level 4.1 in HEVC indicates that video decoding can process data equivalent to an image of about 1920×1080. Therefore, when generating 3D data intended to be processed by a video decoding unit compatible with HEVC level 4.1, the processing resolution setting unit 1003 sets the processing resolution to 1920×1080 and outputs it. The processing performance of the video decoding unit may be received by some communication means, and the processing resolution may be set based on the received processing performance.

<Operation of Point Group Encoding Device 1000>
By linking the constituent functional elements included in the point cloud encoding device 1000 described above, it is possible to generate and output 3D data obtained by compressing and encoding the input point cloud. Point cloud reproduction processing in the point cloud encoding device 1000 is executed in the following procedure.

(S51) The processing resolution setting unit 1003 determines the processing resolution based on the processing performance of the device that decodes the 3D data, and outputs the processing resolution to the image generation unit 100 as a specified resolution.

(S52) The image generation unit 100 generates an integrated image and patch information based on the input point cloud and the specified resolution. The integrated image is output to the video encoding unit 1001, and the patch information is output to the 3D data multiplexing unit 1002.

(S53) The video encoding unit 1001 outputs 2D data generated by compressing and encoding the input integrated image to the 3D data multiplexing device 1002.

(S54) The 3D data multiplexing unit 1002 outputs the 3D data generated by multiplexing the input 2D data and patch information to the point group encoding device 1000. FIG.

According to the point cloud encoding device 1000 described above, regardless of the number and properties of points in the input 3D data, 3D data targeted for a point cloud decoding device equipped with a video decoding unit assuming predetermined computational resources. can generate In particular, since the point cloud encoding device 1000 includes the image generation unit 100, it designates a resolution that can be processed by the computational resources of the video decoding unit, generates an integrated image from 3D data, and encodes the integrated image. 3D data can be constructed by including 2D data that has been processed.

<Configuration of Point Group Decoding Device 2000>
The configuration of the point group decoding device 2000 according to this embodiment will be described based on FIG. 12(b). The point cloud decoding device 2000 includes a point cloud reproducing unit 200, a video decoding unit 2001, and a 3D data demultiplexing unit 2002. The point cloud decoding device 2000 receives the 3D data generated by the point cloud encoding device 1000, restores the original point cloud, and outputs it.

The point cloud reproduction unit 200 is a functional block having the same functions as the point cloud reproduction device 200 described in the second embodiment.

A video decoding unit 2001 restores and outputs an original image from 2D data that has been compression-encoded by a specific method. As a compression encoding method, the same method as that of the video encoding unit 1001 is used. For example, use AVC or HEVC. Note that the same method used to generate the 2D data should be used.

The 3D data demultiplexing unit 2002 demultiplexes the input 3D data, divides it into 2D data and patch information, and outputs them.

<Operation of Point Group Decoding Device 2000>
By linking the constituent functional elements included in the point cloud decoding device 2000 described above, the original point cloud can be restored from the 3D data obtained by compression-encoding the input point cloud and output. The point cloud reproduction process in the point cloud decoding device 2000 is executed in the following procedure.

(S61) The processing resolution setting unit 1003 determines the processing resolution based on the processing performance of the device that decodes the 3D data, and outputs the processing resolution to the image generation unit 100 as a specified resolution.

(S62) The image generator 100 generates an integrated image and patch information based on the input point cloud and the specified resolution. The integrated image is output to the video encoding unit 1001, and the patch information is output to the 3D data multiplexing unit.

(S63) The video encoding unit 1001 outputs 2D data generated by compressing and encoding the input integrated image to the 3D data multiplexing device 1002.

(S64) The 3D data multiplexing unit 1002 outputs the 3D data generated by multiplexing the input 2D data and the patch information to the point group encoding device 1000.

According to the point cloud decoding device 2000 described above, the original point cloud is reproduced by decoding the 3D data generated assuming predetermined computational resources, regardless of the number and properties of points in the input 3D data. can. In particular, since the point cloud decoding device 2000 includes the point cloud reproducing unit 200, it is possible to reproduce the original point cloud from the integrated image generated under the restriction of the designated resolution.

[Embodiment 6]
An embodiment of the present invention will be described below with reference to FIG.

<Configuration of point cloud processing device 3000>
The configuration of a point cloud processing device 3000 according to this embodiment will be described based on FIG. The point cloud processing device 3000 includes an image generation unit 100, a point cloud reproduction unit 200, an integrated image processing unit 3001, and a processing resolution setting unit 3002. The point cloud processing device 3000 corrects the input point cloud by applying predetermined image processing, and outputs the corrected point cloud.

The image generator 100 is a functional block having the same functions as the image generator 100 described in the first embodiment.

The point cloud reproduction unit 200 is a functional block having the same functions as the point cloud reproduction device 200 described in the second embodiment.

The integrated image processing unit 3001 applies two-dimensional image processing of processing to an input image and outputs an image of the application result. A noise removal filter, a smoothing filter, a high frequency enhancement filter, a color adjustment filter, etc. can be applied as the two-dimensional image processing. Also, image conversion processing using a neural network can be applied as the two-dimensional image processing. As an example of such image conversion processing, there is a style transfer that reflects a learned image style.

A processing resolution setting unit 3002 determines and outputs a processing resolution based on the processing performance expected in the integrated image processing unit 3001 .

<Operation of Point Cloud Processing Device 3000>
By linking the constituent functional elements included in the point cloud processing apparatus 3000 described above, it is possible to generate and output a point cloud obtained by applying specific effect processing to an input point cloud. The point cloud conversion process in the point cloud processing device 3000 is executed in the following procedure.

(S61) The processing resolution setting unit 3002 determines the processing resolution based on the performance requirements of the image processing applied in the integrated image processing unit 3001, and outputs the processing resolution to the image generation unit 100 as the specified resolution.

(S62) The image generator 100 generates an integrated image and patch information based on the input point cloud and the specified resolution. The integrated image is output to the integrated image processing unit 3001, and the patch information is output to the point cloud reproducing unit 200.
output to

(S63) The integrated image processing unit 3001 applies predetermined two-dimensional image processing to the input integrated image, and outputs the converted image to the point cloud reproducing unit 200 as an integrated image.

(S64) The point cloud reproduction unit 200 reproduces the point cloud based on the input integrated image and patch information, and outputs the point cloud of the reproduction result to the point cloud processing device 3000. FIG.

According to the point cloud processing apparatus 3000 described above, regardless of the number and properties of points in an input point cloud, it is possible to convert the point cloud using image processing that can be executed with predetermined computational resources. For example, by applying a noise removal filter in the integrated image processing unit 3001, the input point group can be converted into a point group by removing the point group. In particular, since the point cloud processing device 3000 includes the image generation unit 100 and the point cloud reproduction unit 200, it generates an integrated image to be subjected to two-dimensional image processing under the constraint of the designated resolution, and converts the integrated image into You can restore the point cloud from

<Modification 1 of Embodiment 6>
In the point cloud processing device 3000 described in the sixth embodiment, the integrated image processing unit 3001 may select and apply a plurality of types of image processing. Generally, different types of image processing require different computational resources and different image resolutions that can be processed. Therefore, in such a configuration, the processing resolution setting unit 3002 is configured to set different resolutions as designated resolutions according to the type of image processing applied by the integrated image processing unit 3001 and input the specified resolutions to the image generation unit 100 . do. With the above configuration, the point cloud processing device 3000 can switch and execute conversion of various effects on the point cloud according to the application.

<Modification 2 of Embodiment 6>
In the point cloud processing device 3000 described in Embodiment 6, the resolution of an image that can be processed by the integrated image processing unit 3001 depends not only on the image processing method but also on calculation resources such as memory and CPU.
Depending on the implementation, the point cloud processing device 3000 may share computational resources such as memory and CPU with other devices. In such a case, it is preferable that the processing resolution setting unit 3002 acquires the usage status of computational resources such as memory and CPU, and sets the designated resolution according to the usage status. That is, a lower resolution is set when the utilization rate of computational resources is high, and a higher resolution is set when the utilization rate of computational resources is low. With the above configuration, the point cloud processing device 3000 can apply transformations to the point cloud even when available computational resources change.

<Appendix 1 common to Embodiments 1 to 6>
In each of the descriptions of Embodiments 1 to 6, a configuration in which a point group is used as input or output has been described. However, the present invention is also applicable to representation forms of three-dimensional objects other than point clouds. For example, it can also be applied to a format (mesh format) in which a three-dimensional object is represented by meshes and textures. In that case, for example, the cluster division unit 101 outputs clusters expressed in a mesh format instead of point cloud clusters. Also, the function of the patch projection unit 102 may be changed from a function of projecting a point group to derive a patch to a function of projecting a mesh format cluster to derive a patch. Further, for example, in the point cloud reproduction device 200, the function of the patch backprojection unit 204 is changed from the function of deriving the point cloud from the patch to the function of back projecting the patch and deriving the cluster in the mesh format. In addition, the point group restoration unit 205 is a mesh restoration unit that outputs clusters in mesh format. Point cloud clusters and mesh-type clusters are collectively referred to simply as clusters. Also, the point group restoration section and the mesh restoration section are collectively referred to as a three-dimensional object restoration section.

As another representation format, for example, the present invention can be applied to a case where a three-dimensional object is represented in a volume representation format. In volume representation, space is divided into grids, and three-dimensional objects are expressed by describing information indicating the inside and outside of the object, distance to the surface, color, and other attribute information for the voxels obtained by the division. By regarding a voxel as a point whose position is the voxel center, the method described in each embodiment can be applied.

From the above facts, it can be said that the image generation device 100 and the image generation device 300 generate integrated images based on three-dimensional objects expressed in various formats. Also, the point cloud reproduction device 200 and the point cloud reproduction device 400 are three-dimensional object reproduction devices, the point cloud encoding device 1000 is a three-dimensional object encoding device, and the point cloud decoding device 2000 is a three-dimensional object decoding device and a point group processing device. 3000 can also be called a three-dimensional object processing device.

[Embodiment 7]
In the descriptions of the first to sixth embodiments, the apparatus and processing that handle integrated images that can be processed at a specified resolution have been mainly described. However, handling specified resolutions and integrated images is not an essential element in the present invention, and a generalized device that encompasses all embodiments can be defined. As such devices, the 3D expression conversion device 500 and the 3D expression inverse conversion device shown in FIG. 1 will be described below.

FIG. 1(a) is a block diagram showing a three-dimensional representation conversion device 500 implemented based on the present invention. The three-dimensional representation conversion device 500 includes a representation conversion section 501 and a representation conversion information generation section 502 . The 3D conversion device 500 generates and outputs an output 3D representation and representation conversion information based on the input 3D representation and conversion instruction information.

<Definition of 3D expression>
Here, three-dimensional expression is a general term for various forms of expression that express three-dimensional objects. Specifically, the three-dimensional representation can include at least the following representation formats.

(Format 1) Point cloud format: A format that expresses a three-dimensional object as a set of points in a three-dimensional space. It is further classified into a dense point cloud format that restricts the positions of points on a grid and a sparse point cloud format that places points at arbitrary positions in space.

(Format 2) Mesh format: A format in which a three-dimensional object is represented by a mesh, which is a set of faces in a three-dimensional space. Each face that constitutes a mesh is defined by vertex point positions and vertex connection information.

(Format 3) Volume format: A format that expresses a three-dimensional object as a set of voxels obtained by dividing a three-dimensional space into a grid. It is further classified into a voxel occupancy expression format that uses attribute information indicating whether or not a voxel is in a space within a 3D object, and a TSDF (Truncated Signed Distance) format that uses attribute information to indicate the distance between a voxel and a 3D object surface.

(Format 4) Image format: A format in which a three-dimensional object is converted into one or more images and expressed.
The transformation to image is typically using projection based on a given camera or viewpoint, but other transformations can also be used. The integrated image in Embodiments 1 to 6 can be said to be one of three-dimensional representations in image format.

(Format 5) Compressed format: A compressed format of each of the above formats. The 3D data described in the fifth embodiment is an expression obtained by compressing the image format, and is an example of the compression format.

(Form 6) Encrypted form: Encrypted form of each of the above forms.

(Form 7) Composite form: Expression by combining each of the above forms. For example, the above classification includes expressions by combining different forms. An example of this is when the entire three-dimensional object is expressed in a mesh format and a part thereof is expressed in a point cloud format. In addition, expressions by combining the same forms in the above classification are included. For example, dividing a three-dimensional object as described in the third embodiment into a plurality of parts and representing each part in a point cloud format is an example of this.

<Expression conversion unit 501>
The expression conversion unit 501 converts the three-dimensional expression based on the input three-dimensional expression (first three-dimensional expression) and the designated post-conversion characteristics, and converts the three-dimensional expression after conversion (second three-dimensional expression). to output Although post-conversion characteristics are input to the three-dimensional representation conversion device 500, it is also possible to use predetermined conversion characteristics or derive them inside the three-dimensional representation conversion device 500. FIG. Also, the representation conversion information generation unit 502 is not an essential component. However, by generating expression conversion information through the expression conversion information generation unit 502, accurate restoration can be realized when restoring the three-dimensional expression before conversion from the three-dimensional expression after conversion.

Conversions applied in the expression conversion unit 501 include various conversions shown below.

Data volume reduction of 3D representation: Data volume is reduced by spatial or temporal resampling and specific types of data discarding. For example, when the three-dimensional representation is in point cloud format, the data amount is reduced by resampling at a sparser interval than the original data in the spatial dimension. Also, for example, when the three-dimensional representation is in image format, the data amount is reduced by reducing the entire image or a partial area of the image. Also, if the 3D representation is in point cloud format, mesh format, volume format, or image format, reduce the amount of data by reducing the types and precision of attribute information associated with points, vertices, voxels, or pixels. do.

Segmentation of 3D representation: Segmentation in space or time direction. For example, if the three-dimensional representation is in point cloud format, one point cloud is divided into multiple point clouds by dividing the three-dimensional space into multiple spaces. Also, for example, when the three-dimensional representation is in image format, one image is divided into a plurality of images.

3D Representation Format Conversion: Converting from one 3D representation format to another 3D representation format. For example, as described in the first embodiment, the point cloud format is converted into an image format using projection. Also, for example, an encoding process is applied to the three-dimensional expression in the image format to convert it into a compressed format.

<Definition and example of post-conversion characteristics>
Post-conversion properties are information specifying properties of the output three-dimensional representation. As one example, the post-conversion property is computational resource designation information. The computational resource specification information is at least information that explicitly or indirectly indicates various computational resources required for processing the three-dimensional representation after conversion. Computational resources include quantities that directly or indirectly represent the amount of data in the three-dimensional representation. For example, if the three-dimensional representation is in the form of an image, the resolution of the image, the total number of pixels that make up the image, and the type and amount of information associated with each pixel of the image. Also, for example, when the three-dimensional representation is a point cloud format, the number of points, the density of the points, the volume occupied by the point cloud, the volume of the bounding box containing the point cloud, the type of information associated with each point, quantity. Also, for example, when the three-dimensional representation is in the form of a mesh, the number of vertices and faces forming the mesh, the resolution of the texture, and the type and amount of information associated with each vertex can be mentioned. Also, for example, if the three-dimensional representation is in volume format, the number of valid voxels, voxel spacing, volume occupied by the volume, and the type and amount of information associated with each voxel can be mentioned. Further, for example, when the three-dimensional expression is in compressed format or encrypted format, the total amount of data, and the amount of data in units of time and frames can be mentioned.

Computational resources also include information that directly or indirectly indicates the amount of computation and memory required to process the three-dimensional representation. For example, if the three-dimensional representation is in compressed or encrypted form, the algorithm types and algorithm parameters to be applied.

As another example, the post-conversion characteristics are information (expected use information) that directly or indirectly indicates the intended use of the three-dimensional information. The assumed application information includes, for example, observation viewpoint position, observation distance, display resolution, and display frame rate.

<Expression Conversion Information Generation Unit 502>
Expression conversion information generation section 502 outputs information about the conversion applied in expression conversion section 501 as expression conversion information. When the conversion is reduction, the expression conversion information includes the method of reduction, the degree of reduction, the scope of application of reduction, and the like. In addition, when the conversion is division, the expression conversion information includes information indicating the method of division, the number of divisions, and the correspondence between before and after division. In addition, when the conversion is format conversion, the expression conversion information includes information indicating formats before and after conversion.

<Configuration of 3D expression inverse conversion device 600>
FIG. 1(b) is a block diagram showing a 3D representation inverse transform apparatus 600 implemented according to the present invention. 3D expression inverse conversion device 600 includes expression inverse conversion section 601 and expression conversion information interpretation section 602 . The three-dimensional inverse transformation device 600 uses the three-dimensional representation generated by the three-dimensional transformation device 600 as an input three-dimensional representation, and also refers to the input representation transformation information to generate and output an output three-dimensional representation. .

Expression conversion information interpretation section 602 interprets and outputs input expression conversion information. A configuration that does not use expression conversion information as input is also possible. Using representation conversion information as input has the advantage of being able to reproduce the three-dimensional representation more accurately. If the expression conversion information is not used as input, the expression conversion information interpreting unit 602 becomes unnecessary, and may be omitted.

The representation inverse transformation unit 601 restores the input 3D representation (second 3D representation) based on the representation transformation information to generate and output the output 3D representation (third 3D representation). Schematically, the representation inverse transform unit 601 applies a process that corresponds to the inverse transform of the transform applied in the representation transform unit 600 . For example, if the transformation is data reduction, then the inverse transformation is restoration by estimation of the reduced data volume. If point cloud or image downsampling is applied for data reduction, point cloud upsampling can be applied for the inverse transform. Also, for example, if the transform is a split, then the inverse transform can apply a combination of the split three-dimensional representations.

<Correspondence between Embodiment 7 and Embodiment 1>
The image generation device 100 of Embodiment 1 described with reference to FIG. 2 corresponds to a specific example of the three-dimensional representation conversion device 500. FIG. The correspondence will be described below. The first 3D representation of the input is the point cloud and the second 3D representation of the output corresponds to the integrated image. Resolution specification corresponds to post-conversion characteristics, and patch information corresponds to expression conversion information. A combination of the cluster division unit 101, the patch projection unit 102, the patch conversion unit 103, and the patch integration unit 104 corresponds to the expression conversion unit 501. FIG.
The image generating device 100 converts the point cloud format into an image format (patch set) by conversion, and further reduces and integrates some of the patches to generate an image format (integrated image) that is a three-dimensional expression format for output. . The expression conversion information conversion section 502 corresponds to the patch information encoding section 105 .

<Correspondence between Embodiment 7 and Embodiment 2>
The point group reproduction device 200 of the second embodiment described with reference to FIG. 5 corresponds to a specific example of the three-dimensional expression inverse conversion device 600. The correspondence will be described below. A second 3D representation of the input corresponds to the integrated image and a third 3D representation of the output corresponds to the point cloud. Patch information corresponds to expression conversion information. A combination of the patch dividing unit 202, the patch restoring unit 203, the patch back projecting unit 204, the patch back projecting unit 204, and the point group restoring unit 205 corresponds to the expression inverse transforming unit 602. In the point cloud reproduction device 200, the image format (integrated image) is converted into an image format (patch set) by inverse transformation, and a part of the patches are enlarged and the patches are back projected to obtain a three-dimensional representation of the point cloud format. generate a shape. The patch information decoding unit 201 corresponds to the representation conversion information interpretation unit 602 .

<Correspondence between Embodiment 7 and Embodiment 3>
The image generation device 300 of the third embodiment described with reference to FIG. 7 corresponds to a specific example of the 3D representation conversion device 500. FIG. The correspondence will be described below. The first 3D representation of the input is the point cloud and the second 3D representation of the output corresponds to the integrated image. Resolution specification corresponds to post-conversion characteristics, and patch information corresponds to expression conversion information. A combination of the primary division unit 301, the cluster division unit 101, the patch projection unit 102, the patch conversion unit 103, and the patch integration unit 104 corresponds to the expression conversion unit 501. FIG. In the conversion process, the image generation device 300 first divides the three-dimensional expression in the point cloud format into a plurality of pieces, then converts each point cloud into an image format (patch set), and further reduces and integrates some of the patches. to generate an image format (integrated image) that is a three-dimensional representation format of the output. The expression conversion information conversion section 502 corresponds to the patch information encoding section 105 .

<Correspondence between Embodiment 7 and Embodiment 4>
The point cloud reproduction device 400 of the fourth embodiment described with reference to FIG. 10 corresponds to a specific example of the 3D expression inverse conversion device 600. The correspondence will be described below. A second 3D representation of the input corresponds to the integrated image and a third 3D representation of the output corresponds to the point cloud. Patch information corresponds to expression conversion information. A combination of the patch dividing unit 202, the patch restoring unit 203, the patch back projecting unit 204, the patch back projecting unit 204, and the point group restoring unit 205 corresponds to the expression inverse transforming unit 602. The point cloud reproduction device 400 selects a 3D representation suitable for the specified viewpoint information from among a plurality of 3D representations generated by transforming one 3D representation, and converts it into an image format (integrated image ) to an image format (patch set), further enlarge some patches, and back project the patches to generate a point cloud format 3D representation suitable for the specified viewpoint purification method. The patch information decoding unit 201 corresponds to the representation conversion information interpretation unit 602 .

<Correspondence between Embodiment 7 and Embodiment 5>
The image encoding device 1000 of Embodiment 5 described with reference to FIG. 12(a) corresponds to a specific example of the three-dimensional representation conversion device 500. The correspondence will be described below. A first three-dimensional representation of the input is a point cloud and a second three-dimensional representation of the output corresponds to 3D data. The post-conversion characteristics correspond to the resolution specification derived by the processing resolution setting unit 1003, and the expression conversion information corresponds to the patch information included in the 3D data. The image generation unit 100 and the video encoding unit 1001 correspond to the representation conversion unit 501 . In the image encoding device 500, in the process of conversion, the point cloud format three-dimensional representation is first converted into an image format (integrated image), and the integrated image is further compressed to generate a compressed three-dimensional representation.

The point cloud decoding device 2000 of Embodiment 5 described with reference to FIG. The correspondence will be described below. A second three-dimensional representation of the input corresponds to the 3D data and a third three-dimensional representation of the output corresponds to the point cloud. Patch information corresponds to expression conversion information. A combination of the video decoding unit 2001 and the point group reproducing unit 200 corresponds to the expression inverse transforming unit 602 . The point cloud decoding device 600 converts the compressed three-dimensional expression into an image format (integrated image), and converts the integrated image into a point cloud.

<Correspondence between Embodiment 7 and Embodiment 6>
The image processing device 3000 of the sixth embodiment described with reference to FIG. 13 corresponds to a specific example of the 3D expression conversion device 500. The correspondence will be described below. The first 3D representation of the input is a point cloud, and the second 3D representation of the output also corresponds to the point cloud. The representation conversion unit 501 corresponds to the image generation unit 100, integrated image processing unit 3001, and point group reproduction unit 200. FIG. In the point cloud processing device 3000, in the process of conversion, the three-dimensional expression in the point cloud format is first converted into an image format (integrated image). The integrated image is then processed and transformed, and finally the integrated image is inverse transformed to produce a 3D representation in point cloud format.

<Effects of 3D Expression Conversion Device 500>
The three-dimensional representation conversion device 500 described above transforms the first three-dimensional representation based on the designated post-transformation characteristics, and outputs the second three-dimensional representation obtained by the conversion. The output second three-dimensional representation satisfies the specified post-transformation properties. Therefore, even if the input first three-dimensional representation cannot be processed by a device designed to process a three-dimensional representation of a given property, the three-dimensional representation conversion device 500 can be used to correspond to that property. By specifying the post-conversion characteristics to be converted into the second three-dimensional representation, processing becomes possible.

The three-dimensional expression inverse conversion device 600 described above converts the second three-dimensional expression obtained by the conversion by the three-dimensional expression conversion device 500 into a third three-dimensional expression having a property close to the first three-dimensional expression before conversion. A three-dimensional representation can be reconstructed. In addition, by providing the expression conversion information interpretation unit 602, there is an effect that the third three-dimensional expression can be restored more accurately.

[Example of realization by software]
The device group disclosed in this specification (image generation device 100, point cloud reproduction device 200, image generation device 300,
The control blocks of the point cloud reproduction device 400, the point cloud encoding device 1000, the point cloud decoding device 2000, the point cloud processing device 3000, the three-dimensional expression conversion device 500, and the three-dimensional expression inverse conversion device 600) are integrated circuits ( It may be implemented by a logic circuit (hardware) formed in an IC chip or the like, or may be implemented by software.

In the latter case, the devices disclosed herein comprise a computer that executes program instructions, which are software that implements each function. This computer includes, for example, at least one processor (control device) and at least one computer-readable recording medium storing the program. In the computer, the processor reads the program from the recording medium and executes it, thereby achieving the object of the present invention. As the above processor, for example, a CPU (Central
Processing Unit) can be used. As the recording medium, a "non-temporary tangible medium" such as a ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. In addition, a RAM (Random Access Memory) for developing the above program may be further provided. Also, the program may be supplied to the computer via any transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. Note that one aspect of the present invention can also be implemented in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

The device group disclosed in the present invention may be realized by a computer. In this case, the computer is operated as each unit (software element) included in each device included in the device group disclosed in the present invention, thereby A control program for each of the devices described above and a computer-readable recording medium recording it are also included in the scope of the present invention.

The present invention is not limited to the above-described embodiments, but can be modified in various ways within the scope of the claims, and can be obtained by appropriately combining technical means disclosed in different embodiments. is also included in the technical scope of the present invention. Furthermore, new technical features can be formed by combining the technical means disclosed in each embodiment.

100, 300 image generation device, image generation unit 101 cluster division unit 102 patch projection unit 103 patch conversion unit 104 patch integration unit 105 patch information encoding unit 200, 400 point group reproduction device, point group reproduction unit 201 patch information decoding unit 202 Patch division unit 203 Patch restoration unit 204 Patch back projection unit 205 Point group restoration unit 301 Primary division unit 401 Integrated image selection unit 500 3D expression conversion unit 501 Expression conversion unit 502 Expression conversion information generation unit 600 3D expression inverse conversion unit 601 Expression inverse transformation unit 602 Expression transformation information interpretation unit 1000 Point group encoding device 1001 Video encoding unit 1002 3D data multiplexing unit 2000 Point group decoding device 2001 Video decoding unit 2002 3D data demultiplexing units 2003, 3002 Processing resolution setting unit 3000 point cloud processing device 3001 integrated image processing unit

Claims (18)

A three-dimensional representation conversion device that takes as input a first three-dimensional representation representing a three-dimensional object and generates a second three-dimensional representation representing the three-dimensional object,
A representation conversion unit that transforms the first three-dimensional representation into the second three-dimensional representation,
The three-dimensional expression conversion device, wherein the expression conversion unit performs conversion based on post-conversion characteristics that are directly or indirectly designated. 3. The cubic according to claim 1, wherein the conversion in the representation conversion unit is conversion including any of data amount reduction of three-dimensional information, division of three-dimensional information, and format conversion of three-dimensional information. An original representation converter. 3. The three-dimensional expression conversion apparatus according to claim 1, wherein the post-conversion characteristics include at least one of calculation resource specification information and assumed use information. 4. The three-dimensional expression conversion apparatus according to claim 1, further comprising an expression conversion information generating section for generating and outputting expression conversion information relating to the conversion applied by said expression conversion section. The first three-dimensional representation is either point cloud format, mesh format, or volume format,
The second three-dimensional representation is in image format,
The post-conversion characteristics are resolution designation, which is a form of computational resource designation information,
The expression conversion unit is
a cluster dividing unit that divides the first three-dimensional representation into clusters;
a patch projection unit that generates patches from the clusters and generates projection information related to projection;
a patch conversion unit that applies conversion to the patch based on the specified resolution and generates adjustment information related to the conversion;
A patch integration unit that combines the patches to which the transformation is applied to generate an integrated image, and generates integration information related to the integration,
5. The three-dimensional expression conversion apparatus according to claim 4, wherein the expression conversion information generation unit is a patch information encoding unit that generates patch information including projection information, adjustment information, and integration information. . 6. The three-dimensional representation conversion apparatus according to claim 5, wherein said first three-dimensional representation is a point group and said cluster is a point group cluster. 6. The three-dimensional representation conversion apparatus according to claim 5, further comprising a primary division unit that divides the point group into sub-point groups based on the specified resolution. The primary division unit divides the point group into sub-point groups by a plurality of division methods,
8. The image generating apparatus according to claim 7, wherein a plurality of integrated images corresponding to said plurality of division methods are generated. The post-conversion characteristics include viewpoint information, which is a form of assumed usage information,
9. The image generating apparatus according to claim 8, wherein the primary division unit divides the point group into sub-point groups based on the viewpoint included in the viewpoint information. the integrated image includes a geometry integrated image and an attribute integrated image;
10. The image generation apparatus according to claim 5, wherein said resolution specification can specify different resolutions for each of the geometry integrated image and the attribute integrated image. 5. The transformation in the patch transformation unit is any one of patch reduction, patch rotation, patch division and recombination, and a combination of the transformations. Item 11. The image generation device according to Item 10. A three-dimensional representation inverse transformation device for generating a third three-dimensional representation representing the three-dimensional object from a second three-dimensional representation representing the three-dimensional object as input,
a representation inverse transformation unit that transforms the second three-dimensional representation into the third three-dimensional representation;
an expression conversion information interpretation unit that interprets input expression conversion information;
The three-dimensional expression conversion device, wherein the expression inverse conversion unit applies conversion to the second three-dimensional expression based on the expression conversion information. the second three-dimensional representation is an integrated image in the form of an image;
The third three-dimensional representation is either a point cloud format, a mesh format, or a volume format,
The representation conversion information is patch information,
a patch information decoding unit that derives integration information, adjustment information, and projection information from the patch information;
a patch dividing unit that divides the integrated image into a plurality of patches based on the integrated information;
a patch restoration unit that converts each divided patch based on the adjustment information;
a patch backprojection unit that generates clusters from the patches based on the projection information;
13. The 3D inverse transformation device according to claim 12, further comprising a 3D object reconstruction unit that combines a plurality of clusters. Takes viewpoint information as an additional input,
14. The three-dimensional inverse transform according to claim 13, further comprising an integrated image selection unit that selects and outputs a part of the plurality of integrated images as a target integrated image based on the patch information and the viewpoint information. Device. A three-dimensional object encoding device that receives a three-dimensional object as input and generates 3D data corresponding to the three-dimensional object,
a processing resolution setting unit that determines resolution designation;
a cluster dividing unit that divides the three-dimensional object into clusters;
a patch projection unit that generates patches from the clusters and generates projection information related to projection;
a patch conversion unit that applies conversion to the patch based on the specified resolution and generates adjustment information related to the conversion;
a patch integration unit that combines the patches to which the transformation is applied to generate an integrated image, and generates integration information related to the integration;
a patch information encoding unit that generates patch information including the projection information, the adjustment information, and the integration information;
a video encoding unit that applies video encoding to the integrated image to generate 2D data;
A three-dimensional target encoding apparatus, comprising: a 3D data multiplexing unit that multiplexes the 2D data and the patch information to form 3D data. A three-dimensional object decoding device for reconstructing the three-dimensional object with input of 3D data generated by encoding the three-dimensional object,
a 3D data demultiplexer that extracts patch information and 2D data from the 3D data;
a patch information decoding unit that derives integration information, adjustment information, and projection information from the patch information;
a video decoding unit that decodes an integrated image from the 2D data;
a patch dividing unit that divides the integrated image into a plurality of patches based on the integrated information;
a patch restoration unit that converts each divided patch based on the adjustment information;
a patch backprojection unit that generates clusters from the patches based on the projection information;
A three-dimensional object decoding device comprising a three-dimensional object reconstruction unit that combines a plurality of clusters. A three-dimensional object processing device that converts and outputs an input three-dimensional object,
a processing resolution setting unit that determines resolution designation;
an image generation unit that generates an integrated image and patch information based on the resolution designation and the three-dimensional object;
an integrated image processing unit that applies conversion processing to the integrated image and outputs an integrated image after conversion;
A three-dimensional object processing apparatus, comprising: a three-dimensional object reproduction unit for generating a three-dimensional object by inputting the integrated image after conversion and the patch information. The integrated image processing unit converts the integrated image by applying conversion processing selected from a plurality of conversion processing,
18. The three-dimensional object processing apparatus according to claim 17, wherein the processing resolution setting section designates different resolutions according to the type of conversion processing applied in the integrated image processing section.

Images