TW201933284A - Image processing device and method - Google Patents

Abstract

The present disclosure relates to an image processing device and method which make it possible to suppress a reduction in quality resulting from a two-dimensional projection of 3D data. Data for every position included in 3D data representing a three-dimensional structure are projected onto two-dimensional planes in a plurality of layers. Further, data for every position in 3D data projected onto two-dimensional planes in a number of layers indicated by layer number information are projected into a three-dimensional space. The present disclosure is applicable, for example, to information processing devices, image processing devices, electronic equipment, information processing methods and programs.

Description

Image processing device and method

The present disclosure relates to an image processing apparatus and method, and more particularly to an image processing apparatus and method that can suppress a decrease in quality due to two-dimensional projection of 3D data.

Previously, as a coding method for representing a 3D structure such as a point cloud, such as a point cloud, the position and color information of the point cloud are respectively projected to a 2-dimensional plane, and 2 A method of encoding a coding method for a video image (hereinafter also referred to as a video-based approach) has been proposed (for example, refer to Non-Patent Document 1 to Non-Patent Document 3).
[Previous Technical Literature]
[Non-patent literature]

[Non-Patent Document 1] Tim Golla and Reinhard Klein, "Real-time Point Cloud Compression", IEEE, 2015
[Non-Patent Document 2] K. Mammou, "Video-based and Hierarchical Approaches Point Cloud Compression", MPEG m41649, Oct. 2017
[Non-Patent Document 3] "PCC Test Model Category 2 v0", N17248 MPEG output document, October 2017

[Problems to be solved by the invention]

However, the point cloud of the encoding object may have a Point outside the surface of the object due to the characteristics of the noise or camera system. Therefore, sometimes it is difficult to project to a 2-dimensional plane, and the encoding accompanying the projection of the 2-dimensional plane may result in a decrease in quality.

The present disclosure has been developed in view of such a situation, and aims to suppress a decrease in quality due to two-dimensional projection of 3D data.

[Means to solve the problem]

An image processing device according to one aspect of the present invention is an image processing device comprising: a two-dimensional projection unit that projects data of each position included in a 3D structure of a three-dimensional structure onto a plurality of layers 2D plane.

An image processing method according to one aspect of the present technology is an image processing apparatus that projects data of each position included in 3D data of a three-dimensional structure onto a two-dimensional plane of a plurality of layers.

The image processing device according to another aspect of the present technology is an image processing device comprising: a three-dimensional projection unit that is a 3D image that has been projected onto a two-dimensional plane of the number of layers indicated by the layer number information. All the information of each position is projected into the 3-dimensional space.

The image processing method on the other side of the technology is an image processing method, which is data of all the positions of the 3D data on the two-dimensional plane that has been projected onto the number of layers indicated by the layer number information. , projected to 3D space.

In the image processing apparatus and method of one aspect of the present technology, data representing each of the positions included in the 3D data of the three-dimensional structure is projected onto the two-dimensional plane of the complex layer.

In the image processing apparatus and method of the other aspect of the present technology, data of each position of the 3D data that has been projected onto the two-dimensional plane of the number of layers indicated by the layer number information is projected to 3 dimensional space.

[Effect of the invention]

According to the disclosure, information can be processed. In particular, it is possible to suppress a decrease in quality due to 2-dimensional projection of 3D data.

Hereinafter, a mode for carrying out the disclosure (hereinafter referred to as an embodiment) will be described. Further, the description is made in the following order.
Video based method
2. First embodiment (variable layer number)
3. Second embodiment (definition of empty points)
4. Third embodiment (variable depth parameter)
5. Attachment

<1. Video Basics>
<Documents supporting technical content and technical terms, etc.>
The scope of the present invention is not limited to the contents described in the examples, but also includes the contents described in the following non-patent documents which are known at the time of application.

Non-Patent Document 1: (above)
Non-Patent Document 2: (above)
Non-Patent Document 3: (above)
Non-Patent Document 4: TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (International Telecommunication Union), "Advanced video coding for generic audiovisual services", H.264, 04/2017
Non-Patent Document 5: TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (International Telecommunication Union), "High efficiency video coding", H.265, 12/2016
Non-Patent Document 6: Jianle Chen, Elena Alshina, Gary J. Sullivan, Jens-Rainer, Jill Boyce, "Algorithm Description of Joint Exploration Test Model 4", JVET-G1001_v1, Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 7th Meeting: Torino, IT, 13-21 July 2017

That is, the content described in the above non-patent document is also a basis for judging the support requirement. For example, the Quad-Tree Block Structure described in Non-Patent Document 5 and the QTBT (Quad Tree Plus Binary Tree) Block Structure described in Non-Patent Document 6 are still in the present technology even if they are not directly described in the examples. Within the scope of disclosure, and deemed to meet the support requirements of the scope of patent application. Further, for example, the technical terms such as Parsing, Syntax, and Semantics are also the same, and even if they are not directly described in the examples, they are still within the scope of the present technology. Supporting requirements for the scope of patent application.

<point cloud>
Previously, a point cloud of a three-dimensional structure was represented by position information or attribute information of a point group, or a vertex, an edge, and a surface were used, and a polygon representation was used to define a three-dimensional shape of the mesh, etc. presence.

For example, in the case of a point cloud, the three-dimensional structure shown in A of FIG. 1 is represented by a set of points (point groups) of a plurality of points as shown in FIG. 1B. That is, the data of the point cloud is composed of position information or attribute information (for example, color, etc.) of each point of the point group. Therefore, the data structure will be relatively simple, and at the same time, by using enough points, the arbitrary three-dimensional structure can be expressed with sufficient precision.

<Overview of video basic methods>
A video-based approach in which each position of the point cloud and color information, each small field is projected onto a two-dimensional plane, and encoded by a two-dimensional image coding method has been proposed.

In the video basic mode, for example, as shown in FIG. 2, the point cloud that has been input is divided into a plurality of segments (also called fields or plaques), and each field is projected to 2 dimensions. flat. In addition, the data of each location of the point cloud (that is, the data of each point) is composed of position information (Geometry (also known as Depth)) and attribute information (Texture) as described above, and is projected by each field. To the 2-dimensional plane.

Then, the 3D data (point cloud) that has been projected onto the 2-dimensional plane is encoded by a two-dimensional plane image such as AVC (Advanced Video Coding) or HEVC (High Efficiency Video Coding). coding.

<There is a technology related to the basic method of video>
The present technology is described with respect to the basic method of video as explained above. Fig. 3 is a list of the present technology described in each embodiment.

From the top of the table, the first paragraph (excluding the section of the project name) describes the basic method of video in the previous (TMC2). That is, in the previous video basic mode, the 2-dimensional projection of the 3D data is performed on the 2-dimensional plane of the 2 layers (2 layers). This specification is common to all screens (the same projection is performed regardless of which section is used). Therefore, the projection control information used for the control of the projection is transmitted as information of the frame unit from the encoding side to the decoding side.

However, the point cloud of the encoding object may have a Point outside the surface of the object due to the characteristics of the noise or camera system. Therefore, as in the previous method described above, it may be difficult to project a two-dimensional plane of two layers. Therefore, a point that cannot be projected onto a two-dimensional plane occurs, which may result in degradation of data quality due to coding accompanying projection onto a two-dimensional plane.

Further, for example, in the case where the compressed object has the property of Sparse, a projection of a minute unit is required, and in order to encode many small fields, the processing amount and coding efficiency may change. difference.

The table of Fig. 3 is described in the second paragraph (excluding the segment of the item name) from the top, and the present technology described in the first embodiment (the first embodiment) is described. In this method, the number of layers of the two-dimensional plane of the projected 3D data is set to be variable so that the data of all the points (the data of each position) overlapping in the depth direction can be projected. The method of setting the number of layers.

In this case, for example, information indicating the number of layers may be transmitted from the encoding side to the decoding side in each field.

By designing in this way, it is possible to more accurately reproduce the point cloud having a thickness on the surface of the object. That is, it is possible to suppress a decrease in quality due to 2-dimensional projection of 3D data.

The table of Fig. 3 is described in the third paragraph (excluding the segment of the item name) from the top, and the present technology described in the second embodiment (the second embodiment) is described. This method is a method of adding a definition of "empty point" when projecting 3D data to a two-dimensional plane.

In this case, for example, the definition of the pixel value of the point to be deleted on the decoding side may be transmitted from the encoding side to the decoding side.

By designing this way, Sparse's point cloud can be reproduced more correctly. That is, it is not necessary to project a minute unit, and it is possible to suppress an increase in the amount of processing or a decrease in coding efficiency.

The table of Fig. 3 is described in the fourth paragraph (excluding the segment of the item name) from the top, and the present technology described in the third embodiment (Example 3) is described. This method is a method in which the depth parameter of the range of the depth of the 3D data projected onto the two-dimensional plane can be set for each field.

In this case, for example, its depth parameter can also be transmitted from the encoding side to the decoding side.

With this design, the quality of each field can be controlled, and the efficiency of positional information (Geometry) can be suppressed.

<encoding device>
Next, the configuration of each of the above methods will be realized. Fig. 4 is a block diagram showing an example of a configuration of an encoding device to which one aspect of the image processing device of the present technology is applied. The coding apparatus 100 shown in FIG. 4 is a device that projects 3D data such as a point cloud onto a two-dimensional plane and encodes it by a coding method for two-dimensional video.

For example, the encoding device 100 implements the techniques described in Non-Patent Document 1 or Non-Patent Document 6, and encodes 3D data in a method conforming to the specifications described in any of these documents.

In addition, in FIG. 4, the flow of a processing part or a material is mainly shown, and it is not shown in FIG. That is, the encoding device 100 may have a processing unit that is not represented by a block in FIG. 4, and may have a flow of processing or data that is not indicated by an arrow or the like in FIG. This is also the same in other figures of the description of the processing unit and the like in the encoding device 100.

As shown in FIG. 4, the encoding apparatus 100 includes a plaque decomposition unit 111, a packing unit 112, an auxiliary patch information compressing unit 113, a video encoding unit 114, a video encoding unit 115, an OMap encoding unit 116, and a multiplexer 117.

The plaque decomposition unit 111 performs correlation processing for decomposition of 3D data. For example, the plaque decomposition unit 111 acquires 3D data (for example, a point cloud) indicating a three-dimensional structure, which is input to the encoding device 100, and acquires it (arrow 121). Further, the plaque decomposition unit 111 decomposes the acquired 3D data into a plurality of plaques, and the plaque projects the 3D data onto the two-dimensional plane.

The plaque disassembling unit 111 supplies the 3D material projected onto the two-dimensional plane for each plaque to the packing unit 112 (arrow 122). Further, the plaque decomposition unit 111 supplies the information related to the decomposition, that is, the auxiliary plaque information, to the auxiliary plaque information compressing unit 113 (arrow 123).

The packing unit 112 performs related processing of packing the data. For example, the packing unit 112 acquires data of a two-dimensional plane on which 3D data is projected from each plaque supplied from the plaque decomposition unit 111 (arrow 122). Further, the packing unit 112 packs the acquired layers of the two-dimensional plane into mutually different video frames. For example, the packing unit 112 is a position information (Gepmetry) indicating the position of the point, attribute information (Texture) such as color information attached to the position information, and an Occupancy Map indicating the presence or absence of the point. Packed separately into a video frame.

The packing unit 112 supplies the generated video frame to the processing unit (arrow 124) in the subsequent stage. For example, the packing unit 112 supplies the video frame of the generated position information (Geometry) to the video encoding unit 114. Further, for example, the packing unit 112 supplies the video frame of the generated attribute information (Texture) to the video encoding unit 115. Further, for example, the packing unit 112 supplies the generated video frame of the occupied map to the OMap encoding unit 116.

Further, the packing unit 112 supplies the packaged control information to the multiplexer 117 (arrow 125).

The auxiliary patch information compressing unit 113 performs correlation processing for assisting compression of the patch information. For example, the auxiliary patch information compressing unit 113 acquires the data supplied from the plaque decomposition unit 111 (arrow 123). The auxiliary patch information compressing unit 113 encodes (compresses) the auxiliary patch information included in the acquired data. The auxiliary patch information compressing unit 113 supplies the encoded data of the obtained auxiliary patch information to the multiplexer 117 (arrow 126).

The video encoding unit 114 performs a process of encoding the video frame of the position information (Geometry). For example, the video encoding unit 114 acquires the video frame of the position information (Geometry) supplied from the packing unit 112 (arrow 124). Further, the video encoding unit 114 encodes the video frame of the acquired position information (Geometry), for example, by an encoding method for any two-dimensional video such as AVC or HEVC. The video encoding unit 114 supplies the encoded encoded data (coded data of the video frame of the position information (Geometry)) to the multiplexer 117 (arrow 127).

The video encoding unit 115 performs processing related to encoding of the video frame of the attribute information (Texture). For example, the video encoding unit 115 acquires the video frame of the attribute information (Texture) supplied from the packing unit 112 (arrow 124). Further, the video encoding unit 115 encodes the acquired video frame of the attribute information (Texture), for example, by an encoding method for any two-dimensional video such as AVC or HEVC. The video encoding unit 115 supplies the encoded encoded data (encoded data of the video frame of the attribute information) to the multiplexer 117 (arrow 128).

The OMap encoding unit 116 performs a process of encoding the video frame occupying the map. For example, the OMap encoding unit 116 acquires the video frame of the occupied map supplied from the packing unit 112 (arrow 124). Further, the OMap encoding unit 116 encodes the acquired video frame of the occupied map, for example, by an encoding method for any two-dimensional video such as AVC or HEVC. The OMap encoding unit 116 supplies the coded encoded data (encoded data of the video frame occupying the map) to the multiplexer 117 (arrow 129).

The multiplexer 117 is a related process of multiplexing. For example, the multiplexer 117 acquires the encoded data of the auxiliary patch information supplied from the auxiliary patch information compressing unit 113 (arrow 126). Further, for example, the multiplexer 117 acquires control information related to the package supplied from the packing unit 112 (arrow 125). Further, for example, the multiplexer 117 acquires the coded data of the video frame of the position information (Geometry) supplied from the video encoding unit 114 (arrow 127). Further, for example, the multiplexer 117 acquires the coded data of the video frame of the attribute information (Texture) supplied from the video encoding unit 115 (arrow 128). Further, for example, the multiplexer 117 acquires the coded data of the video frame of the occupied map supplied from the OMap encoding unit 116 (arrow 129).

The multiplexer 117 multiplexes the obtained information to generate a bit stream. The multiplexer 117 streams the generated bit stream to the outside of the encoding device 100 (arrow 130).

<plaque decomposition department>
FIG. 5 is a block diagram of a main configuration example of the plaque decomposition unit 111. As shown in FIG. 5, the plaque decomposition unit 111 at this time includes a normal direction estimation unit 151, a segment initial setting unit 152, a segment update unit 153, a two-dimensional projection unit 154, and a pixel distribution analysis unit 155.

The normal direction estimating unit 151 is a correlation process for estimating the normal direction of the surface of the 3D material. For example, the normal direction estimating unit 151 acquires the input 3D material. Further, the normal direction estimating unit 151 estimates the normal direction of the surface of the object presented by the acquired 3D material. For example, the normal direction estimating unit 151 constructs a kd-tree, searches for a nearby area, calculates an optimum approximation plane, and estimates a normal direction. The normal direction estimating unit 151 supplies the estimated result of the normal direction together with other data to the segment initial setting unit 152.

The segment initial setting unit 152 performs correlation processing for initial setting of the segment. For example, the segment initial setting unit 152 acquires the data supplied from the normal direction estimating unit 151. In addition, for example, the segment initial setting unit 152 sets the surface corresponding to the normal direction of the 3D material based on the components of the six directions in the normal direction estimated by the normal direction estimating unit 151. , to classify. The segment initial setting unit 152 supplies the classification result to the segment update unit 153 together with other materials.

The segment update unit 153 performs correlation processing of segment update. For example, the segment update unit 153 acquires the data supplied from the segment initial setting unit 152. Then, the segment update unit 153 is a field that is too small among the segments of the initial settings set by the segment initial setting unit 152, so that it becomes a sufficiently large field. The segment update unit 153 supplies the related information of the updated segment to the two-dimensional projection unit 154 together with other information.

The two-dimensional projection unit 154 performs correlation processing of two-dimensional projection of 3D data. For example, the two-dimensional projection unit 154 acquires the data supplied from the segment update unit 153. Further, the two-dimensional projection unit 154 uses the pixel distribution analysis unit 155 to generate an occupied map of the 3D data included in the acquired data, or project the 3D data or the occupied data to each of the fields to the 2D. Plane and so on. The two-dimensional projection unit 154 supplies the 3D data that has been projected onto the two-dimensional plane to the packing unit 112 together with other materials.

The pixel distribution analysis unit 155 performs correlation processing for analyzing the pixel distribution of the 3D material to be processed by the two-dimensional projection unit 154.

<decoding device>
Fig. 6 is a block diagram showing an example of a configuration of a decoding device to which one aspect of the image processing device of the present technology is applied. The decoding device 200 shown in FIG. 6 is a coded data obtained by projecting 3D data such as a point cloud onto a two-dimensional plane, and decoding it by a decoding method for two-dimensional images, and projecting it to 3 A device for dimensional space.

For example, the decoding device 200 implements the techniques described in Non-Patent Document 1 or Non-Patent Document 6, and decodes the encoded data of the 3D data in a method conforming to the specifications described in any of these documents.

In addition, in FIG. 6, the flow of a processing part or a material is mainly shown, and it is not shown in FIG. That is, in the decoding device 200, there may be a processing unit that is not represented by a block in FIG. 6, and there may be a flow of processing or data that is not indicated by an arrow or the like in FIG. This is also the same in other figures of the description of the processing unit and the like in the decoding device 200.

As shown in FIG. 6, the decoding apparatus 200 includes a demultiplexer 211, an auxiliary patch information decoding unit 212, a video decoding unit 213, a video decoding unit 214, an OMap decoding unit 215, an unpacking unit 216, and a 3D reconstruction. Part 217.

The multiplexer 211 performs correlation processing of inverse multiplexing of data. For example, the demultiplexer 211 is obtained by acquiring a bit stream input to the decoding device 200. This bit stream is supplied, for example, by the encoding device 100. The demultiplexer 211 performs inverse multiplexing on the bit stream, extracts the coded data of the auxiliary patch information, and supplies it to the auxiliary patch information decoding unit 212. Further, the demultiplexer 211 extracts the encoded data of the video frame of the position information (Geometory) from the bit stream by inverse multiplexing, and supplies it to the video decoding unit 213. Further, the demultiplexer 211 extracts the encoded data of the video frame of the attribute information from the bit stream by inverse multiplexing, and supplies it to the video decoding unit 214. Further, the demultiplexer 211 extracts the encoded data of the video frame occupying the map from the bit stream by inverse multiplexing, and supplies it to the OMap decoding unit 215.

The auxiliary patch information decoding unit 212 performs correlation processing for decoding the encoded data of the auxiliary patch information. For example, the auxiliary patch information decoding unit 212 acquires the encoded data of the auxiliary patch information supplied from the demultiplexer 211. Further, the auxiliary patch information decoding unit 212 decodes the encoded data of the auxiliary patch information included in the acquired data. The auxiliary patch information decoding unit 212 supplies the decoded auxiliary patch information to the 3D reconstruction unit 217.

The video decoding unit 213 performs a process of decoding the encoded data of the video frame of the position information (Geometory). For example, the video decoding unit 213 acquires the encoded data of the video frame of the location information (Geometory) supplied from the demultiplexer 211. Further, for example, the video decoding unit 213 decodes the acquired encoded data to obtain a video frame of the location information (Geometory). The video decoding unit 213 supplies the video frame of the position information (Geometory) to the unpacking unit 216.

The video decoding unit 214 performs a process of decoding the encoded data of the video frame of the attribute information (Texture). For example, the video decoding unit 214 acquires the encoded data of the video frame of the attribute information (Texture) supplied from the demultiplexer 211. Further, for example, the video decoding unit 214 decodes the acquired encoded data to obtain a video frame of the attribute information (Texture). The video decoding unit 214 supplies the video frame of the attribute information to the unpacking unit 216.

The OMap decoding unit 215 performs a process of decoding the encoded data of the video frame occupying the map. For example, the OMap decoding unit 215 acquires the encoded data of the video frame of the occupied map supplied from the demultiplexer 211. Further, for example, the OMap decoding unit 215 decodes the acquired encoded data to obtain a video frame occupying the map. The OMap decoding unit 215 supplies the video frame of the occupied map to the unpacking unit 216.

The unpacking unit 216 performs related processing of unpacking. For example, the unpacking unit 216 acquires the video frame of the location information (Geometory) from the video decoding unit 213, acquires the video frame of the attribute information (Texture) from the video decoding unit 214, and acquires the video of the occupied map from the OMap decoding unit 215. Frame. The unpacking unit 216 unpacks these video frames. The unpacking unit 216 supplies the data of the location information (Geometory) obtained by the unpacking, the information of the attribute information (Texture), and the data of the occupied map to the 3D reconstruction unit 217.

The 3D reconstruction unit 217 performs processing related to reconstruction of 3D data. For example, the 3D reconstruction unit 217 is based on the auxiliary patch information supplied from the auxiliary patch information decoding unit 212 or the location information (Geometory) supplied from the unpacking unit 216, and attribute information (Texture). The data and the information on the map are used to reconstruct the 3D data. The 3D reconstruction unit 217 outputs the 3D data thus processed to the outside of the decoding device 200.

The 3D data is supplied to the display unit, for example, and the video is displayed, recorded on a recording medium, or supplied to another device via communication.

<3D Reconstruction Department>
Fig. 7 is a block diagram showing a main configuration example of the 3D reconstruction unit 217 of Fig. 6. As shown in FIG. 7, the 3D reconstruction unit 217 includes a three-dimensional projection unit 251, a pixel distribution analysis unit 252, an inverse segment update unit 253, an inverse segment initial setting unit 254, and an inverse normal direction estimation unit 255.

The three-dimensional projection unit 251 performs projection of 3D data projected onto a two-dimensional plane into a three-dimensional space in each field. The pixel distribution analysis unit 252 performs processing such as analysis of pixel distribution when the projection of the three-dimensional space by the three-dimensional projection unit 251 is performed.

The inverse segment update unit 253 performs inverse processing of the segment update unit 153. The reverse segment initial setting unit 254 performs the inverse processing of the segment initial setting unit 152. The inverse normal direction estimating unit 255 performs the inverse processing of the normal direction estimating unit 151.

<2. First embodiment>
<The number of layers of the 2-dimensional plane of the projected 3D data is variable>
In the previous method, the 3D data, as shown in the example of FIG. 8, is a 2-dimensional plane projected onto two layers (layer 0 and layer 1).

In layer 0 (Layer 0), the data of the point of the 3D data viewed from the projection surface as the surface is projected. In Layer 1 (Layer 1), data from the point 0 at the threshold (Default Th = 4Voxel) and the farthest distance from the layer 0 is projected. If it is far away from the threshold value Th, it is discarded. Further, in the layer 1, the difference value of the distance from the layer 0 is a pixel value.

Because the algorithm of the surface of the object is represented by only two fixed layers, the information of other points is lost (cannot be reproduced). Therefore, a point that cannot be projected onto a two-dimensional plane occurs, which may result in degradation of data quality due to coding accompanying projection onto a two-dimensional plane.

Therefore, the number of layers of the two-dimensional plane of the projected 3D data can be set to be variable. For example, data representing each of the positions included in the 3D material of the 3-dimensional structure is projected onto a 2-dimensional plane of the complex layer. For example, the image processing apparatus includes a two-dimensional projection unit that projects data of each of the positions included in the 3D data of the three-dimensional structure onto a two-dimensional plane of the complex layer.

For example, in the case of FIG. 9, a two-dimensional plane of layer 0 (Layer 0) or even layer 3 (Layer 3) (a two-dimensional plane having more layers than two layers) is set, and 3D data is projected on these layers. By designing in this way, it is possible to more accurately reproduce the point cloud having a thickness on the surface of the object. That is, it is possible to suppress a decrease in quality due to 2-dimensional projection of 3D data.

For example, the two-dimensional projection unit 154 may project data of each position in which the position of the 3D material is superimposed in the depth direction from the projection surface, and project to a mutually different layer of the two-dimensional plane of the plurality of layers.

In the case of the example of Fig. 9, the data superimposed in the depth direction is projected onto the layers different from each other in the layer 0 or the layer 3. With such a design, the position of the 3D material at each position in the depth direction as viewed from the projection surface can be projected onto the 2-dimensional plane. That is, the loss of information can be suppressed. Therefore, it is possible to suppress a decrease in quality due to 2-dimensional projection of 3D data.

Further, for example, the two-dimensional projection unit 154 can generate, for the two-dimensional plane, the same number of layers as the maximum number of pieces of data of each position where the position of the 3D material is superimposed in the depth direction as viewed from the projection surface. .

In the case of the example of Fig. 9, in the 3D data of the Local Bounding Box, the maximum number of data superimposed in the depth direction is 4. Therefore, the 3D data is projected onto a 2-dimensional plane of 4 layers (layer 0 or layer 3).

With this design, the data of each position of the 3D data can be projected onto the 2-dimensional plane. Therefore, the loss of information can be suppressed, and the quality degradation caused by the two-dimensional projection of the 3D data can be suppressed.

Further, in this case, information indicating the number of layers of the two-dimensional plane on which the 3D data is projected by the two-dimensional projection unit 154 may be transmitted in the bit stream. In other words, the multiplexer 117 functioning as the bit stream generating unit generates information including the number of layers indicating the two-dimensional plane on which the 3D data is projected by the two-dimensional projection unit 154, and the video encoding unit. A bit stream of encoded data obtained by encoding a 2-dimensional plane, such as 114.

With such a design, the 3D data of all the layers projected onto the two-dimensional plane can be easily projected onto the three-dimensional space by referring to the information indicating the number of layers of the two-dimensional plane on the decoding side.

<Process of Encoding Process>
An example of the flow of the encoding process executed by the encoding device 100 will be described with reference to the flowchart of FIG.

Once the encoding process is started, the plaque decomposition unit 111 of the encoding device 100 decomposes the 3D data into plaques and projects the data of each plaque onto the two-dimensional plane in step S101. In step S102, the auxiliary patch information compressing unit 113 compresses the auxiliary patch information obtained by the processing of step S101.

In step S103, the packing unit 112 packs the 3D material projected onto the two-dimensional plane by the plaque decomposition unit 111, and packs it into a video frame. In step S104, the video encoding unit 114 encodes the video frame of the position information obtained by the processing of step S103, that is, the geometric video frame, by the encoding method for the two-dimensional video.

In step S105, the video encoding unit 114 encodes the video frame of the attribute information obtained by the processing of step S103, that is, the color video frame, by the encoding method for the two-dimensional image. In step S106, the video encoding unit 114 encodes the occupied map obtained by the processing of step S103 by the encoding method for the two-dimensional video.

In step S107, the multiplexer 117 multiplexes various pieces of information generated as described above to generate a bit stream containing the information.

In step S108, the multiplexer 117 outputs the bit stream generated by the processing of step S107 to the outside of the encoding device 100.

Once the processing of step S108 ends, the encoding process ends.

<Flow of plaque decomposition processing>
Next, an example of the flow of the plaque decomposition processing executed in step S101 of Fig. 10 will be described with reference to the flowchart of Fig. 11 .

When the plaque decomposition processing is started, the normal direction estimating unit 151 estimates the normal direction in step S121. In step S122, the segment initial setting unit 152 performs initial setting of the segmentation. In step S123, the segment update unit 153 updates the segment of the initial state set in step S122 as necessary. In step S124, the two-dimensional projection unit 154 projects the 3D material onto the two-dimensional plane.

Once the processing of step S124 ends, the plaque decomposition processing ends, and the processing returns to FIG.

<Process of 2-dimensional projection processing>
Next, an example of the flow of the two-dimensional projection processing executed in step S124 of Fig. 11 will be described with reference to the flowchart of Fig. 12 .

When the two-dimensional projection processing is started, the two-dimensional projection unit 154 is in step S141, and performs field extraction by segmentation. In step S142, the two-dimensional projection unit 154 is initialized so that the layer number i=0.

In step S143, the pixel distribution analysis unit 155 determines whether or not there is an unprojected pixel (data that is not projected to each position of the 3D material of the two-dimensional plane). If it is determined that there is an unprojected pixel, the processing proceeds to step S144.

In step S144, the two-dimensional projection unit 154 projects the domain to the layer i (Layer i) of the processing target. In step S145, the two-dimensional projection unit 154 adds the variable i (i++). Once the process of step S145 is ended, the process returns to step S143, and the subsequent processes are repeated.

In step S143, if it is determined that the unprojected pixel does not exist (all the pixels in the field have been projected), the processing proceeds to step S146.

In step S146, the two-dimensional projection unit 154 supplies information indicating the number of layers i, and encodes it. Further, in step S147, the two-dimensional projection unit 154 causes the geometric image of the corresponding i-frame portion to be encoded. That is, the two-dimensional projection unit 154 supplies a two-dimensional plane on which 3D data has been projected to the packing unit 112, and the layers are packaged into mutually different frames.

Once the process of step S147 is completed, the two-dimensional projection process ends, and the process returns to FIG.

By performing the respective processes as above, by designing such, it is possible to more accurately reproduce the point cloud having a thickness on the surface of the object. That is, it is possible to suppress a decrease in quality due to 2-dimensional projection of 3D data.

<Reconstruction of 3D data that has been projected onto a 2-dimensional plane with variable number of layers>
On the decoding side, by using the information indicating the number of layers of the two-dimensional plane provided by the encoding side, the 3D data that has been projected onto the two-dimensional plane whose number of layers is variable can be realized as described above. Reconstructed.

That is, the data of each position of the 3D material on the two-dimensional plane that has been projected onto the number of layers indicated by the layer number information is projected to the three-dimensional space. For example, in the image processing device, a three-dimensional projection unit is provided, and data of each position of the 3D material on the two-dimensional plane that has been projected onto the number of layers indicated by the layer number information is projected. To 3D space.

With such a design, reconstruction of 3D data that has been projected onto a two-dimensional plane whose number of layers is variable can be realized. That is, the point cloud having a thickness on the surface of the object can be reproduced more correctly. That is, it is possible to suppress a decrease in quality due to 2-dimensional projection of 3D data.

<Process of decoding processing>
An example of the flow of the decoding process executed by the decoding device 200 will be described with reference to the flowchart of FIG.

Once the decoding process is started, the demultiplexer 211 of the decoding device 200 is in step S201, and the bit stream is inversely multiplexed.

In step S202, the auxiliary patch information decoding unit 212 decodes the auxiliary patch information extracted from the bit stream by the processing of step S201. In step S203, the video decoding unit 213 decodes the encoded data of the geometric video frame (the video frame of the position information) extracted from the bit stream by the processing of step S201.

In step S204, the video decoding unit 214 decodes the coded data of the color video frame (the video frame of the attribute information) extracted from the bit stream by the process of step S201. In step S205, the OMap decoding unit 215 decodes the encoded data of the occupied map extracted from the bit stream by the processing of step S201.

In step S206, the unpacking unit 216 unpacks the geometric video frame, the color video frame, and the occupied map decoded in step S203 to step S205, respectively.

In step S207, the 3D reconstruction unit 217 reconstructs the 3D material such as the point cloud based on the auxiliary patch information obtained in step S202 and various pieces of information obtained in step S206.

Once the processing of step S207 ends, the decoding process ends.

<Process of point cloud re-construction processing>
Next, an example of the flow of the point cloud reconstruction process executed in step S207 of Fig. 13 will be described with reference to the flowchart of Fig. 14.

When the point cloud reconstruction process is started, the three-dimensional projection unit 251 is in step S221 to project the two-dimensional image into the three-dimensional space.

In step S222, the inverse segment update unit 253 updates the segments in the reverse direction and divides the segments that have been assembled.

In step S223, the inverse segmentation initial setting unit 254 performs inverse processing of the initial setting of the segmentation, and aggregates the classified points.

In step S224, the inverse normal direction estimating unit 255 performs inverse processing of the normal direction estimation to reconstruct the point cloud.

Once the process of step S224 ends, the point cloud reconstruction process ends and the process returns to FIG.

<3D projection processing flow>
Next, an example of the flow of the three-dimensional projection processing executed in step S221 of Fig. 14 will be described with reference to the flowchart of Fig. 15 .

When the three-dimensional projection processing is started, the three-dimensional projection unit 251, in step S241, causes the information indicating the number of layers i to be decoded.

In step S242, the three-dimensional projection unit 251 causes the geometric image of the corresponding i-frame to be decoded.

In step S243, the pixel distribution analysis unit 252 performs initialization to make the variable k=0.

In step S244, the pixel distribution analysis unit 252 determines whether or not the variable k<i. If it is determined that k < i, the processing proceeds to step S245.

In step S245, the three-dimensional projection unit 251 projects the 3D material of the layer k (Layer k) into a three-dimensional space.

In step S246, the pixel distribution analysis unit 252 adds the variable k (k++).

Once the process of step S246 ends, the process returns to step S244.

Moreover, if it is determined in step S244 that the variable k<i is not present, the three-dimensional projection processing is terminated, and the processing returns to FIG.

By performing the respective processes as above, reconstruction of the 3D data that has been projected onto the two-dimensional plane whose number of layers is variable can be realized. That is, the point cloud having a thickness on the surface of the object can be reproduced more correctly. That is, it is possible to suppress a decrease in quality due to 2-dimensional projection of 3D data.

<3. Second embodiment>
<Message indicating the value of "empty pixel">
Once the sparse (Sparse) 3D data is projected onto the 2D plane, it will occur in the 2D plane where the 3D data is not projected, that is, the pixel (also referred to as the empty pixel) that is not set to the pixel value. .

In the previous method, in the case where the 3D data is sparse (Sparse), as shown in FIG. 16, such "empty pixels" complement the pixel values. For example, in the case of FIG. 16, for the "empty pixel" of the layer 0 (Layer0), the pixel value is copied from the pixel adjacent to the left side (the pixel complement is performed). The reason for this complement processing is that the coding method for 2D images (such as AVC or HEVC, etc.) does not have the concept of no data (blank).

However, once such a complementary process is performed, a point that does not exist originally is added to the 3D material. Therefore, it is feared that the 3D data will deteriorate. That is, I am afraid that the quality will be degraded due to the 2-dimensional projection of 3D data.

Further, if such a complement processing is not performed, it is necessary to set the field so that "empty pixels" are not generated on the two-dimensional plane on which the 3D data is projected. Therefore, a projection of a small point unit is required, which may lead to an increase in throughput or a decrease in coding efficiency.

Then, the data representing each position included in the 3D data of the 3-dimensional structure is projected onto the 2-dimensional plane, and at each position where the data of each position of the 2-dimensional plane is non-existent, the setting indicates each position. The data is the value that does not exist. For example, in the image processing apparatus, the two-dimensional projection unit is provided to project data of each position included in the 3D data of the three-dimensional structure to a two-dimensional plane, and at each position of the two-dimensional plane. The data is a location where the data does not exist, and the data indicating that each location is a predetermined value that does not exist.

For example, as shown in FIG. 17, the pixel value indicating "empty pixel" on the two-dimensional plane on which the 3D data is projected is a predetermined value X. The value of X can also be set, for example, to a predetermined fixed value that has been predetermined. For example, it can also be set to X=255 (the upper limit of 8 bits).

In this case, the value of X must be signaled in the bit stream (must be notified to the decoding side).

Further, for example, the value of X may be any value of X>D. The D system here indicates the maximum value of the depth of the bounding box. In the pixel value of the 2-dimensional plane, it is impossible to set a value exceeding the maximum value of the depth of the bounding frame. Therefore, it is also possible to use unused values as X.

In this case, on the decoding side, it is possible to determine whether the pixel value set to the 2-dimensional plane is an unused value based on the related information of the size of the bounding frame. Therefore, there is no need to signal the value of X in the bit stream (ie, there is no need to inform the decoding side of the value of X). However, of course, the value of X can be notified to the decoding side.

With such a design, the encoding apparatus 100 can express "empty pixels" without complementing the data, thereby suppressing deterioration of 3D data. That is, it is possible to suppress a decrease in quality due to 2-dimensional projection of 3D data.

Further, with such a design, the encoding apparatus 100 can reduce the number of "pixels that cannot be encoded" in the two-dimensional plane. Therefore, the field can be set larger than when the complementary processing is not performed. Therefore, projection of a minute unit is not required, and an increase in the amount of processing or a decrease in coding efficiency can be suppressed.

<Process of 2-dimensional projection processing>
Also in this case, the encoding process and the plaque decomposition process are performed in the same manner as described in the first embodiment. Therefore, the description of these is omitted.

An example of the flow of the two-dimensional projection processing in this case executed in step S124 of Fig. 11 will be described with reference to the flowchart of Fig. 18.

When the two-dimensional projection processing is started, the two-dimensional projection unit 154 performs the field extraction by the segmentation in step S301. In step S302, the pixel distribution analysis unit 155 determines whether or not it is a field including a sparse point cloud. If it is determined that it is the field including the sparse point cloud, the processing proceeds to step S303.

In step S303, the two-dimensional projection unit 154 transmits information that defines "empty pixels" for the field. That is, the X described above is set and expressed by the X, and has been projected onto the "non-existing point" on the two-dimensional plane. Once the process of step S303 ends, the process proceeds to step S305.

Moreover, if it is determined in step S302 that the field of the sparse point cloud is not included, the processing proceeds to step S304.

In step S304, the two-dimensional projection unit 154 projects the field to each layer of the two-dimensional plane. Once the process of step S304 ends, the process proceeds to step S305.

In step S305, the two-dimensional projection unit 154 causes the geometric image of the field to be encoded.

Once the process of step S305 ends, the 2-dimensional projection process ends and the process returns to FIG.

As described above, by performing the two-dimensional projection processing, the encoding apparatus 100 can express "empty pixels" without complementing the data. Therefore, deterioration of 3D data can be suppressed. That is, it is possible to suppress a decrease in quality due to 2-dimensional projection of 3D data.

Further, with such a design, the encoding apparatus 100 can reduce the number of "pixels that cannot be encoded" in the two-dimensional plane. Therefore, the field can be set larger than when the complementary processing is not performed. Therefore, projection of a minute unit is not required, and an increase in the amount of processing or a decrease in coding efficiency can be suppressed.

<Usage of the value of "empty pixel" that has been transmitted >
On the decoding side, when the 3D data that has been projected onto the 2-dimensional plane is projected into the 3-dimensional space, the pixel value of the "empty pixel" on the two-dimensional plane that is transmitted as described above will be transmitted. (X above) is detected and deleted (so that it is not projected).

That is, in the data of each position included in the 3D data representing the 3-dimensional structure that has been projected onto the 2-dimensional plane, it is indicated that the data of each position is the data of the predetermined value that does not exist. The data is projected into a 3-dimensional space. For example, in the image processing apparatus, the three-dimensional projection unit is provided, and is displayed in the data of each position included in the 3D data representing the three-dimensional structure that has been projected onto the two-dimensional plane. The data of a location is data other than the data of the predetermined value that does not exist, and is projected into a 3-dimensional space.

With this design, it is possible to express "empty pixels" without having to supplement the data, thereby suppressing deterioration of 3D data. That is, it is possible to suppress a decrease in quality due to 2-dimensional projection of 3D data.

Further, by designing this, the number of "pixels that cannot be encoded" in the two-dimensional plane can be reduced, so that the field can be set larger than when the complement processing is not performed alone. Therefore, projection of a minute unit is not required, and an increase in the amount of processing or a decrease in coding efficiency can be suppressed.

<3D projection processing flow>
Also in this case, the decoding process and the point cloud reconstruction process are performed in the same manner as described in the first embodiment. Therefore, the description of these is omitted.

An example of the flow of the three-dimensional projection processing in this case executed in step S221 of Fig. 14 will be described with reference to the flowchart of Fig. 19.

When the three-dimensional projection processing is started, the three-dimensional projection unit 251, in step S321, decodes the geometric image of the domain of the processing target.

In step S322, the pixel distribution analysis unit 252 determines whether or not there is information (a pixel value indicating "empty pixel") in which the "empty pixel" is defined in the geometric image. If it is determined to be present, the process proceeds to step S323.

In step S323, the three-dimensional projection unit 251 deletes the pixel of the detected pixel value indicating "empty pixel" (set it so as not to project into a three-dimensional space). Once the process of step S323 ends, the process proceeds to step S324.

In addition, if it is determined in step S322 that the pixel value indicating "empty pixel" does not exist, the processing of step S323 is omitted, and the processing proceeds to step S324.

That is, the detection of the pixel value indicating "empty pixel" is performed on the geometric image of the domain of the processing object, and if detected, the pixel is deleted.

In step S324, the three-dimensional projection unit 251 projects the geometric image of the domain to be processed into a three-dimensional space.

Once the process of step S324 ends, the 3-dimensional projection process ends and the process returns to FIG.

By performing the three-dimensional projection processing as described above, the decoding apparatus 200 can express "empty pixels" without complementing the data, and thus it is possible to suppress degradation of the 3D data. That is, it is possible to suppress a decrease in quality due to 2-dimensional projection of 3D data.

Further, with such a design, the decoding apparatus 200 can reduce the number of "pixels that cannot be encoded" in the two-dimensional plane. Therefore, the field can be set larger than when the complement processing is not performed alone. Therefore, projection of a minute unit is not required, and an increase in the amount of processing or a decrease in coding efficiency can be suppressed.

<4. Third embodiment>
<Control of depth parameters>
When the 3D data is projected onto the two-dimensional plane, the depth parameter th that controls the depth direction of the 3D data projected onto the two-dimensional plane is used. Since the point within the range specified by the depth parameter th becomes a projection object to the two-dimensional plane, the value of the depth parameter th is related to the length of the depth direction of the local bounding box. For example, if the value of the depth parameter th is larger than the length of the depth direction of the field, then points in other fields may become projection objects. That is, the length of the field in the depth direction must be longer than the depth parameter th.

In the previous method, as shown in Fig. 20, the depth parameter th is controlled in frame units. Therefore, all the depth parameters th in the frame have a common value.

However, for example, the face and the feet, the center and the peripheral portion, the surface density of the point cloud in the frame is not constant, and it is also possible. Therefore, the value of the depth parameter th is not always optimal, and it is feared that the coding efficiency is lowered. For example, it is better to divide the field in the depth direction of the Ming Dynasty, but the value of the depth parameter th will become larger, resulting in an inseparable case.

Therefore, the data of each position of the 3D data representing the 3-dimensional structure is projected to the 2-dimensional plane according to each field defined by the 3-dimensional space, and the setting for each field is limited. The data projected to each of the positions in the depth direction indicated by the depth parameter in the depth direction of the data of each position of the 3D data of the one-dimensional structure projected onto the one-layer layer is projected onto the two-dimensional plane. For example, in the image processing device, the two-dimensional projection unit is provided, and the data of each position of the 3D data representing the three-dimensional structure is projected onto the two-dimensional plane in accordance with each field defined by the three-dimensional space. And each of the fields is set to limit the depth direction indicated by the depth parameter of the range of the depth direction of the data of each position of the 3D data representing the three-dimensional structure that can be projected to one layer. The data for each location is projected onto the 2D plane.

For example, as shown in FIG. 21, it is expanded to transmit the depth parameter for each patch, and each field transmits the position (TH) of the pixel to be projected to the layer. By designing in this way, for example, the depth parameter th can be set for the field 1 and the depth parameter th' can be set for the field 2. Therefore, the efficiency of Layer 1 (Geyer coding efficiency) can be improved.

<Process of 2-dimensional projection processing>
Also in this case, the encoding process and the plaque decomposition process are performed in the same manner as described in the first embodiment. Therefore, the description of these is omitted.

An example of the flow of the two-dimensional projection processing in this case executed in step S124 of Fig. 11 will be described with reference to the flowchart of Fig. 22 .

When the two-dimensional projection processing is started, the pixel distribution analysis unit 155 performs initialization in step S401 so that the field number i=0.

In step S402, the pixel distribution analysis unit 155 determines whether or not there is an unprocessed pixel. If it is determined that there is an unprocessed pixel, the processing proceeds to step S403.

In step S403, the two-dimensional projection unit 154 performs extraction of the field i by segmentation.

In step S404, the two-dimensional projection unit 154 performs adjustment and encoding of the depth parameter (RD determination). That is, the setting of the optimum depth parameter th and the range of the field is performed by the RD determination.

In step S405, the two-dimensional projection unit 154 projects the geometric image of the processing target domain i onto the two-dimensional surface based on the settings made in step S404.

In step S406, the two-dimensional projection unit 154 causes the geometric image that has been projected onto the two-dimensional plane to be encoded.

In step S407, the pixel distribution analysis unit 155 adds the variable i (i++). Once the process of step S407 ends, the process returns to step S402, and the subsequent processes are repeated.

Moreover, if it is determined in step S402 that the unprocessed pixel does not exist, the processing proceeds to step S408.

In step S408, the two-dimensional projection unit 154 causes the Occulusion Map to be encoded. Once the processing of step S408 ends, the two-dimensional projection processing ends, and the processing returns to FIG.

By performing the two-dimensional projection processing as described above, the encoding apparatus 100 can obtain the depth parameter th suitable for each field, and can suppress the reduction in encoding efficiency.

<Use of controlled depth parameters>
On the decoding side, when the 3D data that has been projected onto the two-dimensional plane is projected into the three-dimensional space, the 3D data is projected within the range indicated by the depth parameter th controlled as described above.

That is, according to each field defined by the three-dimensional space, when the data of each position included in the 3D data representing the three-dimensional structure projected onto the two-dimensional plane is projected onto the three-dimensional space, Projecting the data of each position into the 3D space, and setting the depth parameter of the range of the depth direction of the data of each position of the 3D data that can be projected to the 1 layer for each field Within the range of the depth direction indicated. For example, in the image processing device, the three-dimensional projection unit is provided, and each of the 3D data representing the three-dimensional structure that has been projected onto the two-dimensional plane is included in each of the fields defined by the three-dimensional space. When the data of a position is projected into the three-dimensional space, the data of each position is projected into the three-dimensional space, and the set of each field is used to limit the 3D data that can be projected to one layer. The depth direction of the range of the depth direction of the data of each position is within the range of the depth direction.

With this design, the depth parameter th set for each field can be used to project the 3D data into the three-dimensional space, so that the reduction in coding efficiency can be suppressed.

<3D projection processing flow>
Also in this case, the decoding process and the point cloud reconstruction process are performed in the same manner as described in the first embodiment. Therefore, the description of these is omitted.

An example of the flow of the three-dimensional projection processing in this case, which is executed in step S221 of Fig. 14, will be described with reference to the flowchart of Fig. 23.

Once the three-dimensional projection processing is started, the three-dimensional projection unit 251 is caused to decode the Occlusion map in step S421.

In step S422, the pixel distribution analysis unit 252 initializes the field number k=0.

In step S423, the pixel distribution analysis unit 252 determines whether or not the variable k<i. If it is determined that k < i, the processing proceeds to step S424.

In step S424, the three-dimensional projection unit 251 causes the depth parameter th to be decoded. In step S425, the three-dimensional projection unit 251 causes the geometric image to be decoded. In step S426, the three-dimensional projection unit 251 projects the image of the field into a three-dimensional space.

In step S427, the pixel distribution analysis unit 252 adds the variable k (k++). Once the process of step S427 is ended, the process returns to step S423, and the subsequent processes are repeated.

Further, if it is determined in step S423 that k < i is not satisfied, the three-dimensional projection processing is terminated, and the processing returns to FIG.

By performing the three-dimensional projection processing as described above, the decoding apparatus 200 can project the 3D data into the three-dimensional space by using the depth parameter th set for each field, thereby suppressing the reduction in coding efficiency.

<5. Attachment>
<Control Information>
The control information about the present technology described in the above embodiments may be transmitted from the encoding side to the decoding side. For example, control information (e.g., enabled_flag) for applying the above-described technique may be controlled to be permitted (or disabled) for transmission. Also, for example, the scope of the above-described technology (eg, the upper or lower limit of the block size, or both sides, slices, images, sequences, components, viewpoints, layers, etc.) may be used to specify permission (or prohibition). Control information is transmitted.

<computer>
The series of processing described above can be performed by hardware or by software. When a series of processes are executed in software, the program constituting the software can be installed to a computer. Here, the computer system includes a computer that is incorporated in a dedicated hardware, or a general-purpose personal computer that can perform various functions by installing various programs.

Fig. 24 is a block diagram showing a configuration example of a hardware of a computer that executes the above-described series of processes in a program.

In the computer 900 shown in FIG. 24, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are connected to each other via the bus bar 904.

The bus 904 is also connected to the input/output interface 910. The input/output interface 910 is connected to an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive unit 915.

The input unit 911 is formed, for example, by a keyboard, a mouse, a microphone, a touch panel, an input terminal, or the like. The output unit 912 is formed of, for example, a display, a speaker, an output terminal, or the like. The memory unit 913 is formed of, for example, a hard disk, a RAM disk, a non-volatile memory, or the like. The communication unit 914 is formed by, for example, a network interface. The drive unit 915 is driven by a removable medium 921 such as a magnetic disk, a compact disk, a magneto-optical disk, or a semiconductor memory.

In the computer having the above configuration, the program stored in the memory unit 913 is loaded into the RAM 903 through the input/output interface 910 and the bus bar 904, for example, by the CPU 901, and the above-described series of processes can be performed. In the RAM 903, information necessary for the CPU 901 to execute various processes and the like are also appropriately stored.

The program executed by the computer (CPU 901) can be recorded in a removable medium 921 such as a package medium. In this case, the program is attached to the memory unit 913 via the input/output interface 910 by attaching the removable medium 921 to the drive unit 915.

In addition, the program can be provided via a wired or wireless transmission medium such as a regional network, an Internet, or a digital satellite. In this case, the program can be received by the communication unit 914 and installed in the memory unit 913.

In addition to this, the program can be installed in advance in the ROM 902 or the memory unit 913.

<Applicable object of the technology>
Although the above description has been given of the case where the present technology is applied to Voxelization of point cloud data, the present technology is not limited to these examples, and may be applied to Voxelization of 3D data of any specification. In other words, as long as there is no contradiction with the above-described technique, various processing such as encoding and decoding methods, and specifications of various materials such as 3D data or post-data are arbitrary. Further, as long as there is no contradiction with the present technology, a part of the processing or specifications described above may be omitted.

Further, although the encoding device 100 and the decoding device 200 have been described above as an application example of the present technology, the present technology can be applied to any configuration.

For example, the present technology can be applied to a satellite broadcast, a wired TV broadcast of a cable TV, a communication on the Internet, and a transmitter or a receiver at the time of communication with a terminal by communication with a cellular base station (for example, a television receiver or a mobile phone, or a device that records images on or reproduces images from media such as a compact disc, a flash memory, or the like (such as a hard disk recorder or a video camera). A wide range of electronic machines.

Further, for example, the present technology can also be used as a processor such as a system LSI (Large Scale Integration) (for example, a video processor), a module using a complex processor (for example, a video module), or a unit using a complex module (for example). For example, a video unit, a kit (for example, a video set) to which other functions are added to the unit, and the like are implemented as part of the device.

For another example, the present technology can also be applied to a network system composed of a plurality of devices. For example, the present technology can be implemented by sharing a plurality of devices over a network and performing cloud computing for processing together. For example, it can be implemented in a cloud service that provides services related to video (motion video) to any terminal such as a computer, an AV (Audio Visual) device, a portable information processing terminal, and an IoT (Internet of Things) device. This technology.

In addition, in this specification, a system means a collection of a plurality of components (devices, modules (parts), etc.), and all components are not in the same frame. Therefore, a plurality of devices that are housed in individual frames, connected through a network, and one device in which a plurality of modules are housed in one frame are all systems.

<Fields and uses in which the technology can be applied>
The system, the device, the processing unit, and the like to which the present technology is applied can be utilized in, for example, transportation, medical care, theft prevention, agriculture, animal industry, mining, beauty, workshops, home appliances, weather, natural surveillance, and the like. Moreover, its use is also arbitrary.

For example, the present technology may be a system or device for providing for viewing content or the like. For example, the present technology can also be applied to a system or device for traffic use, such as supervision of traffic conditions or automatic driving control. Even, for example, the technology can be applied to systems or devices for security purposes. For another example, the present technology can also be applied to a system or device for automatic control of a machine or the like. Even, for example, the present technology can be applied to systems or devices for use in the agricultural or livestock industry. Further, the present technology can also be applied to, for example, a system or device that monitors natural states such as volcanoes, forests, oceans, and the like, or wildlife. Even, for example, the present technology can be applied to systems or devices for use in sports.

<Other>
In addition, the "flag" in this specification is the information required to identify the complex state, and not only the information used to identify the two states of true (1) or pseudo (0), It also contains information that can identify more than three states. Therefore, the value that the "flag" can take is, for example, a value of 1/0, or a value of 3 or more. In other words, the number of bits constituting the "flag" is arbitrary, and may be 1 bit or a complex bit. Moreover, the identification information (including the flag) is not only included in the form of the bit stream, but also the differential information of the identification information relative to a certain reference information is included in the bit string. In the form of a stream, in this specification, "flag" or "identification information" includes not only the information but also differential information relative to the information as a reference.

Further, various kinds of information (post-design data, etc.) related to the encoded data (bit stream) may be transmitted or recorded in any form as long as it is related to the encoded data. Here, the term "establishing a connection" means, for example, a situation in which the information of the other party may be utilized when the data of one party is processed (a link may be established). That is to say, information that is related to each other can be regarded as one piece of information as a whole or as individual information. For example, information associated with encoded data (images) may also be transmitted on other transmission paths different from the encoded material (images). Further, for example, the information associated with the encoded material (image) may be recorded in another recording medium (or other area of the same recording medium) different from the encoded material (image). In addition, the "establishment of connection" may not be the whole of the information, but part of the information. For example, the image and the information corresponding to the image may be associated with each other using any of a plurality of frames, a frame, or a part of the frame.

In addition, in this specification, "synthesis", "multiplexing", "additional", "integration", "inclusion", "storage", "input", "insertion", "insertion", etc. The term means that, for example, the combination of the coded data and the post-set data into one piece of information and the binding of the plurality of items into one means a method of "establishing a connection" as described above.

Further, the embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the spirit and scope of the invention.

For example, a configuration in which one device (or processing unit) is described may be divided into a plurality of devices (or processing units). On the other hand, the configuration described by the plurality of devices (or processing units) in the above description may be summarized into one device (or processing unit). Further, it is needless to say that the configuration of each device (or each processing unit) may be other than the above. Furthermore, if the entire configuration or operation of the system is substantially the same, a part of the configuration of a certain device (or processing unit) may be included in the configuration of another device (or other processing unit).

Further, for example, the above-described program can be executed in any device. In this case, as long as the device has the necessary functions (function blocks, etc.), necessary information can be obtained.

Further, for example, each step of one flowchart may be executed by one device or may be performed by a plurality of devices. Even when a plurality of processes are included in one step, the plurality of processes may be executed by one device or may be performed by a plurality of devices. In other words, the plurality of processes included in one step can also be performed in a plurality of steps. On the other hand, the processing described in the plural steps can be integrated into one step and executed.

Further, for example, the program executed by the computer and the processing of the steps describing the program may be executed in time series in accordance with the order described in the specification, or may be in parallel, or may be necessary to perform a call. On, but executed individually. That is, as long as no contradiction occurs, the processing of each step can be performed in a different order from the above. Even the processing of the steps describing the program can be performed in parallel with the processing of other programs, or can be performed in combination with the processing of other programs.

Further, for example, a plurality of techniques related to the present technology can be implemented independently as a single unit as long as no contradiction occurs. Of course, any of a plurality of the techniques can be used in combination. For example, part or all of the present technology described in any of the embodiments may be combined with some or all of the present technology described in the other embodiments. Further, some or all of the above-described techniques may be used in combination with other techniques not described above.

Further, the present technology can also adopt the following constitution.
(1) An image processing apparatus having:
The two-dimensional projection unit projects data of each position included in the 3D data of the three-dimensional structure onto a two-dimensional plane of the complex layer.
(2) The image processing device according to (1), wherein
In the second-dimensional projection unit, the position of the 3D data from the projection surface is the position of each position in the depth direction, and is projected on the two-dimensional plane before the multi-layer layer.
(3) The image processing device according to (2), wherein
The pre-recorded two-dimensional projection unit generates the same number of layers as the maximum number of pieces of data of each position of the pre-recorded 3D data from the projection surface as viewed from the projection surface in the depth direction.
(4) The image processing device according to any one of (1) to (3), wherein
Further, the encoding unit is configured to project the 3D data which has been projected by the pre-recorded two-dimensional projection unit to the pre-recorded two-dimensional plane.
(5) The image processing device according to (4), wherein
The preamble encoding unit encodes the position information, the attribute information, and the occupied map of the pre-recorded 3D data that has been projected onto each layer of the pre-recorded two-dimensional plane.
(6) The image processing device according to (4) or (5), wherein
Further, the present invention further includes a bit stream generation unit that generates a bit stream including information indicating a number of layers of a pre-recorded two-dimensional plane on which the 3D data is projected by the two-dimensional projection unit, and The precoding unit encodes the encoded data obtained by encoding the 2D plane.
(7) The image processing device according to any one of (4) to (6), wherein
Further, the packaging unit is configured to project a two-dimensional plane of the pre-recorded 3D data by a two-dimensional projection unit, and package it into a video frame;
The preamble encoding unit is configured to encode a pre-recorded video frame in which a pre-recorded two-dimensional plane is packaged by a pre-packaging unit.
(8) The image processing device according to any one of (1) to (7), wherein
The pre-recorded 2D projection unit projects the 3D data of the predecessor into the pre-recorded 2D plane according to each field.
(9) The image processing device according to any one of (1) to (8) wherein
The pre-recorded 3D data is a point cloud.
(10) An image processing method for projecting data of each position included in a 3D material of a three-dimensional structure onto a two-dimensional plane of a complex layer.

(11) An image processing apparatus comprising:
The three-dimensional projection unit projects data of each position of the 3D data on the two-dimensional plane that has been projected onto the number of layers indicated by the layer number information into a three-dimensional space.
(12) The image processing device according to (11), wherein
It also has a drawing unit for extracting the information of the number of layers in the bit stream which is included in the bit stream;
The pre-recorded three-dimensional projection unit is configured to record all the pre-recorded positions of the pre-recorded 3D data that have been projected on the two-dimensional plane before the number of layers indicated by the number of layers before the extracted portion is extracted. The data is projected to the front 3D space.
(13) The image processing device according to (12), wherein
Further, the decoding unit further includes: decoding the encoded data of the pre-recorded 3D data included in the pre-recorded bit stream that has been projected onto the pre-recorded two-dimensional plane;
The pre-recorded three-dimensional projection unit is configured to decode the pre-recorded data by the pre-decoding unit, and project the data onto all the pre-recorded positions of the pre-recorded 3D data on the two-dimensional plane. Pre-record 3D space.
(14) The image processing device according to (13), wherein
The predecessor decoding unit decodes the positional information, the attribute information, and the coded data of each of the occupied maps of the 3D data of each layer that has been projected onto each of the pre-recorded two-dimensional planes.
(15) The image processing device according to (13) or (14), wherein
Further, the unpacking unit is obtained by decoding the preamble encoded data by the pre-recording unit, and the video frame that has been projected onto the pre-recorded 3D data of the pre-recorded two-dimensional plane is unpacked;
The pre-recorded three-dimensional projection unit is configured to unpack the video frame by the pre-decoding unit, and project the data into all the pre-recorded positions of the pre-recorded 3D data on the two-dimensional plane. Projected to the front 3D space.
(16) The image processing device according to any one of (11) to (15), wherein
The 3D projection unit is projected to the 3D space of the pre-recorded 2D plane in accordance with each of the defined fields.
(17) The image processing device according to any one of (11) to (16), wherein
The pre-recorded 3D data is a point cloud.
(18) An image processing method for projecting data of each position of 3D data on a two-dimensional plane that has been projected onto the number of layers indicated by the layer number information into a three-dimensional space.

(21) An image processing apparatus comprising:
The two-dimensional projection unit projects data of each position included in the 3D data of the three-dimensional structure onto a two-dimensional plane, and records the position of each position before the two-dimensional plane as a non-existent position. Indicates that the information in each of the preceding positions is the value that does not exist.
(22) The image processing device according to (21), wherein
The value stated in the preceding paragraph is a fixed value that has been predetermined.
(23) The image processing device according to (21), wherein
The value set in the preceding paragraph is a value larger than the maximum value of the depth of the 3D data.
(24) The image processing device according to any one of (21) to (23), wherein
Further, the encoding unit is configured to project the 3D data which has been projected by the pre-recorded two-dimensional projection unit to the pre-recorded two-dimensional plane.
(25) The image processing device according to (24), wherein
The preamble encoding unit encodes the position information, the attribute information, and the occupied map of the 3D data that has been projected onto the pre-recorded 2D plane.
(26) The image processing device according to (24) or (25), wherein
Further, the present invention further includes a bit stream generating unit that generates a bit stream including: information indicating a value set in the foregoing, and coded data obtained by encoding the pre-recorded two-dimensional plane by the pre-encoding unit.
(27) The image processing device according to any one of (24) to (26), wherein
Further, the packaging unit is configured to project a two-dimensional plane of the pre-recorded 3D data by a two-dimensional projection unit, and package it into a video frame;
The preamble encoding unit is configured to encode a pre-recorded video frame in which a pre-recorded two-dimensional plane is packaged by a pre-packaging unit.
(28) The image processing device according to any one of (21) to (27), wherein
The pre-recorded 2D projection unit projects the 3D data of the predecessor into the pre-recorded 2D plane according to each field.
(29) The image processing device according to any one of (21) to (28), wherein
The pre-recorded 3D data is a point cloud.
(30) An image processing method for projecting data of each position included in a 3D data representing a 3-dimensional structure onto a 2-dimensional plane, and recording the information of each position before the 2-dimensional plane as a non-existent position On the top, the data indicating that each position of the preceding record is a non-existent value.

(31) An image processing apparatus comprising:
The 3-dimensional projection unit, within the data of each position included in the 3D data representing the 3-dimensional structure that has been projected onto the 2-dimensional plane, will indicate that the data of each position in the preceding record is a non-existent value. The data other than the data is projected into the 3-dimensional space.
(32) The image processing device according to (31), wherein
The value stated in the preceding paragraph is a fixed value that has been predetermined.
(33) The image processing device according to (31), wherein
The value set in the preceding paragraph is a value larger than the maximum value of the depth of the 3D data.
(34) The image processing device according to any one of (31) to (33), wherein
Further, the extracting unit extracts information indicating a value specified in the preceding stream contained in the bit stream;
The pre-recorded three-dimensional projection unit is configured to include, in the data of each position previously included in the pre-recorded 3D data, data that has been extracted by the pre-recording unit and the value of the pre-recorded information indicated by the information. Data, projected to 3D space.
(35) The image processing device according to (34), wherein
Further, the decoding unit further includes: decoding the encoded data of the pre-recorded 3D data included in the pre-recorded bit stream that has been projected onto the pre-recorded two-dimensional plane;
The pre-recorded three-dimensional projection unit is configured to decode the pre-recorded encoded data by the pre-decoding unit, and has been projected onto the pre-recorded 3D data of the pre-recorded two-dimensional plane. In the case, the data other than the data of the value indicated in the previous record indicated by the pre-recording unit is projected to the three-dimensional space.
(36) The image processing device according to (35), wherein
The predecessor decoding unit decodes the positional information, the attribute information, and the coded data of each of the occupied maps of the 3D data of each layer that has been projected onto each of the pre-recorded two-dimensional planes.
(37) The image processing device of (35) or (36), wherein
Further, the unpacking unit is obtained by decoding the preamble encoded data by the pre-recording unit, and the video frame that has been projected onto the pre-recorded 3D data of the pre-recorded two-dimensional plane is unpacked;
The pre-recorded three-dimensional projection unit is configured to unpack the video frame by the pre-decoding unit, and has been projected to the pre-recorded 3D plane of the pre-recorded 2D plane. Within the data, the data other than the data of the previous value indicated by the previous information extracted by the pre-recording department is projected into the 3-dimensional space.
(18) The image processing device according to any one of (31) to (37), wherein
The 3D projection unit is projected to the 3D space of the pre-recorded 2D plane in accordance with each of the defined fields.
(39) The image processing device according to any one of (31) to (38), wherein
The pre-recorded 3D data is a point cloud.
(40) An image processing method in which data representing each position included in a 3D structure of a 3D structure that has been projected onto a 2-dimensional plane is non-existent Projects other than the data of the specified value are projected into the 3-dimensional space.

(41) An image processing apparatus comprising:
The two-dimensional projection unit sets the data of each position of the 3D data representing the three-dimensional structure to each of the fields defined in the three-dimensional space to the two-dimensional plane. It is used to limit the data of each position in the range of the depth in the depth direction indicated by the depth parameter of the depth direction of the data of each position of the 3D data which can be projected to one layer, and project to the pre-record 2D plane.
(42) The image processing device according to (41), wherein
Further, the encoding unit is configured to project the 3D data which has been projected by the pre-recorded two-dimensional projection unit to the pre-recorded two-dimensional plane.
(43) The image processing device according to (42), wherein
The preamble encoding unit encodes the position information, the attribute information, and the occupied map of the 3D data that has been projected onto the pre-recorded 2D plane.
(44) The image processing device of (42) or (43), wherein
A bit stream generation unit is further configured to generate a bit stream, wherein: the depth parameter selected before each field of the preceding record and the coded data obtained by encoding the previous two-dimensional plane by the preceding coding unit .
(45) The image processing device according to any one of (42) to (44), wherein
Further, the packaging unit is configured to project a two-dimensional plane of the pre-recorded 3D data by a two-dimensional projection unit, and package it into a video frame;
The preamble encoding unit is configured to encode a pre-recorded video frame in which a pre-recorded two-dimensional plane is packaged by a pre-packaging unit.
(4) The image processing device according to any one of (41) to (45), wherein
The pre-recorded 3D data is a point cloud.
(47) An image processing method in which each piece of data representing a 3D structure of a 3D structure is projected onto a 2D plane in accordance with a field defined by a 3D space, and will be recorded in each field of the predecessor. It is set to limit the data of each position in the range of the depth direction indicated by the depth parameter in the depth direction of the data indicating the depth direction of the data of each position of the 3D structure which can be projected to one layer, Project to the front 2D plane.

(51) An image processing apparatus comprising:
The three-dimensional projection unit projects the data of each position included in the 3D data representing the three-dimensional structure onto the two-dimensional space according to each field defined by the three-dimensional space to the pre-recorded three-dimensional space. In the third dimension, the data of each position of the pre-record is projected to the depth direction of the data of each position of the pre-recorded 3D data that can be projected to the first layer in each field. The depth parameter of the range is within the range of the depth direction of the preceding paragraph.
(52) The image processing device according to (51), wherein
The utility model further has: an extracting unit, which extracts a previous depth parameter included in the bit stream;
The pre-recorded three-dimensional projection unit is configured to project the data of each position previously included in the 3D data into the range of the depth direction indicated by the depth parameter before being extracted by the pre-recording unit.
(53) The image processing device according to (52), wherein
Further, the decoding unit further includes: decoding the encoded data of the pre-recorded 3D data included in the pre-recorded bit stream that has been projected onto the pre-recorded two-dimensional plane;
The pre-recorded three-dimensional projection unit is configured to decode the data of the pre-recorded data by the pre-recording unit, and project the data into the pre-recorded 3D data of the pre-recorded two-dimensional plane. The depth of the preceding depth direction indicated by the depth parameter is recorded before the extraction of the pre-extraction unit.
(54) The image processing device according to (53), wherein
The predecessor decoding unit decodes the positional information, the attribute information, and the coded data of each of the occupied maps of the 3D data of each layer that has been projected onto each of the pre-recorded two-dimensional planes.
(55) The image processing device according to (53) or (54), wherein
Further, the unpacking unit is obtained by decoding the preamble encoded data by the pre-recording unit, and the video frame that has been projected onto the pre-recorded 3D data of the pre-recorded two-dimensional plane is unpacked;
The pre-recorded three-dimensional projection unit is configured to unpack the video frame by the pre-decoding unit, and project the data into the pre-recorded 3D data of the pre-recorded 2D plane. It has been extracted by the pre-recording unit before the depth direction indicated by the depth parameter.
(56) The image processing device according to any one of (51) to (55), wherein
The pre-recorded 3D data is a point cloud.
(57) An image processing method for projecting data of each position included in a 3D data representing a 3-dimensional structure onto a 2-dimensional plane in accordance with each field defined by a 3-dimensional space to a pre-record 3 In the dimension space, the data of each position of the predecessor is projected into the pre-recorded 3-dimensional space, and the information set in each field of the pre-record is used to limit the pre-recorded 3D data that can be projected to the 1 layer. The depth parameter of the range of the depth direction is within the range of the depth direction indicated in the preceding paragraph.

100‧‧‧ coding device

111‧‧‧Potment Decomposition Department

112‧‧‧Packing Department

113‧‧‧Auxiliary Plaque Information Compression Department

114‧‧•Video Coding Department

115‧‧•Video Coding Department

116‧‧‧OMap Encoding Department

117‧‧‧Multiplexer

151‧‧‧ Normal direction estimation

152‧‧ Section initial setting

153‧‧ Section Update Department

154‧‧2D Dimensional Projector

155‧‧‧Pixel Distribution Analysis Department

200‧‧‧ decoding device

211‧‧ ‧ multiplexer

212‧‧‧Auxiliary plaque information decoding department

213‧‧‧Video Decoding Department

214‧‧‧Video Decoding Department

215‧‧‧OMap Decoding Department

216‧‧·Unpacking Department

217‧‧3D reconstruction department

251‧‧3D projection department

252‧‧‧Pixel Distribution Analysis Department

253‧‧‧ Reverse Segment Update Department

254‧‧‧Inverse Segmentation Initial Setting Department

255‧‧‧Inverse normal direction estimation

900‧‧‧ computer

901‧‧‧CPU

902‧‧‧ROM

903‧‧‧RAM

904‧‧ ‧ busbar

910‧‧‧Import interface

911‧‧‧ Input Department

912‧‧‧Output Department

913‧‧‧Memory Department

914‧‧‧Communication Department

915‧‧‧ drive machine

921‧‧‧Removable media

[Fig. 1] An explanatory diagram of an example of a point cloud.

[Fig. 2] An explanatory diagram of an example of an outline of a video basic mode.

Fig. 3 is a view showing the overall arrangement of the present technology described in each embodiment.

Fig. 4 is a block diagram showing a main configuration example of an encoding device.

Fig. 5 is an explanatory diagram of a main configuration example of a plaque decomposition unit.

Fig. 6 is a block diagram showing a main configuration example of a decoding device.

[Fig. 7] A block diagram of a main configuration example of the 3D reconstruction unit.

[Fig. 8] An illustration of an example of a previous two-dimensional projection.

[Fig. 9] An illustration of an example of a two-dimensional projection to which the present technique is applied.

[Fig. 10] A flowchart for explaining an example of the flow of the encoding process.

[Fig. 11] A flow chart for explaining an example of the flow of the plaque decomposition processing.

[Fig. 12] A flow chart for explaining an example of the flow of the two-dimensional projection processing.

[Fig. 13] A flowchart for explaining an example of the flow of the decoding process.

[Fig. 14] A flow chart for explaining an example of the flow of the point cloud reconstruction process.

[Fig. 15] A flow chart for explaining an example of the flow of the three-dimensional projection processing.

[Fig. 16] An illustration of an example of a previous two-dimensional projection.

[Fig. 17] An illustration of an example of a two-dimensional projection to which the present technique is applied.

[Fig. 18] A flow chart for explaining an example of the flow of the two-dimensional projection processing.

[Fig. 19] A flow chart for explaining an example of the flow of the three-dimensional projection processing.

[Fig. 20] A diagram showing an example of the appearance of the previous two-dimensional projection.

[Fig. 21] An illustration of an example of a two-dimensional projection to which the present technique is applied.

[Fig. 22] A flow chart for explaining an example of the flow of the two-dimensional projection processing.

[Fig. 23] A flow chart for explaining an example of the flow of the three-dimensional projection processing.

[Fig. 24] A block diagram of a main configuration example of a computer.

Claims (20)

An image processing device having: The two-dimensional projection unit projects data of each position included in the 3D data of the three-dimensional structure onto a two-dimensional plane of the complex layer. The image processing device of claim 1, wherein In the second-dimensional projection unit, the position of the 3D data from the projection surface is the position of each position in the depth direction, and is projected on the two-dimensional plane before the multi-layer layer. The image processing device of claim 2, wherein The pre-recorded two-dimensional projection unit generates the same number of layers as the maximum number of pieces of data of each position of the pre-recorded 3D data from the projection surface as viewed from the projection surface in the depth direction. The image processing device of claim 1, wherein The pre-recorded two-dimensional projection unit is provided at a position where the data of each position is not present before the two-dimensional plane, and the data indicating that each position of the previous position is a non-existent value. The image processing device of claim 4, wherein The value stated in the preceding paragraph is a fixed value that has been predetermined. The image processing device of claim 4, wherein The value set in the preceding paragraph is a value larger than the maximum value of the depth of the 3D data. The image processing device of claim 1, wherein The 2D projection section of the pre-recorded 3D data is used to map each field of the pre-recording to the 2D plane according to the position of each position in the 3D space. The data of each position in the range of the preceding depth direction indicated by the depth parameter indicated by the depth parameter in the depth direction of the data of each position of the pre-recorded 3D data which can be projected to the first layer is projected to the pre-recorded two-dimensional plane. The image processing device of claim 1, wherein Further, the encoding unit is configured to project the 3D data which has been projected by the pre-recorded two-dimensional projection unit to the pre-recorded two-dimensional plane. The image processing device of claim 8, wherein Further, the present invention further includes a bit stream generation unit that generates a bit stream including information indicating a number of layers of a pre-recorded two-dimensional plane on which the 3D data is projected by the two-dimensional projection unit, and The precoding unit encodes the encoded data obtained by encoding the 2D plane. The image processing device of claim 1, wherein The pre-recorded 3D data is a point cloud. An image processing method The data representing each of the positions included in the 3D data of the 3-dimensional structure is projected onto the 2-dimensional plane of the complex layer. An image processing device having: The three-dimensional projection unit projects data of each position of the 3D data on the two-dimensional plane that has been projected onto the number of layers indicated by the layer number information into a three-dimensional space. The image processing device of claim 12, wherein The 3D projection part of the pre-recorded 3D data is included in the data of each position before the 3D data, and the data indicating the position of each position in the previous record is the data other than the value of the non-existent value, and is projected to the pre-record 3D. space. The image processing device of claim 13, wherein The value stated in the preceding paragraph is a fixed value that has been predetermined. The image processing device of claim 13, wherein The value set in the preceding paragraph is a value larger than the maximum value of the depth of the 3D data. The image processing device of claim 12, wherein The 3D projection part of the pre-recorded 3D space is used to project each position of the previous 3D data into the 3D space in the pre-recorded 3D space. The data is projected into the pre-recorded 3-dimensional space, and the depth parameter of the range of the depth direction of the data of each position of the 3D data representing the 3-dimensional structure that can be projected to the first layer is limited. The front of the representation is in the depth direction. The image processing device of claim 12, wherein It also has a drawing unit for extracting the information of the number of layers in the bit stream which is included in the bit stream; The pre-recorded three-dimensional projection unit is configured to record all the pre-recorded positions of the pre-recorded 3D data that have been projected on the two-dimensional plane before the number of layers indicated by the number of layers before the extracted portion is extracted. The data is projected to the front 3D space. The image processing device of claim 17, wherein Further, the decoding unit further includes: decoding the encoded data of the pre-recorded 3D data included in the pre-recorded bit stream that has been projected onto the pre-recorded two-dimensional plane; The pre-recorded three-dimensional projection unit is configured to decode the pre-recorded data by the pre-decoding unit, and project the data onto all the pre-recorded positions of the pre-recorded 3D data on the two-dimensional plane. Pre-record 3D space. The image processing device of claim 12, wherein The pre-recorded 3D data is a point cloud. An image processing method The data of each position of the 3D material on the two-dimensional plane that has been projected onto the number of layers indicated by the layer number information is projected into the three-dimensional space.