JP2022080792A - Computer generated hologram generation device, method, and program - Google Patents

Abstract

To allow high-speed generation of CGH videos with small memory consumption.SOLUTION: A point group input unit 101 receives input of 3D point group data used for calculation of a CGH. The input 3D point group is classified (clustered) into some sets (clusters) by a point group clustering unit 102. A correspondence relationship discrimination unit 103 discriminates the correspondence relationship between points for each of corresponding clusters between adjacent or back-and-forth frames. A calculation method determination unit 104 determines a calculation method for light wave propagation related to the current frame as one of a "reuse" method or a "normal calculation" method in units of cluster based on the correspondence relationship. A light wave propagation calculation unit 105 calculates object light wave propagation of the current frame with the determined calculation method. An interference calculation unit 106 calculates the interference between a result of calculation of the object light wave propagation and reference light, and calculates interference fringes. A CGH output unit 107 outputs a CGH.SELECTED DRAWING: Figure 1

Description

The present invention relates to a computer synthetic hologram (CGH) generator, a method and a program, and more particularly to a computer synthetic hologram generator, a method and a program for generating a CGH moving image at high speed.

Holography is a stereoscopic display technology that records and reproduces light from an object (object light) based on the phenomenon of light interference and diffraction. In the holography technique, the wave of light emitted from an object and the reference light emitted from a light source such as a laser are made to interfere with each other, and the object light is recorded as an interference fringe. By shining light on these interference fringes, the light at the time of recording can be reproduced. Since it can faithfully reproduce the light emitted from an object, it is considered to be an ideal 3D display technology that satisfies all the physiological factors of human 3D perception.

Computer Generated Hologram (CGH) is a technology that calculates the propagation and interference of light waves required for hologram calculation as a light wave simulation inside the computer and outputs the interference fringes as electronic data such as images. Is. Compared to analog holograms shot using a photographic plate, it does not require a complicated optical system for shooting, and it can be easily animated by switching the CGH displayed on the LCD one after another. Because of the advantages of, it is expected to be applied to next-generation TVs and XR devices such as VR / AR.

On the other hand, as a problem of CGH, there is a problem that the processing time for calculation is enormous. In particular, as a CGH calculation method, the "point light source method" is famous, in which the object to be recorded is regarded as a set of many point light sources, and the propagation of light from each point light source is calculated and recorded. The calculation processing time is enormous because it is necessary to simulate the propagation of light waves from.

In addition, in order to obtain a sufficient field of view when reproducing holography, an SLM (spatial light modulator) with a fine pixel pitch on the order of 1 μm is required. For example, a liquid crystal display of 1 cm x 1 cm is realized on the order of 1 μm. The number of pixels required to do this is 10000 x 10000 pixels. That is, it is expected that a liquid crystal display having a resolution exceeding 8K will be used in the future. Considering that the calculation time of the point light source method is generally "the number of point light sources x the number of pixels of the hologram surface", it is difficult to perform real-time calculation by inputting a huge number of points, and the CGH generation processing speed is increased. Need to be done.

In the background where high-speed processing time is required, many algorithms for high-speed on the premise of generating moving images have been proposed in view of the advantage of CGH that easy animation can be performed. ..

Patent Document 1 discloses a technique of measuring the amount of change in the entire 3D scene and reusing the object light wave distribution of the previous frame when the amount of change in the entire scene is small. According to this technology, if there are few changes in the entire 3D scene, the process of generating continuous frames can be speeded up by reusing the light waves of the previous frame, rather than calculating each continuous frame independently. be able to.

Patent Document 2 discloses a technique for reducing the calculation time of the point light source method by calculating the movement vector of a 3D object and moving the light wave distribution on the hologram surface based on the movement vector.

Japanese Unexamined Patent Publication No. 2012-008207 Japanese Unexamined Patent Publication No. 10-222046

T. Ichikawa, T. Yoneyama, and Y. Sakamoto, "CGH calculation with the ray tracing method for the Fourier transform optical system," Opt. Express 21, 32019-32031 (2013). Takashi Nishitsuji, Tomoyoshi Shimobaba, Takashi Kakue, Nobuyuki Masuda, Tomoyoshi Ito, "Fast calculation of computer-generated hologram using circular symmetry of zone plate", Optics Express, 20, 27496-27502 (2012) R. Q. Charles, H. Su, M. Kaichun and L. J. Guibas, "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp. 77-85.

In Patent Document 1, when the depth of the point light source changes, it is necessary to subtract the light wave propagated from the depth before the change and add the light wave propagated from the depth after the change. Two light wave propagation calculations are required for the frame to be used. In other words, if 50% or more of the entire scene changes, it will take less time to recalculate the frame from scratch using the point light source method than to reuse it (however, the depth will change). This is not the case when a new point appears or when the point itself disappears).

From this point of view, in Patent Document 1, it is recommended to reuse the previous frame when the critical value of the change ratio in the scene is less than 0.5. That is, when the rate of change in the scene is 0.5 or more, it is difficult to speed up the generation by the technique of Patent Document 1.

In addition, high-speed calculation is realized by using a look-up table (LUT) for calculation processing, but memory consumption is high because it is necessary to prepare a different look-up table for each distance from the hologram surface to the point light source. There was a big problem.

Further, Patent Document 2 does not assume a case where a part of a 3D object cannot be seen due to shielding due to overlapping when 3D objects move, and in a case where such a shielding relationship occurs, efficient calculation is performed. I can't do it.

An object of the present invention is to provide a computer synthetic hologram generator, a method, and a program that solve the above technical problems and enable high-speed generation of CGH moving images with a small memory consumption.

In order to achieve the above object, the present invention calculates the light wave propagation for each point of the 3D point group to obtain the object light wave distribution on the hologram surface, and performs the interference calculation with the reference light wave to make a moving image of the computer composite hologram. The device for generating the above is characterized in that it has the following configurations.

(1) A means for clustering each point of the 3D point cloud for each frame, a means for determining the correspondence relationship of each point belonging to each corresponding cluster between the frames before and after, and a method for calculating the object light wave distribution of the current frame. It is provided with a means for determining either "reuse" or "normal calculation" for each cluster based on the correspondence, and a means for calculating the object light wave distribution of the current frame for each cluster by the determined calculation method. did.

(2) The means for clustering is to cluster each point in the 3D point cloud based on its position.

(3) The means for clustering is to cluster each point in the 3D point cloud based on its position and category.

(4) The means for determining the calculation method is to increase or decrease the frequency of determining the calculation method for each cluster on a frame-by-frame basis based on the category.

(5) Provide a means to estimate the calculation time of the object light wave distribution by each method of "reuse" and "normal calculation" for each cluster of the current frame, and decide on a calculation method with a shorter calculation time for each cluster. I made it.

(6) The means for determining the calculation method is to increase or decrease the frequency of determining the calculation method for each cluster based on the estimation result of the calculation time of the object light wave distribution by "reuse" and "normal calculation".

(7) The means for estimating the calculation time is to represent the calculation time by the number of calculation times of the light wave propagation required for the calculation of the object light wave distribution.

(8) The means for estimating the calculation time is a point with a different color at the position of the point that disappeared from the previous frame to the current frame when the calculation time is represented by the number of calculation times of the light wave propagation required for the calculation of the object light wave distribution. When is appearing, the number of calculations for light wave propagation related to the disappearance and appearance is counted as one.

(1) Since the calculation method of object light wave propagation is determined for each frame for each cluster, the calculation method of "reuse" is performed by collecting the parts where the amount of change in the scene is small even if the amount of change in the entire scene is large in the same cluster. If applied, it will be possible to reduce the calculation time for the entire scene.

(2) By clustering each point in the 3D point cloud based on its position, each scene can be clustered with high selectivity in the part where the amount of change is small and the part where the amount of change is large.

(3) By clustering each point in the 3D point cloud based on its position and category, it becomes possible to cluster each scene into a part with a small amount of change and a part with a large amount of change with higher selectivity.

(4) By increasing or decreasing the frequency of determining the calculation method for each cluster on a frame-by-frame basis based on the category, the time loss required for determining the calculation method can be suppressed while ensuring the appropriate determination of the calculation method. Become.

(5) If you decide on a calculation method that shortens the calculation time for each cluster, it will be possible to shorten the calculation time for the entire scene.

(6) If the frequency of determining the calculation method is increased or decreased for each cluster based on the estimation result of the calculation time of the object light wave distribution by reuse and normal calculation, the calculation method is secured while ensuring an appropriate determination of the calculation method. It will be possible to reduce the time loss required for making a decision.

(7) Since the calculation time of the object light wave distribution is represented by the number of calculations of the light wave propagation required for the calculation of the object light wave distribution, the calculation time can be easily estimated.

(8) If a point with a different color appears at the position of the point that disappeared from the previous frame to the current frame for each cluster, the number of calculations for light wave propagation related to the disappearance and appearance is counted as one, so it is calculated. You will be able to save time.

It is a functional block diagram which showed the structure of the main part of the computer synthetic hologram generation apparatus which concerns on 1st Embodiment of this invention. It is a figure for demonstrating the technique of obtaining a 3D point cloud for each small area by dividing a hologram surface into element holograms and performing ray tracing. It is a figure which showed the classification example of a 3D point cloud. It is a figure which showed the method of discriminating the correspondence relation of the point light source pi between frames. It is the figure which showed schematically the method of estimating the calculation time based on the correspondence relation of the point light source pi for each cluster. It is a figure which showed the output example of CGH. It is a functional block diagram which showed the structure of the main part of the computer synthetic hologram generation apparatus which concerns on 2nd Embodiment of this invention. It is a figure which showed the example which classifies 3D point cloud based on the position and category of each point light source. It is a figure which showed the example which associates each cluster between frames. It is a figure which showed schematically the method of estimating the calculation time. It is a functional block diagram which showed the structure of the main part of the computer synthetic hologram generation apparatus which concerns on 3rd Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing a configuration of a main part of a computer composite hologram (CGH) generator 1 according to the first embodiment of the present invention.

3D point cloud data used for CGH calculation is input to the point cloud input unit 101. The input 3D point cloud is classified (clustered) into several sets (clusters) by the point cloud clustering unit 102. The correspondence relationship determination unit 103 determines the correspondence relationship of each point for each cluster corresponding between adjacent or preceding and following frames.

The calculation method determination unit 104 determines the calculation method of the light wave propagation for the current frame in cluster units based on the correspondence. In the present embodiment, the "reuse" method that uses the calculation result of the object light wave propagation in the past frame, or the point light source method is used to re-use the light wave propagation at all points without using the calculation result in the past frame at all. Determined by one of the "normal calculation" methods of calculation.

The light wave propagation calculation unit 105 calculates the object light wave propagation of the current frame by the calculation method determined above. The interference calculation unit 106 calculates the interference between the calculation result of the object light wave propagation and the reference light, and calculates the interference fringes. The CGH output unit 107 outputs CGH (image data of interference fringes).

Such a CGH generator 1 can be configured by mounting an application (program) that realizes each function on a general-purpose computer or server equipped with a CPU, ROM, RAM, bus, interface, and the like. Alternatively, it can be configured as a dedicated machine or a single-purpose machine in which a part of the application is made into hardware or software.

Next, the functions of the above parts will be described in more detail. The point cloud input unit 101 inputs a 3D point cloud of a scene to be reproduced by holography into the device 1 in order to be handled by a computer. The 3D point cloud is input as a general-purpose format such as a PLY file. In the present embodiment, each point is described as having position information (X, Y, Z) and luminance information (R, G, B), but may further include additional information such as a normal vector.

In addition, as a pre-processing of the point cloud input unit 101, downsampling is performed by deleting the points quantized at the same position using Octree etc. with respect to the position of the input 3D point cloud from the viewpoint of reducing the calculation time. Alternatively, a point at a position hidden by a point in front and invisible when viewed from a specific viewpoint position may be removed in advance.

Further, a polygon model may be input instead of the 3D point cloud data. Non-Patent Document 1 discloses a 3D point cloud acquisition method based on the ray tracing method as a means for obtaining 3D point cloud data from a 3D polygon model for holography, and a polygon model is input based on such a method. It may be converted into a point cloud.

In Non-Patent Document 1, as shown in FIG. 2, the hologram surface is divided into small regions called element holograms, and ray tracing is performed from the center of the small regions to obtain different 3D point clouds for each small region. The technology is disclosed. As a result, when the viewpoint is moved during holography viewing, it is possible to realize a motion parallax in which different parts can be seen depending on the shielding relationship. Therefore, even in this embodiment, a different 3D point cloud is held for each element hologram, and the subsequent flow is an element. The processing may be performed independently for each hologram.

The point cloud clustering unit 102 classifies the input 3D point cloud into a plurality of clusters based on a predetermined index. In this embodiment, the 3D point cloud is classified based on its position (X, Y, Z), but it may be classified based on the information on the position and color (X, Y, Z, R, G, B). .. The classification result is cached in the cache memory 108 as a part of the cluster information 108a.

FIG. 3 is a diagram showing an example of classification of a 3D point cloud by the point cloud clustering unit 102. In this embodiment, each point light source pi constituting the 3D point cloud is located at its position (X, Y, Z). Based on this, it is classified into one of four clusters A to D. In the present embodiment, four clusters A to D are defined in absolute coordinates, and the description will be continued assuming that the positions of the clusters A to D do not change between frames and are fixed.

In the illustrated example, the point light sources p1 to p4 are classified into cluster A, the point light sources p5 and p6 are classified into cluster B, the point light sources p7 to p9 are classified into cluster C, and the point light sources p10 to p13 are classified into cluster D. .. In FIG. 3, each cluster is represented in two dimensions for the sake of simplicity, but in reality, clustering may be performed based on the three-dimensional position of XYZ.

As shown in FIG. 4, the correspondence determination unit 103 indicates that each point light source pi belonging to each cluster in the Nth frame (current frame) belongs to the corresponding cluster in the adjacent N-1th frame (previous frame). The correspondence with the point light source pi is determined based on its position and color.

In FIG. 4, the index (mainly color information) of each point light source pi is represented by hatching, and it is shown that the point light source pi represented by the same hatching has the same color. In this embodiment, the point light sources p1 to p4 of cluster A, the point light sources p5 of cluster B, and the point light sources p10, p12, and p13 of cluster D have the same position and color between adjacent frames. Is determined.

The calculation method determination unit 104 includes the calculation time estimation unit 104a, and determines the calculation method of the object light wave distribution u _N (x, y) applied to the current frame as either “reuse” or “normal calculation” for each cluster. do. In the present embodiment, the calculation time when "reuse" is adopted and the calculation time when "normal calculation" is adopted are estimated and compared, and the calculation method in which the calculation time is shorter is determined for each cluster. ..

The object light wave distribution u _N (x, y) on the hologram surface by I point light sources pi in 3D space is obtained by integrating the light wave distribution si (x, y) by each point light source pi by the following equation (1). The light wave distribution si (x, y) on the hologram surface obtained and propagated from each point light source pi is obtained by the light wave propagation calculation of the following equation (2).

Here, (x, y) indicates the pixel position on the hologram surface on which the light wave is propagated. Ai is the luminance value of the point light source pi, and in the case of the RGB color space, it has independent values for red, green, and blue. ri represents the distance between the point light source pi and the hologram plane (x, y). k represents the wave number calculated from the wavelength of light. Finally, the interference fringes are obtained by inserting the reference light into the object light wave distribution u _N (x, y) and performing the interference calculation.

In the "normal calculation", the light wave distribution si (x, y) by all the point light sources pi is calculated for each cluster based on the above equation (2), and then these are integrated based on the above equation (1) to obtain the hologram surface. Calculate the object light wave distribution u _N (x, y) above.

In "reuse", the object light wave distribution u _N-1 (x, y) calculated for each cluster in the previous frame and the light wave distribution si (x, y) based on the correspondence of each point light source pi in the corresponding cluster of the current frame. The object light wave distribution u _N (x, y) on the hologram surface is obtained by adding or subtracting y).

FIG. 5 is a diagram schematically showing a method in which the calculation time estimation unit 104a estimates the calculation time based on the correspondence of the point light sources pi for each cluster. The object light wave distribution u _{(N, A)} (x, y) from each point light source pi belonging to the cluster A in the Nth frame can be calculated by the following equation (3).

Object light wave distribution from cluster A to cluster D When u _{N, A} (x, y) to u _{N, D} (x, y) are calculated in the same way, the object light wave distribution from all point clouds to be finally obtained. Is calculated by the following equation (4).

Generally, in the calculation of the object light wave distribution of the above equation (1), the light wave propagation calculation of the equation (2) occupies most of the calculation processing time. Therefore, in the present embodiment, the number of calculations in the above equation (2) is counted, and this count number represents the calculation time of the object light wave distribution.

In FIG. 5, since the cluster A in the Nth frame contains 6 point light sources, the number of propagation calculations when the “normal calculation” is applied is 6 times. When "recalculation" is applied, the object light wave distribution u _{(N-1, A)} (x, y) of the corresponding cluster A in the N-1th frame is calculated by the following equation (5).

Focusing on cluster A, the point light sources p1, p2, p3, and p4 match in position and color from the N-1th frame to the Nth frame, and the two point light sources p14 and p15 increase. Therefore, the object light wave distribution u _{N, A} (x, y) of the cluster A in the Nth frame is N frame to the object light wave distribution u _{N-1, A} (x, y) of the cluster A in the N-1th frame. It can be calculated by the following equation (6) by adding the light wave distributions s14 and s15 of the point light sources p14 and p15 that have increased by the eye.

In other words, if the object light wave distribution u _{N, A} (x, y) on the hologram surface of the N-1th frame is saved in the calculation process and used in the light wave distribution calculation of the Nth frame, it will be in the Nth frame. Propagation calculation can be suppressed to 2 times. Therefore, the calculation method applied to cluster A is determined to be "reuse".

Focusing on cluster B, since the current frame contains two point light sources p5 and p17, the number of calculation of light wave propagation when "normal calculation" is applied is twice. On the other hand, when "recalculation" is applied, the point light source p6 disappears from the N-1th frame to the Nth frame, but a new point light source p17 with a different color appears at the same position. ..

In this case, a total of two light wave carrying calculations, one for reducing the light wave distribution of the point light source p6 and the other for adding the light wave distribution of the point light source p17, are required. Since the light wave from the light source p17 is expressed by the following equation (7) according to the above equation (2), the light wave distribution from the point light source p6 before the change is expressed by the following equation (8).

Here, since r6 = r17 on the assumption that the position does not change and only the color changes, the object light wave distribution d _p17－p6 (x, y) on the hologram surface to be added when the color changes is given by the following equation (x, y). It is represented by 9).

In other words, although extra calculation is required for one subtraction (A ₁₇ -A ₆ ), only the color changes because the distance calculation with a large processing time and the calculation of exp (-jkr_i) can be suppressed to one time. It can be seen that the propagation calculation can be counted as one for the light source pi. As a result, the calculation method applied to cluster B is also determined to be "reuse".

Focusing on cluster D, the current frame contains three point light sources p10, p12, and p13, so the number of propagation calculations when "normal calculation" is applied is three. On the other hand, since the point light sources p10, p12, and p13 match from the N-1th frame to the Nth frame and the point light source p11 disappears, the number of propagation calculations by "reuse" is one. .. As a result, the calculation method applied to cluster D is also determined to be "reuse".

On the other hand, focusing on cluster C, since the current frame contains one point light source p16, the number of propagation calculations when "normal calculation" is applied is one. On the other hand, the point light sources p7, p9, and p8 disappear from the N-1th frame to the Nth frame, and a new point light source p11 with a different color appears at the position of the disappeared point light source p8. In this case, the object light wave distribution u _{N, C} (x, y) can be calculated by the following equation (10), so the number of propagation calculations is three. As a result, the calculation method applied to cluster C is determined to be "normal calculation".

As the above equation (2), a different equation may be used as long as it expresses propagation, for example, by using an approximation based on the Fresnel approximation. Further, a method for high-speed calculation of the zone plate as in Non-Patent Document 2 may be used instead of calculating the above equation (2).

Further, in the above embodiment, it has been described that the object light wave distribution due to the point where the position does not change between adjacent frames is reused, but if the position change is small, the quality deterioration can be small even if it is reused. , A threshold value may be set for the amount of change in position, and position changes below the threshold value may be regarded as the same position. This also applies to the color information, and if the colors do not change significantly, they may be regarded as the same color.

The light wave propagation calculation unit 105 calculates the object light wave distribution by the calculation method determined by the calculation method determination unit 104. In the example of FIG. 5, "reuse" is applied to cluster A, cluster B, and cluster D, and "normal calculation" is applied to cluster C, so the following calculation is performed.

From the above calculation results, the number of propagation calculations when "normal calculation" is applied to all clusters is 12, whereas according to this embodiment, it is twice in cluster A and once in cluster B. The number of calculations for light wave propagation can be reduced to less than half, with cluster C once and cluster D once, for a total of five times.

In this way, in this embodiment, the calculation method can be determined for each cluster by dividing the 3D point cloud into a plurality of clusters for each frame, so that even if the amount of change in the entire scene is large, the amount of change in the scene is small. By applying "reuse" to the same cluster, it is possible to reduce the calculation time for the entire frame.

The light wave propagation calculation unit 105 uses the object light wave distributions u _{N, A} (x, y), u _{N, B} (x, y), u _{N, C} (x, y) of each cluster for reuse in the next frame. ), U _{N, D} (x, y) are cached in memory 108 (108b).

It is conceivable to retain the si (x, y) itself calculated by the above equation (2), but in addition to the high resolution of the computer composite hologram exceeding 8K, si (x, y) ) Is required for the number of points × the number of pixels, so that it is difficult to implement it from the viewpoint of memory consumption, especially when it is applied to data consisting of a large-scale point group.

Further, while the point group may be divided into many clusters to reduce the calculation time of light wave propagation, in the present embodiment, the light waves from the point groups constituting each cluster are holographic planes (x, y, respectively). ) Since it is cached as the object light wave distribution above, it is possible that the memory consumption increases in proportion to the number of clusters. Therefore, it is desirable to limit the number of clusters according to the memory resources.

The interference calculation unit 106 performs interference calculation by inserting a reference light wave as a simulation on a computer into the object light wave distribution u _N (x, y) on the hologram surface calculated by the light wave propagation calculation unit 105. For example, the complex amplitude distribution R (x, y) of the light wave when the reference light wave emitted from the point light source is propagated on the hologram surface is expressed by the following equation (16).

Here, Ro is the intensity of the reference light, and r is the distance from the position of the point light source serving as the reference light to the position (x, y) on the hologram surface. The equation of the reference light is not limited to the above equation (16), and different reference light waves such as parallel light may be used. The equation showing the interference between the reference light wave and the object light wave is expressed by the following equation (17).

Here, I (x, y) is the luminance distribution of CGH (interference fringes), and CGH is output by the CGH output unit 107 in the subsequent stage based on the value of I (x, y).

The CGH output unit 107 outputs the interference fringes calculated by the interference calculation unit 106 as image data. An example of an actual CGH is shown in Fig. 6. The format of the image output here is arbitrary, but in general, the image data is often indicated by a certain range of luminance values such as 0-255. On the other hand, since the interference fringes calculated by the interference calculation unit 106 are often not normalized to this range, all the pixels of the interference fringes generated by the interference calculation unit 106 are searched, and the maximum value is 255 and the minimum value is the minimum value. It may have a function to perform normalization such that is set to 0.

Further, regarding the output CGH, each CGH may be output as image data one by one, or the format may be such that an image is displayed in a window at any time.

FIG. 7 is a functional block diagram showing the configuration of the main part of the computer composite hologram generation device 1 according to the second embodiment of the present invention, and the same reference numerals as those described above represent the same or equivalent parts. In this embodiment, each point in the 3D point cloud is classified into a plurality of categories such as "people" and "buildings", and when clustering each point, the category is taken into consideration in addition to the position. There is a feature.

The point cloud acquired by scanning the real world as a 3D point cloud using a known depth sensor or the like is input to the point cloud input unit 101. The point cloud category classification unit 111 categorizes the input 3D point cloud.

In recent years, with the development of deep learning, many methods have been proposed in which Semantic Segmentation can be performed on a 3D point cloud based on prior learning results, for example, as in Non-Patent Document 3. In this embodiment, by using such a method, the input 3D point cloud is classified into categories such as people, cars, and roads. The classification method is not limited to the deep learning-based method, and the category may be classified by another method such as determining the category with reference to the color information of the point cloud.

Further, when a point cloud of a person is recognized at a distant place as shown in FIG. 8, the instances may be divided into segments such as "person 1A" and "person 1B" for segmentation. At this time, the method of dividing the instances may be performed according to an existing clustering method such as the shortest distance method.

The point cloud cluster correspondence calculation unit 112 associates each cluster in the Nth frame with each cluster in the N-1th frame. In the case where there are multiple instances in the same category, the position of the center of gravity of each cluster (person 1A to 1B, person 2A to 2C) is obtained for each frame, and each cluster in the Nth frame is the closest cluster to the N-1th frame. Correspond to.

In this embodiment, each point light source pi in the N-1th frame is a total of three clusters (instances) 1A and 1B classified into the category "person" and one cluster 1A classified into the category "building". It is classified as one of the clusters.

Each point light source pi in the Nth frame is one of three clusters 2A, 2B, 2C (instances) classified in the category "person" and one cluster 2A classified in the category "building", for a total of four clusters. It is classified into.

The point cloud cluster correspondence calculation unit 112 maps each cluster in the Nth frame to the nearest cluster in the N-1th frame based on the position of the center of gravity and the category. In the present embodiment, as shown in FIG. 9, the description is continued assuming that the person 2A is associated with the person 1A, the persons 2B and 2C are associated with the person 1B, and the building 2A is associated with the building 1A.

The correspondence determination unit 103 compares the position and color of each point belonging to the corresponding cluster between adjacent or preceding and following frames, and for each point light source pi of the N frame, any point and position of the N-1th frame and Determine the correspondence between colors that match or not for each cluster.

Similar to the first embodiment, the calculation method determination unit 104 calculates the optical path propagation distribution for each cluster based on the calculation time of the object light wave distribution estimated by the calculation time estimation unit 104a for each cluster. Decide on one of "reuse". Also in this embodiment, the calculation time is represented by the number of calculations of light wave propagation.

In this embodiment, when each point light source pi of the person cluster 2A in the Nth frame is compared with each point of the corresponding person cluster 1A in the N-1th frame, as shown in FIG. 10, four points are located and Both colors match, two new dots appear, and two dots disappear. Therefore, the number of calculations for light wave propagation is 4 for "reuse" and 6 for "normal calculation", and the calculation method is determined to be "reuse".

Comparing each point light source pi of the person cluster 2B in the Nth frame with each point light source pi of the corresponding person cluster 1B in the N-1th frame, there is no point where neither the position nor the color matches. Therefore, the calculation method applied to the person cluster 2B is determined to be "normal calculation".

Comparing each point light source pi of the person cluster 2C in the Nth frame with each point light source pi of the corresponding person cluster 1B in the N-1th frame, there is no point light source pi whose position and color match. Therefore, the calculation method applied to the person cluster 2C is also determined to be "normal calculation".

Comparing each point light source pi of the building cluster 2A in the Nth frame with each point light source pi of the corresponding building cluster 1A in the N-1th frame, the four point light sources pi match both in position and color, and one. Only the positions of the point light source pi match, and one point light source pi disappears. Therefore, the number of calculation of light wave propagation is 2 times for "reuse" and 5 times for "normal calculation", and the calculation method applied to the building cluster 2A is determined to be "reuse".

It should be noted that each cluster in the Nth frame does not necessarily have to be associated with any cluster in the N-1th frame, and the calculation method of the cluster that is not associated is determined as "normal calculation". For example, if a point cloud of the same category does not exist at a short distance in the N-1th frame, it is determined that the cluster is a point cloud newly entered from outside the field of view in the Nth frame, and the association is performed. It does not have to be.

According to this embodiment, each point of the 3D point cloud is clustered in consideration of the category in addition to the position, so that the points of the same object, in other words, the presence or absence of movement and the amount of movement are the same in each cluster. You will be able to classify points. As a result, not only can the number of clusters to which "reuse" is applied be increased, but also the number of points that require calculation of lightwave propagation in the clusters to which "reuse" is applied can be reduced. It is possible to significantly reduce the calculation time for the entire frame.

FIG. 11 is a functional block diagram showing the configuration of the main part of the computer composite hologram generator 1 according to the third embodiment of the present invention, and the same reference numerals as those described above represent the same or equivalent parts. Omit.

In each of the above embodiments, the calculation method of the object light wave distribution is determined for each frame. However, when the 3D point cloud is clustered in consideration of the category as in the second embodiment, it is continuous depending on the cluster. It is empirically recognized that the rate of change of the point cloud does not change significantly between the corresponding clusters of the frames.

For example, a cluster that changed only 10% of the 3D point cloud during the transition from the N-2nd frame to the N-1th frame is 90 of the 3D point cloud from the next N-1th frame to the Nth frame. Sudden changes such as% changes are unlikely to occur in view of the correlation between frames.

Therefore, in this embodiment, in order to reduce the time loss required for calculating the calculation time, the estimation frequency T (> 1) is defined based on the above empirical rule, and the calculation time is estimated once in each T frame. For other frames, the judgment result of the previous frame is adopted as it is.

That is, if the calculation method is determined to be "reuse" based on the correspondence with each point of the N-2 frame for a cluster with the N-1 frame, the Nth frame is calculated from the N-1th frame. When doing so, it is uniformly decided to "reuse" without estimating the calculation time. As a result, the processing of the calculation time estimation unit 104a can be suppressed to one T frame for each cluster, so that the time loss required for the calculation time estimation can be reduced.

The estimated frequency T may be a fixed value, or a different T may be dynamically determined for each cluster. In the present embodiment, the calculation method determination unit 104 includes the estimation frequency determination unit 104b, and determines the estimation frequency T for each cluster.

The estimation frequency determination unit 104b determines the estimation frequency T based on the category determined by the all-point cloud category classification unit 111. For example, the estimated frequency T applied to a category such as a person cluster where the point cloud is likely to change is relatively smaller than the estimated frequency T applied to a category such as a building cluster where the point cloud is unlikely to change. Set to.

Alternatively, the last estimated time of "normal calculation" and "reuse" calculated for each cluster may be compared, and if the time difference is small, T may be set to a value that is relatively small compared to the case where the time difference is large. good. This is based on the fact that if the time difference between each calculation method is small, the superiority or inferiority of each calculation method tends to be reversed in the next frame, so it is desirable to increase the estimation frequency in order to prevent estimation errors from occurring. ..

101 ... Point cloud input unit, 102 ... Point cloud clustering unit, 103 ... Correspondence relationship determination unit, 104 ... Calculation method determination unit, 104a ... Calculation time estimation unit, 104b ... Estimated frequency calculation unit, 105 ... Light wave propagation calculation unit, 106 ... Interference calculation unit, 107 ... CGH output unit, 108 ... Cache memory, 111 ... Point cloud category classification unit, 112 ... Point cloud cluster support calculation unit

Claims (15)

In a device that calculates the light wave propagation for each point in a 3D point group to obtain the object light wave distribution on the hologram surface, and performs interference calculation with the reference light wave to generate a moving image of a computer-synthesized hologram.
A means of clustering each point in a 3D point cloud frame by frame,
A means for determining the correspondence between the points belonging to each corresponding cluster between the preceding and following frames, and
Based on the above correspondence, the calculation method of the object light wave distribution of the current frame is determined for each cluster as either a method of reusing the calculation result of the object light wave distribution of the previous frame or a method of normal calculation without reuse. Means and
A computer-synthesized hologram generator comprising a means for calculating the object light wave distribution of the current frame for each cluster by the above-determined calculation method. The computer-synthesized hologram generator according to claim 1, wherein the clustering means clusters each point in a 3D point cloud based on its position. Further provided with means for categorizing each point in the 3D point cloud,
The computer-synthesized hologram generator according to claim 1, wherein the clustering means clusters each point in a 3D point cloud based on its position and category. The computer-synthesized hologram generator according to claim 2, wherein the means for determining the calculation method is to increase or decrease the frequency of determining the calculation method for each cluster on a frame-by-frame basis based on the category. Each cluster of the current frame is provided with a means for estimating the calculation time of the object light wave distribution by the reuse and normal calculation methods based on the correspondence.
The computer-synthesized hologram generator according to any one of claims 1 to 4, wherein the means for determining the calculation method is determined by the calculation method having a shorter calculation time for each cluster. The means for determining the calculation method according to claim 5, wherein the frequency for determining the calculation method is increased or decreased for each cluster based on the estimation result of the calculation time of the object light wave distribution by the reuse and the normal calculation. Computer synthetic hologram generator. The computer-synthesized hologram generator according to claim 5 or 6, wherein the means for estimating the calculation time is represented by the number of calculations of light wave propagation required for calculating the object light wave distribution. The means for estimating the calculation time counts the number of points appearing or disappearing from the previous frame to the current frame for each cluster based on the position of each point, and the count number represents the number of calculations for light wave propagation. The computer-synthesized hologram generator according to claim 7. The means for estimating the calculation time is to count the number of points appearing or disappearing from the previous frame to the current frame for each cluster based on the position and color of each point, and calculate the number of times of light wave propagation based on the count number. The computer-synthesized hologram generator according to claim 7, wherein the computer is represented. The means for estimating the calculation time is that if a point with a different color appears at the position of the point that disappeared from the previous frame to the current frame for each cluster, the number of calculation times of the light wave propagation related to the disappearance and appearance is 1. The computer-synthesized hologram generator according to claim 9, wherein the number of times is counted. The means for calculating the object light wave distribution of each cluster of the current frame whose calculation method is determined to be reused is changed from the previous frame to the current frame with respect to the calculation result of the object light wave distribution of the corresponding cluster of the previous frame. The computer-synthesized hologram generator according to any one of claims 8 to 10, wherein the calculation is performed by performing a process of adding or subtracting the light wave distributions of the points appearing or disappearing over the counter. In a method in which a computer calculates light wave propagation for each point in a 3D point group to obtain an object light wave distribution on a hologram surface, and performs interference calculation with a reference light wave to generate a moving image of a computer-synthesized hologram.
Cluster each point in the 3D point cloud frame by frame,
Determine the correspondence of each point belonging to each corresponding cluster between the frames before and after,
Based on the above correspondence, the calculation method of the object light wave distribution of the current frame is determined for each cluster as either a method of reusing the calculation result of the object light wave distribution of the previous frame or a method of normal calculation without reuse. ,
A computer-synthesized hologram generation method characterized in that the object light wave distribution of the current frame is calculated for each cluster by the above-determined calculation method. Each point in the above 3D point cloud is categorized and
The computer-synthesized hologram generation method according to claim 12, wherein each point in the 3D point cloud is clustered based on its position and category. In a program that calculates the light wave propagation for each point in a 3D point group to obtain the object light wave distribution on the hologram surface, and performs interference calculation with the reference light wave to generate a moving image of a computer-synthesized hologram.
The procedure for clustering each point in the 3D point cloud frame by frame,
The procedure for determining the correspondence between the points belonging to each corresponding cluster between the preceding and following frames, and the procedure for determining the correspondence between the points belonging to each cluster.
Based on the above correspondence, the calculation method of the object light wave distribution of the current frame is determined for each cluster as either a method of reusing the calculation result of the object light wave distribution of the previous frame or a method of normal calculation without reuse. Procedure and
A computer-synthesized hologram generation program comprising a procedure for calculating an object light wave distribution of a current frame for each cluster by the above-determined calculation method. Further including a procedure for categorizing each point in the above 3D point cloud,
The computer-synthesized hologram generation program according to claim 14, wherein the clustering procedure comprises clustering each point in a 3D point cloud based on its position and category.

Images