JP2022070002A - Feature quantity extraction device, feature quantity extraction method, and program - Google Patents

Abstract

To provide a feature quantity extraction device that enables high-speed and high-accuracy similar image retrieval by extracting a single feature quantity that reflects the shape of a three-dimensional image.SOLUTION: An arithmetic unit 11 generates a projection image in which information indicating the distance between each point of a three-dimensional point group and a center point is projected as depth information of each point, with respect to a spherical surface surrounding three-dimensional point group data centered on the center of gravity of the three-dimensional point group data included in an object image. The arithmetic unit 11 extracts, as a feature quantity, a spherical depth image obtained by expanding the projection image on a two-dimensional plane.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a feature amount extraction device, a feature amount extraction method and a program.

In recent years, a three-dimensional similar image search for searching a similar image similar to a three-dimensional image specified by a user has attracted attention. For example, in fields such as the manufacturing industry, when designing a product, it is possible to improve the efficiency of product design and its estimation by searching for a drawing similar to the design drawing.

In the three-dimensional similar image search, a feature amount representing the feature of the three-dimensional image is extracted from the three-dimensional image, and the similar image is searched based on the feature amount (see Patent Document 1). As a method for extracting features from a three-dimensional image, the D2 (Shape Distribution: distance between two points) method, the SHD (Spherical Harmonics Descriptor: harmonic transform) method, the LFD (Light Field Descriptor: multidirectional shooting) method, and The MFSD (Multi-Fourier Spectra Descriptor) method is known.

Japanese Unexamined Patent Publication No. 2019-091138

The D2 method extracts features by making a histogram of the distribution such as the distance between two points in a three-dimensional image. In the D2 method, a single feature amount can be easily extracted, so that the search speed for similar image search is high. However, since the shape of the three-dimensional image is not reflected in the feature amount, there is a problem that the detection accuracy is low.

In the SHD method, a plurality of spherical shells are generated in a three-dimensional image, and features are extracted based on spherical harmonics corresponding to each spherical shell. In the SHD method, since the feature amount reflecting the shape of the three-dimensional image can be extracted, the detection accuracy is relatively high. However, since it is necessary to calculate the feature amount for each spherical shell, there is a problem that a plurality of feature amounts are required, the feature amount is difficult to handle, and the search speed is slow.

The LFD method extracts a feature amount based on a plurality of two-dimensional images obtained by capturing a three-dimensional image from various directions, and the SHD method is a method of extracting a feature amount of a three-dimensional image in addition to a plurality of feature amounts obtained by the LFD method. It extracts features that combine contours, shadows, and depth. With these methods, the feature amount reflecting the shape of the three-dimensional image can be extracted, so that the detection accuracy is high. However, there is a problem that the number of features is very large and the search speed is very slow.

An object of the present disclosure is to provide a feature amount extraction device, a feature amount extraction method, and a program that enable high-speed and highly accurate similar image search by extracting a single feature amount that reflects the shape of a three-dimensional image. To do.

The feature amount extraction device according to one aspect of the present disclosure is a feature amount extraction device that extracts feature amounts of a three-dimensional image including three-dimensional point group data, and the center point is the center of gravity of the three-dimensional point group data. A projected image is generated in which information indicating the distance between each point of the three-dimensional point group and the center point is projected as depth information of each point on a spherical surface surrounding the three-dimensional point group data, and the projected image is displayed as 2. It has a calculation unit that extracts an image developed on a three-dimensional plane as the feature amount.

According to the present invention, it is possible to extract a single feature amount that reflects the shape of a three-dimensional image, and it is possible to search for similar images at high speed and with high accuracy.

It is a block diagram which shows the feature amount extraction system of one Embodiment of this disclosure. It is a flowchart for demonstrating an example of operation of a server 1. It is a flowchart for demonstrating an example of rotation processing. It is a flowchart for demonstrating an example of interpolation processing. It is a figure which shows an example of interpolation of the point by interpolation processing. It is a flowchart for demonstrating an example of projection processing. It is a flowchart for demonstrating an example of expansion processing. It is a figure for demonstrating an example of an expansion process.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

FIG. 1 is a block diagram showing a feature amount extraction system according to an embodiment of the present disclosure. The feature amount extraction system shown in FIG. 1 includes a server 1 and a user terminal 2. The server 1 and the user terminal 2 are connected to each other so as to be able to communicate with each other. The server 1 and the user terminal 2 may be connected to each other via a communication network such as the Internet.

The server 1 is a feature amount extraction device that extracts the feature amount of the three-dimensional image according to the instruction from the user terminal 2. The user terminal 2 is a terminal device operated by a user who uses the feature amount extraction system, and transmits various instructions to the server 1. The server 1 and the user terminal 2 include, for example, a processor such as a CPU (Central Processing Unit), a memory, an auxiliary storage device, an input device, an output device, and a communication device (none of which is shown) such as a network card. It can be realized with the same hardware configuration as a general computer device.

The server 1 has a calculation unit 11, a processing unit 12, a storage unit 13, and a transmission / reception unit 14.

The calculation unit 11 performs a calculation process of extracting a spherical depth image as a feature amount indicating the characteristics of the target image from the target image which is a three-dimensional image to be processed. The processing unit 12 mainly performs processing other than calculation processing such as acquisition of the target image and storage of the spherical depth image generated by the calculation unit 11. The arithmetic unit 11 and the processing unit 12 may be realized by the processor reading a program recorded in the memory and executing the read program. Further, at least a part of the functions of the arithmetic unit 11 and the processing unit 12 may be realized by one or more hardware circuits (for example, FPGA (Field-Programmable Gate Array) or ASIC (Application Specific Integrated Circuit)).

The storage unit 13 is realized by, for example, a memory and an auxiliary storage device, and stores a program, a target image, a spherical depth image, and the like that define the operation of the calculation unit 11 and the processing unit 12. The transmission / reception unit 14 is realized by, for example, a communication device, and communicates with the user terminal 2.

FIG. 2 is a flowchart for explaining an example of the operation of the server 1.

First, when the transmission / reception unit 14 receives the feature amount extraction instruction instructing the extraction of the feature amount of the target image from the user terminal 2, the processing unit 12 acquires the target image according to the feature amount extraction instruction and calculates the target image. It is passed to the unit 11 (step S101). In the present embodiment, the target image is a three-dimensional image including the three-dimensional point cloud data, and more specifically, it is polygon data formed by polygons having each point of the three-dimensional point cloud data as an apex. The target image may be transmitted from the user terminal 2 together with the feature amount extraction instruction, or may be stored in advance in the storage unit 13.

The calculation unit 11 performs principal component analysis on the three-dimensional point cloud data included in the target image, identifies the principal component of the three-dimensional point cloud data, and rotates the target image based on the principal component. A process (see FIG. 3) is performed (step S102).

The calculation unit 11 sets a spherical surface that surrounds the target image that has undergone rotation processing (step S103). In the present embodiment, the spherical surface surrounding the target image is a spherical surface of an circumscribed sphere circumscribing the 3D point cloud data with the center of gravity of the 3D point cloud data included in the target image as the center point.

The calculation unit 11 performs interpolation processing (see FIGS. 4 and 5) in which points are interpolated on each side of the polygon forming the rotation-processed target image and added to the three-dimensional point cloud data (step S104). ..

The calculation unit 11 generates a projection image in which depth information indicating the depth in the target image of each point (including each interpolated point) of the three-dimensional point group data in the target image is projected onto the spherical surface of the circumscribing sphere. The process (see FIG. 6) is performed (step S105).

The calculation unit 11 extracts an image obtained by expanding the spherical surface of the circumscribed sphere on which the depth information is projected onto a two-dimensional plane as a spherical depth image which is a feature quantity representing the characteristics of the target image (see FIGS. 7 and 8). ) (Step S106).

Then, the processing unit 12 stores the spherical depth image generated by the calculation unit 11 in the storage unit 13 (step S107), and ends the processing. The processing unit 12 may transmit information to the effect that the feature amount has been extracted as response information to the feature amount extraction instruction to the user terminal 2 via the transmission / reception unit 14.

FIG. 3 is a flowchart for explaining an example of the rotation process of step S102 of FIG.

In the rotation process, first, the calculation unit 11 performs principal component analysis on the three-dimensional point group data in the target image, and as the main components of the three-dimensional point group data, the first principal component, the second principal component, and the second principal component. The third principal component is specified (step S201).

The calculation unit 11 sets the first principal component and the third principal component as reference axes, respectively (step S202).

The calculation unit 11 rotates the target image so that the reference axes face each predetermined direction (step S203), and ends the rotation process. In the present embodiment, the calculation unit 11 rotates the target image so that the first principal component faces the x-axis and the third principal component faces the y-axis.

FIG. 4 is a flowchart for explaining an example of the interpolation process of step S104 of FIG.

In the interpolation process, the arithmetic unit 11 determines the interpolation interval for interpolating the points based on the radius of the circumscribed sphere (step S301). For example, the arithmetic unit 11 reduces the interpolation interval as the radius of the circumscribed sphere increases.

The calculation unit 11 calculates the length of each side of each polygon of the target image (step S302). The calculation unit 11 determines the number of points to be interpolated for each side of each polygon based on the length of the side and the interpolation interval (step S303). For example, the arithmetic unit 11 determines the quotient of (side length) / (interpolation distance) as the number.

The calculation unit 11 interpolates on each side of each polygon the number of points determined for that side as points of the three-dimensional point cloud data, and adds the points to the three-dimensional point cloud data (step). S304), the interpolation process is terminated.

FIG. 5 is a diagram showing an example of interpolation of points by interpolation processing. Specifically, FIG. 5A is a diagram showing an example of a target image before the interpolation processing, and FIG. 5B is a diagram showing an example of the target image after the interpolation processing.

Before the interpolation process, as shown in FIG. 5A, the point 22 of the three-dimensional point cloud group data exists only at the apex of each polygon 21 of the target image. On the other hand, after the interpolation processing, as shown in FIG. 5B, not only the point 22 at the apex of each polygon 21 but also the interpolated point 23 exists on each side of each polygon 21. Therefore, the number of points in the target image is increased as compared with that before the interpolation processing.

FIG. 6 is a flowchart for explaining an example of the projection process of step S105 of FIG.

In the projection process, the calculation unit 11 calculates depth information indicating the distance between the point and the center point of the circumscribed sphere as the depth for each point in the target image (step S401).

The calculation unit 11 standardizes the depth information of each point according to a predetermined rule (step S402). Here, the calculation unit 11 normalizes the depth information so that the depth of the point corresponding to the center point of the circumscribed sphere is 0 and the depth of the point corresponding to the surface of the circumscribed sphere is 255.

The calculation unit 11 projects the depth information of each point on the surface of the circumscribed sphere as shading information, and maps the shading information (depth information) to the surface of the circumscribed sphere to generate a projected image (step S403). Here, the calculation unit 11 projects the depth information of each point at the intersection where the straight line connecting the point and the center point of the circumscribed sphere intersects the circumscribed sphere.

Then, the calculation unit 11 determines whether or not there is a point on the surface of the circumscribed sphere on which a plurality of depth information is projected. When there is a point where a plurality of depth information is projected, the calculation unit 11 selects one of the plurality of depth information and deletes the other depth information, so that any one of the plurality of depth information can be obtained. It is projected to generate a projected image (step S404), and the projection process is completed. Here, the calculation unit 11 selects the depth information of the outermost point among the plurality of depth information. The calculation unit 11 may select the depth information of the innermost point among the plurality of depth information. Further, when there is no point where a plurality of depth information is projected, the calculation unit 11 ends the projection process as it is.

FIG. 7 is a flowchart for explaining an example of the expansion process of step S106 of FIG.

In the expansion process, the calculation unit 11 converts the coordinates of each point on the sphere of the circumscribed sphere into polar coordinates (step S501). Here, the coordinates of each point before conversion are represented by Cartesian coordinates (x, y, z), and the first principal component of the three-dimensional point group data faces the x-axis direction by the rotation process (see FIG. 3). The third main component is oriented in the y-axis direction. In this case, the arithmetic unit 11 converts the coordinates of each point into polar coordinates (r, θ, φ) using the following equation 1.

Then, the arithmetic unit 11 converts the polar coordinates of each point of the spherical surface into two-dimensional coordinates (x, y) to expand each point of the spherical surface on the xy plane and generate a spherical depth image (step S502). End the expansion process. Here, the arithmetic unit 11 converts the polar coordinates of each point into two-dimensional coordinates (x, y) using the following equation 2.

FIG. 8 is a diagram for explaining an example of the expansion process. Specifically, FIG. 8A is a diagram showing an example of an image before the expansion process, and FIG. 8B is a diagram showing an example of the image after the expansion process.

As shown in FIG. 8, the projected image (FIG. 8A) in which the shading information is projected as the depth information on the surface of the circumscribed sphere by the expansion process is a two-dimensional grayscale image (FIG. 8 (FIG. 8). b) is converted to).

It is possible to roughly reproduce the projected image by performing the inverse transformation of the transformations shown in Equations 1 and 2 on the spherical depth image which is a feature quantity, and further, the projection is back-projected by the projection process. This makes it possible to roughly reproduce the original target image from the projected image. This indicates that the spherical depth image generally retains features such as the shape of the original target image, and the feature amount loss is small.

As described above, according to the present embodiment, the calculation unit 11 has a three-dimensional point group with respect to a spherical surface surrounding the three-dimensional point group data centered on the center of gravity of the three-dimensional point group data included in the target image. A projected image is generated by projecting information indicating the distance between each point and the center point as depth information of each point. The calculation unit 11 extracts a spherical depth image obtained by expanding the projected image into a two-dimensional plane as a feature amount. Therefore, since it is possible to extract a single spherical depth image that reflects the shape of the target image as a feature amount, it is possible to search for similar images at high speed and with high accuracy.

Further, in the present embodiment, the calculation unit 11 performs principal component analysis on the 3D point group, identifies the principal component of the 3D point group data, and rotates the 3D image based on the principal component. Generate a projected image in this state. Therefore, since it is possible to extract the feature amount by aligning the directions of the target images, it is possible to search for similar images with high accuracy.

Further, in the present embodiment, the calculation unit 11 projects the depth information of each point at the intersection where the straight line connecting the point and the center point of the spherical surface intersects the spherical surface. Therefore, it is possible to extract a feature amount that more accurately reflects the shape of the three-dimensional image.

Further, in the present embodiment, when a plurality of depth information is projected on a point on the spherical surface, the calculation unit 11 projects the depth information of the outermost point among the plurality of depth information. In this case, since it is possible to extract a feature amount that reflects the outer shape of the three-dimensional image, it is possible to search for similar images with high accuracy.

Further, in the present embodiment, the calculation unit 11 interpolates points on each side of each polygon of the target image, adds the interpolated points to the three-dimensional point cloud data, and projects the depth information. Therefore, it is possible to increase the amount of information to be reflected in the feature amount, and it is possible to search for similar images with high accuracy.

Further, in the present embodiment, the spherical surface is the spherical surface of the circumscribed sphere of the three-dimensional point cloud data. Therefore, it is possible to appropriately project the depth of each point of the three-dimensional point cloud data.

The embodiments of the present disclosure described above are examples for the purpose of explaining the present disclosure, and the scope of the present disclosure is not intended to be limited only to those embodiments. One of ordinary skill in the art can implement the present disclosure in various other embodiments without departing from the scope of the present disclosure.

For example, the server 1 may have a function of performing a similar image search for a three-dimensional image using the extracted feature amount.

1: Server, 2: User terminal, 11: Arithmetic unit, 12: Processing unit, 13: Storage unit, 14: Transmission / reception unit

Claims (8)

It is a feature amount extraction device that extracts the feature amount of a 3D image including 3D point cloud data.
Depth information of each point indicating the distance between each point of the three-dimensional point group and the center point with respect to the spherical surface surrounding the three-dimensional point group data with the center of gravity of the three-dimensional point group data as the center point. A feature amount extraction device having a calculation unit for generating a projected image projected as a feature amount and extracting an image obtained by expanding the projected image on a two-dimensional plane as the feature amount. The calculation unit performs principal component analysis on the three-dimensional point group, identifies the main component of the three-dimensional point group data, and rotates the three-dimensional image based on the main component. The feature amount extraction device according to claim 1, which generates a projected image. The feature amount extraction device according to claim 1, wherein the calculation unit projects depth information of each point onto an intersection where a straight line connecting the point and the center point intersects the spherical surface. The feature amount extraction device according to claim 3, wherein the calculation unit projects the depth information of the outermost point among the plurality of depth information when a plurality of the depth information is projected on the intersection. The three-dimensional image is formed by a plurality of polygons having each point of the three-dimensional point cloud data as a vertex.
The feature amount extraction device according to claim 1, wherein the calculation unit interpolates points on each side of each polygon and adds the interpolated points to the three-dimensional point cloud data. The feature amount extraction device according to claim 1, wherein the spherical surface is a spherical surface of a circumscribed sphere of the three-dimensional point cloud data. It is a feature amount extraction method by a feature amount extraction device that extracts a feature amount of a 3D image including 3D point cloud data.
Depth information of each point indicating the distance between each point of the three-dimensional point group and the center point with respect to the spherical surface surrounding the three-dimensional point group data with the center of gravity of the three-dimensional point group data as the center point. Generates a projected image projected as
A feature amount extraction method for extracting an image obtained by expanding the projected image on a two-dimensional plane as the feature amount. It is a program for extracting the features of a 3D image including 3D point cloud data.
Depth information of each point indicating the distance between each point of the three-dimensional point group and the center point with respect to the spherical surface surrounding the three-dimensional point group data with the center of gravity of the three-dimensional point group data as the center point. And the procedure to generate a projected image projected as
A program for causing a computer to execute a procedure of extracting an image obtained by expanding the projected image on a two-dimensional plane as the feature amount.

Images