JP2015176485A - Three-dimensional model characteristic extraction method and three-dimensional model annotation system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a characteristic extraction method for searching for a three-dimensional model by comparing characteristic values of a border pixel pattern of the three-dimensional model being searched (with those of a known three-dimensional model), and to provide an annotation system.SOLUTION: Characteristic values of a boundary pixel pattern is obtained by sorting differences between an observed pixel and adjacent pixels to positive and negative for each border pixel, and extracting a histogram of appearance frequency of each. Comparison of the characteristic values of the border pixel pattern involves assessing dissimilarity, in terms of sum of maximum values of appearance frequency histograms for the border pixels extracted from each depth buffer image, between a known three-dimensional model and a three-dimensional model being searched using all combinations of the multiple viewpoints as an index.

Description

The present invention relates to a method for searching a database of a three-dimensional object model and extracting features, and an annotation system. Annotation of a 3D model is a technology that automatically assigns a plurality of keywords representing the shape of a 3D model.

  In the machine parts manufacturing industry represented by the automobile manufacturing industry, it is a common practice to manufacture a shape model of a three-dimensional object using a three-dimensional CAD / CAM system. Also, in the construction industry, it is widely used to create 3D building models, room models, furniture and tree models in order to simulate the exterior and interior of buildings and the surrounding landscape using CG before construction. Has been done. Furthermore, 3D CG technology is now indispensable for creating animation, movies, commercial films, and the like.

  However, in such an application field, when an elaborate three-dimensional model is first created, much more labor and time are required than two-dimensional drawing. Therefore, in these application fields, a 3D model that has been input and created once by an auxiliary means such as a human or a 3D scanner is stored in the 3D object model database, and a similar 3D object shape model is created. In this case, it is considered that a significant cost reduction can be achieved by reusing an object model having a similar shape.

  The technique described in Patent Document 1 is a method of dividing a polygon into a certain size and defining and searching for feature quantities from the areas of the divided polygons and normal vectors. Since this is an old method that does not perform the normal (normalization), the search depends on the position, orientation, and size, and there is a problem that the search cannot be performed even if the shapes are similar.

  The technique described in Patent Document 2 is a three-dimensional model similarity search method using a two-dimensional image as a search key, and calculates feature amounts based on projection images or cross-sectional images obtained from a plurality of two-dimensional images. Search based on. The feature amount in this patent is a histogram obtained by quantizing each color information value such as RGB, HSV, Lab, etc., which can be calculated for the texture actually attached to the object surface, a shape histogram obtained by quantizing the edge derivative, These include the volume, surface area, vertex distribution, and polygon distribution histogram of a three-dimensional object. For this reason, there is a problem that even if the shapes are similar, those having different colors cannot be searched. Also, as in Patent Document 1, there is a problem such as dependency on projection parameters (cross-section generation parameters) in that there is no facing (normalization) processing and spectrum in the frequency space is not used, and high search accuracy is high. It is thought that it does not come out.

  Non-Patent Document 1 proposes a plurality of methods, but the best search accuracy is the Hybrid Descriptor. This is a two-dimensional Fourier transform of a depth buffer image in six directions to obtain a power spectrum, which is used as a feature value, and a two-dimensional Fourier transform of a silhouette image in three directions to obtain a power spectrum and characterize it. Amount. Then, a vector is emitted from the center until it touches the object surface in a radial manner, and this vector is used as a feature amount. It is a composite feature that combines spectral feature and geometric feature, and achieves high performance. From the system name DESIRE created by Mr. Vranic, it is often called a DESIRE feature (or a DSR472 feature from the published program name). However, the spectrum and geometric features are quite different, complicating index management, and the problem that the shape inside the object cannot be captured from depth buffer images and silhouette images remains. .

  In Non-Patent Document 2, a 256 × 256 silhouette image is generated from 100 or more different directions, 10 Fourier coefficients and 35 Zernike moment coefficients are obtained, and these are used as feature quantities. At the time of similarity calculation, parts of similar silhouette images are compared. This feature amount is referred to as a Light Field Descriptor (or LFD feature amount for short). Since rendering is performed from more than 100 directions, the creation time of the feature amount is extremely necessary. Since the shape is approximated only from the silhouette, there are problems such as being unable to find a fine shape and not being able to capture the features inside the object.

  Non-Patent Document 3 describes a content-based image search method using BIC (Border / Interior pixel Classification). The BIC classifies each pixel as a boundary pixel (Border) if the pixel value differs from any of the four neighboring pixels, and as an internal pixel (Interior) if the pixel value of all of the four neighboring pixels is the same. , And create a histogram based on the appearance frequency of the pixel values. Depth buffer is not used because it is not intended for 3D model search.

JP 2002-41530 A JP 2004-164503 A

D. Vranic, 3D Model Retrieval. Ph.D. thesis, University of Leipzig, 2004. D-Y. Chen, X-P. Tian, Y-T. Shen, M. Ouhyoung, On Visual Similarity Based 3D Model Retrieval.Computer Graphics Forum (2003), 223-232. Stehling, M.A.Nascimento, and A.X. Falcao, A Compact and Efficient Image Retrieval Approach Based on Border / Interior Pixel Classification, "In Proc. Of CIKM'02, pp.102-109, 2002.

  In the feature quantity of the 3D model, like the above-mentioned prior art Non-Patent Documents 1 and 2, the technique of analyzing a 2D image generated from multiple viewpoints from a 3D model had excellent search performance. . In particular, the analysis of the depth buffer image in which the unevenness of the shape of the 3D model is expressed by the shading of the image is suitable for capturing the shape of the 3D model. Due to the nature of the depth buffer image, a feature of the shape appears at the boundary portion where the change in shading is large. However, in Non-Patent Document 1 that handles a depth buffer image, the depth buffer image is analyzed without particularly enhancing the boundary portion.

  On the other hand, in the similarity search of images, there are those that analyze by classifying them into boundary pixels and internal pixels, as in BIC of Non-Patent Document 3. However, the BIC only considers whether or not the pixel of interest and the pixel values in the vicinity of 4 are the same, and does not consider the magnitude of the pixel value. From the viewpoint of capturing irregularities in the shape, the magnitude of the pixel value is important information.

There are also problems with the solution to the arbitrary rotation of the 3D model. In Non-Patent Document 1, the principal axis is obtained by principal component analysis or singular value decomposition, and the rotation of the three-dimensional model is uniquely determined. This method has a problem that even if the shape is similar, the main axis changes due to noise such as a defect. Non-Patent Document 2 relaxes the arbitraryness of rotation by comparing feature quantities of two-dimensional images generated from a large number of viewpoints. When comparing feature amounts, the Euclidean distance of the feature amounts is calculated for the image pair, and the minimum value is set as the distance between the three-dimensional models. As a result, the comparison is made from a single viewpoint, and the arbitraryness of rotation can be solved, but the shapes of the three-dimensional models cannot be accurately compared.

The present invention has been made in order to solve the problems of the prior art described above.

  The feature extraction method of the three-dimensional model according to claim 1 is a method of comparing pixel values of a target pixel and a neighboring pixel in each depth buffer image with respect to depth buffer images at a plurality of viewpoints of a known three-dimensional model. This is a feature extraction method that searches for a 3D model by creating a database of the feature values of the boundary pixel pattern in advance using a different pixel value as the boundary pixel, and comparing the feature values of the boundary pixel pattern of the 3D model to be searched. The boundary pixel pattern feature amount is obtained by classifying the difference between the target pixel and the neighboring pixel in the boundary pixel as positive and negative and extracting a histogram based on the appearance frequency of each of the boundary pixel patterns. Is compared with the sum of the maximum values of histograms based on the appearance frequency of the boundary pixels extracted for each depth buffer image. All combinations of points as an index, and characterized in that to determine the degree of difference between the search target of the three-dimensional model.

  The 3D model search method according to claim 2, wherein the depth buffer image represents unevenness of the 3D model with shading, and the boundary pixel includes the target pixel and the top, bottom, left, and right of the target pixel. What is extracted is a pixel having a different pixel value when comparing pixel values with four neighboring pixels adjacent to.

  The feature extraction method for a three-dimensional model according to claim 3, wherein the depth buffer image is obtained by performing singular value decomposition on a known three-dimensional model and performing posture normalization, and then a predetermined pixel size at a predetermined number of viewpoints. It is generated by the above.

  The feature extraction method of the three-dimensional model according to claim 4, wherein the dissimilarity is calculated by calculating a matrix of sum totals of maximum values of histograms based on appearance frequencies of boundary pixels of each depth buffer image at the number of viewpoints. On the other hand, the difference is the difference in the minimum sum of the combinations obtained by the Hungarian method.

  The three-dimensional model annotation system according to claim 5, wherein the pixel values of the pixel of interest and the neighboring pixels in each depth buffer image are compared for depth buffer images at a plurality of viewpoints of the known three-dimensional model. An annotation system that searches for a 3D model by comparing the feature values of a boundary pixel pattern stored in advance in a database and the feature values of a boundary pixel pattern of a 3D model to be searched, with different values as boundary pixels. A means for generating a depth buffer image at a plurality of viewpoints from a three-dimensional model to be searched; a means for detecting a boundary pixel by comparing pixel values of a target pixel and neighboring pixels for each depth buffer image; For each boundary pixel, the difference between the target pixel and neighboring pixels is classified as positive and negative, and A difference between a known three-dimensional model and a three-dimensional model to be searched, with the total sum of the histograms extracted for each depth buffer image as an index of all combinations of a plurality of viewpoints. And a means for determining.

By the above means, a highly accurate and high-speed 3D model feature extraction method and 3D model annotation system can be realized.

It is explanatory drawing which showed preparation of the histogram by the positive / negative of the difference of an attention pixel and 4 vicinity. It is a flowchart of the three-dimensional model feature extraction method and annotation system according to the present invention. It is explanatory drawing of attitude | position normalization (step 1). It is explanatory drawing of the production | generation (step 2) of a depth buffer image. It is explanatory drawing of the process (step 3) which classify | categorizes into the boundary pixel with respect to a depth buffer image, and an internal pixel. It is explanatory drawing of the process (step 4) of calculating the difference of each pixel of 4 vicinity of an attention pixel, and this attention pixel. It is explanatory drawing of the process (step 5 and step 6) which produces the boundary pattern histogram (BPH) of four types of histograms for every depth buffer image. It is explanatory drawing of the annotation process (step 7) by the k-neighbor method which used BPH as the feature-value.

Embodiments of the present invention will be described with reference to the drawings. The embodiment of the present invention is configured on a computer and executed by causing hardware such as a CPU and a memory to function.

  In the technology of the present invention, a feature value of a three-dimensional model is obtained by analyzing a depth buffer image generated from a plurality of viewpoints from a three-dimensional model (hereinafter also referred to as a three-dimensional shape model or a three-dimensional object). When the depth buffer image is analyzed, a pixel whose pixel value differs from any of the four neighboring pixels is classified as a boundary pixel, and the other pixels are classified as internal pixels. Since the depth buffer image expresses unevenness of the shape with shading, in the technology of the present invention, unlike the BIC (see Non-Patent Document 3), only the boundary pixel in which the shading change appears is the analysis target. Further, in order to capture the unevenness of the shape appearing in the magnitude of the pixel value, the difference between the pixel of interest and each of the four neighboring pixels is divided into positive and negative, and a histogram based on the appearance frequency of the pixel value is created for each.

  As described above, the invention technology can capture the unevenness of the shape of the three-dimensional model for which the background technology has not been sufficiently captured.

  First, posture normalization of the three-dimensional model is performed by Point SVD. In the 3D model, the size, position, and rotation are arbitrary depending on the creator and software. Since these affect the feature quantity of the extracted three-dimensional model, it is necessary to normalize in advance. PointSVD generates random points on the surface of the three-dimensional model, and solves the arbitraryness of position by translating to the origin of the three-dimensional space using the average of these point groups as the center of gravity of the three-dimensional model. Also, the principal axis of the three-dimensional model is obtained by performing singular value decomposition on the sample points. By rotating the main axis along the x-axis, y-axis, and z-axis of the three-dimensional space, the arbitrary rotation is solved. Furthermore, normalization is performed by dividing each vertex coordinate by the maximum distance from the origin to solve the arbitraryness of the size.

Next, a depth buffer image of light and shade L level (for example, 32 levels) is generated with a size of M × M (for example, 256 × 256) pixels from a plurality of viewpoints N _ν (for example, 38 viewpoints with each vertex of geodesic sphere as a viewpoint). To do. In each depth buffer image, a pixel whose pixel value is different from any of the target pixel and any of the four neighbors is detected as a boundary pixel. Then, as shown in FIG. 1, in the boundary pixel, the difference between the target pixel and the four neighbors (Top, Right, Bottom, Left) is classified into positive and negative (Positive, Negative), and a histogram based on the appearance frequency of the pixel value is obtained for each. create. Therefore, the size of the histogram calculated for each depth buffer image is 4 × 2 × L bins. Each histogram is normalized by the sum of the elements.

  FIG. 2 shows a flow of the 3D model feature extraction method and annotation system according to the present invention (hereinafter also referred to as a flow). It consists of seven steps, and each step is explained according to the processing procedure. A large number of three-dimensional shape models are stored in the database, and seven steps are repeated for each shape model.

<Step 1>
Posture normalization is applied to each three-dimensional shape model in the database to eliminate the arbitraryness of “position”, “rotation”, and “size”. The posture-normalized three-dimensional shape model is arranged so that the center of gravity is at the center of a sphere having a radius of 1 (FIG. 3).

<Step 2>
A depth buffer image is generated from each of multiple viewpoints (for example, 38 viewpoints). The posture-normalized sphere is approximated by a triangular patch having 38 vertices, and projected onto a plane perpendicular to the vector from each vertex toward the center of the sphere to generate a depth buffer image (depth buffer image) (FIG. 4). .

<Step 3>
For each depth buffer image, all pixels are classified into boundary pixels (Border) and internal pixels (Interior) (FIG. 5).

<Step 4>
The difference between the target pixel and each pixel value in the vicinity of the target pixel is calculated (FIG. 6).

<Step 5>
Four types of boundary pattern histograms (Border Pixel Pattern Histogram: BPPH) are created for each depth buffer image in accordance with the difference value in Step 4 (FIG. 7).

<Step 6>
The feature quantity obtained by applying the BPPH feature quantity obtained in step 5 to all the depth buffers from the plurality of viewpoints applied in step 2 is used as the shape feature quantity of the three-dimensional shape model and used as an index in the search. To do. Finally, normalization is performed so that the total frequency of the histogram becomes 1.0 (FIG. 7).

<Step 7>
Annotate with the k-neighbor method. Here, as shown in FIG. 8, for example, {vehicle, bicycle} represents the annotation of the three-dimensional shape model with these two labels as training data. In FIG. 8, for an unknown three-dimensional shape model at the center of an arc, k = 3 and a union of labels in the vicinity of 3 is generated as an output label.

When calculating the degree of similarity between 3D models, the 3D model to be a query (search question) is converted into a feature quantity in steps 1 to 6 above, and the distance between the feature quantity and the feature quantity in the database is calculated in step 7. The order of similar shapes is calculated by the above and sorted in ascending order of distance. In this embodiment, this distance calculation is obtained using the Histogram Intersection distance shown in (Equation 1).

  The numerical parameter in the example of the present invention is merely an example of the embodiment, and the flow does not depend on the numerical parameter.

  Further, the step 7 can be applied to a search device and a classification device by changing to the search and the classification instead of the automatic annotation.

  In order to confirm the effectiveness of the 3D model annotation system based on BPH features, we compared it with the conventional method.

First, the dissimilarity is calculated for all combinations of the boundary pixel histograms of a plurality of viewpoints _Nν , and a dissimilarity matrix having a size of _Nν × _Nν is calculated. Then, the difference of the minimum sum of combinations obtained by applying the Hungarian method to the difference matrix is set as the difference of the three-dimensional model. As a result, it is possible to realize the solution of the arbitrary rotation of the 3D model and the shape comparison of the 3D model from a plurality of viewpoints.

  In this experiment, a three-dimensional object of SHREC'12 Generic 3D Dataset and Princeton Shape Benchmark (PSB) was used as test data. SHREC'12 Generic 3D Dataset and PSB are essentially data sets for shape search of a three-dimensional object. However, since a data set for the purpose of annotation of a three-dimensional object has not been proposed so far, a plurality of labels are manually attached to these three-dimensional objects. For example, in the case of a three-dimensional object in the shape of a house, labeling is performed such as building, house, manmade, and architecture. In this experiment, a total of 120 kinds of labels were applied to 1,200 three-dimensional objects of SHREC'12 Generic 3D Dataset. In PSB, 188 types of labeling were performed on 1,814 three-dimensional objects. These data sets are composed of various three-dimensional objects, and general search accuracy can be evaluated.

  These three-dimensional objects were annotated with respect to the respective data using the Leave-one-out method. In this method, for SHREC'12 Generic 3D Dataset, 1,199 three-dimensional objects are used as training data, and the remaining one three-dimensional object is annotated.

  For the label estimation algorithm, a k-nearest neighbor classifier (k-NN) was used. k-NN compares the test data and training data on the feature space, and outputs the class that occupies the largest number among the classes to which training data in the k- vicinity of the test data belongs as the identification result. It is a classifier.

  When k-NN is used for annotation, it is as shown in FIG. A plurality of labels, not classes, are assigned to the training data, and training data labels in the k- vicinity of the test data are output as annotation results. Although k-NN is a simple algorithm, it shows excellent performance in automatic image annotation. This time, the nearest neighbor method was used with k = 1.

  As an evaluation scale, similar to the annotation of an image, Recall, which is the recall for the label, Precision, which is a precision, and F-Measure, which is a harmonic average. It can be said that the higher the value of any evaluation scale, the better the keyword estimation performance.

  As a comparison method, Light Field Descriptor (LFD), DESIRE Descriptor (DESIRE), Border / Interior Pixel Classification (BIC), and Scientific Harmonics Descriptor (HD) described in the related research were used. Since BIC is a feature extraction method of a two-dimensional image, processing was performed on a depth buffer image in the same manner as BPH.

  Table 1 summarizes the precision, recall, and F-value of each feature in SGD. Inventive technology has the best annotation performance on all evaluation scales. Although the relevance rate is higher than that of the prior art, the reproducibility generally having a trade-off relationship with the relevance rate is higher than that of the prior art. This indicates that the inventive technique has both accuracy and completeness in the annotation of the three-dimensional model.

Claims (5)

Features of boundary pixel patterns for depth buffer images at multiple viewpoints of a known three-dimensional model, with pixel values that differ when the pixel values of the target pixel and neighboring pixels in each depth buffer image are compared as boundary pixels A feature extraction method for searching a three-dimensional model by creating a database in advance and comparing feature quantities of boundary pixel patterns of a three-dimensional model to be searched,
The feature amount of the boundary pixel pattern is obtained by classifying the difference between the target pixel and the neighboring pixel in the boundary pixel as positive and negative and extracting a histogram based on the appearance frequency of each pixel
The comparison of the feature values of the boundary pixel pattern is performed by using the sum of the maximum values of the histogram based on the appearance frequency of the boundary pixels extracted for each depth buffer image as an index with all combinations of a plurality of viewpoints as an index. A feature extraction method for a three-dimensional model, characterized by determining a degree of difference.   The depth buffer image is a representation of the unevenness of the three-dimensional model in shades, and the boundary pixel is a comparison of pixel values of the pixel of interest and four neighboring pixels adjacent to the pixel of interest in the upper, lower, left, and right directions. 2. The method of extracting features of a three-dimensional model according to claim 1, wherein the ones having different pixel values are extracted.   3. The depth buffer image according to claim 1, wherein the depth buffer image is generated with a predetermined pixel size at a predetermined number of viewpoints after performing singular value decomposition and normalizing a known three-dimensional model. Feature extraction method for 3D models.   The dissimilarity is calculated by calculating a matrix of the sum of the maximum values of histograms based on the appearance frequency of boundary pixels in each depth buffer image at the number of viewpoints, and the dissimilarity of the minimum sum of combinations obtained by the Hungarian method for the matrix The method for extracting a feature of a three-dimensional model according to any one of claims 1 to 3. The depth buffer images in a plurality of viewpoints of a known three-dimensional model are preliminarily stored in a database with the pixel values of the target pixel and the neighboring pixels in each depth buffer image being different from each other as boundary pixels. An annotation system that searches a 3D model by comparing a feature quantity of a boundary pixel pattern and a feature quantity of a boundary pixel pattern of a 3D model to be searched,
Means for generating a depth buffer image in a plurality of viewpoints from a three-dimensional model to be searched;
Means for comparing the pixel values of the target pixel and neighboring pixels for each depth buffer image to detect a boundary pixel;
A means for classifying the difference between the target pixel and the neighboring pixel into positive and negative for each detected boundary pixel, and extracting a histogram based on the appearance frequency;
Means for determining the degree of difference between a known three-dimensional model and a three-dimensional model to be searched with the sum of the maximum values of the histogram extracted for each depth buffer image as an index of all combinations of a plurality of viewpoints; An annotation system for 3D models.

Images