JP2020135637A - Attitude estimation device, learning device, method, and program - Google Patents

Abstract

To provide an attitude estimation device capable of accurately estimating the posture of the subject captured in the image based on only a learning data collectable at low cost.SOLUTION: An attitude estimation device 150 estimates a relative posture of a given subject captured in a first image with respect to a reference posture. The attitude estimation device 150 has an estimation unit 64 that estimates a relative posture by performing a predetermined process on the first image. The predetermined process is based on a part of the process optimized to geometrically transform the first image into a second image in which the predetermined subject in the reference posture is captured.SELECTED DRAWING: Figure 6

Description

The present invention relates to a posture estimation device, a learning device, a method, and a program, and more particularly to a posture estimation device, a learning device, a method, and a program for estimating the posture of a subject captured in an image.

The progress of image recognition technology is remarkable. In addition to the personal authentication technology using faces and fingerprints that has been put into practical use and the further sophistication of person / face recognition / tracking technology, character recognition / object recognition technology that operates on general-purpose devices such as smartphones has also become widespread. It is also being used for new services such as O2O services and navigation services.

One of the applications that has been attracting attention recently is the use of robots as "eyes". In the manufacturing industry, the introduction of factory automation by robots equipped with image recognition functions has been promoted for a long time, but with the recent progress of robot AI technology, transportation / inventory management, transportation / transportation at retail / logistics sites It is expected to expand into fields that require a higher level of recognition.

A typical image recognition technique is a technique of giving a name label (hereinafter, simply referred to as a label) to the image itself or the subject reflected in the image itself. For example, if an apple appears in a certain image, the desirable operation of the image recognition technology when this is input is to output the label "apple" or to output the label "apple", or to capture the apple in the image, that is, the pixel. Assign the label "apple" to the set.

On the other hand, in the image recognition technology that can be provided in the robot as described above, it is often insufficient to output the label in this way. For example, as an example of using a robot in a retailer, let's consider a situation in which a product on an article shelf is grasped and transported and then moved to another product shelf. At this time, in order to complete the task, the robot roughly (1) identifies the product to be transferred from the various products on the goods shelf, and (2) grasps the product. , (3) Move and transport to the target product shelf, and (4) Arrange in the desired layout. However, the image recognition technology can recognize the goods shelves, goods, and goods shelves, and in addition to this, especially in relation to (2) and (4), the posture of the object for grasping and displaying the object. That is, it is necessary to be able to accurately recognize the position, angle, and size. The typical image recognition technology as described above does not have a function of estimating the posture of such an object, and a technique for estimating the posture of the object is required separately.

In view of such problems, an image recognition technique for estimating the posture of an object has been invented and disclosed.

Non-Patent Document 1 discloses a posture estimation method based on Scale Invariant Feature Transform (SIFT) features and generalized Hough transform. In this posture estimation method, by analyzing the brightness value of the image, a large number of partial regions having a remarkable change in brightness are extracted, and the change in brightness of each of these partial regions is invariant with respect to size and rotation. Expressed as a feature vector (SIFT feature). Next, the Euclidean distances between SIFT features are measured for the partial regions included in two different images of the same object, and the partial regions between different images having small values are obtained as correspondence candidates. Furthermore, if it is a partial region obtained from the same object, the partial region that is a candidate for correspondence is based on the assumption that the change in posture between the corresponding partial regions on the object is consistent regardless of the shooting viewpoint. Find the "deviation" of the position, posture, and size between them. Assuming that the set of corresponding subregions obtained from the same object constitutes a histogram of the deviations, assuming that the deviations are consistent, they may be concentrated in a very small number of bins. is assumed. Therefore, only the correspondence candidates distributed in the frequently occurring bins are regarded as truly effective correspondences, and the deviation corresponding to the bins is estimated as the posture parameter. This posture represents the relative posture change of the objects shown in the two compared images, and if the posture of one object is used as the reference posture, the posture of the other can be known.

Patent Document 1 discloses an improved technique of Non-Patent Document 1. It is the same as finding the correspondence candidates of the partial area based on the SIFT feature and calculating the deviation of these positions, postures, and sizes to judge the suitability of the correspondence, but when evaluating the deviation, the three-dimensional rotation angle is used. thinking. As a result, it is possible to estimate the posture more finely than the technique of Non-Patent Document 1.

The technique disclosed in Non-Patent Document 2 is different from the above two documents, and discloses a technique based on machine learning using a convolutional neural network (CNN). A large amount of learning image data (for example, 120,000 sheets) in which the vertices of eight rectangular parallelepipeds surrounding the object appearing on the image are recorded is prepared in advance, and an image is input based on this data. At that time, the posture estimation is made possible by learning the CNN that predicts the vertices of the eight rectangular parallelepipeds surrounding the object in the image based on the error back propagation method.

JP-A-2015-95156

David G. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints”, International Journal of Computer Vision, pp.91-110, 2004. Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, and Stan Birchfield. Deep Object Pose Optimization for Semantic Robotic Grasping of Household Objects, Proceedings of Conference on Robot Learning, pp. 306-316, 2018.

From a broad perspective, the existing invention estimates the relative orientation by analyzing the correspondence between two different images of the same object (Non-Patent Document 1, Patent Document 1). Alternatively, is it possible to construct a posture estimator by learning CNN based on a large amount of learning data that records information representing the posture of an image and an object reflected therein (such as the vertices of an eight rectangular parallelepiped) (Non-Patent Document 2). ) Is adopted.

However, the techniques described in Patent Document 1 and Non-Patent Document 1 have a problem that they must always hold an image of an object of the type to be estimated in advance. Assuming that there are 10 types of objects whose postures are to be estimated, the image for which the posture is to be estimated is compared with at least 10 images of the above 10 types of objects taken in the reference postures. It is necessary to estimate the relative posture. In this principle, in order to estimate the posture of an object, it cannot be applied unless an image showing the object as a reference posture is held, that is, it cannot be applied to an unknown object. was there.

Further, since the posture estimation is performed by the relative posture change between the two images, there is also a problem that the time required for the posture estimation process increases according to the number of types of objects.

Further, the method described in Non-Patent Document 2 has a problem that a large amount of learning data in which information representing a posture is recorded is required for learning CNN. Information representing the posture (in the case of Non-Patent Document 2, the vertices of eight rectangular parallelepipeds) must have fineness and accuracy at the pixel level of the image, and it is costly to give such information. It takes. In addition, since CNN generally requires a large amount of learning data, it is expected that the cost of constructing the learning data will be extremely high.

That is, until now, no image recognition technique has been invented that can estimate the posture of an unknown object with high accuracy based only on learning data that can be collected at low cost.

The present invention has been made to solve the above problems, and is capable of accurately estimating the posture of a subject captured in an image based only on learning data that can be collected at low cost. It is intended to provide estimation devices, methods, and programs.

Another object of the present invention is to provide a learning device capable of performing learning for accurately estimating the posture of a subject captured in an image based only on learning data that can be collected at low cost.

In order to achieve the above object, the posture estimation device according to the first aspect is a posture estimation device that estimates the relative posture of a predetermined subject captured in the first image with respect to the reference posture. It has an estimation unit that estimates the relative posture by performing a predetermined process on the first image, and the predetermined process is a second image in which the predetermined subject in the reference posture is imaged. It is based on some of the processes optimized to geometrically transform the first image into an image.

The learning device according to the second aspect is a learning device that learns a process of estimating a posture of a predetermined subject captured in the first image relative to a reference posture, and is imaged in the first image. A second image in which the predetermined subject in the reference posture is imaged by estimating the relative posture of the predetermined subject with respect to the reference posture and using the estimated relative posture. Learning to estimate the relative posture and learn the geometric transformation so that the estimation unit that geometrically transforms the first image, the result of the geometric transformation by the estimation unit, and the second image match. It is composed of parts and parts.

The posture estimation method according to the third aspect is a posture estimation method for estimating the relative posture of a predetermined subject captured in the first image with respect to the reference posture, and the estimation unit uses the first image. The predetermined process includes estimating the relative posture by performing a predetermined process on the subject, and the predetermined process is performed on the second image in which the predetermined subject in the reference posture is captured. It is based on some of the processing optimized to geometrically transform the image in.

The program according to the fourth aspect is a program for estimating the relative posture of a predetermined subject captured in the first image with respect to the reference posture, and is used by a computer with respect to the first image. It is a program for executing the estimation of the relative posture by performing a predetermined process, and the predetermined process is performed on a second image in which the predetermined subject in the reference posture is captured. It is based on some of the processes optimized to geometrically transform the first image.

According to the posture estimation device, method, and program according to one aspect of the present invention, the posture of the subject captured in the image can be accurately estimated based only on the learning data that can be collected at low cost. The effect is obtained.

Further, according to the learning device according to one aspect of the present invention, learning for accurately estimating the posture of the subject captured in the image can be performed based only on the learning data that can be collected at low cost. , The effect can be obtained.

It is a block diagram which shows the structure of the image converter with a geometric transformation layer. It is a block diagram which shows the structure of the geometric transformation layer. It is a block diagram which shows the structure of the image converter without a geometric transformation layer. It is a flowchart which shows the flow of the learning process which concerns on embodiment of this invention. It is a block diagram which shows the structure of the learning apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the posture estimation apparatus which concerns on embodiment of this invention. It is a schematic block diagram of an example of a computer functioning as a learning device or a posture estimation device. It is a figure which shows the experimental result.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<< Processing overview >>
The posture estimation method in the present embodiment adopts a configuration in which a neural network that outputs a posture parameter, a posture conversion image, and a feature amount when an image is received as an input is used as an image converter. The attitude parameter referred to here is a value representing the attitude of an object, and more specifically, it is an element of an affine transformation matrix or a projective transformation matrix, which is a matrix that gives the position, size, and angle of an object, for example. The posture is a value indicating what kind of rigid body motion is given to the object from the reference posture. The posture conversion image means an image after the posture conversion obtained when the posture conversion by the posture parameter is applied to the input image. The feature quantity is a vector representation of the features of the object appearing in the input image, and is used to specify the name of the object.

As the configuration of the image converter, any configuration can be adopted as long as the above input / output relationship can be realized, but an example of the configuration will be described.

FIG. 1 is a diagram showing an example of the configuration of the image converter 10. This configuration is based on CNN and is composed of a combination of four types of layers: a folding layer, a pooling layer, a fully connected layer, and a geometric conversion layer. For example, the geometric conversion layer 11, the folding layer 12, and the pooling layer 13 It is composed of a folding layer 14, a pooling layer 15, and fully connected layers 16 and 17. The convolution layer, the pooling layer, and the fully connected layer are widely known and known layers, and for example, those described in Reference 1 may be used.

[Reference 1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet Classification with Deep Convoutional Neural Networks. Proceedings of Neural Information Processing Systems, 2012.

Further, regarding the geometric transformation layer 11, any layer can be used as long as it can output a posture conversion image obtained by applying the posture parameter and the image conversion based on the posture parameter to the input image when the image is received as an input. It doesn't matter what. For example, the following known layers can be used.

[Reference 2] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial Transformer Networks. Proceedings of Neural Information Processing Systems, 2015.

For example, as shown in FIG. 2, the geometric transformation layer 11 estimates the attitude parameter θ represented by the affine transformation matrix or the projective transformation matrix from the image U input to the geometric transformation layer 11. It is composed of a layer 11A and an image conversion layer 11B for obtaining and outputting a posture-transformed image V by applying the estimated posture parameter θ to the image U. The attitude parameter estimation layer 11A is composed of a neural network that receives an image U as an input and outputs an element of an affine transformation matrix or a projective transformation matrix of an object reflected in the image U. The geometric transformation layer 11 configured in this way is suitable because it is possible to directly estimate how much the object reflected in the input image is deviated from the reference posture in terms of position, size, and angle. ..

Of course, the configuration of the neural network of the image converter 10 in the present invention is not limited to this, and any configuration may be adopted as long as the above input / output requirements are satisfied. Preferably, an L2 normalized layer is added to the final layer. By doing so, the feature amount can be made robust, which is preferable.

Further, for example, the configuration shown in FIG. 1 has a structure in which the combination of the convolution layer and the pooling layer is repeated twice, and then two fully bonded layers are stacked. Instead of the two fully bonded layers, The global pooling layer disclosed in Reference 3 or Reference 4 may be used.

[Reference 3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, Proceedings of Conference on Computer Vision and Pattern Recognition, 2016.

[Reference 4] Giorgos Tolias, Ronan Sicre, and Herve Jegou. Particular Object Retrieval with Integral Max-pooling of CNN Activations. ArXiv Preprint: https://arxiv.org/abs/1511.05879, 2015.

As described above, it is preferable to use a configuration without full coupling because it is possible to always obtain features having the same number of dimensions regardless of the size of the input image.

Hereinafter, for the sake of simplicity, the description will be continued by taking the case of using the image converter 10 shown in FIG. 1 as an example.

The posture estimation method in the present embodiment roughly classifies two different processes, that is, a posture estimation process for obtaining a feature amount and a posture using the image converter 10 and a learning process for learning the image converter 10.

Hereinafter, the learning process will be described in detail first, and then the posture estimation process will be described in detail.

<< Learning process >>
The learning process is a process that needs to be performed at least once before performing posture estimation, and more specifically, the weight of the neural network, which is a parameter of the image converter 10, is appropriately determined based on the training data. It is a process to do.

In executing this learning process, in the image converter 10 (neural network) described above, the image converter 10A excluding the geometric transformation layer is also used. As an example, FIG. 3 shows the configuration of the neural network of the image converter 10 of FIG. 1 excluding the geometric transformation layer 11. Hereinafter, the image converter 10 configured by the neural network having the geometric transformation layer 11 will be referred to as a model 10 with a geometric transformation layer, and the image converter 10A without it will be referred to as a model 10A without a geometric transformation layer. Here, the parameters other than the geometric transformation layer 11 are common to both models. That is, in the examples of FIGS. 1 and 3, the parameters of the corresponding convolution layer and the fully connected layer all have the same value.

In order to execute the learning process in the present embodiment, it is necessary to prepare the learning image data in advance. This learning image data is composed of a set of triplets of images. And this triad, the image I ^a you shoot an object at the reference ^position, an image I ^p of photographs depicting the same subject as the image I ^a in a different ^orientation, the image I ⁿ in the burst different subject from the image I ^a three It is assumed that the image data for learning is composed of a set of three sets consisting of a combination of different images.

Such a triplet of image data for learning can be automatically configured if there is an image set that satisfies the following requirements (a) to (d).

(A) Including one or more images, each image is a representation of at least one object.

(B) The entire learning image data includes two or more types of objects, and there are at least two or more images of the same type of objects.

(C) An object image that serves as a reference posture exists and can be identified for all types of objects.

(D) Images of the same type of object can be distinguished from each other.

As long as the above requirements (a) to (d) are satisfied, for example, the above three sets can be configured by the following processes (1) to (3).

(1) Select one image as a reference attitude randomly, which is an image I ^a.

(2) An image ^Ip that is not the reference posture and shows the same subject as the image I ^a is randomly selected and used as the image I ^p .

(3) image ^I a, randomly selects an image taken of ^{I p} and another object, this is the image ^{I n.}

By repeating the processes (1) to (3) until the desired number of sets is obtained, the image data for learning can be constructed.

Actually, the means for preparing the learning image data that satisfies the above requirements is irrelevant to the main point of the present invention, and any means may be used for preparing. It may be prepared manually or automatically.

In the presence of the above-mentioned learning image data, the learning process according to the embodiment of the present invention is executed by the following steps. FIG. 4 shows an outline of the learning process in this embodiment.

First, in step S301, applying a geometric transformation layer-model 10 for the image ^{I p,} obtaining the image ^{I pt,} and the feature amount ^{f pt} is an image ^{I p} after the geometric transformation.

Subsequently, in step S302, the model 10A without the geometric transformation layer is applied to each of the image I ^a and the image In ⁿ , and the corresponding feature quantities f ^a and f ⁿ are obtained.

Subsequently, in step S303, the appearance loss is obtained based on the image ^Ipt and the image I ^a .

Subsequently in step S304, the feature amount ^{f a,} the feature amount ^{f pt,} based on the feature quantity ^{f n,} calculates a characteristic quantity loss.

Subsequently, in step S305, the parameters of the model 10 with the geometric transformation layer are updated so as to reduce the weighted sum of the appearance loss and the feature amount loss, and the parameters are copied to the model 10A without the geometric transformation layer.

The above steps S301 to S305 are repeated until a predetermined end condition is satisfied, and when the end condition is satisfied, the learning process is ended.

Steps S301 and S302 can be immediately executed by applying the model 10 with the geometric transformation layer and the model 10A without the geometric transformation layer described above to each image. Hereinafter, the processing details of steps S303 to S305 will be described.

<Processing details of each process>
Hereinafter, an example of the detailed processing of each processing in the present embodiment will be described.

[Step S303: Appearance loss calculation process]

In this process, the image I ^a include the same ^subject, and, on the basis of the I ^pt, obtaining the appearance losses where a appearance difference as the image.

Through step S301, a posture-converted image ^Ipt obtained by subjecting the image ^Ip included in the learning image data to a posture-transformed image by the geometric transformation layer 11. Image I ^{a and} I ^p have a copy of the same subject, since different What are each captured position, ideally, the image I ^pt after the posture change, the same appearance as the image I ^a It is preferable that it is.

From this concept, the total sum of the distance of the pixel value of the image I ^a and image I ^pt, used as the appearance loss L ^a. The distance of the pixel value can be obtained as follows, for example, by the L1 distance.

(1)

Here, I ^a _i (x, y, c) or I ^pt _i (x, y, c) represents the pixel value of the channel c at the x and y positions of the image I ^a _i or I ^pt _i , respectively, and X _i , Y _i , and C _i represent the domain of x, the domain of y, and the domain of c for the set of I ^a _i and I ^pt _i , respectively. N is a constant of the number of triplets included in the learning image data or less.

The appearance loss _{L a} takes a smaller value image ^{I pt} _i and image ^I _{a i} after the posture conversion have the same ^appearance, becomes 0 in the case of ^_I pt _i ⁼ _{I a} i. That is, by learning the parameters of the geometric transformation layer 11 so that this value is smaller than the different images I ^p _i, even photographed object in any orientation, always photographing the subject at the reference posture and it will be able to perform the posture change as close to the image I ^a _i has, as a result, the attitude parameters are the it is possible to estimate the difference between the reference attitude.

Appearance loss is not necessarily may not be the form of the above formula (1), takes a smaller value image ^{I pt} _i and image ^I _{a i} have the same ^appearance, 0 in the case of ^_I pt _i ⁼ _{I a} i Any form may be taken as long as it becomes. For example, the distance to be used does not have to be the L1 distance, and for example, the L2 distance may be used. Further, X _i , Y _i , and C _i do not necessarily have to indicate the entire image. If the area in which the object is captured is given in some form in advance, the calculation may be limited to that range. Normally, the image often shows something other than the target subject, such as the background, but from the perspective of finding the change in the posture of the object, it is better not to include information on pixels that are not related to the subject. From such a viewpoint, it is preferable that the domain is set as limited to the area where the subject is captured as much as possible.

The above is the process performed in step S303.

[Step S304: Feature loss calculation process]
In this process, the feature amount loss required for learning the feature amount is obtained. In terms of image recognition, the image ^I a include the same ^subject, the feature amount ^f a of ^{I pt, f pt} is close to each other as a vector, also, conversely, an image ^I a containing different subject, the ^{I n} wherein It is preferable that the quantities f ^a and f ⁿ are far from each other as vectors. By obtaining such a feature amount, it becomes possible to search for the same subject based on the closeness of the feature amount, or to estimate the name of the subject by an identification method such as the K-nearest neighbor method.

The loss for learning such a feature amount can be in various forms, and for example, the following loss can be used.

(2)

Here, || x || ² ₂ is the L2 norm of x, m is a real value of 0 or more, and may be set to any value, but may be set to 0.3, for example.

The feature quantity loss Lf takes the feature quantity ^{f pt} _i is the minimum value if Chikashikere m or more in the feature value ^f _{a i} in L2 distance sense than the feature quantity ^f _{n i} 0, otherwise, 0 Take a larger value than. Therefore, by learning the parameters of the model 10 with the geometric transformation layer so that this value is small for various triplets, it is possible to output a feature amount closer to an image of the same subject.

Alternatively, the following feature amount loss may be used.

(3)

Thus feature quantity loss Lf defined becomes a smaller value closer in terms of the feature ^{f pt} _i and ^f _{a i} are the same L2 distance, also means the feature amount ^f _{n i} and ^f _{a i} is L2 distance The farther it is, the smaller the value is taken. Otherwise, it takes a value greater than 0.

The feature amount loss does not necessarily have to be in the form of the above equation (2) or equation (3). For example, the norm does not have to be the L2 norm, and for example, the L1 norm may be used.

The above is the process performed in step S304.

[Step S304: Model parameter update]

In this process, step S303, S304 in the determined appearance loss L _a, and on the basis of the feature quantity loss L _f, updates the parameters of the geometric transformation layer-model 10 so as to reduce these weighted sum L.

Specifically, L is defined as follows.

(4)

λ is a parameter that balances the two losses, and any real value may be set. For example, λ = 100 may be set.

L _a, L _f both in view of the piecewise it is differentiable with respect to the neural network parameters, it is possible learned by gradient method. For example, when learning based on the stochastic gradient descent method, if a parameter of the model 10 with a geometric transformation layer is set to w, one step per step.

(5)

The parameter w may be updated based on. The derivative value of L with respect to any parameter w can be calculated by the error back propagation method. Of course, an improvement method of a general stochastic gradient descent method such as using a momentum term or using weight attenuation may be introduced, or another gradient descent method may be used.

After updating the values of all the parameters based on the above, all the parameters except the geometric transformation layer 11 are copied to the model 10A without the geometric transformation layer.

The above is the process performed in step S305.

As described above, the processes from S301 to S305 are repeated until the end condition is satisfied. The end condition may be set arbitrarily, but for example, "end after repeating a predetermined number of times (for example, 100 times)", "end if the error phenomenon is within a certain range for a certain number of repetitions". "And so on.

The details of the learning process in the example of this embodiment have been described above.

<< Posture estimation processing >>
Subsequently, the posture estimation process of the posture estimation method in the example of the present embodiment will be described.

If the model 10 with the geometric transformation layer that has been trained is used, the posture estimation process is very simple. Specifically, an image may be input to the model 10 with a geometric transformation layer, and the posture parameters output by the geometric transformation layer 11 may be obtained. In one example of the present embodiment, the attitude parameter is assumed to be an affine transformation matrix or a projective transformation matrix, but in both cases, the size, position (translation), and angle of the object can be easily determined from the matrix. It is possible to ask.

Further, if the model 10 with the geometric transformation layer obtained in the example of the present embodiment can output the feature amount at the same time, as described above, if this feature amount is used, for example, the K-means method is known. It is also possible to recognize the name of the subject using the method.

The above is the posture estimation process of the posture estimation method in the example of the present embodiment.

<< Device configuration >>
Hereinafter, the learning device and the posture estimation device according to the embodiment of the present invention will be described with reference to the drawings.

FIG. 5 is a block diagram showing the configuration of the learning device 100. The learning device 100 shown in the figure is an image converter 10 that estimates a posture parameter representing a posture relative to a reference posture of a predetermined subject captured in the input first image according to the learning process described above. learn. The learning device 100 includes an input unit 20 and a calculation unit 30. The calculation unit 30 includes a model storage unit 32, an estimation unit 34, and a learning unit 38. The model storage unit 32 may be inside or outside the learning device 100, but in the present embodiment, the model storage unit 32 is arranged inside.

The input unit 20 receives two or more first images in which a predetermined subject is captured, and two or more third images in which a subject different from the predetermined subject is captured.

The model storage unit 32 stores the parameters of the image converter 10 and the image converter 10A.

The estimation unit 34 estimates the posture parameter representing the posture relative to the reference posture of the predetermined subject captured in the first image by the image converter 10, and uses the estimated posture parameter to perform the first. By geometrically transforming the image of, the second image in which a predetermined subject in the reference posture is captured is obtained, and the feature amount of the second image is extracted.

Further, the estimation unit 34 extracts the feature amount of the first image by the image converter 10A. Further, the estimation unit 34 extracts the feature amount of the third image, which is an image in which a subject different from the subject of the first image is captured, by the image converter 10A.

The learning unit 38 divides at least two or more first images of the same subject into two groups including at least one image, and divides the images in one group into the geometric conversion layer 11 of the image converter 10A. Is an image in which the sum of the distances between the pixel values of the second image and the pixel values of the images of the other group obtained by applying the above is the minimum, and a subject different from the subject of the first image is captured. The distance between the feature amount obtained by the image converter 10A with respect to the third image and the feature amount obtained by the image converter 10 with respect to the first image becomes long, and the first image becomes On the other hand, the above equations (1) to (5) are made so that the distance between the feature amount obtained by the image converter 10 and the feature amount obtained by the image converter 10A is close to the first image. According to this, the image converters 10 and 10A are learned, and the model storage unit 32 is updated.

Each process of the estimation unit 34 and the learning unit 38 is repeated until a predetermined end condition is satisfied.

FIG. 6 is a block diagram showing the configuration of the posture estimation device 150. The posture estimation device 150 shown in the figure estimates a posture parameter representing the posture of a predetermined subject captured in the input first image relative to the reference posture according to the posture estimation process described above, and at the same time, estimates the posture parameter. Recognize the subject of one image. The posture estimation device 150 includes an input unit 50, a calculation unit 60, and an output unit 70. The calculation unit 60 includes a model storage unit 62, an estimation unit 64, a reference image database 66, and a collation unit 68. The model storage unit 62 and the reference image database 66 may be inside or outside the posture estimation device 150, but in the present embodiment, the model storage unit 62 and the reference image database 66 are arranged inside.

The input unit 50 receives a first image in which a predetermined subject is captured, which is an estimation target.

The model storage unit 62 stores the parameters of the image converter 10 learned by the learning device 100.

The estimation unit 64 estimates the posture parameter representing the posture relative to the reference posture of the predetermined subject captured in the first image by the image converter 10, and uses the estimated posture parameter to perform the first. By geometrically transforming the image of, the second image in which a predetermined subject in the reference posture is captured is obtained, and the feature amount of the second image is extracted.

The reference image database 66 stores a feature amount for each of the reference images in which the name of the subject, which is the subject information, is known. The feature amount of the reference image shall be extracted in advance by the image converter 10 or the image converter 10A.

Further, the collation unit 68 recognizes the name of the subject of the first image by collating the feature amount extracted by the estimation unit 64 with the feature amount of the reference image.

The output unit 70 outputs the estimated posture parameters, the second image, and the names of the subjects in the first image.

Each of the learning device 100 and the posture estimation device 150 is realized by the computer 84 shown in FIG. 7 as an example. The computer 84 includes a CPU 86, a memory 88, a storage unit 92 that stores the program 90, a display unit 94 that includes a monitor, and an input unit 96 that includes a keyboard and a mouse. The CPU 86, the memory 88, the storage unit 92, the display unit 94, and the input unit 96 are connected to each other via the bus 98.

The storage unit 92 is realized by an HDD, SSD, flash memory, or the like. The storage unit 92 stores a program 90 for causing the computer 84 to function as the learning device 100 or the posture estimation device 150. The CPU 86 reads the program 90 from the storage unit 92, expands the program 90 into the memory 88, and executes the program 90. The program 90 may be stored and provided on a computer-readable medium.

<< Experimental Results >>
The results of an experiment in which the learning process and the attitude estimation process are performed using the learning processing method and the attitude estimation processing method constructed according to the example of the embodiment of the present technology described above are shown.

In this experiment, the recognition accuracy and the posture estimation error were evaluated using an image group including 12939 sets in which 10 kinds of characters were photographed in various postures. Here, in the triplet, a seed image in which the characters are photographed in the reference posture, an affine transformation image obtained by geometrically transforming the seed image by an affine transformation matrix obtained in advance, and a character different from the seed image are photographed. It consists of a seed image.

The learning process was performed using 11559 sets of the above three sets of image data for learning, and the posture estimation process was performed using the remaining 1380 sets as the evaluation data.

Further, as a conventional technique as a comparative example, a learning process and a posture estimation process were similarly performed on a technique using an image converter configured by using the CNN described in Reference 2.

FIG. 8 shows the results of recognition accuracy and attitude estimation error. Here, the recognition accuracy is the ratio of images in which the type of characters can be correctly recognized, and takes a value of 0 to 1, and the higher the value, the better. The attitude estimation error is an estimation error with respect to the true attitude (affine transformation matrix), and takes a value of 0 or more, and the smaller the value, the better. As is clear from this figure, according to this technique, it is possible to recognize the conventional technique with extremely high accuracy.

As described above, according to the posture estimation device according to the embodiment of the present invention, by optimizing the first image to be geometrically transformed into the second image in which the subject in the reference posture is captured. It is possible to accurately estimate the posture of the subject captured in the image based only on the learning data that can be collected at low cost.

According to the learning device according to the embodiment of the present invention, it is possible to collect at low cost by optimizing the first image to be geometrically transformed into the second image in which the subject in the reference posture is captured. Learning for accurately estimating the posture of the subject captured in the image can be performed based only on the learning data.

Further, the image converter is composed of a neural network provided with a geometric transformation layer, and learning of the geometric transformation layer involves inputting two different images A and B in which the same subject is photographed, and inputting the same subject to one image B. Only the attitude-transformed image B is obtained by applying the geometric transformation layer, and the pixel values of the attitude-transformed image B and the image A to which the other geometry-transforming layer is not applied are matched. By using the geometric transformation layer learned in this way, it is possible to convert the object in the image B so that the appearance always matches the image A regardless of the posture. According to this principle, if the image A includes an object captured in the reference posture, the same subject image captured in an arbitrary posture can be brought closer to the reference posture. At the same time, this geometric transformation layer is configured to output the attitude parameters used for the transformation, and has the attitude parameters to estimate how much the object in the image B deviates from the reference attitude. It is possible.

In addition, learning data to which explicit posture information is added based on the principle of learning a geometric transformation layer that transforms an image so that it always becomes an image showing a reference posture as long as there is an image of the same subject. Even if there is no such thing, the posture can be estimated only by the criteria of whether or not the appearances of the images match. This means that it is possible to learn by learning data that can be collected at a much lower cost than collecting learning data to which explicit attitude information is given, and this means that learning can be performed by learning data such as Non-Patent Document 2. Can be learned at extremely low cost.

Further, the geometric transformation layer is applied to an input image to estimate an affine transformation matrix or a projective transformation matrix, and the geometric transformation layer is applied to form an image converter as a neural network for obtaining a geometric transformation image. Therefore, it is possible to accurately obtain the position, size, and angle. This is because the affine transformation matrix and the projective transformation matrix are matrices that transform the position, size, and angle. With this configuration, it is possible to directly estimate how much the image B is deviated from the image A having the reference posture in terms of position, size, and angle.

Further, the learning device according to the embodiment of the present invention requires only the same subject image and does not use information on what kind of object the object is. That is, unlike Non-Patent Document 1 or Patent Document 1, the posture can be estimated with high accuracy even for an unknown object.

As described above, according to the embodiment of the present invention, a posture estimation device, a method, and a program capable of estimating a posture with high accuracy even for an unknown object based only on learning data that can be collected at low cost are provided. be able to.

The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

For example, the posture estimation device may estimate the posture parameters in each of a plurality of regions of the input image. For example, the input image may be divided into areas and input to the image converter, and the posture parameters may be estimated for each area.

Further, although the case where the learning device and the posture estimation device are configured as separate devices has been described as an example, they may be configured as one device.

Further, the posture estimation method and the learning method in the embodiment of the present invention may be configured by a computer, a server, or the like equipped with a general-purpose arithmetic processing unit, a storage device, or the like, and each processing may be executed by a program. This program is stored in a storage device, can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or can be provided through a network. Of course, any other component does not have to be realized by a single computer or server, but may be distributed and realized by a plurality of computers connected by a network.

Although the embodiments of the present invention have been described above with reference to the drawings, it is clear that the embodiments are merely examples of the present invention and the present invention is not limited to the above embodiments. is there. Therefore, components may be added, omitted, replaced, or otherwise modified without departing from the technical idea and scope of the present invention.

10 Image converter, model with geometric transformation layer 10A Image converter, model without geometric transformation layer 11 Geometric transformation layer 11A Attitude parameter estimation layer 11B Image transformation layer 20, 50 Input unit 30, 60 Calculation unit 32, 62 Model storage unit 34 , 64 Estimating unit 38 Learning unit 66 Reference image database 68 Matching unit 70 Output unit 100 Learning device 150 Geometry estimation device

Claims (8)

A posture estimation device that estimates the posture of a predetermined subject captured in the first image relative to the reference posture.
It has an estimation unit that estimates the relative posture by performing a predetermined process on the first image.
The predetermined process is based on a part of a process optimized to geometrically transform the first image into a second image in which the predetermined subject in the reference posture is captured. .. The process optimized for geometric transformation is one optimized for geometrically transforming the first image into the second image, and has a process of estimating the relative posture. The posture estimation device according to claim 1. The posture estimation device according to claim 1 or 2, wherein the process optimized for geometric transformation is STN (Spatial Transferner Networks). The estimation unit further performs a process of extracting the feature amount of the image geometrically transformed by the process optimized for the geometric transformation.
The process of extracting the feature amount of the image is
Optimal to perform the geometric conversion on the feature amount obtained by performing the process of extracting the feature amount of the third image, which is an image in which a subject different from the predetermined subject is captured, and the first image. The distance from the feature amount obtained by performing the conversion process and the process of extracting the feature amount of the image becomes long, and
A process for extracting the feature amount of the first image and a feature amount obtained by performing the process optimized for geometrically transforming the first image and the process of extracting the feature amount of the image. The posture estimation device according to any one of claims 1 to 3, which is optimized so that the distance from the feature amount obtained by performing the above is close. By collating the feature amount obtained by the estimation unit with the feature amount obtained by performing a process of extracting the feature amount of the image with respect to the reference image to which the subject information is added, the first The posture estimation device according to claim 4, further comprising a collating unit that recognizes a predetermined subject captured in an image. It is a learning device that learns the process of estimating the posture of a predetermined subject captured in the first image relative to the reference posture.
The relative posture of the predetermined subject captured in the first image with respect to the reference posture is estimated, and the predetermined subject in the reference posture is imaged using the estimated relative posture. An estimation unit that geometrically transforms the first image into the second image
A learning unit that learns the estimation of the relative posture and the geometric transformation so that the result of the geometric transformation by the estimation unit matches the second image.
Learning device including. It is a posture estimation method that estimates the posture of a predetermined subject captured in the first image relative to the reference posture.
The estimation unit includes estimating the relative posture by performing a predetermined process on the first image.
The predetermined process is based on a part of a process optimized for geometrically transforming the first image into a second image in which the predetermined subject in the reference posture is captured. .. A program for estimating the relative posture of a predetermined subject captured in the first image with respect to the reference posture.
On the computer
It is a program for executing the estimation of the relative posture by performing a predetermined process on the first image.
The predetermined process is a program based on a part of a process optimized to geometrically transform the first image into a second image in which the predetermined subject in the reference posture is captured.