JP6188345B2 - Information processing apparatus and information processing method - Google Patents

Description

  The present invention relates to a technique for estimating the position and orientation of a target object in an image.

  A lot of research and development has been conducted on image recognition in which an object is to detect a target object from an image of the target object. Image recognition technology has been applied to various fields and has been used for many real problems such as face recognition and parts recognition in factories.

  From the viewpoint of image pattern recognition, research has been conducted on how to classify input information. For example, various methods such as a neural network, SVM, Randomized Tree, and FERN according to Non-Patent Document 1 have been proposed. A learning image is required when generating a discriminator in these methods.

  In recent industrial visual recognition, there is a need to recognize a target object having a high degree of freedom of posture in a three-dimensional manner, for example, detecting piled parts. In order to detect a three-dimensional posture, learning images corresponding to various postures of the target object are required. In recognition tasks aimed at picking parts by a robot, the posture information of the target object is extremely important.

  The posture corresponding to the learning image is expressed by parameters such as Euler angles and quaternions. However, it is difficult to prepare a learning image of a target object that makes such posture known as a real image. In Patent Document 1, it is necessary to take an image with a handheld camera while measuring the relative position and orientation of the object to be photographed and the camera, and display the moving direction, rotation angle, etc. of the camera with respect to the position and orientation where the number of photographing is insufficient. A method that can obtain a complete omnidirectional image data set is proposed.

  On the other hand, it is often performed to generate a CG image of an arbitrary posture based on three-dimensional model data such as CAD and use it as a learning image. In the CG using the three-dimensional model, the viewpoint can be set freely, so that it is easy to generate a large amount of learning images as compared with the real-photographing. In particular, when the position and orientation of a target object is determined using distance information such as a depth map, it can be handled as a learning image using distance information obtained from a three-dimensional model.

JP 2011-198349 A

M.M. Ozuyal, et al. "Fast Keypoint Recognition using Random Ferns", IEEE Trans. on Pattern Analysis and Machine Intelligence, (2010).

  When generating a CG image from a three-dimensional model, it is necessary to set a viewpoint such as where an image viewed from the model is generated. The viewpoint for generating the learning image is set discretely, and the learning image of each posture is obtained by generating an image of the target object obtained when viewed from the position of the set viewpoint with CG. It will be.

  When the degree of freedom of posture of the target object is high, the posture of the target object in the input image at the time of detection hardly matches the posture of the discretely defined learning image. Therefore, the discriminator determines which posture is most similar to the posture used for learning. For this purpose, it is desirable that the features used in the classifier change smoothly between neighboring postures.

  However, depending on the shape of the target object, there may be a sudden change in the characteristics in the vicinity of a certain posture. In the vicinity of such a posture, the range that can be handled by the discriminator is extremely narrow, which causes a reduction in recognition accuracy regarding the posture.

  The present invention has been made in view of such problems, and provides a technique for suppressing a decrease in recognition accuracy related to a posture in the vicinity of a specific posture class when performing posture estimation of a target object in an input image. The purpose is to do.

In order to achieve the object of the present invention, for example, the information processing apparatus of the present invention is an information processing apparatus that generates a learning image to be given to a discriminator for learning a discriminator that estimates the posture of a target object. There,
Setting means for setting at least one viewpoint for the shape model of the target object;
Deriving means for deriving the flatness of the shape model from the distribution of distance values from the viewpoint to each region in the shape model when the shape model is observed from the set viewpoint;
Generating means for generating, as the learning image, an image of the shape model obtained by viewing the shape model from the set viewpoint when the flatness derived from the set viewpoint is equal to or less than a threshold value. It is characterized by.

  With the configuration of the present invention, it is possible to suppress a decrease in recognition accuracy related to a posture in the vicinity of a specific posture class when performing posture estimation of the target object in the input image.

The figure explaining the image generation using a geodesic dome. The figure explaining the process which creates a tree from an input image. The flowchart of the process of posture recognition of a target object. The figure explaining the rapid change of a characteristic. The figure explaining the branch state of the tree by the shift | offset | difference of a viewpoint. The figure which shows the score of a posture class. The flowchart of the process for learning a discriminator. The flowchart which shows the detail of the process in step S1300. The figure explaining a ray frequency map. The figure explaining a near viewpoint position. The flowchart which shows the detail of the process in step S1300. The figure explaining the addition of the bias to a learning image. The block diagram which shows the hardware structural example of information processing apparatus.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
Before describing this embodiment, first, problems of each embodiment including the present embodiment will be described. In order to generate a CG image of a three-dimensional model in various postures, it is first necessary to set a viewpoint for observing the three-dimensional model. The setting of the viewpoint is typically performed by a geodetic dome.

  As shown in FIG. 1, the learning image is a three-dimensional model with various roll angles at each point (each viewpoint A403) on the geodetic dome A401 centered on the object center A404 of the three-dimensional model of the target object. Is obtained as an image group A402. An index is given for each combination (posture class) of the viewpoint and the roll angle at the viewpoint. For example, when the number of viewpoints is 72 and the roll angle is changed every 30 degrees at each viewpoint (while rotating in-plane), the three-dimensional model is observed and a learning image is generated. The number of learning images to be generated is 72 × 12 = 864. However, as a dictionary, 72 × 12 = 864 classifiers are learned.

  In the following description, the number of viewpoints is N, the number of rotations of the roll angle (number of in-plane rotations for one viewpoint) is Nr, the viewpoint index is n (n = 1... N), and the in-plane rotation index is Let r (r = 1... Nr). Also, an index of a class (posture class) determined by a combination of the viewpoint n and the in-plane rotation r is represented by ν = 1... N × Nr. Since the attitude class ν has a one-to-one correspondence with the combination of the viewpoint n and the in-plane rotation r, the class index may be described as ν [n, r].

  Here, an identification method for performing posture recognition of a target object by a depth map using a tree type classifier that performs branching by two-point comparison, represented by Randomized Tree, will be described with reference to the flowchart of FIG. Note that the processing according to the flowchart of FIG. 3 is executed by a device such as a PC (personal computer). Of course, this apparatus holds a computer program for causing the CPU to execute the process according to the flowchart of FIG. 3, and the CPU executes the process according to the flowchart of FIG. To do.

  First, in the input image reading step in step S2100, an image showing the target object is acquired as an input image from an imaging device (not shown) or a storage device. Here, this input image is a so-called distance image (depth information) in which the pixel value of each pixel is a distance value (distance information) from the imaging device that has captured the input image to the target. Map). As a method for acquiring distance information using an imaging device, a stereo camera or a device using spatial coding is conceivable. However, the method is not limited to these methods, and an input image is also depth-dedicated as described later. It is not limited to maps.

  Next, in the position / orientation estimation step in step S2200, a tree (to be described later) using the images in the window while moving the window position (search position A200) in the raster scan order with respect to the input image A500 illustrated in FIG. Create

  The identification process is performed using a plurality of trees m (m = 1... M) indicated by A300. Here, M is the number of trees. The basic operation of each tree is as follows. With respect to the input image, each node of the tree branches according to the depth value between two different points on the input image. The respective positions (relative coordinates from the search position A200) of the two points A110 and A120 to be compared at the i-th node A312 of the m-th tree A310 shown in FIG. 2 are expressed as XmiB = (xmiB, ymiB). , XmiL = (xmiL, ymiL). Then, assuming that the depth values at each of the two points are D (XmiB) and D (XmiL), if the following expression (1) is satisfied, the process proceeds to the lower right node of the tree. If (2) is satisfied, the process proceeds to the lower left node of the tree.

  Starting from the root node A311 with respect to one input image, when the branch reaches the end node A313, the branch related to the tree ends. Each end node of each tree m is assigned a posture class index ν (ν = 1... N × Nr), and the number of votes for the posture class by the plurality of trees is used as a score of each posture class.

  When the input image I is given and the index of the posture class assigned to the tree m is IDXm (I), the score SCORE (ν, I) regarding the posture class ν is expressed by the following equation (3) ).

  Here, δ (A) is a function that returns 1 when the condition A is true and returns 0 when the condition A is false. After performing detection processing for all search positions A200, peak detection and threshold processing regarding scores are performed to determine the estimated position and orientation of the target object.

  Next, a classifier learning method using this tree will be briefly described. First, learning images corresponding to all posture indexes given as identification classes are prepared. At each node, the positions of the two points to be compared are determined randomly, and branch processing similar to that at the time of detection is performed. As a result of branching, when only a learning image corresponding to one posture index remains in the next node, that node is set as a terminal node and a posture index is allocated. If the end node is determined for all the learning images, the learning for the tree ends.

  In the depth learning image generated from the viewpoint allocated at equal intervals by the geodetic dome, there may be a case where a sudden change occurs in the feature depending on the shape of the target object. For example, as shown in FIG. 4, consider a case where a normal A420 related to a surface A410 having a large area on a three-dimensional model is parallel to a line-of-sight direction A440 connecting the viewpoint A430 (viewpoint A) and the object coordinate origin. In the image A450 obtained from the viewpoint A430, the depth values of the pixels in the area A411 corresponding to the surface A410 are all the same depth value. In this image, even if any combination of two points taken from the area A411 is compared, the condition of Expression (2) is always satisfied. Such a state is a singular point that occurs only when the line-of-sight direction A440 and the large-area surface A410 are completely orthogonal to each other even in a wide posture space.

  Like the viewpoint A431 (viewpoint A ″) in FIG. 5, even if it is slightly deviated from the viewpoint A, the depth value of the area A412 corresponding to the surface A410 in the obtained image A451 has a gradient. The branching state changes greatly.

  FIG. 6 (a) shows the viewpoints when the viewpoint position moves along viewpoints A432 (viewpoint Z), A433 (viewpoint B), and A434 (viewpoint C) arranged at equal intervals with viewpoint A430 and viewpoint A430. This is a plot of the score of the posture class corresponding to the viewpoint. However, in order to simplify the description, in-plane rotation is not considered here.

  B132, B133, and B134 are score curves of posture classes corresponding to the viewpoint A432, the viewpoint A433, and the viewpoint A434, respectively. Since the singular points as described above do not pass near these viewpoints, the score curve changes smoothly. On the other hand, the score curve of the posture class corresponding to the viewpoint A430 is a peaky curve like B130. Therefore, in the region like B110, the recognition accuracy regarding the posture is lowered.

  When the target object is an industrial product, the target object is often composed of a plane, and the target object coordinate system in CAD data is often set according to some reference plane. This situation can easily occur.

  Next, an embodiment relating to a learning method for solving the above-described problem will be described. First, a hardware configuration example of the information processing apparatus according to the present embodiment will be described with reference to the block diagram of FIG.

  The CPU 1351 executes each process using a computer program and data stored in the RAM 1352 and the ROM 1353 to control the operation of the entire information processing apparatus, and executes each process described later as what the information processing apparatus performs. To do.

  The RAM 1352 has an area for temporarily storing the computer program and data loaded from the external storage device 1356 and the computer program and data received from the external device via the I / F (interface) 1357. Further, the RAM 1352 has a work area used when the CPU 1351 executes various processes. That is, the RAM 1352 can provide various areas as appropriate.

  The ROM 1353 stores setting data for the information processing apparatus, a boot program, and the like. The operation unit 1354 is configured by a keyboard, a mouse, and the like, and various instructions can be input to the CPU 1351 by an operator of the information processing apparatus. The display unit 1355 is configured by a CRT, a liquid crystal screen, or the like, and can display a processing result by the CPU 1351 with an image, text, or the like.

  The external storage device 1356 is a mass information storage device represented by a hard disk drive device. The external storage device 1356 stores an OS (Operating System) and computer programs and data for causing the CPU 1351 to execute processes described later as those performed by the information processing apparatus. The external storage device 1356 also stores what is treated as known information in the following description. Computer programs and data stored in the external storage device 1356 are appropriately loaded into the RAM 1352 under the control of the CPU 1351, and are processed by the CPU 1351.

  An external device can be connected to the I / F 1357, and the information processing apparatus can perform data communication with the external device via the I / F 1357. For example, in the following description, an input image to be processed and other information may be acquired from an external device via the I / F 1357, and a processing result by the information processing device may be acquired via the I / F 1357. May be sent to.

  Each of the above parts is connected to a bus 1358. Note that the configuration shown in FIG. 13 is merely an example, and the configuration shown in FIG. 13 may be changed or modified as appropriate as long as processing equivalent to the processing described below can be realized. In the following description, the information processing apparatus according to the present embodiment is assumed to be a computer such as a PC. For example, a digital camera may be used.

  Next, a series of processes for learning a discriminator for estimating the posture of the target object will be described with reference to FIG. In the following description, a classifier that performs posture recognition of a target object using a depth map will be described as an example.

  Note that a computer program and data for causing the CPU 1351 to execute the process according to the flowchart of FIG. 7A are stored in the external storage device 1356. This computer program and data are appropriately loaded into the RAM 1352 under the control of the CPU 1351. When the CPU 1351 executes processing using the loaded computer program and data, the information processing apparatus executes processing according to the flowchart of FIG.

  First, in the model reading step in step S1100, the CPU 1351 loads the shape model of the target object that is the target of posture estimation from the external storage device 1356 into the RAM 1352. This shape model is a three-dimensional model such as CAD that imitates the shape of the target object, and includes information necessary for obtaining a depth map as a learning image, and includes surface information of the target object. .

  Next, in the viewpoint setting step in step S1200, the CPU 1351 sets a viewpoint for observing the shape model. The setting of the viewpoint is typically performed using a geodetic dome as shown in FIG. The viewpoint setting by the geodetic dome has an advantage that it is easy to handle as a posture class because the distances between the adjacent viewpoints are all equal. However, the viewpoint setting method is not limited to the method using the geodetic dome, and any method may be adopted as long as the viewpoint can be set at a plurality of positions surrounding the shape model.

  The total number of viewpoints set is N, the index of the viewpoint is n = 1... N, the target object coordinate system (one point in the shape model is the origin, and three axes orthogonal to each other at the origin are the x, y, and z axes, respectively. The position vector of the viewpoint n in the coordinate system) is assumed to be Vn = (Xvn, Yvn, Zvn). Further, it is assumed that the origin of the geodetic dome coincides with the object center A404 of the shape model. Vn is a unit vector that defines only the direction of the viewpoint. Therefore, the norm of Vn for all viewpoints n is 1.

  The process of the update process in step S1300 will be described with reference to the flowchart of FIG. In the flatness calculation step in step S1311, the CPU 1351 generates a depth map of the shape model based on the posture class ν given by the viewpoint n and the in-plane rotation r. This generated depth map is defined as a depth image Iν. That is, an image of the shape model observed from the viewpoint with the index n at the roll angle with the index r (in-plane rotation with the index r) is generated as the depth image Iν. A depth image corresponding to each posture class is created in advance and stored in the external storage device 1356, and a depth image corresponding to a combination of an index for a viewpoint and an index for in-plane rotation is acquired from the depth image. It doesn't matter.

  Here, a region where the shape model is reflected in the depth image Iν is Tν, and an area (number of pixels) of the region Tν is Sν. At this time, for each posture class, a histogram of depth values in the depth image corresponding to the posture class is created. The histogram calculated here is obtained by normalizing the frequency of each depth value obtained from the depth image corresponding to each posture class by Sν in the depth image. The range of the histogram is from the minimum value to the maximum value of the depth values in all the depth images Iν (ν = 1... N × Nr). The bin width is set according to the depth resolution of the distance measuring device when performing the detection process using the depth image. For example, if the depth measurement device has a depth resolution of 100 nm, the bin width is set to 100 nm as well. From the bin width and the histogram range, the bin number K can be easily obtained. In the histogram generated from the depth image Iν corresponding to the posture class ν, when the frequency value in the k (= 1... K) bin is represented as Hν (k), the flatness of the viewpoint n defined as follows: Pn is calculated (derived).

  Here, Νn is a set of classes obtained by the in-plane rotation variation regarding the viewpoint n, and the number of elements is Nr. That is, the flatness Pn is the maximum value of the total frequency values for each bin in each histogram created from images of the shape model viewed from the viewpoint n at different roll angles. It becomes an evaluation value.

  The case where the flatness Pn is large means that the ratio of the same depth value is large in the depth image at the viewpoint n, and the plane occupying a large area with respect to the shape model is orthogonal to the line-of-sight direction. Means.

  Next, if the flatness Pn of all viewpoints is equal to or less than the threshold εn, the process proceeds to step S1400. On the other hand, if even one of the flatness Pn of all viewpoints is larger than the threshold value εn, the process proceeds to step S1312.

  Here, the threshold value εn is a threshold value that determines how many pixels having the same depth value may exist in the region Tn, and is set to satisfy 0 <εn <1. The threshold εn is a value such as 0.3, and may be set to a uniform value for all viewpoints. Or you may set with the value depending on the size of the area | region of the shape model in a depth image, such as (epsilon) n / 1 / (root) Sn.

  Next, in the viewpoint moving step in step S1312, the CPU 1351 newly sets a viewpoint group that is different from the viewpoint set in step S1200 and surrounds the shape model. Here, the newly set viewpoint group means a viewpoint group obtained by rotating all viewpoints set in step S1200 in the same direction with the origin in the target object coordinate system as the center. If the rotation transformation is R, the position Vn of the viewpoint n in the target object coordinate system is updated as follows.

  Here, T represents the shift of the matrix. The rotational transformation R can be expressed as follows, for example, when expressed by an xyz system Euler angle.

  The rotation transformation R for moving the viewpoint may be given as a random number. For example, the values of θx, θy, and θz in Equation (6) are each randomly selected between 0 and 180 deg. Alternatively, a value proportional to the minimum angle θd between the viewpoints on the geodetic dome may be given. For example, θx, θy, θz = θd / 10 may be set.

  Then, after resetting the viewpoint in step S1312, the process returns to step S1311. In step S1311, the flatness Pn for each viewpoint is obtained again using the reset viewpoint.

  In the learning image output step in step S1400, the CPU 1351 outputs the depth image determined by the above processing and corresponding to each posture class as a learning image to a memory such as the external storage device 1356 or the RAM 1352. The output destination is not limited to a specific output destination.

  In the classifier generation step in step S1500, the CPU 1351 performs a learning process for a classifier that identifies the posture of the target object, using the output depth image for each posture class as a learning image. Since the procedure related to learning of the discriminator has been described above, description thereof is omitted here.

  In the following embodiments including this embodiment, a tree without pruning has been described as an example, but the type of classifier is not limited to this. Pruning may be performed at the time of tree learning, or FERN shown in Non-Patent Document 1 may be used. In the case of these classifiers, classes are not uniquely assigned to the terminal nodes, and the ratio of learning images remaining in the terminal nodes during learning is obtained as an estimated score for each class. Assuming that the score for the class ν obtained at the terminal node of the tree m is obtained by Pm (ν | I) when the input image I is given, the score of the entire classifier is given by the following equation (7). .

  Or it may be given as follows.

  Further, in each of the subsequent embodiments including this embodiment, the learning image by the depth map is specifically described as an example, but the present invention is not limited to the depth map. For example, a luminance image by CG rendering may be used.

  In this way, by updating the viewpoint so that the plane constituting the target object is not orthogonal to the viewpoint direction, it is possible to suppress a decrease in discrimination power near a specific posture class. Specifically, it will be described as follows.

  By the method described above, the viewpoints A, B, C, and Z in FIG. 6 move to the viewpoints A ′, B ′, C ′, and Z ′, respectively. Thereby, the score curves of the posture classes of the viewpoints A ′, B ′, C ′, and Z ′ are respectively as B140, B143, B144, and B142 in FIG. 6B, and the viewpoint A in FIG. It is possible to suppress a decrease in posture recognition accuracy like the region B110 in the vicinity of.

  Note that the configuration described in the present embodiment is merely an example as described above, and is merely an example of the configuration described below. That is, the information processing apparatus according to the present embodiment is an information processing apparatus that generates a learning image to be given to a discriminator in order to learn a discriminator that estimates the posture of the target object. In this information processing apparatus, at least one viewpoint is set for the shape model of the target object. When the shape model is observed from the set viewpoint, when the set viewpoint and the shape model satisfy a predetermined condition, an image of the shape model viewed from the set viewpoint is learned. It generates as an image.

[Second Embodiment]
In the first embodiment, the viewpoint update is terminated when the flatness of all viewpoints is equal to or less than the threshold value. However, a more suitable viewpoint may be selected from various viewpoint settings. Processing in that case will be described with reference to FIGS. 7A and 8B. In the second and subsequent embodiments, only differences from the first embodiment will be described, and descriptions of the same points as in the first embodiment will be omitted.

The processes in step S1100 and step S1200 are the same as in the first embodiment. However, in this embodiment, in addition to that, in step S1200, the set of N viewpoints set is called a view set, initialized to t = 0, and the set view set here is set as an initial view set U ₀ . To do.

In step S1300, the process shown in FIG. 8B is executed. The flatness calculating step in step S1321, in the same manner as in step S1311, calculates a flatness Pn of respective viewpoints n included in the viewpoint set _{U t.} This calculation follows the following equation (9).

Here, Νn is a set of classes obtained by the in-plane rotation variation with respect to the viewpoint n. Then, as shown in the following formula (10), the largest flatness _{Pt, max} of the flatness Pn of each viewpoint is set as the maximum flatness _{Pt, max} .

  Next, if a predetermined number T of viewpoint sets are generated, the process proceeds to a viewpoint selection step in step S1323. If a predetermined number T of viewpoint sets have not yet been generated, the process proceeds to step S1322. Proceed to the process. The predetermined number T may be set to a value such as 10, for example. A larger T is better, but is set in a trade-off with the processing time required for step S1321.

In step S1322, by applying a predetermined rotation transformation on the viewpoint set _{U t,} and generates a new perspective set _{U t + 1.} As in the first embodiment, this rotation conversion may be given randomly, or may be given as a value proportional to the minimum angle θd between the viewpoints on the geodetic dome. Then, the process returns to step S1321, and the maximum flatness is obtained using this new viewpoint set.

When the process proceeds to step S1323, since a predetermined number T of viewpoint sets are generated, among the predetermined number T of viewpoint sets, the viewpoint set U _c for which the smallest maximum flatness is obtained is specified ( Formula (11)).

In step S1400 and subsequent steps, the depth image corresponding to each posture class based on each viewpoint in the specified viewpoint set U _{c is} set as a learning image. Since the subsequent processing is the same as in the first embodiment, description thereof is omitted.

[Third Embodiment]
In the first embodiment and the second embodiment, the viewpoint is selected based on the flatness, but the viewpoint may be selected so that the ratio of the plane orthogonal to the line of sight decreases. Processing in that case will be described with reference to FIGS. 7 (a) and 8 (c).

  The processes in step S1100 and step S1200 are the same as in the first embodiment. In step S1300, the process shown in FIG. 8C is executed. In the normal frequency map generation step in step S1331, a normal frequency map is generated from the shape model to indicate which direction the plane on the target object is facing. Specifically, the normal frequency map is generated as follows.

  First, regarding Np polygons constituting the shape model (shape model of the target object), the area Sp of the polygon p (= 1... Np) and the normal direction normalization vector np (| np) of the polygon p in the target object coordinate system. | = 1) is obtained. Let Rp = Sp / Sall be an area normalized by the total area of all polygons. This represents the area ratio of the polygon p in the surface area of the entire shape model. A normal frequency F (Ψ) with respect to one point n (Ψ) on the unit sphere obtained when an arbitrary rotational transformation Ψ is performed on the reference axis n0 = (1, 0, 0) on the unit sphere is expressed as follows. Define as follows.

  Here, δ (A) is a function that returns 1 when the condition A is true and returns 0 when the condition A is false. Since Np or more variations of the rotational transformation Ψ do not exist, if F (Ψ) is calculated for rotational transformation variations from n0 to np (p = 1... Np), a discrete normal as a set of F (Ψ). A frequency map will be obtained. A specific example is shown in FIG.

  The reference axis n0 is represented by C100. A vector obtained by multiplying the reference axis C100 by the rotational transformation Ψ represented by C300 is C200. The sum of the area ratios of the polygons in the surface area of the entire target object with respect to the polygons C410 and C420 having normalized normal vectors C210 and C220 in the same direction as the vector C200 is calculated by the above equation (12). The frequency F (Ψ) shown is obtained.

  The following evaluation value En is calculated from the obtained frequency F (Ψ).

  Here, ang (a, b) represents an angle formed by the vector a and the vector b. Since Vn and n (Ψ) are both unit vectors, the following expression is satisfied.

  In the viewpoint optimization step in step S1332, the evaluation value En obtained as described above is used to change the viewpoint or the viewpoint in the same manner as the flatness Pn in the first and second embodiments described above. By selecting, the position of the viewpoint is determined. Since the subsequent processing is the same as in the first embodiment and the second embodiment, a description thereof will be omitted.

[Fourth Embodiment]
As in the third embodiment, when the ratio of the plane orthogonal to the line of sight is used as the evaluation value, the viewpoint may be sequentially updated so that the ratio decreases. Processing in that case will be described with reference to FIGS. 7 (a) and 8 (c).

  The processes in step S1100 and step S1200 are the same as in the first embodiment. In step S1300, the process shown in FIG. 8C is executed. In the normal frequency map generation step in step S1331, the normal frequency F (Ψ) is obtained in the same manner as in the third embodiment.

  Next, in the viewpoint optimization step in step S1332, the viewpoint is updated using the obtained normal frequency map. First, the cost En (θ) when the rotation transformation R (θ) is performed on the viewpoint position Vn = (Xvn, Yvn, Zvn) (n = 1... N) set in step S1200 is expressed as follows. Define as follows.

  Here, ang (a, b) represents an angle formed by the vector a and the vector b. Since Vn and n (Ψ) are both unit vectors, the following equation (16) is satisfied.

  η is a positive constant that determines the width of the kernel, and is set to η = 2 or the like. The state where the cost En is large means that a plane having a large area is nearly orthogonal to a line-of-sight direction connecting the shape model center from the viewpoint n. That is, if the rotational transformation R (θ) can be set so as to reduce the cost En, the viewpoint can be updated so that the number of planes orthogonal to the line-of-sight direction is reduced.

  The energy function E when the rotation transformation R (θ) is performed on the viewpoint set in step S1200 is defined as follows.

  If this is solved by the gradient method, the update formula of θ is as follows.

  Here, ζ is an update coefficient set in a range of 0 <ζ <1, and ζ = 0.1 is set. The rotation conversion R (θ) is usually set to no rotation (R = unit matrix) by default, but is not limited to the initial value. Starting from the initial value of the rotation conversion R (θ), the rotation conversion R (θ) is updated by sequentially executing Expression (18). When the change of the rotational transformation R (θ) becomes smaller than a predetermined threshold, it is considered that the rotation has converged, and the sequential calculation is terminated. Then, as shown in Expression (19), the rotation conversion R (θ) at the time of convergence is performed for all the viewpoints Vn (n = 1... N) set in step S1200, and the viewpoint Vn is updated.

  When all the viewpoints are updated, the process proceeds to step S1400. In step S1400, the depth image corresponding to each posture class determined by the above processing is output as a learning image to a memory such as the external storage device 1356 or the RAM 1352. Since the subsequent processing is as described in the first embodiment, description thereof is omitted.

[Fifth Embodiment]
Whether or not the state of the classifier at the neighboring viewpoint does not change extremely with respect to the internal state of the classifier at the set viewpoint may be set as the viewpoint update condition. Processing in that case will be described with reference to FIGS. 7B and 8D.

  The processes in step S1100 and step S1200 are the same as in the first embodiment. In the learning image generation step in step S1700, a depth image corresponding to each posture class is acquired as a learning image. The depth image acquisition method corresponding to each posture class is as described in the first embodiment. The processing in step S1500 is as described in the first embodiment.

  Next, the process performed in step S1300 will be described with reference to the flowchart of FIG. 8D showing details of the process. In the viewpoint variation image generation step in step S1341, a near viewpoint that is slightly moved from the currently set viewpoint is set, and an image of a shape model when viewed from the vicinity viewpoint is generated. Here, the near viewpoint is a viewpoint that is slightly shifted from Vn with respect to the set viewpoint Vn (n = 1,..., N). If A460 in FIG. 10 is Vn, the near viewpoint position Vnj (j = 1... J: J = 6 in FIG. 10) represented by A461 is obtained by shifting A460 by a minute angle δ represented by A470. Here, the small angle δ is a small value close to 0, and is set to a value smaller than the angle between the viewpoint Vn and the adjacent viewpoint. For example, if the angle between Vn and the adjacent viewpoint Vn ′ is 7 deg, δ = 3 deg is set. The adjacent viewpoint Vn ′ is defined as follows.

  Here, ang (Vn, Vk) is an angle formed by the viewpoint Vn and the viewpoint Vk when viewed from the center of the geodetic dome. Here, since both Vn and Vk are represented by unit vectors, the following equation (21) is satisfied.

  It is desirable to set a plurality of neighboring viewpoints around each viewpoint Vn. As shown in FIG. 10, several points are set at equal angular intervals around Vn, for example, 6 points at 60 deg intervals (J = 6), etc. May be set. An image I [nj, r] (n = 1... N, j = 1... J, r = 1... Nr) of the shape model viewed from the set near viewpoint direction is generated.

  In the comparison step in step S1342, the state of the discriminator is compared between the set viewpoint and its neighboring viewpoints to see how much has changed. The learning image for the viewpoint Vn generated in step S1700 and the image for the viewpoint near the viewpoint Vn generated in step S1341 are input to the classifier generated in step S1500. At that time, the degree of similarity regarding the state of each classifier is calculated. Various similarities can be considered. For example, when considering an example using a tree type discriminator, it is as follows.

  When the voting score by a plurality of trees for the posture class ν [n, r] corresponding to the in-plane rotation r of the viewpoint n with respect to the input image I is SCORE (ν [n, r], I), the viewpoint Vn and the neighboring viewpoint The similarity Sim (n, j) of Vnj is defined as follows.

  Here, I [n, r] is a learning image of the posture class corresponding to the in-plane rotation r of the viewpoint n. Otherwise, if the classifier is a pruned tree or FERN, the score for the class ν [n, r] obtained at the terminal node of the tree m is Pm (ν [n, r] | I) In some cases, the information amount of the entire discriminator can be used as the similarity (formula (23)).

  The minimum value of the similarity between the viewpoint n and the near viewpoint is set as the comparison scale Ln of the viewpoint n (Expression (24)).

  The scale L for all viewpoints of the similarity to the nearby viewpoint is calculated by any one of the total sum of Ln (Expression (25)), the sum of squares (Expression (26)), and the minimum value (Expression (27)).

  In addition, the scale L may be defined using the amount of information (relative entropy) of the Cullback-Liber (formula (28)).

  If the scale L obtained by any of these methods is equal to or smaller than the threshold value θL, the process proceeds to step S1800. On the other hand, when the scale L is larger than the threshold value θL, the process proceeds to step S1343. In the viewpoint moving step in step S1343, the viewpoint is moved by performing the same process as the process in step S1312 described in the first embodiment.

  In step S1800, it is determined whether the viewpoint is moved by performing the process in step S1343. If the result of this determination is that the viewpoint has not been moved, the processing in FIG. On the other hand, if the viewpoint is moved, the process returns to step S1700, and the subsequent steps are executed using the moved new viewpoint.

  Thereby, the learning image by the viewpoint set when the process according to the flowchart of FIG.7 (b) is complete | finished can be output as a learning image finally given to a discriminator.

  In the above-described processing, a predetermined number T of viewpoint sets may be generated in advance, and all may be compared and the one with the smallest scale L may be selected. In that case, the details of the processing in step S1300 are as shown in FIG.

First, the view point set that is set in step S1200 holds the U _0, keep initialization and number of repetitions t = 0. The processing in step S1341 is as described above. If a predetermined number T of viewpoint sets have not been generated, t is incremented by 1, and the process of step S1343 is performed in the same manner as described above, and the obtained viewpoint set U _t is held. The predetermined number T is set in the same manner as in the second embodiment. When the predetermined number T of viewpoint sets are generated, the process proceeds to step S1344.

In the viewpoint selection step in step S1344, the aforementioned scale L is calculated for each of the viewpoint sets U ₀ , U ₁ ... U _T. Then, when the scale for the viewpoint set U _t is Lt, the minimum value Lc is specified among L0, L1... LT, and the viewpoint set U _c corresponding to the specified minimum value Lc is selected as the final viewpoint. (Formula (29)).

The classifier learned from the learning image corresponding to the selected viewpoint set U _c is set as the final classifier. The value of the scale Lt may be obtained in step S1341 instead of step S1344. In that case, if the scale Lt is held so that it can be used in step S1344, only one of the viewpoint set U _t and the viewpoint set corresponding to the previous minimum value is compared by comparing Lt with the previous minimum value. You should leave For this reason, it is not necessary to retain all of the generated learning images and neighboring viewpoint images.

[Sixth Embodiment]
The state of the discriminator obtained by learning from the learning image at the set viewpoint may be used as the viewpoint update condition. Processing in that case will be described with reference to FIGS. 7B and 11A. The processes in step S1100, step S1200, step S1700, and step S1500 are the same as those in the first embodiment.

  The process in step S1300 will be described with reference to the flowchart of FIG. 11A showing details of the process. In the discriminator determining step in step S1351, it is determined how much the internal state of the discriminator obtained by learning is biased. Here, various biases can be considered. For example, if the bias in the conditional branch is considered as a scale, the scale Bν for the posture class ν can be defined as follows.

  Here, ρm (Iν, m, d) is a branch at a node at a depth d (d = 1,, D) when the learning image Iν is input to the tree m (m = 1,, M). Assume that the function returns 0 when the condition of the expression (1) is satisfied, and returns 1 when the condition of the expression (2) is satisfied. However, D is the number of branches from the root node to the end node of the tree. The value Cν, m represented by the expression (31) is a value obtained by counting branches according to the condition of the expression (1) in the tree m. If the branch of each node is completely random, Cν, m follows a binomial distribution of B (D, 0.5). Therefore, the scale Bν, which is a value obtained by normalizing Cν, m, can take a value of 0 at the minimum and 1 at the maximum, and takes a value close to 0.5 if there is no bias in the branch. The maximum value of the scale Bν when the learning image Iν (ν = 1,, N × Nr) is input is defined as Bmax (Formula (32)).

  If the obtained scale Bmax is greater than the predetermined threshold value θ, the process proceeds to step S1353, and if it is equal to or less than the threshold value, the process proceeds to step S1800. The threshold value is set in a range of 0.5 <θ <1, and is given as θ = 0.7, for example. Alternatively, robustness when noise is added to the learning image may be used as a measure of the bias. An image obtained by adding W kinds of white noise to the learning image Iν is denoted by Iν, w (w = 1... W). At this time, the scale Bν is defined as follows.

  If this value is small, the posture ν is robust against noise. In the same manner as in the equation (32), the maximum value of the scale Bν is set as the scale Bmax. Although noise is given to the learning image here, robustness can be measured even if noise is added at the time of branching of each node.

  In the viewpoint movement process in step S1353, the viewpoint is moved by performing the same process as the process in step S1312 described in the first embodiment. Thereby, the learning image by the viewpoint set when the process according to the flowchart of FIG.7 (b) is complete | finished can be output as a learning image finally given to a discriminator.

  In the above-described processing, a predetermined number T of viewpoint sets may be generated in advance, and all of them may be compared and the one with the smallest scale Bmax may be selected. In that case, details of the processing in step S1300 are as shown in FIG.

First, the view point set that is set in step S1200 holds the U _0, keep initialization and number of repetitions t = 0. The processing in step S1351 is as described above. If a predetermined number T of viewpoint sets have not been generated, t is incremented by 1, and the process of step S1353 is performed in the same manner as described above, and the obtained viewpoint set U _t is held. The predetermined number T is set in the same manner as in the second embodiment. If the predetermined number T of viewpoint sets are generated, the process proceeds to step S1354.

In the viewpoint selection step in step S1354, the aforementioned scale Bmax is calculated for each of the viewpoint sets U ₀ , U ₁ ... U _T. Then, when the scale for the viewpoint set U _t is Bt, max, the minimum value Bc, max is specified from B0, max, B1, max ... BL, max, and the viewpoint corresponding to the specified minimum value Bc, max is specified. The set U _c is selected as the final viewpoint (formula (34)).

Then, the classifier learned from the learning image corresponding to the selected viewpoint set U _c is set as a final classifier.

[Seventh Embodiment]
You may add the viewpoint from which the characteristic changes rapidly with respect to the viewpoint obtained in 1st-6th embodiment. In that case, the following processing is performed at the end of step S1300 in each embodiment.

  The flatness Pn is calculated for all the viewpoints generated in step S1200 or step S1300. Then, viewpoints for which the calculated flatness Pn is greater than the threshold value εn described in the first embodiment are added to the set of viewpoints obtained in step S1300. In other words, an image of the shape model obtained by viewing the shape model at different roll angles from the position where the evaluation value is greater than the threshold value among the positions (each viewpoint) is output as a learning image. As a result, the viewpoint Y represented by B441 in FIG. 6C is added, and the discrimination ability in the specific posture is interpolated as indicated by the score curve B141.

[Eighth Embodiment]
In the case of a classifier based on a two-point comparison, if it is found that the posture that becomes a singular point is included in the set viewpoint, a bias may be applied to the learning image itself. Processing in that case will be described with reference to FIG.

  The processing in each of step S1100, step S1200, and step S1700 is as described above. In the learning image update step in step S1600, first, it is determined whether or not a learning image of a posture class that is a singular point is included in the learning image. As the determination criterion, the flatness Pn described in the first embodiment is used. If at least one of all viewpoints n = 1... N is greater than the threshold value .epsilon.n, a bias is applied to all learning images. The bias given to the learning image is given linearly with respect to the image coordinates as shown in FIG. 12, for example. C600 is the learning image Iν, and it is assumed that the depth when the image is cut at the cutting position C610 is given by C620. At this time, by applying a linear bias, the depth of the cross section is updated like C630. When the depth value at the position X = (x, y) of the learning image coordinate system of the learning image Iν (the coordinate system having one point in the learning image (for example, the upper left corner) as the origin) is Dν (X), As a depth value to which a bias is added, Dν (X) is updated as follows.

  The bias function f (X) is a linear function with respect to X, and is defined as follows.

  However, a, b, and c are constants. The constants a and b representing the gradient are determined by the image resolution in the work area and the shortest viewpoint angle of the geodetic dome. For example, if the image resolution is 1 mm / pixel and the shortest viewpoint angle is 7 deg, a = 0.1, b = 0, c = 0, and the like are set.

  In step S1500, a learning process for a discriminator for identifying the posture of the target object is performed using the learning image updated in step S1600. When discriminating the input image I using the discriminator obtained by learning, a bias similar to the above is applied to the entire image. If the x-axis and y-axis in the input image coordinate system are in the same direction with respect to the x-axis and y-axis in the learning image coordinate system, the bias of equation (36) may be given in the same manner. That is, the depth value D (XS) at the position XS = (xS, yS) in the input image is updated as follows, and then the search by the discriminator is performed.

  Considering the bifurcation by the two-point comparison, only the magnitude relation of the depth is important, so the difference in the origin position between the input image and the learning image has no effect. As a result, the depth is artificially given to the state where the large surface A410 as shown in FIG. 4 is orthogonal to the axis of the line-of-sight direction A440, so that only the condition of Expression (2) is satisfied. Special situations can be avoided.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (8)

An information processing apparatus for generating a learning image to be given to a discriminator for learning a discriminator for estimating a posture of a target object,
Setting means for setting at least one viewpoint for the shape model of the target object;
Deriving means for deriving the flatness of the shape model from the distribution of distance values from the viewpoint to each region in the shape model when the shape model is observed from the set viewpoint;
Generating means for generating, as the learning image, an image of the shape model obtained by viewing the shape model from the set viewpoint when the flatness derived from the set viewpoint is equal to or less than a threshold value. An information processing apparatus characterized by the above. Further, when the flatness obtained by the derivation means is larger than the threshold value, the change means for changing the viewpoint position,
2. The information processing according to claim 1 , wherein the generation unit generates an image of the shape model obtained by viewing the shape model from the viewpoint at the viewpoint position changed by the changing unit as the learning image. apparatus. An information processing apparatus for generating a learning image to be given to a discriminator for learning a discriminator for estimating a posture of a target object,
Setting means for setting at least one viewpoint for the shape model of the target object;
A derivation means for deriving an evaluation value representing the percentage of the surface and each surface constituting the viewing direction and the shape model when observing the shape model from the set viewpoint orthogonal,
Generating means for generating, as the learning image, an image of the shape model obtained by viewing the shape model from a viewpoint in which the evaluation value is set at a position greater than a threshold value among a plurality of positions ;
The information processing apparatus comprising: a. Furthermore, based on the evaluation value obtained by the derivation means, the update means for updating the position,
Said generating means, according to claim 3, characterized in that the image of the shape model seen the shape model from a viewpoint set in the position updated by said updating means, for generating as the learning image Information processing device. Moreover, using said learning image processing apparatus according to any one of claims 1 to 4, characterized in that it comprises means for performing learning of the classifier. An information processing method performed by an information processing apparatus that generates a learning image to be given to a discriminator in order to learn a discriminator that estimates the posture of a target object,
A setting step in which the setting unit of the information processing apparatus sets at least one viewpoint for the shape model of the target object;
The derivation means of the information processing device derives the flatness of the shape model from the distribution of distance values from the viewpoint to each region in the shape model when the shape model is observed from the set viewpoint. A derivation process;
When the flatness derived at the set viewpoint is less than or equal to a threshold value , the generation unit of the information processing apparatus converts the shape model image obtained by viewing the shape model from the set viewpoint. An information processing method comprising: a generation step of generating as An information processing method performed by an information processing apparatus that generates a learning image to be given to a discriminator in order to learn a discriminator that estimates the posture of a target object,
A setting step in which the setting unit of the information processing apparatus sets at least one viewpoint for the shape model of the target object;
A derivation step in which the derivation means of the information processing device derives an evaluation value that represents a ratio of a plane in which the line-of-sight direction when the shape model is observed from the set viewpoint and each surface constituting the shape model is orthogonal When,
A generation step in which the generation unit of the information processing apparatus generates, as the learning image, an image of the shape model obtained by viewing the shape model from a viewpoint in which the evaluation value is set to a position larger than a threshold value among a plurality of positions. When
An information processing method comprising: A computer program for causing a computer to function as each unit of the information processing apparatus according to any one of claims 1 to 5 .

Images