JP2018036770A - Position attitude estimation device, position attitude estimation method, and position attitude estimation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enhance an estimation accuracy of the position attitude of an object based on an image.SOLUTION: A position attitude estimation device includes: a first extraction part; a global area determination part; a second extraction part; and a specifying part. The first extraction part extracts local feature information around a feature point of an object image imaging an object which is an estimation object of a position attitude, and the local feature information around the feature point of a registered image in which the object is shown in a predetermined position attitude. The global area setting part determines the global area including a plurality of feature points for the object image and the registered image, respectively. The second extraction part extracts global area feature information based on the feature point of the object image and the global area and the global area feature information based on the feature point of the registered image and the global area. The specifying part specifies a group of pairs of feature points showing the correspondence between the feature point of the object image and the feature point of the registered image on the basis of the similarity of the local feature information and the similarity of the global feature information in the pairs of feature points of the feature point of object image and the feature point of the registered image.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a position / orientation estimation apparatus, a position / orientation estimation method, and a position / orientation estimation program.

  In factories that assemble various products, the work contents are instructed to the workers by augmented reality (AR) or the like, thereby supporting the efficiency of the work by the workers. As one type of assist technology of this type, a technology for estimating the position and orientation (hereinafter referred to as “position and orientation”) of an object based on an image obtained by imaging the object is known.

  A technique for estimating the position and orientation of an object based on an image is applied to, for example, a work line for attaching a label to a specified position of an object conveyed by a conveyance path such as a belt conveyor. The position and orientation of the object to be conveyed is estimated, and an image showing the label application position calculated based on the position and orientation is directly projected onto the object, so that the correct application according to the position and orientation of the object is applied to the worker. A position (specified position) can be presented.

  When estimating the position and orientation of an object based on an image, a target image obtained by imaging the object for which the position and orientation is estimated and a registered image in which the object is captured at a predetermined position and orientation are used. The position and orientation of the target object in the target image are estimated based on the correspondence between the feature points. The correspondence between the feature point of the target image and the feature point of the registered image is specified based on the similarity of the image information around the feature point (local region). In addition, there is a method of using a camera and a distance sensor to specify the correspondence between feature points based on feature information obtained by adding unevenness information (depth differential value) to image information around the feature points detected from the image. It is known (see, for example, Patent Document 1). Furthermore, a method of adding feature information that is strong against geometric transformation by registering a set of feature points extracted from an object in an image as a geometric figure is known (for example, non-patent literature). 1).

JP 2011-174879 A

Lamdan, Y. and Wolfson, H.J .: "Geometric Hashing: A General And Efficient Model-based Recognition Scheme", Computer Vision. Second International Conference on, pp.238-249, 1988

  It is difficult to distinguish between feature points with small differences in image information in the local area by the method of identifying the correspondence between the feature points based on the image information and unevenness information in the local area around the feature points detected from the image It is. Therefore, when there is a repetitive pattern or the like on the surface of the object in the image, the similarity of local image information at a plurality of feature points increases, and it becomes difficult to specify the correspondence between the feature points. . That is, when the similarity of local feature information at a plurality of feature points detected from one image is high, it is difficult to specify the correspondence between the feature points of the target image and the feature points of the registered image. The estimation accuracy of the position and orientation of the target object is reduced.

  In one aspect, an object of the present invention is to improve the estimation accuracy of the position and orientation of an object based on an image.

  In one aspect, the position / orientation estimation apparatus includes a detection unit, a first extraction unit, a global region determination unit, a second extraction unit, a specification unit, and a calculation unit. The detection unit detects a feature point from a target image in which a target object whose position and orientation is to be estimated is captured and a registered image in which the target object is captured in a predetermined position and orientation. The first extraction unit selects a feature point indicating the feature of the target object from among a plurality of feature points detected from the target image, and local feature information indicating a feature around the selected feature point in the target image; Local feature information indicating features around a plurality of feature points in the registered image is extracted. The global region setting unit sets a global region including a plurality of the feature points in each of the target image and the registered image. The second extraction unit extracts global feature information in the target image based on the feature points and the global region of the target image, and extracts global feature information in the registered image based on the feature points and the global region of the registered image To do. The identifying unit determines whether the feature point of the target image and the registered image are based on the similarity of the local feature information and the similarity of the global feature information in the feature point pair of the feature point of the target image and the feature point of the registered image. A feature point pair group indicating the correspondence with the feature point is specified. The calculation unit calculates the position and orientation of the target object shown in the target image based on the identified feature point pair group.

  According to the above aspect, the estimation accuracy of the position and orientation of the object based on the image is improved.

It is a figure which shows the example of application of a position and orientation estimation apparatus. It is a figure which shows the functional structure of the position and orientation estimation apparatus which concerns on 1st Embodiment. It is a figure which shows the structure of the feature point pair group extraction part which concerns on 1st Embodiment. It is a flowchart explaining the process which the position and orientation estimation apparatus which concerns on 1st Embodiment performs. It is a flowchart explaining the content of the feature point detection process. It is a flowchart explaining the content of the global region determination process which concerns on 1st Embodiment. It is a flowchart explaining the content of the feature point pair group extraction process which concerns on 1st Embodiment. It is a flowchart explaining the content of the global feature information extraction process which concerns on 1st Embodiment. It is a figure which shows the example of a registration image and a target image. It is a figure explaining the extraction method of local feature information. It is a figure explaining the determination method of the global area | region which concerns on 1st Embodiment. It is a figure explaining the global feature information of a target image. It is a figure explaining the global feature information of a registration image. It is a figure which shows the example of the target image feature information and registered image feature information after extracting global feature information. It is a figure explaining the calculation method of a position and orientation, and the conversion method of a label sticking position. It is a figure explaining the modification of the extraction method of global feature information. It is a flowchart explaining the content of the global area | region determination process which concerns on 2nd Embodiment. It is a flowchart (the 1) explaining the content of the feature point pair group extraction process which concerns on 2nd Embodiment. It is a flowchart (the 2) explaining the content of the feature point pair group extraction process which concerns on 2nd Embodiment. It is a flowchart (the 3) explaining the content of the feature point pair group extraction process which concerns on 2nd Embodiment. It is a figure explaining the determination method of the global area | region which concerns on 2nd Embodiment. It is a figure explaining the extraction method of the global feature information in a target image. It is a figure explaining the extraction method of the global feature information in a registration image. It is a figure which shows the example of the target image feature information and registered image feature information after extracting global feature information. It is a figure explaining the production | generation method of the set of vertex pairs. It is a figure explaining the relationship between the rotational symmetry of a global area | region, and the set of the vertex pair which can be produced | generated. It is a figure explaining the relationship with the global feature information compared with a vertex pair. It is a flowchart (the 1) explaining the content of the feature point pair group extraction process which concerns on 3rd Embodiment. It is a flowchart (the 2) explaining the content of the feature point pair group extraction process which concerns on 3rd Embodiment. It is a figure which shows the functional structure of the position and orientation estimation apparatus which concerns on 4th Embodiment. It is a flowchart explaining the process which the position and orientation estimation apparatus which concerns on 4th Embodiment performs. It is a figure explaining the method of removing an obstruction from a target image. It is a figure which shows the hardware constitutions of a computer.

[First Embodiment]
The position / orientation estimation apparatus according to the present embodiment is applied to, for example, estimation of the position / orientation of an object flowing on a line such as a belt conveyor in a work line such as a factory. Here, the position and orientation of the target means information including the position and orientation of the target. One example of this type of work line is a work line for attaching a label to an object.

FIG. 1 is a diagram illustrating an application example of the position and orientation estimation apparatus.
As shown in FIG. 1, the position / orientation estimation apparatus 1 is connected to an imaging apparatus 2 and a projector 3.

  The imaging device 2 captures an image of the object 5 conveyed by the belt conveyor 4. In the belt conveyor 4 shown in FIG. 1, the right end is the upstream end. The object 5 placed on the conveyor belt 401 at the upstream end of the belt conveyor 4 moves to the left. The imaging device 2 is installed above the conveyor belt 401 in a direction for imaging the conveyor belt 401. The imaging device 2 performs fixed-point imaging of a predetermined imaging area 410 in the conveyance path in which the conveyance belt 401 moves around, and sequentially transmits the captured images to the position / orientation estimation device 1.

  The position / orientation estimation apparatus 1 detects an object that has passed through the imaging area 410 based on an image obtained by imaging the object 5 that passes through the imaging area 410 and an image of the object that has been imaged in advance at a predetermined position and orientation. 5 is estimated. In addition, the position / orientation estimation apparatus 1 according to the present embodiment generates display data of an image 6 indicating the label application position on the object 5 based on the estimated position / orientation of the object 5 and outputs the display data to the projector 3. .

  The projector 3 projects the image 6 onto the object 5 based on the display data received from the position / orientation estimation apparatus 1. The projector 3 projects an image 6 indicating the label application position on the object 5 at a timing when the object 5 conveyed by the conveyance belt 401 passes in front of the worker 8 applying the label 7. As a result, the worker 8 can know the label application position on the object 5 without confirming with a paper instruction or the like, and the work can be made more efficient.

  In the work line 10 illustrated in FIG. 1, the position / orientation estimation apparatus 1 determines the position / orientation of the object 5 </ b> A within a period from when the object 5 </ b> A passes through the imaging region 410 to reach the worker 8. The display data of the image 6 is generated and output to the projector 3. For this reason, the projector 3 projects the image 6 indicating the label application position on the object 5A onto the object 5A at the timing when the object 5A reaches the worker 8.

  In addition, when the object 5B located on the upstream side of the imaging area 410 in FIG. 1 passes through the imaging area 410 and reaches the worker 8, the projector 3 displays an image 6 corresponding to the position and orientation of the object 5B. Is projected onto the object 5B.

  The position and orientation of the object 5 placed on the transport belt 401 is not always constant. For example, the object 5A and the object 5B shown in FIG. 1 have different side directions on the label attachment surface. Therefore, the label sticking position of the object 5A is different from the label sticking position of the object 5B. Therefore, by estimating the position and orientation for each object 5 and projecting the label application position, the worker 8 correctly puts the label 7 on the predetermined position of each object 5 according to the projected image 6 of the application position. It becomes possible to affix.

FIG. 2 is a diagram illustrating a functional configuration of the position / orientation estimation apparatus according to the first embodiment.
As illustrated in FIG. 2, the position / orientation estimation apparatus 1 according to the present embodiment includes a position / orientation estimation unit 110, a storage unit 120, and a projection image generation unit 130.

  The position / orientation estimation unit 110 estimates the position / orientation of the object shown in the image acquired from the imaging device 2. The position / orientation estimation unit 110 includes an image acquisition unit 111, a feature point detection unit 112, a local feature information extraction unit 113, a global region determination unit 114, a feature point pair group extraction unit 115, and a position / orientation calculation unit 116. ,including.

  The storage unit 120 stores data referred to by the position / orientation estimation unit 110, calculated data, extracted data, and data referred to by the projection image generation unit 130. The storage unit 120 stores various data including a registered image 121, a target image 122, registered image feature information 123, target image feature information 124, similarity information 125, and a label pasting position 129. . Here, the registered image 121 is an image of an object captured in advance at a predetermined position and orientation, and the target image 122 is an image showing the object whose position and orientation is estimated (in other words, a belt imaged by the imaging device 2). The image of the object conveyed by the conveyor 4). The storage unit 120 is not limited to the registered image 121 for one type of object, and may store a plurality of registered images obtained by individually capturing a plurality of types of objects.

  The projection image generation unit 130 generates an image to be projected on the object based on the estimation result of the position and orientation of the object in the position and orientation estimation unit 110 and the label pasting position 129 stored in the storage unit 120.

  As described above, the position / orientation estimation unit 110 of the position / orientation estimation apparatus 1 according to the present embodiment includes the image acquisition unit 111, the feature point detection unit 112, the local feature information extraction unit 113, and the global region determination unit 114. , A feature point pair group extraction unit 115 and a position / orientation calculation unit 116.

  The image acquisition unit 111 acquires an image (target image 122) of the imaging range 401 imaged by the imaging device 2.

  The feature point detection unit 112 detects a feature point from a target image 121 in which the target object 5 that is a position and orientation estimation target is shown, and a registered image 121 in which the target object 5 is shown in a predetermined position and orientation. Part. The position of the feature point of the registered image 121 is stored in the storage unit 120 as one of the registered image feature information 123, and the position of the feature point of the target image 122 is stored in the storage unit 120 as one of the target image feature information 124. .

  The local feature information extraction unit 113 is a first extraction unit that extracts local feature information used for estimating the position and orientation of the target object from each of the registered image 121 and the target image 122. The local feature information is information representing features (for example, color distribution, edges, etc.) on the image around the feature points detected from the image (local region). The local feature information extraction unit 130 extracts local feature information for all feature points detected from the registered image 121 and local feature information for all feature points detected from the target image 122. Further, the local feature information extraction unit 130 excludes feature points that are unnecessary for estimation of the position and orientation of the target object, such as feature points of the background region in the target image 122, based on the extracted local feature information of each feature point. . The local feature information of each feature point of the registered image 121 is associated with the position of the feature point and stored in the storage unit 120 as one of the registered image feature information 123. The local feature information of each feature point after removing unnecessary feature points in the target image 122 is associated with the position of the feature point and stored in the storage unit 120 as one of the registration screen feature information 124.

  The global area determination unit 114 determines a global area to be set for each of the registered image and the target image according to a predetermined condition. The global region determination unit 114 includes a global region setting unit (not shown) that sets a global region including a plurality of feature points in each of the registered image and the target image. The global region determination unit 114 according to the present embodiment sets a region surrounded by the outline of a minimum circumscribed figure (minimum circumscribed polygon) including all feature points in the image as a global region. The global region of the registered image 121 is stored in the storage unit 120 as one of the registered image feature information 123, and the global region of the target screen 122 is stored in the storage unit 120 as one of the registered screen feature information 124.

  The feature point pair group extraction unit 115 extracts a set of feature point pairs (a feature point pair group) representing a correspondence relationship between the feature points of the target image 122 and the feature points of the registered image 121. The feature point pair extraction unit 115 generates based on the similarity of the local feature information and the similarity of the global feature information calculated for each feature point pair based on the feature points of the target image 122 and the feature points of the registered image 121. A feature point pair group is extracted from a set of a plurality of obtained feature point pairs. Here, the global feature information is feature information that is different from local feature information about each feature point, that is, information representing the feature amount for each feature point in the image calculated based on the global information. The feature point pair group extraction unit 115 according to the present embodiment uses the distance from the feature point to the center of gravity of the global region calculated for each feature point in the image as the global feature information. The feature point pair extraction unit 115 calculates the feature point similarity for each set of feature points based on the similarity of the local feature information and the similarity of the global feature information, and uses the calculated feature point similarity as a feature point. The information is stored in the similarity information 125 of the storage unit 120 in association with the set of pairs. Then, after calculating the feature point similarity with all of a plurality of sets of feature point pairs that can be generated, the feature point pair group extraction unit 115 characterizes the set of feature point pairs having the maximum feature point similarity. Decide on a point-pair group.

  The position and orientation calculation unit 116 is based on the positional relationship between the feature points of the target image 122 and the feature points of the registered image 121 in each feature point pair included in the feature point pair group extracted by the feature point pair group extraction unit 115. The calculation unit calculates the position and orientation of the object shown in the target image 122. In the present embodiment, the position / orientation calculation unit 116 calculates, as information indicating the position / orientation of the object, a conversion formula (conversion matrix) that converts the position of the feature point of the registered image 121 into the position of the feature point of the target image 122. To do. That is, the projection image generation unit 130 converts the label application position 129 in the registered image 121 to the label application position in the target image 122 using the conversion matrix calculated by the position / orientation calculation unit 116, and an image indicating the label application position of the target object Is generated.

Next, the functional configuration of the feature point pair group extraction unit 115 will be described in detail with reference to FIG.
FIG. 3 is a diagram illustrating a configuration of the feature point pair group extraction unit according to the first embodiment.

  As shown in FIG. 3, the feature point pair group extraction unit 115 according to this embodiment includes a global feature information extraction unit 115A, a set creation unit 115B, a local feature information comparison unit 115C, and a global feature information comparison unit 115D. And a similarity calculation unit 115E. The feature point pair group extraction unit 115 includes a calculation processing control unit 115F and a feature point pair group determination unit 115G.

  The global feature information extraction unit 115A is a second extraction unit that extracts, from each of the target image 122 and the registered image 121, global feature information characterized by a feature point and a base point determined based on the global region. The global feature information extraction unit 115A according to the present embodiment extracts (calculates) the distance from the feature point to the base point as the global feature information using the center of gravity of the global region as the base point.

  The set generation unit 115B generates a set ψ of feature point pairs based on the feature points of the target image 122 and the feature points of the registered image 121.

  The local feature information comparison unit 115C compares the local feature information of the feature point of the target image 122 with the local feature information of the feature point of the registered image 121 for each feature point pair included in the feature point pair set ψ. The similarity of local feature information is calculated.

  The global feature information comparison unit 115D compares the global feature information of the feature point of the target image 122 with the global feature information of the feature point of the registered image 121 for each feature point pair included in the feature point pair set ψ. The similarity of global feature information is calculated.

  The similarity calculation unit 115E calculates the feature points in the feature point pair set ψ based on the local feature information similarity and the global feature information similarity calculated for each feature point pair included in the feature point pair set ψ. An evaluation value (feature point similarity) is calculated for the correspondence relationship.

  The calculation processing control unit 115F controls the feature point similarity calculation processing by the set creation unit 115B, the local feature information comparison unit 115C, the global feature information comparison unit 115D, and the similarity calculation unit 115E. The calculation processing control unit 115F calculates the feature point similarity for all sets of feature point pairs that can be generated, the set creation unit 115B, the local feature information comparison unit 115C, the global feature information comparison unit 115D, and the similarity calculation. The unit 115E repeatedly performs the feature point similarity calculation process.

  The feature point pair group determination unit 115G determines a feature point pair set having the highest feature point similarity among a plurality of feature point pair sets for which feature point similarity is calculated, as a feature point of the target image 122 and a registered image. A set of feature point pairs (a feature point pair group) representing a correspondence relationship with 121 feature points is determined.

  That is, the feature point pair group extraction unit 115 is roughly divided into a second extraction unit (global feature information extraction unit 115A) that extracts global feature information and a specifying unit. Here, the specifying unit determines the feature of the target image based on the similarity of the local feature information and the similarity of the global feature information in the feature point pair of the feature point of the target image 122 and the feature point of the registered image 121. A feature point pair group indicating the correspondence between the points and the feature points of the registered image is specified. The specifying unit includes a set generation unit 115B, a local feature information comparison unit 115C, a global feature information comparison unit 115D, a calculation processing control unit 115D, and a feature point pair group determination unit 115G.

  For example, the position / orientation estimation apparatus 1 according to the present embodiment repeatedly performs the processes of steps S1 to S9 illustrated in FIG. 4 at predetermined time intervals. Note that the position / orientation estimation apparatus 1 may repeatedly perform the series of processes in steps S1 to S9 as a single processing unit, or may perform the processes in steps S1 to S9 in a pipeline.

  FIG. 4 is a flowchart for explaining processing performed by the position and orientation estimation apparatus according to the first embodiment.

  As shown in FIG. 4, the position / orientation estimation apparatus 1 according to the present embodiment first acquires a target image obtained by imaging a target from the imaging apparatus 2 (step S <b> 1). The process of step S1 is performed by the image acquisition unit 111.

  Next, the position / orientation estimation apparatus 1 performs a feature point detection process (step S2) for detecting a feature point from the target image and the registered image. The feature point detection unit 112 performs the process of step S2. The feature point detection unit 112 detects a feature point from each of the captured image and the registered image according to a known feature point detection method in the field of image processing. For example, the feature point detection unit 112 calculates feature points by Speeded Up Robust Features (SURF) or Scale Invariant Feature Transform (SIFT).

  The feature point detection unit 112 stores the position of the feature point detected from the registered image in the storage unit 120 as one of the registered image feature information 123, and the position of the feature point detected from the target image is one of the target image feature information 124. Stored in the storage unit 120.

  Note that the feature point detection unit 112 registers, for example, the feature points of the registered image detected in the detection process immediately after the start of position and orientation estimation for a certain type of object until the type of the object is changed. It may be a feature point of an image. That is, the feature point detection unit 112 may detect the feature point from the registered image for the target object whose position and orientation are estimated only once after the start of the position and orientation estimation for a certain type of target object. In this case, the feature point detection unit 112 performs only the process of detecting the feature point from the target image in the second and subsequent processes of the position and orientation estimation process for one type of target object. In addition, when storing the position of the feature point of the target image that is the current processing target in the storage unit 120 as one of the target image feature information 124, the feature point detection unit 112 may erase past information. Alternatively, it may be stored in an identifiable state for each target image.

  Next, the position / orientation estimation apparatus 1 determines whether or not the object is shown in the target image (step S3). The determination in step S3 is performed by, for example, the feature point detection unit 112. The feature point detection unit 112 determines whether or not the target object is shown in the target image based on the detection result of the feature point with respect to the target image in step S2. For example, when a value (N−M) obtained by subtracting the total number M of feature points of the registered image from the total number N of feature points of the target image is smaller than the threshold value TH1, the feature point detection unit 112 captures the target object in the target image. Judge that it is not. The threshold value TH1 is, for example, a negative value (TH1 <0) set based on the total number M of feature points of the registered image. When it is determined that the target object is not shown in the target image (step S3; NO), the position / orientation estimation apparatus 1 ends the process for the target image that is currently the processing target.

  On the other hand, when the value (N−M) obtained by subtracting the total number M of feature points of the registered image from the total number N of feature points of the target image is equal to or greater than the threshold value TH1, the feature point extraction unit 112 captures the target object in the target image. It is determined that When it is determined that the target object is captured in the target image (step S3; YES), the position / orientation estimation apparatus 1 next extracts local feature information of the detected feature point from each of the target image and the registered image. (Step S4). The local feature information extraction unit 113 performs the process of step S4. The local feature information extraction unit 113 obtains local feature information representing the feature of the image around the feature point (local region) from each of the target image and the registered image according to a known local feature information extraction method in the field of image processing. Extract. For example, the local feature information extraction unit 113 extracts local feature information of each feature point by SURF or SIFT. The local feature information extraction unit 113 stores the local feature information about the feature points of the target image in the storage unit 120 as one of the target image feature information 124, and the local feature information about the feature points of the registered image as registered image feature information. One of 123 is stored in the storage unit 120.

  Note that the local feature information extraction unit 113, for example, obtains local feature information about the feature points of the registered image extracted in the extraction process immediately after the start of position / orientation estimation for a certain type of target object. Until it is changed, local feature information of feature points of the registered image may be used. That is, the local feature information extraction unit 113 extracts the local feature information from the registered image for the target object whose position and orientation is estimated only once immediately after the start of the position and orientation estimation for a certain type of target object. Good. In this case, the local feature information extraction unit 113 performs only the process of extracting the local feature information from the target image in the second and subsequent steps in the position and orientation estimation process for one type of target object.

  Next, the position / orientation estimation apparatus 1 excludes feature points that are unnecessary for position / orientation estimation based on the local feature information (step S5). For example, the local feature information extraction unit 113 performs the process of step S5. Based on the extracted local feature information, the local feature information extraction unit 113 is detected from a background region (for example, a transport belt 401 that transports the target object 5) among a plurality of feature points detected from the target image. Feature points are excluded as feature points that are unnecessary for position and orientation estimation. For example, the local feature information extraction unit 113 calculates, for each of a plurality of feature points of the target image, the similarity of the local feature information with one or more feature points in the registered image, and the similarity is equal to or less than a threshold value. Exclude feature points. When there is a feature point to be excluded, the local feature information extraction unit 113 deletes the position of the corresponding feature point and the local feature information from the target image feature information 124.

  Next, the position / orientation estimation apparatus 1 performs a global region determination process (step S6) for determining the global region of the target image and the registered image. The global area determination process in step S6 is performed by the global area determination unit 114. The global region determination unit 114 according to the present embodiment determines the global region of the target image based on the position of the feature point of the target image after removing unnecessary feature points, and based on the position of the feature point of the registered image. To determine the global area of the registered image. The global area determination unit 114 according to the present embodiment calculates a minimum circumscribed polygon including all feature points in each of the target image and the registered image, and sets the area surrounded by the outline of the minimum circumscribed polygon as the global area. decide. The global region determination unit 114 stores information representing the global region of the target image in the storage unit 120 as one of the target image feature information 124, and information representing the global region of the registered image as one of the registered image feature information 123. Store in the storage unit 120.

  Next, the position / orientation estimation apparatus 1 performs a feature point group extraction process (step S7) for extracting a feature point pair group representing a correspondence relationship between the feature points of the target image and the feature points of the registered image. The process of step S7 is performed by the feature point pair group extraction unit 115. The feature point pair group extraction unit 115 generates a plurality of sets of feature point pairs based on the feature points of the target image and the feature points of the registered image, and selects a feature point pair group from the plurality of sets of feature point pairs. Extract. The feature point pair group extraction unit 115 calculates the similarity of the feature point pairs in units of sets based on the similarity of the local feature information and the similarity of the global feature information for each set of feature points. The highest feature point pair set is determined as a feature point pair group. The global feature information is information representing the relationship between the feature points and the global region in each of the target image and the registered image, and is extracted (calculated) by the global feature information extraction unit 115A of the feature point pair group extraction unit 115. The global feature information extraction unit 115A according to the present embodiment calculates the center of gravity of the global region for each of the target image and the registered image, and calculates the distance from the feature point to the center of gravity of the global region for each feature point I do. That is, in the feature point pair group extraction processing according to the present embodiment, the distance from the feature point calculated for each feature point to the center of gravity of the global region is used as the global feature information. Each time the feature point pair group extraction unit 115 calculates the similarity of the feature point pair based on the local feature information extracted for each feature point and the global feature information, the calculated similarity is used as a set of feature point pairs. Corresponding information is stored in the similarity information 125. Then, after calculating the similarity for all of the sets of feature point pairs that can be generated, the feature point pair group extraction unit 115 has the maximum similarity from the set of feature point pairs accumulated in the similarity information 125. A set of feature point pairs is specified, and the set of feature point pairs is determined as a feature point pair group. It should be noted that the feature point pair group having the maximum similarity in the feature point pair extraction unit 115 is used as the feature point pair group because the higher the similarity is, the larger the calculated value is. This is a case of calculating the degree. In other words, when the similarity is calculated according to a similarity calculation method in which the similarity is a positive value and the calculated value decreases as the similarity increases, the feature point pair group extraction unit 115 determines that the similarity is the minimum value. A set of feature point pairs (closest to 0) is determined as a feature point pair group.

  Next, the position / orientation estimation apparatus 1 calculates the position / orientation of the target object in the target image based on the determined feature point pair group (step S8). The position / orientation calculation unit 116 performs the process of step S8. Note that the position / orientation calculation unit 116 according to the present embodiment calculates a conversion matrix that converts the position of the feature point of the registered image into the position of the feature point of the target image as the position / orientation of the target object. The position / orientation calculation unit 116 converts the transformation matrix based on the positional relationship between the feature points of the target image and the feature points of the registered image in each of a plurality of feature point pairs included in the feature point pair group according to a known coordinate transformation method. Is calculated.

  Next, the position / orientation estimation apparatus 1 generates and outputs an image to be projected onto the object based on the label attaching position 129 of the storage unit 120 and the calculated position / orientation (conversion matrix) of the object (step S9). ). The projection image generation unit 130 performs the process in step S9. The label application position 129 includes information indicating the label application position on the object in the registered image. The projection image generation unit 130 reads the label application position 129 in the storage unit 120, and uses the transformation matrix calculated by the position / orientation calculation unit 116 to indicate the coordinates indicating the label application position in the registered image and the label application position in the target image. Convert to the coordinates to represent. Thereafter, the projection image generation unit 130 generates an image in which the label pasting position is displayed at the coordinates representing the label pasting position in the target image, and outputs the image to the projector 3. When the image generated by the projection image generation unit 130 is output, the position / orientation estimation apparatus 1 ends the process on the target image that is the current processing target.

  As described above, in the process of estimating the position and orientation of the target object according to this embodiment, the similarity of the local feature information extracted from around the feature point of the target object, the centroid and the feature point of the global area set in the image The position and orientation of the object is estimated based on the similarity of the global feature information that represents the relationship between

  Hereinafter, with reference to FIG. 5 to FIG. 8, each process of steps S2, S6, and S7 in the flowchart of FIG. 4 will be described in detail.

  The feature point detection unit 112 performs the process (feature point detection process) in step S2 in the flowchart of FIG. The feature point detection unit 112 performs the process shown in FIG. 5 as the feature point detection process.

FIG. 5 is a flowchart for explaining the content of the feature point detection process.
The feature point detection unit 112 first detects a feature point from the target image and stores it in the storage unit 120 (step S201). In step S201, the feature point detection unit 112 detects feature points P _n (n = 1, 2,..., N ′) from the target image according to a known detection method, and determines the position of the detected feature points as the target image. It is stored (stored) in the storage unit 120 as one of the feature information 124.

  Next, the feature point detection unit 112 refers to the registered image feature information 123 and determines whether or not a feature point of the registered image has been detected. When the feature point of the registered image has been detected (step S202; YES), the feature point detection unit 112 ends the feature point detection process.

On the other hand, when the feature point of the registered image has not been detected yet (step S202; NO), the feature point detecting unit 112 next detects the feature point from the registered image and stores it in the storage unit 120 (step S203). In step S203, the feature point detection unit 112 detects feature points PU _m (m = 1, 2,..., M) from the registered image according to a known detection method, and determines the position of the detected feature points as the target image feature. The information is stored (held) in the storage unit 120 as one piece of information 124.

  The feature point detection unit 112 performs the process of detecting the feature point of the target image and the process of detecting the feature point of the registered image according to the same feature point detection method (algorithm). Therefore, the processing result (the number and position of the detected feature points) when the process of detecting the feature points from the registered image in which the position and orientation of the target object are determined is almost the same. Therefore, when the feature point of the registered image has already been detected in the process of detecting the position and orientation of one type of target object, the feature point detection unit is omitted by omitting the feature point detection process (step S202). The processing load 112 can be reduced.

  Although description with reference to the flowchart is omitted, in the process of extracting local feature information of each feature point (step S4), the local feature information extraction unit 113 obtains the local feature information in the same procedure as in the flowchart of FIG. Extract. That is, when the local feature information about the feature point of the feature image has already been extracted, the local feature information extraction unit 113 omits the process of extracting the local feature information about the feature point of the feature image, and the feature point of the target image Only the process of extracting local feature information about is performed.

  After performing the process of extracting local feature information (step S4) and the process of excluding feature points unnecessary for estimating the position and orientation of the target object (step S5), the position and orientation estimation apparatus 1 performs the global region of step S6. Perform decision processing. The global area determination process is performed by the global area determination unit 114. The global area determination unit 114 according to the present embodiment performs the process shown in FIG. 6 as the global area determination process.

  FIG. 6 is a flowchart for explaining the contents of the global area determination processing according to the first embodiment.

First, the global area determination unit 114 calculates a minimum circumscribed polygon including each feature point P _n in the target image, and stores the minimum circumscribed polygon in the storage unit 120 as the global area D of the target image (step S601). . In step S601, the global area determination unit 114 determines all the features based on the position information of the feature points P _n (n = 1, 2,..., N) of the target image stored in the target image feature information 124. Calculate the minimum circumscribed polygon containing the points. Here, the feature points used for the calculation of the minimum circumscribed polygon are the feature points of the target image after the feature points unnecessary for the position and orientation estimation are removed in step S5. That is, the total number N of feature points Pn used in step S601 is N <N ′ when one or more feature points are excluded in step S5, and N when no feature points are excluded in step S5. = N '. The global area determination unit 114 calculates a minimum circumscribed polygon including all feature points of the target image according to a known calculation method. The global region determination unit 114 determines the region surrounded by the calculated contour of the minimum circumscribed polygon as the global region D of the target image, and stores the information representing the global region D as one of the target image feature information 124. To store.

  Next, the global area determination unit 114 refers to the registered image feature information 123 and determines whether or not the global area DU of the registered image has been determined (step S602). When the global area DU of the registered image has been determined (step S602; YES), the global area determination unit 114 ends the global area determination process.

On the other hand, when the global area DU of the registered image has not yet been determined (step S602; NO), the global area determining unit 114 next calculates a minimum circumscribed polygon including each feature point PU _m in the registered image, The minimum circumscribed polygon is held in the storage unit 120 as the global area DU of the registered image (step S603). In step S <b> 603, the global area determination unit 114 determines all feature points based on the position information of the registered image feature points PU _m (m = 1, 2,..., M) stored in the registered image feature information 123. Compute the minimum circumscribed polygon containing. The global area determination unit 114 calculates a minimum circumscribed polygon including all feature points of the registered image according to the same calculation method as that used in step S601. In this case, the global area determination unit 114 determines the area surrounded by the calculated contour of the minimum circumscribed polygon as the global area DU of the registered image, and uses the information representing the global area DU as one of the registered image feature information 123. The data is stored in the storage unit 120 and the global area determination process is terminated.

  Note that the flowchart of FIG. 5 is merely an example of a global area determination process according to the present embodiment. In the global area determination processing according to the present embodiment, for example, the determination in step S602 may be performed first. In this case, if the global area DU of the registered image has been determined (step S602; YES), the global area determination unit 114 performs only the process of step S601. On the other hand, when the global area DU of the registered image has not yet been determined (step S602; YES), the global area determination unit 114 performs the process of step S601 and the process of step S603.

  After performing the global region determination process, the position / orientation estimation apparatus 1 performs the feature point pair group extraction process in step S7. The feature point pair group extraction processing is performed by the feature point pair group extraction unit 115. The feature point pair group extraction unit 115 according to the present embodiment performs the processing shown in FIGS. 7 and 8 as feature point pair group extraction processing.

  FIG. 7 is a flowchart for explaining the content of the feature point pair group extraction processing according to the first embodiment. FIG. 8 is a flowchart for explaining the contents of the global feature information extraction processing according to the first embodiment.

  As shown in FIG. 7, the feature point pair group extraction unit 115 first performs global feature information extraction processing (step S701) for extracting global feature information from each of the target image and the feature image. The process of step S701 is performed by the global feature information extraction unit 115A of the feature point pair group extraction unit 115. The global feature information extraction unit 115 according to the present embodiment calculates the distance from the feature point to the center of gravity of the global region D for each feature point of the target image, and from the feature point to the global region DU for each feature point of the registered image. And calculating the distance to the center of gravity (see FIG. 8). The global feature information extraction unit 115 sets the distance from the feature point calculated for each feature point of the target image to the center of gravity of the global region D as global feature information for each feature point of the target image, and uses the global feature information as the target image feature It is stored in the storage unit 120 as one piece of information 124. Similarly, the global feature information extraction unit 115 sets the distance from the feature point calculated for each feature point of the registered image to the center of gravity of the global region DU as the global feature information for each feature point of the registered image, and uses the global feature information. It is stored in the storage unit 120 as one of the registered image feature information 123.

  Next, the feature point pair group extraction unit 115 sets a variable k for identifying a set of feature point pairs to 0 (step S702). The calculation process control unit 115F performs the process of step S702.

  After step S702, the feature point pair group extraction unit 115 calculates feature point similarity in each of a plurality of sets of feature point pairs based on the local feature information and the global feature information (steps S703 to S707). )I do. The processing in steps S703 to S707 is performed by the set generation unit 115B, the local feature information comparison unit 115C, the global feature information comparison unit 115D, and the similarity calculation unit 115E under the control of the calculation processing control unit 115F.

When the process of step S702 is completed, the feature point pair group extraction unit 115 next generates a set of feature points ψ _k based on the feature points of the target image and the feature points of the registered image (step S703). The set generation unit 115B performs the process of step S703. The set generation unit 115B is configured to generate a feature point of the target image and a feature point of the registered image according to a known generation method for generating a set of element pairs by elements included in the first set and elements included in the second set. Generates a set ψ _k of feature point pairs. Note that the set generation unit 115B can generate based on the total number N of feature points of the target image and the total number M of feature points of the registered image in one process of step S703 according to a predetermined generation rule (generation algorithm). A set of a set of feature point pairs is generated.

Next, the feature point pair group extraction unit 115 calculates the similarity SL _{n, m} of local feature information and the similarity SC _{n, m of} global feature information in each feature point pair included in the set ψ _k of feature point pairs. Is calculated (step S704). The local feature information comparison unit 115C and the global feature information comparison unit 115D perform the processing in step S704. For each feature point pair, the local feature information comparison unit 115C compares the local feature information of the feature point of the target image stored in the target image feature information 124 and the feature point of the registered image 121 stored in the registered image feature information 123. The local feature information is compared to calculate the similarity SL _{n, m} of the local feature information. The local feature information comparison unit 115C calculates the similarity SL _{n, m} between the local feature information of the feature point of the target image 122 and the local feature information of the feature point of the registered image 121 according to a known similarity calculation method. The global feature information comparison unit 115D, for each feature point pair, the global feature information of the feature point of the target image stored in the target image feature information 124 and the global feature information of the registered image stored in the registered image feature information 123. The similarity SCn _{, m} of the global feature information is calculated by comparing with the feature information. The global feature information comparison unit 115D has a relationship between the similarity between the compared global feature information and the calculated similarity SC _{n, m} , the similarity between the compared local feature information, and the calculated similarity SL. _The similarity SC _{n, m} is calculated by a calculation method having the same relationship as that of _{n, m} . For example, when the compared local feature information is more similar, the similarity SL _{n, m} value is larger, and the global feature information comparing unit 115D is more similar to the compared global feature information as the similarity SC. _The similarity SC _{n, m} is calculated by a calculation method in which the value of _{n, m} increases.

Next, the feature point pair group extraction unit 115 performs feature points on the set ψ _k of feature point pairs based on the similarity SL _{n, m} of the local feature information and the similarity SC _{n, m of} the global feature information. A similarity score _Sk is calculated (step S705). The similarity calculation unit 115E performs the process of step S705. The similarity calculation unit 115E calculates, for example, the feature point similarity S _k for the feature point pair set ψ _k by the following equation (1).

SL _{n, m} and SC _{n, m} in equation (1) are the similarity of the local feature information in each feature point pair (P _n , PU _m ) included in the set ψ _k of feature point pairs, and the global feature, respectively. It is the similarity of information. In addition, α and β in the formula (1) are arbitrary constants, respectively.

The similarity calculation unit 115E associates the calculated feature point similarity S _k with the feature point set ψ _k and stores it in the storage unit 120 as one of the similarity information 129, and calculates the feature point similarity S _k . This is notified to the calculation processing control unit 115F.

When the feature point similarity S _k for the set [psi _k feature point pairs are calculated, calculation control section 115F determines whether to generate all the set of feature point pairs that can generate (step S706) . In step S706, the calculation processing control unit 115F calculates the total number of sets that can be generated based on the total number N of feature points of the target image and the total number M of feature points of the registered image, and a variable k that identifies the generated set. It is determined whether or not all sets have been generated. For example, when there are K sets of feature point pairs that can be generated, the calculation processing control unit 115F determines whether or not k ≧ K−1 (since the variable k starts from 0). When k <K−1, the calculation processing control unit 115F determines that there is a set of feature point pairs that have not been generated.

  If there is a set of feature point pairs that have not been generated (step S706; NO), the calculation processing control unit 115F updates the variable k that identifies the set of feature point pairs to k + 1 (step S707). Thereafter, the calculation processing control unit 115F causes the set generation unit 115B, the local feature information comparison unit 115C, the global feature information comparison unit 115D, and the similarity calculation unit 115E to perform the processes of steps S703 to S705.

  On the other hand, when all sets of feature point pairs have been generated (step S706; YES), the calculation processing control unit 115F next causes the feature point pair group determination unit 115G to determine the feature point pair group. The feature point pair group determination unit 115G refers to the similarity information 125 of the storage unit 120, and determines a set of feature point pairs having the maximum feature point similarity as a feature point pair group (step S708). When the feature point pair group determination unit 115G determines the feature point pair group, the feature point pair group extraction process ends.

Note that the feature point pair set having the maximum feature point similarity in step S708 is determined as a feature point pair group by a calculation method in which the higher the similarity, the larger the value calculated. _This is a case where _k is calculated. That is, the similarity is a positive value, and when calculating the feature point similarity S _k by calculating how the value of the degree of similarity is calculated higher decreases, the feature point pair group determination unit 115G, in step S708 Then, a set of feature point pairs having a minimum feature point similarity is determined as a feature point pair group.

  As described above, in the feature point pair group extraction processing according to the present embodiment, the similarity of the local feature information and the similarity of the global feature information are calculated for each feature point pair included in the set of feature points. Based on the similarity, a feature point similarity for evaluating the similarity of the feature point pair in the entire set is calculated.

  Here, the global feature information extraction processing (step S701) according to the present embodiment will be described again with reference to FIG. FIG. 8 is a flowchart for explaining the contents of the global feature information extraction processing according to the first embodiment.

  As shown in FIG. 8, in the global feature information extraction process, the global feature information extraction unit 115A first calculates the center of gravity G of the global region D of the target image (step S70101). In step S70101, the global feature information extraction unit 115A calculates the centroid G of the global region D of the target image according to a known calculation method.

Next, the global feature information extraction unit 115A calculates a distance C _n between the feature point P _n and the center of gravity G for each feature point P _n of the target image, and the storage unit 120 uses the calculated distance C _n as the global feature information. (Step S70102). In step S70102, the global feature information extraction unit 115A calculates a distance C _n from the position (pixel) of the feature point P _{n to} the position (pixel) of the center of gravity G in the target image. The global feature information extraction unit 115A stores the calculated distance C _n in the target image registration information 124 of the storage unit 120 as the global feature information of each feature point.

  Next, the global feature information extraction unit 115A determines whether or not the global feature information for each feature point of the registered image has been extracted (step S70103). When the global feature information for each feature point of the registered image has been extracted (step S70103; YES), the global feature information extraction unit 115A ends the global feature information extraction process.

On the other hand, if the global feature information has not yet been extracted (step S70103; NO), the global feature information extraction unit 115A next calculates the center of gravity GU of the global region DU of the registered image (step S70104). In step S70104, the global feature information extraction unit 115A calculates the center of gravity GU of the global region DU of the registered image according to a known calculation method. After step S70104, the global feature information extraction unit 115A calculates the distance CU _m between the feature point PU _m and the center of gravity GU for each feature point PU _m of the registered image, and stores the calculated distance CU _m as the global feature information. The data is held in the unit 120 (step S70105). In step S70105, the global feature information extraction unit 115A calculates a distance CU _m between the position (pixel) of the feature point PU _{m and} the position (pixel) of the center of gravity GU in the registered image. The global feature information extraction unit 115A stores the calculated distance CU _m as global feature information of each feature point in the registered image feature information 123 of the storage unit 120, and ends the global feature information extraction process.

  Hereinafter, with reference to FIGS. 9 to 15, the position / orientation estimation processing of the object according to the present embodiment will be described in detail. In the following description, the position / orientation estimation process of the object 5 in the work line 10 for attaching the label 7 to a predetermined position of the object 5 shown in FIG.

FIG. 9 is a diagram illustrating an example of a registered image and a target image.
FIG. 9A shows an example of a registered image 121 of the object 5. The registered image 121 is an image obtained by imaging a substantially square label sticking surface of the object 5 from the front direction, and the outer periphery of the registered image 121 substantially matches the outline of the label sticking surface of the object 5. Three triangular concave portions 501, 502, and 503 are formed on the label affixing surface of the object 5 shown in the registered image 10.

  The registered image 121 has the horizontal direction (x direction) of the image parallel to the conveying direction of the object 5 on the work line 10, and the object 5 shown in the registered image 121 has a predetermined rule at the time of conveyance. Position and posture. That is, in the work line 10, the first concave portion 501, the second concave portion 502, and the third concave portion 503 are the lower left, upper right, and lower right of the label application surface, respectively, and one of the label application surfaces. It is defined that the object 5 is conveyed in a position and orientation in which the opposite side is parallel to the x direction.

  Thus, in the work line 10 for applying a label to a predetermined position of the target object 5, the position and orientation of the target object 5 at the time of transportation are defined, and the line (conveyor belt 401) is set to the position and orientation according to the regulation. ) Place the object 5 on top. However, in the actual work line 10, for example, as shown in FIG. 9B, the opposite side that makes the position and orientation of the object 5 parallel to the transport direction (x direction) on the label attachment surface is the transport direction. The position and orientation may be inclined by the angle θ1.

  FIG. 9B shows an example of a target image 122 obtained by imaging the target object 5 that is transported by the transport belt 401. The position and orientation of the object 5 in the target image 122 in FIG. 9B is different from the position and orientation in the registered image 121 (that is, the specified position and orientation).

  When the position and orientation of the target object 5 to which the label is applied are different from the specified position and orientation, the label application position projected on the target object 5 changes. For this reason, in the work line 10 for applying a label, the label application position to be projected on the object 5 is changed according to the position and orientation of the object 5 to be conveyed. In the present embodiment, the target object conveyed by the conveyor belt 401 based on the correspondence between the feature point of the target object 5 in the registered image 121 and the characteristic point of the target object 5 in the target image 122 captured by the imaging device 2. 5 is estimated (calculated). In the process of estimating the position and orientation of the object 5 according to the present embodiment, first, feature points are detected from the registered image 121 and the target image 122 (step S2), and local feature information about the detected feature points is extracted (step S2). Step S4).

FIG. 10 is a diagram illustrating a method for extracting local feature information.
In the position / orientation estimation apparatus 1 according to the present embodiment, the feature point detection unit 112 detects feature points from the registered image 121 and the target image 122 (step S2). The feature point detection unit 112 performs a process of detecting the feature point PU _m from the registered image 121 (step S203) and a process of detecting the feature point P _n from the target image 122 (step S201) according to a known detection method. .

FIG. 10A shows three feature points PU _m (m = 1, 2, 3) detected from the registered image 121. The positions of the three feature points PU _m detected from the registered image 121 are respectively the positions of the feature points PU _m from the origin when, for example, the pixel at the upper left corner in the registered image 121 is the origin (0.0). This is expressed by the number of pixels in the x direction up to the position (pixel) and the number of pixels in the y direction.

On the other hand, FIG. 10B shows five feature points P _n (n = 1 to 5) detected from the target image 122. The positions of the five feature points P _n detected from the target image 122 are the positions of the feature points P _n from the origin when the pixel at the upper left corner in the target image 122 is the origin (0.0), for example. This is expressed by the number of pixels in the x direction up to the position (pixel) and the number of pixels in the y direction.

  After detecting the feature points from the registered image 121 and the target image 122, the position / orientation estimation apparatus 1 determines whether or not the target object 5 is shown in the target image 122 (step S3). The process of detecting feature points from the registered image 121 (step S203) and the process of detecting feature points from the target image 122 (step S201) are performed by the same detection method (detection algorithm). Therefore, when a target object is shown in the target image 122, the number of feature points detected from the target image 122 is substantially equal to the number of feature points detected from the registered image 121. Therefore, in step S3, as described above, the difference (N−M) between the number N of feature points detected from the target image 122 and the number M of feature points detected from the registered image 121 is greater than or equal to the threshold value TH1. In some cases, it is determined that the target object 5 is reflected in the target image 122.

When the target object 5 is reflected in the target image 122, the local feature information extraction unit 113 then performs local feature information on each feature point PU _m of the registered image 121 and each feature point P _n of the target image 122. The local feature information is extracted (step S4). The local feature information extraction unit 113 performs processing for extracting local feature information for each feature point of the registered image 121 and processing for extracting local feature information for each feature point of the target image 122 according to a known extraction method. Do.

In the process of extracting local feature information for each feature point of the registered image 121, the local feature information extracting unit 113, for example, includes feature points PU ₁ and PU ₂ in the registered image 121 as shown in FIG. , PU ₃ , and local regions E ₁ , E ₂ , E ₃ are set. Then, the local feature information extraction unit 113 selects each feature point PU ₁ , PU ₂ , PU ₃ based on information such as pixel values of the pixels in the local regions E ₁ , E ₂ , E ₃ in the registered image 121. Extract local feature information. Further, the process of extracting the local feature information about the feature points of the target image 122, the local feature information extraction unit 113, for example, as shown in (b) of FIG. 10, the feature points in the target image 122 P ₁ ~ Local regions E _{1 to} E ₅ centering on each of P ₅ are set. Then, the local feature information extraction unit 133 extracts local feature information of the feature points P _{1 to} P ₅ based on information such as pixel values of the pixels in the local regions E _{1 to} E ₅ in the target image 122. . The local regions E _{1 to} E ₅ are not limited to the circles shown in FIGS. 10A and 10B, but may be squares or other polygons. The shape and size of the local region may be appropriately set based on the unevenness or hole formation pattern on the surface of the object 5, a coating pattern (printing pattern), or the like.

By the way, in the target image 122, as shown in FIG. 10B and the like, an object other than the target object 5 such as the transport belt 401 is reflected around the target object 5 transported by the transport belt 401. Yes. Therefore, in the process of detecting the feature points from the target image 122, the feature points P ₁ , P ₂ and P _{3 on} the surface of the target object 5 are separated from the target object 5 as shown in FIG. The feature points P ₄ and P ₅ on the surface of the object (conveyor belt 401) may be detected. As described above, the feature points detected from the target image 122 may include feature points detected from the outside of the region in which the target object 5 is reflected (in other words, the background region in the target image 122). Since the feature points detected from the background region in the target image 122 do not correspond to the feature points of the target object 5 detected from the registered image, it is not necessary for estimating the position and orientation of the target object 5. Also, as shown in FIG. 10B, the image information in the local areas E ₄ and E ₅ set for the feature points detected from the background area of the target image 122 and the target 5 are detected. Similarity is low with the image information in the local regions E _{1 to} E ₃ set for the feature points. Therefore, in the present embodiment, based on the local feature information, a process (step S5) of excluding feature points that are unnecessary for estimating the position and orientation of the target object 5 from the feature points of the target image 122 is performed. As described above, the outer periphery of the registered image 121 substantially matches the contour of the object 5. Therefore, it can be considered that only the object 5 is shown in the registered image 121. Therefore, all of the plurality of feature points PU _m detected from the registered image 121 can be regarded as feature points detected from the object 5.

In the process of step S _< b _> 5, the local feature information extraction unit 113 calculates the similarity of the local feature information with one or more feature points PU _m of the registered image 121 for each feature point P _n of the target image 122. The local regions E _{1 to} E ₃ corresponding to the feature points P _{1 to} P ₃ of the target object 5 among the feature points P _{1 to} P ₅ of the target image 122 are respectively shown in FIG. The tops of the triangular recesses 501 to 503 in the object 5 are included. Similarly, the local areas E _{1 to} E ₃ in the registered image 121 include vertices of triangular recesses 501 to 503 in the object 5, respectively, as illustrated in FIG. Therefore, the local feature information of the feature points P ₁ , P ₂ , and P ₃ of the object in the target image 122 is similar to the local feature information of the feature points PU ₁ , PU ₂ , and PU _{3 in} the registered image 121, respectively. The degree becomes higher. On the other hand, the local feature information of the feature points P ₄ and P _{5 in} the background region of the target image 122 has low similarity with the local feature information of the feature points PU ₁ , PU ₂ , and PU ₃ in the registered image 121, respectively. Therefore, in the present embodiment, among the feature points of the target image 122, feature points whose similarity of local feature information with the feature points of the registered image 121 is lower than the threshold are determined as feature points of the background region, that is, the target object. Is regarded as a feature point of another object and excluded from the feature points of the target image 122. That is, in the processing in step S5, the feature points _{P 4} and _{P 5} is detected from the target image 122 shown in (b) of FIG. 10, be excluded from the feature point of the target image 122. Thereby, in the process of step S6 and step S7, the feature points of the target image 122 are the three feature points P ₁ , P ₂ , and P _{3 that} have high similarity in local feature information with the feature points of the registered image 121. It becomes.

However, as can be seen from FIGS. 10A and 10B, the three recesses 501, 502, and 503 of the object 5 have substantially the same outer shape (contour) and are centered on each feature point. Features in local regions E ₁ , E ₂ , and E ₃ are similar. For this reason, it is difficult to determine which feature points of the registered image 121 correspond to the feature points P ₁ , P ₂ , and P ₃ of the target image 122 only by the similarity of the local feature information. . Therefore, in the position and orientation estimation processing according to the present embodiment, the global feature information representing the relationship between the feature points and the global region set in the image is used, and the feature points P ₁ , P ₂ , and P ₃ is, determines whether the corresponding characteristic points of the respective registration images 121 throat. For this reason, in the position / orientation estimation processing according to the present embodiment, after the feature points unnecessary for estimating the position / orientation of the target object 5 are excluded, the global region determination for determining the global region of the target image 122 and the registered image 121 is performed. Processing (step S6) is performed.

FIG. 11 is a diagram for explaining a method of determining a global area according to the first embodiment.
In the global area determination process according to the present embodiment, a minimum circumscribed polygon including all feature points in each image is set as the global area D of the target image 122 and the global area DU of the registered image 121.

FIG. 11A shows an example of the global area DU set in the registered image 121. When the number of feature points detected from the registered image 121 is three, the minimum circumscribed polygon including the _three feature points PU ₁ , PU ₂ , and PU ₃ is three feature points PU ₁ , PU ₂ , and a triangle whose vertices are the PU _3. Therefore, the global region determination unit 114 determines a region surrounded by a triangle having the _three feature points PU ₁ , PU ₂ , and PU ₃ as vertices as the global region DU of the registered image 121.

FIG. 11B shows an example of the global region D set in the target image 122. Target image 122 in FIG. 11 (b) is the same as the target image 122 shown in (b) of FIG. 10, the feature point _{P 4} and _{P 5} of the background area are excluded by the processing of step S5 . That is, the feature points used for estimating the position and orientation of the target object 5 in the target image 122 are the three feature points P ₁ , P ₂ , and P ₃ of the target object 5. For this reason, the minimum circumscribed polygon including the _three feature points P ₁ , P ₂ , and P ₃ in the target image 122 is a triangle having the _three feature points P ₁ , P ₂ , and P ₃ as vertices. Become. Therefore, the global region determination unit 114 determines the region surrounded by the triangle having the _three feature points P ₁ , P ₂ , and P ₃ as vertices as the global region D of the target image 122.

  After determining the global region D of the target image 122 and the global region DU of the registered image 121, in the position and orientation estimation apparatus 1, the feature point pair group extraction unit 115 performs a feature point pair group extraction process (step S <b> 7). In the feature point pair group extraction processing, first, global feature information extraction processing (step S701) for extracting feature quantities (global feature information) different from local feature information based on the relationship between feature points and the global region is performed. . In the feature point group extraction process in the present embodiment, as described above, the distance between the feature point and the center of gravity of the global region is extracted as the global feature information.

  FIG. 12 is a diagram illustrating the global feature information of the target image. FIG. 13 is a diagram illustrating global feature information of a registered image. FIG. 14 is a diagram illustrating an example of target image feature information and registered image feature information after the global feature information is extracted.

  FIG. 12 is an enlarged view of the global region D set in the target image 122 of FIG. 11B and the partial rectangular region 122A including the periphery thereof.

In the global feature information extraction processing according to the present embodiment, processing for extracting global feature information for the feature points of the target image 122 and processing for extracting global feature information for the feature points of the registered image 121 are performed. In the process of extracting the global feature information about the feature points of the target image 122, the center of gravity G of the global region D of the target image 122 is calculated (step S70101), and the distance from the feature point to the center of gravity G is calculated for each feature point. (Step S70102). As shown in FIG. 12, when the global region D in the target image 122 (partial rectangular region 122A) is a triangle, the center of gravity G of the global region D has three lines connecting the midpoint of each side and the opposite vertex. It will be a crossing point. The global feature information about each feature point P ₁ , P ₂ , and P ₃ of the target image 122 is the distance C ₁ from the feature point P ₁ to the center of gravity G and the distance from the feature point P ₂ to the center of gravity G, respectively. C ₂ and the distance C ₃ from the feature point P ₃ to the center of gravity G.

Further, in the process of extracting the global feature information about the feature points of the registered image 121, the center of gravity GU of the global region DU of the registered image 121 is calculated (step S70104), and the distance from the feature point to the center of gravity G for each feature point. Is calculated (step S70104). As shown in FIG. 13, when the global area DU of the registered image 121 is a triangle, the center of gravity GU of the global area DU is a point where three lines connecting the midpoint of each side and the opposite vertex intersect. The global feature information about each feature point PU ₁ , PU ₂ , and PU ₃ of the registered image 121 is the distance CU ₁ from the feature point PU ₁ to the center of gravity GU, and the distance from the feature point PU ₂ to the center of gravity GU, respectively. CU ₂ and the distance CU ₃ from the feature point PU ₃ to the center of gravity GU.

Therefore, when the global feature information extraction process (step S701) is completed, the target image feature information 124 includes feature points P _n , local feature information L _n, and a table 1101 shown in FIG. The global feature information C _n is stored in association with each other. Similarly, when the global feature information extraction process (step S701) is finished, the registered image feature information 123 includes a feature point PU _m , local feature information LU _m, and a table 1102 shown in FIG. , Global feature information CU _m is stored in association with each other.

When the number of feature points detected from the target image 121 is N (N> 2), the local feature information L _{1 is included in} each of the _N feature points P ₁ , P ₂ ,. , _L 2, ···, and _{L N,} global feature information _{C 1, C 2, ···,} and a _{C N} associated. Similarly, when there are M feature points detected from the registered image 121 (M> 0), the local feature information LU is included in each of the _M feature points PU ₁ , PU ₂ ,. _{1, LU} 2, ···, and LU _M, global feature information _{CU 1, CU 2, ···,} and the CU _M associated.

  After extracting the global feature information of each feature point in each of the target image 122 and the registered image 121, the feature point pair group extraction unit 115 performs the processing from step S <b> 702 onward, and the feature point of the target image and the feature point of the registered image A feature point pair group that represents the correspondence relationship is extracted (determined).

In order to extract a feature point pair group, the feature point pair group extraction unit 115 generates a set ψ _k of feature point pairs based on the feature points of the target image 122 and the feature points of the registered image 121 in the set generation unit 115B (step S704). The set generation unit 115B has M features detected from the registered image at each of the N feature points P _n (n = 1, 2,..., N) detected from the target image according to a predetermined generation rule. A set ψ _k of feature point pairs in which any one of the points PU _m (m = 1, 2,..., M) is associated is generated. When k = 0 and M ≧ N, the set generation unit 115B includes, for example, a set ψ _k = {(P ₁ , PU ₁ ), (P ₂ , PU ₂ ),... including N feature point pairs. , (P _N , PU _M )}. When k = 0 and M <N, the set generation unit 115B includes, for example, a set ψ _k = {(P ₁ , PU ₁ ), (P ₂ , PU ₂ ),. .., (P _{N = M} , P _UM )}.

When k ≠ 0, the set generation unit 115B, for example, adds M feature points PU _m (m = 1, N) to each of the N feature points P _n (n = 1, 2,..., N). 2, ..., according to the method of generating the permutation in the case of associating M) on a one-to-one, to produce a set [psi _k which is not yet produced.

After generating the set ψ _k of feature point pairs, the feature point pair group extraction unit 115 calculates the similarity SL _{n, m} of the local feature information of each feature point pair in the local feature information comparison unit 115C, and the global feature The information comparison unit 115D calculates the similarity SC _{n, m} of the global feature information of each feature point pair.

The local feature information is information representing the feature of the image in the local region calculated based on the pixel value of the pixel in the local region including the feature point. Therefore, the local feature information comparison unit 115C, for each feature point pair, the local feature information L _n of the feature point P _n of the target image 122 and the feature of the registered image 121 according to a known similarity calculation method in the field of image processing. similarity _{SL n} the local feature information LU _m at point PU _m, to calculate the _m.

On the other hand, the global feature information is information representing the distance from the position of the feature point in the image to the position of the center of gravity of the global region. Therefore, for example, for each feature point pair, the global feature information comparison unit 115D calculates the global feature information C _n of the feature point P _n of the target image 122 and the global feature information CU _m of the feature point PU _m of the registered image 121. Based on the difference, the similarity SC _{n, m} of the global feature information is calculated.

After calculating the similarity SL _{n, m} of the local feature information and the similarity SC _{n, m} of the global feature information, the feature point pair group extraction unit 115 performs the feature point pair set ψ _k on the similarity calculation unit 115E. The feature point similarity _Sk is calculated. As described above, the feature point similarity S _k is a value for evaluating the correspondence between the feature points in the set of feature points ψ _k , and the similarity SL _{n, m of} the local feature information and the similarity of the global feature information It may be calculated based on the degree SC _{n, m} . For this reason, in this embodiment, for example, the feature point similarity S _k for the set ψ _k of feature point pairs is calculated by the above equation (1). That is, in this embodiment, a value obtained by multiplying the sum of the similarity SL _{n, m} of the local feature information of each feature point pair included in the feature point pair set ψ _k by the coefficient α and the global feature of each feature point pair. The sum of the information similarity SC _{n, m} and the value obtained by multiplying by the coefficient β is _{defined as} the feature point similarity S _k of the set ψ _k of feature point pairs.

The feature point pair group extraction unit 115 according to the present embodiment is a set of a plurality of (K) feature point pairs ψ _k (k = 0, 1,...) That can be generated under the control of the calculation processing control unit 115F. .., K−1), the feature point similarity _Sk is calculated. After that, the feature point pair group extraction unit 115 extracts a set of feature point pairs having the maximum feature point similarity in the feature point pair group determination unit 115F, and uses the set of feature point pairs as the feature points of the target image 122. And a pair of feature points representing the correspondence between the feature points of the registered image 121.

After the feature point pair group extraction unit 115 determines (extracts) the feature point pair group, the position / orientation estimation apparatus 1 is reflected in the target image 122 by the position / orientation calculation unit 116 based on the determined feature point pair group. The position and orientation of the target object 5 is calculated (step S8). As described above, the position / orientation calculation unit 116 according to the present embodiment is based on the position of the feature point of the target image 122 and the position of the feature point of the registered image 121 that are feature point pairs in the feature point pair group. Then, a conversion matrix for converting the position of each point (each pixel) in the registered image 121 to the position of each point in the target image 122 is calculated. The position / orientation calculation unit 116, for example, 6 in the following formula (2) based on the position of the feature point of the target image 122 and the position of the feature point of the registered image 121 that are feature point pairs in the feature point pair group. The coefficients a, b, c, d, t _x , and _ty are determined.

Coefficients in equation (2) a, b, c , d, t x, in determining _{t y} substitutes coordinates representing the position of the feature point of the registered image 121 in (x, y). Further, in the feature point pair group, the coordinates representing the position of the feature point of the target image 122 that is a feature point pair with the feature point of the registered image 121 are substituted into (x ′, y ′). The position / orientation calculation unit 116 thus substitutes the coordinates representing the position of the feature point in each of the plurality of feature point pairs into (x, y) and (x ′, y ′) in Expression (2). A determinant is generated, and coefficients a, b, c, d, t _x , and _ty are determined based on the generated determinant.

  After calculating the position and orientation (coefficient of the transformation matrix) of the target object 5, the position and orientation estimation apparatus 1 generates an image to be projected on the target object 5 in the projection image generation unit 130 and outputs the image to the projector 3 (step S9). The projection image generation unit 130 reads the label application position in the registered image 121 from the label application position 129 in the storage unit 120, and converts the read application position into the label application position in the target image 122 using a conversion matrix.

FIG. 15 is a diagram for explaining a position and orientation calculation method and a label application position conversion method.
A feature point pair group (feature point) representing the correspondence between the feature points P ₁ , P ₂ , and P _{3 of} the target image 122 shown in FIG. 15 and the feature points PU ₁ , PU ₂ , and PU ₃ of the registered image 121. Ψ) is set to ψ = {(P ₁ , PU ₁ ), (P ₂ , PU ₂ ), (P ₃ , PU ₃ )}. For this reason, the position / orientation calculation unit 116 determines the positions of the three feature points PU ₁ , PU ₂ , and PU _{3 in} the registered image 121 as the three feature points P ₁ , P ₂ , and P in the target image 122, respectively. A conversion matrix to be converted into position ₃ is calculated. Although only three feature points are shown in FIG. 15, a larger number of feature points are actually detected. In the transformation matrix of Expression (2) calculated by the position and orientation calculation unit 116, the matrix of 2 rows and 2 columns of the first term on the right side is a linear transformation component, and the second term on the right side is a translation component. Therefore, the position of each point (pixel) in the registered image 121 is converted into the position of one point in the region where the target object 5 is shown in the target image 122 by the conversion matrix.

  That is, the point (XU1, YU1) which is the upper left corner of the label pasting area 504 in the registered image 121 is converted into a point (X1, Y1) in the area where the object 5 is reflected in the target image 122 by the conversion matrix. Is done. Similarly, a point (XU4, YU4) which is the lower right corner of the label pasting area 504 in the registered image 121 is a point (X4, Y4) in the area where the object 5 is reflected in the target image 122 by the transformation matrix. Converted. Furthermore, although illustration is omitted, in the registration image 121, the position of the point that becomes the upper right corner and the point that becomes the lower left corner of the label affixing region 504 are within the region in which the object 5 is reflected in the target image 122, respectively. Converted to 1 point.

  In this way, by converting the coordinates representing the position of the label pasting area 504 in the registered image 121 using the conversion matrix, the registered image generating unit 130 calculates the label pasting area 504 in the target object 5 shown in the target image 122. It becomes possible to do. Therefore, the registered image generation unit 130 can generate an image indicating an appropriate label attaching position according to the position and orientation of the target object 5 shown in the target image 122. Therefore, by projecting the image generated by the registered image generation unit 130 onto the target object 5 shown in the target image 122 by the projector 3, it is possible to present an appropriate label attaching position to the worker 8. Become.

  As described above, the position / orientation estimation apparatus 1 according to the present embodiment includes a local feature similarity of feature points detected from each of the target image 122 and the registered image 121 and a global region including a plurality of feature points. The position and orientation of the object 5 are estimated based on the feature similarity to At this time, the minimum circumscribed polygon including a plurality of feature points detected from one image is set as the global region, and the distance from the feature point to the centroid of the global region is the feature point for the global region (global feature information) As a result, a difference occurs in the global feature information for each feature point. Therefore, even when the local feature (local feature information) similarity between a plurality of feature points detected from one image is high, the feature point of the target image and the registered image are based on the difference of the global feature information. It is possible to specify the correspondence with the feature points. That is, according to the present embodiment, even when the similarity of local features (local feature information) about a plurality of feature points detected from one target image is high, the target object shown in the target image It is possible to estimate the correct position and orientation. Therefore, according to this embodiment, the estimation accuracy at the time of estimating the position and orientation of the target object shown in the target image based on the target image obtained by capturing the target object and the registered image is improved. Therefore, for example, in a work line or the like that targets a product in which the similarity of the local feature information of the detected feature points becomes high due to the uniformity of the design and the installation of an intake port, etc. It is possible to correctly present the position and orientation of the product.

Note that the flowchart in FIG. 7 is merely an example of a feature point group extraction process in the position and orientation estimation process according to the present embodiment. The feature point pair group extraction processing according to the present embodiment is not limited to the procedure shown in FIG. For example, in the feature point pair group extraction process, each time the feature point similarity S _k is calculated for the set of feature point pairs ψ _k , the feature point similarity is compared with the maximum value, and the feature point similarity becomes the maximum value. Processing that holds only a set of feature point pairs may be used. In this way, by holding only a set of feature points having the maximum feature point similarity in the feature point pair group extraction process, the amount of information stored in the similarity information 125 of the storage unit 120 is reduced and stored. An increase in the capacity of the unit 120 can be suppressed.

  Furthermore, the feature point pair group extraction process may be a process of calculating the feature point similarity for each set of feature points after generating all the sets of feature points that can be generated, for example.

In addition, in the process of calculating the feature point similarity S _k for the set ψ _k of feature point pairs, the feature point similarity may be calculated not only by the above equation (1) but also by other equations. For example, the feature point similarity S _k may be calculated by the following formula (3) or (4).

Expression (2) calculates the feature point similarity S _k based on the sum of the similarity SL _{n, m} of the local feature information and the sum of the reciprocal (1 / SC _{n, m} ) of the similarity of the global feature information. It is an example of the calculation formula in the case of doing. By calculating the feature point similarity S _{k according} to equation (2), the influence of the deterioration of the similarity SC _{n, m} of the global feature information on the feature point similarity S _k is increased, in other words, the feature point similarity. It becomes possible to increase the contribution of the similarity SC _{n, m} of the global feature information in S _k . Meanwhile, Equation (3) is a local inverse (1 / _{SL n, m)} of the similarity of the feature information and the total sum of similarity _{SC n} global feature _information, a feature point similarity _{S k} based on the sum of _m It is an example of the calculation formula in the case of calculating. By calculating the feature point similarity S _{k according} to the expression (3), the influence of the deterioration of the similarity SL _{n, m} of the local feature information on the feature point similarity S _k , in other words, the local in the feature point similarity S _k The contribution degree of the similarity SL _{n, m} of the feature information can be increased.

  Further, the flowchart of FIG. 8, FIG. 12, and FIG. 13 are merely examples of the global feature information extraction method according to the present embodiment. The global feature information according to the present embodiment is not limited to the distance between the feature point and the center of gravity of the global region including a plurality of feature points. For example, as shown in FIG. 16, the feature point is lowered from the feature point to the opposite side of the global region. It may be the length of the perpendicular line.

FIG. 16 is a diagram illustrating a modification of the global feature information extraction method.
FIG. 16 shows, as a modification of the global feature information extraction method, a global feature based on the feature points P ₁ , P ₂ and P ₃ and the global region D in the target image 122 (partially enlarged region 122A) of FIG. The modification of the method of extracting information is shown. As shown in FIG. 16, when there are three feature points used for position and orientation estimation in the target image, the global region D is a triangular shape having _three feature points P ₁ , P ₂ , and P ₃ as vertices. It becomes an area. In the modification illustrated here, the length C _{n of the} perpendicular line extending from the feature point P _n to the side of the global region D is set as described above. Therefore, global feature information extraction unit 115A, for each feature point, and calculates the distance C _n from the position of the feature point P _n a perpendicular line is dropped on opposite sides of the feature point P _n to the position of intersection of the opposite sides and the vertical. In this way, even when the length C _n of the perpendicular line extending from the feature point P _n to the side of the global region D is used as the global feature information, there is a difference in the global feature information of each feature point P _n in one image. Arise. For this reason, the local feature information of a plurality of feature points Pn in one image is similar by using, as the global feature information, the length C _n of the perpendicular line extending from the feature point P _n to the side of the global region D. Even in this case, it is possible to specify the feature point pair group.

  Further, as illustrated in the present embodiment, when the global regions D and DU are triangular, the global feature information is not limited to the above values, and for example, any one of the centroid, inner center, outer center, and side center from the feature point It is good also as distance to.

  In addition, the work line 10 in FIG. 1 is merely an application example of the position and orientation estimation apparatus 1 according to the present embodiment. The position / orientation estimation apparatus 1 according to the present embodiment includes an object image 122 based on a target image 122 obtained by imaging the object 5 whose position / orientation is estimated and a registered image 121 of the object 5 obtained by imaging at a predetermined position / orientation. The present invention can be applied to various apparatuses and systems that estimate the position and orientation of the target object 5.

[Second Embodiment]
In this embodiment, a method for extracting global feature information used for position and orientation estimation, which is different from the first embodiment, will be described.

  The functional configuration of the position / orientation estimation apparatus according to the present embodiment may be the same as the functional configuration of the position / orientation estimation apparatus 1 according to the first embodiment. That is, the position / orientation estimation apparatus 1 according to this embodiment includes a position / orientation estimation unit 110, a storage unit 120, and a projection image generation unit 130, as illustrated in FIG. The position / orientation estimation unit 110 includes an image acquisition unit 111, a feature point detection unit 112, a local feature information extraction unit 113, a global region determination unit 114, a feature point pair group extraction unit 115, and a position / orientation calculation unit 116. (See FIG. 2). The feature point pair group extraction unit 115 includes a global feature information extraction unit 115A, a set creation unit 115B, a local feature information comparison unit 115C, a global feature information comparison unit 115D, a similarity calculation unit 115E, and a calculation processing control unit. 115F and a feature point pair group determination unit 115G (see FIG. 3).

  Further, as shown in FIG. 2, the storage unit 120 includes the registered image 121, the target image 122, the registered image feature information 123, the target image feature information 124, the similarity information 125, and the label pasting position 129. The various data including are stored.

  In addition, the global area | region determination part 114 of the position and orientation estimation apparatus 1 which concerns on this embodiment makes the outline of the target object 5 reflected in the image a global area | region. In addition, the global feature information extraction unit 115A of the position / orientation estimation apparatus 1 according to the present embodiment uses a plurality of vertices of the global region as base points, and for each feature point, extracts the distance from the feature point to each base point as global feature information. (Calculate). Furthermore, the feature point pair group extraction unit 115 of the position / orientation estimation apparatus 1 according to the present embodiment generates a plurality of sets of base point pairs based on the base point of the target image and the base point of the registered image, and extracts each base point pair set. The feature point pair group that maximizes the feature point similarity is determined from the candidate feature point pair groups.

  For example, the position / orientation estimation apparatus 1 according to the present embodiment repeatedly performs the processes of steps S1 to S9 illustrated in FIG. 4 at predetermined time intervals. The position / orientation estimation apparatus 1 may repeatedly perform a series of processes in steps S1 to S9 as one processing unit, or may perform the processes in steps S1 to S9 in a pipeline.

  Note that the position / orientation estimation apparatus 1 according to the present embodiment performs the process shown in FIG. 17 as the global region determination process in step S6 of the processes in steps S1 to S9.

  FIG. 17 is a flowchart for explaining the contents of the global area determination processing according to the second embodiment.

  Also in the present embodiment, the global region determination process in step S6 is performed by the global region determination unit 114 of the position and orientation estimation apparatus 1. As illustrated in FIG. 17, the global region determination unit 114 first extracts the contour of the target object 5 in the target image 122, and stores the region surrounded by the contour of the target object 5 as the global region D of the target image. (Step S621). In step S621, the global region determination unit 114 extracts the contour of the target object 5 in the target image 122 according to a known contour extraction method in the field of image processing. For example, the global region determination unit 114 performs edge detection processing on the target image 122 and detects (extracts) the contour of the target object 5. The global region determination unit 114 stores information representing the global region D extracted from the target image 122 in the storage unit 120 as one of the target image feature information 124.

  Next, the global area determination unit 114 refers to the registered image feature information 123 and determines whether or not the global area DU of the registered image 121 has been determined (step S622). When the global area DU of the registered image 121 has been determined (step S622; YES), the global area determination unit 114 ends the global area determination process.

  On the other hand, when the global area DU of the registered image 121 has not yet been determined (step S622; NO), the global area determining unit 114 next extracts the contour of the object 5 in the registered image 121, and The area surrounded by the outline is held in the storage unit 120 as the global area DU of the registered image (step S623). When the outer periphery of the registered image 121 matches the contour of the object 5 as shown in FIG. 9A, in step S613, the global area determining unit 114 changes the outer periphery of the registered image 121 to the global area DU. decide. On the other hand, when the outer periphery of the registered image 121 and the contour of the target object 5 do not match, in step S613, the global region determining unit 114 performs the same process as in step S611 to extract the contour of the target object 5. To do. When the process of step S613 is performed, the global region determination unit 114 determines the contour of the extracted object 5 as the global region DU of the registered image 121, and sets information indicating the global region DU as one of the registered image feature information 123. To store in the storage unit 120, and the global area determination process is terminated.

  After performing the global region determination process, the position / orientation estimation apparatus 1 performs the feature point pair group extraction process of step S7 in the flowchart of FIG. The feature point pair group extraction processing is performed by the feature point pair group extraction unit 115. The feature point pair group extraction unit 115 according to the present embodiment performs the processes shown in FIGS. 18A to 18C as the feature point pair group extraction process.

  FIG. 18A is a flowchart (part 1) for explaining the contents of the feature point pair group extraction processing according to the second embodiment. FIG. 18B is a flowchart (part 2) illustrating the contents of the feature point pair group extraction processing according to the second embodiment. FIG. 18C is a flowchart (part 3) for explaining the content of the feature point pair group extraction processing according to the second embodiment.

Feature point pair group extraction unit 115 according to the present embodiment, as shown in FIG. 18A, first detects the vertex _{G i} of the global area D of the target image (step S721). The processing in step S721 is performed by the global feature information extraction unit 115A. Global feature information extraction unit 115A, in accordance with known peak detecting method, for detecting the vertex G _i of the global area D of the target image 122. At this time, the global feature information extraction unit 115A sets, for example, one of a plurality of vertices detected from the target image 122 to G _{i = 1} , and the remaining vertices from the vertex G _{i = 1} to the global region D An identifier i indicating the order of appearance of vertices when the contour is rotated once in a clockwise direction is assigned.

After execution of step S721, the global feature information extraction unit 115A continues, the feature point for each _{P n} of the target image, calculates the distance _{C n, i} to each vertex _{G i} from the feature point _{P n,} the The distance C _{n, i} is stored in the storage unit 120 as the global feature information of the target image (step S722). In step S722, the global feature information extraction unit 115A calculates a distance C _{n, i} from the position (pixel) of the feature point P _{n to} the position (pixel) of the vertex G _i in the image. The global feature information extraction unit 115A stores the calculated distance C _{n, i} in the target image registration information 124 as global feature information of each feature point P _n .

When the process of step S722 is completed, the global feature information extraction unit 115A next refers to the registered image feature information 123 and determines whether or not the vertices of the global region DU and the global feature information of the registered image have been extracted. (Step S723). If the vertex and global feature information of the global area DU of the registered image have not yet been extracted (step S723; NO), the global feature information extraction unit 115A next detects the vertex GU _j of the global area DU of the registered image. (Step S724). In step S724, the global feature information extraction unit 115A detects the vertex GU _j of the global region DU of the registered image 121 by the same processing as in step S711. At this time, the global feature information extraction unit 115A sets, for example, one of a plurality of vertices detected from the registered image as GU _{j = 1} , and the remaining vertices from the vertex GU _{j = 1} to the outline of the global region DU. Is assigned an identifier j indicating the order of appearance of the vertices when the circle is rotated clockwise.

After execution of step S724, the global feature information extraction unit 115A is followed, for each feature point PU _m of the reference image, calculates the distance _{CU m, j} to each vertex GU _j from the feature point PU _m, the The distance CU _{m, j} is stored in the storage unit 120 as the global feature information of the registered image (step S725). In step S725, the global feature information extraction unit 115A calculates a distance CU _{m, j} from the position (pixel) of the feature point PU _{m to} the position (pixel) of the vertex GU _{j in} the registered image. The global feature information extraction unit 115A stores the calculated distance CU _{m, j} in the registered image registration information 123 as global feature information of each feature point.

When the processes of steps S724 and S725 are completed, the feature point pair group extraction unit 115 then sets a variable p for identifying a set of vertex pairs in the global region to 0 (step S726). When the vertex GU _j and the global feature information (distance CU _{m, j} ) of the global region DU of the registered image have been determined (step S723; YES), the feature point pair group extraction unit 115 performs steps S724 and S725. The process of step S726 is performed by skipping the process. The calculation process control unit 115F performs the process of step S726.

  After the processing of step S726, the feature point pair group extraction unit 115 generates a feature point pair group that represents the correspondence between the feature points of the target image and the feature points of the registered image based on the local feature information and the global feature information. An extraction (identification) process (steps S727 to S737) is performed. The processes in steps S727 to S737 are performed under the control of the calculation processing control unit 115F, the set generation unit 115B, the local feature information comparison unit 115C, the global feature information comparison unit 115D, the similarity calculation unit 115E, and the feature points. Performed by the pair determination unit 115G.

After the process of step S726, the feature point pair group extraction unit 115 then, as shown in FIG. 18B, the vertex G _i of the global area D of the target image 122 and the vertex GU _j of the global area DU of the registered image 121 A set ω _p of vertex pairs by is generated (step S727). The set generation unit 115B performs the process of step S727. The set generation unit 115B, according to a known generation method for generating a set of element pairs of the elements included in the first set and the elements included in the second set, the vertex G _i of the global region of the target image and the registered image A set of vertex pairs ω _p with vertices GU _{i in} the global region is generated. In one process of step S727, the set generation unit 115B generates one set of sets of vertex pairs that can be generated according to a predetermined generation rule (generation algorithm).

  Next, the feature point pair group extraction unit 115 sets a variable k for identifying a set of feature point pairs to k = 0 (step S728). The calculation process control unit 115F performs the process of step S728.

Next, the feature point pair group extraction unit 115 generates a set ψ _k of feature point pairs based on the feature point P _n of the target image and the feature point PU _m of the registered image (step S729). The set generation unit 115B performs the process of step S729. Similar to the first embodiment, the set generation unit 115B generates a set ψ _k of feature point pairs of the feature points P _n of the target image and the feature points PU _{m of the} registered image according to a known generation method. Note that in one process of step S729, the set generation unit 115B can generate based on the total number N of feature points of the target image and the total number M of feature points of the registered image according to a predetermined generation rule (generation algorithm). A set of a set of feature point pairs is generated.

Next, the feature point pair group extraction unit 115 calculates the similarity SL _{n, m} of local feature information and the similarity SC _{n, m of} global feature information in each feature point pair included in the set ψ _k of feature point pairs. Is calculated (step S730). The local feature information comparison unit 115C and the global feature information comparison unit 115D perform the process in step S730. For each feature point pair, the local feature information comparison unit 115C compares the local feature information L _n of the feature point P _n of the target image stored in the target image feature information 124 and the registered image stored in the registered image feature information 123. by comparing the local feature information LU _m feature points PU _m, the similarity is calculated _{SL n, m} of the local feature information. For each feature point pair, the global feature information comparison unit 115D performs the global feature information C _n of the feature point P _n of the target image stored in the target image feature information 124 and the registered image stored in the registered image feature information 123. compares the global feature information CU _m feature points PU _m, it calculates the global feature information similarity _{SC n,} the _m.

In the process of calculating the similarity of the global feature information in the process of step S730, based on the correspondence between the vertex G _i of the target image in the vertex pair set ω _p and the vertex GU _{j of the} registered image, Compare global feature information. For example, when the vertex pair set ω _{p includes} (G ₁ , GU ₁ ) vertex pairs, in the process of calculating the global feature information similarity SC _1,1 for the feature point pair (P ₁ , PU ₁ ) The distance from the feature point P ₁ to the vertex G ₁ is compared with the distance from the feature point PU ₁ to the vertex GU ₁ . Further, for example, when the vertex pair set ω _{p includes} (G ₁ , GU ₃ ) vertex pairs, the similarity SC ₁ , ₁ of the global feature information for the feature point pair (P ₁ , PU ₁ ) is calculated. In the processing, the distance C _1,1 from the feature point P ₁ to the vertex G ₁ is compared with the distance CU _1,3 from the feature point PU ₁ to the vertex GU ₃ .

Next, the feature point pair group extraction unit 115 performs feature points on the set ψ _k of feature point pairs based on the similarity SL _{n, m} of the local feature information and the similarity SC _{n, m of} the global feature information. calculating a similarity _{S k} (step S731). The process of step S731 is performed by the similarity calculation unit 115E. For example, the similarity calculation unit 115E calculates the feature point similarity S _k for the set ψ _k by the above equation (1). Similarity calculation unit 115E is configured to store the similarity information 125 in the storage unit 120 the feature point similarity S _k for the calculated set [psi _k in correspondence with the set [psi _k, was calculated feature point similarity S _k This is notified to the calculation processing control unit 115F.

When the feature point similarity S _k for the set [psi _k feature point pairs are calculated, calculation control section 115F determines whether to generate all the set of feature point pairs that can generate (step S732) . In step S732, the calculation processing control unit 115F calculates the total number of sets that can be generated calculated based on the total number N of feature points of the target image and the total number M of feature points of the registered image, and a variable k that identifies the generated set. It is determined whether or not all sets have been generated. For example, when there are K sets of feature point pairs that can be generated, the calculation processing control unit 115F determines whether or not k ≧ K−1 (since the variable k starts from 0). When k <K−1, the calculation processing control unit 115F determines that there is a set of feature point pairs that have not been generated.

  When there is a set of feature point pairs that have not been generated (step S732; NO), the calculation processing control unit 115F updates the variable k that identifies the set of feature point pairs to k + 1 (step S733). Thereafter, the calculation processing control unit 115F causes the set generation unit 115B, the local feature information comparison unit 115C, the global feature information comparison unit 115D, and the similarity calculation unit 115E to perform the processes of steps S729 to S732.

On the other hand, when all sets of feature point pairs that can be generated are generated (step S732; YES), the calculation processing control unit 115F next causes the feature point pair group determination unit 115G to determine feature point pair group candidates. The feature point pair group determination unit 115G refers to the similarity information 125, and the feature point similarity among the sets of a plurality of feature point pairs whose feature point similarity is calculated in the state of the vertex pair set ω _p is the maximum. A set of feature point pairs as values is determined as a candidate feature point group and stored in the storage unit 120 (step S733). Feature point pair group determination unit 115G is configured to store the candidate [psi _p and the feature point similarity S _p of the feature point pair group for collection omega _p pair of vertices in similarity information 125, a candidate of the feature point pair group The calculation processing control unit 115F is notified of the determination.

If candidates [psi _p of the feature point pair group for collection omega _p pair of vertices is determined, calculation control section 115F is, as shown in FIG. 18C, the determination whether to generate all set of vertices pairs capable of producing (Step S735). In step S735, the calculation processing control unit 115F compares the total number of sets that can be generated calculated based on the shapes of the global areas D and DU with the value of the variable p that identifies the generated set, and sets all sets It is determined whether or not has been generated. For example, when there are P sets of vertex pairs that can be generated, the calculation processing control unit 115F determines whether or not p ≧ P−1 (since the variable p starts from 0). When p <P-1, the calculation processing control unit 115F determines that there is a set of vertex pairs that have not been generated.

  When there is a set of vertex pairs that have not been generated (step S735; NO), the calculation processing control unit 115F updates the variable p for identifying the set of vertex pairs to p + 1 (step S736). Thereafter, the calculation processing control unit 115F performs the processing of steps S727 to S734 to the set generation unit 115B, the local feature information comparison unit 115C, the global feature information comparison unit 115D, the similarity calculation unit 115E, and the feature point pair group determination unit 115G. To do.

  On the other hand, when all sets of vertex pairs that can be generated are generated (step S735; YES), the calculation processing control unit 115F next causes the feature point pair group determination unit 115G to determine a feature point pair group. The feature point pair group determination unit 115G refers to the similarity information 125, and determines a feature point pair group candidate having the maximum feature point similarity among the feature point pair group candidates as a feature point pair group. (Step S737). When the feature point pair group determination unit 115G determines the feature point pair group, the feature point pair group extraction process ends.

  Thus, in the feature point pair group extraction processing according to the present embodiment, the similarity of the local feature information and the similarity of the global feature information are calculated for each feature point pair included in the generated feature point pair set, Based on these similarities, a value representing the similarity of feature point pairs in each set is calculated. At this time, in the feature point pair group extraction processing according to the present embodiment, the outline of the object in the image is set as the global region, and the distance from the feature point to the vertex of the global region calculated for each feature point is set as the global feature information. To do.

  Hereinafter, with reference to FIG. 19 to FIG. 23, the estimation process of the position and orientation of the object according to the present embodiment will be specifically described. In the following description, a case where the position and orientation of the target object 5 is estimated based on the registered image 121 and the target image 122 shown in FIG. Moreover, in the following description, detailed description about the part which may be the same process as the process demonstrated in 1st Embodiment among the estimation processes of the position and orientation of the target object 5 which concerns on this embodiment is abbreviate | omitted.

FIG. 19 is a diagram for explaining a method of determining a global area according to the second embodiment.
In the global region determination processing according to the present embodiment, the region surrounded by the contour of the target object 5 in each of the target image 122 and the registered image 121 is determined as the global region D of the target image 122 and the global region DU of the registered image 121. To do.

FIG. 19A shows an example of the global area DU of the registered image 121. When the outer periphery of the registered image 121 matches the contour of the object 5, the entire registered image 121 becomes the global region DU. In this case, for example, the global area determination unit 114 stores the position of the pixel that is the outer periphery of the registered image 121 in the registered image feature information 123 as information representing the global area DU of the registered image 121. When the entire registered image 121 is a global region DU, the global region DU is a region including the feature points PU ₁ , PU ₂ , and PU ₃ detected from the registered image 121.

FIG. 19B shows an example of the global region D set in the target image 122. In the target image 122, the conveyance belt 401 and the like are shown in addition to the target object 5. For this reason, the global region determination unit 114 detects, for example, the edge of the object 5, that is, the boundary between the object 5 and the background (such as the conveyor belt 401) according to a known edge detection method. The global area determination unit 114 stores the position of the pixel representing the detected edge (boundary) in the target image feature information 124 as information indicating the global area D of the target image 122. When the region surrounded by the contour of the target object 5 in the target image 122 is a global region D, the global region D includes feature points P ₁ , P ₂ , and P ₃ detected from the target object 5 in the target image 122. It becomes an area to include.

  After determining the global region D of the target image 122 and the global region DU of the registered image 121, in the position and orientation estimation apparatus 1, the feature point pair group extraction unit 115 performs a feature point pair group extraction process (step S <b> 7). In the feature point pair group extraction processing, first, a feature amount (global feature information) different from the local feature information is extracted based on the relationship between the feature points and the global region. In the feature point group extraction process in the present embodiment, as described above, the distance between the feature point and the vertex of the global region is extracted as the global feature information.

  FIG. 20 is a diagram illustrating a method for extracting global feature information from a target image. FIG. 21 is a diagram illustrating a method for extracting global feature information from a registered image. FIG. 22 is a diagram illustrating an example of target image feature information and registered image feature information after the global feature information is extracted.

FIG 20 illustrates a method of extracting global features information about the feature points P ₂ detected from the target image 122 of (b) in FIG. 19.

When the outer shape of the target object 5 in the target image 122 is approximately square, the global region D is approximately square. Therefore, the global feature information extraction unit 115A detects _four vertices G _{1 to} G ₄ from the global region D of the target image 122 in the process of step S721. In this case, global feature information extraction unit 115A is one of the four vertices, for example, the value of y-coordinate is the first vertex G ₁ vertex is the minimum value. Also, the global feature information extraction unit 115A has the remaining three vertices, according to order of appearance of the vertices when the one round the contour of the global area D in the clockwise direction from the first vertex G _1, the second vertex G ₂ , a third vertex G ₃ , and a fourth vertex G ₄ .

Then, the global feature information extraction unit 114 calculates the the feature point for each _{P n} detected, the distance _{C n,} 1 _~C n, ₄ from the feature point _{P n} to each vertex _G 1 ~G ₄ from the target image 122 Then, the calculated distances C _{n, 1 to} C _{n, 4} are set as the global feature information about the feature point P _n (step S722). FIG. 20 shows global feature information (distance C _{2,1 to} C _2,4 ) about the feature point P ₂ that is one of the feature points detected from the target image 122.

Next, the global feature information extraction unit 115 </ b> A extracts the global feature information of the registered image 121. Global feature information extraction unit 115A extracts four vertices _GU 1 _{~GU 4} from the global area DU of the registered image 121 in the process of step S724. In this case, global feature information extraction unit 115A, for example, as shown in FIG. 21, the apex of the upper left corner of the four vertexes and the first vertex GU _1. Also, the global feature information extraction unit 115A has the remaining three vertices, according to order of appearance of the vertex when the edge of the global heap DU and one round clockwise from the first vertex GU _1, second vertex GU ₂ , a third vertex GU ₃ , and a fourth vertex GU ₄ .

Then, the global feature information extraction unit 114 calculates for each feature point PU _m detected from the registration image, the distance _{CU m,} 1 _~CU m, ₄ from the feature point PU _m to each vertex _GU 1 _{~GU 4} The calculated distances CU _{m, 1 to} CU _{m, 4} are set as global feature information about the feature point PU _m (step S725). FIG. 21 shows global feature information (distance CU _{2,1 to} CU _2,4 ) for the feature point PU ₂ that is one of the feature points detected from the registered image 121.

Therefore, when the processes of steps S721 and S722 are performed, the target image feature information 124 includes feature points P _n , local feature information L _n , and global feature information as in the table 1103 illustrated in FIG. Information that associates C _{n, 1 to} C _{n, 4} is stored. Similarly, when the processes in steps S724 and S725 are performed, the registered image feature information 123 includes the feature point PU _m , the local feature information LU _m, and the global feature as in the table 1104 shown in FIG. Information that associates information CU _{m, 1 to} CU _{m, 4} is stored.

When the number of feature points P _n detected from the target image 122 is N, the local feature information L ₁ , L _{2 is included in} each of the _N feature points P ₁ , P ₂ ,. ,..., L _N and global feature information C _{1, i} , C _{2, i} ,..., C _{N, i} (i = 1 to I) are associated with each other. Further, when there are M feature points PU _m detected from the registered image 121, local feature information LU ₁ , LU _{2 is included in} each of the _M feature points PU ₁ , PU ₂ ,. ,..., LU _M and global feature information CU _{1, j} , CU _{2, j} ,..., CU _{M, j} (j = 1 to J) are associated with each other.

After extracting the global feature information of each feature point in each of the target image 122 and the registered image 121, the feature point pair group extraction unit 115 generates a vertex pair set ω _p in the set generation unit 115A (step S727). Set generation unit 115B in accordance with a predetermined generation rule to each of the I-number of vertices G _i in the global area D of the target image, a pair of one of the J feature points GU _j in the global heap DU registered image 1 A set ω _{p of} vertex pairs associated with each other is generated. Since the registered image 121 and the target image 122 include objects having the same shape, the number I of vertices in the global area D of the target image 122 and the number J of vertices in the global area DU of the registered image 121 are the same number. (I = J). However, there is a possibility that I ≠ J depending on the position and orientation of the target object 5 shown in the target image 122. Therefore, when p = 0 and J ≧ I, the set generation unit 115B includes, for example, a set ω _p = {(G ₁ , GU ₁ ), (G ₂ , GU ₂ ), including I vertex pairs. ., (G _I , GU _{J = I} )} is generated. On the other hand, when k = 0 and J <I, the set generation unit 115B includes, for example, a set ω _p = {(G ₁ , GU ₁ ), (G ₂ , GU ₂ ), including J vertex pairs. · to produce a _{(G I = J, GU J} )}.

When p ≠ 0, the set generation unit 115B shifts the GU _j of the global area D of the registered image 121 to be associated with each vertex G _i of the global area D of the target image 122 by p vertices clockwise. Let it be a vertex.

  FIG. 23 is a diagram for explaining a method for generating a set of vertex pairs. FIG. 24 is a diagram for explaining the relationship between the rotational symmetry of the global region and the set of vertex pairs that can be generated.

23A shows a table 1105 for explaining the correspondence between the vertices G _{1 to} G ₄ of the global area D of the target image 122 and the vertices GU _{1 to} GU ₄ of the global area DU of the registered image 121. Show. As described above, the vertices G _{1 to} G ₄ of the global area D of the target image 122 are assigned identifiers i in the order of appearance of the vertices when the outline of the global area D is made one round clockwise from the first vertex G _1. (See FIG. 20). Similarly, the vertex _GU 1 _{~GU 4} global regions DU of the registered image 121, the identifier j by applying the appearance order of the vertices of when one round the contour of the global heap DU clockwise from the first vertex GU ₁ (See FIG. 21). Also, assuming that the global areas D and DU are substantially square, the vertex G1 of the global area D of the target image 122 is represented by four vertices of the global area DU of the registered image 121 only by the information representing the global areas D and DU. It is difficult to determine which vertex of GU _{1 to} GU ₄ corresponds to. For this reason, in the present embodiment, as shown in FIGS. 18B and 18C, a plurality of vertex pair sets ω _p are generated by the vertex G _i of the target image 122 and the vertex GU _j of the registered image 121, respectively. In this case, feature point pair group candidates are extracted.

When the global areas D and DU of each image are substantially square and identifiers i and j are assigned according to the above rules, the correspondence between the vertex G _i of the target image 122 and the vertex GU _j of the registered image 121 is The patterns are limited to the four patterns shown in the table 1105 in FIG. Pattern 1 and the apex _{G 1} of the target image 122, a pattern in which the vertex GU ₁ of the reference image 121 corresponding, pattern 2 and the apex _{G 1} of the target image 122, and the vertex GU ₂ registration image 121 corresponding It is a pattern. Pattern 3 vertex _{G 1} of the target image 122, a pattern in which the vertex GU ₃ registered image 121 corresponding pattern 4 and the apex _{G 1} of the target image 122, and the vertex GU ₄ of the reference image 121 corresponding It is a pattern.

Therefore, when generating the set ω _{p of} vertex pairs, the set generation unit 115B sequentially generates four patterns from pattern 1 to pattern 4. For example, when p = 0, the set generation unit 115B generates a set of vertex pairs ω _{p = 0 in} which the vertices G _i and GU _j are associated with each other in the pattern 1. When p = 1, the set generation unit 115B generates, for example, a set of vertex pairs ω _{p = 1 in} which the vertices G _i and GU _j are associated with each other in the pattern 2. That is, the set generation unit 115B sequentially generates, for example, four sets of vertex pairs ω _{0 to} ω ₃ shown in the table 1106 in FIG.

When the global regions D and DU representing the contour of the target object 5 are square (four-fold symmetry), the first vertex G ₁ in the target image 122 is any of the _four vertices GU _{1 to} GU 4 in the registered image 121. Both can be vertex pairs. Therefore, when the global areas D and DU are square, there are four sets of vertex pairs ω _p as shown in FIG. On the other hand, when the global areas D and DU representing the contour of the target object 5 are rectangular (two-fold symmetry), the first vertex G ₁ in the global area D of the target image is registered as shown in FIG. Only two vertices GU ₁ and GU ₃ in the global region DU of the image can be a vertex pair. That is, when the contour of the object 5 (for example, the label sticking surface) is a rectangle, vertices (for example, the vertex G ₁ and the vertex GU ₂ ) whose positional relationship between the short side and the long side connected to the vertex are opposite to each other are opposite. There is no correspondence. Therefore, it is preferable that the calculation processing control unit 115F and the set generation unit 115B change the number of combinations of vertex pair sets that can be generated according to the shapes of the global regions D and DU. Vertex _G 1 ~G ₄ global area D, when the relationship shown in the global heap DU apex _GU 1 of ～GU ₄ Togazu 24 (a), a set omega _p pair of vertices that may be generated, FIG. The two sets ω ₀ and ω ₁ shown in the table 1107 in FIG. Thus, when the rotational symmetry of the global areas D and DU is low, the combination of the set of vertex pairs ω _p to be generated (in other words, the number of repetitions of steps S727 to S734) can be reduced, and the feature point pairs It becomes possible to reduce the processing load of the group extraction process.

After generating the vertex pair set ω _p , as described in the first embodiment, the feature point pair group extraction unit 115 generates a set of feature point pairs ψ _{k, and} for each feature point pair, a local feature is generated. The similarity of information and the similarity of global feature information are calculated. The similarity of the local feature information of the feature point pair may be calculated according to a known similarity calculation method as described in the first embodiment.

On the other hand, the global feature information in the present embodiment is the distance from the feature point to each of a plurality of vertices in the global region as described above. That is, as shown in FIGS. 22A and 22B, there are a plurality of pieces of global feature information for one feature point. Also, as shown in FIGS. 20 and 21, a plurality of pieces of global feature information for one feature point are combinations of different values depending on the position of the feature point. Therefore, the similarity of the global feature information in the feature point pair varies depending on the correspondence between the plurality of global feature information in the feature points of the target image and the plurality of global feature information in the feature points of the registered image. Therefore, when calculating the similarity of the global feature information of the feature point pair in this embodiment, based on the vertex pair set ω _p , a plurality of global feature information at the feature points of the target image and the registered image A correspondence relationship between a plurality of global feature information at the feature points is determined.

FIG. 25 is a diagram for explaining the relationship between vertex pairs and global feature information to be compared.
For example, set generation unit 115B is a vertex _G 1 ~G ₄ in the target image 122 in FIG. 20, the vertex _GU 1 _{~GU 4} collectively ω _p = _{(G 1 pair of vertices by the registered image 121 in FIG. 21, Suppose that GU ₁ ), (G ₂ , GU ₂ ), (G ₃ , GU ₃ ), (G ₄ , GU ₄ )} are generated. In this case, the global feature information similarity SC _2,2 for the feature point pair (P ₂ , PU ₂ ) between the feature point P ₂ in the target image 122 in FIG. 20 and the feature point PU ₂ in the registered image 121 in FIG. Is calculated as follows, for example.

As shown in the table 1107 in FIG. 25A, the global feature information comparison unit 115D _first follows the distance C from the feature point P ₂ to the vertex G ₁ in the target image according to the vertex pair (G ₁ , GU ₁ ). ₂ and ₁ are compared with the distance CU _2,1 from the feature point PU ₂ to the vertex GU ₁ in the registered image. Next, the global feature information comparing unit 115D follows the distance C _2,2 from the feature point P ₂ to the vertex G ₂ in the target image and the feature point PU ₂ in the registered image according to the vertex pair (G ₂ , GU ₂ ). The distance CU _2,2 to the vertex GU ₂ is compared. Next, the global feature information comparison unit 115D follows the vertex pair (G ₃ , GU ₃ ) from the distance C _2,3 from the feature point P ₂ to the vertex G ₃ in the target image and the feature point PU ₂ in the registered image. The distances CU ₂ and ₃ to the vertex GU ₃ are compared. Then, the global feature information comparing unit 115D in accordance with the vertex pair _{(G 4,} GU 4), the distance _{C 2, 4} from the feature point _{P 2} in the target image to the apex _{G 4,} from the feature point PU ₂ in the registration image The distances CU ₂ and ₄ to the vertex GU ₂ are compared. Finally, the global feature information comparing unit 115D is based on the distance comparison result in each of the four sets of vertices pairs described above, the feature point P _2, the global feature information for the feature point pairs between the feature point PU ₂ The similarity SC _2,2 is calculated.

Further, for example, the set generation unit 115B sets the vertex pair set ω _p = {(G ₁ , GU ₃ ), (G ₂ , GU ₄ ), (G ₃ , GU ₁ ), (G ₄ , GU ₂ )}. When generated, the similarity SC _2,2 of the global feature information for the feature point pair of the feature point P ₂ and the feature point PU ₂ is calculated as follows.

Global feature information comparing unit 115D, as shown in table 1108 of (b) in FIG. 25, first, the distance in accordance with the vertex pair _(G 1, GU _3), from the feature point _{P 2} in the target image to vertex _{G 1} C ₂ and ₁ are compared with the distance CU _2,3 from the feature point PU ₂ to the vertex GU ₃ in the registered image. Next, the global feature information comparison unit 115D follows the distance C _2,2 from the feature point P ₂ to the vertex G ₂ in the target image and the feature point PU ₂ in the registered image according to the vertex pair (G ₂ , GU ₄ ). The distance CU _2,4 to the vertex GU ₄ is compared. Next, the global area feature comparison unit 115D follows the vertex pair (G ₃ , GU ₁ ) from the distance C _2,3 from the feature point P ₂ to the vertex G ₃ in the target image and the feature point PU ₂ in the registered image. The distance CU _2,1 to the vertex GU ₁ is compared. Then, the global feature information comparing unit 115D in accordance with the vertex pair _(G 4, GU _2), the distance _{C 2, 4} from the feature point _{P 2} in the target image to the apex _{G 4,} from the feature point PU ₂ in the registration image The distance CU _2,2 to the vertex GU ₂ is compared. Finally, the global feature information comparing unit 115D is based on the distance comparison result in each of the four sets of vertices pairs described above, the feature point P _2, the global feature information for the feature point pairs between the feature point PU ₂ The similarity SC _2,2 is calculated.

After calculating the similarity of the local feature information and the similarity of the global feature information of each feature point pair included in the set ψ _k of feature point pairs, the feature point pair group extraction unit 115 performs the feature points in the similarity calculation unit 115E. It calculates a feature point similarity S _k for the set [psi _k pairs. Similarity calculation unit 115E, for example, to calculate the feature point similarity _{S k} by equation (1). Then, when the feature point similarity score S _k is calculated for all of the feature point pair sets ψ _k that can be generated in a state where a set of vertex pairs ω _p is generated, the feature point pair group extraction unit 115 In the feature point pair group determination unit 115G, a set of feature point pairs having the maximum feature point similarity is extracted. The feature point pair group determination unit 115G stores the extracted set of feature point pairs in the similarity information 125 in association with the vertex pair set ω _p and the feature point similarity as candidate feature point pairs ( Step S734). Note that the feature point pair set having the maximum feature point similarity in the feature point pair group determination unit 115G is used as a feature point pair group candidate because the calculated value increases as the similarity increases. This is a case where the feature point similarity _Sk is calculated. That is, the similarity is a positive value, and when calculating the feature point similarity S _k by calculating how the value of the degree of similarity is calculated higher decreases, the feature point pair group determination unit 115G, feature points similar A set of feature point pairs having a minimum degree is set as a candidate feature point group.

After that, the feature point pair group extraction unit 115 repeats the processes of steps S727 to S734 until the feature point pair group candidates are extracted from all of the vertex pair sets ω _p that can be generated. Then, the feature point pair group extraction unit 115 determines the correspondence between the feature point of the target image and the feature point of the registered image for the candidate having the maximum feature point similarity among the extracted feature point pair group candidates. The feature point pair group to be represented is specified (step S737). The feature point pair group extraction unit 115 specifies the feature point pair versus group candidate having the maximum feature point similarity as the feature point pair group. The calculation method increases the calculated value as the similarity degree increases. This is a case where the feature point similarity _Sk is calculated. That is, when the feature point similarity _Sk is calculated by a calculation method in which the similarity is a positive value and the calculated value is smaller as the similarity is higher, the feature point pair group extraction unit 115 calculates the feature point similarity. A feature point pair group having a minimum degree is defined as a feature point pair group.

After specifying the feature point pair group, the position / orientation estimation apparatus 1 calculates the position / orientation of the object shown in the object image in the position / orientation calculation unit 116 (step S8). As described in the first embodiment, the position and orientation calculation unit 116 determines the position of each feature point PU _m in the registered image 121 based on the feature point correspondence in the feature point pair group in the target image 122. A determinant to be converted into the position of the point P _n is calculated.

  After calculating the determinant, the position / orientation estimation apparatus 1 uses the projection image generation unit 130 to calculate the object 5 shown in the target image 122 based on the calculated determinant and the label pasting position 129 of the storage unit 120. The label attaching position at is calculated (step S9). Furthermore, the projection image generation unit 130 generates an image 6 to be projected onto the target object 5 shown in the target image 122 based on the calculated pasting position, and outputs it to the projector 3 (step S9).

  As described above, the position / orientation estimation apparatus 1 according to the present embodiment performs local feature similarity on feature points detected from each of the target image and the registered image, and features for a global region including a plurality of feature points. The position and orientation of the object are estimated based on the similarity. In the present embodiment, the global feature information for each feature point is differentiated by defining the outline of the object in the image as the global region and the distance from the feature point to the vertex of the global region as the global feature information. Cause it to occur. Therefore, even when the local feature (local feature information) similarity between a plurality of feature points detected from one image is high, the feature point of the target image and the registered image are based on the difference of the global feature information. It is possible to specify the correspondence with the feature points. That is, according to the present embodiment, even when the similarity of local features (local feature information) about a plurality of feature points detected from one target image is high, the target object shown in the target image It is possible to calculate the correct position and orientation. Therefore, according to this embodiment, the estimation accuracy at the time of estimating the position and orientation of the target object shown in the target image based on the target image obtained by capturing the target object and the registered image is improved. Therefore, for example, in a work line or the like that targets a product in which the similarity of the local feature information of the detected feature points becomes high due to the uniformity of the design and the installation of an intake port, etc. It is possible to correctly present the position and orientation of the product.

  In the present embodiment, as described above, the outline of the object shown in the image is the global region, and the distance from the feature point to the vertex (base point) of the global region is the global feature information. Therefore, according to this embodiment, even when a large number of feature points are detected from one image, the global region and the global feature information are extracted with the same processing load as when the number of detected feature points is small. It becomes possible to determine the base point when Therefore, according to the present embodiment, it is possible to suppress an increase in processing load in the process of determining the global region and the base point.

Note that the flowcharts of FIGS. 18A to 18C are merely an example of feature point pair group extraction processing according to the present embodiment. The content of the feature point pair group extraction processing according to the present embodiment is not limited to the procedure shown in FIGS. 18A to 18C and can be changed as appropriate. For example, the processes in steps S729 to S734 of FIG. 18B are compared with the maximum value of the feature point similarity every time the feature point similarity S _k is calculated for the set ψ _k of the feature point pairs, and the feature point similarity is maximized. Processing that holds only a set of feature point pairs as values may be used. In this way, by holding only a set of feature points having the maximum feature point similarity in the feature point pair group extraction process, the amount of information stored in the similarity information 125 of the storage unit 120 is reduced and stored. An increase in the capacity of the unit 120 can be suppressed.

Furthermore, the processing of steps S729 to S734 in FIG. 18 includes, for example, a process of calculating a feature point similarity S _k for each set of feature points ψ _k after generating all sets ψ of possible feature point pairs. It may be.

In addition, the process of calculating the feature point similarity S _k for the set ψ _k of feature point pairs is not limited to the above-described equation (1), and for example, the feature points by the above-described equation (3) or equation (4). The similarity score S _k may be calculated.

  In addition, the global feature information according to the present embodiment is not limited to the distance between the feature point and the vertex of the global region (the contour of the object), but, for example, the length of a perpendicular line extending from the feature point to each side of the global region It is good.

[Third Embodiment]
In the present embodiment, another example of a method for extracting a feature point pair group based on the similarity of local feature information of feature point pairs and the similarity of global feature information will be described.

  The functional configuration of the position / orientation estimation apparatus according to the present embodiment may be the same as the functional configuration of the position / orientation estimation apparatus 1 according to the first embodiment. That is, the position / orientation estimation apparatus 1 according to this embodiment includes a position / orientation estimation unit 110, a storage unit 120, and a projection image generation unit 130, as illustrated in FIG. The position / orientation estimation unit 110 includes an image acquisition unit 111, a feature point detection unit 112, a local feature information extraction unit 113, a global region determination unit 114, a feature point pair group extraction unit 115, and a position / orientation calculation unit 116. (See FIG. 2). The feature point pair group extraction unit 115 includes a global feature information extraction unit 115A, a set creation unit 115B, a local feature information comparison unit 115C, a global feature information comparison unit 115D, a similarity calculation unit 115E, and a calculation processing control unit. 115F and a feature point pair group determination unit 115G (see FIG. 3).

  Further, as shown in FIG. 2, the storage unit 120 includes the registered image 121, the target image 122, the registered image feature information 123, the target image feature information 124, the similarity information 125, and the label pasting position 129. The various data including are stored.

  Note that, as in the first embodiment, the global region determination unit 114 of the position / orientation estimation apparatus 1 according to the present embodiment calculates a minimum circumscribed polygon including a plurality of feature points detected from an image, and calculates the minimum circumscribed polygon. The polygon is verified to the global areas D and DU. Further, the global feature information extraction unit 115A of the position / orientation estimation apparatus 1 according to the present embodiment extracts the distance from the feature point to each base point as the global feature information for each feature point using the center of gravity of the global region as the base point ( calculate).

  For example, the position / orientation estimation apparatus 1 according to the present embodiment repeatedly performs the processes of steps S1 to S9 illustrated in FIG. 4 at predetermined time intervals. The position / orientation estimation apparatus 1 may repeatedly perform a series of processes in steps S1 to S9 as one processing unit, or may perform the processes in steps S1 to S9 in a pipeline.

  Note that the position / orientation estimation apparatus 1 according to the present embodiment performs the process shown in FIGS. 26A and 26B as the feature point pair group extraction process in step S6 of the processes in steps S1 to S9. Also in this embodiment, the feature point pair group extraction process is performed by the feature point pair group extraction unit 115.

  FIG. 26A is a flowchart (part 1) for explaining the contents of the feature point pair group extraction processing according to the third embodiment. FIG. 26B is a flowchart (part 2) for explaining the contents of the feature point pair group extraction processing according to the third embodiment.

  As shown in FIG. 26A, the feature point pair group extraction unit 115 according to the present embodiment first performs global feature information extraction processing (step S741) for extracting global feature information from each of the target image and the feature image. The process of step S741 is performed by the global feature information extraction unit 115A of the feature point pair group extraction unit 115. The global feature information extraction unit 115 according to the present embodiment calculates the distance from the feature point to the center of gravity of the global region D for each feature point of the target image, and from the feature point to the global region DU for each feature point of the registered image. And calculating the distance to the center of gravity (see FIG. 8). The global feature information extraction unit 115 sets the distance from the feature point calculated for each feature point of the target image to the center of gravity of the global region D as global feature information for each feature point of the target image, and uses the global feature information as the target image feature It is stored in the storage unit 120 as one piece of information 124. Similarly, the global feature information extraction unit 115 sets the distance from the feature point calculated for each feature point of the registered image to the center of gravity of the global region DU as the global feature information for each feature point of the registered image, and uses the global feature information. It is stored in the storage unit 120 as one of the registered image feature information 123.

  Next, the feature point pair group extraction unit 115 sets a variable k for identifying a set of feature point pairs to 0 (step S742). The calculation process control unit 115F performs the process of step S742.

  After step S742, the feature point pair group extraction unit 115 performs processing (steps S743 to S745) for calculating the feature point similarity in each of a plurality of sets of feature point pairs based on the local feature information. The processes in steps S743 to S745 are performed by the set generation unit 115B, the local feature information comparison unit 115C, and the similarity calculation unit 115E under the control of the calculation processing control unit 115F.

When the process of step S742 is completed, the feature point pair group extraction unit 115 next generates a set of feature points ψ _k based on the feature points of the target image and the feature points of the registered image (step S743). The set generation unit 115B performs the process of step S743. As described in the first embodiment, the set generation unit 115B generates a set ψ _k of feature point pairs based on the feature points of the target image and the feature points of the registered image according to a known generation method. Note that the set generation unit 115B can generate based on the total number N of feature points of the target image and the total number M of feature points of the registered image in one process of step S743 according to a predetermined generation rule (generation algorithm). A set of a set of feature point pairs is generated.

Next, the feature point pair group extraction unit 115 calculates the similarity SL _{n, m} of local feature information in each feature point pair included in the set ψ _k of feature point pairs (step S744). The local feature information comparison unit 115C performs the process of step S744. For each feature point pair, the local feature information comparison unit 115C compares the local feature information of the feature point of the target image stored in the target image feature information 124 and the feature point of the registered image 121 stored in the registered image feature information 123. The local feature information is compared to calculate the similarity SL _{n, m} of the local feature information. The local feature information comparison unit 115C calculates the similarity SL _{n, m} between the local feature information of the feature point of the target image 122 and the local feature information of the feature point of the registered image 121 according to a known similarity calculation method.

Next, the feature point pair group extraction unit 115 calculates the feature point similarity S _k for the set ψ _k of feature point pairs based on the similarity SL _{n, m} of the local feature information (step S745). The similarity calculation unit 115E performs the process of step S745. For example, the similarity calculation unit 115E calculates the feature point similarity S _k for the set ψ _k of feature point pairs by the following equation (5).

The similarity calculation unit 115E associates the calculated feature point similarity S _k with the feature point set ψ _k and stores it in the storage unit 120 as one of the similarity information 125, and calculates the feature point similarity S _k . This is notified to the calculation processing control unit 115F.

When the feature point similarity S _k for the set [psi _k feature point pairs are calculated, calculation control section 115F determines whether to generate all the set of feature point pairs that can generate (step S746) . In step S746, the calculation processing control unit 115F calculates the total number of sets that can be generated calculated based on the total number N of feature points of the target image and the total number M of feature points of the registered image, and a variable k that identifies the generated set. It is determined whether or not all sets have been generated. For example, when there are K sets of feature point pairs that can be generated, the calculation processing control unit 115F determines whether or not k ≧ K−1 (since the variable k starts from 0). When k <K−1, the calculation processing control unit 115F determines that there is a set of feature point pairs that have not been generated.

  When there is a set of feature point pairs that have not been generated (step S746; NO), the calculation processing control unit 115F updates the variable k that identifies the set of feature point pairs to k + 1 (step S747). Thereafter, the calculation process control unit 115F causes the set generation unit 115B, the local feature information comparison unit 115C, and the similarity calculation unit 115E to perform the processes of steps S743 to S745.

On the other hand, when all sets of feature point pairs have been generated (step S746; YES), the calculation processing control unit 115F next causes the feature point pair group determination unit 115G to determine the feature point pair group. The feature point pair group determination unit 115G refers to the similarity information 125 of the storage unit 120, and determines a set of feature point pairs having the maximum feature point similarity as a feature point pair group (step S748). Note that the feature point pair determining unit 115G determines the feature point pair set having the maximum feature point similarity as the feature point pair group by a calculation method in which the higher the similarity, the larger the calculated value. This is a case where the feature point similarity _Sk is calculated. That is, the similarity is a positive value, and when calculating the feature point similarity S _k by calculating how the value of the degree of similarity is calculated higher decreases, the feature point pair group determination unit 115G, feature points similar A set of feature point pairs having a minimum degree is determined as a feature point pair group.

  After the feature point pair group is determined, the feature point pair group extraction unit 115, as shown in FIG. 26B, of the feature point pairs included in the feature point pair group has a low similarity in global feature information. Is performed (steps S749 to S752). The processes in steps S749 to S752 are performed by the global feature information comparison unit 115D and the feature point pair group determination unit 115G under the control of the calculation processing control unit 115F.

  When the feature point pair group is determined, the feature point pair group extraction unit 115 next selects one feature point pair from the feature point pair group candidates as shown in FIG. 26B (step S749). The calculation process control unit 115F performs the process of step S749. The calculation processing control unit 115F selects one feature point pair from among a plurality of feature point pairs included in the feature point pair group according to a predetermined selection rule. The selection rule is, for example, a rule of selecting a feature point pair with the earliest registration order in the feature point pair group from among unselected feature point pairs.

Next, the feature point pair group extraction unit 115 calculates the similarity SC _{n, m} of the global feature information in the selected feature point pair (step S750). The process of step S750 is performed by the global feature information comparison unit 115D. As described in the first embodiment, the global feature information comparison unit 115D has the global feature information C _n for the feature point P _n of the target image in the feature point pair and the global feature for the feature point PU _m of the registered image. Based on the difference with the information PU _m , the similarity SC _{n, m of} the global feature information is set. At this time, the global feature information comparison unit 115D calculates the similarity SC _{n, m} according to a predetermined function in which the value of the similarity SC _{n, m} of the global feature information increases as the difference in the global feature information decreases.

Next, the feature point pair group extraction unit 115 determines whether or not the calculated similarity SC _{n, m of} the global feature information is _{equal to} or greater than a threshold (step S751). The determination in step S751 is performed by, for example, the feature point pair group determination unit 115G. When the feature point P _n of the target image in the selected feature point pair and the feature point PU _m of the registered image are a correct combination, the similarity SC _{n, m} of the global feature information for the feature point pair becomes high. That is, when the similarity SC _{n, m} of the global feature information for the selected feature point pair is low, the combination of the feature point P _n of the target image and the feature point PU _m of the registered image in the feature point pair is the feature It can be determined that the combination included in the point pair group is inappropriate. Therefore, when the calculated similarity SC _{n, m of} the global feature information is smaller than the threshold (step S751; NO), the feature point pair group determination unit 115G excludes the selected feature point pair from the feature point pair group ( Step S752). After excluding the feature point pair, the feature point pair group determination unit 115G notifies the calculation processing control unit 115F that the process of step S752 has been completed. On the other hand, when the similarity SC _n , _m of the global feature information of the selected feature point pair is equal to or greater than the threshold (step S751; YES), the feature point pair group determination unit 115G calculates that the process of step S752 is skipped. The processing control unit 115F is notified.

  Upon receiving the notification from the feature point pair group determination unit 115G, the calculation processing control unit 115F determines whether all the feature point pairs included in the feature point pair group have been selected (step S753). When there is an unselected feature point pair (step S753; NO), the calculation processing control unit 115F selects an unselected feature point pair (step S749), and determines whether or not to exclude the feature point pair. Processing for determination (steps S750 to S752) is performed.

On the other hand, when all feature points included in the feature point pair group are selected (step S753;), the calculation processing control unit 115F next causes the feature point pair group determination unit 115G to determine the feature point pair group. The feature point pair group determination unit 115G corrects the feature point pair group after the feature point pair having a low similarity SC _n , _m of the global feature information is excluded between the feature point of the target image and the feature point of the registered image. A pair of feature points representing the relationship is output (step S754). When the feature point pair group determination unit 115G determines the feature point pair group, the feature point pair group extraction process according to the present embodiment is completed.

  As described above, in the feature point pair group extraction processing according to this embodiment, after determining a feature point pair group based on the similarity of local feature information in each feature point pair included in the set of feature point pairs, A feature point pair having a low similarity of global feature information is excluded from the point pair group. The local feature information is information representing the feature on the image in the local region in the vicinity of the feature point including the feature point. For this reason, even if the feature point pair does not represent the correct correspondence between the feature point of the target image and the feature point of the registered image, if the feature similarity on the local image is high, the feature point of the target image May be included in a pair of feature points representing a correct correspondence between the feature points and the registered image feature points.

  On the other hand, the base point set based on the global region including a plurality of feature points (for example, the center of gravity of the global region) is substantially coincident with the position in the region where the target object of the target image is reflected in the registered image. To do. Therefore, when the distance from the feature point to the base point is the global feature information, the similarity of the global feature information is low unless the correspondence between the feature point of the target image and the feature point of the registered image is correct. Therefore, by excluding the feature point pairs with low similarity of the global target image included in the feature point pair group specified based on the local feature information, the feature point of the target image and the feature point of the registered image are excluded from the feature point pair group. It is possible to exclude a pair of feature points whose correspondence with is incorrect. Based on the similarity of the global feature information, it is possible to exclude a feature point pair whose feature point correspondence is not correct from a feature point pair group (a set of feature point pairs) used for position and orientation estimation. Therefore, according to this embodiment, the estimation accuracy of the position and orientation of the object is improved.

[Fourth Embodiment]
FIG. 27 is a diagram illustrating a functional configuration of a position and orientation estimation apparatus according to the fourth embodiment.

  As illustrated in FIG. 27, the position / orientation estimation apparatus 1 according to this embodiment includes a position / orientation estimation unit 110, a storage unit 120, and a projection image generation unit 130.

  The position / orientation estimation unit 110 includes an image acquisition unit 111, a feature point detection unit 112, a local feature information extraction unit 113, a global region determination unit 114, a feature point pair group extraction unit 115, and a position / orientation calculation unit 116. ,including. The feature point pair group extraction unit 115 includes a global feature information extraction unit 115A, a set creation unit 115B, a local feature information comparison unit 115C, a global feature information comparison unit 115D, a similarity calculation unit 115E, and a calculation processing control unit. 115F and a feature point pair group determination unit 115G (see FIG. 3). Each of the above-described units in the position / orientation estimation unit 110 of the position / orientation estimation apparatus 1 according to the present embodiment has the functions described in the first to third embodiments.

  The position / orientation estimation unit 110 of the position / orientation estimation apparatus 1 according to the present embodiment further includes an image editing unit 117 in addition to the above-described units. The image editing unit 117 detects an object that shields the target object 5 such as the arm of the operator 8 that is reflected in the target image acquired from the imaging device 2, and converts the target image into an image from which the shielded object has been removed. To edit.

  As illustrated in FIG. 27, the storage unit 120 includes a registered image 121, a target image 122, registered image feature information 123, target image feature information 124, similarity information 125, and a label pasting position 129. Various types of data are stored. It is stored (stored) in the storage unit 120. These pieces of information are information as described in the first to third embodiments.

  Based on the position and orientation of the target object 5 calculated by the position and orientation estimation unit 110 and the label attaching position 129 of the storage unit 120, the projection image generation unit 130 projects an image to be projected onto the target object 5 shown in the target image. It is generated and output to the projector 3.

  For example, the position / orientation estimation apparatus 1 according to the present embodiment repeatedly performs the processes of steps S1, S11, and S2 to S9 shown in FIG. 28 at predetermined time intervals. The position / orientation estimation apparatus 1 may repeatedly perform a series of processes of steps S1, S11, and S2 to S9 as a single processing unit, or pipeline the processes of steps S1, S11, and S2 to S9. You may go.

  FIG. 28 is a flowchart for describing processing performed by the position and orientation estimation apparatus according to the fourth embodiment.

  As shown in FIG. 28, the position / orientation estimation apparatus 1 according to the present embodiment first acquires a target image obtained by capturing an image of an object from the imaging apparatus 2 (step S1). The process of step S1 is performed by the image acquisition unit 111.

Next, the position / orientation estimation apparatus 1 removes the shielding object from the target image (step S11). The image editing unit 117 performs the process in step S11. For example, the image editing unit 117 extracts Histograms of Oriented Gradients (HOG) features from the target image, and detects the presence or absence of an object different from the target object and the background object moving in a predetermined direction in the target image. When there is an object (shielding object) different from the object and the background object in the target image, the image editing unit 117 does not capture the past shielding object in the partial area where the shielding object is reflected in the target image. Replace with the corresponding partial region in the target image. When there is an obstruction in the target image, the image editing unit 117 stores the target image from which the obstruction has been removed in the storage unit 120. Next, the position and orientation estimation device 1 detects feature points from the target image and the registered image. A feature point detection process (step S2) is performed. The feature point detection unit 112 performs the process of step S2. As described in the first embodiment, the feature point detection unit 112 calculates the feature points by, for example, SURF or SIFT.

  Of the processes performed by the position / orientation estimation apparatus 1 according to the present embodiment, the processes of steps S2 to S9 may be any of the processes described in the first to third embodiments. Therefore, in this embodiment, the description about the content of the process of step S2 to S9 of FIG. 28 is abbreviate | omitted.

  Thus, in the process of estimating the position and orientation of the target object according to the present embodiment, when another object (shielding object) different from the target object and the background object (for example, the conveyor belt 401) exists in the target image. Then, an editing process is performed on the target image to remove the shielding object. For this reason, for example, the arm part of the worker 8 on the work line 10 is reflected in the target image, an increase in feature points unnecessary for position and orientation estimation, and a part or all of the target object 5 is hidden in the arm part. Therefore, it is possible to suppress the deterioration of the position and orientation estimation accuracy due to the occurrence.

  Further, in the process of estimating the position and orientation of the target object according to the present embodiment, the similarity between the local feature information extracted from around the feature point of the target object, the center of gravity of the global region set in the image, and the feature point Based on the similarity of the global feature information representing the relationship, the position and orientation of the object are estimated. Therefore, similarly to the position / orientation estimation process described in the first to third embodiments, the position / orientation estimation accuracy can be improved.

FIG. 29 is a diagram for explaining a method of removing the shielding object from the target image.
Below 29, the target image _{122 t-1} captured at time t-1, and the target image 122 _t captured at time t, the three target image and the target image _{122 t + 1} captured at time t + 1 Show. In addition, above the three target images 122 _t−1 , 122 _t , and 122 _{t + 1} , the state of the imaging region 410 at the time when the target images are captured is shown.

The object existing in the imaging region 410 at the time t-1 is only the object 5 except for the conveyor belt 401 which is a predetermined background object. Therefore, except for the background object, only the object 5 is shown in the object image 122 _t-1 captured at time t-1.

On the other hand, at the time t, the arm portion 801 of the worker different from the target object 5 and the predetermined background object (conveyance belt 401) exists in the imaging region 410. Therefore, the target image 122t is captured at time t, at the time acquired from the image pickup device 2, the arm portion 801 of the operator to the position shown by dotted lines in the image 112 in _t is captured. As described above, when an object (worker's arm 801) different from the object 5 and the background is captured, a plurality of feature points are detected from the arm 801 in the feature point detection process (step S2). There is a possibility that. Therefore, the position and orientation estimation apparatus 1 according to the present embodiment, the image 112 _t of such arms 801, when detecting a different object (obstacle) is the object 5 and a predetermined background object, the shield Remove. Specifically, specify a rectangular partial regions BL1 the arm portion 801 is captured in the image 112 _t, the partial region BL1 past target image (for example, time t-1 target image _{122 t-1} captured in) Is replaced with the image of the corresponding partial region BL0. As a result, it is possible to remove the worker's arm 801 from the target image 122t imaged at time t and prevent a large number of feature points from being detected from the background area.

Further, at time t + 1, there is an arm portion 802 of the worker that is different from the object 5 and the predetermined background object (conveyor belt 401) in the imaging region 410. Moreover, at time t + 1, a part of the arm 802 of the worker shields a part of the object 5. For this reason, when the target image 122 _{t + 1} captured at time t is acquired from the imaging device 2, the arm portion 802 of the worker is reflected at the position indicated by the dotted line in the image 112 _{t + 1} . The image is missing. As described above, when an object (worker's arm 802) different from the object 5 and the background object is captured, a plurality of feature points are detected from the arm 801 in the feature point detection process (step S2). There is a possibility that. Furthermore, when a part of the target object 5 is missing, the feature point detection process may fail to detect the feature point of the target object 5 and may fail to estimate the position and orientation of the target object 5. Even in such a case, the position / orientation estimation apparatus 1 according to the present embodiment removes an object (shielding object) different from the target object 5 and the predetermined background object, such as the arm 802 detected from the image 112 _t-1. To do. Specifically, a rectangular partial area BL3 in which a part of the arm 802 and the object 5 in the image 112 _{t + 1} is shown is specified, and the partial area BL3 is captured in the past target image (for example, a target image captured at time t). The image of the corresponding partial region BL2 at 122 _t ) is replaced. At this time, for example, the image editing unit 117 cuts out the corresponding partial region BL2 whose position and size of the region in which the object 5 is reflected is closest to the partial region BL3 from the corresponding partial regions in the plurality of past target images. Then, the partial region BL3 is replaced. Thereby, it is possible to remove the worker's arm 802 from the target image 122 _{t + 1} captured at time t + 1, and to prevent a situation in which many feature points are detected from the background area. In addition, since a part of the target object 5 missing by the arm portion 802 in the target image 122 _{t + 1} acquired from the imaging device 2 is complemented from the other target image, a reduction in accuracy of estimation of the position and orientation of the target object 5 is prevented. It becomes possible.

  The position / orientation estimation apparatus 1 described in each of the above embodiments can be realized by, for example, a computer and a program executed by the computer. Hereinafter, the position / orientation estimation apparatus 1 realized by a computer and a program will be described with reference to FIG.

FIG. 30 is a diagram illustrating a hardware configuration of a computer.
As shown in FIG. 30, the computer 15 includes a processor 1501, a main storage device 1502, an auxiliary storage device 1503, an input device 1504, an output device 1505, an input / output interface 1506, a communication control device 1507, and a medium. A driving device 1508. These elements 1501 to 1508 in the computer 15 are connected to each other by a bus 1510 so that data can be exchanged between the elements.

  The processor 1501 is a central processing unit (CPU), a micro processing unit (MPU), or the like. The processor 1501 controls the overall operation of the computer 15 by executing various programs including an operating system. Further, the processor 1501 performs, for example, each process illustrated in FIGS.

  The main storage device 1502 includes a read only memory (ROM) and a random access memory (RAM) not shown. In the ROM of the main storage device 1502, for example, a predetermined basic control program read by the processor 1501 when the computer 15 is started is recorded in advance. The RAM of the main storage device 1502 is used as a working storage area as needed when the processor 1501 executes various programs. The RAM of the main storage device 1502 can be used as a storage unit that stores, for example, registered image feature information 123, target image feature information 124, similarity information 125, and the like in the position and orientation estimation device 1.

  The auxiliary storage device 1503 is a storage device having a larger capacity than the RAM of the main storage device 1502, such as a hard disk drive (HDD) and a non-volatile memory such as a flash memory (including a solid state drive (SSD)). is there. The auxiliary storage device 1503 can be used to store various programs executed by the processor 1501 and various data. The auxiliary storage device 1503 can be used for storing, for example, a position / orientation estimation program including the processes shown in FIGS. Further, the auxiliary storage device 1503 can be used as the storage unit 120 that stores, for example, the registered image 121, the captured image 122, the registered image feature information 123, the target image feature information 124, the similarity information 125, the label pasting position 129, and the like. It is.

  The input device 1504 is, for example, a keyboard device or a touch panel device. When an operator (user) of the computer 15 performs a predetermined operation on the input device 1504, the input device 1504 transmits input information associated with the operation content to the processor 1501. The input device 1504 can be used, for example, for inputting a command for starting a process for estimating the position and orientation of an object, a command related to other processes that can be executed by the computer 15, and for inputting various setting values.

  The output device 1505 includes, for example, a device such as a liquid crystal display device, a sound reproduction device such as a speaker, and the projector 3 that projects an image on the object 5.

  The input / output interface 1506 connects the computer 15 and other electronic devices. The input / output interface 1506 includes, for example, a universal serial bus (USB) standard connector. The input / output interface 1506 can be used, for example, for connection between the computer 15 and the imaging device 2 that images the object 5.

  The communication control device 1507 is a device that connects the computer 15 to a network such as the Internet and controls various communications between the computer 15 and other electronic devices via the network. The communication control device 1507 can be used, for example, for communication between the computer 15 and a monitoring system that monitors the work status in the work line 10 from a remote location. When the computer 15 is installed in a management facility or the like different from the installation location of the work line 10, the communication control device 1507 includes the imaging device 2 and the projector 3 installed at the installation location of the work line 10, It can be used for communication with the computer 15.

  The medium driving device 1508 reads programs and data recorded in the portable storage medium 16 and writes data stored in the auxiliary storage device 1503 to the portable storage medium 16. For the medium driving device 1508, for example, a memory card reader / writer corresponding to one type or a plurality of types of standards can be used. When a memory card reader / writer is used as the medium driving device 1508, the portable storage medium 16 is a memory card (flash memory) conforming to a standard supported by the memory card reader / writer, for example, Secure Digital (SD) standard. ) Etc. can be used. As the portable recording medium 16, for example, a flash memory provided with a USB standard connector can be used. Further, when the computer 15 is equipped with an optical disk drive that can be used as the medium driving device 1508, various optical disks that can be recognized by the optical disk drive can be used as the portable recording medium 16. Examples of the optical disc that can be used as the portable recording medium 16 include a Compact Disc (CD), a Digital Versatile Disc (DVD), and a Blu-ray Disc (Blu-ray is a registered trademark). The portable recording medium 16 can be used for storing, for example, a position / orientation estimation program including the processes shown in FIGS. Further, the portable recording medium 16 is used as the storage unit 120 that stores, for example, the registered image 121, the captured image 122, the registered image feature information 123, the target image feature information 124, the similarity information 125, the label pasting position 129, and the like. Is possible.

  When the operator inputs an instruction to start the position / orientation estimation process of the object to the computer 15 using the input device 1504 or the like, the position stored by the processor 1501 in a non-temporary recording medium such as the auxiliary storage device 1503 or the like. Read and execute the posture estimation program. In this process, the processor 1501 functions (operates) as the position / orientation estimation unit 110 and the projection image generation unit 130 in the position / orientation estimation apparatus 1 of FIGS. The RAM of the main storage device 1502, the auxiliary storage device 1503, and the like function as the storage unit 120 in the position / orientation estimation device 1 of FIGS.

  Note that the computer 15 operated as the position / orientation estimation apparatus 1 does not need to include all the elements 1501 to 1508 illustrated in FIG. 30, and some elements may be omitted depending on applications and conditions. For example, the computer 15 may be one in which the communication control device 1507 and the medium driving device 1508 are omitted.

The following additional notes are disclosed for each of the embodiments described above.
(Appendix 1)
A detection unit that detects a feature point from a target image in which a target object that is an estimation target of a position and orientation is captured, and a registered image in which the target object is captured in a predetermined position and orientation;
The feature point indicating the feature of the target object is selected from among the plurality of feature points detected from the target image, and the local feature information indicating the features around the selected feature point in the target image, and the registration A first extraction unit for extracting local feature information indicating features around the plurality of feature points in the image;
A global region setting unit for setting a global region including a plurality of the feature points in each of the target image and the registered image;
The global feature information in the target image is extracted based on the feature point of the target image and the global region, and the global feature information in the registered image is extracted based on the feature point of the registered image and the global region. A second extraction unit for extracting;
The feature point of the target image based on the similarity of the local feature information and the similarity of the global feature information in a feature point pair of the feature point of the target image and the feature point of the registered image And a specifying unit for specifying a feature point pair group indicating a correspondence relationship between the registered image and the feature point of the registered image;
A calculation unit that calculates a position and orientation of the object in the target image based on the identified feature point pair group;
A position / orientation estimation apparatus comprising:
(Appendix 2)
The global region setting unit determines a minimum circumscribed polygon including the plurality of feature points in the target image as the global region of the target image, and determines a minimum circumscribed polygon including the plurality of feature points in the registered image. Determining the global area of the registered image;
The position and orientation estimation apparatus according to Supplementary Note 1, wherein
(Appendix 3)
The second extraction unit sets a base point based on the shape of the global region, and a distance from the feature point to the base point in the target image, and a distance from the feature point to the base point in the registered image Is extracted as the global feature information,
The position and orientation estimation apparatus according to Supplementary Note 2, wherein
(Appendix 4)
The second extraction unit sets a center of gravity of the global region as the base point;
The position and orientation estimation apparatus according to Supplementary Note 3, wherein
(Appendix 5)
The global region setting unit determines a triangular region including a plurality of the feature points in the target image as the global region of the target image, and sets a triangular region including the plurality of feature points in the registered image to the registered image. Determine the global area,
The second extraction unit is based on any one of the center of gravity, inner center, outer center, vertical center, and side center of the global region, the distance from the feature point to the base point in the target image, and the registered image A distance from the feature point to the base point is extracted as the global feature information;
The position and orientation estimation apparatus according to Supplementary Note 1, wherein
(Appendix 6)
The global area setting unit determines an area surrounded by the outline of the target object in the target image as the global area of the target image, and registers the area surrounded by the outline of the target object in the registered image. To determine the global region of the image,
The position and orientation estimation apparatus according to Supplementary Note 1, wherein
(Appendix 7)
The global area setting unit sets the polygonal global area in the target image and the registered image,
The second extraction unit is configured to determine a distance from the feature point to the vertex of the global area in the target image and a distance from the feature point to the vertex of the global area in the registered image. Extract as,
The position and orientation estimation apparatus according to Supplementary Note 1, wherein
(Appendix 8)
The global area setting unit sets the polygonal global area in each of the target image and the registered image,
The second extraction unit includes: a length of a perpendicular drawn from the feature point in the target image to a side of the global region; and a length of a perpendicular drawn from the feature point in the registered image to the side of the global region. Extracting as the second feature information;
The position and orientation estimation apparatus according to Supplementary Note 1, wherein
(Appendix 9)
The specifying unit generates a plurality of sets of feature point pairs in which the feature points of the registered image are associated with the feature points of the target image on a one-to-one basis, For each set of feature points, a feature point similarity is calculated based on the similarity of the local feature information in the feature point pair included in the set of feature points and the similarity of the global feature information. A set of the feature point pairs that maximizes the calculated feature point similarity is specified as the feature point pair group;
The position and orientation estimation apparatus according to Supplementary Note 1, wherein
(Appendix 10)
The specifying unit specifies the feature point pair after specifying the feature point pair group based on the similarity of the local feature information in the feature point pair by the feature point of the target image and the feature point of the registered image. Of the feature point pairs included in the point pair group, the feature point pair whose similarity of the global feature information is lower than a threshold is excluded from the feature point pair group,
The calculation unit calculates the position and orientation of the object shown in the target image based on the feature point pair group excluding feature points whose similarity of the global feature information is lower than a threshold value.
The position and orientation estimation apparatus according to Supplementary Note 1, wherein
(Appendix 11)
The first extraction unit extracts the local feature information of all feature points detected from the target image, and similarity between the extracted local feature information and the local feature information of the feature points of the registered image Selecting the feature point whose degree is higher than a threshold as the feature point of the object;
The position and orientation estimation apparatus according to Supplementary Note 1, wherein
(Appendix 12)
A storage unit for storing the target image;
An object different from the object specified as the object and the background object is detected from the target image, and a partial region including the detected object in the target image is stored in the target image stored in the storage unit. An image editing section that replaces a partial area corresponding to the area;
The position and orientation estimation apparatus according to appendix 1, further comprising:
(Appendix 13)
Computer
A feature point is detected from a target image in which a target object whose position and orientation are to be estimated is recorded and a registered image in which the target object is captured in a predetermined position and orientation;
The feature point indicating the feature of the target object is selected from among the plurality of feature points detected from the target image, and the local feature information indicating the features around the selected feature point in the target image, and the registration Extracting local feature information indicating features around the plurality of feature points in the image;
A global region including a plurality of the feature points is set for each of the target image and the registered image,
The global feature information in the target image is extracted based on the feature point of the target image and the global region, and the global feature information in the registered image is extracted based on the feature point of the registered image and the global region. Extract and
The feature point of the target image based on the similarity of the local feature information and the similarity of the global feature information in a feature point pair of the feature point of the target image and the feature point of the registered image And a feature point pair group indicating a correspondence relationship between the registered image and the feature point of the registered image,
Based on the specified feature point pair group, the position and orientation of the object shown in the target image is calculated.
A position and orientation estimation method characterized by executing processing.
(Appendix 14)
A feature point is detected from a target image in which a target object whose position and orientation are to be estimated is recorded and a registered image in which the target object is captured in a predetermined position and orientation;
The feature point indicating the feature of the target object is selected from among the plurality of feature points detected from the target image, and the local feature information indicating the features around the selected feature point in the target image, and the registration Extracting local feature information indicating features around the plurality of feature points in the image;
A global region including a plurality of the feature points is set for each of the target image and the registered image,
The global feature information in the target image is extracted based on the feature point of the target image and the global region, and the global feature information in the registered image is extracted based on the feature point of the registered image and the global region. Extract and
The feature point of the target image based on the similarity of the local feature information and the similarity of the global feature information in a feature point pair of the feature point of the target image and the feature point of the registered image And a feature point pair group indicating a correspondence relationship between the registered image and the feature point of the registered image,
Based on the specified feature point pair group, the position and orientation of the object shown in the target image is calculated.
A position and orientation estimation program for causing a computer to execute processing.

DESCRIPTION OF SYMBOLS 1 Position and orientation estimation apparatus 110 Position and orientation estimation part 111 Image acquisition part 112 Feature point detection part 113 Local feature information extraction part 114 Global area determination part 115 Feature point pair group extraction part 115A Global feature information extraction part 115B Set generation part 115C Local feature Information comparison unit 115D Global feature information comparison unit 115F Similarity calculation unit 115G Calculation processing control unit 115F Feature point pair group determination unit 116 Position and orientation calculation unit 117 Image editing unit 120 Storage unit 121 Registered image 122 Target image 123 Registered image feature information 124 Target image feature information 125 Similarity information 129 Label application position 130 Projected image generation unit 2 Imaging device 3 Projector 4 Belt conveyor 401 Conveying belt 410 Imaging regions 5, 5A, 5B, 5U Object 6 Projected image 7 Label 8 Worker 801 802 Arm 10 Work line 15 Compu Data 1501 processor 1502 processor storage 1503 auxiliary memory 1504 input device 1505 output devices 1506 output interface 1507 communication controller 1508 medium drive device 16 portable recording medium

Claims (13)

A detection unit that detects a feature point from a target image in which a target object that is an estimation target of a position and orientation is captured, and a registered image in which the target object is captured in a predetermined position and orientation;
The feature point indicating the feature of the target object is selected from among the plurality of feature points detected from the target image, and the local feature information indicating the features around the selected feature point in the target image, and the registration A first extraction unit for extracting local feature information indicating features around the plurality of feature points in the image;
A global region setting unit for setting a global region including a plurality of the feature points in each of the target image and the registered image;
The global feature information in the target image is extracted based on the feature point of the target image and the global region, and the global feature information in the registered image is extracted based on the feature point of the registered image and the global region. A second extraction unit for extracting;
The feature point of the target image based on the similarity of the local feature information and the similarity of the global feature information in a feature point pair of the feature point of the target image and the feature point of the registered image And a specifying unit for specifying a feature point pair group indicating a correspondence relationship between the registered image and the feature point of the registered image;
A calculation unit that calculates a position and orientation of the object in the target image based on the identified feature point pair group;
A position / orientation estimation apparatus comprising: The global region setting unit determines a minimum circumscribed polygon including the plurality of feature points in the target image as the global region of the target image, and determines a minimum circumscribed polygon including the plurality of feature points in the registered image. Determining the global area of the registered image;
The position and orientation estimation apparatus according to claim 1. The second extraction unit sets a base point based on the shape of the global region, and a distance from the feature point to the base point in the target image, and a distance from the feature point to the base point in the registered image Is extracted as the global feature information,
The position and orientation estimation apparatus according to claim 2. The second extraction unit sets a center of gravity of the global region as the base point;
The position and orientation estimation apparatus according to claim 3. The global area setting unit determines an area surrounded by the outline of the target object in the target image as the global area of the target image, and registers the area surrounded by the outline of the target object in the registered image. To determine the global region of the image,
The position and orientation estimation apparatus according to claim 1. The global area setting unit sets the polygonal global area in the target image and the registered image,
The second extraction unit is configured to determine a distance from the feature point to the vertex of the global area in the target image and a distance from the feature point to the vertex of the global area in the registered image. Extract as,
The position and orientation estimation apparatus according to claim 1. The global area setting unit sets the polygonal global area in each of the target image and the registered image,
The second extraction unit includes: a length of a perpendicular drawn from the feature point in the target image to a side of the global region; and a length of a perpendicular drawn from the feature point in the registered image to the side of the global region. Extracting as the second feature information;
The position and orientation estimation apparatus according to claim 1. The specifying unit generates a plurality of sets of feature point pairs in which the feature points of the registered image are associated with the feature points of the target image on a one-to-one basis, For each set of feature points, a feature point similarity is calculated based on the similarity of the local feature information in the feature point pair included in the set of feature points and the similarity of the global feature information. A set of the feature point pairs that maximizes the calculated feature point similarity is specified as the feature point pair group;
The position and orientation estimation apparatus according to claim 1. The specifying unit specifies the feature point pair after specifying the feature point pair group based on the similarity of the local feature information in the feature point pair by the feature point of the target image and the feature point of the registered image. Of the feature point pairs included in the point pair group, the feature point pair whose similarity of the global feature information is lower than a threshold is excluded from the feature point pair group,
The calculation unit calculates the position and orientation of the object shown in the target image based on the feature point pair group excluding feature points whose similarity of the global feature information is lower than a threshold value.
The position and orientation estimation apparatus according to claim 1. The first extraction unit extracts the local feature information of all feature points detected from the target image, and similarity between the extracted local feature information and the local feature information of the feature points of the registered image Selecting the feature point whose degree is higher than a threshold as the feature point of the object;
The position and orientation estimation apparatus according to claim 1. A storage unit for storing the target image;
An object different from the object specified as the object and the background object is detected from the target image, and a partial region including the detected object in the target image is stored in the target image stored in the storage unit. An image editing section that replaces a partial area corresponding to the area;
The position / orientation estimation apparatus according to claim 1, further comprising: Computer
A feature point is detected from a target image in which a target object whose position and orientation are to be estimated is recorded and a registered image in which the target object is captured in a predetermined position and orientation;
The feature point indicating the feature of the target object is selected from among the plurality of feature points detected from the target image, and the local feature information indicating the features around the selected feature point in the target image, and the registration Extracting local feature information indicating features around the plurality of feature points in the image;
A global region including a plurality of the feature points is set for each of the target image and the registered image,
The global feature information in the target image is extracted based on the feature point of the target image and the global region, and the global feature information in the registered image is extracted based on the feature point of the registered image and the global region. Extract and
The feature point of the target image based on the similarity of the local feature information and the similarity of the global feature information in a feature point pair of the feature point of the target image and the feature point of the registered image And a feature point pair group indicating a correspondence relationship between the registered image and the feature point of the registered image,
Based on the specified feature point pair group, the position and orientation of the object shown in the target image is calculated.
A position and orientation estimation method characterized by executing processing. A feature point is detected from a target image in which a target object whose position and orientation are to be estimated is recorded and a registered image in which the target object is captured in a predetermined position and orientation;
The feature point indicating the feature of the target object is selected from among the plurality of feature points detected from the target image, and the local feature information indicating the features around the selected feature point in the target image, and the registration Extracting local feature information indicating features around the plurality of feature points in the image;
A global region including a plurality of the feature points is set for each of the target image and the registered image,
The global feature information in the target image is extracted based on the feature point of the target image and the global region, and the global feature information in the registered image is extracted based on the feature point of the registered image and the global region. Extract and
The feature point of the target image based on the similarity of the local feature information and the similarity of the global feature information in a feature point pair of the feature point of the target image and the feature point of the registered image And a feature point pair group indicating a correspondence relationship between the registered image and the feature point of the registered image,
Based on the specified feature point pair group, the position and orientation of the object shown in the target image is calculated.
A position and orientation estimation program for causing a computer to execute processing.

Images