JP2015203680A - Information processing device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method that selects an image serving as a factor in performance improvement in a position pose estimation using a plurality of images.SOLUTION: An information processing device is configured to: plurally acquire an image having an object body photographed; execute at least one of recognition processing of the object body and position pose estimation processing of the object body on the basis of information on at least one image of the acquired images; execute at least one of the recognition processing of the object body and the position pose estimation processing of the object body on the basis of information on at least two images of the plurality of acquired images; generate first performance information about processing to be executed by first execution means; generate second performance information about processing to be executed by second execution means; and cause display means to display the first performance information and the second performance information.

Description

  The present invention relates to a method for recognizing and displaying an estimation of a position and orientation of an object in an image based on an image obtained by imaging the object.

  At production sites, robotic automation is being promoted to improve production efficiency. In picking or assembling parts by a robot, it is necessary to recognize the target part and further acquire the position and orientation. As a method for realizing this, a method using an image obtained by photographing a target part with a camera has been developed.

  Patent Document 1 discloses a method for improving the accuracy of recognition by using images taken by a plurality of cameras. In this method, multiple stereo image pairs are selected from images captured by multiple cameras, and multiple target objects are recognized using distance values calculated for each stereo image pair. It is improving.

JP 2000-94374 A

However, a plurality of images forming a stereo image pair may include images containing a lot of noise. Also, the use of a plurality of images increases the processing time, and the task may not be completed within a predetermined time.
The present invention has been made in view of the above problems, and an object of the present invention is to select an image suitable for deriving a position and orientation from a plurality of images in recognition or position and orientation estimation using a plurality of images.

  The information processing apparatus according to the present invention includes, for example, an image acquisition unit that acquires a plurality of images obtained by capturing a target object, and a recognition process for the target object based on information about at least one of the acquired images. The target object recognition process based on the first execution means for executing at least one of the process of estimating the position and orientation of the target object and information on at least two of the acquired plurality of images. And second execution means for executing at least one of the process for estimating the position and orientation of the target object, and first performance information for generating the first performance information relating to the process executed by the first execution means. Performance information generating means, second performance information generating means for generating second performance information relating to processing executed by the second execution means, the first performance information and the second sex And a display control means for displaying on the display means and information.

  In recognition or position / orientation estimation using a plurality of images, an image suitable for deriving the position and orientation can be selected from the plurality of images.

It is a figure which shows the structure of the system in 1st Embodiment. It is a figure which shows the structural example of the system using the information processing apparatus of this invention. It is a figure explaining the process sequence of 1st Embodiment. It is a figure which shows the example of GUI in 1st Embodiment. It is a figure which shows the example of GUI in the modification 1. It is a figure which shows the structure of the system in 2nd Embodiment. It is a figure explaining the process sequence of 2nd Embodiment. It is a figure which shows the example of GUI in 2nd Embodiment. It is a figure which shows the structure of the system in 3rd Embodiment. It is a figure explaining the process sequence of 3rd Embodiment. It is a figure which shows the example of GUI in 3rd Embodiment. It is a figure which shows the hardware constitutions of the information processing apparatus in this invention.

  Prior to describing each embodiment according to the present invention, a hardware configuration in which the information processing apparatus shown in each embodiment is mounted will be described with reference to FIG.

  FIG. 12 is a hardware configuration diagram of the information device in the present embodiment. In the figure, a CPU 1210 comprehensively controls devices connected via a bus 1200. The CPU 1210 reads and executes processing steps and programs stored in a read only memory (ROM) 1220. The operating system (OS) and other processing programs, device drivers, and the like according to the present embodiment are stored in the ROM 1220, temporarily stored in a random access memory (RAM) 1230, and executed as appropriate by the CPU 1210. The input I / F 1240 is input as an input signal in a format that can be processed by the information processing apparatus 1 from an external device (display device, operation device, or the like). The output I / F 1250 outputs an output signal to an external device (display device) in a format that can be processed by the display device.

  Each of these functional units is realized by the CPU 1210 expanding a program stored in the ROM 1220 in the RAM 1230 and executing processing according to each flowchart described later. Further, for example, when hardware is configured as an alternative to software processing using the CPU 1210, arithmetic units and circuits corresponding to the processing of each functional unit described here may be configured.

(First embodiment)
In the first embodiment, a method for presenting the user with performance related to position / orientation estimation when the position / orientation estimation of the target object is performed based on images obtained by capturing the target object with a plurality of cameras will be described. More specifically, position and orientation estimation is performed based on each captured image and a plurality of images, and information regarding the performance of each position and orientation estimation is displayed on a graphic user interface (hereinafter referred to as GUI). This makes it possible to easily check and compare the performance of position and orientation estimation performed using each image and a plurality of images, and to identify the image or camera that is the cause of the performance degradation or improvement. You can do it. In addition, it becomes possible to remove or change the arrangement of the camera identified as the cause of the performance degradation. It is possible to perform position and orientation estimation adapted to changing conditions such as the shape of the target object, how it is placed, and how it hits the ambient light.

  FIG. 1 shows a system configuration in this embodiment. The information processing apparatus 100 includes an image acquisition unit 101, a first measurement unit 102, a second measurement unit 103, a first performance information generation unit 104, a second performance information generation unit 105, a first output unit 106, A second output unit 107. The information processing apparatus 100 is connected to an external display device 120 via an interface.

  FIG. 2 is a diagram showing an apparatus configuration and arrangement for taking an image acquired by the image acquisition unit 101, and includes a projector 1 and a plurality of cameras 2 to 5. FIG. 2 shows a case where the projector is composed of one projector and four cameras. The plurality of cameras are connected to the information processing apparatus 100 via the image acquisition unit 101.

  The image acquisition unit 101 acquires a luminance image obtained by photographing the target object with a plurality of cameras 2 to 5. However, other methods may be used as long as images obtained by capturing the target object from a plurality of viewpoints can be acquired. For example, an image obtained by capturing an image of a target object at a plurality of imaging devices or a plurality of viewpoint positions once held in an external storage device connected to the image information processing device 100 may be input. Further, the acquired luminance image may be a color image or a monochrome image.

  The first measurement unit 102 estimates the position and orientation of the target object based on each image acquired by the image acquisition unit 101. A specific method will be described later.

  The second measurement unit 103 estimates the position and orientation of the target object based on the plurality of images acquired by the image acquisition unit 101. A specific method will be described later.

  The first performance information generation unit 104 generates performance information (first performance information) related to the position / orientation estimation process executed by the first measurement unit 102. The performance information is the accuracy of position / orientation estimation and the processing time required for position / orientation estimation. The first performance information generation unit 104 sends the generated first performance information to the first output unit 106.

  The second performance information generation unit 105 generates performance information (second performance information) related to position and orientation estimation performed by the second measurement unit 103. The performance information is the position / orientation estimation accuracy and the processing time required for position / orientation estimation, similar to that generated by the first performance information generating unit 104. The second performance information generation unit 105 sends the generated second performance information to the second output unit 107.

  The first output unit 106 outputs the performance information generated by the first performance information generation unit 104 and causes the display device 102 to display the performance information. In this respect, the first output unit 106 functions as a display control unit. As a result, the user can confirm (view) the first performance information via the display device.

  The second output unit 107 outputs the performance information generated by the second performance information generation unit 105 and causes the display device 120 to display the performance information. In this respect, the second output unit 107 functions as a display control unit. As a result, the user can check (view) the second performance information via the display device 120.

  The display device 120 is, for example, a liquid crystal display or a CRT display. The display device 120 has a display screen and displays performance information received from the first output unit 106 and the second output unit 107 on the display screen. In the present embodiment, an example will be described in which a GUI is displayed on the display screen of the display device 120, and first performance information and second performance information are displayed in the GUI area. Accordingly, there is an effect that the user can more easily visually recognize the first performance information and the second performance information.

  FIG. 3 is a flowchart showing a processing procedure of the present embodiment. Hereinafter, the processing of this embodiment will be described in detail with reference to the flowchart of FIG.

(Step S301)
In step S301, the image acquisition unit 101 acquires an image obtained by capturing a target object. The image acquired by the image acquisition unit 101 is a luminance image obtained by photographing the target object irradiated with the pattern projected by the projector 1. The image acquisition unit 101 sends the acquired image to the first measurement unit 102 and the second measurement unit 103.

(Step S302)
In step S302, the first measurement unit 102 estimates the position and orientation of the target object based on each image acquired by the image acquisition unit 101 in step S301. Specifically, the position and orientation of the target object are estimated using the method disclosed in Non-Patent Document 1 below.

David A. Simon, Martial Hebert, Take Kanade, “Real-time 3-D Pose Estimating Using a High-Speed Range Sensor”, Proc. 1994 IEEE International Conference on Robotics and Automation (ICRA'94), pp. 2235-2241, 1994.

  Specifically, the position and orientation of the target object using a model fitting technique that fits a 3D model of the target object represented by a surface or a set of 3D points to the distance value obtained by measuring the target object Is estimated. The distance value of the target object is calculated by using the principle of triangulation by associating the pattern projected by the projector 1 with the projection pattern detected from each image. That is, in this step, the first measurement unit 102 estimates the position and orientation of the target object for each image (for each pair of the cameras 2 to 5 and the projector 1). The first measurement unit 102 sends the estimated position / orientation data of the target object to the first performance information generation unit 104.

(Step S303)
In step S303, the first performance information generation unit 104 generates performance information related to the position and orientation estimation performed in step S302. The first performance information generated in the present embodiment is assumed to be position / orientation estimation accuracy and processing time required for position / orientation estimation. In the present embodiment, the performance information related to the position / orientation estimation is used as the position / orientation estimation accuracy and processing time. However, other information may be used as long as it is information related to the position / orientation estimation performance of the target object. For example, the estimated position and orientation may be used. Alternatively, it may be information on whether or not the calculation of position and orientation has been broken (success or failure of position and orientation estimation).

  The accuracy of position and orientation estimation is the degree of fitting between the distance value of the target object and the three-dimensional model of the target object. The fitting degree is, for example, the maximum value of the distance to the surface or the three-dimensional point held by the three-dimensional shape model associated with the distance value of the target object. In this case, the smaller the maximum distance value, the higher the accuracy of position and orientation estimation.

  The processing time is a time required for the position / orientation estimation process in the first measurement unit 102. Furthermore, the processing time may include the time for capturing the target object and the time for inputting the captured image to the image acquisition unit 101. The first performance information generation unit 104 sends the generated performance information to the first output unit 106.

(Step S304)
In step S304, the first output unit 106 outputs the first performance information related to the position / orientation estimation generated in step S303 and the image acquired by the image acquisition unit 101 to the display device 120 and causes the display device 120 to display the first performance information. . Then, the display device 120 receives the captured image and the first performance information, and displays the captured image and the first performance information in the display area 450. The first performance information in the present embodiment is the position and orientation estimation accuracy and processing time as described above. Further, in order to identify the object that is the target of position and orientation estimation, an image in which a rectangular texture is drawn and superimposed in the vicinity of the position of the object on the captured image is displayed. The texture may be drawn near the position of the object on the captured image, and the shape may be other. For example, a circular shape or a star shape may be used.

(Step S305)
In step S305, the second measurement unit 103 estimates the position and orientation of the target object based on the plurality of images acquired by the image acquisition unit 101 in step S301. As a method of selecting a plurality of images, all the images may be selected, or a combination of images may be determined in advance. A specific method is as follows. First, as in step S303, the distance value of the target object is calculated for each image. The distance value of the target object is calculated by using the principle of triangulation by associating the projection pattern of the projector 1 with the projection pattern identified from each image. Next, using all the calculated distance values, the position and orientation of the target object are estimated using the method disclosed in Non-Patent Document 1. In other words, using all the distance values obtained from cameras of multiple viewpoints (integrating), the 3D structure of the object is represented, and the model of the object is fitted to the 3D structure, thereby estimating the position and orientation I do. In the present embodiment, it has been described that the position and orientation of the target object are estimated using all the distance values calculated for each image (for each pair of the projector 1 and the cameras 2 to 5). However, the present invention is not limited to this, and other methods for estimating the position and orientation based on the distance value obtained by integrating the distance values calculated for each image may be used. For example, the position and orientation of the target object may be estimated based on a thinning process performed so that the density in the three-dimensional space is uniform for all distance values. Further, by avoiding the use of the distance value that greatly deviates from the average value of the distance values in the predetermined area for the alignment, more robust position and orientation estimation can be performed.

(Step S306)
In step S306, the second performance information generation unit 105 generates second performance information related to the position / orientation estimation of the target object performed based on the plurality of images in step S305. The second performance information in the present embodiment is the accuracy of position / orientation estimation and the processing time required for position / orientation estimation, as described in step S303. The second output unit outputs the second performance information generated by the second performance information generation unit to the display device 120.

(Step S307)
In step S307, the second output unit 107 outputs the performance information regarding the position / orientation estimation of the target object generated in step S306 and the image acquired by the image acquisition unit 101 to the display device 120. The display device 120 receives the captured image and the second performance information, and displays the captured image and the second performance information in the display area 460. The performance information is the accuracy and processing time of position and orientation estimation as described above. In addition, in order to present the object that is the target of position and orientation estimation, an image in which a rectangular texture is drawn and superimposed in the vicinity of the position of the object on the captured image is displayed. The shape of the texture is not limited to a rectangle, and may be, for example, a circle or a star shape.

  FIG. 4 shows an example represented as a GUI 400 in the display screen of the display device 120.

  The display device 120 includes images 410 to 413 acquired by the camera 2, the camera 3, the camera 4, and the camera 5, accuracy of position / orientation estimation performed based on each image, and processing time 420 to 423 required for position / orientation estimation. Is displayed. Further, in order to specify the object 401 that is the target of position and orientation estimation, an image in which elliptical textures 430 to 433 are drawn and superimposed is displayed near the position of the object on the captured image.

  The display device 120 displays the accuracy of the position / orientation estimation performed based on all the images acquired by the camera 2, the camera 3, the camera 4, and the camera 5 and the time 424 required for the position / orientation estimation. Further, in order to specify the object 401 that is the target of position and orientation estimation, an elliptical texture 434 is drawn and superimposed on the vicinity of the object on the image 414 obtained by pasting the images acquired by the cameras 2 to 5. Display an image. The image to be displayed may be any image acquired by the cameras 2 to 5 or may not be displayed.

  Note that the number of objects for position estimation is not limited to one, and a plurality of objects arranged in an aligned or stacked state may be used.

  As described above, by executing the series of processing from step S301 to step S307, it is possible to present to the user information related to the performance of position and orientation estimation performed based on each captured image and a plurality of images. become. Accordingly, it is possible to easily check and compare the performance of position and orientation estimation performed for each image and the performance of position and orientation estimation performed using a plurality of images.

<Variation of position and orientation estimation method>
In the present embodiment, the method for estimating the position and orientation of the target object using the distance value calculated based on the image has been described. However, the present invention is not limited to this, and other methods may be used as long as the position and orientation of the target object can be estimated based on the image.

  For example, the position and orientation of the target object may be estimated using features extracted from the image. For example, the position and orientation of the object are estimated by matching a learning image obtained by photographing the target object from a plurality of viewpoints and an image feature detected from the acquired image.

  Alternatively, the position / orientation estimation of the target object may be performed using both the distance value calculated based on the image and the image feature extracted from the image. For example, the accurate position and orientation of the target object are estimated by simultaneously fitting a three-dimensional model of the target object represented by a plurality of planes and line segments to the edge features extracted from the distance value and the image.

<Image variations>
In the present embodiment, the image acquired by the image acquisition unit 101 has been described as a luminance image obtained by capturing a target object on which a pattern is projected by a projector. However, when position and orientation estimation is performed using features extracted from the image, a luminance image obtained by capturing a target object on which no pattern is projected may be used. Alternatively, both a luminance image obtained by photographing a target object on which a pattern is projected and a luminance image obtained by photographing a target object on which no pattern is projected may be input.

<Variation of position and orientation estimation accuracy>
In this embodiment, the position / orientation estimation accuracy is set to the maximum value of the distance to the surface or the three-dimensional point held by the three-dimensional shape model associated with the distance value of the target object. However, the present invention is not limited to this, and any other index may be used as long as it indicates the accuracy of position and orientation estimation. For example, the average value or variance of the distance to the surface or the three-dimensional point held by the three-dimensional shape model associated with the distance value of the target object may be used. Or it is good also as an average value and dispersion | distribution of the distance on an image, projecting on an image based on the position and orientation which estimated the point on the model matched with a distance point. Alternatively, it may be the maximum value, average value, or variance of the distance between the feature extracted from the image and the feature of the learning data associated with it. Or it is good also as the maximum value or dispersion | distribution of the some position and orientation calculated by implementing the position and orientation estimation of a target object in multiple times.

<Modification 1>
In the first embodiment, the position / orientation estimation accuracy and processing time are displayed on the GUI as information related to the position / orientation estimation performance generated by the first performance information generation unit 104 and the second performance information generation unit 105. . In this modification, a computer graphic (hereinafter, CG) superimposed on the acquired image is displayed on the GUI. It is assumed that the CG to be superimposed is drawn based on a position and orientation obtained by estimating a three-dimensional model of a target object expressed by points, line segments, and surfaces. Accordingly, the user can visually determine the performance of position and orientation estimation by observing the degree of overlap between the target object on the image and the CG of the target object superimposed on the image.

  FIG. 5 is an example when a CG of the target object drawn and superimposed on the image is displayed on the GUI 400 of the display device 120. The display device 120 displays an image drawn by superimposing the CGs 434 to 437 of the target object on the images taken by the camera 1, the camera 2, the camera 3, and the camera 5.

  The display device 120 displays an image in which the CG 438 of the target object is drawn and superimposed on the image 414 obtained by pasting all the images taken by the cameras 2 to 5.

  By observing the degree of overlap between the CGs 434 to 437 and the target objects on the images 410 to 414, the user can visually determine the performance of position and orientation estimation. For example, the CG 436 matches the target object on the image 412, but the CG 437 is shifted from the target object on the image 413. From this, it can be determined that the accuracy of position and orientation estimation using the image 412 is high. On the other hand, the accuracy of position and orientation estimation using the image 413 can be determined to be lower than the accuracy of position and orientation estimation using the image 412.

  As described above, the user can visually determine the accuracy of the position / orientation estimation of the target object by drawing and superimposing and displaying the CG of the target object on the image based on the estimated position / orientation. It becomes possible.

<Modification 2>
In the first embodiment, the first measuring unit 102 and the second measuring unit 103 estimate the position and orientation of the target object. In the present modification, the first measurement unit and the second measurement unit perform recognition processing that specifies the presence or absence and type of the target object, not estimation of the position and orientation.

  A processing procedure in the present modification will be described with reference to FIG.

(Step S301)
In step S301, the image acquisition unit 101 acquires an image obtained by capturing a target object. The acquired image is a luminance image obtained by photographing the target object.

(Step S302)
In step S302, the first measurement unit 102 recognizes the target object based on each image acquired by the image acquisition unit 101 in step S301. The recognition process is performed by the method described in Non-Patent Document 2 below.

"SCHMID, C .; R. MOHR: 'Local grayvalue invariants for image retry' IEEE PAMI 19 May 1997, pages 530-534".
Specifically, the target object is recognized using a template matching method using features extracted from the image.

(Step S303)
In step S303, the first performance information generation unit 104 generates performance information related to the recognition process performed in step S302. In this modification, the performance information regarding recognition is recognition accuracy and processing time required for recognition. The recognition accuracy is the Hausdorff distance between the input image and the template image collated with the input image. The processing time required for recognition is the time required for recognition processing performed by the first measurement unit 102. Furthermore, the processing time may include the time for capturing the target object and the time for the image acquisition unit 101 to acquire the captured image.

(Step S304)
In step S <b> 304, the first output unit 106 displays the first performance information related to recognition generated in step 303 on the GUI set in the display device 120. As described above, the performance information related to recognition is the recognition accuracy and the processing time required for recognition. In addition, in order to present the object that is the target of position and orientation estimation, an image in which a rectangular texture is drawn and superimposed in the vicinity of the position of the object on the captured image is displayed. The texture may be drawn near the position of the object on the captured image, and the shape may be other. For example, a circular shape or a star shape may be used. However, other information may be used as long as the information indicates the performance related to recognition. For example, the presence / absence of the target object may be used as the performance information. Alternatively, when a plurality of objects are recognized at the same time, the number of recognized objects may be used as performance information.

(Step S305)
In step S305, the second measurement unit 109 recognizes the target object based on a plurality of images (two or more selected images) acquired by the image acquisition unit 101 in step S301. A specific method is as follows. First, the image feature of the target object is extracted for each image. Then, from all the extracted image features, a feature extracted by duplication in a plurality of images is selected. Then, the target object is recognized by the method disclosed in Non-Patent Document 2 using the selected image. However, other methods that can recognize a target object based on image features extracted from a plurality of images may be used. For example, the target object may be recognized using all image features extracted from a plurality of images.

(Step S306)
In step S <b> 306, the second performance information generation unit 105 generates second performance information related to recognition performed by the second measurement unit 103. The performance information related to recognition is as described in step S303.

(Step 307)
In step S <b> 307, the second output unit 107 displays the performance information related to recognition generated in step 306 and the captured image on the display area 460 of the display device 120. As described above, the performance information related to recognition is the recognition accuracy and the processing time required for recognition. In addition, in order to present an object to be recognized, an image in which a rectangular texture is drawn and superimposed in the vicinity of the position of the object on the captured image is displayed. The shape of the texture is not limited to a rectangle, and may be, for example, a circle or a star shape.

  As described above, by executing the series of processing from step 301 to step 307, performance information related to recognition performed for each image and performance information related to recognition performed using a plurality of images are displayed on the GUI. can do. Thereby, the user can confirm and compare recognition performance when using each image and a plurality of images.

<Modification 3>
In the first embodiment and the first and second modifications, the performance related to the recognition of the target object or the position and orientation estimation performed based on each image and a plurality of images is displayed. Further, for a camera that causes a performance degradation, a GUI that prompts the user to calibrate parameters indicating the characteristics of the camera and the placement information of the camera or to remove the camera or the image may be displayed. The camera that causes the performance degradation is a camera in which the position / orientation estimation performance estimated based on the image captured by the camera does not satisfy a predetermined performance or the camera having the lowest performance.

(Second Embodiment)
In the first embodiment, the method for displaying the performance related to the position / orientation estimation performed based on each image and the performance related to the position / orientation estimation performed based on a plurality of images has been described. In the present embodiment, a method for displaying performance related to position / orientation estimation and selecting an image used for position / orientation estimation from a plurality of input images will be described. Accordingly, the user can select an image to be used for position / orientation estimation while confirming the performance related to position / orientation estimation, and can perform position / orientation estimation with high accuracy. It is possible to perform position and orientation estimation adapted to changing conditions such as the shape of the target object, how it is placed, and how it hits the ambient light. Further, the processing time required for position and orientation estimation can be kept within the processing time determined by the tact time of the assembly work.

  FIG. 6 shows a system configuration in the present embodiment. The configuration of the information processing apparatus 600 according to the present embodiment includes an image selection unit 610 in addition to the configuration of the information processing apparatus 100 described with reference to FIG. 1 in the first embodiment. Since the image acquisition unit 601 to the second output unit 607 having other configurations have the same functions as the image acquisition unit 101 to the second output unit 107, only the image selection unit 610 will be described here. In addition, the display device 620 connected to the information processing device 600 has a function equivalent to that of the display device 120 described in the first embodiment. Further, the information processing apparatus 600 is connected to an external operation device 620. The operation device 620 is a mouse or a keyboard, and the user can instruct the image selection unit 610 via the operation device. Further, when the display screen of the display device 620 has a touch panel function, the touch panel function functions as an operation device.

  The image selection unit 610 selects an image used for processing in the second measurement unit from among a plurality of images acquired by the image acquisition unit 601. In the present embodiment, as in the first embodiment, the second measurement unit performs position and orientation estimation of the target object, and thus the image selection unit 610 selects an image to be used for position and orientation estimation processing.

  FIG. 7 is a flowchart showing the processing procedure of this embodiment. The processing in Steps 701 to 704 and Steps 706 and 707 is the same as the processing described in the first embodiment, and thus description thereof is omitted here.

(Step 708)
In step S708, the image selection unit 610 determines whether to perform image selection. This determination may be made by the user using the operation device 611 or based on a predetermined rule such that image selection is always performed a predetermined number of times. If the image selection unit 610 determines in step S708 to select an image, the process proceeds to step 709. If not selected, this process is terminated.

(Step 709)
In step S709, the image selection unit 610 selects an image to be used for position and orientation estimation based on an operation from the user via the operation device.

  FIG. 8 is an example of the GUI 800 according to the second embodiment displayed on the display screen of the display device 620.

  The display device 620 displays the position / orientation estimation accuracy performed based on the images 810 to 813 captured by the camera 2, the camera 3, the camera 4, and the camera 5 and the processing time 820 to 823 required for the position / orientation estimation.

  The image selection unit 610 selects the selection buttons 810 to 830 set in the GUI 800 via the operation device, and selects whether to use the position and orientation estimation for each image captured by the camera. In the example of FIG. 8, the images taken by the camera 2 and the camera 5 are not used, and the images taken by the camera 2 and the camera 3 are selected to be used. In the present embodiment, for each image input to the image acquisition unit 601, a setting is made as to whether to use for position and orientation estimation. However, other setting methods may be used as long as an image used for position and orientation estimation can be selected. For example, whether or not to use for position and orientation estimation may be collectively set for all images to be selected.

  The display device 620 uses the selection buttons 824 to 827 to display the position / orientation estimation accuracy performed based on the images taken by the camera 2 and the camera 3 selected by the user and the processing time 824 required for the position / orientation estimation. .

(Step 705)
In step S <b> 705, the second measurement unit 603 estimates the position and orientation of the target object based on the image selected in step 701.

<Image selection variation (selection method)>
In the present embodiment, for each image acquired by the image acquisition unit 601, whether to use for position and orientation estimation is set. In addition to this, a position and orientation estimation processing method may be selected for each image. For example, a processing method may be selected from a position / orientation estimation method using image features, a position / orientation estimation method using distance values, or a position / orientation estimation method using both image features and distance values. By doing so, it is possible to perform position and orientation estimation adapted to changing conditions such as the shape of the target object, how it is placed, and how it hits the ambient light. Further, the position / orientation estimation processing method may be set collectively for all the images to be selected.

<Modification 2-1>
In the first embodiment and Modifications 1-1 to 1-3, the first measurement unit performs position and orientation estimation based on each image. However, the present invention is not limited to this, and position and orientation estimation may be performed based on two or more images (one or more may be used). In this case, the first performance information generation unit generates performance information related to the position and orientation performed based on two or more images, and the display device 620 displays the performance information.

  The display device 620 displays the estimation accuracy and processing time 820 of position and orientation estimation performed based on images taken by the camera 2 and the camera 3. Further, the display device 620 displays the estimation accuracy and processing time 821 of the position / orientation estimation performed based on the images taken by the camera 3 and the camera 4. Further, the display device 620 displays the estimation accuracy and the processing time 822 of the position / orientation estimation performed based on the image captured by the camera 4. Further, the display device 620 displays the estimation accuracy and processing time 823 of the position / orientation estimation performed based on the images taken by the camera 4 and the camera 5. In addition, these captured images 810 to 813 are displayed. The images 810, 811 and 813 are images obtained by pasting captured images. For example, the image 810 is an image obtained by pasting together images taken by the cameras 2 and 3.

  The image selection unit 610 selects a camera to be used when the user selects the selection buttons 824 to 827.

  The display device 620 displays the accuracy and processing time 824 of position and orientation estimation performed based on images taken by the camera 3, the camera 4, and the camera 5. In addition, an image 814 in which images taken by the camera 3, the camera 4, and the camera 5 are combined is displayed.

  Note that the captured images 810, 811, 813, and 814 may display individual images constituting the combined images instead of the combined images. For example, the image 810 displays an image taken by the camera 2 or an image taken by the camera 3. Alternatively, the image display itself may not be performed.

(Third embodiment)
In the first embodiment, the method for displaying the performance related to the position / orientation estimation performed based on each image and the performance related to the position / orientation estimation performed based on a plurality of images has been described. In the second embodiment, the method for selecting an image used for position and orientation estimation from a plurality of input images has been described. In the third embodiment, a method will be described in which conditions related to position and orientation estimation are input, and the performance related to position and orientation estimation when the input conditions are satisfied is displayed. This makes it possible to estimate the position and orientation that satisfy the conditions desired by the user. Further, it is possible to perform position and orientation estimation adapted to changing conditions such as the shape of the target object, how it is placed, and how it hits the ambient light.

  FIG. 9 shows the configuration of the information processing apparatus 900 in this embodiment. The information processing apparatus 900 holds a condition acquisition unit 910 in addition to the configuration described with reference to FIG. 1 in the first embodiment. Other than the condition acquisition unit 910, the second measurement unit 903, the second performance information generation unit 905, and the second output unit are the same as those described in the first embodiment, and a description thereof is omitted here. .

  The condition acquisition unit 910 acquires conditions related to position and orientation estimation. In the present embodiment, the condition regarding the position / orientation estimation to be acquired is the lower limit value of the position / orientation estimation accuracy.

  The second measurement unit 903 generates a plurality of image combinations used for position and orientation estimation from the plurality of input images, and performs position and orientation estimation for each generated image combination. The position and orientation estimation method is as described in the first embodiment.

  The second performance information generation unit 905 generates performance information related to position / orientation estimation for the results of the plurality of position / orientation estimations performed by the second measurement unit 903. Furthermore, it is determined whether the generated performance information satisfies the condition acquired by the condition acquisition unit 910. Specifically, it is determined whether the accuracy of position / orientation estimation falls within the acquired estimation accuracy threshold. As a result of the determination, only the position / orientation measurement result with the accuracy determined to be within the acquired threshold value is sent to the second output unit 907.

  The second output unit 907 outputs only the performance related to the position / orientation estimation determined to satisfy the condition input in the performance information generation unit 905 among the performance information generated by the second performance information generation unit 905.

  FIG. 10 is a flowchart showing a processing procedure in the present embodiment. Steps 1001 to 1006 are the same as steps S301 to S306 described in the first embodiment, and thus description thereof is omitted here.

(Step 1007)
In step S <b> 1007, the condition acquisition unit 910 acquires a position / orientation estimation accuracy threshold value via the set GUI. In this embodiment, what the user inputs via the operation device is acquired. The condition acquisition unit 910 sends the acquired condition to the second output unit 909.

(Step 1008)
In step S1008, the second output unit 907 acquires the accuracy of the position / orientation measurement generated in step S1006 and determines whether the acquired accuracy is within the threshold acquired in step S1007. Then, performance information related to position / orientation estimation determined to be within the threshold is selected and output to the display device 920.

  FIG. 11 is an example showing the GUI 1100 set in the display device 920 and the condition acquisition unit 910. The display device 920 displays the performance information specified by the performance information display switching button 1128 from among a plurality of performance information related to position and orientation estimation determined to satisfy the condition. Further, information 1124 to 1127 indicating whether or not each image is used is displayed in order to specify the image used for the position and orientation estimation that is the target of the performance information displayed on the display device 920. In this example, performance information of position and orientation estimation using the cameras B and C is displayed in the display area 907.

  The condition acquisition unit 910 acquires an estimation accuracy condition input by the user via the operation device in the input box 1129 set in the GUI 1100.

  The display device 920 displays position / orientation estimation accuracy and processing times 1120 to 1123 required for position / orientation estimation performed based on the images 1110 to 1113 captured by the camera 2, the camera 3, the camera 4, and the camera 5, respectively.

  The display device 920 displays the position / orientation estimation accuracy and the processing time required for the position / orientation estimation based on the images taken by the cameras 3 and 4 in the display area 1107.

<Variations of input conditions>
In the present embodiment, the input condition is the lower limit value of the estimation accuracy. However, other conditions may be used as long as they are conditions related to position and orientation estimation. For example, an upper limit value of processing time required for position and orientation estimation may be specified. In this case, when the processing time required for position / orientation estimation falls below the upper limit value of the designated processing time, it is determined that the condition is satisfied. Alternatively, both the lower limit value of the estimation accuracy and the upper limit value of the processing time may be input as conditions.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

<Effect of each embodiment>
In the first embodiment, a method has been described in which position and orientation estimation is performed based on each captured image or a plurality of images, and information on the performance of each position and orientation estimation is displayed on the GUI. This makes it possible to easily check and compare the performance of position and orientation estimation performed using each image and a plurality of images, and to identify the image or camera that is the cause of the performance degradation or improvement. You can do it. In addition, it becomes possible to remove or change the arrangement of the camera identified as the cause of the performance degradation.

  In the second embodiment, a method of displaying the performance of position / orientation estimation based on each captured image or a plurality of images and selecting an image to be used for position / orientation estimation from a plurality of input images. explained. Accordingly, the user can select an image to be used for position / orientation estimation while confirming the performance, and thus can perform position / orientation estimation with high performance. Further, the processing time required for position and orientation estimation can be kept within the processing time determined by the tact time of the assembly work.

  In the third embodiment, a method for acquiring conditions related to position / orientation estimation and acquiring performance related to position / orientation estimation when the acquired conditions are satisfied will be described. This makes it possible to estimate the position and orientation that satisfy the conditions desired by the user.

<Definition>
The image acquisition unit of the present invention may input an image from a camera that has captured a target object connected to the information processing apparatus 100, or connected to the information processing apparatus 100 are images of the target object captured from a plurality of viewpoints. You may input what was once held in an external storage device.

  In the first and second measurement units of the present invention, recognition or position / orientation estimation may be performed based on a distance value calculated based on an image. Alternatively, recognition or position / orientation estimation may be performed based on image features extracted from the image. Alternatively, recognition or position / orientation estimation may be performed based on both the distance value calculated based on the image and the image feature extracted from the image.

  In the 1st and 2nd performance information generation part in this invention, the performance information regarding the position and orientation estimation implemented in the 1st and 2nd measurement part is produced | generated. The performance information related to the position / orientation estimation may be anything as long as it is information indicating the position / orientation performance. For example, the position / orientation estimation accuracy or processing time is used. Alternatively, it may be information on the estimated position and orientation, the presence or absence of the target object, and whether or not the calculation of position and orientation estimation or recognition has broken.

  The first and second output units in the present invention display the performance information generated by the first and second performance information generation units in the display area of the display device, respectively. The display method may display the accuracy of position / orientation estimation or the numerical value of the processing time on the GUI, or draw based on the estimated position / orientation of the 3D model indicating the shape of the target object and superimpose it on the input image. May be displayed.

DESCRIPTION OF SYMBOLS 101 Image acquisition part 102 1st measurement part 103 2nd measurement part 104 1st performance information generation part 105 2nd performance information generation part 106 1st output part 107 2nd output part

Claims (14)

Image acquisition means for acquiring a plurality of images obtained by imaging a target object;
First execution means for executing at least one of recognition processing of the target object and position / orientation estimation processing of the target object based on at least one of the acquired images;
Second execution means for executing at least one of the target object recognition process and the position / posture estimation process of the target object based on at least two or more images of the plurality obtained.
First performance information generation means for generating first performance information relating to processing executed by the first execution means;
Second performance information generation means for generating second performance information relating to processing executed by the second execution means;
Display control means for causing the display means to display the first performance information and the second performance information;
An information processing apparatus comprising: Furthermore, the image acquisition unit includes an image selection unit that selects at least two images to be used for processing by the second execution unit from a plurality of images acquired by the image acquisition unit,
The second position / orientation estimation unit executes at least one of the target object recognition process and the target object position / orientation estimation process based on the image selected by the image selection unit. The information processing apparatus according to claim 1. Furthermore, the image acquired by the image acquisition means has means for selecting a processing method in the second position and orientation estimation means,
The information processing apparatus according to claim 1, wherein the second measurement unit performs the process based on the selected processing method.   4. The method according to claim 1, wherein the first performance information includes at least one of accuracy of processing executed by the first execution unit, processing time, and success / failure of the processing. The information processing apparatus according to item 1.   5. The method according to claim 1, wherein the second performance information includes at least one of accuracy, processing time, and success / failure of processing executed by the second execution unit. The information processing apparatus according to item 1. Furthermore, it comprises a condition acquisition means for acquiring a condition relating to processing executed by at least one of the first execution means and the second execution means,
The information processing apparatus according to claim 1, wherein the display control unit causes the display unit to display performance information that satisfies the acquired condition.   The condition acquisition means includes at least one of processing accuracy and processing time executed by at least one of the first execution means and the second execution means. The information processing apparatus according to claim 6.   The information processing apparatus according to claim 1, wherein the image acquisition unit acquires images captured by a plurality of imaging apparatuses that are arranged at different positions.   The information processing apparatus according to claim 1, wherein the image acquisition unit acquires an image including a pattern projected by a projection apparatus.   The information processing apparatus according to claim 1, further comprising the display unit.   The display control means further causes the display means to display an image in which a three-dimensional model of the target object is drawn based on a result of processing by the first execution means or the second execution means. The information processing apparatus according to any one of claims 1 to 9 Furthermore, a projection device that projects a pattern, and a plurality of imaging devices that capture an image on which the pattern is projected,
The information processing apparatus according to claim 1, wherein the image acquisition unit acquires a plurality of images on which the pattern imaged by the imaging apparatus is projected. An image acquisition process for acquiring a plurality of images of the target object;
A first execution step of executing at least one of the target object recognition process and the target object position and orientation estimation process based on information about at least one of the acquired images; ,
A second execution step of executing at least one of the target object recognition process and the target object position and orientation estimation process based on information about at least two images of the plurality obtained ,
A first performance information generation step for generating first performance information related to the processing executed in the first execution step;
A second performance information generation step for generating second performance information relating to the processing executed in the second execution step;
A display control step of causing the display means to display the first performance information and the second performance information;
An information processing apparatus comprising:   A program for causing a computer device to function as each unit of the information processing device according to any one of claims 1 to 12, when the computer device is executed.

Images