JP6425405B2 - INFORMATION PROCESSING APPARATUS, METHOD, AND PROGRAM - Google Patents

Description

  The present invention relates to a method of recognizing and displaying position and orientation estimation of an object in an image based on an image obtained by capturing the object.

  At production sites, automation by robots is in progress to improve production efficiency. In picking and assembling a part by a robot, it is necessary to recognize a target part and to acquire the position and orientation. As a method of realizing this, a method using an image obtained by photographing a target part with a camera has been developed.

  Patent Document 1 discloses a method of improving the accuracy of recognition by using images captured by a plurality of cameras. In this method, a plurality of stereo image pairs are selected from images captured by a plurality of cameras, and recognition accuracy is improved by performing multiple recognitions of the target object using distance values calculated for each stereo image pair. I am improving.

Unexamined-Japanese-Patent No. 2000-94374

However, a plurality of images forming a stereo image pair may include an image that is rich in noise. Further, using a plurality of images may increase the processing time, making it impossible to finish the task within a predetermined time.
The present invention has been made in view of the above problems, and it is an object of the present invention to select an image suitable for deriving a position and orientation from a plurality of images in recognition or position and orientation estimation using a plurality of images.

  According to one aspect of the present invention, based on an image acquisition unit for acquiring a plurality of images obtained by imaging a target object with each of a plurality of imaging devices in an information processing apparatus, and based on each image of the plurality of acquired images First execution means for executing any one of the target object recognition process and the target object position and orientation estimation process, and the first execution for each of the plurality of imaging devices A first performance information generation unit configured to generate first performance information related to processing executed by the unit; and the first performance information displayed on the display unit in association with each of the plurality of imaging devices Display control means, selection means for selecting two or more imaging devices based on a user's instruction from the plurality of imaging devices in which the first performance information is displayed in association with each, and the selected two More imaging equipment And a second performance information generation unit configured to generate second performance information related to the processing to be executed by the second execution unit, based on the image captured in step And second display control means for displaying the second performance information on the display means.

  In recognition or position and orientation estimation using a plurality of images, an image suitable for deriving a position and orientation can be selected from the plurality of images.

It is a figure showing composition of a system in a 1st embodiment. It is a figure which shows the structural example of the system using the information processing apparatus of this invention. It is a figure explaining the process sequence of 1st Embodiment. It is a figure which shows the example of GUI in 1st Embodiment. It is a figure which shows the example of GUI in the modification 1. FIG. It is a figure which shows the structure of the system in 2nd Embodiment. It is a figure explaining the process sequence of 2nd Embodiment. It is a figure which shows the example of GUI in 2nd Embodiment. It is a figure which shows the structure of the system in 3rd Embodiment. It is a figure explaining the process sequence of 3rd Embodiment. It is a figure which shows the example of GUI in 3rd Embodiment. It is a figure which shows the hardware constitutions of the information processing apparatus in this invention.

  Prior to describing each embodiment according to the present invention, a hardware configuration on which the information processing apparatus shown in each embodiment is mounted will be described with reference to FIG.

  FIG. 12 is a hardware configuration diagram of the information device in the present embodiment. In the same figure, a CPU 1210 centrally controls each device connected via a bus 1200. The CPU 1210 reads out and executes processing steps and programs stored in a read only memory (ROM) 1220. An operating system (OS), processing programs according to this embodiment, device drivers and the like according to the present embodiment are stored in the ROM 1220, temporarily stored in a random access memory (RAM) 1230, and appropriately executed by the CPU 1210. Also, the input I / F 1240 is input as an input signal in a format that can be processed by the information processing apparatus 1 from an external device (display device, operation device, etc.). The output I / F 1250 outputs an output signal to an external device (display device) in a format that can be processed by the display device.

  These functional units are realized by the CPU 1210 developing a program stored in the ROM 1220 in the RAM 1230 and executing processing according to each flowchart to be described later. Further, for example, when hardware is configured as a substitute for software processing using the CPU 1210, an arithmetic unit or a circuit corresponding to the processing of each functional unit described here may be configured.

First Embodiment
In the first embodiment, a method for presenting the performance related to position and orientation estimation to the user when position and orientation estimation of the target object is performed based on images obtained by capturing the target object with a plurality of cameras will be described. More specifically, position and orientation estimation is performed based on each captured image and a plurality of images, and information regarding the performance of each position and orientation estimation is displayed on a graphic user interface (hereinafter referred to as a GUI). This makes it possible to easily confirm and compare the performance of position and orientation estimation performed using each image and a plurality of images, and the identification of the image or camera that is the cause of the performance degradation or improvement is You will be able to do it. In addition, it becomes possible to remove or change the camera specified as the cause of the performance degradation. It is possible to perform position and orientation estimation adapted to changing conditions such as the shape and placement of the target object, and how the ambient light strikes.

  FIG. 1 shows the configuration of a system in the present embodiment. The information processing apparatus 100 includes an image acquisition unit 101, a first measurement unit 102, a second measurement unit 103, a first performance information generation unit 104, a second performance information generation unit 105, and a first output unit 106. And a second output unit 107. The information processing apparatus 100 is connected to an external display device 120 via an interface.

  FIG. 2 is a diagram showing an apparatus configuration and arrangement for capturing an image acquired by the image acquisition unit 101, which includes a projector 1 and a plurality of cameras 2-5. FIG. 2 shows a case where one projector and four cameras are provided. The plurality of cameras are connected to the information processing apparatus 100 via the image acquisition unit 101.

  The image acquisition unit 101 acquires a luminance image obtained by photographing a target object with a plurality of cameras 2 to 5. However, any other method may be used as long as an image obtained by photographing the target object from a plurality of viewpoints can be acquired. For example, an image obtained by photographing a target object at a plurality of imaging devices or a plurality of viewpoints may be input once stored in an external storage device connected to the image information processing apparatus 100. Further, the luminance image to be acquired may be a color image or a monochrome image.

  The first measurement unit 102 estimates the position and orientation of the target object based on the images acquired by the image acquisition unit 101. The specific method will be described later.

  The second measurement unit 103 estimates the position and orientation of the target object based on the plurality of images acquired by the image acquisition unit 101. The specific method will be described later.

  The first performance information generation unit 104 generates performance information (first performance information) related to the position and orientation estimation process performed by the first measurement unit 102. The performance information is the accuracy of position and orientation estimation and the processing time required for position and orientation estimation. The first performance information generation unit 104 sends the generated first performance information to the first output unit 106.

  The second performance information generation unit 105 generates performance information (second performance information) related to position and orientation estimation performed by the second measurement unit 103. The performance information is, like the one generated by the first performance information generation unit 104, the accuracy of the position and orientation estimation and the processing time required for the position and orientation estimation. The second performance information generation unit 105 sends the generated second performance information to the second output unit 107.

  The first output unit 106 outputs the performance information generated by the first performance information generation unit 104 and causes the display device 102 to display the information. In this regard, the first output unit 106 functions as a display control unit. Thereby, the user can confirm (view) the first performance information via the display device.

  The second output unit 107 outputs the performance information generated by the second performance information generation unit 105 and causes the display device 120 to display the same. In this regard, the second output unit 107 functions as a display control unit. Thus, the user can confirm (view) the second performance information via the display device 120.

  The display device 120 is, for example, a liquid crystal display or a CRT display. The display device 120 has a display screen, and displays the performance information received from the first output unit 106 and the second output unit 107 on the display screen. In the present embodiment, a GUI is displayed on the display screen of the display device 120, and the first performance information and the second performance information are displayed in the area of the GUI. This has the effect that the user can more easily view the first performance information and the second performance information.

  FIG. 3 is a flowchart showing the processing procedure of this embodiment. The process of the present embodiment will be described in detail below with reference to the flowchart of FIG.

(Step S301)
In step S301, the image acquisition unit 101 acquires an image obtained by photographing a target object. The image acquired by the image acquisition unit 101 is a luminance image obtained by photographing the target object irradiated with the pattern projected by the projector 1. The image acquisition unit 101 sends the acquired image to the first measurement unit 102 and the second measurement unit 103.

(Step S302)
In step S302, the first measurement unit 102 estimates the position and orientation of the target object based on each of the images acquired by the image acquisition unit 101 in step S301. Specifically, the position and orientation of the target object are estimated using the method disclosed in Non-Patent Document 1 below.

David A. Simon, Martial Hebert, Takeo Kanade, "Real-time 3-D Pose Estimation Using a High-Speed Range Sensor", Proc. 1994 IEEE International Conference on Robotics and Automation (ICRA '94), pp. 2235-2241, 1994.

  Specifically, the position and orientation of the target object are obtained using a method of model fitting in which a three-dimensional model of the target object represented by a plane or a set of three-dimensional points is fitted to the distance value obtained by measuring the target object. Estimate The distance value of the target object is calculated by using the principle of triangulation in association with the pattern projected by the projector 1 and the projection pattern detected from each image. That is, in this step, the first measurement unit 102 estimates the position and orientation of the target object for each image (for each pair of the cameras 2 to 5 and the projector 1). The first measurement unit 102 sends the estimated position and orientation data of the target object to the first performance information generation unit 104.

(Step S303)
In step S303, the first performance information generation unit 104 generates performance information related to the position and orientation estimation performed in step S302. The first performance information generated in this embodiment is the accuracy of position and orientation estimation and the processing time required for position and orientation estimation. In the present embodiment, the performance information on position and orientation estimation is the position and orientation estimation accuracy and the processing time, but other information may be used as long as it is information on the performance of position and orientation estimation of the target object. For example, the estimated position and orientation may be used. Alternatively, it may be information on whether or not calculation of position and orientation estimation has failed (success or failure of position and orientation estimation).

  The accuracy of position and orientation estimation is the degree of fitting between the distance value of the target object and the three-dimensional model of the target object. The degree of fitting is, for example, the maximum value of the distance to the plane or three-dimensional point held by the three-dimensional shape model corresponding to the distance value of the target object. In this case, the smaller the maximum value of the distance is, the higher the accuracy of position and orientation estimation is.

  The processing time is the time required for the process of position and orientation estimation in the first measurement unit 102. Furthermore, the processing time may include the time for photographing the target object and the time for inputting the photographed image to the image acquisition unit 101. The first performance information generation unit 104 sends the generated performance information to the first output unit 106.

(Step S304)
In step S304, the first output unit 106 causes the display device 120 to display the first performance information on position and orientation estimation generated in step S303 and the image acquired by the image acquisition unit 101 on the display device 120. . Then, the display device 120 receives the captured image and the first performance information, and displays the captured image and the first performance information in the display area 450. The first performance information in the present embodiment is the accuracy and processing time of position and orientation estimation as described above. Further, in order to specify an object which has been subjected to position and orientation estimation, a rectangular texture is drawn near the position of the object on the photographed image, and the superimposed image is displayed. The texture may be drawn near the position of the object on the photographed image, and the shape may be other. For example, it may be circular or star-shaped.

(Step S305)
In step S305, the second measurement unit 103 estimates the position and orientation of the target object based on the plurality of images acquired by the image acquisition unit 101 in step S301. As a method of selecting a plurality of images, all images may be selected, or a combination of images may be determined in advance. The specific method is as follows. First, as in step S303, the distance value of the target object is calculated for each image. The distance value of the target object is calculated by using the principle of triangulation in association with the projection pattern of the projector 1 and the projection pattern identified from each image. Next, using the calculated distance values, the position and orientation of the target object are estimated using the method disclosed in Non-Patent Document 1. That is, by using (integrating) all distance values obtained from cameras of a plurality of viewpoints to express the three-dimensional structure of the object and fitting the model of the object to the three-dimensional structure, position and orientation estimation is performed. I do. In the present embodiment, it has been described that the position and orientation of the target object are estimated using all the distance values calculated for each image (for each pair of the projector 1 and the cameras 2 to 5). However, the present invention is not limited to this, and another method of estimating the position and orientation based on a distance value obtained by integrating the distance values calculated for each image may be used. For example, the position and orientation of the target object may be estimated on the basis of one in which thinning processing is performed such that the density in three-dimensional space is uniform for all distance values. In addition, more robust position and orientation estimation can be performed by not using a distance value that largely deviates from the average value of the distance values in the predetermined area for alignment.

(Step S306)
In step S306, the second performance information generation unit 105 generates, in step S305, second performance information on position and orientation estimation of the target object performed based on the plurality of images. The second performance information in this embodiment is the accuracy of the position and orientation estimation and the processing time required for the position and orientation estimation, as in the case described in step S303. The second output unit outputs the second performance information generated by the second performance information generation unit to the display device 120.

(Step S307)
In step S307, the second output unit 107 outputs, to the display device 120, the performance information on the position and orientation estimation of the target object generated in step S306 and the image acquired by the image acquisition unit 101. Then, the display device 120 receives the captured image and the second performance information, and displays the captured image and the second performance information in the display area 460. The performance information is the accuracy and processing time of position and orientation estimation as described above. Further, in order to present an object which has been subjected to position and orientation estimation, a rectangular texture is drawn near the position of the object on the photographed image, and the superimposed image is displayed. The shape of the texture is not limited to a rectangle, and may be, for example, a circle or a star.

  FIG. 4 shows an example represented as a GUI 400 in the display screen of the display device 120.

  The display device 120 includes images 410 to 413 acquired by the camera 2, the camera 3, the camera 4 and the camera 5, and processing time 420 to 423 required for position and orientation estimation accuracy and position and orientation estimation performed based on the respective images. Display Further, in order to specify the object 401 which is the target of position and orientation estimation, an elliptical shaped texture 430 to 433 is drawn near the position of the object on the photographed image, and the superimposed image is displayed.

  The display device 120 displays the accuracy of the position and orientation estimation performed based on all the images acquired by the camera 2, the camera 3, the camera 4 and the camera 5 and the time 424 required for the position and orientation estimation. In addition, in order to specify the object 401 which is the target of position and orientation estimation, an elliptical texture 434 is drawn and superimposed in the vicinity of the object 414 on the image 414 obtained by combining the images acquired by the cameras 2 to 5. Display an image. The image to be displayed may be any of the images acquired by the cameras 2 to 5 or may not be displayed.

  The object to be subjected to position estimation may not be one, but may be a plurality of objects arranged in an aligned or stacked state.

  As described above, by executing the series of processes in steps S301 to S307, it is possible to present the user with information on the performance of position and orientation estimation performed based on each captured image and a plurality of images. become. As a result, the performance of position and orientation estimation performed for each image and the performance of position and orientation estimation performed using a plurality of images can be easily confirmed and compared.

<Variation of position and orientation estimation method>
In the present embodiment, the method of estimating the position and orientation of the target object using the distance value calculated based on the image has been described. However, the present invention is not limited to this, and another method may be used as long as the position and orientation of the target object can be estimated based on the image.

  For example, position and orientation of a target object may be estimated using features extracted from an image. For example, the position and orientation of an object is estimated by matching a learning image obtained by photographing a target object at a plurality of viewpoints and an image feature detected from the acquired image.

  Alternatively, the position and orientation of the target object may be estimated using the distance value calculated based on the image and the image feature extracted from the image in combination. For example, by simultaneously fitting a three-dimensional model of a target object represented by a plurality of planes and line segments to edge features extracted from distance values and images, the accurate position and orientation of the target object are estimated.

<Variation of image>
In the present embodiment, the image acquired by the image acquisition unit 101 has been described as a luminance image obtained by photographing a target object on which a pattern is projected by the projector. However, in the case of performing position and orientation estimation using features extracted from an image, a luminance image obtained by capturing an object on which a pattern is not projected may be used. Alternatively, both a luminance image obtained by photographing a target object on which a pattern is projected and a luminance image obtained by photographing a target object on which a pattern is not projected may be input.

<Variation of position and attitude estimation accuracy>
In the present embodiment, the position and orientation estimation accuracy is set to the maximum value of the distance to a plane or a three-dimensional point held by the three-dimensional shape model corresponding to the distance value of the target object. However, the present invention is not limited to this, and any index indicating the accuracy of position and orientation estimation may be used. For example, the average value or the variance of the distances to the plane or three-dimensional point held by the three-dimensional shape model corresponding to the distance value of the target object may be used. Alternatively, the point on the model corresponding to the distance point may be projected on the image based on the estimated position and orientation, and the average value or the variance of the distance on the image may be used. Alternatively, the maximum value, the average value, or the variance of the distance between the feature extracted from the image and the feature of the corresponding learning data may be used. Alternatively, the maximum value or the variance of the plurality of positions and orientations calculated by performing the position and orientation estimation of the target object a plurality of times may be used.

<Modification 1>
In the first embodiment, as information on position / posture estimation performance generated by the first performance information generation unit 104 and the second performance information generation unit 105, the accuracy of position / posture estimation and processing time are displayed on the GUI . In the present modification, an image obtained by superimposing a computer graphic (hereinafter, CG) on the acquired image is displayed on the GUI. CG to be superimposed is assumed to be drawn based on the estimated position and orientation of a three-dimensional model of a target object represented by points, line segments and planes. Thus, the user can visually judge the performance of position and orientation estimation by observing the degree of overlap between the target object on the image and the CG of the target object superimposed on the image.

  FIG. 5 is an example when the CG of the target object is drawn and superimposed on the image and displayed on the GUI 400 of the display device 120. The display device 120 displays an image drawn by superimposing CG 434 to 437 of the target object on the images captured by the camera 1, the camera 2, the camera 3 and the camera 5.

  The display device 120 displays the CG 438 of the target object on the image 414 obtained by combining all the images captured by the cameras 2 to 5 and superimposing the image.

  By observing the degree of overlap between the CG 434 to 437 and the target object on the images 410 to 414, the user can visually judge the performance of position and orientation estimation. For example, the CG 436 matches the target object on the image 412, but the CG 437 deviates from the target object on the image 413. From this, it can be determined that the accuracy of position and orientation estimation using the image 412 is high. On the other hand, it can be determined that the accuracy of the position and orientation estimation using the image 413 is lower than the accuracy of the position and orientation estimation using the image 412.

  As described above, the user can visually determine the accuracy of the position and orientation estimation of the target object by drawing and superimposing the CG of the target object on the image based on the estimated position and orientation. It becomes possible.

<Modification 2>
In the first embodiment, in the first measurement unit 102 and the second measurement unit 103, the position and orientation of the target object are estimated. In the present modification, the first measurement unit and the second measurement unit do not estimate the position and orientation, but perform recognition processing for specifying the presence or absence and the type of the target object.

  The processing procedure in this modification will be described using FIG.

(Step S301)
In step S301, the image acquisition unit 101 acquires an image obtained by photographing a target object. The image to be acquired is a luminance image obtained by photographing the target object.

(Step S302)
In step S302, the first measurement unit 102 recognizes a target object based on each image acquired by the image acquisition unit 101 in step S301. The recognition process is performed by the method described in Non-Patent Document 2 below.

"SCHMID, C. R. MOHR: 'Local gray value invariants for image retrieval' IEEE PAMI 19 May 1997, pages 530-534".
Specifically, the target object is recognized using a template matching method using features extracted from the image.

(Step S303)
In step S303, the first performance information generation unit 104 generates performance information related to the recognition process performed in step S302. In the present modification, performance information related to recognition is recognition accuracy and processing time required for recognition. The recognition accuracy is the Hausdorff distance between the input image and the template image collated with the input image. The processing time required for recognition is the time required for the recognition processing performed by the first measurement unit 102. Furthermore, the processing time may include the time for photographing the target object and the time for the image acquisition unit 101 to acquire the photographed image.

(Step S304)
In step S304, the first output unit 106 causes the GUI set in the display device 120 to display the first performance information related to the recognition generated in step S303. The performance information related to recognition is, as described above, the recognition accuracy and the processing time required for recognition. Further, in order to present an object which has been subjected to position and orientation estimation, a rectangular texture is drawn near the position of the object on the photographed image, and the superimposed image is displayed. The texture may be drawn near the position of the object on the photographed image, and the shape may be other. For example, it may be circular or star-shaped. However, other information may be used as long as the information indicates the performance related to recognition. For example, the presence or absence of a target object may be used as the performance information. Alternatively, when recognizing a plurality of objects at the same time, the number of recognized objects may be used as the performance information.

(Step S305)
In step S305, the second measurement unit 109 recognizes the target object based on the plurality of images (two or more selected images) acquired by the image acquisition unit 101 in step S301. The specific method is as follows. First, image features of the target object are extracted for each image. Then, among all of the extracted image features, features extracted redundantly in a plurality of images are selected. Then, using the selected image, the target object is recognized by the method disclosed in Non-Patent Document 2. However, it may be another method capable of recognizing a target object based on image features extracted from a plurality of images. For example, recognition of a target object may be performed using all image features extracted from a plurality of images.

(Step S306)
In step S306, the second performance information generation unit 105 generates second performance information related to the recognition performed by the second measurement unit 103. The performance information related to recognition is as described in step S303.

(Step 307)
In step S307, the second output unit 107 causes the display area 460 of the display device 120 to display the performance information on the recognition generated in step 306 and the captured image. The performance information related to recognition is, as described above, the recognition accuracy and the processing time required for recognition. Also, in order to present an object that has been the object of recognition, a rectangular texture is drawn near the position of the object on the captured image, and the superimposed image is displayed. The shape of the texture is not limited to a rectangle, and may be, for example, a circle or a star.

  As described above, by executing the series of processes of step 301 to step 307, performance information on recognition performed for each image and performance information on recognition performed using a plurality of images are displayed on the GUI can do. Thereby, the user can confirm and compare the recognition performance when using each image and a plurality of images.

<Modification 3>
In the first embodiment and the first and second modifications, the performance regarding recognition or position / orientation estimation of a target object performed based on each image and a plurality of images is displayed. Furthermore, with regard to a camera that causes performance degradation, calibration of parameters indicating the camera characteristics and arrangement information of the camera, and a GUI prompting removal of the camera or an image may be displayed. The camera that causes the performance degradation is a camera in which the performance of the position and orientation estimation estimated based on the image captured by the camera does not meet the predetermined performance, or the camera with the lowest performance.

Second Embodiment
In the first embodiment, the method for displaying the performance related to position and orientation estimation performed based on each image and the performance related to position and orientation estimation performed based on a plurality of images has been described. In this embodiment, a method of displaying the performance regarding position and orientation estimation and selecting an image to be used for position and orientation estimation from among a plurality of input images will be described. As a result, the user can select an image to be used for position and orientation estimation while confirming the performance related to position and orientation estimation, and highly accurate position and orientation estimation can be performed. It is possible to perform position and orientation estimation adapted to changing conditions such as the shape and placement of the target object, and how the ambient light strikes. Further, the processing time required for position and orientation estimation can be kept within the processing time determined by the tact time of the assembling operation.

  FIG. 6 shows the configuration of a system in the present embodiment. The configuration of the information processing apparatus 600 in the present embodiment includes an image selection unit 610 in addition to the configuration of the information processing apparatus 100 described with reference to FIG. 1 in the first embodiment. The image acquisition unit 601 to the second output unit 607, which have other configurations, have the same functions as the image acquisition unit 101 to the second output unit 107, so only the image selection unit 610 will be described here. The display device 620 connected to the information processing device 600 has the same function as the display device 120 described in the first embodiment. Further, the information processing apparatus 600 is connected to an external operating device 620. The operation device 620 is a mouse or a keyboard, and the user can instruct the image selection unit 610 via the operation device. When the display screen of the display device 620 has a touch panel function, the touch panel function functions as an operating device.

  The image selection unit 610 selects an image to be used for processing in the second measurement unit from among the plurality of images acquired by the image acquisition unit 601. In the present embodiment, as in the first embodiment, since the second measurement unit estimates the position and orientation of the target object, the image selection unit 610 selects an image used for the process of position and orientation estimation.

  FIG. 7 is a flowchart showing the processing procedure of the present embodiment. The processes in steps 701 to 704 and steps 706 and 707 are the same as the processes described in the first embodiment, and thus the description thereof is omitted here.

(Step 708)
In step S708, the image selection unit 610 determines whether to perform image selection. This determination may be performed by the user using the operation device 611, or may be performed a predetermined number of times based on a predetermined rule such as image selection. If it is determined in step S 708 that the image selection unit 610 selects an image, the process proceeds to step 709. If not selected, this processing ends.

(Step 709)
In step S709, the image selection unit 610 selects an image to be used for position and orientation estimation based on an operation from the user via the operation device.

  FIG. 8 is an example of the GUI 800 in the second embodiment displayed on the display screen of the display device 620.

  The display device 620 displays the position and orientation estimation accuracy and the processing time 820 to 823 required for the position and orientation estimation performed based on the images 810 to 813 captured by the cameras 2, 3, 4 and 5 respectively.

  The image selection unit 610 selects the selection buttons 810 to 830 set in the GUI 800 via the operation device, and selects whether to use for position and orientation estimation for each image captured by the camera. In the example of FIG. 8, the photographed images of the cameras 2 and 5 are not used, and the photographed images of the cameras 2 and 3 are selected for use. In the present embodiment, whether or not to use for position and orientation estimation is set for each image input to the image acquisition unit 601. However, another setting method may be used as long as an image used for position and orientation estimation can be selected. For example, whether or not to use for position and orientation estimation may be set collectively for all images to be selected.

  The display device 620 displays the accuracy of the position and orientation estimation and the processing time 824 required for the position and orientation estimation performed based on the images captured by the camera 2 and the camera 3 selected by the user using the selection buttons 824 to 827. .

(Step 705)
In step S705, the second measuring unit 603 estimates the position and orientation of the target object based on the image selected in step S701.

<Variation of image selection (selection method)>
In the present embodiment, for each image acquired by the image acquisition unit 601, it is set whether to use for position and orientation estimation. In addition to this, the processing method of position and orientation estimation may be selected for each image. For example, the processing method may be selected from a position and orientation estimation method using an image feature, a position and orientation estimation method using a distance value, or a position and orientation estimation method using both an image feature and a distance value. In this way, it is possible to perform position and orientation estimation adapted to changing conditions such as the shape and placement of the target object and the way in which ambient light strikes. Further, the processing method of position and orientation estimation may be set collectively for all images to be selected.

<Modification 2-1>
In the first embodiment and the modified examples 1-1 to 1-3, in the first measurement unit, position and orientation estimation is performed based on each image. However, the present invention is not limited to this, and position and orientation estimation may be performed based on two or more images (one or more may be sufficient). In this case, the first performance information generation unit generates performance information related to the position and orientation performed based on the two or more images, and the display device 620 displays the performance information.

  The display device 620 displays the estimation accuracy of the position and orientation estimation and the processing time 820 performed based on the images captured by the camera 2 and the camera 3. The display device 620 also displays the estimation accuracy of the position and orientation estimation and the processing time 821 performed based on the images captured by the camera 3 and the camera 4. The display device 620 also displays the estimation accuracy of the position and orientation estimation and the processing time 822 performed based on the image captured by the camera 4. The display device 620 also displays the estimation accuracy of the position and orientation estimation and the processing time 823 performed based on the images captured by the camera 4 and the camera 5. Also, these photographed images 810 to 813 are displayed. Regarding the images 810, 811 and 813, taken images are combined. For example, the image 810 is an image obtained by combining the images captured by the cameras 2 and 3.

  The image selection unit 610 selects a camera to be used by the user selecting the selection buttons 824 to 827.

  The display device 620 displays the accuracy of position and orientation estimation and the processing time 824 performed based on the images captured by the camera 3, the camera 4 and the camera 5. Further, an image 814 obtained by combining the images captured by the camera 3, the camera 4 and the camera 5 is displayed.

  The photographed images 810, 811, 813, and 814 may display individual images constituting the pasted image instead of the pasted image. For example, in the image 810, an image captured by the camera 2 or an image captured by the camera 3 is displayed. Alternatively, the image display itself may not be performed.

Third Embodiment
In the first embodiment, the method for displaying the performance related to position and orientation estimation performed based on each image and the performance related to position and orientation estimation performed based on a plurality of images has been described. In the second embodiment, the method of selecting an image used for position and orientation estimation from among a plurality of input images has been described. In the third embodiment, a method for inputting a condition regarding position and orientation estimation and displaying performance regarding position and orientation estimation when the input condition is satisfied will be described. This enables position and orientation estimation to meet the conditions desired by the user. In addition, it is possible to perform position and orientation estimation adapted to changing conditions such as the shape and placement of the target object and the way in which ambient light strikes.

  FIG. 9 shows the configuration of the information processing apparatus 900 in the present embodiment. The information processing apparatus 900 holds a condition acquisition unit 910 in addition to the configuration described using FIG. 1 in the first embodiment. The components other than the condition acquisition unit 910, the second measurement unit 903, the second performance information generation unit 905, and the second output unit are the same as those described in the first embodiment, and thus the description thereof is omitted here. .

  The condition acquisition unit 910 acquires conditions regarding position and orientation estimation. In the present embodiment, the condition regarding position and orientation estimation to be acquired is the lower limit value of the position and orientation estimation accuracy.

  The second measurement unit 903 generates a plurality of combinations of images to be used for position and orientation estimation from a plurality of input images, and performs position and orientation estimation for each combination of the generated images. The position and orientation estimation method is as described in the first embodiment.

  The second performance information generation unit 905 generates performance information on position and orientation estimation with respect to the results of the plurality of position and orientation estimations performed by the second measurement unit 903. Furthermore, it is determined whether the generated performance information satisfies the condition acquired by the condition acquisition unit 910. Specifically, it is determined whether the accuracy of the position and orientation estimation falls within the obtained estimation accuracy threshold. As a result of the determination, only the position and orientation measurement result of the accuracy determined to be within the acquired threshold value is sent to the second output unit 907.

  The second output unit 907 outputs, of the performance information generated by the second performance information generation unit 905, only the performance related to position and orientation estimation that is determined to satisfy the condition input by the performance information generation unit 905.

  FIG. 10 is a flowchart showing the processing procedure in the present embodiment. Steps 1001 to 1006 are the same as steps S301 to S306 described in the first embodiment, and thus the description thereof is omitted here.

(Step 1007)
In step S1007, the condition acquisition unit 910 acquires the threshold of the position and orientation estimation accuracy through the set GUI. In the present embodiment, the input by the user via the operation device is acquired. The condition acquisition unit 910 sends the acquired condition to the second output unit 909.

(Step 1008)
In step S1008, the second output unit 907 acquires the accuracy of the position and orientation measurement generated in step S1006, and determines whether the acquired accuracy falls within the threshold acquired in step S1007. Then, performance information on position and orientation estimation that is determined to be within the threshold is selected and output to the display device 920.

  FIG. 11 is an example showing the GUI 1100 set in the display device 920 and the condition acquisition unit 910. The display device 920 displays the performance information specified by the performance information display switching button 1128 among the performance information on the plurality of position and orientation estimations determined to satisfy the condition. Further, information 1124 to 1127 indicating whether or not each image has been used is displayed in order to specify the image used for position and orientation estimation that is the target of the performance information displayed on the display device 920. In this example, performance information of position and orientation estimation using the cameras B and C is displayed in the display area 907.

  The condition acquisition unit 910 acquires the condition of the estimation accuracy input by the user via the operation device in the input box 1129 set in the GUI 1100.

  The display device 920 displays the position and orientation estimation accuracy and the processing time 1120 to 1123 required for the position and orientation estimation performed based on each of the camera 2, the camera 3, the camera 4 and the images 1110 to 1113 captured by the camera 5.

  The display device 920 displays the position and orientation estimation accuracy and the processing time required for the position and orientation estimation on the display area 1107 based on the images captured by the cameras 3 and 4.

<Variation of input condition>
In the present embodiment, the input condition is the lower limit value of the estimation accuracy. However, other conditions may be used as long as the conditions are related to position and orientation estimation. For example, the upper limit value of the processing time required for position and orientation estimation may be designated. In this case, when the processing time required for position and orientation estimation falls below the specified upper limit of the processing time, it is determined that the condition is satisfied. Alternatively, both the lower limit value of the estimation accuracy and the upper limit value of the processing time may be input as a condition.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, software (program) for realizing the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU or the like) of the system or apparatus reads the program. It is a process to execute.

<Effect of each embodiment>
In the first embodiment, the method of performing position and orientation estimation based on each captured image or a plurality of images and displaying information on the performance of each position and orientation estimation on the GUI has been described. This makes it possible to easily confirm and compare the performance of position and orientation estimation performed using each image and a plurality of images, and the identification of the image or camera that is the cause of the performance degradation or improvement is You will be able to do it. In addition, it becomes possible to remove or change the camera specified as the cause of the performance degradation.

  In the second embodiment, a method of displaying the performance of position and orientation estimation performed based on each captured image or a plurality of images, and selecting an image to be used for position and orientation estimation from among a plurality of input images explained. As a result, the user can select an image to be used for position and orientation estimation while confirming the performance, so that high-performance position and orientation estimation can be performed. Further, the processing time required for position and orientation estimation can be kept within the processing time determined by the tact time of the assembling operation.

  In the third embodiment, a method for acquiring conditions for position and orientation estimation and displaying performance for position and orientation estimation when the acquired conditions are satisfied will be described. This enables position and orientation estimation to meet the conditions desired by the user.

<Definition>
The image acquisition means of the present invention may input an image from a camera that has photographed the target object connected to the information processing apparatus 100, or has connected to the information processing apparatus 100 an image that has photographed the target object from a plurality of viewpoints. You may input what was once hold | maintained to the external storage device.

  The first and second measurement units in the present invention may perform recognition or position and orientation estimation based on the distance value calculated based on the image. Alternatively, recognition or position and orientation estimation may be performed based on image features extracted from an image. Alternatively, recognition or position and orientation estimation may be performed based on both of the distance value calculated based on the image and the image feature extracted from the image.

  The first and second performance information generation units in the present invention generate performance information on position and orientation estimation performed by the first and second measurement units. The performance information related to position and orientation estimation may be any information that indicates the performance of the position and orientation. For example, the accuracy or processing time of position and orientation estimation is used. Alternatively, it may be information on the estimated position and orientation, the presence or absence of the target object, and whether or not the calculation of position and orientation estimation or recognition is broken.

  The first and second output units in the present invention respectively display the performance information generated by the first and second performance information generation units in the display area of the display device. The display method may display the accuracy of the position and orientation estimation or the numerical value of the processing time on the GUI, or draw the three-dimensional model indicating the shape of the target object based on the estimated position and orientation and superimpose it on the input image. May be displayed.

101 image acquisition unit 102 first measurement unit 103 second measurement unit 104 first performance information generation unit 105 second performance information generation unit 106 first output unit 107 second output unit

Claims (12)

An image acquisition unit that acquires a plurality of images obtained by imaging a target object with each of a plurality of imaging devices ;
A first execution unit that executes any one of the target object recognition process and the target object position and orientation estimation process based on each of the plurality of acquired images;
First performance information generation means for generating first performance information related to processing executed by the first execution means for each of the plurality of imaging devices ;
First display control means for causing the display means to display the first performance information in association with each of the plurality of imaging devices ;
A selection unit configured to select two or more imaging devices based on a user's instruction from the plurality of imaging devices in which the first performance information is displayed in association with each other;
A second execution unit that executes the one process based on the images captured by the two or more selected imaging devices;
Second performance information generation means for generating second performance information related to processing executed by the second execution means;
An information processing apparatus comprising: second display control means for causing the display means to display the second performance information . The second display control means, together with the second performance information, displays, for the display means, whether or not the first performance information and the selection means are selected for each of the plurality of imaging devices. The information processing apparatus according to claim 1, wherein the information is displayed . The image processing apparatus further comprises means for selecting a processing method in the second execution means for the image acquired by the image acquisition means.
The information processing apparatus according to claim 1, wherein the second execution unit executes the one process based on the selected processing method. The method according to any one of claims 1 to 3, wherein the first performance information includes at least one of accuracy of processing executed by the first execution unit, processing time, and success or failure of processing. The information processing apparatus according to claim 1. The said 2nd performance information contains at least one among the precision of the process which said 2nd execution means performs, processing time, and success or failure of a process, The any one of the Claims 1 to 4 characterized by the above-mentioned. The information processing apparatus according to claim 1. The information processing apparatus according to any one of claims 1 to 5 , wherein the image acquisition unit acquires images captured by a plurality of imaging devices arranged at different positions. The information processing apparatus according to any one of claims 1 to 6 , wherein the image acquisition unit acquires an image including a pattern projected by a projection device. The information processing apparatus according to any one of claims 1 to 7 , further comprising the display unit. The first display control means or the second display control means further includes an image in which a three-dimensional model of the target object is drawn based on a result of processing by the first execution means or the second execution means. The information processing apparatus according to any one of claims 1 to 8 , characterized in that: And a plurality of imaging devices for capturing an image on which the pattern is projected, and a projection device for projecting a pattern, and
The information processing apparatus according to any one of claims 1 to 9 , wherein the image acquisition unit acquires a plurality of images on which the pattern captured by the imaging device is projected. An image acquisition step of acquiring a plurality of images obtained by imaging a target object with each of a plurality of imaging devices ;
A first execution step of executing any one of the target object recognition processing and the target object position and orientation estimation processing based on each of the plurality of acquired images;
A first performance information generation step of generating first performance information related to processing executed in the first execution step for each of the plurality of imaging devices ;
A first display control step of causing the display unit to display the first performance information in association with each of the plurality of imaging devices ;
A selection step of selecting two or more imaging devices based on a user's instruction from the plurality of imaging devices in which the first performance information is displayed in association with each other;
A second execution step of executing the one process based on the images captured by the two or more selected imaging devices;
A second performance information generation step of generating second performance information related to processing executed by the second execution means;
The information processing method characterized by comprising a <br/> a second display control step of displaying the second performance information to said display means. The program for functioning this computer apparatus as each means of the information processing apparatus of any one of Claim 1 thru | or 10 by being performed by computer.

Images