JPH09178426A - Position posture recognition device of object to be recognized and position posture recognition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To measure position and posture of objects like a cylindrical column or a circular cone and a circular truncated trapezoid from camera image by calculating a projected image of a three-dimensional model of an object to be recognized, correcting three-dimensional position and posture, and comparing and checking it with the object to be recognized. SOLUTION: A TV camera 1 photographes a region in the vicinity of an object like circular column which is an object to be recognized and outputs as an image signal. An image input part 2 digitalizes the image signal. An image feature sampling part 3 samples features of an image in the area of the image requested by a recognition processing control part 8 for digital image from the input part 2 or a model projected image generation part 6. A linear line detection part 4 detects a linear line which becomes a profile line of the object and a candidate of a meridian from the results of sampling of the sampling part 3 to transmit the position information to the generation part 6. The generation part 6 generates a model projected image in accordance with camera parameters of a camera-parameter storage part 5. A fitting verification part 7 compares the model projected image which is generated with the feature sampled image sampled by the sampling part 3 to find a model projected image by which they agree.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a position / orientation recognizing device and a position / orientation recognizing method for recognizing a work target of a cylindrical cone-shaped object and its position / orientation by an image in, for example, intelligent robot control.

[0002]

2. Description of the Related Art With the development of robot technology, robots have come to be used not only in industrial fields but also in various fields. In recent years, research has been actively conducted in which robots perform work such as parts replacement and maintenance of equipment in dangerous environments such as high places.

A robot (work robot for dangerous environment) for performing work such as parts replacement and maintenance of equipment in such a dangerous environment is a system in which operation of the robot is remotely controlled by human hands. Is taken,
Since there is a problem in terms of handling, such as requiring labor and requiring skill in operation, it is possible to mount a TV camera on the work robot, autonomously recognize a target object, and perform work. There is a strong demand for the realization of autonomous work robots.

In order to realize an autonomous work robot, a position and orientation recognition device for recognizing a three-dimensional position and orientation of a work target from an input image by image processing is indispensable. If the shape of the target object is known, the position and orientation of the object can be measured even by a monocular camera, and studies on monocular object recognition are being actively conducted.

In particular, there are many artificial objects having a columnar shape or a conical shape, and the recognition of their positions and postures is an important factor for realizing an autonomous work robot.

As one application example of the autonomous work robot, for example, consider a robot that performs distribution line work on a utility pole. The shape of an object to be recognized by the robot for work is limited. is there. Among them, the pillars and insulators that are considered to be the most important recognition targets have a cylindrical or conical shape.

As a method for recognizing a three-dimensional position and orientation of an object based on image information of a known object such as an insulator on a telephone pole, which is photographed by a monocular camera, a target object is selected from the image. A method of extracting a two-dimensional image feature point indicating the feature of the shape and recognizing the three-dimensional position and orientation of the object from the correspondence with the feature point of the target object is often used.

However, when recognizing an object having a cylindrical or conical shape by extracting the characteristic points, the target object is smooth, so that there are no protrusions or corner points, and the characteristic points are extracted from the image. There is a problem that it is difficult to extract.

There is also a method of extracting characteristic straight lines or characteristic curves instead of extracting characteristic points and performing recognition based on the correspondence with the recognition target object. For example, when the target object has a cylindrical shape, an elliptical shape or parallel lines are extracted and the object is recognized from the shape.

In this case, in order to perform accurate measurement, it is desirable that the contour line of the elliptical shape is extracted in the entire circumference.
However, if objects are in contact with the top and bottom of the cylinder,
It is not possible to extract the contour line of the entire circumference, but it is often extracted as a part of an elliptic arc, which makes it difficult to accurately measure the position and orientation.

Further, the contour lines on both sides of the cylinder or the cone are not contour lines appearing as ridgelines peculiar to the object, but are contour lines called shielding contour lines. Especially, when the difference between the target object region and the background region is not clear, Even if the image processing for extracting the elliptical contour line is used as it is, the extraction is difficult.

[0012] In addition, in these methods using a combination of feature points and feature lines, if a large number of candidate feature points and feature lines are extracted from the screen, the number of combinations to be tried tends to be enormous, resulting in a process. There is a problem that it takes time. In order to ensure the extraction of contour lines, a large number of candidates are prepared by image processing, so it is necessary to try combinations even for candidates that are unrelated to the recognition target, and the processing is redundant accordingly. is there.

As a method for recognizing an object by limiting an object to a cylindrical shape, Japanese Patent Publication No. 6-56287 (Goto: Position detecting device for cylindrical object) and Japanese Patent Publication No. 6-4817.
The technique disclosed in Japanese Patent Publication No. 3 (Inamoto: Cylindrical measurement method) is already known.

In these methods, the processing is facilitated by artificially giving the target object a characteristic suitable for image processing. For example, the technique for detecting the position of a cylindrical object disclosed in Japanese Patent Publication No. 6-56287 is a method of generating a shadow by projecting active illumination and limiting conditions of a projected image. The cylinder measuring technique disclosed in Japanese Patent Publication No. 48173 is a method of coating a marker on an object.

However, as a technique for recognizing the three-dimensional position and orientation of an object required by an autonomous work robot, it is possible to recognize the image taken in a natural environment. Desirable,
In addition, it is desirable not to artificially modify the subject.

As another method for recognizing the three-dimensional position and orientation of an object, there is a method of capturing a projected image as a figure and applying a pattern matching method in a figure. In this case, by comparing the shape of the projected image of the target object with the pattern of the appearance of the target object stored in advance,
Recognize the position and orientation of an object.

However, when an object having a cylindrical or conical shape can freely change its position and orientation in a three-dimensional manner, the appearance of the object has the following five types of change elements. That is, there are a total of five types of change factors: “size”, “inclination on the screen”, “inclination in the depth direction”, “position in the center axis direction”, and “position in the direction perpendicular to the center axis”. And, considering all the changing elements, the pattern of appearance is infinite, so when using the graphical pattern matching method,
It was necessary to limit the prepared pattern by fixing the size or fixing the rotation in the depth direction.

[0018]

In order to realize an autonomous work robot, the three-dimensional position and orientation of an object are recognized from an image captured by the robot itself, and necessary control is performed based on the information. However, the technique for recognizing the three-dimensional position and orientation of the object is to be able to recognize an image taken in a natural environment. Is desirable.

As a method applicable to such a condition, there is a method of capturing a projected image as a figure and applying a pattern matching method in a graphic form. In this case, the shape of the projected image of the target object is stored in advance. Since the position and orientation of the object are recognized by comparing the appearance pattern of the target object that is present, the pattern of the appearance of the target object that must be prepared becomes infinite, which is not realistic. Therefore, the conditions will be limited, but it will not be practically applicable.

That is, even when considering applying the pattern matching method to a cylindrical or conical object, if the cylindrical or conical object can freely change its position and orientation three-dimensionally, To see the subject,
There are a total of five types of change factors: “size”, “screen tilt”, “depth tilt”, “center axis position”, and “center axis vertical position”. The pattern of appearance is infinite considering the changing elements of
When the graphic pattern matching method is used, the prepared pattern is limited by fixing the size or fixing the rotation in the depth direction. However, if such a restriction is added, the image captured by the actual robot is not bound to such a frame, and therefore the pattern matching is less likely to occur, and the image cannot be practically used.

In the case of a robot whose purpose is specified such as attachment and detachment of specific parts as the work robot, the shape of the object to be handled may be fixed, and particularly when targeting an artificial product, a cylinder or If a cone-shaped object can measure its position and orientation from camera images, realizing an autonomous work robot is not a dream.

Therefore, the object of the present invention is to
A position / orientation recognizing apparatus and a position / orientation recognizing method capable of measuring the position and orientation of a cylinder, a cone, or a truncated cone-shaped object from a camera image.

[0023]

In order to achieve the above object, the present invention provides an input means for inputting an image of an object to be recognized having a specific shape, and an object to be recognized having the specific shape.
A projection image for holding a three-dimensional model, calculating a projection image of this three-dimensional model, correcting the three-dimensional position and orientation of the three-dimensional model based on the information on the degree of fitting, and comparing and collating with the recognition target object. Model projection image generating means, image feature extracting means for extracting image features of the recognition target object from the image obtained from the input means, and from the projection image obtained by the model projection image generating means, and this image A fit determination means for determining the degree of fit by comparing and comparing the image feature of the recognition target object obtained by the feature extraction means with the projection image obtained by the model projection image generation means, and obtaining information indicating the degree of fit. And the model projection image generation means obtains the position / orientation information of the best three-dimensional model projection image of the degree of fit determined by the fit determination means. Characterized by comprising an output means for.

Further, input means for inputting an image of an object to be recognized having a specific shape and a three-dimensional model of the object to be recognized having the specific shape are held, and a projection image of this three-dimensional model is calculated and is applicable. 3 based on degree information
A model projection image generation means for correcting the three-dimensional position and orientation of the three-dimensional model to obtain a projection image for comparison and collation with the recognition target object; and an image obtained from the input means, and to the model projection image generation means. Image feature extraction means for extracting the image feature of the recognition target object from the obtained projection image, means for extracting straight line information from the image feature of the recognition target object obtained by the image feature extraction means, and the image feature extraction Among the image characteristics of the recognition target object obtained by the means, the information of the straight line portion detected by the straight line detection means and the projection image obtained by the model projection image generation means are collated and compared to determine the degree of fitting. , Fit determination means for obtaining information indicating the fit degree, and position / orientation information of the best three-dimensional model projected image of the fit degree determined by the fit determination means , Characterized by comprising an output means for outputting the obtained from the model projection image generation means.

[0025]

BEST MODE FOR CARRYING OUT THE INVENTION In the present invention, the image feature of an image obtained from an image pickup means for picking up a recognition target is compared with the prepared projection image of a three-dimensional model of the recognition target to verify the degree of fitting. Then, the projection image obtained by sequentially correcting the position and orientation of the model according to the verification result and the image feature are collated again to verify the degree of fit, and the projection image that best fits is obtained and obtained. The three-dimensional position and orientation of the recognition target object is recognized by obtaining information on the position and orientation of the recognition target from the projection image that best fits.

Specifically, the image feature extracting means extracts the image feature of the recognition target object from the image of the recognition target object having the specific shape. Further, the model projection image generation means,
The projection image of this three-dimensional model is calculated using the held three-dimensional model of the recognition target object of the specific shape, and the three-dimensional position of the three-dimensional model is calculated based on the degree of fitting determined by the fitting determining means. A projection image for correcting the posture and comparing and collating with the recognition target object is obtained. Then, the fit determination means compares the image feature of the recognition target object obtained by the image feature extraction means with the projection image obtained by the model projection image generation means to determine the degree of fit.

Then, the output means obtains and outputs from the model projection image generation means the position / orientation information of the best three-dimensional model projection image of the fitting degree among the judgment results of the fitting degree judged by the fitting judgment means.

Further, when the straight line portion extracting means is provided, the straight line portion extracting means extracts the straight line portion from the image feature of the recognition target object obtained by the image feature extracting means, and the fitting determining means determines the straight line portion. The image feature and the projection image obtained by the model projection image generating means are collated and compared to determine the degree of fitting.

Then, the output means obtains and outputs from the model projection image generation means the position / orientation information of the best three-dimensional model projection image of the fitting degree among the judgment results of the fitting degree judged by the fitting judgment means.

In this way, the image information of the image of the recognition target is obtained, the image feature of the image obtained from this image information is extracted, and the extracted image feature is projected onto the prepared three-dimensional model of the recognition target. Compare the image, verify the degree of fit, sequentially correct the position and orientation of the model,
The three-dimensional position and orientation of the recognition target object is recognized by obtaining the projection image that best fits and obtaining the position and orientation from the obtained projection image that best fits.

Therefore, it becomes possible to measure the position and orientation of an object to be recognized in the shape of a cylinder, a cone, or a truncated cone from an image, which opens the way to the realization of an autonomous work robot.

Specific examples of the present invention will be described below with reference to the drawings.

(Specific Example 1) FIG. 1 shows a schematic configuration of a columnar shape recognizing device to which the position / orientation recognizing device for a cylindrical object of the present invention is applied, and a flow of information.

In this example, an example of a columnar object recognizing device will be described, and examples of a conical object and a truncated cone object will be described in Example 4.

First, the configuration of the present apparatus shown in FIG. 1 will be described.

A position / orientation recognizing device for recognizing a cylindrical shape is
The V camera 1, the image input unit 2, the image feature extraction unit 3, the straight line detection unit 4, the camera parameter storage unit 5, the model projection image generation unit 6, the fitting verification unit 7, the recognition processing control unit 8, and the result output unit 9. .

Of these, the TV camera 1 is installed so that a cylindrical object to be recognized is input, photographs a region near the target object, and outputs the photographed image as an image signal. .

The image input section 2 is for digitizing and inputting the image signal transmitted from the TV camera 1, and the image feature extraction section 3 is for the digital image obtained from the image input section 2 and Alternatively, the image feature in the image area required by the recognition processing control unit 8 is extracted from the digital image obtained from the model projection image generation unit 6, and the extraction result is held in the image holding memory, or the model projection is performed. It has a function of extracting image features from the model projection image generated by the image generation unit 6.

Further, the straight line detecting section 4 detects straight lines which are the contour line candidates in the cylindrical axis direction of the cylindrical conical object and the meridian candidates of the conical object from the extraction result of the image feature extracting section 3,
The camera parameter storage unit 5 has a function of transmitting the position information to the model projection image generation unit 6, and the camera parameter storage unit 5 inputs the TV camera 1 and the image which are necessary when the model projection image generation unit 6 generates the model projection image. The camera parameters of the camera system including the unit 2 are stored. That is, the model projection image generation unit 6 needs to generate a model projection image on the screen (screen of the subject image) with the same positional relationship as the subject image captured by the TV camera 1, so that the calculation process necessary for the projection is performed. , Which is a camera parameter, and moves the coordinates (X, Y, Z) indicating the positional relationship in the three-dimensional space of the camera system to the projected position (x, y) on the screen. This is a parameter indicating the projection relationship at the time.

The model projection image generation unit 6 holds a three-dimensional model of the recognition target object, and approximates the posture, size, etc. of the projection image of the recognition target object according to the camera parameters of the camera parameter storage unit 5. Produces a projected image,
The detection result of the straight line detection unit 4, the extraction result of the image feature extraction unit 3, and the input image are corrected so that the verification unit 7 can compare the alignment and the tilt on the screen to obtain the corrected model projection image. It has functions such as generating.

The fitting verification unit 7 compares the model projection image generated by the model projection image generation unit 6 with the result (feature extraction image) extracted by the image feature extraction unit 3 from the input image, and matches the model projection image. It has the function of finding. The recognition processing control unit 8 plays a central role in the control of the present invention and performs a series of recognition processing procedure control. The recognition processing procedure will be described in detail later.

The result output unit 9 is a fitting verification unit 7,
The model projection image generated by the model projection image generation unit 6 and the feature extraction image extracted by the image feature extraction unit 3 are the recognition results based on the position (three-dimensional position) and posture information of the model projection image determined to match. It has a function of outputting as information.

Here, the detailed structure of each section will be described.

The camera parameters of the camera input system consisting of the TV camera 1 and the image input section 2 are measured in advance and stored in the camera parameter storage section 5. The camera parameters are
It is a parameter indicating the projection relationship from the coordinates (X, Y, Z) in the three-dimensional space to the projection position (x, y) on the screen.
For example, the following formulas (1) and (2) have 12 constants {p
_i } i = 1, 2, ..., 12 expresses the projection relationship.

[0045]

[Equation 1]

These parameters {p _i } i = 1, 2,
..., 12 can be obtained by taking 6 or more points with known three-dimensional coordinates, measuring their projected positions, and considering equations (1) and (2) as linear equations and obtaining them by the least squares method. Good.

Alternatively, the camera position vector C (the camera position here indicates the position of the focal point of the camera), the focal length F, the optical axis direction vector A of the camera, and the image center position (x ₀ , y ₀ ). Even if is given, the parameters {p _i } i = 1, 2, ..., 12 can be obtained from the following expressions.

First, from the optical axis direction vector A of the camera,
An X direction vector U ₀ and a Y direction vector V ₀ of the camera light receiving element surface are determined. These may be determined so that the vector U ₀ , the vector A, and the vector V ₀ are in the right-handed orthogonal coordinate system. At this time, the parameter is calculated by the following formula. It should be noted that in the following equations, vector elements are represented by adding an arrow vector code on the element code.

[0049]

[Equation 2]

Here, (.,.) Indicates the inner product of the vectors.

It is also easy to convert these 12 parameters into physical camera parameters. The camera position vector C, the focal length F, the optical axis direction vector A of the camera, and the image center position (x ₀ , y ₀ ) can be calculated by the following equations.

[0052]

(Equation 3) In this specific example, the camera parameters will be described below based on the expressions (1) and (2).

Next, the image feature extraction unit 3 will be described.

The image feature extraction unit 3 converts the input image into an edge intensity image by performing edge extraction processing on the digital image obtained through the image input unit 2, and outputs this as a feature extraction image. Alternatively, the edge is detected from the model projection image generated by the model projection image generation unit 6, and this is output as a feature extraction image. Here, when converting the input image into the edge strength image, it is not necessary to perform processing on the entire image, and it is sufficient to perform processing only on the partial image required by the straight line detection unit 4 and the fitting verification unit 7.

The edge extraction method is applicable to the verification unit 7
An appropriate extraction method can be applied according to the position and direction of the outer contour line to be fitted.

For example, when trying to fit the contour line of the cylindrical object in the direction of the cylinder axis, generally, it is assumed that the input TV camera 1 is attached to the ground at an angle horizontal to the image. It is desired to extract vertical edges. In this case, an edge extraction method having directivity may be applied to the edge in the assumed direction. Further, for example, when trying to fit the upper and lower outer contour lines of the cylindrical object, that is, the contour lines extracted as a part of the elliptic arc, it is preferable to apply an edge extraction method having no directivity in the direction.

The edge extraction method having directivity at the edge of the direction includes, for example, the following methods.

Now, in the image 30 of the cylindrical object,
As shown in FIG. 7, regions (A) and (B) are set to the left and right of the assumed boundary line 32 for the pixel 31 of interest. 3
3 is the area (A), and 34 is the area (B). Border line 32
If there is an inclination on the screen, the coordinates may be exchanged to set the area (A) on the left side and the area (B) on the right side of the boundary line 32. The number of pixels in the left area (A) labeled with reference numeral 33 and the number of pixels in the right area (B) labeled with reference numeral 34 do not have to be the same.

Here, as shown in FIG. 7, the number of pixels in the entire region is N _T , the average luminance in the entire region is m _T , the variance in the entire region is σ _T , and the region indicated by reference numeral 33 (A). The number of pixels in the area is N _A , the average luminance in the area (A) is m _A , and reference numeral 34
Area indicated by the number of pixels in (B) the N _B, also the region (B)
Let m _{B be} the average luminance in the inside and P _i be the luminance at the position i, and the separation degree η between the regions (A) and (B) is defined by the following equation.

[0060]

(Equation 4)

This separation η is calculated for each pixel,
An edge extraction method having directivity in a direction can be performed by setting a position having a large value as an edge position.

For example, in the case of a occluded contour line of an object, such as a contour line of a cylindrical object in the cylinder axis direction, the area (A),
By setting (B) to a shape that is long in the vertical direction and calculating the degree of separation η, it becomes possible to extract edges that were difficult to extract in the past.

On the other hand, when extracting the upper and lower contour lines of a cylindrical object, that is, a part of an elliptic arc, an edge extraction method having no directivity in the direction is used. For example, FIG.
Edge strength is calculated by preparing two types of masks for the X direction and the Y direction as shown in (b), performing a mask operation on the pixel of interest using these masks, and calculating the sum of squares thereof. Can be asked.

In this case, the edge direction can be obtained by using the intensity ratio of the mask calculation value for the X-direction mask and the mask calculation value for the Y-direction mask. How to use the information on the edge direction will be described in a specific example 2 described later.

In the following description, the outer contour line of the cylindrical object in the cylinder axis direction will be expressed as a meridian, and will be expressed as a meridian edge in the image, a meridian candidate in the image, or the like.

The straight line detector 4 detects straight lines which are candidates for the meridian of the cylindrical object to be recognized from the edge intensity image output which is the feature extraction image extracted by the image feature extractor 3.

Detailed procedures will be described in detail in the description of the processing flow.

The model projection image generator 6 will be described.

In this example, the model projection image generation unit 6
Holds a three-dimensional model of the recognition target object, for example, as a wireframe model of a polygonal prism as indicated by reference numeral M in FIG. The example shown in FIG. 9 shows that when the recognition target object has a cylindrical shape, the model of the recognition target object is approximated by the model M in which the cylinder is a regular cuboid. The position information of the model M is held as three-dimensional coordinates of each vertex.

At this time, the model projection image is calculated by the following procedure.

First, each vertex {P _i } i of the polygonal column of the model
= 1, 2, ..., N, the projection positions {p _i } i = 1, 2 ,. . . , N
Ask for. Next, the projection image of each side is obtained by linearly approximating between the projection positions of each vertex in the image.

In this specific example, the outer contour line of the object to be recognized and the outer contour line of the projected image of the model are fitted and collated, so that
In this case, the outer contour line of the model projection image is obtained by selecting the outer contour line on the image from the generated model projection image.

To obtain the outer contour line of the cylindrical object exactly according to the projections of the equations (1) and (2) requires a very large amount of calculation, but the model is approximated by a polygonal cylinder in this way. By doing so, it becomes possible to generate a cylindrical approximate contour projection image with a small amount of calculation.

Further, the model projection image generating section 6 holds the model as information on three-dimensional coordinates. Therefore, it is easy to spatially move and rotate the model, and it is possible to easily generate a projected image corresponding to each of the five types of changing elements (see FIG. 10) of how the model looks.

For example, in order to give the model a rotation corresponding to the inclination θ on the screen, a three-dimensional vector J passing through the current center position of the object to be recognized is obtained from the viewpoint position obtained by the equation (9), and its axis is rotated. The rotation of θ should be given to.

If the number of corners of the model is approached infinitely, the distance between the vertices on the screen becomes less than 1 pixel. In this case, the fitting on the screen is equivalent to fitting the projection image of a strict cylindrical model.

As the approximate angle of the model increases, the accuracy of the approximation increases, but on the other hand, the amount of calculation for processing increases. Conversely, if the approximate angle of the model decreases, the accuracy of the approximation increases. It will decrease, but the calculation amount will decrease. Therefore, in consideration of these conditions, in this specific example, 20
It is approximated by a polygon. However, it should be noted that it is not limited to this.

As described above, the model projection image generator 6
Since a model having a three-dimensional position and orientation is held and its projected image is generated according to the camera parameters of the input system, the contour projected image of the model projected image generation unit 6 and the extraction of the image feature extraction unit 3 are performed. If the edge positions coincide with each other, the actual position and orientation of the recognition target should coincide with the position and orientation of the model of the model projection image generation unit 6, and three-dimensional measurement is possible.

In this example, the method of generating the model projection image of the model projection image generation unit 6 by the method of approximating the cylinder model with the polygonal cylinder model has been described. ), By solving the function form on the cylindrical image by the conversion formula of the formula (2), the contour projection image (model projection image) can be obtained.

The fitting verification unit 7 verifies whether the model projection image generated by the model projection image generation unit 6 matches the feature extraction image obtained by the image feature extraction unit 3. This verification is performed by the above feature extraction. The fitting scale of the model projection image to the image is calculated, and the projection position of the model projection image and the degree of coincidence between the two are obtained.

For that purpose, first, the fitting scale is calculated by integrating the image feature amount of each pixel position on the outer contour line in the model projection image generated by the model projection image generation unit 6. Then, the model projection image with the best fitting scale is selected as the recognition result. There are several possible methods for setting this fitting scale, so we will discuss it later.

By performing such a series of operations under the control and management of the recognition processing control unit 8, the operation of the recognition processing control unit 8 will be described together with the flow of processing.

The result output unit 9 is a format in which the recognition result of the fitting verification unit 7 is set by the user (for example, displaying the fitting result on an image, or numerical data indicating the position and orientation).
Output as In other words, in the case of obtaining numerical data as an output, if the model projection image with the best fitting scale is known, the model projection image generation unit 6 already has the model position, orientation, etc. as data. By outputting the recognition result, the result output unit 9 can output the recognition result as data and can be used as recognition information of the position and orientation of the recognition target of the autonomous work robot. Becomes

The operation of the present system having the above configuration will be described.

As an operating environment of the autonomous work robot equipped with the present system, distribution line construction on a utility pole will be described as an example. In this operating environment, the autonomous work robot needs to recognize and position the work object. For example, a large number of cylindrical objects such as insulators exist on the electric pole, and the position of the object is to be specified.

The autonomous work robot is mounted on the aerial work platform of the work vehicle. First, the operator installs the work vehicle near the power pole and raises the work platform above the power pole. Then, considering the range of motion of the robot and the influence of obstacles,
Place the robot in the proper position. Since the surrounding environment varies from place to place, it is not possible to install the robot at the exact same position for each work, and the placement on the utility pole also differs from place to place. However, since the work to be performed is predetermined, it is possible to decide in advance what the work target object needs to be recognized, and it is possible to set the position and posture of the robot so that the object fits on the screen. it can.

In such a case, it is possible to assume that the object to be recognized by the recognition system is within the screen, and also the spatial range in which the object can exist. can do. This is because, as described above, the operator installs the robot so that the positional relationship between the robot and the target is within a certain range in relation to the movable range of the robot.

FIG. 2 is a flow chart of this specific example.
It shows the flow of the entire processing. Further, FIGS. 3 to 6 are conceptual diagrams of processing progress when the present system is applied to an insulator position specifying device on a telephone pole according to the flowchart of FIG. It should be noted that in FIGS. 3 to 6, only the contour lines related to the insulator are clearly drawn for the sake of illustration.
It should be noted in advance that in the actual input image, the contour line is not so clearly extracted, and a large number of lines other than the contour line of the insulator are extracted. In addition, some insulators have a colored band (band formed of paint) around the waist. In this case, the colored band around the waist of a cylindrical insulator is shown as an arc-shaped contour line in the image. The image state becomes recognizable.

The flow of the following processing control is controlled by the recognition processing control unit 8 and each component carries out its operation under the control of the recognition processing control unit 8 to obtain a desired result. The details will be described in order.

The recognition process is started from image capturing.
That is, the image input process is performed in step S101, which is a process in which the image input unit 2 digitizes and captures the video signal captured and output by the TV camera 1. As a result, if the image captured by the TV camera 1 is an image as shown in FIG. 3, this image is digitized by the image input unit 2 and input to the image feature extraction unit 3.

When the image is captured, step S10 is performed next.
At 2, the initial position estimation process is performed. This initial position estimation processing function is a function of the recognition processing control unit 8, and the recognition processing control unit 8 performs three-dimensional (3D) between the camera (TV camera 1) and the recognition target for the image captured. (Estimation) As a result of providing information on the positional relationship to the model projection image generation unit 6, the model projection image generation unit 6 can determine the initial position and orientation of the recognition target object.

There are various setting methods for the initial position and posture. For example, if the approximate positional relationship between the camera and the recognition target object is known, the three-dimensional position and orientation of the object may be estimated from that. Alternatively, the setting may be made by a human, for example, a method of pointing from the image with a mouse.
Generally, it is considered that a plurality of objects having the same shape as the recognition target exist in the image. For example, in the case of the above-mentioned autonomous work robot for distribution line construction, it is considered that many insulators are photographed.

In such a case, a situation in which the captured image is displayed on the display, and a human selects a real recognition target from the displayed image with a pointing device such as a mouse and gives an instruction to the system is conceivable. In many cases, the position on the screen can be obtained by some method. Therefore, by giving information to the recognition processing control unit 8 by these artificial instruction operations, the recognition processing control unit 8 generates information on the estimated initial position of the object with respect to the camera and gives it to the model projection image generation unit 6. You can

However, there may be a situation in which such prior information cannot be used at all. In such a case, the recognition target may be set to be, for example, the center of the image.

When the initial position is estimated and set from the position in the image, it is necessary to convert the information of the two-dimensional position into the information of the three-dimensional position and the posture and to estimate the initial position and the posture. This will be explained next.

In order to convert the two-dimensional position information in the image into the three-dimensional position, the data of the insufficient depth direction position is set in advance, and the three-dimensional position coordinate is calculated by the following procedure based on the camera parameters. To calculate.

Now, as shown in FIG. 11, a plane Q perpendicular to the camera optical axis vector A is located at a position away from the camera focus position vector F by d in the optical axis direction of the camera optical axis vector A.
Suppose The element of the camera focus position vector F is F =
(F _x , F _y , F _Z ), and letting the elements of the camera optical axis vector A be A = (A _x , A _y , _AZ ), the position vector H of a straight line point on the optical axis on the plane Q = (H _x , H _y ,
H _Z ) is represented by the following formula.

(H _x , H _y , H _Z ) = (F _x + A _x d, F _y + A _y d, F _Z + _AZ d) (16) Further, the point H is perpendicular to the optical axis vector A. The equation of the plane Q is given by the following equation.

(X−H _x ) A _x + (Y−H _y ) A _y + (Z−H _Z ) A _Z = 0 (17) On the other hand, the center position of the recognition target object on the screen is C (x _c , y
It is assumed that the distance from the camera to the object to be recognized is d given by _c ). At this time, since the center position (X, Y, Z) of the recognition target object in space satisfies the projection equations of the equations (1) and (2) and exists on the plane Q, the equation (16 ) And equation (17) must be satisfied.
Therefore, by combining these equations, the following simultaneous equations are derived.

[0100]

(Equation 5)

By solving the simultaneous equations of equation (18), the distance from the camera to the object is d, and the projection position on the screen is (x _c , y _c ).
Y, Z) is required.

By the above procedure, the initial estimated position and orientation of the recognition target object can be set.

When the initial position estimating process for setting the initial estimated position and orientation of the object to be recognized is completed, the next step S10 is performed.
3, the axial edge extraction processing is started. In step S103, a projection image of the initial estimated position of the recognition target model is generated,
The process of extracting the axial edge of the projected image is performed.

That is, since the initial estimated position and orientation of the recognition target object are set, a model projection image corresponding to the current state of the recognition target object can be created.
The model projection image generation unit 6 uses the camera parameters stored in the camera parameter storage unit 5 and the approximate model of the recognition target object prepared in advance to calculate the projection image of the recognition target object at the initial position. Of the object to be recognized (modeled from the object to be recognized), and then the created projection image is given to the image feature extraction unit 3, whereby the image feature extraction unit 3 determines the edge of the projection image in the axial direction. Extract processing.

At this time, the area from which the edge is extracted is the area a1 surrounded by the dotted line shown in FIG. 4, which may be a small area near the meridian projection position of the estimated model projection image. Since the “direction” of the meridian of the estimated model projection image can also be estimated at the same time, an edge extraction method having directivity in that direction is applied. The method of extracting edges having directivity in the direction has been described above.

The size of the small area to be subjected to edge extraction may be set in advance by the number of vertical and horizontal pixels, or may be set according to the size of the projected image of the recognition target model. For example, the size of the region in the vertical direction may be set to twice the estimated meridian length, and the length of the region in the horizontal direction may be set to ½ of the horizontal width of the projected image. The processing is performed for each of the small areas set around the center positions of the right and left meridians of the projected image.

When the edge extraction process is completed in this way, the process moves to step S104 which is the meridian candidate extraction process step. In step S104, the straight line detection unit 4 extracts meridian candidates from the edge image extracted by the image feature extraction unit 3 in the process of step S103.

The extraction of meridian candidates starts with the extraction of the meridian position.

Extraction of the meridian position is performed as follows, as shown in the flow chart of FIG. First, the area where the edge extraction has been performed is divided into partial areas 81 as shown in FIG. Then, this is filtered (step S81 in FIG. 12C). Next, one point is selected from each of the upper end and the lower end of the partial area 81 for which edge extraction has been performed, and one of these selected points is used as the start point 8
2, the other end point 83, the start point 82 and the end point 83 are connected by a straight line 84, and the edge strength of the pixels on the straight line 84 is integrated (step S82 in FIG. 12C).

At the upper end and the lower end of the partial area 81, there are points corresponding to the number of constituent pixels at the end position. Therefore, similar processing is performed for all possible combinations of the upper and lower points.

The edge strengths of the pixels on the straight line 84 are integrated as shown in FIG. That is, if the number of pixels at the upper end used for integration is M and the number of pixels at the lower end is N, then M × N
Secure storage area for. In this storage area, each combination connecting the positions of the upper end and the lower end of the partial area 81 corresponds to a specific one array element.
In the xN storage area, the value of the edge strength of the pixel on the straight line 84 is added to the position of the array element corresponding to each straight line 84, and the data is placed. That is, the value of the edge strength of the pixel passing through the straight line 84 is voted for the position of the array element corresponding to each straight line 84. When the voting value is written in the M × N storage area in this way, as a result, a histogram as shown in FIG. 12B is obtained (step S83 in FIG. 12C).

If the pixel is an edge pixel, the pixel at that pixel position has a high edge strength. Therefore, in the voting result of FIG. 12B, the straight line corresponding to the pixel having a large voting value is likely to be an edge.

Therefore, a predetermined number of positions of the two-dimensional local maximum values in this M × N region are selected in descending order of the integrated value, and they are used as meridian candidates (step in FIG. 12C). S84). Then, the position of the meridian candidate is known by the backward calculation of the starting point position and the ending point position of the meridian (step S84 in FIG. 12C). The same process is performed on both the left and right meridians of the image, the meridian candidates on both sides are obtained, and their positions are known.

An example of the states of the meridian candidates on both sides of the insulator thus obtained is as shown in FIG. 5, and b1 to bn in FIG.
Is a meridian candidate sought by.

When the candidates for both the right and left meridians of the image are known, the position correction of the model projection image for fitting verification and the fitting verification are performed for the combination of these meridian candidates on both sides.

At this time, if there are a large number of left and right meridian candidates, there is a fact that "the left and right meridians are parallel". Therefore, this knowledge is used, and only those combinations whose angles are less than a certain value are used. By correcting the position of the model projection image for the fitting verification and performing the fitting verification, redundant processing can be omitted.

Also, if the range in which the model exists is known to some extent, it is possible to omit a combination in which the widths of the left and right meridian candidates are extremely wide, or an extremely narrow combination.
Redundant processing can be omitted.

The position correction of the model projection image for the fitting verification and the fitting verification are performed in the next step S105.
It is the processing from to step S109.

First, in step S105, a combination of meridians is created. In the process here, one combination is created for each of the left and right meridian candidates thus obtained. Then, the processing from step S106 to step S109 is repeated for all the combinations thus obtained.

The processing in step S106 is model position correction processing. In step S106, the position of the model is corrected for each combination of left and right meridians.

When a cylindrical shape is projected, assuming that it is a projection of the cylindrical object to be recognized, the positional relationship between the camera and the recognized object when the cylindrical object is viewed from the direction of its axis. 13 is shown in FIG.

Here, the radius R of the circle in FIG. 13 is the radius R of the cylindrical shape. By setting the combination of the left and right meridians, the positions A'and B'of the meridians in the image become known, while the focal point F of the camera and the direction of the optical axis can be obtained from equations (9) to (12). Therefore, the central angle 2θ can be correctly obtained. Since the triangle ACF and the triangle BCF are congruent, the distance d from the camera to the object can be calculated by the equation (19).

[0123]

(Equation 6)

By considering the center position of the left and right meridians to be the center position in the image of the recognition target, the depth d is used to calculate the 3 of the recognition target from the equation (18) as in step S102.
Dimensional coordinates can be obtained. Once the three-dimensional coordinates of the recognition target are obtained, the position of the model can be aligned with the recognition target using this.

As a result, of the five types of change factors of the appearance of the object when the position and orientation of a cylindrical or conical object can be freely changed three-dimensionally, the "size" matches and the "center axis" And the vertical position ”.

In the model in which the positions are matched, "inclination on the screen" is applied by the following procedure.

First, the inclination of the new projected image on the screen is calculated. For this purpose, for example, the inclination on the screen of the line connecting the projection position of the center position of the upper end surface of the recognition target and the projection position of the center position of the lower end surface may be checked. Here, when there is a difference between the average inclination of the left and right meridians thus obtained and the inclination of the projected image on the screen, the recognition target model is rotated with the optical axis obtained by the equation (11) as the rotation axis, To match. With this, 5
Among the changing factors of the species, the "inclination on the screen" matches.

By the processing so far, three change elements out of the five kinds of change elements of the appearance of the model are determined. And the remaining parameters are "position in the axial direction of the center"
And "rotation angle in the depth direction".

Next, the edge extraction processing of the elliptic portion in step S107 is performed. In step S107, a projection image of the corrected recognition target model position is created, and the upper end surface of the projection image,
Image features in the region near the bottom edge are extracted. In this case, since the elliptical contour line of the cylindrical object is extracted, an edge extraction method having no directionality is applied here, unlike the processing in step S103.

Next, the model fitting in step S108
Move to verification processing. This process is performed by the applicable verification unit 7. By applying the process of step S108 under the control of the recognition process control unit 8, the process of applying the contour lines on the upper and lower ellipses of the model projection image to the edge extraction result in the image and the process of matching are performed. Done.

The fitting in step S108 is a process for determining the two types of parameters remaining in the processes up to the previous step. In other words, the two remaining parameters (two undetermined parameters) are the "position in the direction of the central axis of the cylindrical object" and the "rotation angle in the depth direction".
It is possible to determine the two types of parameters by generating various types of model projection images according to these two types of parameters and performing fitting verification.

The specific contents of the processing are as follows. The recognition target model changes the model position and posture according to these two types of parameters. The first is an axial translation of a cylindrical object. The axial direction of the object is determined in step S102.
Similar to the equations (16) and (2), a plane Q ′ passing through the center of the object and perpendicular to the camera optical axis is obtained, and the axis of the object is projected on the plane. Second,
This is rotation in the depth direction, and this rotation axis is set as an axis that lies on the above-mentioned plane Q ′ and passes through the center of the object and is perpendicular to the translation direction.

By changing these two types of parameters two-dimensionally, the fitting projection between the model projected image and the image feature is verified, and the optimum fitting position and angle are obtained.

The parameter change range and the number of divisions are set as follows. The parameter change range is the entire area where the projected image of the recognition target object can exist. In reality, as in the case where the initial position and orientation of the recognition target object could be set in step S102, the orientation and position of the object can often be assumed within a certain range.

For example, if the camera does not photograph an object from extremely above or below, the rotation angle can limit the existing region within the range of 180 degrees. If the camera can distinguish whether it is "looking up", "looking down", or "looking from sideways", set the change area within a range of at most 90 degrees. It is enough.

On the other hand, it is not necessary to change the position of the object in the axial direction to the area outside the screen, and the range of change can be limited depending on the size of the screen.

As described above, these two types of parameters can limit the change range.

The number of divisions of the parameters may be set so that the position change of the projected image in the image is less than 1 pixel. Finer changes in position and orientation are not relevant to the fit in the image. Therefore, the number of divisions can be limited to a finite number.

From the above, the change range of the parameters and the number of divisions are determined. Therefore, model projection images are generated for all the combinations of these two types of parameters, and the fitting verification is performed.

The applicable verification is as follows. A scale of judgment is required for the applicable verification, but the following scale is adopted here.

It is now assumed that the pixel positions on the outer contour line of the model projection image are {C _i (c _ix , c _iy )} i = 1, 2, ..., M. In addition, it is assumed that the edge strength of the pixel position (c _ix , c _iy ) of the edge image output by the image feature extraction unit 3 is {E (c _ix , c _iy )} i = 1, 2 ,. At this time, the scale of the fit is defined by the following equation.

[0142]

(Equation 7)

That is, the values of the edge strength are integrated as they are, and it is judged that the higher the value, the better the fit.

This integration of the fitting scale may be performed for all the pixels of the outer contour line of the model transmitted from the model projection image generation unit 6, but in this step S108, attention is paid only to the elliptical contour line. Therefore, it is also possible to integrate the fitting scale only for the contour line on the ellipse.

The above steps S106 to S1
By the procedure up to 08, the three-dimensional position and orientation of the model that matched one pair of left and right meridian candidates were obtained. This procedure is similarly performed for all combinations of meridian candidates created in step S105.

If all combinations of meridian candidates have been completed (step S109), then step S109.
The processing moves to 110 (optimal position / orientation selection processing).
Then, by the processing here, the position and orientation of the model having the maximum fitting scale is selected from the positions and orientations of all the models thus far obtained, and this is used as the final recognition result. Then, as a result of this recognition, the optimum position and orientation can be selected. FIG. 6 shows the state of the model in which the selected fitting scale is maximum. The model in the wire frame format shown by adding a symbol M in FIG. 6 is the selected model.

In this way, when the fitting verification unit 7 completes the selection of the optimum position and orientation, the result recognition unit 7 outputs the final recognition result in step S111, which is the output process. The process ends.

When an insulator having a colored band on the body is to be recognized as shown in FIGS. 3 to 6, the cylindrical recognition target model is not applied as shown in FIG. The recognition process of this system can be applied by setting the model by recognizing this paint color band as a columnar recognition target. In this case, it is also possible to use the information of the color of the paint band and the color of the surface of the insulator, and as described in Specific Example 2 below, the color of the cylindrical recognition target object and the recognition target If the information on the color of the object in contact with the upper and lower sides can be used, more stable processing can be realized.

In this way, as a method of obtaining information on the three-dimensional position and orientation of the recognition target object from the image of the cylindrical recognition target included in the image captured by the image pickup means (for example, TV camera) The feature (outline image) of the columnar recognition target object included in the image is extracted, the straight line portion is extracted from this, and a cylindrical model having the same shape as the recognition target object is prepared, and the extraction is performed. The contour image of the projection image of the model is aligned with the image of the straight line part of the recognition target object, the posture is adjusted to check the fitting state, and the projection image that best fits is selected, and the posture state or position The information of is acquired.

Therefore, it is possible to realize a position / orientation recognition apparatus which can measure the position and orientation of a cylindrical object whose appearance changes three-dimensionally from a two-dimensional image. By this, if the shape to be recognized such as a column is specified, the position and orientation can be specified from the image, so that it is possible to control the autonomous work robot that attaches and detaches the object having the specific shape. Will be available.

The basic concept of this system has been described above. In this system, one of the elements that plays an important role in recognizing the position and orientation of an object is a measure of fit, and there are various modifications, so this will be described as another specific example. .

(Specific Example 2) In this specific example 2, a modification of the equation (20), which is a scale expression of the fit, is shown. In the above-described specific example 1, the scale of the fit is the integrated value of the edge strength values, but here, the method is one in which the integration is performed only at the position where the peak of the edge strength is taken.

The peak position may be discriminated by the following procedure, for example.

For example, in the processing of step S103 of FIG. 2 described in the first specific example, for the determination of the peak position when the edge extraction in the cylinder axis direction is performed, [1] in the lateral direction of the image within the extracted region Set the pixel of interest, [2]
If the edge strength of the pixel of interest is greater than the edge strength of both sides, it is stored as the peak position, and [3] it is the peak position among the pixels searched in the horizontal direction, and
The position having the maximum edge strength is determined as the maximum peak position.

It suffices to perform the procedure described below.

On the other hand, in the case of determining the peak position of the edge extracted by the edge extraction of the ellipse portion in step S107, if the direction for searching the target pixel is the direction perpendicular to the contour line of the model projection image generating unit 6. The same processing can be performed.

A method of integrating the scales only with the maximum peak position is conceivable, or a method of integrating all the extracted peak positions with each other is also conceivable.

It is also possible to use not only the edge strength of the output of the image feature extraction unit 3 but also the direction of the edge.

For example, if it is possible to use as information that the object to be recognized having a cylindrical shape is brighter (whiter) than the vertically adjacent objects, a method of integrating only the edges close to the desired edge direction can be considered. . For example, by adding a weight W (c _ix , c _{i y} ) to the integration of (Equation 8),

(Equation 8)

Then, the closer the extracted edge direction is to the desired edge direction, the larger the weight should be given.

For example, as the weight, an inner product of the estimated contour direction and the edge direction obtained by the image feature extraction unit 3 can be used.

Specific Example 3 Next, another modified example of the equation (20), which is a scale expression of the fit, will be described as a specific example 3.

In the above-mentioned specific example 1, importance is attached to the measurement accuracy,
In this method, the parameter division number in step S108 is set so that the positional change of the projected image on the screen is less than one pixel. However, if the required measurement accuracy is not so high, the number of divisions need not be set very finely. Therefore, the configuration of a simplified method in which the number of divisions is reduced and the measurement accuracy is lowered is also useful depending on the application.

In other words, in this case, the number of projection image patterns to be fitted to the same change range is small, so that the processing time can be greatly shortened and high-speed processing can be achieved.

On the other hand, on the other hand, if the pixel positions used for the integration of the judgment of the equation (20) are misaligned between the patterns and the integration is performed as it is, the integration result itself becomes unreliable and the peak value is lost. The accuracy of the judgment becomes unreliable. This is because the positional deviation between the patterns in the case of using the simplified method may reach several pixels.

For this reason, in this example, the integration method of equation (20), which is a fitting scale equation, is changed in consideration of this point.

For example, when the contour line position of the model projected image changes by a maximum of 2 pixels between each pattern, a mask as shown in FIG. 14 is prepared and this mask is used for the pixel at the contour line position. The edge image is masked and the values are integrated.

As described above, by adopting the method of performing the mask calculation on the edge image using the mask and integrating the values, the contour line position is reduced even if the number of trial patterns is reduced to simplify the process. It is possible to perform highly reliable fitting verification simply by changing the size and shape of the mask in accordance with the change amount of

An example of the case where the recognition target object has a cylindrical shape has been described in the concrete example 1, but the present invention can be applied even when the recognition target object has a conical shape or a truncated cone shape.
An example thereof will be described as Specific Example 4.

(Specific Example 4) In this specific example 4, a specific example of a position / orientation recognition apparatus for a conical object and a truncated cone object will be described. The configuration of the device may be the same as the configuration described in FIG. The processing flowchart is also basically the same as in FIG. However, although the recognition target is the cylindrical object in the first specific example described above, since the target here is a conical object or a truncated cone-shaped object, the meridian extraction portion and the left and right meridian candidates are accordingly changed. It is necessary to change the content of the parameter that is obtained when it is decided. That is, what is different from the first specific example is the part of the meridian extraction and the contents of the parameters obtained when the left and right meridian candidates are determined.

Only the parts different from the first embodiment will be described below.

In the recognition of the cylindrical object, the left and right meridians are used as the contour lines in the axial direction, but in order to recognize the conical object, this may be replaced with the meridian of the conical object obtained by the projection of the model. FIG. 15 shows the shapes in the case of a cone and a truncated cone object.

FIG. 15A shows a conical object, and FIG.
(B) is a truncated cone object. When feature extraction is performed on these objects, meridian edges and ellipse edges are obtained. 110
Is the meridian edge, and 111 is the ellipse edge.

Although the names of the edges to be extracted are different from those in the first specific example, the method of extracting the meridian edges and the ellipse edges is exactly the same.

That is, in the processing of step S103 of FIG. 2, the initial estimated position and direction of the meridian edge are estimated,
The meridian extraction area is set according to the position and direction. Next, in step S104, a vote is made to obtain the position and direction of the meridian edge.

However, it is only necessary to note that the right and left meridians are not always parallel, and therefore it is necessary to perform processing according to each direction.

In the recognition of the cylindrical object described in the first specific example, in step S106, the "inclination on the screen" and the "axis" of the object are selected from the five types of change elements in the appearance of FIG. Determine the three elements of the model's "inclination on the screen", "axis center vertical position", and "size" according to the three elements "vertical center position" and "size". I was able to.

However, in recognizing a conical object or a truncated cone-shaped object, the combination of changing elements found from the left and right meridians changes. Then, when the left and right meridians are extracted, the "position of the apex in the screen" is determined for the conical object, and, as in Example 1, the "center position in the direction perpendicular to the axis" and the "tilt on the screen". Is decided. The position in the depth direction is not determined from the widths of the left and right meridians, but the constraint condition of the position of the three-dimensional center of gravity of the recognition target model is obtained.

This will be described with reference to FIG. In the conical object, when the left and right meridian positions and directions are determined, the axial position and size of the cone in the screen are related. That is, it is assumed that there are a plurality of conical objects 121a to 121n in the screen 120 as shown in FIG. Then, it is assumed that those vertex positions are 122.

Such a cone having the same apex position indicates that "the apparent size is large and the center position in the screen is far from the apex" and "the apparent size is small and the center position in the screen is small". Is near the apex of the cone-shaped object. ”There are innumerable intermediate stages.

However, since the projection positions of the vertices are fixed on the screen, the three-dimensional position of the recognition target model 121 is such that the vertex position TP is constrained on the constraint line L as shown in FIG. 16B. Will change in the depth direction.
In other words, among the five types of changing elements of appearance, this constraint condition exists between the "position in the depth direction" and the "position in the axial direction", so these have only one degree of freedom in total. Absent.

Summarizing the above, the factors that change the appearance of the conical object obtained from the left and right meridian candidates are "center position in the direction perpendicular to the axis", "tilt on the screen", and "center in the depth and axial direction". There are three types of conditions, which restrict the position ". Therefore, in step S106, the “inclination on the screen” is found from the left and right meridian candidates, and the number of undetermined parameters is 2 at this stage.
As a seed, this is the same as in Example 1.

Therefore, the subsequent steps can be processed in exactly the same way as in the first embodiment. Step S108
Two undetermined parameters, that is, "rotation angle in the depth direction" and "center position in the screen in the axial direction" are obtained by voting in. At this time, the center position on the screen in the axial direction may be the three-dimensional center position of the model in the depth direction, because the constraint condition exists between these elements, the degree of freedom is "1" in total. Because.

A truncated cone-shaped object can be processed in exactly the same manner as a cone-shaped object. In the case of a truncated cone-shaped object, as shown in FIG. 17, an intersection point obtained by extending left and right meridians may be set as a virtual position of the apex, and this position may be used.

The recognition target model may be exactly the same as that shown in FIG. 9, but when a frustoconical object is used as a model, it is obvious that the calculation will be easier if the vertex position information is also stored. However, the vertex position in the case of using the truncated cone as a model is related only to the calculation and may not be related to the projection image creation.

The fitting of the ellipse edge in step S108 is the same as in the first specific example. The difference is that there is only one ellipse edge in the case of a conical shape, and there are two ellipse edges in the case of a truncated cone shape.

In this example, the recognition processing control unit 8
By performing the fitting order in the order from the contour line in the axial direction of the cylindrical cone shape to the contour line in the circumferential direction of the cylindrical cone shape, the fitting is efficiently performed corresponding to the five types of variable elements of the cylindrical cone shape. It will be possible.

(Specific Example 5) Specific Example 5 will be described below.
In this specific example 5, as shown in FIG. 3 to FIG. 6, in the configuration diagram of FIG. 1 when the recognition method of the present invention is applied when an insulator having a colored band on the body is a recognition target. Another embodiment of the applicable verification unit 7 will be described.

In the specific example 1, the specific example 2, and the specific example 3, the fitting scale is used as the edge integrated value, and only the magnitude of the value is used to judge whether or not the fitting is performed. Specific Example 5
Is a method of using luminance information (luminance information on an image) on the surface of a paint band or the like for verification of fitting.

The contents described in this example can be said to be another form of step S110 in the flowchart of FIG. The effect of the recognition method having the configuration of this specific example will be briefly described as follows.

In this example, a cylindrical object as shown in FIG. 3 is recognized. In the present invention, the outline of the object to be recognized and the projected image of the recognition model are applied for recognition. If there are few lines appearing as contour lines in this object, for example,
Vertical lines are reflected on the surface of the object due to shadows,
When there are many linear objects in the background, the fitting candidate in which these "false lines" and the original contour line are combined may have a larger fit scale of the equation (20).

Even in such a case, the "correct answer" candidate has a sufficiently large fitting scale as compared with many other (mostly unapplied) candidates. In other words, it is only necessary to perform a verification process that rejects candidates that fit the “false line” from the minority candidates with a sufficiently large fitting scale and leaves the “correct answer” candidates. In order to enable correct recognition even in such a case, the information of the area is also used to enhance the processing of the fitting verification unit 7.

That is, the present specific example 5 is a specific example in which the verification process is enhanced.

FIG. 18 is a configuration diagram for realizing the fitting verification unit of the present specific example 5. FIG. 19 is similar to FIG.
8 is a flowchart of the process of the applicable verification unit in the configuration diagram of FIG. 8, and FIG. 20 is a flowchart showing an example of the verification procedure on the profile of steps S1907 and S1908 in the flowchart of FIG.

Each step will be described below.

Now, it is assumed that the processing up to step S109 in the flowchart of FIG. 2 is completed, and the position and orientation of the model having the maximum fitting scale are obtained for all combinations of meridian candidates. The verification process starts from here, but the verification process control unit 188 of FIG. 18 applies the verification data of the video feature data from the video feature extraction unit 3 of FIG. 1 and the fitting candidate data of the model projection image generation unit 6 of FIG. The projection position on the image of the fitting candidate is input as the value of the fitting candidate, and the fitting scale value of the fitting candidate is input as the fitting verification candidate data from the recognition processing control unit 8 of FIG. 1 (step S1).
901).

At this stage, the position and orientation of each fitting candidate are corrected so that the model projection image fits to the edge positions of the meridian and ellipse. Therefore, the difference between the positions of the candidates is the left and right end points. Appears in the position. Next, FIG.
A table showing the positional relationship of each candidate as shown in 2 is created. That is, in step S1902, a list of sharing candidates for the right end point position is created and stored in the right end point list storage unit 182.

Similarly, in step S1903, a shared list of left end point positions is created and stored in the left shared list storage unit 183.

The verification process from step S1906 to step S1911 is sequentially performed on all fitting candidates. Steps S1904, S1905, and S1912 are control procedures for processing all candidates.

First, in step S1906, a small area for verification is set for a candidate to be verified, and its brightness value data is loaded. As shown in FIG. 21, a small area for verification is set along the boundary of the line of the insulator band (referred to as a profile in the following description) with a white region and a black region sandwiching the boundary of the line of the insulator. Then, set this in a strip shape from the left end position to the right end position. The vertical length of this small region is set to, for example, half the width of the band in the image, and the horizontal width is set to 1 pixel or 2 pixels. From the brightness value distributions of the white area and the black area, the average brightness of each area and the expression (13)
~ Separation degree η between regions is calculated based on Expression (15).

The verification processing for the verification small area is performed separately for the left and right sides. Therefore, the center position, the right end position, and the left end position are set in the small area in FIG. Step S19
The verification procedure on the left profile is executed in 07, and the verification procedure on the right profile is executed in step S1908.

The order of these two verification procedures may be exchanged. Details of the verification procedure on the profile will be described later. In step S1909, it is determined whether the verification result on the left profile and the verification result on the right profile are both passed. If both are passed, the entire verification result is also passed (step S1910), and if one is a rejection decision, The overall verification result is also rejected (step S1911).

Step S1 for all fit candidates
When the procedures up to 912 are completed, step S19
In step 13, one candidate having the largest fitting scale is selected from the candidates judged to be acceptable, and this is set as the final recognition result.

Next, the verification procedure on the profile will be described.

FIG. 20 is a flowchart of the verification procedure on the profile. This flowchart shows that when the object is photographed with a black and white camera (monochrome image camera), the paint color band of the insulator appears black (that is, the brightness is low).
It is explained assuming that.

[0206] In this verification procedure, the determination status is "PAS".
Three types, S "," REJECT ", and" DOUBT "are used. First, in step S2001, the determination state is" PAS. "
S ”is set. In step S2002, the verification position is set to FIG.
The center position as shown in 1 is set. The following steps S2003 to S2014 are repeated from the verification start point (center position) to the verification end point (right end position or left end position).

In step S2004, it is determined whether the average brightness of the black region is larger than the set value. If the average brightness is larger than the set value, the state is not changed and the process proceeds to step S2014. This set value is a value for determining whether the image brightness is saturated.

If the average luminance of the black area is larger than this value, that portion causes specular reflection and the image luminance is saturated, so the state is not changed as an exceptional process.

In step S2005, the average brightness of the black area and the average brightness of the white area are compared. If the average brightness of the black area is larger, it is out of alignment with the original fitting position.
The state is set to "REJECT" (rejection) (step S20
10) and proceeds to step S2014. If the average brightness of the black area is not higher than the average brightness of the white area, step S
Proceed to 2006.

In step S2006, it is determined whether the degree of separation between the black area and the white area is smaller than the set value. This set value is set to a relatively low value (for example, 0.5) as the degree of separation. If the degree of separation is high, step S200
Proceed to 8. If the degree of separation is low, the process proceeds to step S2007.

Since the position and orientation of the verification candidate fitting candidate for which a "Yes" determination is made in step S2006 are doubtful as to the correct answer, the end point shared list (18 in FIG. 18).
2 and 183), it is checked whether or not there is a candidate that has an endpoint on one side at the current verification position and shares the endpoint on the other side with the current verification target (step S2007).

The endpoint sharing list refers to the column of fitting candidates of the validation target in the left endpoint sharing list when the right profile is being verified, and the right endpoint sharing list is being verified when the left profile is being validated. Refer to the column of fit candidates.

If there is no candidate that shares the end point at the current verification position, it is determined that "separation degree is lowered only for this position for some reason", and the determination state is "not changed".
(Step S2011), and the process proceeds to step S2014. Possible reasons here are, for example, a mark on the surface of the insulator and a cable or the like which is present in the front so as to hide the object.

The fact that there is a candidate that has one end point at the current verification position and shares the other end point with the current verification target means that the current verification position is regarded as the recognition target end point and the model is applied. It means that there was a candidate. That is, this indicates that there is a possibility that the meridian to be recognized may exist at this position. For this reason, the candidate currently being verified is determined to be a suspicious candidate, and the determination state is set to “DOUBT”,
(Step S2012), the process proceeds to step S2014.

In step S2008, the determination state is "D.
If it is OUBT ", the process proceeds to step S2009, and if not, the process proceeds to step S2014. In step S2009, it is determined whether the degree of separation between the black region and the white region is larger than the set value. It is determined that the degree is a relatively high value (for example, 0.8), etc. If the degree of separation is high, the process proceeds to step S2013 and the state is set to "PA".
Change to SS ″ and proceed to step S2014. If the degree of separation is low, proceed directly to step S2014.

The meaning of the determination made in step S2009 is as follows. The fact that the state was "DOUBT" was judged to be a suspicious candidate in the profile at the middle position, but when it was seen that the separation degree was sufficiently high again, the reason why the separation degree in the middle was There is a possibility that the candidate to be verified is the correct answer just because Therefore, the determination state is returned to "PASS".

In step S2014, one unit is moved to the outside of the verification position, and the process returns to step S2003. For right side verification, go to the right; for left side verification, go to the left.

[0218] Steps S2003 to S201
4 is repeated for all the verification positions, and when the processing is completed up to the verification end point, the process proceeds to step S2015. Step S20
It is determined in 15 whether the determination state is "PASS", and if the result is "PASS", the final determination result is passed (step S2016), and otherwise, "rejected" (step) S2017). This is the end of the flowchart.

The candidate whose judgment state has become “DOUBT” is recovered if the separation degree has recovered to a sufficiently high level on the way (steps S2008, S2009, S20).
13). However, if not, it is rejected. With such a method, even if the degree of separation may be lowered in the middle of the region for some reason, it is possible to make a pass judgment for a candidate that is well fit as a whole.

As described above, as in the fifth specific example, the processing of the fitting verification unit 7 is enhanced and the information of the area is used in addition to the information of the contour line so that the shadow of the surface of the object or the false background It is possible to select a correct answer candidate instead of the candidate fitted to the line.

Although various concrete examples have been described above, in short, the present invention obtains image information of an image of a recognition target, extracts image features of an image obtained from this image information, and extracts the extracted image features. , Comparing the prepared projection image of the three-dimensional model of the recognition object, verifying the degree of fitting, sequentially correcting the position and orientation of the model, obtaining the projection image that best fits, and obtaining the best projected image. By determining the position and orientation from the applicable projected image,
It is designed to recognize the three-dimensional position and orientation of the recognition target object. Therefore, the position and orientation of the recognition target object of a specific shape such as a cylinder, a cone, or a truncated cone are measured from an image. This will open the way to the realization of autonomous work robots.

Therefore, the present invention is an optimum position / orientation recognition apparatus applied to, for example, an autonomous work robot or an intelligent robot, and it shows image characteristics of an image obtained from a TV camera attached to the robot and a recognition target model inside a computer. By comparing the projected images, verifying the fit, and successively correcting the position and orientation of the in-computer model, the projected image that best fits is obtained, and the position and orientation are obtained from the projected image. It becomes possible to recognize the three-dimensional position and orientation of the object.

The present invention is not limited to the above-mentioned specific examples and can be carried out in various modified forms.

[0224]

As described in detail above, according to the present invention, an object is recognized only from the image of the object to be recognized without attaching any marker to the object to be recognized having a specific shape, for example, a cylinder or a cone. It is possible to recognize or measure the three-dimensional position and orientation of the robot, and to play the role of the autonomous robot vision.

[Brief description of the drawings]

FIG. 1 is a diagram for explaining a specific example of the present invention,
The block diagram explaining the structure of the specific example 1 of this invention.

FIG. 2 is a diagram for explaining a specific example of the present invention,
3 is a flowchart of processing in a specific example based on the configuration of FIG. 1.

FIG. 3 is a diagram for explaining a specific example of the present invention,
The figure which shows the input image example input from a TV camera.

FIG. 4 is a diagram for explaining a specific example of the present invention,
The figure which shows the example of the image feature extraction area | region for axial edge extraction.

FIG. 5 is a diagram for explaining a specific example of the present invention,
The figure which shows the example of the extracted meridian line candidate.

FIG. 6 is a diagram for explaining a specific example of the present invention,
The figure which shows the final fitting result.

FIG. 7 is a diagram for explaining a specific example of the present invention,
The figure for demonstrating the edge extraction method which has directivity in the edge of the direction applied in this invention.

FIG. 8 is a diagram for explaining a specific example of the present invention,
The figure which shows the mask example used for the mask calculation for extracting a part of elliptical arc used in the specific example of this invention.

FIG. 9 is a diagram for explaining a specific example of the present invention,
The figure which shows the example of a three-dimensional model of the recognition target object used in the specific example of this invention.

FIG. 10 is a diagram for explaining a specific example of the present invention, and is a diagram showing five types of changing elements in the appearance of the model for explaining the specific example of the present invention.

FIG. 11 is a diagram for explaining a specific example of the present invention, which is a three-dimensional diagram from a two-dimensional position for explaining the specific example of the present invention.
Theoretical explanatory drawing which performs conversion to a dimensional position.

FIG. 12 is a diagram for explaining a specific example of the present invention, and an explanatory diagram of a meridian extraction method used in the specific example of the present invention.

FIG. 13 is a diagram for explaining the specific example of the present invention, and is a diagram for explaining the positional relationship between the left and right meridians used in the specific example of the present invention, the target object, and the camera.

FIG. 14 is a diagram for explaining a specific example of the present invention, in which the maximum position of the contour line of the model projection image is 2 between each pattern.
The figure which shows the example of a mask for mask calculation for contour line position extraction in case a pixel changes.

FIG. 15 is a diagram for explaining a specific example of the present invention, and is an explanatory diagram of edges of a conical object and a truncated cone object that are targeted in another specific example of the present invention.

FIG. 16 is a diagram for explaining a concrete example of the present invention, and is a diagram for explaining a constraint relationship by apexes of a conical object and a truncated cone object, which are another specific example of the present invention.

FIG. 17 is a diagram for explaining a specific example of the present invention, and is an explanatory diagram of a virtual apex position of a truncated cone-shaped object.

FIG. 18 is a diagram for explaining the specific example of the present invention, and is a configuration diagram for realizing the fitting verification unit of the specific example 5;

FIG. 19 is a diagram for explaining the fifth example of the present invention, which is a flowchart of the process of the fitting verification unit in the configuration diagram of FIG. 18;

FIG. 20 is a diagram for explaining a specific example of the present invention, which is step S19 in the flowchart of FIG. 19;
07, a flow chart showing an example of the verification procedure on the profile of S1908.

FIG. 21 is a diagram for explaining the fifth example of the present invention and is a diagram for explaining an example of setting a small area for verification.

FIG. 22 is a diagram for explaining the fifth example of the present invention and is a diagram showing an example of the positional relationship of each candidate.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... TV camera 2 ... Image input part 3 ... Image feature extraction part, 4 ... Straight line detection part 5 ... Camera parameter storage part, 6 ... Model projection image generation part 7 ... Fit verification part, 8 ... Recognition process control part 9 ... Result Output section.

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Kyoichi Okamoto 1-1-30 Oyodo Naka, Kita-ku, Osaka-shi, Osaka In-house Toshiba Kansai Branch

Claims (8)

[Claims] 1. An input unit for inputting an image of a recognition target object having a specific shape, and a three-dimensional model of the recognition target object having the specific shape,
A projected image of the three-dimensional model is calculated, and a projected image for correcting and comparing the three-dimensional position and orientation of the three-dimensional model with the recognition target object is obtained based on the information indicating the degree of fitting. Generating means, image feature extracting means for extracting the image feature of the recognition target object from the image obtained from the input means and from the projection image obtained by the model projection image generating means, and the image feature extracting means A matching determination unit that determines the degree of fit by comparing and comparing the image feature of the recognized object and the projection image obtained by the model projection image generation unit, and a fit determination unit that obtains information indicating the degree of fit. Outputting means for obtaining and outputting the position / orientation information of the best three-dimensional model projection image of the degree of fitting determined by the means from the model projection image generating means. , Position and orientation recognition device of the recognition target object, characterized by comprising. 2. An input unit for inputting an image of a cylindrical object to be recognized, a three-dimensional model of the object to be recognized, a projection image of the three-dimensional model being calculated, and information about the degree of fitting. Model projection image generation means for correcting the three-dimensional position and orientation of the three-dimensional model based on the above to obtain a projection image for comparison and collation with the recognition target object, and from the image obtained from the input means, and the model projection Image feature extraction means for extracting the image feature of the recognition target object from the projection image obtained by the image generation means, image feature of the recognition target object obtained by the image feature extraction means, and the model projection image generation means The obtained projection images are collated and compared to determine the degree of fit, and a fit determining means for obtaining information indicating the fit degree, and the fit determining means determines the fit. Position and orientation information of the best three-dimensional model projected image of about stuck Te and position and orientation recognition device of the recognition target object, characterized by comprising an output means for outputting the obtained from the model projection image generation means. 3. An input unit for inputting an image of a recognition target object having a conical or truncated cone shape, a three-dimensional model of the recognition target object, and a projection image of the three-dimensional model, Model projection image generation means for correcting the three-dimensional position and orientation of the three-dimensional model based on information about the degree of fitting to obtain a projection image for comparison and collation with the recognition target object; and an image obtained from the input means, Also, an image feature extraction means for extracting the image feature of the recognition target object from the projection image obtained by the model projection image generation means, the image feature of the recognition target object obtained by the image feature extraction means, and the model projection The fit determination means for determining the degree of fit by comparing and comparing the projected images obtained by the image generation means, and the fit determination means for obtaining information indicating the fit degree. The position / orientation of the recognition target object, comprising: an output unit that obtains and outputs the position / orientation information of the best three-dimensional model projection image of the degree of fitting determined by the model projection image generation unit. Recognition device. 4. An input unit for inputting an image of a recognition target object having a specific shape, and a three-dimensional model of the recognition target object having the specific shape.
Model projection image generation for calculating a projection image of the three-dimensional model and correcting the three-dimensional position and orientation of the three-dimensional model based on information about the degree of fitting to obtain a projection image for comparison and collation with the recognition target object Means, image feature extraction means for extracting image features of the recognition target object from the image obtained from the input means, and from the projection image obtained by the model projection image generation means, and the image feature extraction means. Means for extracting straight line information from the image feature of the recognition target object, information of the straight line portion detected by the straight line detection means among the image features of the recognition target object obtained by the image feature extraction means, and the model projection The fit determination means for determining the degree of fit by comparing and comparing the projected images obtained by the image generation means, and the fit determination means for obtaining information indicating the fit degree. An output unit that obtains and outputs the position / orientation information of the best three-dimensional model projection image of the degree of fitting determined by the disconnection unit from the model projection image generation unit. Position and orientation recognition device. 5. An input unit for inputting an image of a cylindrical object to be recognized, a three-dimensional model of the object to be recognized, a projection image of the three-dimensional model being calculated, and information about the degree of fitting. Model projection image generation means for correcting the three-dimensional position and orientation of the three-dimensional model based on the above to obtain a projection image for comparison and collation with the recognition target object, and from the image obtained from the input means, and the model projection Image feature extraction means for extracting the image feature of the recognition target object from the projection image obtained by the image generation means, and means for extracting straight line information from the image feature of the recognition target object obtained by the image feature extraction means, Of the image features of the recognition target object obtained by the image feature extraction means, information on the straight line portion detected by the straight line detection means and the projection image obtained by the model projection image generation means The fitting / judging means for obtaining the information indicating the fitting degree by comparing and comparing to determine the fitting degree, and the position / orientation information of the best 3D model projected image of the fitting degree, which is judged by the fitting judging means, A position / orientation recognition apparatus for a recognition target object, comprising: an output unit that obtains and outputs the model projection image generation unit. 6. Obtaining image information of an image of a recognition target, extracting image features of the image obtained from this image information, and using the extracted image feature as a projection image of the prepared three-dimensional model of the recognition target. By comparing, the degree of fit is verified, the position and orientation of the model are sequentially corrected, the projection image that best fits is obtained, and the position and orientation are obtained from the obtained projection image that best fits. ,
A position / orientation recognition method of a recognition target object, which is characterized by recognizing a three-dimensional position and orientation of the recognition target object. 7. An image characteristic of an image obtained from an image pickup means for picking up a recognition target is compared with a projection image of a prepared three-dimensional model of the recognition target, the degree of fitting is verified, and this verification result is obtained. Correspondingly, the projection image obtained by sequentially correcting the position and orientation of the model and the image feature are collated again to verify the degree of fit, and the projection image that best fits is obtained, and the projection image that best fits is obtained. A position / orientation recognition method, which comprises recognizing a three-dimensional position and orientation of a recognition target object by obtaining information on the position and orientation of the recognition target. 8. Obtaining image information of an image of a recognition target, extracting image features of an image obtained from this image information, extracting straight line portions from the extracted image features, and extracting the extracted straight line portions, By comparing with the prepared projection image of the three-dimensional model of the recognition target, the degree of the fitting is verified,
By three-dimensionally correcting the position and orientation of the model to obtain a projection image that best fits, and obtaining the position and orientation from the projection image that best fits, the three-dimensional position and orientation of the recognition target object A method for recognizing the position and orientation of an object to be recognized, which is characterized by recognizing.