JPH02226376A - Picture analyzing system - Google Patents

Abstract

PURPOSE:To extract the position of an unhidden object by dividing the objective area of a picture into plural areas having the same character, searching the relation of before and behind positions for plural surfaces corresponding to these plural areas, converting the picture to information including the relation of the before and behind positions for the plural surfaces and extracting the desired surface. CONSTITUTION:An area dividing means 12 divides the area into the objective areas in the picture, namely, into the surfaces and labels are attached to the respective surfaces. Then, a boundary is extracted from the surface of the same label. A structuring means 13 extracts a T-shaped connecting part, where three boundaries are crossed, among the nodes of the boundary and the overlapping relation of three surfaces is calculated in each T-shaped connecting point. After that, the relation in the T-shaped connecting point is converted to the overlapping relation between two surfaces and further, the overlapping relation between two surfaces is propagated to all the surfaces existing in the picture. Then, the surface relation of the whole picture is searched in the overlapping relation. An unhidden surface extracting means 14 extracts the unhidden surface not to be overlapped to any surface out of the structuring information of the surface relation obtained by surface relation searching means 1. Thus, the position of the unhidden object can be extracted.

Description

[Detailed Description of the Invention] [Summary] Image analysis method for extracting the position of an unoccluded object The position of an unoccluded object is extracted using a method that uses image structuring, regardless of the object represented by the image. A region dividing means for dividing a target region of an image into a plurality of regions having the same properties, and a region dividing means that divides a target region of an image into a plurality of regions having the same property, and a region dividing means that searches for a front-back relationship of a plurality of surfaces corresponding to the plurality of regions, structuring means for converting into information containing information, and surface extraction means for extracting a desired surface based on the output of the structuring means.

[Industrial application field]

The present invention relates to image recognition.2. More specifically, the present invention relates to an image analysis method for extracting the position of an unoccluded object.

It is extremely important to perform so-called object recognition (computer vision), which recognizes three-dimensional models by analyzing images from televisions and cameras. In a three-dimensional world, there are problems such as objects having different shapes depending on the viewing direction, objects being hidden by other objects, and sizes changing depending on the viewing direction. , it is important to understand the images. Technology for measuring the position of an unobstructed object that is in front of other objects from an image is important in the field of identifying objects that are in front of other objects in an environment where objects are mixed. For example, in a self-driving robot, the m-image input from a camera installed on the robot is used to measure the position of an unobstructed object in front of the robot, and the robot arm uses technology to avoid obstacles. It can be used in fields such as bin picking technology, which picks up a group of parts in order from the top.

[Conventional technology]

FIG. 26 (b) is an explanatory diagram of a conventional image analysis method for extracting the position of an unoccluded object. The same figure (al
1 is a scene in a three-dimensional world. 2 and 3
are input units for inputting binocular stereoscopic vision, that is, a stereo right image and a left image, respectively, and 4 and 5 are feature extraction units for extracting features of the right image and left image, respectively. 6
is an element matching section, and 7 is a distance calculation section.

8 and 9 in the figure [bl are examples of the left image and right image, respectively. In the left image, A is an object that is not hidden, that is, it is in front of other objects, and F is also an object that is not hidden. In the right image as well, a and f are objects that are not hidden, and their positions are slightly shifted compared to the left image. C, B, D, E+ in the left image are images with i occlusion, and similarly in the right image C, d, e are objects with occlusion corresponding to C, B, DE in the left image, respectively. be.

This conventional method uses binocular stereoscopic vision (i.e., stereo), and associates a group of features, i.e., line or surface elements, present in the left and right images based on constraint conditions such as epipolar conditions. , the position and depth of the associated elements are extracted using the principle of triangulation.

FIG. 27 is an explanatory diagram showing a conventional recognition method based on comparison with a model. This is another method in the prior art, and is a method for recognizing unoccluded objects by comparison with a target model, that is, a template model. In the same figure, 10
is an image of a scene, and there are objects without occlusion and objects with occlusion. 11 is a target model. In this method, a model of the object to be extracted is prepared in advance, the position of the object is determined by extracting features that are well associated with that model from the input image, and the depth is determined by applying ultrasonic waves etc. to the object at that position. It is something to be measured.

[Problem to be solved by the invention]

In the binocular stereoscopic method, all elements that satisfy the constraint conditions are matched, so the more complex the environment (i.e. the scene) to be handled, such as the large number of objects, the more elements to be matched, and the more the image becomes A problem has arisen in that the calculation time for the analysis becomes large. On the other hand, in the method based on model matching, it is difficult to select a model that allows easy and accurate matching between image features and the model, and since the model is generally fixed,
A problem has arisen in that the range of use of the method is limited to the range where the model can be used.

An object of the present invention is to extract the position of an unoccluded object using a method that uses image structuring, regardless of the object represented by the image.

[Means to solve the problem]

FIG. 1 is a block diagram of the present invention.

The area dividing means 12 divides the image into target areas, that is, planes, labels each plane, and extracts boundaries from the labeled planes of the same labeled plane, and the structuring unit 13 divides the image into target areas, that is, planes, and extracts boundaries from the labeled planes of the same labeled plane. After extracting the T-shaped connection where the three boundaries intersect and determining the overlapping relationship between the three surfaces at each T-shaped connection point, the relationship at the T-type connection point is converted to the overlapping relationship between the two surfaces, and then the relationship between the two surfaces is calculated. The non-hidden surface extracting means 14 propagates the overlapping relationship between the two surfaces to all the surfaces existing in the image, searches for the surface relationship of the entire image in the overlapping relationship, and the non-hidden surface extracting means 14 extracts the structure of the surface relationship found by the surface relation searching means 13. Extracts unoccluded surfaces that do not overlap with any other surfaces from the converted information.

[For production]

In the present invention, not only two-dimensional features such as area and perimeter length can be obtained from an image, but also areas with the same features that can be easily predicted from the image, that is, three-dimensional features such as the vertical relationship of the image between surfaces,
In particular, we extract the overlapping relationship of the three planes at the T-shaped connection point,
By characterizing the degree of overlapping of surfaces as image structuring and extracting unoccluded objects as surfaces that do not have front, back, or top surfaces, the position of unoccluded objects can be extracted regardless of the complexity of the image scene. This makes it possible.
l [Example] Hereinafter, an example of the present invention will be described with reference to the drawings.

FIG. 2A is a block diagram of a bin picking system according to the image analysis method of the present invention. 19 is a three-dimensional scene that contains various objects inside.

For simplicity, it is assumed that the target scene for bin picking includes several types of simple shapes such as industrial parts or rigid bodies such as rods, and that none of the objects are entangled with each other and are piled up. 20-a is an arm of the robot, and 20-b is an arm control section. 21 is a depth measuring section, 22 is a sensor, 23 is a camera, and 24 is a sensor positioning section. It is assumed that the transverse axes of the sensor 22 and camera 23 are aligned. This device is an object position measuring device that searches for an unconcealed object that is not hidden by other objects in the target scene 19 by processing the image input from the television camera 23, and determines the position of the object. The sensor positioning unit 24 aligns the non-concealed object, the sensor 22 senses it, the depth measurement unit 21 measures it, and the arm is controlled based on the measurement results. The arm control unit 20-b moves the arm 20-a to lift the uppermost object. In order to do this, in the present invention, the image taken out from the television camera 23 is manually input to the object position measuring device. Then, the region dividing unit 25 divides the image into regions of the same density or texture, labels each region, and extracts boundaries from the surfaces with the same label. Then, the structuring unit 26 focuses on the intersection points of the boundaries, extracts a T-shaped connection point where three boundaries intersect among the intersection points, calculates the overlapping relationship of the three surfaces at the T-shaped connection point, and then Convert the relationship into an overlapping relationship between two faces and structure it. Then, the overlapping relationship between the two surfaces is propagated to all the surfaces existing in the image, and the surface relationships regarding the surfaces of the entire image in the overlapping relationship are structured. Then, a non-hidden surface extracting section 27 extracts a surface that does not have an upper surface, extracts an object that does not overlap with any object, that is, a surface, and provides it to the sensor positioning section 24. Then, the sensor 22 detects this, and the arm control section 20-b moves the arm via the depth measuring section 21.

FIG. 2B is an explanatory diagram of the image structuring process of the present invention. The image shown in FIG. 5A is a grayscale image, and the image is divided into regions (surfaces) having the same pattern (texture), and each region is labeled. Then, boundaries are extracted from faces with the same label. Next, as shown in (2) and (2) in FIG. 2(b), the local overlapping relationship between each surface and the adjacent surface is determined. That is, first, attention is paid to the intersection points of the boundaries, and T-shaped connection points where the boundaries intersect are extracted from among the intersection points. The T-shaped connection point shown at the top of the figure is the intersection of three planes (A, B, and C), and plane A is the intersection of plane B.
It can be seen that it is located further forward in the depth direction than C and C. In this way, since the T-type connection provides the overlapping relationship of the three-dimensional surfaces (the presence or absence of concealment) as visual information, this information is used to find the overlapping relationship of the three surfaces that make up the T-type connection. After determining the overlapping relationship between the three surfaces at each T-shaped connection point, the relationship at the connection point is converted into the overlapping relationship between the two surfaces.

Then, as shown in (C) in the same figure, the overlapping relationship between the two surfaces is propagated to all the surfaces existing in the image, and the sequence of the entire image in the overlapping relationship is brought into perspective. In other words, as shown in Figure (d), if nodes are made to correspond to surfaces and their context is expressed as one ink, that is, a directed branch, for example, "house", "grass", and "entotsu" are subordinate. There is a roof, grass, and a car's body above it, leaves and wheels above it, and Miki above it.
It is possible to express the structure of an image with a graph.

As described above, surfaces that do not have upper surfaces can be easily extracted, and surfaces that do not overlap with any surface, that is, objects that are not hidden, can be extracted.

FIG. 3 is a configuration diagram of the area dividing section of the present invention.

In the figure, in response to the output of the noise removal unit 28, a texture analysis unit 29 extracts parts of the same texture, a boundary line extraction unit 30 detects boundaries, and a region labeling unit attaches the same label to the same area. .

FIG. 4 is a processing flow of the area dividing section of the present invention. 33
When the flow starts at 2, the sensor positioning section 2 starts at 333.
4, it is determined whether there is an instruction to capture an image, and if so, the image is captured from the television camera 23 in step 334. A noise removal section 28 removes noise from the image input from the television camera 23, and a texture analysis section 29 divides the image into areas with the same shading and pattern, that is, the same texture.

This becomes the texture analysis process of S35. Here, an area having the same texture is called a surface, and after texture analysis, in step 336, a label unique to that surface is assigned to all pixel values belonging to the surface. An image whose entire surface is labeled is called a label image.

At the same time as the label image is generated, an image is generated in which only the boundary lines of each surface are extracted by 4-connection. That is, in S37, the position of each surface is extracted, and this is transferred to the non-obscured surface extraction section 27. Then, in step 338, a boundary line is extracted from the texture-analyzed image to generate a line drawing.

Here, the pixel value indicating the boundary line is assumed to be 1. The label image and the line drawing are images of the same size, and the boundary label of the same label corresponds to the boundary of the line drawing. After extracting the boundary line, the process returns to S33 and waits for an instruction from the sensor positioning unit 24. That is, the processing of the region dividing section 25 starts its operation only when there is an operation command from the sensor positioning section 24. If no instructions are given, the system will wait until instructions are given, but if no surface is extracted through texture analysis, all operations of this system will end.

FIG. 5 is a configuration diagram of the structuring section of the present invention. In the figure, 39 is a 3-plane connection point extraction unit, 40 is a T-type connection determination unit that extracts a T-type connection point where 3 boundaries intersect among the intersection points of 3-plane connection points, and 41 is a 3-plane connection determination unit at each T-type connection point. After determining the overlapping relationship between the surfaces, the relationship at the connection point is converted to the overlapping relationship between the two surfaces, and the overlapping relationship between the two surfaces is further propagated to all surfaces existing in the image, and the entire image in the overlapping relationship is This is a surface relationship search unit that performs structuring.

FIG. 6 is a processing flowchart of the structuring section of the present invention. When the flow starts, it is determined in step 342 whether a label image and a line drawing have been input from the area dividing unit 25, and if they have been input, processing of the line drawing, that is, template matching is performed in step S43. For example, by manipulating and matching line drawings using the four types of templates shown in FIG. 8, the position on the image of the connection point where the three surfaces meet, that is, the connection point of the three surfaces, is determined. Then, in 344, the relationship between adjacent connection points is extracted. That is, the boundary line is traced and it is detected whether the traced pixel is a connection point.

In S45, the labels of the three sides constituting the connection point are determined.

346 outputs three-sided connection information. Then, a T-shaped room at the connection point is performed to search for the relationship between the two surfaces, and then the relationship between the surfaces is searched for the entire image. When the process is finished, return to S42,
If no image has been input, it will be in a waiting state.

FIG. 7 is a configuration diagram of the three-plane connection point extraction section. Reference numeral 43 denotes a template matching section, which performs processing to find three-plane connection points on a line drawing expressed by four connections.

FIG. 8 is an example diagram showing an image area connected by three boundary lines. In the case of 4-connected boundaries, the connecting points of 3 boundary lines are expressed in the form shown in FIG. Therefore, matching is performed to see if the surrounding pixels around the coordinates (X, Y) form the four types of patterns shown in FIG. This makes it possible to detect three-sided connection points.

FIG. 9 is a processing flowchart of template matching. The image is a two-dimensional image in the X direction and the Y direction, as shown at 46, and the processing ends when the image size is MAX. If it starts at 344, it initializes at 345. Let X be 2, Y be 2, and index i be 1.

If Y has not reached MAX in step 346, the process goes to step S47, where it is checked whether surrounding pixels centered on X and Y correspond to any of the four types. That is, these four types are templates, and it is checked whether a given four-connected line drawing corresponds to any of these patterns. Note that it is assumed that the pixel indicated by diagonal lines in the figure has a value of 1.

In S47, if any one of these four types matches, the coordinate is X.

Store Y in the memory at 349 and label its position as J. That is, the position of the three-plane connection point where the three planes meet is labeled J8. Increment i and X in S50 and 851 respectively, but if X reaches MAX in S52 (T), that is, if one line is finished, S5
3, but if the first line is not finished (F), the same Y
In other words, the pixels in the same row are advanced one by one. That is, the process returns to S46 and template matching is performed in S47. In this 347 template matching comparison, if no match is found, the process directly proceeds to S51 and the coordinate is incremented by 1 in the X direction. In this way, in S52, the process proceeds with Y fixed until the end of one line, and when X reaches MAX in S52, in S53, X is initialized to 2, Y is advanced by 1, and the process proceeds to S46. Return to the next line and perform template matching in the same way.

After template matching is completed, boundary lines are then traced. FIG. 10 is a flowchart of boundary line tracking processing.

FIG. 11 (al is an example diagram of ■template matching section 43 (FIG. 7) and ■boundary line tracing section 44 (FIG. 7). In the same figure, between 52 and 33 centering on J3,
There are three boundaries between 36 and 53, J, and J3, and there are connection points other than J3 at J2. There are three points: J6° and J7. Tracing this boundary is the process of ① boundary line tracing 44 in FIG. That information is issued as boundary information. In the flowchart of FIG. 10, when the flow starts, 357
Initialize i to 1 with . If i is not MAX+1 at 358, it enters S59 and returns Jl, for example, Jl in FIG. 11 (al).
Check whether all three boundary armors from 3 have been tracked. If 8" has been traced, i is incremented in S61 and the process returns to 358. If it has not been traced, the remaining boundary line is traced in S60, and the traced pixels are transferred to other connection points, e.g. ,J2 ,Ji, ,J
Determine whether it is any one connection point of q. If it is a connection point, store the label of that connection point in 364,
Return to S59. If the pixel tracked in step 362 is not a connection point, the process goes to step S63 to continue tracking and leave a tracked history. That is, the boundary line information is saved and the process returns to S62. S5
When you return to step 9, if you have traced all the boundaries, S
The process returns to S58 via 61.

When i reaches MAX+1, the process returns to the main routine. The program then enters the process 45 in FIG. 7, and enters the process (2) of determining the labels of the three sides constituting the connection point.

FIG. 11(b) shows a label image in the vicinity of a pixel, and in the same figure, labels are examined clockwise from an arbitrary pixel around the three-sided connection point, that is, from .

FIG. 12 is a flowchart of the process of labeling the three sides constituting the connection points. When processing starts, i is set to 1 in 364.
Initialize it with 365 and check whether i becomes MAX+1.
Judge by. If it is not the last one, the process goes to S66, sets the counter value to O, initializes ■ to 1, and makes the stack empty. Then, it enters 367 and checks whether the value of the pixel I around Jl, for example J3, is the same as the value stored in the stack. That is, if there is already a label value in the stack, there is no need to enter a new label number in the stack, so the process jumps to S70. However, if it is not a value stored in the stack, the label is added to the stack at 868 and the pixel value is added to the stack. is stored, and the count is incremented in S69.

Then, in S70, ■ is incremented, and in S71, it is detected whether the counter value has reached 3. Counter value is 3
If not, the pixels around the connection point are numbered clockwise as shown in the figure on the right (see FIG. 11(b)). Here, 1 is a number indicating the position of the surrounding pixels, and the counter (COUNT) value corresponds to the three planes forming the connection point. 3
When the counter value reaches 3 in step 71, the process enters step 372 and stores the three values stored in the stack as the labels of the connection point J1, that is, the three sides constituting the connection point. Then, in step 373, i is incremented and the process is repeated until the end.

The process of labeling the three surfaces described above will be explained in more detail using FIG. 11(C).

For all three surface connection points, three boundary lines with that point as an end point are traced through four neighboring pixels until each of them reaches another connection point. The traced boundary line is held as a pixel string, and is held as boundary line information together with the label of the connection point that is the end point. Furthermore, the connection between adjacent connection points via a boundary line is also recorded as one of the next pieces of information.

Based on the coordinate position of the three-plane connection point on the image, the labels of the three planes constituting the three-plane connection point are determined from the label image.

Specifically, surrounding pixel values are checked clockwise starting from an arbitrary pixel around the pixel at the connection point. The first three types of labels that appear at this time are the labels that constitute the three-sided connection point. This allows you to know the labels of the faces that make up the connection points. Next, it is determined which faces constitute which boundary lines, and the labels of the three faces at each connection point are expressed in the form of a list of three terms. Compare the faces that make up adjacent connection points.

For example, when comparing the lists of connection point J3 and its adjacent connection point J2, the common planes are A, and A3. From this, it can be seen that either of the surfaces Al and A3 constitutes the boundary line J2J3J or the boundary line J2J3J7. Similarly, when comparing the list with the adjacent connection point J6, one of the planes AA and the boundary line J2 Jz J
It can be seen that this is the surface forming the boundary line Jb Ja J7.

From these two results, surfaces A+, A3, and As are the boundary lines J2 Jll, Jll, J2 J3 J, respectively.
t, it can be seen that it constitutes JbJ3J7.

In general, the boundary line cad formed by plane B at connection point a
If (c, d are the labels of the connection points) are expressed as (B cd), for example, the connection point J3 will be as shown in Fig. 11 (c+).This will be expressed for all connection points and as three-sided connection point information. Hand over to the next process.

FIG. 11 (dl is an example diagram of three-sided connection point information.

The information about the connection point J is that the connection imaginary name is J3, and the surface configuration is that A1 is the boundary JzJz and A3 is J2J? , As has Jb J7.

FIG. 13 is an embodiment diagram of the structuring section of the present invention. In the figure, 74 is a straight line approximation part, 75 is a T-shaped running part of the connecting point, and 76 is a part for determining the overlap of three planes at the l connecting point. FIG. 14 is a functional block diagram of the structuring section of the present invention. When the flow starts at S77, first 378
It is determined whether the three-sided connection point information has been entered manually, and if it has not been entered, the system is in a waiting state. If it is entered,
At step 379, a straight line is drawn, and at step S80, a T-shaped chamber is performed at the connection point, and at step S81, the overlapping relationship of the three surfaces at one connection point is determined. Then, a two-sided relationship is searched for in S82, and a multi-sided relationship is searched for in S83. This processing first requires linear approximation processing.

The boundary line extending from the connection point is approximated as a straight line based on the boundary information received from the previous process, and a straight line extending directly from the connection point is determined. For example, the straight line approximation method is Duda and Hart's Spli.
The tting method is used.

FIG. 15 is a functional block diagram of (1) linear approximation update processing of the present invention. When the flow starts in S84, i is initialized to 1 in S85, and in S86 it is checked whether i has reached the number of labels at the connection point (MAX) + 1. If it has not reached, in S87, Ji and It is checked whether all the boundary lines between adjacent connection points have been processed, and if they have been processed, i is incremented and the process returns to S86.

If it has not been processed, the process goes to S88, where one of the boundaries between Jl and the adjacent connection point is approximated by a straight line.

FIG. 16 is an example of the linear approximation method. Line segment J, J
, so that the connection point J3 is on the line segment that connects the point J2 on the perpendicular line of , and the point Jl on the perpendicular line of the line segment J3J6, and further 1.
With J3 on the perpendicular line of J2J2, straight line J3
J'', J3 J', J3 J7 are obtained.

FIG. 17 is a functional block diagram of the linear approximation method used in the present invention. In this straight line approximation method, a straight line approximation of the boundary line between Jl and the adjacent connection point 33 is obtained. S8
When the flow starts at step 9, the equation of the straight line between J and J is found at 390, and the point farthest from the straight line between Jl and 33 is found from the boundary line information between J and 38 at S91, and the point that is farthest from the straight line between J and 33 is found. Let the distance be L. If L is less than or equal to the threshold in S92, J is set to JIl' in S93, but if L is greater than or equal to the threshold, j is replaced with J8 in 394, and the process returns to S90.

∎ Once the linear approximation is completed, the flow changes to ∎ the T-shaped chamber at the connecting point and ② the overlapping relationship between the three sides of that point.

FIG. 18 is a functional block diagram for deriving the overlapping relationship between the T-shaped chamber and the three sides of the connection point of the present invention. It's something you can buy. When the flow starts in S89, the equation of the straight line between J and 38 is determined in 390, and in S91, the point farthest from the straight line between J and 33 is determined from the boundary line information between J and 38, and the Let the distance be L. Then, if L is less than the dark value in S92, J is set to J % in S93, but if it is more than the threshold value, j is replaced with J8 in 394, and the process returns to S90.

∎ When the linear approximation is completed, the flow changes to ∎ the T-shaped window of the connection point and ③ the overlapping relationship between the three sides of that point.

FIG. 18 is a functional block diagram for deriving the overlapping relationship between the T-shaped window and the three sides of the connection point according to the present invention. When the flowchart starts in S95, i is initialized to 1 in 396, and 397
If i is not the number of labels at the connection point + 1, then 898 is entered. Here, J obtained by linear approximation, 3 straight cotton J from
,J,'.

The angle (α
, β, y), and in S99, the largest angle among them is found. Then, the connection point sequence that forms the largest angle at 5100 is determined. Furthermore, in step 5101, the upper surface and the lower surface are determined by combining the three surface connection point information. Then, in step 5102, 1 is incremented, and the process returns to S97.

Using the results of the straight-line approximation, find the angle between adjacent boundary lines at the connection point. Within those angles πrad(
= 180°) and select the 2 that makes up that angle.
Find the boundary line to which the straight line belongs. This boundary line belongs to the upper surface.

FIG. 19 is an embodiment of a T-shaped window at the connection point of the present invention. Two straight lines at the connection point J3, J3J2 and JffJ'
, 2 straight lines J3 J2. J3 J? and 2 straight lines J3 J'
, J3J7 are respectively aa2+23, and since al is closest to πrad, the boundary cotton = Iz
-Js jb is the boundary line belonging to the top surface.

FIG. 20(a) and (bl) are embodiment diagrams showing the overlapping of three surfaces at one connection point of the present invention.

As shown in FIG. 4A, the label of the top surface is obtained from the three surface connection point information received from the previous process and the boundary line belonging to the top surface. For example, it can be seen that the surface A+ is the upper surface of the surfaces An and As. The relationship between surface A3 and surface A is unknown at this point. This information is expressed in the form of a list as shown in FIG. 20 (bl) and transferred to the next process as T-type connection point information 105.

Next, the overlapping relationship between the two surfaces is determined based on the relationship between each connection point.

FIG. 21 is a configuration diagram of the surface relationship search processing section of the present invention. In the same figure, 103 is ■Search for relationships between 22 surfaces, 104
■ is the inter-surface relationship search section.

FIG. 22 is a diagram showing an example of searching for a relationship between two surfaces according to the present invention. In (a) of the same figure, (
) on the leftmost side (A3, AI, A6).

As, ...) indicates that it is above the other two surfaces. Also, in the same figure (bl is a tree structure graph expressing this T-shaped connection point information, the same labels are connected by thin solid lines. If you remove the overlap of these labels and organize them, the same figure (C)
become. Here, thick arrows indicate overlapping relationships, and dotted lines indicate unknown relationships.

In this way, all information is extracted for each surface from the T-type connection information, and the top surface of each surface is determined, as well as surfaces whose relationship is unknown even though they are in an adjacent relationship. in particular,
As an output, two-surface relationship information shown in FIG. 22(d) is prepared in which each surface is associated with the upper surface of that surface and adjacent surfaces with unknown relationships. At this time, all initial values are NUL
It is L. For example, the top surface of A2 is A3, and if the relationship is unknown, it is NUL.
It becomes L. One item (ie, the top surface) of the three-item list of T-type connection information is registered in the second and third items of the two-plane relationship information. After the registration for the top surface is completed, for each surface,
If that surface is not registered as an upper surface among adjacent surfaces that are not registered in the registered upper surface, that adjacent surface is registered in the column of unknown relationship.

For example, surface A has an unknown relation to surface A4.

FIG. 23 is a functional block diagram of the search for the relationship between two surfaces according to the present invention. When the flow starts at 5106, 2
Set the frame for inter-plane relationship information. That is, for each surface, a column for describing the upper surface (upper surface column) and a column for describing adjacent surfaces whose relationship is unknown (unknown relationship column) are provided, and NULL is set as an initial value in each column. At 8108, it is determined whether all the T-type connection point information has been used, and if it has not been used,
At 5109, the surface of the first term is entered in the upper column of the relationship information between two surfaces in the surface of the second term of the T-type connection point information. and,
Move to 5110, check whether the 1st item is entered in the unknown relationship column, and if it is, 5111
Delete the first item from the unknown relationship column and enter 5111. If it is not filled in, skip 5111 and enter 5.
Enter ill'. In 8111', it is checked whether or not the surface of the third term is entered in the top column or the unknown relationship column in the relationship information between two planes of the surface of the second term. If it is filled in, skip 5ill'', if it is not filled out, skip 5I.
Enter LL" 1. Fill out the 3rd column in the Unknown relationship column. 5
Enter 112. In step 5112, in the third term of the T-type connection point information, the first term is entered in the top field using 1.2 inter-plane relationship information. Then, move to 5l13, check whether the first item is entered in the unknown relationship column,
If not, the process returns directly to 3108, and if it has, the first term is deleted from the unknown relationship column in 5l14, and then the process moves to 8114'. In step 5114', it is checked whether or not the surface of the second term is entered in the top surface column or the relation unknown column in the relationship information between the two surfaces of the surface of the third term.

If it is filled in, skip 5114'', and if not filled in, go to 5114'', enter the second term in the unrelated column, and move to 8108. Next, processing for searching for relationships between entire surfaces is started.

FIG. 24 is a diagram showing an embodiment of the inter-surface relationship search process of the present invention. For example, suppose that in the embodiment diagram of FIG. 22, the relationship between the surface A4 and the surface A is unknown. In Figure 24, (
(a) is a tree-structured graph resulting from searching for relationships between two surfaces. When each relationship is propagated to all surfaces and surfaces with unknown relationships are integrated, the graph shown in (b) is obtained. Then, as shown in Figure 24 (C1), the top surface of one surface is registered all at once.However, two overlapping surfaces (Ae) are registered as one surface.In this way, the relationship information between the two surfaces The information that has been corrected is called inter-surface relationship information.

FIG. 25 is a functional block diagram of the polyhedral relationship search process of the present invention. When the flow starts in 5114, it is first checked in 5115 whether all the relationship information between the two sides has been used. If not, the process moves to 8116, and it is checked whether the relationship unknown column is NULL. If NULL, return; if not, enter 5117, combine the faces in the unknown relationship column as one face, and enter the OR of the top faces of each face in the top face column. Then, the unknown relationship column for the surfaces grouped together in 8118 is set to NULL. Furthermore, in step 5119, the surface that was in the upper surface field is deleted from the relationship information between the two surfaces, and the process returns to step 5115.

Finally, the non-hidden surface extracting section 27 shown in FIG. 2A searches for the inter-surface relation information from the structuring section 26,
Search for a surface that is ULL (non-concealed surface).

If such a non-concealed surface is found, the position of the surface is transferred to the sensor positioning section 24. If multiple non-obscuring surfaces exist, they are transferred sequentially. Then, the sensor positioning unit 24 determines that if the arm control unit 2o-b is in the completed state,
The sensor is aligned with the position of the non-concealed surface stored in the buffer and operated.

When the data in the buffer runs out, the area dividing unit 2
5 to capture an image from the television camera 23.

The depth measurement section 21 converts the depth obtained from the sensor (distance from the sensor to the object) into a distance from the arm to the object, and transfers it to the arm control section 20-b.

The arm control unit 20-b is located at the non-concealing surface position (X).

Move the arm according to the Y direction) and depth (Z direction),
Grab an object on an unconcealed surface and move it to a designated location. after that,
The end of the operation is notified to the sensor positioning unit 24.

〔Effect of the invention〕

In the present invention, compared to conventional methods, images can be characterized using a simpler algorithm that extracts the overlapping relationship between surfaces. That is, the present invention can easily extract the position of the non-obscuring surface without depending on the object, and extracts the overlapping relationship of the surfaces from the T-shaped connection point where the boundary lines of the three surfaces intersect using a method to create an image. can be characterized three-dimensionally. and,
If the above method is implemented in an object position measuring device, there will be a sensor positioning section that positions the sensor based on the data output from the device, a depth measuring section that transfers the information received from the sensor to the arm, It is possible to construct a pin picking system consisting of an arm control section that moves a robot arm based on three-dimensional position information obtained from each section.

[Brief explanation of the drawing]

Fig. 1 is a block diagram of the present invention, Fig. 2A is a block diagram of a bin picking system according to the image analysis method of the present invention, Fig. 2B is an explanatory diagram of image structuring processing of the present invention, and Fig. 3 is a block diagram of the present invention. FIG. 4 is a processing flowchart of the area division section of the present invention; FIG. 5 is a block diagram of the structuring section of the present invention; FIG. 6 is a processing flowchart of the structuring section of the present invention; Fig. 7 is a block diagram of the 3-plane connection point extracting unit, Fig. 8 is an example of an image area connected by 3 boundary lines, Fig. 9 is a processing flowchart of template matching, and Fig. 10 is a flowchart of the template matching process. Flowchart of boundary line tracking processing, 11th
Figure fa) is an example diagram of the boundary line, Figure 11 (bl is the label image near the pixel, Figure 11 (C
) is a diagram showing the processing with labels on the three sides, Figure 11 (dl is an example diagram of connection point information on the three sides, Figure 12 is a flowchart of the process with labels on the three sides constituting the connection points, and Figure 13 is a diagram showing the process). 14 is a functional block diagram of the structuring section of the present invention. FIG. 15 is a functional block diagram of the linear approximation update process of the present invention. FIG. 16 is a straight line. An example diagram of the approximation method, FIG. 17 is a functional block diagram of the linear approximation method used in the present invention, FIG. 18 is a functional block diagram for determining the T-type of the connection point and deriving the overlapping relationship of three surfaces according to the present invention, FIG. 19 is an example diagram of the T-type determination of a connection point according to the present invention, FIG. 20 is an example diagram showing the overlap of three surfaces at one connection point according to the present invention, FIG. Block diagram of surface relationship search processing unit, No. 22
The figure is an example diagram of the dihedral quantity relationship search of the present invention, Fig. 23,
, are functional block diagrams of the two-plane relationship search of the present invention, FIG. 24 (at, fb), and (c) are an example diagram of the full-plane relationship search process of the present invention, and FIG. 25 is the multi-plane relationship search process of the present invention. Functional block diagram of relationship search processing, Fig. 26(a), (bl is an explanatory diagram of a conventional image analysis method for extracting the position of an object without obscurity, Fig. 27 is a conventional recognition method by matching with a model) 12... Area dividing means, 13... Structuring means, 14... Non-obscured surface extraction means, 15... Area labeling means, 16... Boundary extraction means, 17. ... T-type connection determination means, 18... conversion means.

Claims (1)

[Claims] 1) Region dividing means (12) for dividing a target region of an image into a plurality of regions having the same properties; A structuring means (13) for converting into information including the context of a plurality of surfaces, and a surface extraction means (14) for extracting a desired surface based on the output of the structuring means (13). Featured image analysis method. 2) A region dividing means (12) that divides the target area in the image into a plurality of planes and extracts the boundaries of each plane, extracts a T-shaped connection point where three boundaries intersect among the intersection points of the boundaries, and After determining the overlapping relationship between the three surfaces at the T-shaped connection point, the relationship at the T-type connection point is converted to the overlapping relationship between two surfaces, and the overlapping relationship between the two surfaces is then converted to the overlapping relationship between all the surfaces existing in the image. a structuring means (13) for searching the surface relationship of the entire image based on the overlapping relationship between the two surfaces; An image analysis method characterized by comprising a non-concealing surface extraction means (14) for extracting surfaces that do not overlap with each other. 3) The area dividing means (12) includes: means (28) for removing noise from the image; texture analysis means (29) for determining the same texture area of the image from which the noise has been removed; and a boundary line of the same texture area. It consists of a means for extracting (30) and a means for attaching the same label to the same texture area (31), which divides the image into areas with the same texture, determines the boundaries of the areas, and attaches a label to each texture area. The image analysis method according to claim 1, characterized in that: 4) The structuring means (13) is a means for linearly approximating the boundaries of each region obtained from the region dividing means (12) to extract three-plane connection points where three planes overlap.
39); means (39) for extracting T-type connection point information indicating a T-type connection relationship at the three-plane connection points; and extracting T-type connection point information indicating the T-type connection relationship at the three-plane connection points. a T-type connection determination means (40) for searching for a sequential relationship between two surfaces based on the T-type connection point information; 2. The image analysis method according to claim 1, further comprising an inter-surface relationship searching means (104) for determining a vertical relationship between the entire surfaces by determining between the surfaces. 5) Area labeling means (1) that divides the image of the object into planes that are areas of the same texture and labels each area.
5), and a boundary extraction means (16) that extracts boundaries from surfaces with the same label.
), a T-type connection determination means (17) for extracting a T-type connection point where three boundaries intersect from among the intersection points of the boundaries, and after determining the overlapping relationship of the three planes at each T-type connection point, Conversion means (1
8), a surface relationship search means for propagating the overlapping relationship between the two surfaces to all surfaces existing in the image and searching for the vertical relationship of the surfaces in the entire image; and concealment based on the output of the surface relationship search means. 1. An object detection device comprising: a non-concealing object extracting means for extracting an object without a hidden object; and a means for detecting the position of the non-concealing object and the distance to the object based on the output of the non-concealing object extracting means.