JP3154501B2 - Object recognition method and object recognition device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an object recognizing method and an object recognizing apparatus for discriminating a subject by analyzing an image obtained by an image pickup means by a computer.

[0002]

2. Description of the Related Art Conventionally, as an object recognizing device for judging an image obtained by an image pickup means, there has been a device using only one still image photographed with the posture of an image pickup system fixed. In such a device, an object has a different apparent shape depending on the viewing direction. Therefore, an apparatus that determines an object by viewing the object from only one direction stores an apparent shape when the object is viewed from the entire periphery of the object so that the object can be determined from any direction. Then, when recognizing the object,
The object is determined by finding a shape similar to the obtained image from the stored apparent shapes.

[0003] Further, there has been an apparatus for discriminating using a plurality of still images taken by moving an imaging system. In the case of this device, the position to which the imaging system is moved is determined in advance, shooting is performed at all of the determined positions, and a shape similar to the apparent shape obtained from each movement position is obtained.
A search is performed from the stored apparent shapes, and the object is determined based on a combination of the results.

[0004]

In the above prior art, the back side of the object cannot be seen when the posture of the imaging system is fixed. Also, an object in an image may obstruct another object. If such a state is referred to as "hidden", the object may not be discriminated if occlusion occurs.

[0005] In a conventional apparatus using only one still image, occlusion cannot be avoided. Further, even in a device using a plurality of still images photographed from different positions, hiding cannot be completely avoided if the moving position is set in advance. Therefore, in any device,
There is a possibility that occlusion may occur, and it may not be possible to determine this depending on the arrangement of the objects.

[0006] When this occlusion occurs, in order to try to discriminate from the remaining portion, it is necessary to extract as much discrimination information as possible from limited image information, and to guess an object using information on a shooting situation and a shooting location. Processing is required. In addition, a large storage capacity and a large amount of calculation are required for that purpose.

Since the photographing position is set in advance, it is not possible to control the direction in which the object is observed, and it is necessary to store the appearance viewed from all directions. Therefore, a large storage capacity is required. On the other hand, when the view from a certain viewpoint is complicated, the storage method in the computer is also complicated, and there are many problems such as a large amount of calculation required to handle it.

An object of the present invention is to solve the above-mentioned problems in the prior art, in which a hidden object can be easily recognized by moving a photographing system according to an arrangement state of the object. It is an object to provide an object recognition method and an object recognition device.

[0009]

According to the present invention, in order to make it possible to recognize a hidden object, as shown in FIG. In the object recognition method that analyzes and determines the shape of the object, the shape and color of the image information of the object to be photographed are compared with the three-dimensional shape of the known object, the apparent shape, and the knowledge information of the color of the surface of the object. Based on whether the image information of the captured object is information of an image of what kind of object is viewed from which direction, and from the result of this determination, if the shape of the object can be determined, When the processing is completed and the object shape cannot be determined due to lack of image information at the time of determination, a shooting position for acquiring the missing image information is obtained, and the imaging device 12 is moved according to the information of the shooting position. The object Captured from the photographing position image information is missing, supplemented with image information missing, characterized in that to determine the shape of the object scene object.

Further, in the object recognizing apparatus that executes this method, a knowledge information storage unit 11 for storing knowledge information of a three-dimensional shape, an apparent shape, and a color of an object surface relating to a known object, and an image of the object to be photographed. Imager 12 that obtains image information by using
And a movable device 13 to which the imaging device 12 is attached to arbitrarily change a photographing position; and image information including color information from the imaging device 12, and an image of an apparent shape of the photographed object is analyzed. An image analysis unit 14 for extracting information; and an image information of the object based on the shape and color of the image information from the image analysis unit 14 and the knowledge information on the known object. It is determined whether the information of the image viewed from, from the result of this determination, if the shape of the object can be determined, the object recognition process ends,
If the object shape cannot be determined due to lack of image information at the time of determination, the coordinate information of the photographing position by the imaging device 12 for obtaining the missing image information necessary for recognition of the object is obtained and output. The object recognizing unit 15 repeats the acquisition of the missing image information, the comparison of the image information, and the verification until the missing image information supplemented and collated is verified and the shape of the object can be determined. And a movable device control section 16 for controlling the movable device 13 in accordance with coordinate information of a photographing position, wherein the shape of the object is determined.

[0011]

With this configuration, in the object recognition method, the shape and color of the image information of the photographed object and the knowledge information such as the three-dimensional shape, apparent shape, and color of the surface of the object relating to the known object are obtained. Based on the comparison, the image information of the photographed object is determined as to what kind of object is viewed from which direction, and from the result of the determination, the shape of the object can be determined. If so, end the process,
When the object shape cannot be determined due to lack of image information at the time of the determination, a shooting position for acquiring the missing image information is obtained, and the imaging device 12 is moved according to the information of the shooting position, and the object to be shot is moved. An image is taken from the shooting position where the image information was insufficient, and the missing image information is supplemented to determine the shape of the object.

In the object recognizing device, the movable device 13 is operated to initially shoot the image pickup device 12 at the object to be photographed, and an apparent image including color information of the object to be photographed is obtained from the obtained image information. Image information of the shape of the image
4, the obtained image information of the object and the three-dimensional shape, the apparent shape, and the knowledge information of the color of the surface of the object from the knowledge information storage unit 11 are stored in the object recognition unit 1.
5. The shape and color are compared and compared according to 5 to determine whether the image information of the object is information of an image in which the object is viewed from which direction, and from the result of this determination, the shape of the object Is determined, the process ends.
If the shape of the object cannot be determined due to lack of information such as occlusion, the coordinate information of the shooting position to obtain the missing image information necessary for object recognition is obtained.
Operate the movable device 13 via the movable device control unit 16,
The photographing position is changed to the required direction, the object to be photographed is again photographed by the image pickup device 12, and the missing image information is replenished to the object recognizing unit 15 so that the image information of the apparent shape of the object to be known is obtained. Acquisition of missing image information, and collation and verification of image information are repeated until the knowledge information on the object is compared and verified again to verify the shape of the object.

[0013]

DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following embodiments of the present invention, a case where a cup with a handle is recognized will be described.

FIG. 2 is a view showing a robot arm in the embodiment. Here, reference numeral 21 denotes a video camera, which shoots an object. Reference numeral 22 denotes a six-axis robot arm, to which the video camera 21 is attached at the tip, and turns the video camera 21 in an arbitrary direction.

FIG. 3 is a block diagram showing the apparatus according to the embodiment. A processing unit 23 analyzes image information obtained by photographing a subject (not shown) with the video camera 21 and determines what the subject is. When sufficient image information for determining the subject is not obtained in the processing device 23, the robot arm 22 is operated to change the direction of the video camera 21 with respect to the subject, and the image is captured. Information can be supplemented.

The processing device 23 includes an image analyzing unit 24 for analyzing image information, an object recognizing unit 26 for judging a subject by comparing the image analysis result with knowledge information about the object stored in a memory 25, When the image information is insufficient for the determination of the object, the coordinate information of the necessary photographing position is
And an arm control unit 27 for receiving a control signal of the robot arm 22 from the control unit 6.

The image analysis unit 24 extracts the apparent information of the object observed in the image by examining the color information of the image obtained from the video camera 21, and provides the analysis result to the object recognition unit 26. The partial result of the object recognition unit 26 is used for analysis processing. Investigation of the color information involves calculating the statistics of the brightness, saturation, and hue, and calculating the spatial differential value and the spatial integral value. Further, for points showing specific colors, differential values, and integral values, the spatial distribution is geometrically examined. The analysis process changes the process applied to the image depending on the object to be recognized and the appearance of the object. The analysis process is applied only to a necessary and sufficient portion in the image. The partial result of the object recognition may be used for selecting an analysis method and determining an applied part.

As shown in FIG. 4, the image analyzer 24 stores an image memory 31 for storing images obtained from the video camera 21 and a brightness histogram, an average and a standard deviation for the whole or a part of the image. A color information statistic calculation unit 32, a region dividing unit 33 that binarizes an image with a certain threshold value as a boundary, performs erosion, dilation, dilation, and erosion and removes a small region; On the other hand, a straight line or a curve is applied as a sequence of partial points, and a straight line, a straight line for detecting a curve and a curve detecting unit 34, and a geometric relationship between a straight line of an area contour and a shape expressed by a curve are examined. A geometric shape judging unit 35 for judging a geometric shape such as a circle or a circle; a center of gravity calculating unit 36 for calculating a center of gravity based on coordinate information of an area constituting point; Make up A geometric figure center calculation unit 37 for calculating the coordinates of the center of the figure, such as using the coordinates of a point, and using the coordinates of the start and end points of each side for a rectangle, and the color information of the constituent points for a specific area And an area color discriminator 38 for discriminating the color of the area from the average value.

The memory 25 stores knowledge information on a known object. The memory 25 includes a storage unit 41 that stores knowledge about the three-dimensional shape of the object, a storage unit 42 that stores knowledge about the apparent shape of the object, and a storage unit 43 that stores knowledge about the color of the surface of the object. . For example, the shape of the cup as viewed from above, below, laterally, obliquely, and the like, and the bottom, side, and inner colors of the cup are stored.

The object recognizing unit 26 determines what kind of object is viewed from which direction based on the result of the image analyzing unit 24. If the object cannot be recognized due to lack of image information at the time of determination, what kind of missing information is determined, and how to change the attitude (or orientation) of the video camera 21 in order to obtain the information It is determined whether or not it is good, and the coordinate information of the position where a new image is to be taken is given to the arm control unit 27. Whether or not the imaging position to be changed is correct is to predict the image acquired from the new imaging position based on the partial result of the object recognition, and to collate with the image acquired by actually changing the imaging position. Verify.

As shown in FIG. 5, the object recognizing section 26 includes a collating section 45 for collating knowledge about a known object and an image analysis result, an attitude judging section 46 for judging an imaging attitude, and image information for object recognition. The camera includes an insufficient image determining unit 47 for determining whether there is a shortage, and a position determination verifying unit 48 for determining or verifying the moving position of the video camera 21.

The collating unit 45 collates the shape information of the circle and the quadrangle obtained by the image analyzing unit 24 and the color information of those figures with the knowledge information stored in the memory 25, and determines the observed object. Determine what it is. For example,
Collation of cup appearance and surface color with knowledge information.

The posture determining section 46 determines the posture of the video camera 21 with respect to the object based on the result of the collating section 45. For example, the relative positional relationship between the cup and the video camera 21 is determined.

The missing image determining section 47 determines what information is missing for recognizing the object based on the result of the matching section 45. For example, it is determined whether the image acquired when determining the position of the handle and the bottom surface of the cup is sufficient.

The position determination verification unit 48 determines where to move to obtain the missing image information based on the partial results of the posture determination unit 46 and the missing image determination unit 47, Then, the image obtained from the new moving position is predicted, and the moving position is verified by comparing the image with the image obtained by actually moving. For example, when the bottom, inside, and both sides of the cup are not visible, how to change the posture is determined, and the changed position is verified.

The arm control unit 27 converts the coordinate information of the photographing position given from the object recognizing unit 26 into a control signal for the robot arm 22 and controls the robot arm 22 to thereby control the video signal attached to the robot arm 22. The posture (or direction) of the camera 21 is changed.

The processing procedure of the embodiment configured as described above is as follows.
As shown in FIG. 6, first, the imaging system including the video camera 21 and the robot arm 22 is moved to a position set in advance, and an object is photographed (step 51). Color information is extracted from the image information of the photographed subject, and brightness, saturation, hue statistics, spatial differential values, and spatial integral values are calculated for the color information, and a point indicating a specific color, and For points showing specific spatial differential values and spatial integral values, the spatial distribution in the image is examined, and the surface pattern observed in the image and the geometric information of the figure observed in the image are extracted. Then, based on the information, the line and surface colors and patterns observed in the visual field, and the position and orientation in the three-dimensional space are calculated (step 52). Based on this result, what the subject is,
Determine how they are arranged (Step 5
3). If the subject can be determined, the process ends.

If the subject cannot be determined, the part hidden and not visible at the position photographed in step 51 becomes visible based on the result of partial recognition in step 53, and the subject can be determined. A candidate for the orientation of the imaging system from which information can be obtained is calculated. Then, from those candidates, a posture that can obtain the richest information about the subject by changing the posture is determined. At the same time, the image photographed after the change is predicted (step 54). According to the obtained posture candidates, the posture of the imaging system is changed and the image is taken again (step 55). The image captured again is compared with the image predicted in step 54 (step 56). If the captured image does not match the predicted image, the process returns to step 54, where the next candidate posture is obtained, and the image after capturing is predicted. If the captured image and the predicted image match, the process returns to step 52, and performs the processing necessary for subject discrimination, and repeats these processes until the subject can be discriminated.

In the embodiment configured as described above, for example, when the cup 61 having the handle 61a shown in FIG. 7 is determined, the color of the side, the handle, the bottom, and the inside of the cup 61 is known. It will be stored in advance. If the cup 61 is placed on a desk and it is placed upside down instead of in a normal position, or if it is lying down, if it looks down from directly above the cup 61, FIG. It looks like a shape composed of a circular part and a small rectangle as shown in FIG.
It looks like a shape with a handle 61a attached to a rectangular portion as shown in FIG.

In the object recognition in this case, first, the video camera 21 is moved so as to look above the desk and directly below the desk to photograph the cup 61. A portion having a color other than the cup 61 is removed from the captured image as a background. To remove this background, a color information statistic calculation unit 32 calculates a brightness histogram for the entire image, determines a brightness value that divides the histogram into two, and a region division unit 33 uses the value as a threshold to perform a binary Then, the background is removed from the image by separating the cup 61 from the background.

It is checked whether the contour of the remaining portion has a shape as shown in FIG. 8A or a shape as shown in FIG. In this shape determination, a straight line or a curve is extracted from a point sequence corresponding to the outline of the portion from which the background has been removed, using the straight line and curve detecting unit 34. Next, it is determined from the straight line or the curve obtained by the region geometric shape determining unit 35 whether the contour is a quadrangle or a circle.

In the case of the shape as shown in FIG. 8A, the color discriminating section 38 of the area checks whether the color of the circular portion of the area from which the background has been removed is the color of the bottom of the cup or the color of the inside stored in advance. Based on the result, the position of the bottom surface of the cup 61 is determined. Next, the center of gravity of the region from which the background has been removed is calculated by the center of gravity calculating unit 36 of the region, and the center of the circular portion of the region from which the background has been removed is calculated by the center calculating unit 37 of the geometric figure.
Then, as shown in FIG. 9, the position of the handle 61a is determined from the direction of displacement between the center of gravity of the region and the center of the circular portion.

In the case of the shape as shown in FIG. 8 (b), the region geometric shape judging section 35 determines the position of the long side and the short side of the rectangle and the position on the image with respect to the rectangular portion of the region from which the background has been removed. Calculate the slope of the rectangle. Based on the result, the video camera 21 is moved so that the cup 61 can see a plane perpendicular to the video camera 21. That is, as shown in FIG. 10, first, the video camera 21 is tilted to the two short sides of the rectangle from the initial position and the moving position 1 or the moving position 2 is moved.
Change to If the inside of the cup 61 becomes visible, it is expected that an image obtained by rotating FIG. 7 will be obtained. Whether the image obtained by defeating the video camera 21 is the predicted image, or whether the color of the newly seen portion is the color of the bottom surface of the cup 61 or the color of the inside stored in advance. A check is made to determine the position of the bottom surface of the cup 61. Next, the center of gravity of the region from which the background has been removed is calculated by the center of gravity calculating unit 36 of the region, and the center of the rectangle of the region from which the background has been removed is calculated by the center calculating unit 37 of the geometric figure. Then, as shown in FIG. 11, the position of the handle 61a is determined from the direction of displacement between the center of gravity of the region and the center of the rectangle.

As described above, the video camera 21 is
And the handle 61a of the cup 61 can be always observed by looking down the cup 61 directly below. When the cup 61 has fallen, the bottom surface cannot be seen at first, but by moving the video camera 21, the bottom surface of the cup 61 can be determined. In addition, such processing is configured by relatively simple image processing, such as determination of the shape of a contour, calculation of the center position of a rectangle, the center position of a circle, the position of the center of gravity of a region, and determination of the color of a region. Since it is not necessary to judge the shape of the curve in order to recognize the handle 61a, the processing can be easily performed in a short time.

[0035]

As described above, according to the present invention, in the object recognition method, the shape and color of the photographed object are compared with the image information of the photographed object and the knowledge information on the known object. The image information is used to determine what kind of object is viewed from which direction. From the result of this determination, if the shape of the object can be determined, the process ends. If the object shape cannot be determined due to lack, the shooting position of the missing image information is obtained, and the imaging device is operated according to the obtained shooting position information,
Take an image of the object, supplement the missing image information,
The shape of the object can be determined, and the object can be moved to an imaging position according to the arrangement of the object, so that hiding of the object can be avoided. Therefore, it is only necessary to extract minimum necessary discrimination information according to each photographing situation from a plurality of images at different photographing positions, thereby reducing the amount of analysis processing and the analysis time. Further, since the missing image information can be supplemented and determined, the total number of pieces of image information representing the apparent shape of the subject to be stored as knowledge information can be reduced.

In the object recognition device, the movable device 13 is operated to initially shoot the image pickup device 12 at the object, and an image of the apparent shape of the object captured from the obtained image information. The information is extracted by the image analyzing unit 14, and the obtained image information including the color information of the object to be obtained and the knowledge information of the known object from the knowledge information storage unit 11 such as the three-dimensional shape, the apparent shape, the color of the surface of the object, To the object recognition unit 1
5. The shape and color are compared and compared according to 5 to determine whether the image information of the object is information of an image in which the object is viewed from which direction, and from the result of this determination, the shape of the object If the object shape can not be determined due to lack of information such as occlusion at the time of determination and the object shape cannot be determined, the shooting position for acquiring the insufficient image information necessary for object recognition , The movable device 13 is operated via the movable device control unit 16, the photographing position is changed to a required direction, the object is again photographed by the image pickup device 12, and the object recognizing unit 15 has a shortage. Replenishing the image information that has been, the image information of the apparent shape including the color information in the object to be compared and the knowledge information about the known object again,
This process can be repeated until the shape of the object can be determined. Therefore, the posture of the image pickup device 12 is determined based on the information obtained by the determination process, and the posture is changed. The apparent information of the object can be replenished until the object can be identified, and the indistinguishable state due to the hiding of the object can be avoided. In addition, since the movable device 13 is operated to direct the image pickup device 12 so that the hidden surface of the object can be photographed, the minimum necessary for a plurality of images at different photographing positions according to each photographing situation can be obtained. It is only necessary to extract the discrimination information of, and the analysis time can be reduced. Further, since the optimum comparison plane can be photographed, the appearance of the object to be stored can be reduced, and the storage capacity can be reduced.

[Brief description of the drawings]

FIG. 1 is a principle configuration diagram of the present invention.

FIG. 2 is a perspective view showing a robot arm to which a camera is attached in the embodiment.

FIG. 3 is a block diagram of an embodiment apparatus.

FIG. 4 is a block diagram of an image analysis unit in the embodiment.

FIG. 5 is a block diagram of an object recognition unit in the embodiment.

FIG. 6 is a flowchart showing a processing procedure in the embodiment.

FIG. 7 is a perspective view showing the cup when the handle in the embodiment is viewed to the right.

FIG. 8 shows the apparent shape of the cup in the embodiment, wherein (a) is a case where the cup is looked down from directly above the cup.
(B) is a case where the cup is viewed from the side.

FIG. 9 is an explanatory diagram showing the position of the handle in the case where the cup looks circular in the embodiment.

FIG. 10 is an explanatory diagram showing movement of a camera when a cup is falling on a desk in the embodiment.

FIG. 11 is an explanatory diagram showing the position of the handle in the case where the cup looks rectangular in the embodiment.

[Explanation of symbols]

 Reference Signs List 11 Knowledge information storage unit 12 Image pickup device 13 Mobile device 14 Image analysis unit 15 Object recognition unit 16 Mobile device control unit

Continuation of the front page (72) Inventor Shigeru Sasaki 1015 Uedanaka, Nakahara-ku, Kawasaki-shi, Kanagawa Prefecture Inside Fujitsu Limited (56) References Shokodo, 1990.10.30 Published: Artificial Intelligence Series, Volume 11, Volume 7 Distribution book “Robot Vision”, Chapter 7.3, pages 163-181

Claims (2)

(57) [Claims] We claim: 1. have you <br/> the object recognition method to determine the imaging device (12) analyzed Hiutsushi object shape of the image information obtained by photographing by the image information of the shooting object and shooting and Based on a comparison of the shape and color with the knowledge information of the three-dimensional shape, the apparent shape, and the color of the surface of the known object, the image information of the object to be photographed is from which direction the image is viewed. Information, and from the result of this determination, the shape of the object
If the shape can be determined, the process ends. If the image information is insufficient at the time of the determination and the object shape cannot be determined, a shooting position for acquiring the missing image information is obtained. Moving the imaging device (12) in accordance with the above, taking an image of the object from a shooting position where image information was insufficient, supplementing the missing image information, and determining the shape of the object. Object recognition method to be featured. 2. A knowledge information storage unit (11) for storing knowledge information of a three-dimensional shape, an apparent shape, and a color of an object surface relating to a known object, and an imager (12) for photographing an object to obtain image information. ), A movable device (13) to which the imaging device (12) is attached to arbitrarily change a photographing position, and image information including color information from the imaging device (12) is analyzed to obtain a photographed subject. An image analysis unit (14) for extracting image information of an apparent shape of the object; and the object to be captured based on a comparison of the shape and color between the image information from the image analysis unit (14) and the knowledge information on the known object. image information is determined what image information looking object from any direction, from the results of this determination, the
When the shape of the object can be determined, the object recognition process is terminated.
If the image information is insufficient at the time of the determination and the object shape cannot be determined , the image pickup device (12) for acquiring the insufficient image information necessary for recognizing the object. The coordinate information of the shooting position is obtained and output, and the supplemented missing image information is collated and verified, and the image of the subject is verified .
Acquisition of missing image information until the shape can be determined
And collation of the image information, the object recognition unit repeated verification (1
5) and a movable device control section (16) for controlling the movable device (13) in accordance with the coordinate information of the photographing position .
An object recognition device characterized by determining a shape .