JPH04259076A - Method and device for recognizing object - Google Patents

Abstract

PURPOSE:To easily recognize an object provided with an invisible part by moving a photographing system corresponding to the arranging status of the object in a method and device for recognizing object to be photographed which discriminates the object by analyzing an image acquired by an image pickup means with a computer. CONSTITUTION:In an object recognizing device which discriminates the shape of the object to be photographed by analyzing image information obtained by photographing with an image pickup machine 12, the photographic position of insufficient image information is decided based on comparison between the image information of a photographed image and knowledge information with respect to a known object, and the image pickup machine 12 is moved according to the information of the photographic position, and photographing is performed from the photographic position where the image information of the object to be photographed is insufficient, and the insufficient image information is supplemented, thereby, the shape of the object to be photographed can be discriminated.

Description

[Detailed description of the invention]

0001

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an object recognition method and an object recognition apparatus for identifying a photographed object by analyzing an image obtained by an imaging means using a computer.

0002

2. Description of the Related Art Conventionally, as an object recognition device for discriminating images captured by an imaging means, there has been a device that uses only a single still image taken with the imaging system fixed in position. In such a device, the apparent shape of an object changes depending on the viewing direction. Therefore, in a device that discriminates an object by viewing it from only one direction, the apparent shape of the object when viewed from all around is stored so that the object can be discriminated from any direction. When recognizing a photographed object, the object is determined by finding a shape similar to the obtained image from among the stored apparent shapes.

[0003] There has also been an apparatus that performs discrimination using a plurality of still images taken by moving an imaging system. In the case of this device, the position to which the imaging system will be moved is determined in advance, images are taken at all of the determined positions, and a shape similar to the apparent shape obtained from each movement position is captured.
It searches among the memorized apparent shapes and identifies objects based on the combination of results.

0004

SUMMARY OF THE INVENTION In the conventional technique described above, when the posture of the imaging system is fixed, the back side of the object cannot be seen. Also, some objects in the image may block other objects. If such a state is referred to as "hiddenness," when the object is hidden, it may not be possible to distinguish the object.

[0005] Hiding cannot be avoided with conventional devices that use only one still image. Furthermore, even in a device that uses a plurality of still images taken from different positions, hiding cannot be completely avoided if the moving position is set in advance. Therefore, in any device,
Hidden objects may occur, and depending on the placement of objects, it may be impossible to distinguish between them.

[0006] When this occlusion occurs, in order to try to distinguish it from the remaining parts, complicated processes such as estimating the object using information about the shooting situation and the shooting location are performed in order to extract as much discrimination information as possible from the limited image information. processing is required. Moreover, this requires a large amount of storage capacity and calculation amount.

[0007] Since the photographing position is set in advance, it is not possible to control the direction in which the object is observed, and it is necessary to memorize how it looks from all directions, thus requiring a large amount of storage capacity. In addition, when the appearance from a certain viewpoint is complicated, the storage method within the computer becomes complicated, and there are many problems such as a large amount of calculation is required to handle it.

[0008] The present invention is intended to solve the above-mentioned problems in the conventional technology, and makes it possible to easily recognize hidden objects by moving the photographing system according to the arrangement of the objects. The object of the present invention is to provide an object recognition method and an object recognition device.

[0009]

[Means for Solving the Problems] In order to recognize hidden objects, the present invention uses image information obtained by photographing with an imaging device 12 in an object recognition method, as shown in FIG. An object recognition device that analyzes and determines the shape of a photographed object determines the photographing position of the missing image information based on a comparison between the image information of the already photographed image and the knowledge information about the known object, and then images according to the information on this photographing position. Machine 12
The present invention is characterized in that the shape of the object is determined by moving the object, taking an image from a shooting position where image information of the object is lacking, and replenishing the missing image information.

[0010] In addition, an object recognition device that executes this method includes a knowledge information storage section 11 that stores knowledge information regarding a known object, an image pickup device 12 that obtains image information by photographing an object, and a knowledge information storage section 11 that stores knowledge information about known objects; A movable device 13 that can be attached to change the photographing position arbitrarily, an image analysis unit 14 that analyzes image information and extracts apparent information of the photographed object, and a device that acquires image information necessary for object recognition. an object recognition unit 15 that determines and outputs the coordinate information of the photographing position by the imaging device 12 and recognizes the object by comparing the image analysis result of the photographed object with knowledge information; and the movable device according to the coordinate information of the photographing position. movable device control unit 16 that controls 13;
It is characterized by having the following.

[0011]

[Operation] With this configuration, in the object recognition method, the photographing position of the missing image information is determined based on the comparison between the image information of the already photographed image and the knowledge information about the known object, and the determined photographing position is determined. The imaging device 12 is operated according to the position information to image the object from the shooting position where the image information was missing, and to image the hidden surface due to the arrangement of the object to capture the missing image information. After obtaining the information, the shape of the object is determined.

[0012] In the object recognition device, the movable device 13 is operated to point the image pickup device 12 at the object to take an initial photograph, and the apparent information of the photographed object is extracted from the obtained image information by the image analysis section. The object recognition section 15 compares and checks the apparent information of the object extracted and obtained by the object recognition section 14 with the knowledge information about the known object from the knowledge information storage section 11. Coordinate information for acquiring the image information necessary for recognition is determined, the movable device 13 is operated via the movable device control unit 16, the photographing position is changed to the required orientation, and the subject is photographed again by the image pickup device 12. The object is photographed, the supplemented image information is added to the object recognition unit 15, and the apparent information of the object and the knowledge information about the known object are again compared and verified to recognize the object.

[0013]

[Embodiment] In the following embodiment of the present invention, a case will be described in which a cup with a handle is recognized.

FIG. 2 is a diagram showing a robot arm in this embodiment. Here, 21 is a video camera that photographs objects. 22 is a six-axis robot arm, a video camera 21 is attached to the tip, and the video camera 21 is directed in any direction.

FIG. 3 is a block diagram showing an embodiment of the apparatus. 23 is a processing device that analyzes image information obtained by photographing a subject (not shown) with the video camera 21 and determines what the subject is. If this processing device 23 does not obtain enough image information for object determination, the robot arm 22 is operated to change the direction of the video camera 21 with respect to the object and take images to obtain the images necessary for object determination. Information can be supplemented.

The processing device 23 includes an image analysis unit 24 that analyzes image information, an object recognition unit 26 that determines the subject by comparing the image analysis results with knowledge information about the object stored in the memory 25, and If there is insufficient image information for determination, the coordinate information of the necessary shooting position is sent to the object recognition unit 2.
6 and obtains a control signal for the robot arm 22.

The image analysis section 24 extracts the apparent information of the object observed in the image by examining the color information of the image obtained from the video camera 21, and provides the analysis result to the object recognition section 26 at the same time. Partial results from the object recognition unit 26 are used for analysis processing. In order to investigate color information, statistics of brightness, saturation, and hue are calculated, as well as spatial differential values and spatial integral values. Furthermore, the spatial distribution of points exhibiting specific colors, differential values, and integral values is investigated geometrically. The analysis process changes the process applied to the image depending on the object to be recognized and how it looks, and the analysis process is applied only to necessary and sufficient parts of the image. In addition, partial results of object recognition may be used to select an analysis method and decide which part to apply.

As shown in FIG. 4, this image analysis unit 24 includes an image memory 31 for storing images obtained from the video camera 21, and a brightness histogram, average, and standard deviation for the whole or part of the image. A color information statistics calculation unit 32 that calculates the color information, an area division unit 33 that binarizes an image using a certain threshold as a boundary, and then performs degeneration, dilation, dilation, and degeneration to remove small areas; A straight line and curve detection unit 34 detects straight lines and curves as a sequence of fitted partial points by applying straight lines or curves to the area, and a straight line and curve detection unit 34 detects straight lines and curves as a matched partial point sequence, and a straight line and curve detection unit 34 that checks the geometric relationship between the straight line and curved shape of the area outline and determines whether the area is a quadrilateral. A region geometric shape determination unit 35 determines the geometric shape such as whether it is a circle or a circle, a region centroid calculation unit 36 calculates the center of gravity based on the coordinate information of the region constituent points, and a region A geometric figure center calculation unit 37 that calculates the coordinates of the center of the figure by using the coordinates of each point constituting the figure, or by using the coordinates of the start and end points of each side for a quadrilateral, and the It also includes a region color discrimination section 38 that discriminates the color of the region from point color information and its average value.

Memory 25 stores knowledge information regarding known objects. The memory 25 includes a storage section 41 that stores knowledge about the three-dimensional shape of the object, a storage section 42 that stores knowledge about the apparent shape of the object, and a storage section 43 that stores knowledge about the color of the object's surface. . For example, the shape of the cup when viewed from above, below, side, diagonally, etc., and the color of the bottom, side, and interior of the cup are stored.

The object recognition section 26 determines, based on the results of the image analysis section 24, what kind of object is being viewed and from which direction. If the object cannot be recognized due to lack of image information at the time of determination, determine what information is missing and how to change the posture (or direction) of the video camera 21 in order to acquire that information. The coordinate information of the new imaging position is then given to the arm control unit 27. Whether the changed imaging position is correct or not is determined by predicting the image that will be acquired from the new imaging position based on the partial results of object recognition, and comparing it with the image that was actually acquired after changing the imaging position. Verify.

As shown in FIG. 5, the object recognition unit 26 includes a collation unit 45 that collates knowledge about the known object with the image analysis results, an attitude determination unit 46 that determines the imaging orientation, and a system that collects image information in object recognition. It includes a missing image determining unit 47 that determines whether there is a shortage, and a position determination verification unit 48 that determines or verifies the moving position of the video camera 21.

The collation unit 45 collates the shape information of circles and rectangles obtained by the image analysis unit 24 and the color information of these figures with the knowledge information stored in the memory 25 to determine whether the observed object is determine what it is. For example, matching the appearance of the cup and the color of its surface with knowledge information.

[0023] The attitude determining unit 46 determines the attitude of the video camera 21 with respect to the object based on the result of the matching unit 45. For example, the relative positional relationship between the cup and the video camera 21 is determined.

The missing image determining unit 47 determines, based on the results of the matching unit 45, what information is missing in order to recognize the object. For example, it is determined whether the acquired images are sufficient to determine the positions of the bottom surface and handle of the cup.

The position determination verification section 48 determines, based on the partial results of the posture determination section 46 and the missing image determination section 47, where to move in order to acquire the missing image information. Then, the moving position is verified by predicting an image to be acquired from the new moving position and comparing it with the image obtained by actually moving. For example, if both the bottom and the inside of the cup cannot be seen, it is determined how the posture should be changed and the changed position is verified.

The arm control unit 27 converts the coordinate information of the photographing position given from the object recognition unit 26 into a control signal for the robot arm 22 and controls the robot arm 22 to control the video attached to the robot arm 22. The attitude (or direction) of the camera 21 is changed.

The processing procedure of the embodiment configured as described above is as follows:
As shown in FIG. 6, first, an imaging system consisting of a video camera 21 and a robot arm 22 is moved to a preset position and a subject is photographed (step 51). Color information is extracted from the image information of the photographed subject, and the brightness, saturation, hue statistics, spatial differential value, and spatial integral value are calculated for the color information, and the points that indicate a specific color, , for points showing specific spatial differential values and spatial integral values, we examine the spatial distribution in the image and extract the pattern of the surface observed in the image and the geometric information of the figure observed in the image. Then, based on this information, the lines observed within the field of view, the color and pattern of the surface, and the position and orientation in the three-dimensional space are calculated (step 52). Based on this result, what is the subject?
Determine how it is arranged (step 53
). If the object can be identified, the process ends.

If the object cannot be identified, based on the results of partial recognition in step 53, the parts that were hidden at the photographed position in step 51 become visible, and the object can be identified. Calculate candidate postures of the imaging system that will allow information to be obtained. Then, from among these candidates, the posture that allows the most abundant information about the subject to be obtained by changing the posture is determined. At the same time, the image taken after the change is predicted (step 54). According to the obtained orientation candidates, the orientation of the imaging system is changed and images are taken again (step 55). The image obtained by photographing again is compared with the image predicted in step 54 (
Step 56). If the photographed image and the predicted image do not match, the process returns to step 54, the next candidate posture is determined, and the image after photographing is predicted. If the photographed image and the predicted image match, the process returns to step 52 and processes necessary to identify the subject are performed, and these processes are repeated until the subject can be identified.

In the embodiment configured as described above, for example, when the cup 61 with the handle 61a shown in FIG. I will store it in advance. When the cup 61 is placed on a desk, even if it is not placed in the usual way, and even if it is placed upside down or fallen down, if you look directly down from above the cup 61, if you look down from the top of the cup 61, Does it look like a shape consisting of a circular part and a small rectangle as shown in (a), or as shown in Fig. 8(b)?
It looks like a shape with a handle 61a attached to a rectangular part, or a shape where these parts are rotated within a two-dimensional plane.

In object recognition in this case, first, the video camera 21 is moved above the desk so as to look directly below the desk, and the cup 61 is photographed. From the thus photographed image, the portions having colors other than the cup 61 are removed as the background. To remove this background, the color information statistics calculation unit 32 calculates a brightness histogram for the entire image, determines a brightness value that divides the histogram into two, and then the area division unit 33 uses this value as a threshold to create a binary value. The cup 61 and the background are separated, and the background is removed from the image.

[0031] Is the outline of the remaining part as shown in Fig. 8(a) or as shown in Fig. 8(b)?
Examine the apparent shape. For this shape determination, the straight line and curve detection unit 34 is used to extract a straight line or curved line from a sequence of points corresponding to the contour of the portion from which the background has been removed. Next, it is determined whether the contour is a rectangle or a circle based on the obtained straight line or curved line by the region geometric shape determination unit 35.

In the case of the shape shown in FIG. 8(a), the area color determination unit 38 checks whether the color of the circular part of the area from which the background has been removed is the previously stored color of the bottom of the cup or the color of the interior; Based on the result, the position of the bottom surface of the cup 61 is determined. Next, the center of gravity calculation unit 36 calculates the center of gravity of the area from which the background has been removed, and the center of geometric figure calculation unit 37 calculates the center of the circular portion of the area from which the background has been removed. Then, as shown in FIG. 9, the position of the handle 61a is determined from the direction of deviation between the center of gravity of the area and the center of the circular portion.

In the case of a shape as shown in FIG. 8(b), the region geometric shape determination unit 35 determines the positions of the long and short sides of the rectangle and the position of the rectangle on the image with respect to the rectangular part of the region from which the background has been removed. Calculate the slope of the rectangle. Based on the result, the video camera 21 is moved so that the cup 61 can see the surface perpendicular to the video camera 21. That is, FIG.
As shown in , first, the video camera 21 is tilted from the initial position to the two short sides of the rectangle to change to the movement position 1 or the movement position 2. If the inside of the cup 61 becomes visible, it is predicted that a rotated image of FIG. 7 will be obtained. Then, check whether the image obtained by tilting the video camera 21 is the predicted image, or whether the newly visible color is the color of the bottom or inside of the cup 61 that has been stored in advance. Examine and determine the position of the bottom of the cup 61. Next, the center of gravity calculation unit 36 calculates the center of gravity of the area from which the background has been removed, and the center of geometric figure calculation unit 37 calculates the center of the rectangle of the area from which the background has been removed. Then, as shown in FIG. 11, the position of the handle 61a is determined from the direction of deviation between the center of gravity of the area and the center of the rectangle.

In this way, the video camera 21 is attached to the cup 61.
By positioning the handle 61a above the cup 61 and looking down at the cup 61, the handle 61a of the cup 61 can be observed without fail. When the cup 61 is fallen down, the bottom surface cannot be seen at first, but by moving the video camera 21, the bottom surface of the cup 61 can be determined. Furthermore, such processing involves relatively simple image processing, such as determining the shape of an outline, calculating the center position of a rectangle, the center position of a circle, and the center of gravity of an area, and determining the color of an area. Since there is no need to judge the shape of the curve in order to recognize the handle 61a, the process can be easily performed in a short time.

[0035]

As described above, in the object recognition method of the present invention, the photographing position of the missing image information is determined based on the comparison between the image information of the already photographed image and the knowledge information about the known object. By operating the imaging device according to the information on the photographed position, the object can be imaged from the missing photographing position, the missing image information can be supplemented, and the shape of the object can be determined. It is possible to move to an imaging position according to the arrangement of the object, and avoid hiding the object. Therefore, for multiple images taken at different shooting positions,
It is only necessary to extract the minimum necessary discrimination information according to each photographing situation, and it is possible to reduce the amount of analysis processing and shorten the analysis time. Further, since missing image information can be supplemented for discrimination, the total number of image information representing the apparent shape of the object to be stored as knowledge information can be reduced.

In the object recognition device, the movable device 13 is operated to point the image pickup device 12 at the object to take an initial photograph, and the apparent information of the photographed object is extracted from the obtained image information by the image analysis section. The object recognition unit 15 compares and matches the apparent information of the object extracted and obtained by the object recognition unit 14 with the knowledge information about the known object from the knowledge information storage unit 11;
If information is insufficient, such as when the object is hidden, coordinate information for acquiring image information necessary for object recognition is determined, and the movable device 13 is operated via the movable device control unit 16 to change the shooting position to the required position. The object is photographed again by the imaging device 12 after changing its orientation, and supplementary image information extracted by the image analysis section 14 is added to the object recognition section 15.
Since the object can be recognized by comparing the apparent information of the object and the knowledge information about the known object again, the attitude of the imaging device 12 is determined and changed based on the information obtained by the discrimination process. It is possible to supplement the apparent information of the object until it becomes possible to distinguish the photographed object by photographing the photographed object, and it is possible to avoid an indistinguishable state due to the occurrence of the hidden object. In addition, since the movable device 13 can be operated to direct the imaging device 12 so that the hidden side of the object can be photographed, it is possible to handle multiple images at different photographing positions to the minimum required amount according to each photographing situation. It is only necessary to extract the discrimination information, and the analysis time can be shortened. Furthermore, since the optimum comparison surface can be photographed, the appearance of the object to be stored can be reduced, and the storage capacity can be reduced.

[Brief explanation of the drawing]

FIG. 1 is a diagram showing the principle configuration of the present invention.

FIG. 2 is a perspective view showing a robot arm with a camera attached in the example.

FIG. 3 is a block diagram of an example device.

FIG. 4 is a block diagram of an image analysis section in the embodiment.

FIG. 5 is a block diagram of an object recognition unit in the embodiment.

FIG. 6 is a flowchart showing the processing procedure in the embodiment.

FIG. 7 is a perspective view showing the cup when the handle in the embodiment is viewed from the right.

FIG. 8 shows the apparent shape of the cup in the example; (a) shows the shape of the cup when viewed directly below from above;
b) is when the cup is viewed from the side.

FIG. 9 is an explanatory diagram showing the position of the handle when the cup appears circular in the embodiment.

FIG. 10 is an explanatory diagram showing the movement of the camera when the cup falls down on the desk in the example.

FIG. 11 is an explanatory diagram showing the position of the handle when the cup appears rectangular in the embodiment.

[Explanation of symbols]

11 Knowledge information storage unit 12 Imaging device 13. Movable device 14 Image analysis department 15 Object recognition section 16　Movable device control section

Claims (2)

[Claims] Claim 1: In an object recognition device that analyzes image information obtained by photographing with an imaging device (12) and determines the shape of a photographed object, the image information of an already photographed image is compared with knowledge information about a known object. Based on this, determine the shooting position of the missing image information, move the imaging device (12) according to the information of this shooting position, take an image from the shooting position where the image information of the object was missing, and remove the missing image information. An object recognition method characterized in that the shape of a photographed object is determined by supplementing the . 2. A knowledge information storage unit (11) that stores knowledge information about a known object; an image pickup device (12) that photographs the object to obtain image information; a movable device (13) that can be changed arbitrarily; an image analysis unit (14) that analyzes image information and extracts apparent information about a photographed object; an object recognition unit (15) that determines and outputs the coordinate information of the photographing position by the imaging device (12) and recognizes the object by comparing the image analysis result of the photographed object with knowledge information; The movable device (
13) A movable device control section (16) for controlling the object recognition device.