JPH10232937A - Method and instrument for measuring meal tray - Google Patents

Abstract

PROBLEM TO BE SOLVED: To measure the three-dimension position of even a meal tray which has none of a featured pattern, a shape, and a mark to be measured and to stably take a measurement even if measurement conditions such as lighting conditions change. SOLUTION: Images of a meal tray 10 photographed by cameras 1a and 1b are stored in an object image memory 4a, and a correlation arithmetic means 5 performs correlation operation between the object images and reference images to find straight-line parameters of the edge of the tray and the shadow of the edge and/or the bottom of the tray. Further, an end point detector 8 finds the end points of the segment of the edge of a meal tray separate part from an image of the meal tray separate part. A processing means 6 measures the three- dimensional position of the meal tray on the basis of the straight-line parameter and the end points of the edge to find a gripping position of a manipulator. Further, an R corrector 9 corrects the positions of the end points of the segment in the object image to be measured according to the size of the roundness of the corners of the tray.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION In Japan, the aging of society is rapidly progressing, and the number of elderly people who need nursing care is also increasing rapidly. On the other hand, the population that will be responsible for the labor force is relatively decreasing, and the shortage of human resources in hospitals and welfare facilities is particularly acute. Also, due to changes in the social structure, the number of homes that can provide nursing homes for the elderly as in the past is decreasing.
Under such circumstances, research and development of technology for supplementing care manpower is urgent.

[0002] In hospitals and welfare facilities, there is a problem that work is concentrated because meals must be served / set down to each patient particularly during mealtime. In order to deal with this, the present applicants have developed a “meal transport robot system for the elderly / disabled”. In this system, an autonomous traveling robot having a manipulator distributes / sets a meal tray (hereinafter abbreviated as a tray) from a meal delivery wagon to a bed, which has been conventionally performed manually.

[0003] In the above-mentioned meal transport robot, the position of the tray on the overbed table of each bed is undefined at the time of the lower table, and the position of the tray must be manipulated to grasp the tray with the robot manipulator. It must be measured with the required accuracy. As a measurement method, a method such as a laser that causes harm to humans cannot be used, so that image measurement or the like must be used. The present invention relates to a method and an apparatus for measuring a meal tray in a meal transport robot that serves and distributes meals in a facility described above. In particular, the present invention has no characteristic shape, pattern, mark, or the like to be measured. The present invention relates to a meal tray measuring method and device capable of measuring the position of a tray and stably measuring and holding the tray even when measurement conditions such as brightness and direction of illumination change. is there.

[0004]

2. Description of the Related Art Conventionally, as a method of measuring the position of a tray in the above-described meal transport robot, there has been a method of measuring a characteristic pattern or shape that can be a target of the tray measurement. Further, when it is difficult to find a target pattern or shape suitable for measurement on the meal tray, there is a method for solving this problem by adding a mark. For example, as shown in FIG. 44 (a), when a notch in a separation portion of a tray is set as a measurement target, or when there is no pattern or shape suitable for the measurement target in the tray, as shown in FIG. A mark is attached to the tray, and the mark is set as a measurement target. The notch or the mark is photographed with two cameras, position data thereof is obtained, and the three-dimensional position of the measurement target tray is obtained from the position data. Was calculated to measure the position of the tray.

FIG. 45 is a view showing the arrangement of cameras in the above-mentioned meal transport robot. As shown in FIG. 45, two cameras are attached to the tip of the manipulator, and a tray is photographed by the cameras. The data is sent to a meal tray measuring device provided on the main body side of the meal transport robot, and the position is measured on the tray. FIG. 46 is a diagram showing a configuration of a conventional tray measuring device. As shown in FIG. 1, a conventional tray measuring device includes two cameras 11a and 11a.
b, the camera switching device 12, the image input device 13,
The target image memory 14, the reference image memory 15, the correlation operation processing device 16, and the CPU 17 were provided.

FIG. 47 is a flowchart showing a tray measuring process in the above-mentioned conventional tray measuring device. As shown in the drawing, an image captured by a first camera 11a is selected by a camera switcher 12, an analog image signal is digitized by an image input device 13, and a target image memory 14
(Step S1). Then, the correlation operation processing unit 16 calculates a correlation value between all the regions of the target image stored in the target image memory 14 and the reference image (template image) of the measurement target (for example, a mark) stored in the reference image memory 15. Calculation is performed to obtain the position (point 1) of the region having the highest correlation (step S2). Next, the position (point 2) of the measurement target in the image captured by the second camera 11b
Is obtained in the same manner as above (steps S3 and S4), and the point 1 is obtained.
The three-dimensional position of the measurement target is calculated from the data of the position of the point 2 and the data of the relative positions of the two cameras calibrated in advance (step S5).

[0007]

Since the above-mentioned meal transport robot system for the elderly and handicapped is used especially in hospitals and welfare facilities, it is necessary to minimize the cost of remodeling buildings when introducing the system. No. Therefore, it is desirable that meal trays and the like can be used without modifying the materials used in the facility. However, there are many trays in which characteristic patterns and shapes that can be measured are not seen. Therefore, the existing tray must be replaced with a tray having a characteristic pattern or shape that can be measured, or the existing tray must be marked.

[0008] Replacing the tray has a problem of cost. Marking the tray requires man-hours for making the mark and maintenance for the case where the mark disappears due to the cleaning of the tray. There is. Also, tableware on the tray,
If the pattern / shape or mark of the tray to be measured is concealed, there is a problem that measurement becomes impossible.
In addition, bedside lighting conditions differ between day and night,
It is different between the window side and the corridor side. Also, the color of the overbed table varies. Such a change in conditions makes the tray measurement more difficult.

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and a first object of the present invention is to provide a meal tray which can measure a characteristic pattern shape or a tray having no mark which can be measured. To provide a measurement method and apparatus. A second object of the present invention is to provide a meal tray measurement method and apparatus that can stably measure and hold a tray even when measurement conditions such as brightness and direction of illumination change.

[0010]

FIG. 1 is a block diagram showing the principle of the first, second, fifth and sixth aspects of the present invention. In the figure, 1
Reference numerals a and 1b denote first and second cameras, and reference numeral 10 denotes a meal tray to be measured. Reference numeral 2 denotes a camera switch for switching between the first and second cameras, 3 denotes a means for inputting a camera image, 4a denotes a target image memory for storing target images captured by the cameras 1a and 1b, and 4b denotes a reference image. A reference image memory for storing, 5 is a correlation operation means for performing a correlation operation between the target image and the reference image, 6 is a processing means for obtaining a three-dimensional position of the tray based on the correlation operation result, and 7 is an edge of the tray based on the correlation operation result. 8 is an end point detecting means for obtaining an end point of a line segment, and 9 is an end point detection means for obtaining an end point in a horizontal direction on an image of a measurement target when the corner of the meal tray is round. R correction means for correcting the position of the end point in the left-right direction in the space to be measured from the position.

FIG. 2 is a diagram showing the principle of the present invention. In the figure, 2 ′ is a stereo image acquisition device, 3
Is a means for inputting a camera image, 4a is a camera 1a, 1b
A target image memory for storing a target image taken by
4b is a reference image memory for storing the reference image, 5 is a correlation operation means for performing a correlation operation between the target image and the reference image, 6 is a processing means for obtaining a three-dimensional position of the tray based on the correlation operation result, and 7 is a result of the correlation operation Is an end point detecting means for obtaining an end point of the line segment, and 9 is the R correction means described above.

Numeral 41 is a position calculating means for calculating an end point position in the space from the end point on the image obtained by the end point detecting means 8, and 42 is a tray edge, a shadow of the tray edge, and a tray space from the tray bottom. Attitude calculation means for obtaining the upper attitude, 43 is a grip correction means for obtaining the grip position of the tray by the manipulator from the end point position in space and the attitude of the tray in space, and 44 is one of two stereo images. A corresponding point detecting means for calculating an end point position on the other image from an upper end point position and an edge on the other image; 45, a tray bottom detecting means for detecting a tray bottom;
6 is a color detecting means for detecting an area of the same color as the tray, 47 is a straight line determining means for determining the length of the straight line, 48 is a tray edge, a shadow of the tray edge, and a tray posture obtained from the tray bottom. This is a posture comparing means.

FIG. 3 is a diagram showing the principle of the present invention. In the figure, 2 ′ is a stereo image acquisition device, 3
Is a means for inputting a camera image, 4a is a camera 1a, 1b
A target image memory for storing a target image taken by
4b is a reference image memory for storing the reference image, 5 is a correlation operation means for performing a correlation operation between the target image and the reference image, 6 is a processing means for obtaining a three-dimensional position of the tray based on the correlation operation result, and 7 is a result of the correlation operation A straight line detecting means for determining a straight line based on the line;
0 is an intercept calculating means for obtaining an intercept on the edge of the tray corresponding to the visual field of the image, and 51 is a straight posture calculating means.

In the first, second, fifth and sixth aspects of the present invention, as shown in FIG. 1, the above-mentioned problem is solved as follows. (1) By performing a correlation operation between the images of the first and second cameras and the reference image of the edge, a straight line parameter of the edge of the meal tray in the image of each camera is obtained. The position of the end point of the edge of the separation portion of the meal tray is determined from the image of the second camera, and the straight line parameter of the edge of the meal tray obtained from the image of the camera and the position of the end point of the edge of the separation portion of the tray Is measured.

(2) By performing a correlation operation between the images of the first and second cameras and the reference image of the edge, the straight line parameters of the edge of the meal tray in the images of the respective cameras are obtained. And / or the position of the end point of the edge of the meal tray is obtained from the image of the second camera. The position of the end point in the left-right direction in the space of the measurement target is corrected, and the three-dimensional meal tray is obtained from the straight line parameter of the edge of the meal tray obtained from the image of the camera and the corrected position of the end point of the edge of the tray. Measure the position.

In the third and seventh aspects of the present invention, as shown in FIG. 2, the above-mentioned problem is solved as follows. (3) By performing a correlation operation between the image of the camera acquired from the stereo image acquisition device and the reference image of the edge, the edge of the meal tray in each image is obtained, and the attitude of the meal tray in space is determined from each edge. Determining the position of the end point of the edge of the separated portion of the meal tray in each image, obtaining the position in the space of the end point of the edge of the separated portion from the position of the end point in each image, The position and posture of the meal tray are measured from the upper posture and the position in the space of the end point of the edge of the separated portion.

According to claims 4 and 8 of the present invention, as shown in FIG. 3, the above-mentioned problem is solved as follows. (4) The straight line of the edge of the meal tray is detected from the two images acquired from the stereo image acquisition device, and the position and orientation of the edge in the space of the meal tray are obtained. The corresponding section of the edge is obtained, and the gripping position of the meal tray by the manipulator is calculated from the position of the section of the edge.

In the invention of the first, second, fifth and sixth aspects of the present invention, since the three-dimensional position of the meal tray is measured as described in the above (1) and (2), it can be a measurement target. It is possible to measure the three-dimensional position of a tray having no characteristic pattern or mark. In particular, by detecting the end point of the edge of the separation portion of the tray, when the tray is gripped by the manipulator, the grip point can be directly measured. In addition, the camera position is corrected by correcting the position of the horizontal end point in the space of the measurement target from the position of the camera, the roundness of the corner of the meal tray, and the position of the horizontal end point on the image of the measurement target. The position of the end point can be accurately obtained irrespective of the positional relationship between the meal tray.

In the third and seventh aspects of the present invention,
With the configuration as described in (3) above, the position in the space of the end point of the meal tray and the attitude in the space of the meal tray can be obtained, and the gripping point by the manipulator can be obtained with high accuracy. Further, by detecting the shadow of the edge of the meal tray and the bottom of the meal tray to determine the attitude of the meal tray in space, the position of the meal tray is stably measured with respect to changes in measurement conditions such as lighting conditions. be able to. Further, by detecting an area of the same color as the meal tray, the contour of the tray can be measured stably.

In the invention of claims 4 and 8 of the present invention,
With the configuration as described in (4) above, the gripping point by the manipulator can be obtained even for a meal tray having no separation part.

[0021]

FIG. 4 and FIG. 5 are diagrams showing a measurement object of a tray in an embodiment of the present invention. In the present invention, when measuring the position of the tray, FIG. 4 and FIG.
(A) end points of the tray front edge and the separated portion of the tray, (b) tray front and side edges, (c) end points of the tray front edge and front edge,
(D) The front edge and the left and right side edges of the tray, or (e) The front edge and the left and right end points of the front edge of the tray are measured, and these are photographed by two cameras.
The three-dimensional position of the tray is obtained by measuring the position.

FIGS. 4A to 4C show cases where the distance between the camera and the tray is short or a long focal length lens is used, and FIGS. 5D and 5E show the opposite cases. . FIG.
5 (a) to 5 (c) can measure with higher accuracy as compared with FIGS. 5 (d) and 5 (e), but have a feature that the visual field control is more difficult. FIGS. 4B and 5D show a method of measuring the edges of two or three trays. FIG. 4 (a)
(C) and FIG. 5 (e) show a method of measuring a separation portion between the front edge of the tray and the front edge and an end point forming a tray corner. It can be measured during the process of detecting the end points of the constituent line segments. In particular, FIG. 4A has an advantage that the gripping point when the tray is gripped by the manipulator of the robot can be directly measured.

FIG. 6 is a diagram showing a positional relationship between two cameras and a tray in the embodiment of the present invention. As shown in the figure, the tray 10 is photographed from two different viewpoints by two cameras 11a and 11b to obtain images a and b in the vicinity of the measurement target, and to obtain the position of the measurement target from these images. To calculate the three-dimensional position of the tray. Less than,
An embodiment of the present invention will be described with reference to FIGS.

(1) First Embodiment FIG. 7 is a view showing a configuration of a tray measuring apparatus according to a first embodiment of the present invention. In this embodiment, as shown in FIG. An example is shown in which three-dimensional measurement of a tray having a separated portion is performed by measuring an edge and a tray separated portion. 21, the same components as those shown in FIG. 21 are denoted by the same reference numerals, and in the present embodiment, the linear components are detected from the photographed images in the components shown in FIG. A straight line detector 18 and a line segment end point detector 19 for detecting the end point of the line segment are added.

FIG. 8 is a flowchart showing the overall processing of this embodiment.
FIG. 9 is a flowchart showing the tray front edge end point detection processing, and FIGS. 10 to 14 are diagrams for explaining the measurement processing in the present embodiment. The present embodiment will be described with reference to FIG. First, the camera 11a is selected by the camera switching unit 12, and an image captured by the camera 11a is input to the target image memory 14 by the image input device 13 (step S1 in FIG. 8).
Next, a tray separation portion is detected (step S in FIG. 8).
2). FIGS. 10A and 10B are diagrams for explaining the tray separation portion detection processing. FIG. 10A is a reference image, and FIG.
a shows the image of the tray separation portion taken by a,
FIGS. T1 and T2 correspond to T1 and T2 in FIG. 6, respectively.

The reference image (template image) of the separated portion shown in FIG. 1A is stored in the reference image memory 15 in advance. Then, in each detection area of the search range of the target image shown in FIG. 3B, the correlation operation processing device 16 calculates the correlation value between the template image of the separation part and each detection area, and calculates the maximum value of the template image and the correlation value. Detect high areas. Next, the tray front edge is detected (step S3 in FIG. 8). As shown in FIG. 11A, the template image of the edge is stored in the reference image memory 15 in advance. Then, as shown in FIG. 4B, the correlation operation processing device 16 applies the edge template image to each of the detection areas in the search range of the target image from the central area of the tray separation part detected in step S2. The correlation value of the detection area is obtained, and the area having the highest correlation value with the template image is detected.

Next, the straight line detector 18 obtains the straight line parameters of the tray front edge (straight line 1) (step S4 in FIG. 8). That is, as shown in FIG. 12 (a), the template image used in step S3 is used, and as shown in FIG. 12 (b), the position of the tray front edge detected in step S3 is centered on the tray front edge. Along with this, a local correlation operation is performed between the template image and each of the detection areas within the search range at two or more positions, and an area having the highest correlation with the edge template image is detected. Then, a straight line parameter of the front edge of the tray on the target image is obtained from the center positions of a plurality of areas having a high correlation with the template image by a straight line approximation using the least squares method.
That is, the coefficients (a, b, c) at the approximate straight line ax + by = c by the least squares method are obtained.

Next, the line segment end point detector 19 redetects the end point of the tray front part of the tray separation part with higher accuracy (step S5 in FIG. 8). As shown in FIG. 13A, a straight line parallel to the detected front edge of the tray and below a certain pixel is set as a processing target, and pixel values on the straight line on the processing target are read from the target image memory 14, and the CPU 1
7 is stored in the memory. The tray separation portion is a portion on which the pixel value changes on this straight line.
As shown in (b), the X coordinate of the portion where the pixel value changes is detected by a method such as differentiating this straight line.

FIG. 9 is a flow chart showing the detection processing of the above separated portion. First, in FIG. 14A, the camera 11a
The intersections y1 and y2 with the left and right ends of the image are obtained from the straight line parameters of the front edge of the tray in the image (step S1 in FIG. 9). Next, a straight line lower than the straight line connecting the intersections y1 and y2 by σ pixels is obtained (step S2).
Then, the value of the pixel on the straight line connecting y1 + σ and y2 + σ is read from the target image memory and stored in the buffer b1 in the CPU 17 (step S3). Next, the product in the buffer b1 is multiplied by the differential filter prepared in advance, and the result is stored in another buffer b2 (step S4). Next, the positions of the maximum value and the minimum value of the data stored in the buffer b2 are obtained (step S5).

Then, the position of the minimum value [FIG.
(A) corresponds to the X coordinate at the left end of the tray separation portion, and the position of the maximum value (B in FIG. 13B) also corresponds to the X coordinate at the right end (step S6). When the X coordinate A at the left end and the X coordinate B at the right end of the tray separation portion are obtained as described above, the point 1 in the image of the camera 11a is obtained from A and B. For example, as shown in FIG. 14, the point of intersection of the center of the tray separation portion and the front edge of the tray is point 1.

With the above processing, the position of the object to be measured in the image of the camera 11a is obtained. Next, the camera 11b is selected by the camera switch 12, and the image captured by the camera 11b is input to the target image memory 14 by the image input device 13 (step S6 in FIG. 8). Next, the front edge (straight line 2) of the tray in the image of the camera 11b is detected by the straight line detector 18 (step S).
7).

Here, since the position of the camera 11a is known, the position of the tray front edge in the image of the camera 11b can be estimated from the positions of the cameras 11a and 11b. Therefore, to detect the tray front edge in the image of the camera 11b, the point 1 of the camera 11a
As described above, the correlation operation is performed using the range that can exist in the image b as the search range. In the template image,
A tray front edge image (see FIG. 11) obtained from the image of the camera 1 and the like are used. Of course, the same processing as that for the camera 11a and the image may be performed. However, when the search range is narrowed by the above-described method, the calculation amount of the correlation calculation is small, so that the processing can be performed at high speed.

Next, the straight line parameters of the front edge are obtained in the same manner as described for the camera 11a (step S in FIG. 8).
8). Next, a measurement target (point 2) in the camera 11b is obtained (step S9). Since the measurement target (point 1) in the camera 11a is already known, the camera 11a
, The straight line parameters of the front edge of the tray in the camera 11b, the camera 11a and the camera 1
Point 2 can be calculated from the positional relationship of 1b. Finally, the three-dimensional position of the tray is calculated from points 1 and 2. That is, the three-dimensional position of the measurement target can be calculated from the positional relationship of the camera and the points 1 and 2, and the three-dimensional position and direction of the front edge of the tray can be similarly calculated from the straight lines 1 and 2.

(2) Second Embodiment FIG. 15 is a view showing a configuration of a tray measuring device according to a second embodiment of the present invention. In this embodiment, as shown in FIG.
By measuring the tray front edge and side edge,
An example is shown in which three-dimensional measurement of a tray having a separation portion is performed. 4, the same components as those shown in FIG. 4 are denoted by the same reference numerals, and in this embodiment, the line segment end point detector 19 is removed from the components shown in FIG. The other configuration is the same as that of FIG. 4 described above.

FIG. 16 is a flowchart showing the overall processing of the present embodiment. The present embodiment will be described with reference to FIG. First, the camera 11a is selected by the camera switcher 12, and an image captured by the camera 11a is input to the target image memory 14 by the image input device 13 (step S in FIG. 16).
1). Next, the tray separation portion is detected using the reference image (template image) of the separation portion as in the first embodiment described above (step S2). Next, the linear detector 1
8, a straight line parameter of the tray front edge (straight line 1) is obtained (step S3). That is, as described above,
The tray front edge is detected by calculating the correlation value between the edge template image and each detection area, and a local correlation operation is performed between the edge template image and each detection area within the search range along the tray front edge, As described above, the front edge of the tray (straight line 1) is detected by the straight line detector 18.
To obtain the linear parameters of

Next, a tray side edge straight line parameter (straight line 2) is detected by the same method as that for obtaining the tray front edge straight line parameter (step S4). Next, the camera 11b is selected by the camera switch 12, and the tray front edge straight line parameter (straight line 3) and the tray side edge straight line parameter (straight line 4) are detected in the same manner as described above (steps S5, S6, S7). ). Then, the positional relationship of the camera and the straight line 1 obtained as described above,
The three-dimensional position of the tray is calculated from the straight lines 2, 3, and 4 (step S8).

(3) Third Embodiment FIG. 17 is a diagram showing a configuration of a third embodiment of the present invention.
In the present embodiment, an R corrector 20 is added to the tray measuring device of the embodiment of FIG. 4, and the other configuration is the same as that of FIG. The R corrector 20 is means for correcting the tray separation portion to be measured in the first embodiment according to the shape of the tray and the position of the camera, and the X coordinate of the left end of the tray separation portion measured by the camera. A, from the X coordinate B at the right end and the roundness (R) of the corner of the tray,
The X coordinate in the space to be measured is corrected.

That is, as shown in FIG. 18A, when the camera 11a (or the camera 11b, the camera 11a will be described in the following description) is not provided in front of the tray separation portion, an image is taken by the camera 11a. The left end position P1 of the separated portion obtained from the image and the left end position P2 of the true separated portion are shifted by L. Therefore, in the present embodiment, the measurement error generated when the corner of the tray separation portion is rounded as described above is corrected by the R corrector 20.

FIG. 18B is a diagram for explaining the principle of correction by the R corrector 20. In the figure, the coordinate system is a right-handed system in which the optical axis of the camera 11a is the Z axis.
1 is obtained, and the radius of the roundness of the corner of the tray is r. Here, the position of P1 in the image of the camera 11a is (x _p1 ^C1 , y _p1 ^C1 ) (unit pixel), and the position of the optical axis Z in the image of the camera 11a is (x
_c ^C1 , y _c ^C1 ) (unit pixel), and the angle occupied by one pixel of the camera 11a is ω, and the angle θ ′ between the optical axis and the direction of P1 is as follows. θ ′ = ω (x _p1 ^C1 −x _c ^C1 )

The direction vector of the tray front edge is obtained from the tray front edge obtained from the images taken by the cameras 11a and 11b. Assuming that the x component of this direction vector is x _e and the z component is z _e ,
The angle φ between the camera 11a and the front edge of the tray is as follows. _{^{φ = tan -1 (z e /}} x e) Therefore, looking at the tray corner angle θ is as follows. θ = θ '+ φ

From the above θ and φ, the distance L between P1 and P2 is shown in FIG.
As shown in FIG. 8 (b), when divided into L1 and L2,
As apparent from FIG. L1 = {rr (sin θ / cos θ)} (sin θ / cos
θ) L2 = {(r / cos θ) −r} / cos θ L = L1 + L2 Therefore, P2 is obtained as follows. P2 = (x _p2 , z _p2 ) = (x _p1 + L / cos φ, z _p1 + L
/ Cos φ) where P1 = (x _p1 , z _p1 )

FIG. 19 is a flowchart showing the R correction processing. First, the tray front edge P1 in the camera 11a
An angle θ ′ formed between the optical axis and the direction of P1 is determined from the end point of the camera and the optical axis of the camera (step S1 in FIG. 19). Next, an angle φ that the tray front edge forms with the optical axis is obtained from the tray front edges of the cameras 11a and 11b (step S2). Next, the angle θ is determined from the angle θ ′ and the angle φ (step S3), and the size r of the R of the tray and the angle θ
More distance L is obtained. Finally, the coordinate value of P1 and the distance L
And the angle φ, the coordinate value of P2 is obtained.

FIG. 20 is a flowchart showing the overall processing of the present embodiment, and the measurement processing in the present embodiment will be described with reference to FIG. In FIG. 20, similar to the first embodiment,
First, the camera 11a is selected by the camera switching unit 12, and an image captured by the camera 11a is input to the target image memory 14 by the image input device 13 (step S1 in FIG. 20). Next, the correlation value between the reference image (template image) of the separation portion and each detection region in the search range of the target image is obtained, and the tray separation portion is detected by detecting the region having the highest correlation value (step S2). . Next, as described above, the tray front edge is detected, and the straight line parameters of the tray front edge are obtained (steps S3 and S4).

Next, as described above, the tray separation portion (the end point of the tray front portion: point 1) is detected (step S).
5). Next, the camera 11 is switched by the camera switch 12.
b, and the image captured by the camera 11b is input to the target image memory 14 by the image input device 13 (step S6). Next, the front edge of the tray in the image of the camera 11b is detected (step S7), and a straight line parameter of the front edge is obtained (step S8). Then, a measurement target (point 2) in the camera 11b is obtained (step S9), and the measurement result is R-corrected by the R corrector 20 (step S10). Calculate the three-dimensional position.

(4) Fourth Embodiment As shown in FIG. 4C, this embodiment is an embodiment in which the tray front edge and the end point of the front edge are subjected to three-dimensional measurement of the tray. The configuration of the tray measuring device of the present embodiment is the same as that of FIG. 7 of the first embodiment. The measurement of the front edge and the end point of the front edge in the present embodiment can be performed by the same processing as in the first embodiment. That is, as described above, first, the front edge of the tray is detected by performing a correlation operation between the image of the camera 11a and the template image of the edge, and a straight line parameter of the front edge is obtained. Next, as described above, the line segment end point detector 19 detects the end point of the front edge.

Similarly, the same processing as described above is performed on the image of the camera 11b, the front edge is detected, and the end point of the front edge is detected. Then, the position of the front edge and its end point obtained by the camera 11a and the camera 1
The three-dimensional position of the tray is measured from the position of the front edge and the end point obtained in 1b. This embodiment can be applied to the case where the visual field cannot be obtained as in the first embodiment within the movable range of the manipulator.

(5) Fifth Embodiment As shown in FIG. 5D, this embodiment is an embodiment in which the tray front edge and the left and right side edges are measured as three-dimensional measurement of the tray. The configuration of the tray measuring device of the present embodiment is the same as that of FIG. 15 of the second embodiment. The measurement of the front edge and the left and right side edges in the present embodiment can be performed by the same processing as in the second embodiment, and can be performed from the front edge and the left and right side edges measured from the images of the cameras 11a and 11b. The three-dimensional position of the tray can be measured. The present embodiment can be applied to a case where the field of view of the camera can be widened, and the field of view can be easily controlled as compared with the second embodiment.

(6) Sixth Embodiment In this embodiment, as shown in FIG. 5 (e), three-dimensional measurement of a tray is performed by measuring the end points of the tray front edge and the left and right front edges. An example is shown, and the configuration of the tray measuring apparatus of the present embodiment is the same as that of FIG. 7 of the first embodiment, and the tray is measured by the same processing as the above-described first and fourth embodiments. The dimensional position can be measured. The present embodiment can be applied to a case where the field of view of the camera can be widened, and the field of view can be easily controlled as compared with the first embodiment. The three-dimensional position of the meal tray can be measured by appropriately combining the first to sixth embodiments. In the fourth embodiment, when measuring the end point of the edge, the end point of the edge obtained from the camera image is corrected by applying the R correction shown in the third embodiment, so that the edge point is accurately measured. The end point position can be obtained.

(7) Seventh Embodiment In the above-described embodiment, the case where the three-dimensional position of the tray is measured by measuring the tray separation portion or the end point of the edge of the tray has been described. Further, an embodiment is shown in which the posture of the tray can be calculated and the gripping point of the manipulator can be obtained. FIG. 21 is a diagram showing the configuration of the seventh embodiment of the present invention, and the same components as those shown in FIG. 17 are denoted by the same reference numerals.

In FIG. 21, reference numeral 21 denotes a stereo image acquisition unit. The stereo image acquisition device 21 may be composed of two cameras 11a and 11b and a camera switching device 12 as shown in the above embodiment, or may use one camera and utilize a prism or the like. By sequentially acquiring two images having different viewing directions, two stereo images having different viewpoints can be obtained. In the following description, a case where a stereo image is acquired using two cameras, a first camera and a second camera, will be described as an example for ease of explanation. Reference numerals 13, 14, and 15 denote the above-described image input device, target image memory, reference image memory,
16 is a correlation processing operation device, and 17 is a CPU. Also,
Reference numerals 18, 19, and 20 denote a linear detector for detecting a linear portion of the edge of the tray from the photographed image, a line segment end point detector for detecting an end of a tray separation portion in front of the tray, and a tray to be measured. This is an R corrector for correcting the separated portion according to the shape of the tray and the position of the camera. In this embodiment, in addition to the above configuration, the corresponding point calculator 22,
A position calculator 23, a posture calculator 24, and a grip corrector 25 are provided.

The corresponding point calculator 22 calculates a tray separation portion in another camera image from a point P1 corresponding to the tray separation portion obtained from the first camera image and a tray edge L2 obtained from another camera image. Is calculated. FIG. 22 shows the corresponding point P in the corresponding point calculator 22.
FIG. 4 is a diagram for explaining the second calculation method, wherein CCD1 is an image projection surface of a first camera, and CCD2 is an image projection surface of a second camera. F1 and F2 represent upper and lower limits of the range where the tray exists, O1 and O2 represent the origin positions of the cameras, and P1 represents the image projection surface CCD of the first camera.
1, a point corresponding to the tray separation portion projected on L
Reference numeral 2 denotes an edge of the tray projected on the image projection surface CCD2 of the second camera.

In the figure, the point P2 can be obtained as follows. First, a straight line connecting the origin O1 and the point P1 is extended to determine intersections m1 and m2 between the planes F1 and F2. Next, the intersections m1 and m2 and the origin O2 of the second camera are connected by a straight line, and the intersections between the straight line and the image projection surface CCD2 of the second camera are determined as CC1 and CC2. The point P2 is obtained by finding the intersection of a straight line connecting the intersections CC1 and CC2 with the edge L2 of the tray projected on the image projection plane CCD2.

The position calculator 23 calculates the position P3 in the space of the end point of the tray separation portion from the points P1 and P2. FIG. 23 is a diagram for explaining a method of obtaining the position P3.
CCD1 is the image projection surface of the first camera, CCD2 is 2
The image projection plane of the second camera, P1 and P2 are points corresponding to the tray separation portion on the image projection planes CCD1 and CCD2, respectively, and O1 and O2 are the origin positions of the camera. In order to obtain the position P3 in the space of the end point of the tray separation portion, as shown in FIG. 6, the origin positions O1, O2 of the camera and the points P1, P2
With a straight line. Then, the intersection is obtained and the position is set as the position P3 in the space of the end point of the tray separation portion. If the two straight lines do not intersect, for example, the midpoint of the point where the two straight lines come closest is P3.

The posture calculator 24 calculates the posture L3 of the front contour line of the tray from the tray edge L1 obtained from the camera image viewed from a certain viewpoint and the tray edge L2 obtained from the camera image viewed from another viewpoint. I do. FIGS. 24 (a) and 24 (b) are diagrams for explaining a method of calculating the posture of the above-mentioned front contour line of the tray.
CD2 is the image projection surface of the second camera, L1 and L2 are the image projection surfaces CCD1 and the edge of the tray on CCD2, respectively.
L3 is a direction vector of the edge of the tray in space.

In the figure, the posture L of the contour line at the front of the tray is shown.
The direction vector [l, m, n] of No. 3 can be obtained as follows. As shown in FIG. 2B, an outer product V1 of the vector of the edge L1 and the normal vector C1 of the image projection plane CCD1 is obtained. Similarly, an outer product V2 of the vector of the edge L2 and the normal vector C2 of the image projection plane CCD2 is obtained. Outer product V of outer product V1 and outer product V2 obtained as described above
Calculate 3. The outer product V3 matches the direction vector of L3. The gripping corrector 25 obtains a point P4 for gripping the tray based on the distance between the position P3 'obtained by correcting the position P3 by the R corrector 20 and the point gripped by the manipulator.

FIG. 25 is a diagram showing the relationship between the gripping points of the manipulator and the measurement points. For example, when the movable range of the manipulator is narrow and the manipulator does not reach the measurement point, the manipulator grips a point different from the measurement point as shown in FIG. It is necessary to calculate the gripping position by the container 25. In FIG. 25, the measurement point P3 [x3, y3, z3]
If the distance between the object and the grip point P4 is L and the vector of the posture of the tray is L3 [11, 12, 23], the grip point P4 is a point obtained by moving the measurement point P3 along the measurement point P3 by L along the L3. Therefore, the gripping position P4 can be calculated by the following equation. P4 = [x3 + L · 11, y3 + L · 12, z3 + L ·
l3]

FIG. 26 is a diagram showing the flow of processing in this embodiment, and this embodiment will be described with reference to FIG.
The relative position and orientation of the first camera and the second camera are calibrated in advance, and the video taken by the first camera is selected by the switch, and stored in the target image memory 14 via the image input device 13. (Steps S1 and S2). FIG.
(A) is a figure which shows an example of the image image | photographed by the 1st camera. In the figure, P1 is an endpoint corresponding to a tray separation part, and L1 is an edge of a tray. From the above image,
The edge of the tray is detected by the linear detector 18 to obtain L1 (step S3). Further, the line segment end point detector 19
An end point P1 corresponding to the tray separation portion determined by L1 and the tray separation portion is obtained (step S4).

Similarly, the video captured by the second camera is selected by the switch, and stored in the target image memory 14 via the image input device 13 (steps S5 and S6). FIG. 27B is a diagram illustrating an example of an image captured by the second camera. In FIG. 27B, P2 denotes an end point corresponding to the tray separation portion, and L2 denotes an edge of the tray. Similarly, the edge of the tray is detected by the straight line detector 18 from the image, and L2 is obtained (step S7). Then, the P1 and L are calculated by the corresponding point detector 22 in the manner described with reference to FIG.
2, an end point P2 corresponding to the tray separation portion is obtained. (Step S8).

Next, the position calculator 23 executes the processing shown in FIG.
The position P3 in the space of the end point of the tray separation portion is calculated from the points P1 and P2 by the method described in Step 3 (Step S9).
Then, the posture calculator 24 calculates the posture L3 of the front contour line of the tray from L1 and L2 by the method described with reference to FIG. 24 (step S10). As shown in FIG. 27, the tray used in this embodiment is
Since R of 0 mm is attached, R correction is performed by the R corrector 20 as described above, and an R correction value P3 'of the position P3 in the space of the end point of the tray separation part is obtained (step S).
11). Further, assuming that the distance between the point gripped by the manipulator and the point P3 ′ is, for example, 140 mm, the gripping corrector 25 moves the point P3 ′ in the L3 direction by one as described above.
By moving by 40 mm, a point P4 at which the manipulator grips the meal tray is obtained (step S12).

In the above embodiment, the case where the measurement conditions such as the brightness and the direction of the illumination are not taken into account has been described. However, as described above, the illumination conditions at the bedside are different between day and night, and the window side and the corridor side are different. But different. Also, the color of the overbed table varies. Such a change in the condition affects the appearance of the edge of the tray, and there is a problem that an image on the tray may or may not be detected depending on the condition. For example, in FIG. 27, the entire edge of the tray can be detected, but in FIGS. 29 and 32 described below, the edge of the tray can be detected only between the separated portions. Further, since the acquired image includes distortion of the lens, accurate measurement may not be possible from a partially detected straight line. In order to cope with the above problem, in the following embodiments, the measurement conditions such as the brightness and direction of the illumination are detected by detecting the shadow of the edge of the tray, the bottom of the tray, and the area of the same color as the color of the tray. Even if it changes, the tray can be measured stably.

(8) Eighth Embodiment This embodiment shows an embodiment in which the tray posture is calculated by detecting the shadow of the tray and the tray is measured. The system configuration of this embodiment is shown. Is the same as that of the seventh embodiment shown in FIG. 21. In this embodiment, the straight line detector 18 detects the straight line portion of the edge of the tray and detects the straight line portion of the shadow of the edge of the tray. The attitude of the tray is calculated by the attitude calculator 24 based on the shadow of the edge of the tray.

FIG. 28 is a diagram showing the flow of processing in this embodiment, and this embodiment will be described with reference to FIG. The video taken by the first camera is selected by the switch,
It is stored in the target image memory 14 via the image input device 13 (steps S1 and S2). FIG. 29A is a diagram showing an example of an image taken by the first camera, in which P1 is an end point corresponding to the tray separation portion, L1 is the edge of the tray, and M1 is the front edge of the tray. Is the shadow of From the above image, the edge of the tray is detected by the straight line detector 18,
L1 is obtained (step S3). In addition, the line segment end point detector 1
According to 9, an end point P1 corresponding to the tray separation portion determined by the above L1 and the tray separation portion is obtained (step S4).
Next, the shadow of the edge of the tray is detected by the straight line detector 18 to obtain M1 (step S5).

Similarly, the video captured by the second camera is selected by the switch, and stored in the target image memory 14 via the image input device 13 (steps S6 and S7). FIG. 29B is a diagram illustrating an example of an image captured by the second camera. In FIG. 29B, P2 denotes an end point corresponding to the tray separation portion, L2 denotes the edge of the tray, and M2 denotes the shadow of the edge of the tray. It is. Similarly, the edge of the tray is detected by the straight line detector 18 from the above image to obtain L2 (step S2).
8). Then, the corresponding point detector 22 obtains an end point P2 corresponding to the tray separation portion from the above P1 and L2 by the method described in FIG. (Step S8). Further, the shadow of the edge of the tray is detected, and M2 is obtained (step S10).

Then, the position calculator 23 calculates
3, the position P3 in the space of the end point of the tray separation portion is calculated from the points P1 and P2 (Step S1).
1). Then, the posture calculator 24 calculates the posture L3 of the tray from the shadows M1 and M2 by the method described with reference to FIG.
Is calculated (step S12). Further, the R correction is performed by the R corrector 20 as described above, and the R correction value P3 ′ of the position P3 in the space of the end point of the tray separation part is obtained (step S13), and the manual correction is performed by the grip corrector 25 as described above. A point P4 at which the purator grips the meal tray is determined (step S14).

(9) Ninth Embodiment This embodiment shows an embodiment in which the position of the tray is calculated by detecting the bottom of the tray, and the tray is measured. FIG. 30 shows the ninth embodiment. The configuration is shown. The structure of this embodiment is the same as that shown in FIG. 21 except that a tray bottom detector 26 is provided. The other structure is the same as that shown in FIG. To detect the bottom of the tray, and based on the detected bottom of the tray, the posture calculator 2
Then, the posture of the tray is calculated according to 4.

FIG. 31 is a diagram showing the flow of processing in this embodiment, and this embodiment will be described with reference to FIG. The video taken by the first camera is selected by the switch,
It is stored in the target image memory 14 via the image input device 13 (steps S1 and S2). FIG. 32A is a diagram illustrating an example of an image captured by the first camera, in which P1 is an end point corresponding to a tray separation portion, L1 is an edge of the tray, and M1 is a bottom of the tray. . From the above image,
The edge of the tray is detected by the linear detector 18 to obtain L1 (step S3). Further, the line segment end point detector 19
An end point P1 corresponding to the tray separation portion determined by L1 and the tray separation portion is obtained (step S4). Next, the tray bottom is detected by the tray bottom detector 26 to obtain M1 (step S5).

Similarly, the image picked up by the second camera is selected by the switch, and stored in the target image memory 14 via the image input device 13 (steps S6 and S7). FIG. 32B is a diagram illustrating an example of an image captured by the second camera, in which P2 is an end point corresponding to the tray separation portion, L2 is the edge of the tray, and M2 is the bottom of the tray. . Similarly, the edge of the tray is detected by the straight line detector 18 from the image, and L2 is obtained (step S8). Then, the corresponding point detector 22 obtains an end point P2 corresponding to the tray separation portion from the above P1 and L2 by the method described in FIG. (Step S8). Further, the tray bottom is detected by the tray bottom detector 26 to obtain M2 (step S10).

Then, the position calculator 23 calculates
3, the position P3 in the space of the end point of the tray separation portion is calculated from the points P1 and P2 (Step S1).
1). Then, the attitude calculator 24 calculates the attitude L3 of the tray from the bottoms M1 and M2 of the tray by the method described with reference to FIG. 24 (step S12). Further, the R correction is performed by the R corrector 20 as described above, and the R correction value P3 ′ of the position P3 in the space of the end point of the tray separation portion is obtained (step S13).
The point P at which the manipulator grips the meal tray by 5
4 is obtained (step S14).

(10) Tenth Embodiment This embodiment shows an embodiment in which the edge of the tray is detected by detecting an area having the same color as the tray color. FIG. 33 shows the configuration of this embodiment. . The configuration of this embodiment is the same as that shown in FIG. 21 except that a color detector 27 is provided. The other configuration is the same as that shown in FIG. And the edges L1 and L2 of the tray are detected.

The color detector 27 detects an area of the same color as the tray as follows. A reference area R storing data of all pixel values (R, G, B) = (0, 0, 0) is prepared. By specifying the color of red, a correlation operation is performed between the reference area R and the target area S of the image data captured by the camera to obtain a distortion value DR. Then, IR = [distortion value DR] / [the number of pixels included in the area (for example, 256)] is calculated to obtain IR (average luminance of red). By specifying the color green, a correlation operation is performed between the reference region R and the target region S of the image data captured by the camera to obtain a distortion value DG. Then, IG = [distortion value DG] / [the number of pixels contained in the area (for example, 256)] is calculated, and IG (green average luminance) is obtained. A blue color is designated, and a correlation operation is performed between the reference region R and the target region S of the image data captured by the camera to obtain a distortion value DB. Then, IB = [distortion value DB] / [number of pixels included in the area (for example, 256)] is calculated, and IB (average luminance of blue) is obtained. The range of RGB of the color of the tray is set in advance by the above method in FIG.
4, and the IR,
It is determined whether IG and IB fall in the area shown in FIG. Then, within the range shown in FIG.
If IG and IB are entered, the area is determined to be the same color as the tray color.

FIG. 35 is a diagram showing the flow of processing in this embodiment, and this embodiment will be described with reference to FIG. The video taken by the first camera is selected by the switch,
It is stored in the target image memory 14 via the image input device 13 (steps S1 and S2). From the above image, color detector 2
7, the tray color is detected, and an area having the same color as the tray color is detected (step S3). Then, the edge L1 of the tray is detected by the straight line detector 18 from the area of the same color as the tray color (step S4). Further, the line segment end point detector 19
An end point P1 corresponding to the tray separation portion determined by L1 and the tray separation portion is obtained (step S5).

Similarly, the video captured by the second camera is selected by the switch, and stored in the target image memory 14 via the image input device 13 (steps S6 and S7). Similarly, the tray color is detected from the image by the color detector 27, and an area having the same color as the tray color is detected (step S).
8). Then, the linear detector 1 is extracted from the area of the same color as the tray color.
8, the edge L2 of the tray is detected (step S9).
Then, the corresponding point detector 22 obtains an end point P2 corresponding to the tray separation portion from the above P1 and L2 by the method described in FIG. (Step S10).

Next, the position calculator 23 executes the processing shown in FIG.
3, the position P3 in the space of the end point of the tray separation portion is calculated from the points P1 and P2 (Step S1).
1). Then, the attitude calculator 24 calculates the attitude L3 of the tray from the bottoms M1 and M2 of the tray by the method described with reference to FIG. 24 (step S12). Further, the R correction is performed by the R corrector 20 as described above, and the R correction value P3 ′ of the position P3 in the space of the end point of the tray separation portion is obtained (step S13).
The point P at which the manipulator grips the meal tray by 5
4 is obtained (step S16).

(11) Eleventh Embodiment In this embodiment, the tray bottom is detected, the tray edge is detected, and the tray attitude is calculated from the tray bottom and the tray edge. FIG. 36 shows an embodiment in which it is possible to confirm that the tray is correctly measured by comparing the postures of the trays. FIG. 36 shows the configuration of this embodiment. The configuration of the present embodiment is the same as that shown in FIG. 30 except that an attitude comparator 28 is provided, and the other configuration is the same as that shown in FIG. Attitude comparator 28
Are the tray orientation L3 (direction vector of the tray edge: unit vector) determined from the tray edge and the tray attitude M3 (direction vector of the tray bottom:
The absolute value of the inner product of the unit vector) is determined, and it is checked whether this value is within a predetermined threshold. When the absolute value is within the predetermined threshold, the posture L3 of the tray obtained from the edge of the tray and the bottom of the tray are used. It is determined that the obtained tray attitude M3 matches.

FIG. 37 is a diagram showing the flow of processing in the present embodiment. The present embodiment will be described with reference to FIG. The video taken by the first camera is selected by the switch,
It is stored in the target image memory 14 via the image input device 13 (steps S1 and S2). From the image, the edge L1 of the tray is detected by the straight line detector 18 (Step S).
3). Further, the line segment end point detector 19 obtains an end point P1 corresponding to the tray separation part determined by the above L1 and the tray separation part (step S4). Further, the bottom of the tray is detected by the tray bottom detector 26 to obtain M1 (step S5).

Similarly, the video captured by the second camera is selected by the switch, and stored in the target image memory 14 via the image input device 13 (steps S6 and S7). Similarly, the edge L2 of the tray is detected by the straight line detector 18 from the image (step S8). Then, the P1 and L are calculated by the corresponding point detector 22 in the manner described with reference to FIG.
2, an end point P2 corresponding to the tray separation portion is obtained. (Step S9). Further, the bottom of the tray is connected to the tray bottom detector 2.
6 to obtain M2 (step S10). Next, the position calculator 23 calculates the position P3 in the space of the end point of the tray separation portion from the points P1 and P2 by the method described in FIG. 23 (step S11). Further, the attitude calculator 24 calculates the tray attitude L3 from the tray edges L1 and L2 by the method described with reference to FIG. 24 (step S12), and the attitude calculator 24 calculates the tray bottom M
The posture M3 of the tray is calculated from M1 and M2 (step S
13).

Then, the postures L3 and M3 of the trays are compared by the posture comparator 28 as described above (Step S).
14) If the two do not match, for example, the processing ends as an error and the tray measurement is performed again. If they are substantially the same, the R correction is performed by the R corrector 20 as described above, and the R correction value P3 'of the position P3 in the space of the end point of the tray separation portion is obtained (step S1).
5) As described above, a point P4 at which the manipulator grips the meal tray is obtained by the gripping corrector 25 (step S16).

(12) Twelfth Embodiment This embodiment is directed to an embodiment in which the edge of the tray, the bottom of the tray, and the shadow of the tray are detected, and the posture which is most clearly visible is selected to calculate the posture. FIG. 38 shows the configuration of this embodiment. The configuration of the present embodiment is such that a straight line discriminator 29 is added to the one shown in FIG. 36 to discriminate the line that is most clearly seen. The other configurations are the same as those shown in FIG. Is the same. The straight line discriminator 29 determines the length of the straight portion with respect to the detected edge of the tray, the bottom of the tray, and the shadow of the tray, and determines that the length of the straight portion is greater than or equal to a predetermined value. Determine the line.

FIG. 39 is a diagram for explaining a method of determining the length of a straight line by the straight line determiner 29. As shown in the figure, the correlation between the edge reference image and the target area S is obtained, and the target image area S having the highest correlation with the edge reference image is sequentially obtained.
Then, the shift amounts α1, α2, of the center coordinates of the target image area
.., Αn are sequentially obtained, and β = | (αn−1−αn) − (α2
−α1) | is calculated. When β obtained as described above exceeds a predetermined threshold value, it is determined that the straight line portion has ended.

FIG. 40 is a diagram showing a flow of processing in the present embodiment, and the present embodiment will be described with reference to FIG.
The video captured by the first camera is selected by the switch, and stored in the target image memory 14 via the image input device 13 (steps S1 and S2). From the image, the edge L1 of the tray is detected by the straight line detector 18 (Step S).
3). Further, the line segment end point detector 19 obtains an end point P1 corresponding to the tray separation part determined by the above L1 and the tray separation part (step S4). Further, the bottom of the tray is detected by the tray bottom detector 26 to obtain M1 (step S5). The shadow of the edge of the tray is detected by the straight line detector 18 to obtain N1 (step S6).

Similarly, the video captured by the second camera is selected by the switch, and stored in the target image memory 14 via the image input device 13 (steps S7, S8). Similarly, the edge L2 of the tray is detected from the image by the straight line detector 18 (step S9). Then, the P1 and L are calculated by the corresponding point detector 22 in the manner described with reference to FIG.
2, an end point P2 corresponding to the tray separation portion is obtained. (Step S10). Further, the bottom of the tray is detected by the tray bottom detector 26 to obtain M2 (step S11). Further, the shadow of the edge of the tray is detected by the straight line detector 18 and N
2 is obtained (step S12).

Then, the position calculator 23 calculates the points P1,
The position P3 in the space of the end point of the tray separation part is calculated from P2 (step S13). The posture calculator 24 calculates the posture L3 of the tray from the edges L1 and L2 of the tray (step S14).
It is determined whether L2 is clearly visible (step S1).
5). If the edges L1 and L2 are clearly visible, the process proceeds to step S21, where the R correction is performed by the R corrector 20, and the R correction value P3 ′ of the position P3 in the space of the end point of the tray separation part is obtained. As described above, the point P4 at which the manipulator grips the meal tray is obtained by the grip corrector 25 (step S22). If the edges L1 and L2 are not clearly visible, the flow goes to step S16, where the straight line discriminator 29 determines that the bottom M1 of the tray is the shadow N of the edge of the tray.
It is determined whether it is clearly visible from 1 or not.

If the bottom M1 of the tray is clearly visible from the shadow N1 of the edge of the tray, the flow goes to step S17, where the posture is calculated using the bottoms M1 and M2 of the tray to obtain the posture M3 of the tray. In S19, the posture comparator 28 compares the posture L3 obtained from the edge of the tray with the posture M3 obtained from the bottom of the tray. If the postures of the two are substantially the same, the process proceeds to step S21, and the R correction and the grip correction are performed as described above. In step S16, the bottom M1 of the tray is the shadow N of the edge of the tray.
If it is determined that the tray is not clearly visible, the process proceeds to step S18, where the posture is calculated based on the shadows N1 and N2 of the edges of the tray, and the posture N3 of the tray is obtained.
At 20, the attitude comparator 28 compares the attitude L3 obtained from the edge of the tray with the attitude N3 obtained from the shadow of the edge of the tray. If the postures of the two are substantially the same, the process proceeds to step S21, and the R correction and the grip correction are performed as described above.

(13) Thirteenth Embodiment In the above embodiment, an embodiment has been described in which the tray separation portion is measured and the grip point is obtained. However, even if there is no separation portion on the tray or even if there is a separation portion, the camera is used. In some cases, the camera is not in a position where shooting can be performed. This embodiment shows an embodiment in which even in the above case, only the edge of the tray can be detected and the tray can be gripped by the manipulator.

FIG. 41 is a diagram for explaining the tray measuring method according to the present embodiment. In the figure, CCD1 is the image projection surface of the first camera, and CCD2 is the image projection surface of the second camera. O1 and O2 are the origin positions of the respective cameras, L1 is the edge of the edge of the tray on CCD1, L2 is C
The edge of the edge of the tray on CD2, L3 is the edge of the tray in space, and P1, P2, P3, and P4 are the four corner points of CCD1. P5 is an intercept between the plane including the points O1, P1 and P2 and L3, P6 is an intercept between the plane including the points O1, P3 and P4 and L3, and P7 is a middle point between P5 and P6.

In this embodiment, the gripping position is obtained as follows. The edges L1 and L2 of the tray are detected by the first camera and the second camera, and the posture L3 of the tray is obtained. The intercept P5 between the plane including the origin O1 and the two corners P1 and P2 of the first camera and L3 and the intercept P6 between the plane including the origin O1 and the two corners P3 and P4 of the camera and L3 are obtained. The distance A between the slices P5 and P6 is compared with the width B of the gripper of the manipulator. If A> B, the tray is grasped by the manipulator.
For example, a middle point P7 between the six points is set as a grip point.

FIG. 42 is a diagram showing the configuration of this embodiment.
The basic configuration of this embodiment is the same as that of the above-described embodiment. In this embodiment, in addition to the above-described straight line detector 18 and straight line discriminator 29, a straight line position / posture calculator 30 and an intercept position calculator 31 are provided. Is provided. FIG. 43 is a diagram showing a flow of processing in the present embodiment, and the present embodiment will be described with reference to FIG. The video captured by the first camera is selected by the switch, and stored in the target image memory 14 via the image input device 13 (steps S1 and S2). From the above image, the edge L1 of the tray is detected by the straight line detector 18 (step S3), and it is determined whether the edge L1 of the tray detected by the straight line discriminator 29 is visible in the full field of view of the first camera (step S3). Step S4).

If the edge L1 of the tray is not visible in the full field of view of the first camera, the measurement cannot be performed by this method, so the flow goes to step S5 to perform measurement by another method.
If the field of view is full, step S
The process goes to step S6, where the video taken by the second camera is selected by the switch, and is stored in the target image memory 14 via the image input device 13 (steps S6 and S7). Similarly, the edge L2 of the tray is detected by the straight line detector 18 from the image (step S8). Then, the posture L3 of the edge of the tray is obtained from the above L1 and L2 by the linear position / posture calculator 30 (step S9). Then, the intercept position calculator 31
Thus, the intercepts P5 and P6 are calculated as described with reference to FIG. Then, it is determined whether or not the tray can be gripped by the manipulator based on the distance between the sections P5 and P6 and the width of the gripper of the manipulator. If the tray can be gripped, the gripping point is determined (step S10).

[0089]

As described above, the following effects can be obtained in the present invention. (1) The three-dimensional position of the meal tray is measured by obtaining the straight line parameter of the edge of the meal tray and the position of the end point of the edge of the tray separation portion from the images of the first and second cameras. It is possible to measure a characteristic pattern shape that can be a measurement target or a three-dimensional position of a tray having no mark. In particular, by detecting the end point of the edge of the separation portion of the tray, when the tray is gripped by the manipulator, the grip point can be directly measured. Also, the position of the end point in the horizontal direction in the space of the measurement target is corrected based on the position of the camera, the roundness of the corner of the meal tray, and the position of the end point in the horizontal direction on the image of the measurement target. Therefore, the position of the end point can be accurately obtained regardless of the position of the camera.

(2) The position in the space of the end point of the meal tray and the attitude of the meal tray are measured from the image obtained from the stereo image acquisition device, and the gripping point of the manipulator is obtained. It is possible to measure a three-dimensional position of a tray having no typical pattern or a mark without a mark, and hold the tray with a manipulator. Further, by detecting the shadow of the edge of the meal tray and the bottom of the meal tray to determine the attitude of the meal tray in space, the position of the meal tray is measured stably with respect to changes in measurement conditions such as lighting conditions. be able to. Further, by detecting an area of the same color as the meal tray, the contour of the tray can be measured stably.

(3) 2 obtained from the stereo image obtaining unit
The straight line of the edge of the meal tray is detected from the one image, the position and orientation of the edge in the space of the meal tray are obtained, and the corresponding section of the edge in the space is obtained from the field of view of one image. By calculating the gripping position of the meal tray by the manipulator from the position of the section of the camera, the meal tray is gripped by the manipulator even if there is no separation part or the separation part does not fall within the field of view of the camera. It is possible to do.

[Brief description of the drawings]

FIG. 1 is a principle configuration diagram (1) of the present invention.

FIG. 2 is a principle configuration diagram (2) of the present invention.

FIG. 3 is a principle configuration diagram (3) of the present invention.

FIG. 4 is a diagram showing a measurement target according to the embodiment of the present invention.

FIG. 5 is a diagram (continued) illustrating a measurement target in the example of the present invention.

FIG. 6 is a diagram showing a positional relationship between two cameras and a tray in the embodiment of the present invention.

FIG. 7 is a diagram illustrating a configuration of a tray measuring device according to the first embodiment of the present invention.

FIG. 8 is a flowchart showing the overall processing of the first embodiment of the present invention.

FIG. 9 is a flowchart illustrating tray front edge end point detection processing in the first embodiment.

FIG. 10 is an explanatory diagram of tray separation portion detection in the first embodiment.

FIG. 11 is an explanatory diagram of tray front edge detection in the first embodiment.

FIG. 12 is an explanatory diagram of detecting the inclination of the front edge of the tray in the first embodiment.

FIG. 13 is an explanatory diagram of re-detection of a tray separation portion in the first embodiment.

FIG. 14 is an explanatory diagram of how to set measurement points in the first embodiment.

FIG. 15 is a diagram illustrating a configuration of a tray measuring device according to a second embodiment of the present invention.

FIG. 16 is a flowchart showing the overall processing of the second embodiment of the present invention.

FIG. 17 is a diagram illustrating a configuration of a tray measuring device according to a third embodiment of the present invention.

FIG. 18 is a diagram illustrating the principle of R correction according to the third embodiment of the present invention.

FIG. 19 is a flowchart showing an R correction process according to the third embodiment of the present invention.

FIG. 20 is a flowchart showing the overall processing of the third embodiment of the present invention.

FIG. 21 is a diagram illustrating a configuration of a tray measuring device according to a seventh embodiment of the present invention.

FIG. 22 is a diagram illustrating a method of calculating corresponding points in a corresponding point calculator.

FIG. 23 is a diagram for explaining how to obtain a position P3 in the position calculator.

FIG. 24 is a diagram illustrating a method of calculating the tray attitude by the attitude calculator.

FIG. 25 is a diagram showing a relationship between a gripping point of the manipulator and a measurement point.

FIG. 26 is a diagram showing a processing flow of a seventh embodiment of the present invention.

FIG. 27 is a diagram illustrating an example of an image captured by a camera in the seventh embodiment.

FIG. 28 is a diagram showing the flow of the process of the eighth embodiment of the present invention.

FIG. 29 is a diagram illustrating an example of an image captured by a camera in the eighth embodiment.

FIG. 30 is a diagram illustrating a configuration of a tray measuring device according to a ninth embodiment of the present invention.

FIG. 31 is a diagram showing the flow of the process of the ninth embodiment of the present invention.

FIG. 32 is a diagram illustrating an example of an image captured by a camera in the ninth embodiment.

FIG. 33 is a diagram illustrating a configuration of a tray measuring device according to a tenth embodiment of the present invention.

FIG. 34 is a diagram for explaining color detection in a tenth embodiment of the present invention.

FIG. 35 is a diagram showing the flow of the process of the tenth embodiment of the present invention.

FIG. 36 is a diagram illustrating a configuration of a tray measuring device according to an eleventh embodiment of the present invention.

FIG. 37 is a diagram showing a flow of processing of the eleventh embodiment of the present invention.

FIG. 38 is a diagram showing a configuration of a tray measuring device according to a twelfth embodiment of the present invention.

FIG. 39 is a diagram illustrating a discriminating method using a straight line discriminator according to a twelfth embodiment of the present invention.

FIG. 40 is a diagram showing a processing flow of the twelfth embodiment of the present invention.

FIG. 41 is a diagram illustrating a tray measuring method according to a thirteenth embodiment of the present invention.

FIG. 42 is a diagram illustrating a configuration of a tray measuring device according to a thirteenth embodiment of the present invention.

FIG. 43 is a diagram showing a flow of processing in a thirteenth embodiment of the present invention.

FIG. 44 is a diagram showing a measurement target in a conventional example.

FIG. 45 is a diagram showing an arrangement of cameras in the meal transport robot.

FIG. 46 is a diagram showing a configuration of a conventional tray measuring device.

FIG. 47 is a flowchart showing tray measurement processing in a conventional tray measurement device.

[Explanation of symbols]

 1a, 1b Camera 2 Camera Switcher 2 'Stereo Image Acquisition Device 3 Image Input Means 4a Target Image Memory 4b Reference Image Memory 5 Correlation Calculation Means 6 Processing Means 7 Straight Line Detection Means 8 End Point Detection Means 9 R Correction Means 10 Meal Tray 11a, 11b Camera 12 Camera switcher 13 Image input device 14 Target image memory 15 Reference image memory 16 Correlation processor 17 CPU 18 Straight line detector 19 Line segment end point detector 20 R corrector 21 Stereo image acquirer 22 Corresponding point calculator 23 Position Calculator 24 Posture Calculator 25 Grip Corrector 26 Tray Bottom Detector 27 Color Detector 28 Posture Comparator 29 Straight Line Discriminator 30 Linear Position and Posture Calculator 31 Intercept Calculator 41 Position Calculator 42 Posture Calculator 43 Grip Corrector 44 Corresponding point detecting means 45 Tray bottom detecting means 46 Detection means 47 linearly discriminating means 48 position comparing means 50 intercept calculation unit 51 linearly posture calculating means

Claims (8)

[Claims] 1. A method for measuring a three-dimensional position of a meal tray from images taken by first and second cameras, the method comprising: calculating a correlation between an image of the first and second cameras and a reference image of an edge. To obtain the straight line parameters of the edge of the meal tray in the image of each camera, and also find the position of the end point of the edge of the separated portion of the meal tray from the image of the first and / or the second camera, A meal tray measuring method, comprising: measuring a three-dimensional position of the meal tray from a straight line parameter of an edge of the meal tray obtained from the image of the camera and a position of an end point of the edge of the tray separation portion. 2. A method of measuring a three-dimensional position of a meal tray from images taken by first and second cameras, the method comprising: calculating a correlation between an image of the first and second cameras and a reference image of an edge. Is performed, the straight line parameter of the edge of the meal tray in the image of each camera is obtained, and the position of the end point of the edge of the meal tray is obtained from the image of the first and / or second camera, and The position of the end point in the horizontal direction in the space of the measurement target was corrected from the position, the roundness of the corner of the meal tray, and the position of the end point in the horizontal direction on the image of the measurement target, and obtained from the image of the camera described above. A meal tray meter for measuring a three-dimensional position of the meal tray from the straight line parameters of the edge of the meal tray and the corrected position of the end point of the edge of the tray; Measurement method. 3. A method for measuring a three-dimensional position of a meal tray from two images acquired from a stereo image acquiring device, wherein a correlation between a camera image acquired from the stereo image acquiring device and an edge reference image is provided. By performing the calculation, the edge of the meal tray in each image is obtained, the attitude in the space of the meal tray is obtained from each edge, and the position of the end point of the edge of the separation portion of the meal tray in each image is obtained. From the position of the end point in each image, the position in the space of the end point of the edge of the separated part is obtained from the position of the end point in each image, A meal tray measurement method characterized by measuring a posture. 4. A method for measuring a three-dimensional position of a meal tray from two images acquired from a stereo image acquisition device, wherein a straight line of an edge of a meal tray is obtained from two images acquired from the stereo image acquisition device. Is detected, the position and orientation of the edge in the space of the meal tray are obtained, the corresponding section of the edge in the space is obtained from the field of view of one image, and the meal tray by the manipulator is obtained from the position of the section of the edge. A meal tray measuring method, comprising: calculating a gripping position of a meal tray. 5. A first and second camera for photographing a meal tray, a camera switch for switching between the first and second cameras, and a target image memory for storing images of the first and second cameras. A reference image of a separated portion of the meal tray, a reference image memory storing a reference image of an edge portion of the tray, a target image of the first and second cameras stored in the target image memory, and a reference image memory Correlation calculating means for performing a correlation calculation with the reference image stored in the storage means, and a straight line detecting means for calculating a straight line parameter of an edge of the meal tray in the images of the first and second cameras based on the correlation calculation result of the correlation calculating means. Line segment end point detecting means for obtaining an end point of a line segment of an edge of the meal tray separation portion from an image of the meal tray separation portion of the first and / or second camera; A meal tray based on the straight line parameters of the edge of the meal tray in the images of the first and second cameras obtained by the straight line detecting means and the end points of the edges of the separated portions of the meal tray obtained by the line segment end point detecting means. A meal tray measuring device comprising processing means for measuring the three-dimensional position of the food tray. 6. A first and second camera for photographing a meal tray, a camera switcher for switching between the first and second cameras, and a target image memory for storing images of the first and second cameras. A reference image memory storing a reference image of an edge portion of the meal tray; a correlation between target images of the first and second cameras stored in the target image memory; and a reference image stored in the reference image memory. Correlation calculation means for performing calculation, straight line detection means for obtaining a straight line parameter of an edge of a meal tray in images of the first and second cameras based on a result of the correlation calculation by the correlation calculation means, and first and / or second Line segment end point detection means for obtaining the end point of the line segment of the edge of the meal tray from the image of the camera, and the end point of the line segment is determined by the camera position and the roundness of the corner of the tray. R correction means for correcting the position of an end point in the left-right direction on the image of the measurement target to a position in the left-right direction in the space of the measurement target, and images of the first and second cameras obtained by the straight line detection means A meal tray measurement unit for measuring a three-dimensional position of the meal tray based on the straight line parameter of the edge of the meal tray in the above and the end point of the edge of the meal tray obtained by the R correction means. apparatus. 7. A stereo image acquisition device for photographing a meal tray, a target image memory for storing the two acquired images, a reference image memory for storing a reference image of the meal tray, and a reference image memory stored in the target image memory. Correlation calculation means for performing a correlation calculation between the two target images and the reference image stored in the reference image memory; straight line detection means for obtaining a straight line in each image based on the correlation calculation result of the correlation calculation means; A posture calculating means for calculating a posture of the straight line in space from a straight line in each of the images; and a line segment end point for obtaining an end point of a line segment of an edge of a meal tray for one of the two images. Detecting means, the end point in the image obtained by the line segment end point detecting means, and the straight line of the edge of the meal tray on the other image obtained by the straight line detecting means, Corresponding point calculating means for calculating the end point of the line segment of the edge of the meal tray on the image of the above, and obtaining the position of the end point in space from the end points in the respective images obtained by the line segment end point detecting means and the corresponding point calculating means. A position calculator, a three-dimensional position of the meal tray is measured based on the position in space obtained by the position calculator, and a position of an end point in space obtained by the position calculator, and a gripping position by the manipulator A meal tray measuring device comprising processing means for obtaining 8. A stereo image acquisition device for photographing a meal tray, a target image memory for storing the two acquired images, a reference image memory for storing a reference image of the meal tray, and a target image memory stored in the target image memory. Calculating means for calculating a correlation between the two target images and the reference image stored in the reference image memory; and detecting a straight line for obtaining an edge of a meal tray in each image based on a result of the correlation calculation by the correlation calculating means. Means for determining whether a straight line portion of the edge is within the field of view of the image, and a straight line attitude calculating means for calculating an attitude in space of the edge from the edges on the two images. When the straight line portion of the edge is in the full field of view of the image, the corresponding intercept of the edge is determined from the field of view of one of the two images. When the slice calculating means to be determined and the straight line discriminating means determine that the edge of the meal tray appears to fill the field of view of the image, the manipulator grips the meal tray from the position of the slice calculated by the slice calculating means. A meal tray measuring device, comprising: processing means for calculating a position.