JP5453859B2 - Robot system - Google Patents

Description

  The present invention relates to a robot system, and more particularly to a robot system provided with a camera for photographing a target article.

  Conventionally, for example, when inspecting the appearance of a target article, a robot having a camera at the tip of an arm is used. The target article is photographed by a camera mounted on the tip of the robot arm. In the case of a robot system including a robot on which such a camera is mounted, it is necessary to perform so-called teaching in which the camera is set to accurately move to a position to be inspected (Patent Document 1 regarding camera teaching). reference). Since the camera position is not accurate only by the appearance position of the camera, it is determined whether or not the camera position is appropriate based on a photographed image photographed by the camera.

  However, depending on, for example, the performance of the camera or the control device or the communication means connecting the camera and the control device, it takes time to transmit and acquire the captured image. There may be a time difference between them. As described above, when there is a time difference between actual image capturing and display, a delay also occurs in an operation instruction from the control device to the robot. Therefore, excessive input is performed for the operation of the camera mounted on the robot, and so-called overshoot and undershoot are likely to occur. As a result, there arises a problem that fine adjustment of the position of the camera mounted on the robot becomes difficult. On the other hand, in order to precisely adjust the position of the camera, high performance is required for the camera, the control device, and the communication means.

JP-A-9-321080

An object of the present invention does not depend on the ability of the communication unit is that the accurate alignment of the camera mounted on the robot to provide easy robotic system.

In the first, second, or third aspect of the invention, the image synthesizing unit creates a synthesized image based on the planar image data stored in the storage unit as a plurality of grouped data groups. When finely adjusting the position of the camera mounted on the robot, a photographed image including at least one target article is acquired immediately before that. The image composition means creates a composite image that approximates the image currently captured by the camera, using the captured image including the target article. Therefore, when the image currently captured by the camera cannot be displayed on the display means, that is, the time difference between the image captured by the camera and the image displayed on the display means due to the capability of the camera, control device or communication means. Even when this occurs, the robot can be instructed to operate by referring to the composite image displayed on the display means corresponding to the image currently captured by the camera. Therefore, without depending on the ability of the communication unit, the accurate alignment of the camera mounted on the robot can be easily performed.

  According to the first aspect of the present invention, when the storage area for storing new planar image data is insufficient in the storage means, the data updating means deletes the predetermined planar image data and creates a free storage area. New plane image data is stored. In other words, the data updating means is the plane image data having the oldest selection time selected for creating a composite image from the most frequent data group belonging to the largest number of plane image data belonging to the data group stored in the storage means. Is deleted. The most frequent data group having the largest number of plane image data belonging to it is highly likely to include a lot of plane image data with similar focal coordinates, that is, similar images. Then, the plane image data having the oldest selection time, that is, the lowest selection frequency is deleted. That is, the plane image data having a low contribution to the creation of the composite image is deleted, and new plane image data is added. As a result, plane image data serving as a basis for a composite image necessary for operation of the robot is ensured without increasing the storage area. Further, by deleting the plane image data based on the above-mentioned conditions, the plane image data having a low contribution to the creation of the composite image is reduced, and the time for selecting the desired plane image data for the creation of the composite image is Shortened. Therefore, a captured image can be substituted by a composite image that is quickly created, and accurate positioning of a camera mounted on a robot can be easily performed.

  In the invention described in claim 2, when the storage area for storing new planar image data is insufficient in the storage means, the data updating means deletes the predetermined planar image data and newly adds a new storage area. Planar image data is stored. That is, the data update unit selects a specific data group to which the new plane image data belongs from among a plurality of data groups stored in the storage unit. Then, the data update unit deletes the plane image data having the closest focal coordinate to the average value of the focal coordinates in all the planar image data belonging to the specific data group. There is a high possibility that a lot of plane image data having approximate focal coordinates exist in the range close to the average value of the focal coordinates of the specific data group. Therefore, it is considered that the planar image data whose focal point coordinates are closest to the average value in the specific data group can be substituted by other planar image data whose focal point coordinates are close. That is, the plane image data that has a small influence on the creation of the composite image is deleted, and new plane image data is added. As a result, plane image data serving as a basis for a composite image necessary for operation of the robot is ensured without increasing the storage area. In addition, by deleting plane image data that has a small effect on the creation of the composite image, the plane image data that has a small effect on the creation of the composite image is reduced, and the desired plane image data is selected for creating the composite image. The time to do is shortened. Therefore, a captured image can be substituted by a composite image that is quickly created, and accurate positioning of a camera mounted on a robot can be easily performed.

  In the invention described in claim 3, when the storage area for storing the new plane image data is insufficient in the storage means, the data update means deletes the predetermined plane image data and creates a new one in the free storage area. Planar image data is stored. That is, the data update unit selects a specific data group to which the new plane image data belongs from among a plurality of data groups stored in the storage unit. Then, the data updating unit deletes the image having the oldest shooting time from the plane image data in the specific data group. There is a high possibility that the plane image data with the oldest shooting time has been acquired at a position close to the subsequent shooting of the image. For this reason, it is considered that the oldest plane image data in the specific data group can be substituted by plane image data photographed thereafter. That is, the plane image data that has a small influence on the creation of the composite image is deleted, and new plane image data is added. As a result, plane image data serving as a basis for a composite image necessary for operation of the robot is ensured without increasing the storage area. In addition, by deleting plane image data that has a small effect on the creation of the composite image, the plane image data that has a small effect on the creation of the composite image is reduced, and the desired plane image data is selected for creating the composite image. The time to do is shortened. Therefore, a captured image can be substituted by a composite image that is quickly created, and accurate positioning of a camera mounted on a robot can be easily performed.

The block diagram which shows the robot system by 1st Embodiment of this invention. Schematic showing the robot system according to the first embodiment Schematic diagram showing the data structure of planar image data Schematic diagram showing the relationship between the camera, workpiece, and field plane Schematic diagram showing the relationship between the camera, workpiece, field plane, crossed image, and composite image The schematic diagram which shows the display of the display part by 1st Embodiment The schematic diagram which shows the data group in the robot system by 1st Embodiment. Schematic which shows the flow of a process in the robot system by 1st Embodiment. Schematic which shows the flow of a data update process in the robot system by 1st Embodiment. Schematic which shows the flow of a data update process in the robot system by 2nd Embodiment. The schematic diagram which shows the data group in the robot system by 2nd Embodiment Schematic which shows the flow of a data update process in the robot system by 3rd Embodiment. The schematic diagram which shows the data group in the robot system by 3rd Embodiment

Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that, in a plurality of embodiments, substantially the same components are denoted by the same reference numerals, and description thereof is omitted.
(First embodiment)
The robot system according to the first embodiment is shown in FIGS. The robot system 10 includes a robot 11, a control device 12, and a communication unit 13 as communication means. The robot 11 according to the first embodiment is applied to a visual inspection for inspecting whether or not the work 14 that is a target article is correctly assembled. The robot 11 is provided in an inspection facility 15 that is remotely located from the control device 12. The inspection facility 15 is formed in a tunnel shape, for example, and has a transport unit 16 at the bottom. The workpiece 14 is transported to a predetermined position of the inspection facility 15 by the transport unit 16. When the work 14 is transported to a predetermined position, the transport unit 16 stops. As a result, the appearance of the workpiece 14 is inspected by the robot 11. The robot 11 is provided on the top side of the inspection facility 15.

  The robot 11 is configured as a six-axis vertical articulated robot, for example. As is well known, the robot 11 has an arm 17. Each arm 17 is driven by a driving force from a servo motor 18 that is an actuator. The arm 17 has a camera 19 as an end effector at the tip. Between the servo motor 18 and the camera 19 of the arm 17, a driving force transmission mechanism such as a speed reduction mechanism and a link (not shown) is provided. Thereby, the arm 17 provided with the camera 19 at the tip is driven by the driving force from the servo motor 18.

  The control device 12 includes a robot controller 21 and a personal computer (hereinafter “PC”) 22 which is an electronic computer. The robot controller 21 and the personal computer 22 constitute a control device 12 as shown in FIG. The robot controller 21 and the personal computer 22 constituting the control device 12 may be arranged apart from each other such that the robot controller 21 is in the vicinity of the robot 11 as shown in FIG. May be arranged. The control device 12 includes a control unit 23 as shown in FIG. The control unit 23 is mainly configured by a microcomputer including a CPU, a ROM, a RAM, and the like (not shown). The control unit 23 drives the servo motor 18 of each axis of the robot 11 via the servo control unit 24 of the robot controller 21 based on a robot control program input and stored in advance. Thereby, the control device 12 controls the operation of the robot 11. The control device 12 has an input unit 25 connected to the personal computer 22. The input unit 25 includes, for example, manually input devices such as a keyboard 26, a mouse 27, and a joystick 28, and a magnetic or optical input device 29 such as an HDD or a DVD. The operator 30 can create a robot control program describing the operation procedure of the robot 11 according to the application by executing the programming software on the personal computer 22. In this case, the operator 30 can create a robot control program relatively easily by combining or modifying package instructions according to the application.

  The communication unit 13 connects the robot 11 and the control device 12 in a communicable manner. In the case of this embodiment, the communication unit 13 includes a first cable 31 for connecting the robot 11 and the robot controller 21 and a second cable 32 for connecting the robot controller 21 and the personal computer 22 as shown in FIG. The robot 11 and the control device 12 are connected by wire. Note that the communication unit 13 is not limited to wired communication, and may employ wireless communication, for example. By connecting the robot 11 and the control device 12 by the communication unit 13, the servo motor 18 that drives each axis of the robot 11 and the camera 19 that captures the workpiece 14 are controlled by the control device 12. In addition, an image captured by the camera 19 is transmitted to the control device 12 via the communication unit 13.

  The robot controller 21 of the control device 12 controls the movement of the arm 17 of the robot 11 in accordance with an inspection program executed on the personal computer 22. When the work 14 flowing through the transport unit 16 stops at a predetermined position of the inspection facility 15, the work 14 is taken by the camera 19. The robot controller 21 performs imaging while driving the arm 17 to move the camera 19 to a plurality of target points having different preset three-dimensional positions and orientations. The target point at which the camera 19 attached to the robot 11 is driven is taught in advance and stored in the robot controller 21 or the personal computer 22 as position data. In teaching, an entry prohibition area of the robot 11 is also set.

The camera 19 has, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and photographs the workpiece 14. The camera 19 outputs the captured image as an electrical signal for each pixel to the control device 12 via the communication unit 13. In the camera 19, for example, optical conditions at the time of photographing such as the size of the visual field, the focus, or the aperture are automatically adjusted. Further, the camera coordinate system that is the coordinate system of the image captured by the camera 19 and the robot coordinate system that is the coordinate system of the position of the robot 11 are mutually correlated by coordinate transformation, that is, calibration.

  In addition to the above, the control device 12 includes a planar image data creation unit 33, a storage unit 34, a group identification unit 35, an image composition unit 36, a display unit 37, and a data update unit 38 as shown in FIG. The planar image data creation unit 33, the group identification unit 35, the image composition unit 36, and the data update unit 38 may have a configuration that functions as software by a computer program executed by the control unit 23, or a configuration that functions as hardware. Good. In the present embodiment, an example in which the planar image data creation unit 33, the storage unit 34, the group identification unit 35, the image synthesis unit 36, the display unit 37, and the data update unit 38 are provided in the personal computer 22 has been described. A part or all of the above may be provided in the robot controller 21.

  The planar image data creation unit 33 creates planar image data 40 as shown in FIG. The planar image data 40 is centered on the focal coordinates 41 of the captured image captured by the camera 19, and the coordinate data 43 indicating the captured image 42 captured by the camera 19 and the focal coordinates 41 when the captured image 42 is captured. And the direction data 44 indicating the direction in which the photographed image 42 is photographed are created in association with each other. The planar image data creation unit 33 acquires the captured image 42 created from the electrical signal for each pixel from the camera 19 via the communication unit 13. At this time, the plane image data creation unit 33 uses the focal coordinates 41 of the camera 19 correlated with the position of the arm 17 from the robot 11 or the robot controller 21 as coordinate data, and the captured image 42 correlated with the angle of the arm 17. Acquire the direction of shooting as direction data. Further, the planar image data creation unit 33 acquires the time when the captured image 42 was captured as time data 45. FIG. 3 schematically shows the data structure of the planar image data 40 and is different from the actual planar image data 40.

  The storage unit 34 includes, for example, a non-volatile memory and an HDD. The storage unit 34 stores the planar image data 40 created by the planar image data creation unit 33. In this case, the group specifying unit 35 groups the plane image data 40 having focal coordinates approximated within a preset setting range, and stores them in the storage unit 34 as a plurality of data groups. The planar image data 40 is acquired over the entire movable region of the robot 11. Therefore, the group specifying unit 35 groups the acquired plane image data 40 whose focal coordinates are close to each other and stores them in the storage unit 34. The storage unit 34 may be configured together with the ROM and RAM of the control unit 23. In addition, grouping by the group specifying unit 35 not only groups objects having similar focal coordinates, but also divides an area to be imaged by the camera 19, that is, a space to be imaged, into a plurality of group areas in advance. Grouping may be performed so that the planar image data 40 having the focal point coordinates included in the region is grouped into one group.

The image composition unit 36 creates a composite image from the planar image data 40 stored in the storage unit 34. Specifically, the image composition unit 36 selects the selected plane image data from the plurality of plane image data 40 stored in the storage unit 34. The selected plane image data intersects the field plane 51 as shown in FIG. 5 when the virtual field plane 51 photographed by the camera 19 is moved in the optical axis extending direction as shown in FIG. This is plane image data of the intersecting images 52, 53, and 54. That is, the image compositing unit 36 selects, as the selected plane image data, the plane image data of the intersecting images 52, 53, and 54 that intersect the field plane 51 from the plurality of plane image data stored in the storage unit 34. Then, the image synthesizing unit 36, intersection image 52, 53 and 54 are taken by photographing the focal coordinates and intersection images 52, 53 of the cross image 52, 53, 54 based on the selected plane image data selected orientation The image is projected onto the field plane 51 as it is. The image composition unit 36 creates an image projected on the field plane 51 as a composite image 55 of the work 14 taken. The image composition unit 36 creates the composite image 55 and assigns the selected time when the selected plane image data is selected as the intersecting images 52, 53, 54 to the selected plane image data and stores it in the storage unit 34. The display unit 37 is, for example, a liquid crystal display that is integral with or separate from the personal computer 22. As shown in FIG. 6, the display unit 37 displays an image captured by the camera 19 or a composite image created by the image composition unit 36 as a display image 371.

  The data update unit 38 updates and stores the planar image data 40 stored in the storage unit 34. In the case of the first embodiment, when the storage area for storing new plane image data 40 is insufficient in the storage unit 34, the data update unit 38 starts from the most frequent data group having the largest number of plane image data belonging to the data group. The plane image data with the oldest selection time is deleted, and new plane image data 40 is stored in a free storage area. Since the camera 19 captures new images every moment, the captured images are sent to the control device 12 one after another. On the other hand, since the storage area of the storage unit 34 is finite, not all the plane image data 40 can be stored. Therefore, when the storage capacity of the storage unit 34 is insufficient, the data update unit 38 deletes the plane image data with the oldest selection time from the most frequent data group.

  The plane image data 40 created by the plane image data creation unit 33 is specified and stored in one of the data groups 61, 62, and 63 as shown in FIG. Stored in the unit 34. Of the plurality of data groups 61, 62, 63, the one having the largest number of plane image data is the most data group 62. Therefore, it is considered that the most data group 62 includes a lot of plane image data 40 whose focal coordinates are close to each other. The most frequent data group 62 includes from the oldest flat image data 65 to the newest flat image data 66 selected by the image composition unit 36. Of these, the plane image data 65 with the oldest selection time is least frequently used in the most data group 62, and it can be considered that the other plane image data 40 that approximates the focal point coordinates 41 can be substituted. That is, it is considered that the planar image data 65 having the oldest selection time in the most data group 62 has a low utility value for creating a composite image in the image composition unit 36. Therefore, when the storage capacity of the storage unit 34 is insufficient, the data update unit 38 deletes the plane image data 65 having the oldest selection time from the most frequent data group 62 to secure a storage area. The data update unit 38 stores the new plane image data 40 acquired in the reserved storage area.

Next, the operation of the robot system 10 having the above configuration will be described with reference to FIG.
Prior to the inspection of the workpiece 14 to be inspected, the robot system 10 performs teaching for setting the photographing position of the workpiece 14. Teaching is performed by executing a teaching program included in the inspection program on the personal computer 22. The operator 30 moves the camera 19 attached to the arm 17 of the robot 11 to a predetermined photographing position by operating the joystick 28 of the input unit 25 while executing the teaching program on the personal computer 22. In this case, the camera 19 captures an image every second or tens of minutes. Therefore, the captured image captured by the camera 19 is displayed as a moving image on the display unit 37 of the control device 12.

On the other hand, the etc. communication speed of the communication unit 13, between the image and the photographed image which is captured by the camera 19 are displayed on the display unit 37, occurs a time lag. For example, there is a case where an image photographed at the focal coordinates (x0, y0, z0) is displayed on the display unit 37 even though the camera 19 is photographed at the focal coordinates (x1, y1, z1). is there. Since the operator 30 operates the robot 11 while viewing the image displayed on the display unit 37, when the temporal difference between the shooting with the camera 19 and the display on the display unit 37 increases, The discrepancy between the operation of the joystick 28 as the input unit 25 and the image displayed on the display unit 37 increases. As a result, the input to the joystick 28 serving as the input unit 25 becomes excessive, so-called overshoot occurs in the operation of the robot 11, and the uncomfortable feeling of operation increases. Therefore, in the first embodiment, the difference between the image captured by the camera 19 and the image displayed on the display unit 37 is reduced by the processing based on FIG.

  The arm 17 of the robot 11, that is, the camera 19 attached to the arm 17, is moved by the operation of the input unit 25 by the operator 30 (S101). Input from the input unit 25 is transmitted from the personal computer 22 of the control device 12 to the robot 11 via the robot controller 21. The robot controller 21 drives the servo motor 18 of the robot 11 by the servo control unit 24 according to the input from the input unit 25. As a result, the arm 17 of the robot 11 is moved to a position corresponding to the input from the input unit 25 by the servo motor 18. As a result, the camera 19 attached to the arm 17 also moves to the shooting position corresponding to the input.

  The camera 19 shoots the workpiece 14 at a predetermined time interval such as every several tenths of a second while moving with the arm 17 (S102). At this time, the camera 19 acquires the focal point coordinates and the shooting direction of the shot image together with the shot image of the workpiece 14 (S103). For example, the camera 19 acquires the focal point coordinates and the shooting direction of the shot image from the robot controller 21 or the like. Each time the work 14 is photographed at a predetermined time interval, the camera 19 acquires the focal point coordinates and the photographing direction of the photographed image.

  The acquired captured image, and the focal position and orientation of the captured image are transferred to the control device 12 via the communication unit 13 (S104). The planar image data creation unit 33 acquires the captured image, the focal coordinates of the captured image, and the captured orientation from the camera 19 via the communication unit 13, and creates the planar image data 40. As shown in FIG. 3, the plane image data creation unit 33 is centered on the focal point coordinate 41, the captured image 42, the coordinate data 43 that is the focal point coordinate of the captured image 42, and the direction data that is the direction in which the captured image 42 is captured. 44 to create the plane image data 40. At this time, the planar image data creation unit 33 adds the time at which the captured image 42 was captured to the planar image data 40 as time data. When the plane image data 40 is created, the image composition unit 36 determines whether or not the captured image 42 has been acquired by the control device 12 within a predetermined time (S105). That is, the image composition unit 36 determines whether or not the photographed image 42 of the photographed work 14 has been transferred to the control device 12 within a predetermined time after photographing the work 14 with the camera 19.

  When a photographed image of the workpiece 14 is transferred to the control device 12 within a predetermined time after photographing with the camera 19, the photographed image of the workpiece 14 photographed with the camera 19 and the image displayed on the display unit 37 are displayed. The difference between the two is small. That is, the difference between the photographed image photographed by the camera 19 and the image displayed on the display unit 37 is so small that the operator 30 cannot recognize it. In this case, the above-described overshoot of the arm 17 to which the camera 19 is attached and an uncomfortable feeling of operation hardly occur. For this reason, when the image data captured by the camera 19 is acquired by the control device 12 within a predetermined time after shooting (S105: Yes), the image composition unit 36 captures the image by the camera 19 and transfers it to the control device 12. The captured image is displayed on the display unit 37 as it is (S106). At this time, the time difference for preventing overshoot and uncomfortable feeling is, for example, about several milliseconds or less. Of the data constituting the planar image data 40, the coordinate data 43, the direction data 44, and the time data 45 are text data, whereas the captured image 42 is image data. Therefore, the coordinate data 43, the direction data 44, and the time data 45 are transferred from the camera 19 to the control device 12 in a shorter time than the captured image 42. Thereby, the image composition unit 36 acquires the time from the image capturing by the camera 19 to the data transfer related to the captured image 42 based on, for example, the time data 45 transferred from the camera 19.

  On the other hand, if the photographed image of the workpiece 14 is not acquired by the control device 12 within a predetermined time from the photographing by the camera 19, the photographed image of the workpiece 14 photographed by the camera 19 and the display unit 37 are displayed. There is a difference between the image and an overshoot or discomfort. Therefore, when the data of the captured image captured by the camera 19 is not acquired by the control device 12 within a predetermined time from the capturing (S105: No), the image composition unit 36 is based on the focal coordinates 41 of the captured image and the capturing direction. A composite image is created (S107).

  As described above, when the difference between the time taken for capturing the captured image by the camera 19 and the time for displaying the captured image on the display unit 37 becomes large, an operation failure such as overshoot is likely to occur in the operation of the camera 19. For this reason, the image compositing unit 36 uses the planar image data already stored in the storage unit 34 when the captured image data is not acquired by the control device 12 within a predetermined time from the shooting by the camera 19. A virtual composite image corresponding to the photographed image taken in step 1 is created. In this case, the composite image does not need to completely match the captured image captured by the camera 19, and is a virtual image that does not cause an operation failure such as an overshoot in the operation of the camera 19 by the input from the input unit 25. Anything is acceptable.

  Specifically, the image composition unit 36 creates a composite image in the following procedure. As shown in FIG. 4, the image composition unit 36 moves the visual field plane 51 photographed by the camera 19 in the optical axis extending direction. Here, the field plane 51 captured by the camera 19 corresponds to the field plane 51 of the captured image 42 transferred from the camera 19 to the control device 12 in S104. The image compositing unit 36 selects the plane image data of the intersecting images 52, 53, and 54 that intersect the field plane 51 as shown in FIG. 5 from the plurality of plane image data 40 stored in the storage unit 34. Select as data. That is, the image composition unit 36 selects the intersecting images 52, 53, and 54 that intersect the visual field plane 51 of the camera 19, and sets the planar image data of the intersecting images 52, 53, and 54 as selected planar image data. Then, the image composition unit 36 acquires coordinate data related to the focal coordinates of the intersecting images 52, 53, and 54 included in the selected selected plane image data and direction data related to the shooting direction. Here, the intersecting images 52, 53 and 54 are selected based on the planar image data 40 of the captured image stored in the storage unit 34. Therefore, the plane image data corresponding to the selected intersection images 52, 53, and 54 includes focal coordinate data and direction data acquired when the images were captured in the past. The image composition unit 36 projects the acquired cross images 52, 53, 54 onto the visual field plane 51 as they are when the cross images 52, 53, 54 are photographed to create a composite image 55. Thereby, a composite image 55 that approximates the captured image captured by the camera 19 in S102 is created in a pseudo manner. Further, the image composition unit 36 assigns the selected time selected as the selected time data 46 as shown in FIG. 3 to the selected plane image data corresponding to the selected intersecting images 52, 53, 54 and stores it in the storage unit 34. Let

  When the composite image 55 is created in the image composition unit 36, the display unit 37 displays the created composite image 55 (S108). Thereby, the operator 30 operates the input unit 25 with reference to the composite image 55 corresponding to the captured image captured in S102 displayed on the display unit 37. The composite image 55 displayed on the display unit 37 simulates the captured image of the work 14 captured in S102 as described above. For this reason, the image displayed on the display unit 37 in S108 approximates at least the position at which the workpiece 14 is imaged, that is, the focal coordinates and the imaged direction. As a result, although the captured image actually captured by the camera 19 in S102 and the composite image displayed on the display unit 37 in S108 are strictly different, the captured focal coordinates and orientations are approximated. That is, when a so-called frame drop occurs in the moving image shot by the camera 19, the image composition unit 36 creates and inserts a pseudo image corresponding to the frame that has been dropped. Therefore, the operator 30 who operates the camera 19 can operate the input unit 25 without receiving a sense of incongruity due to a time difference from the image displayed on the display unit 37.

  The group identifying unit 35 identifies a group of captured images, that is, captured images 42 (S109). Even when there is a time difference between the image capturing by the camera 19 and the display on the display unit 37, the control device 12 receives the captured image captured by the camera 19 in S102 from the camera 19 at any time. The group specifying unit 35 groups the plane image data 40 corresponding to the captured image received from the camera 19 in this way. Specifically, the group specifying unit 35 groups the plane image data 40 having the focal coordinates 41 that are approximated within a preset setting range. The group specifying unit 35 stores the planar image data 40 grouped as shown in FIG. 7 as a plurality of data groups 61, 62, and 63. The planar image data 40 is acquired over the entire movable region of the robot 11. Therefore, the group specifying unit 35 groups the acquired plane image data 40 whose focal coordinates are close to each other. The group specifying unit 35 specifies a group every time the captured image taken in S102 is acquired from the camera 19, classifies it into one of the data groups 61, 62, 63, and stores it in the storage unit 34.

  When the group of photographed images is specified, the data update unit 38 determines whether or not the storage area of the storage unit 34 can store the planar image data of the photographed image that has been photographed in S102 and identified in S109. (S110). As described above, the storage area of the storage unit 34 is finite. Therefore, when the storage of the plane image data 40 including the captured image 42 captured by the camera 19 is repeated, the storage area of the storage unit 34 is saturated at any time. Therefore, each time the flat image data 40 of the captured image is acquired from the camera 19, the data update unit 38 determines whether or not a storage area capable of storing the flat image data 40 remains in the storage unit 34. When the data updating unit 38 determines that the storage area remains in the storage unit 34 in S110 (S110: Yes), the storage unit 34 stores the planar image data 40 of the captured image that was captured in S102 and the group was specified in S109. Store in a free storage area.

  On the other hand, if the data update unit 38 determines in S110 that no storage area remains in the storage unit 34 (S110: No), the data update unit 38 executes processing based on FIG. That is, the data update unit 38 selects the most frequent data group 62 from the plurality of data groups 61, 62, and 63 in the storage unit 34, and deletes the plane image data 65 having the oldest selection time (S201). Specifically, it is as follows. As described in S <b> 109, the plane image data 40 captured by the camera 19 is grouped in the group specifying unit 35 whose focal coordinates 41 are approximate to each other. Therefore, the data updating unit 38 selects the most frequent data group 62 in which the number of the planar image data 40 included is the largest from the data groups 61, 62, 63 of the planar image data 40 stored in the storage unit 34. Since a plurality of data groups 61, 62, and 63 are set according to the focal point coordinate 41, one of these is the most frequent data group 62.

  The data update unit 38 extracts the plane image data 65 having the oldest selection time from the plane image data 40 included in the selected most data group 62. Each data group 61, 63 including the most frequent data group 62 includes a plurality of plane image data 40. The selection time is included in the planar image data 40 as selection time data 46. Therefore, the data update unit 38 extracts, from the plurality of plane image data 40 included in the most data group 62, the plane image data 65 having the oldest time selected as the selected plane image data in the creation of the composite image 55 in S107. . The data update unit 38 deletes the extracted planar image data 65 with the oldest selection time from the storage area of the storage unit 34. In this case, the deletion includes, for example, writing “0 data” in the corresponding storage area to delete the plane image data 40 itself, and simply allowing the corresponding storage area to be overwritten. As described above, the plane image data 65 having the oldest selection time among the plane image data 40 included in the most frequent data group 62 can be substituted by other plane image data included in the most frequent data group 62 and deleted. It is considered that the influence on the creation of the composite image 55 is small. Therefore, when the new storage area remaining in the storage unit 34 is insufficient, the data update unit 38 deletes the plane image data 65 having the oldest selection time from the plurality of plane image data 40 included in the most frequent data group 62. .

  The data update unit 38 stores the plane image data 40 of the image captured in S102 in the vacant storage area of the storage unit 34 by deleting the plane image data 65 having the oldest selection time included in the most data group 62. (S202). As a result, the planar image data 40 shot in S102 and the group specified in S109 is stored in the storage area of the storage unit 34. When the plane image data 40 is stored in the storage unit 34 in S202, the process returns to S101 and the above processing is repeated.

  In the first embodiment described above, the image composition unit 36 creates the composite image 55 based on the planar image data 40 stored in the storage unit 34 as a plurality of grouped data groups 61, 62, 63. . When the position of the camera 19 mounted on the robot 11 is finely adjusted, an image including at least one of the workpieces 14 is taken immediately before that. The image composition unit 36 uses the photographed image including the work 14 to create a composite image 55 that approximates the image currently photographed by the camera 19. Therefore, when the image currently captured by the camera 19 cannot be displayed on the display unit 37, that is, the time difference between the image captured by the capability of the camera 19, the control device 12, or the communication unit 13 and the displayed image. Even when this occurs, the operator 30 can operate the robot 11 with reference to the composite image 55 displayed on the display unit 37 corresponding to the image currently captured by the camera 19. As a result, the uncomfortable feeling of the operation of the robot 11 such as excessive input to the input unit 25 is reduced. Therefore, accurate alignment of the camera 19 mounted on the robot 11 can be easily performed without depending on the capabilities of the camera 19, the control device 12, and the communication unit 13.

  In the first embodiment, when the storage area for storing new planar image data 40 is insufficient in the storage unit 34, the data update unit 38 stores a plurality of data groups 61 and 62 stored in the storage unit 34. , 63, the largest data group 62 having the largest number of plane image data 40 belonging thereto is selected. Then, the data update unit 38 extracts the plane image data 65 with the oldest selection time selected for creating the composite image 55 from the most frequent data group 62 and deletes it. The most frequent data group 62 is likely to contain a lot of planar image data 40 in which the focal coordinates 41 are approximate and the photographing direction is approximate, that is, similar images. Then, the flat image data 40 having the oldest selection time, that is, the lowest selection frequency is deleted. That is, the planar image data 40 having a low contribution to the creation of the composite image 55 is deleted, and new planar image data 40 corresponding to the image photographed in S102 is added. As a result, the planar image data 40 that is the basis of the composite image 55 necessary for the operation of the robot 11 is secured without increasing the storage area of the storage unit 34. Further, by deleting the plane image data 40 based on the above-described conditions, the plane image data 40 having a low contribution to the creation of the composite image 55 is reduced, and the desired plane image data 40 for creating the composite image 55 is reduced. The time for selecting is shortened. Furthermore, by deleting the planar image data 40 having a low contribution to the creation of the composite image 55, the composite image 55 does not disappear unexpectedly during the operation of the camera 19. Therefore, the captured image can be substituted by the composite image 55 that is quickly created, and accurate alignment of the camera 19 mounted on the robot 11 can be easily performed.

(Second Embodiment)
A robot system according to the second embodiment will be described. The robot system 10 according to the second embodiment has the same configuration as that of the first embodiment, but the processing flow in the data update unit 38 is different from that of the first embodiment. Therefore, the flow of processing in the data updating unit 38 of the second embodiment will be described based on FIG.
The processes from S101 to S111 in the operation of the robot system 10 shown in FIG. 8 are the same as those in the first embodiment. If the data update unit 38 determines in S110 that the storage area does not remain in the storage unit 34 (S110: No), the specific data group is selected, and the planar image closest to the average value of the focal coordinates of the specific data group is selected. Data is deleted (S301). Specifically, it is as follows. As described in S109, the plane image data 40 captured by the camera 19 is grouped in the group specifying unit 35 whose focal coordinates are similar to each other. Therefore, the data updating unit 38, as shown in FIG. 11, from the data groups 71, 72, and 73 of the plane image data 40 stored in the storage unit 34, the latest plane that has been shot in S102 and whose group has been specified in S109. The data group 72 to which the image data 40a belongs is selected as the specific data group 72.

  From the planar image data 40 included in the selected specific data group 72, the data updating unit 38 has a planar image whose focal coordinates 41 are closest to the average value of the focal coordinates of all the planar image data 40 included in the specific data group 72. Data 74 is extracted. Then, the data update unit 38 deletes the extracted planar image data 74 whose focal point coordinates are closest to the average value from the specific data group 72. It is considered that a plurality of pieces of planar image data 40 included in the specific data group 72 are distributed near the average value of the focal coordinates. Therefore, it is considered that the plane image data 74 that is closest to the average value of the focal coordinates can be substituted by the other plane image data 40. Therefore, the data update unit 38 deletes the plane image data 74 whose focal point coordinates are closest to the average value of the specific data group 72 from the plurality of plane image data 40 included in the specific data group 72.

  The data update unit 38 deletes the plane image data 74 closest to the average value of the focal coordinates of the specific data group 72, thereby storing the latest plane image data 40a captured in S102 in an empty storage area of the storage unit 34. (S302). As a result, the latest planar image data 40a photographed in S102 and identified in S109 is stored in the storage area of the storage unit 34. When the plane image data 40 is stored in the storage unit 34 in S302, the process returns to S101 and the above processing is repeated.

  In the second embodiment described above, when the storage area for storing the latest planar image data 40 a is insufficient in the storage unit 34, the data update unit 38 stores a plurality of data groups 71 stored in the storage unit 34. The specific data group 72 to which the latest planar image data 40a belongs is selected from 72 and 73. Then, the data update unit 38 deletes the plane image data 74 having the closest focal coordinates to the average value of the focal coordinates in all the planar image data 40 belonging to the specific data group 72. In the range close to the average value of the focal coordinates of the specific data group 72, there is a high possibility that a lot of planar image data 40 having approximate focal coordinates exists. Therefore, it is considered that the planar image data 74 whose focal point coordinates are closest to the average value in the specific data group 72 can be substituted by other planar image data 40 whose focal point coordinates are close. That is, the plane image data 74 that has a small influence on the creation of the composite image 55 is deleted in the storage area of the storage unit 34, and the latest plane image data 40a is added. As a result, the planar image data 40 that is the basis of the composite image 55 necessary for the operation of the robot 11 is secured without increasing the storage area of the storage unit 34. Further, by deleting the plane image data 74 that has a small influence on the creation of the composite image 55, the plane image data 40 that has a small influence on the creation of the composite image 55 is reduced. The time for selecting the planar image data 40 is shortened. Furthermore, by deleting the plane image data 74 having a low contribution to the creation of the composite image 55, the composite image 55 will not disappear unexpectedly during the operation of the camera 19. Therefore, the captured image can be substituted by the composite image 55 that is quickly created, and accurate alignment of the camera 19 mounted on the robot 11 can be easily performed.

(Third embodiment)
A robot system according to a third embodiment will be described. The robot system 10 according to the third embodiment has the same configuration as that of the first embodiment, but the processing flow in the data update unit 38 is different from that of the first embodiment. Therefore, the flow of processing in the data update unit 38 of the third embodiment will be described based on FIG.
The processes from S101 to S111 in the operation of the robot system 10 shown in FIG. 8 are the same as those in the first embodiment. If the data update unit 38 determines in S110 that no storage area remains in the storage unit 34 (S110: No), the data update unit 38 selects a specific data group and deletes the oldest planar image data from the specific data group ( S401). Specifically, it is as follows. As described in S109, the plane image data 40 captured by the camera 19 is grouped in the group specifying unit 35 whose focal coordinates are similar to each other. Therefore, the data update unit 38, as shown in FIG. 13, from the data groups 81, 82, and 83 of the plane image data 40 stored in the storage unit 34, the latest plane that was shot in S102 and the group was specified in S109. The data group 82 to which the image data 40b belongs is selected as the specific data group 82.

  The data update unit 38 extracts the oldest flat image data 84 with the oldest shooting time from the flat image data 40 included in the selected specific data group 82. Then, the data update unit 38 deletes the plane image data 84 with the oldest shooting time from the specific data group 82. As for the plane image data 40 included in the plurality of specific data groups 82, the older one is, the higher the possibility that the photographing is repeated at a position closer to the subsequent photographing of the image by the camera 19. Therefore, it is considered that the plane image data 84 with the oldest shooting time can be substituted by other plane image data 40. Therefore, the data update unit 38 deletes the plane image data 84 having the oldest shooting time from the plurality of plane image data 40 included in the specific data group 82.

  The data update unit 38 deletes the plane image data 84 with the oldest shooting time included in the specific data group 82, so that the plane image data 40 b of the latest image shot in S <b> 102 is stored in an empty storage area of the storage unit 34. Store (S402). As a result, the latest planar image data 40b that was shot in S102 and whose group was specified in S109 is stored in the storage area of the storage unit 34. When the plane image data 40 is stored in the storage unit 34 in S402, the process returns to S101 and the above processing is repeated.

  In the third embodiment, when the storage area for storing the latest planar image data 40 b is insufficient in the storage unit 34, the data update unit 38 has a plurality of data groups 81, 82, 83 stored in the storage unit 34. The specific data group 82 to which the latest plane image data 40b belongs is selected. Then, the data update unit 38 deletes the plane image data 84 having the oldest shooting time from the plane image data 40 in the specific data group 82. The flat image data 84 with the oldest shooting time is highly likely to have been acquired in a position close to the subsequent image shooting. Therefore, it is considered that the oldest flat image data 84 in the specific data group 82 can be substituted by the flat image data 40 captured thereafter. That is, in the storage unit 34, the plane image data 84 that has a small influence on the creation of the composite image 55 is deleted, and the latest plane image data 40b is added. As a result, the planar image data 40 that is the basis of the composite image 55 necessary for the operation of the robot 11 is secured without increasing the storage area of the storage unit 34. In addition, by deleting the plane image data 84 that has a small influence on the creation of the composite image 55, the plane image data 40 that has a small influence on the creation of the composite image 55 is reduced. The time for selecting the planar image data 40 is shortened. Furthermore, by deleting the plane image data 84 having a low contribution to the creation of the composite image 55, the composite image 55 will not disappear unexpectedly during the operation of the camera 19. Therefore, the captured image can be substituted by the composite image 55 that is quickly created, and accurate alignment of the camera 19 mounted on the robot 11 can be easily performed.

  The present invention described above is not limited to the above-described embodiment, and can be applied to various embodiments without departing from the gist thereof.

  10 is a robot system, 11 is a robot, 12 is a control device, 13 is a communication unit (communication means), 17 is an arm, 19 is a camera, 21 is a robot controller (control device), 22 is a personal computer (control device), and 33 is Planar image data creation unit (planar image data creation unit), 34 is a storage unit (storage unit), 36 is an image synthesis unit (image synthesis unit), 37 is a display unit (display unit), and 38 is a data update unit (data update unit). Means).

Claims (3)

A robot system comprising a robot having a camera at the tip of an arm, and a control device disposed remotely from the robot via communication means,
The control device includes:
A plane that creates plane image data centered on the focal coordinates by associating the captured image captured by the camera and the focal coordinates of the captured image when the captured image is captured and the direction in which the captured image was captured. Image data creation means;
Storage means for grouping a plurality of the plane image data having the focal coordinates approximated in a preset setting range from the plane image data created by the plane image data creation means and storing them as a plurality of data groups;
When the captured image captured by the camera cannot be acquired within a predetermined time, the field plane captured by the camera is moved in the optical axis extending direction from the planar image data stored in the storage means. Plane image data of the intersecting image that intersects the field plane when selected as the selected plane image data, and the focal coordinates of the intersecting image based on the selected selected plane image data and the direction in which the intersecting image was captured are selected. An image composition unit that projects the projected image onto the field plane as it is when the crossed image is captured to create a composite image, and stores the selected plane image data in the storage unit with a selection time selected.
When the captured image captured by the camera cannot be acquired within the predetermined time, the combined image combined by the image combining means is displayed , and the captured image captured by the camera is displayed within the predetermined time. Display means for displaying an actual image taken by the camera when acquired ,
When a storage area for storing new planar image data is insufficient in the storage unit, the selection time is selected from the most frequent data group to which the number of the planar image data belongs among the plurality of data groups stored in the storage unit. Data updating means for deleting the oldest plane image data and storing the new plane image data in an empty storage area ,
The predetermined time is a value that is small enough that the difference between the captured image captured by the camera and the image displayed on the display means does not cause a sense of incongruity due to a delay in data transfer via the communication means. A robot system characterized by being set . A robot system comprising a robot having a camera at the tip of an arm, and a control device disposed remotely from the robot via communication means,
The control device includes:
A plane that creates plane image data centered on the focal coordinates by associating the captured image captured by the camera and the focal coordinates of the captured image when the captured image is captured and the direction in which the captured image was captured. Image data creation means;
Storage means for grouping a plurality of the plane image data having the focal coordinates approximated in a preset setting range from the plane image data created by the plane image data creation means and storing them as a plurality of data groups;
When the captured image captured by the camera cannot be acquired within a predetermined time, the field plane captured by the camera is moved in the optical axis extending direction from the planar image data stored in the storage means. Plane image data of the intersecting image that intersects the field plane when selected as the selected plane image data, and the focal coordinates of the intersecting image based on the selected selected plane image data and the direction in which the intersecting image was captured are selected. An image composition unit that projects the projected image onto the field plane as it is when the crossed image is captured to create a composite image, and stores the selected plane image data in the storage unit with a selection time selected.
When the captured image captured by the camera cannot be acquired within the predetermined time, the combined image combined by the image combining means is displayed , and the captured image captured by the camera is displayed within the predetermined time. Display means for displaying an actual image taken by the camera when acquired ,
When a storage area for storing new plane image data is insufficient in the storage unit, a specific data group to which the new plane image data belongs is selected from the plurality of data groups stored in the storage unit, and focal coordinates are selected. Data update means for deleting the plane image data closest to the average value of the focal coordinates in all the plane image data belonging to the specific data group, and storing the new plane image data in a vacant storage area ,
The predetermined time is a value that is small enough that the difference between the captured image captured by the camera and the image displayed on the display means does not cause a sense of incongruity due to a delay in data transfer via the communication means. A robot system characterized by being set . A robot system comprising a robot having a camera at the tip of an arm, and a control device disposed remotely from the robot via communication means,
The control device includes:
A plane that creates plane image data centered on the focal coordinates by associating the captured image captured by the camera and the focal coordinates of the captured image when the captured image is captured and the direction in which the captured image was captured. Image data creation means;
Storage means for grouping a plurality of the plane image data having the focal coordinates approximated in a preset setting range from the plane image data created by the plane image data creation means and storing them as a plurality of data groups;
When the captured image captured by the camera cannot be acquired within a predetermined time, the field plane captured by the camera is moved in the optical axis extending direction from the planar image data stored in the storage means. Plane image data of the intersecting image that intersects the field plane when selected as the selected plane image data, and the focal coordinates of the intersecting image based on the selected selected plane image data and the direction in which the intersecting image was captured are selected. An image composition unit that projects the projected image onto the field plane as it is when the crossed image is captured to create a composite image, and stores the selected plane image data in the storage unit with a selection time selected.
When the captured image captured by the camera cannot be acquired within the predetermined time, the combined image combined by the image combining means is displayed , and the captured image captured by the camera is displayed within the predetermined time. Display means for displaying an actual image taken by the camera when acquired ,
When a storage area for storing new plane image data is insufficient in the storage unit, a specific data group to which the new plane image data belongs is selected from a plurality of data groups stored in the storage unit, and an imaging time is selected. Data update means for deleting the oldest plane image data from all the plane image data belonging to the specific data group, and storing the new plane image data in an empty storage area ,
The predetermined time is a value that is small enough that the difference between the captured image captured by the camera and the image displayed on the display means does not cause a sense of incongruity due to a delay in data transfer via the communication means. A robot system characterized by being set .

Images