JP5213183B2 - Robot control system and robot control program - Google Patents

Description

  The present invention relates to a robot control system and a robot control program for controlling a robot capable of performing an object transfer operation.

  In recent years, housework support robots (see, for example, Patent Document 1) capable of cleaning and moping have come to be used in ordinary homes. In addition, since a robot that can carry an object has been developed, it is considered that the day when a robot capable of carrying an object for use in a general home will be developed.

JP 2006-331434 A

"NyARToolKit", [online], [Search June 16, 2009], Internet <URL: http://sourceforge.jp/projects/nyartoolkit/> "Efficient graph-based image segmentation", International Journal of Computer Vision, vol. 59, no. 2, pp. 167-181, 2004 "Fast visual search using focused color matching-active search", System and Computers In Japan, vol. 31, no. 9, pp. 81-88, 2000

  Accordingly, the object of the present invention is to provide a robot control system that can suitably control a robot that can carry objects and a robot control program that can realize (construct) such a robot control system by using the robot control system. There is to do.

  In order to solve the above-described problems, a robot control system according to the present invention includes an imaging unit that repeatedly generates image data of a robot use space that is a space for functioning a robot capable of performing an object transfer operation, and the imaging unit includes: Based on each image data to be generated, a display means for displaying an image showing the current state of the robot usage space on the screen, and an image segmentation process is performed on the image data generated by the photographing means, whereby the image data Segmentation means for dividing the image shown in FIG. 2 into a plurality of segments, and user interface means for allowing the user to specify a transport object that is an object to be transported by the robot, Processing by the segmentation means when a specified range specification operation is performed By using the results, an image composed of a segment group in which a portion of a predetermined ratio or more is included in the range specified by the range specifying operation from the images displayed on the screen is used as the transport object image. User interface means for performing a search process and determining that an object whose image at that time is the conveyance object image is designated as the conveyance object; and the conveyance object searched by the user interface means Robot control means for causing the robot to carry out the carrying work on the carrying object using an image.

That is, the robot control system of the present invention is a system that displays an image showing the current state of the robot use space (hereinafter referred to as a camera image) on the screen based on the image data from the photographing means, and also the camera image. When the user designates a certain range on the screen on which is displayed, the image segmentation processing result for the camera image is used, and a portion of the camera image includes a portion of a predetermined ratio or more within the range designated by the user. This is a system that searches for an image composed of a group of segments as a transport object image (an image at the current time of an object designated by the user as a transport object).

  Therefore, in this robot control system, when the user designates an object displayed on the screen in a form that does not overlap with another object as a transport object, the object is displayed on the screen in a form that overlaps with another object. In any case where the user designates the object as the transport object, the transport object image is accurately searched (specified).

  If the transport object image can be accurately searched, information (information on the current position of the transport object) necessary for causing the robot to perform the transport operation of the transport object can be obtained based on the transport object image. Therefore, according to the robot control system of the present invention, it is possible to suitably control a robot capable of carrying an object.

  In realizing the robot control system of the present invention, as the robot control means, “from several continuous segments included in the image displayed on the screen using the processing result by the segmentation means”. A plurality of comparison control images are specified, a comparison control image most similar to the transport object image is searched from the specified plurality of comparison control images, and the position of the searched comparison control image on the screen From the above, “means for calculating current position information of the object to be transported in the robot use space” used for controlling the robot can be employed. If such a robot control means is adopted, a robot control system capable of accurately grasping the current position (current position information) of the object to be transported (less likely to lose the object to be transported) can be realized. become.

  The robot control system of the present invention can be realized as a type of system in which the transfer location (transfer destination) of the transfer object is designated / set in advance, or the transfer control location of the transfer object is specified for each transfer operation. System (“the user interface means is means for allowing the user to specify the transport object and a transport location where the transport object is to be transported, and the robot control means It is also possible to realize a system that is a means for causing the robot to perform a transport operation for transporting the transport object to the transport location using an image.

  When the robot control system according to the present invention is realized as the latter type, as a user interface means, “a transport location designation icon that allows a drag-and-drop operation after searching for the transport object image is displayed. When a drag-and-drop operation is performed on the transfer location designation icon displayed on the screen, a location in the robot use space corresponding to the drop position of the transfer location designation icon is designated as the transfer location. "Means to judge" can be adopted. Further, as means for user interface, “means for displaying an icon of the transport object image as the transport location designation icon” may be employed.

  In addition, there is a case where “there is no room for placing the object to be transported at the designated transport location even though the transport location designation icon is dropped at a position not overlapping with the image of another object”. In order to avoid such a situation, as a user interface means, “during the drag-and-drop operation of the transport location designation icon, the size of the transport location designation icon corresponds to the display position of the transport location designation icon. Means having a size changing function for changing the size of the image of the conveyance object when the conveyance object is present at a position in the robot use space may be employed.

  In addition, in order to prevent the user from feeling uncomfortable due to a subtle change in the size of the transport location designation icon during the drag-and-drop operation of the transport location designation icon, the above size change is performed as a user interface means. In addition to means having a function, "means for turning on / off the size changing function according to the posture of the photographing means with respect to the robot use space" may be employed.

  Further, in order to make the system easy to set up, the robot control system according to the present invention is changed to “the image data based on the shape information indicating the shape of the specific part of the robot and the image data generated by the photographing means”. Conversion information for obtaining a coordinate system coordinate having a coordinate plane parallel to the traveling surface of the robot, corresponding to each pixel, in the robot use space, from the position of each pixel in the image indicated by Conversion information calculating means for calculating the image information, and the robot control means analyzes the image data generated by the photographing means using the conversion information calculated by the conversion information calculation means, whereby the robot Means for calculating the current position information of the robot and the current position information of the object to be transported in the robot use space. That "it is also possible to be realized as a system.

  Then, the robot control program of the present invention uses the computer to display the robot on the screen based on image data from a digital camera directed to a robot use space that is a space for functioning a robot capable of carrying an object. Display means for displaying an image showing the current state of the use space, and segmentation means for dividing the image indicated by the image data into a plurality of segments by performing image segmentation processing on the image data from the digital camera. User interface means for allowing a user to designate a transport object that is an object to be transported by the robot, and when a range designation operation for designating a certain range on the screen is performed, by the segmentation means It is displayed on the screen using the processing result. From the image, a process of searching for an image composed of a segment group including a portion of a predetermined ratio or more within the range specified by the range specifying operation is performed as a transport target image, and the image at that time is the transport target User interface means for determining that an object that is an object image has been designated as the transport object, and transport work for the transport object using the transport object image searched by the user interface means. The apparatus can be made to function as an apparatus including a robot control means to be performed by the robot.

  Therefore, by using this robot control program, it is possible to realize a robot control system (a system corresponding to the robot control system according to claim 1) capable of suitably controlling a robot capable of carrying an object.

  According to the present invention, there is provided a robot control system that can suitably control a robot that can carry an object, and a robot control program that can realize (construct) such a robot control system by using the robot control system. I can do it.

1 is a configuration diagram of a robot control system according to a first embodiment of the present invention. It is an external view of the robot controlled by the robot control system according to the first embodiment. It is explanatory drawing of the processing content of the image segmentation process which the control apparatus in the robot control system which concerns on 1st Embodiment performs. It is a flowchart of the main process which the control apparatus in the robot control system which concerns on 1st Embodiment performs. It is the figure which showed an example of the image displayed on a touch screen during execution of the main process (conveyance instruction reception process). It is a figure for demonstrating the content of the D & D operation response process performed within the main process (conveyance instruction reception process). It is a flowchart of the robot control process which the control apparatus in the robot control system which concerns on 1st Embodiment performs. It is explanatory drawing of the color histogram which the control apparatus in the robot control system which concerns on 1st Embodiment calculates. It is a flowchart of the 2nd D & D operation response process which the control apparatus in the robot control system which concerns on 2nd Embodiment of this invention performs.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<< First Embodiment >>
First, the outline of the robot control system 10 according to the first embodiment of the present invention will be described with reference to FIGS. 1 and 2.

  As shown in FIG. 1, the robot control system 10 according to the present embodiment includes a robot 15, a control device 11, a camera 12 and a touch screen display 13 connected to the control device 11 with cables. I have.

  The robot 15 included in the robot control system 10 is a robot obtained by modifying iRobot Create (trademark of iRobot Corporation) marketed as a robot platform so that wireless control by Bluetooth (trademark of Bluetooth-SIG Inc.) is possible. is there.

  As shown in FIG. 2, at the center of the upper surface of the robot 15, a plate-like robot marker on which a marker graphic having a shape having no rotational symmetry is drawn is in the vertical direction (vertical direction in the figure). And the front-rear direction of the robot 15 (vertical direction in the figure), and the plate surface and the travel surface of the robot 15 (the floor surface of the room in which the robot 15 performs the object transfer operation; hereinafter referred to as the robot travel surface) It is attached in a posture that is parallel to the notation.

  The camera 12 (FIG. 1) is a device (a device called a PC camera, a Web camera, or the like) that can periodically output image data (color image data) about a subject (photographing target). At the time of setting up the robot control system 10, the camera 12 is fixed to a position near the ceiling of the room where the robot 15 is to carry the object in such a posture that the state / situation of the floor of the room can be photographed obliquely from above. Is done. In the present embodiment, “object transfer work” means “the object on the floor is moved by moving the object in the direction of the object while being in contact with the object on the floor”. Work to do.

  The touch screen display 13 is a display device (a display commercially available as a peripheral device of a computer) including a touch screen 14 that can be operated with a finger. The touch screen display 13 has a built-in speaker.

  The control device 11 controls the camera 12, the touch screen display 13 and the robot 15 in an integrated manner, thereby controlling the robot control system 10 as follows: “The user performs a desired“ object transport operation ”by operating the touch screen 14. This is a device that functions as a “system that can be performed by the robot 15”.

The control device 11 includes a robot control program, which is a program prepared for the system, on a computer having a communication function by Bluetooth, and a marker graphic indicating the shape (size and vertical direction of each part) of the marker graphic described above. The information is installed from a portable recording medium (not shown).

  Next, the operation of the robot control system 10 will be described.

  In the following description, a room in which the robot 15 performs an object transfer operation is referred to as a robot use space, and is referred to as a Z-axis that matches the optical axis of the lens of the camera 12, the vertical direction of the camera 12, and the horizontal direction. A three-dimensional orthogonal coordinate system having a parallel Y axis and X axis is referred to as a camera coordinate system. Further, a two-dimensional orthogonal coordinate system having a coordinate axis that coincides with the left side of an image photographed by the camera 12 (an image in which image data from the camera 12 represents its contents) and a coordinate axis that coincides with the upper side of the image. This is expressed as an image coordinate system, and a two-dimensional orthogonal coordinate system (having a coordinate plane that coincides with the robot traveling plane) set on the robot traveling plane is denoted as a control coordinate system.

  When the robot control program is started, the control device 11 causes the touch screen display 13 to display an image based on each image data from the camera 12 (each image taken by the camera 12 is sequentially displayed on the touch screen 14. Display process). Further, the control device 11 is in a state of periodically executing the robot coordinate calculation process and the image segmentation process.

The robot coordinate calculation process periodically executed by the control device 11 analyzes the image data from the camera 12 by using the above-described marker graphic information, so that the robot marker (more precisely, the marker graphic on the robot marker) is analyzed. This is a process of grasping the position and orientation and calculating the following two pieces of information based on the grasp result.
Robot coordinates that are the coordinates (control coordinate system coordinates) of “a point obtained by orthogonally projecting the center point of the robot marker onto the coordinate plane (robot traveling plane) of the control coordinate system”. Direction vector) which is an orthogonal projection of a vector pointing in the upward direction on the coordinate plane of the control coordinate system.

  The robot coordinate calculation process is a process for calculating a “coordinate conversion matrix for converting image coordinate system coordinates to control coordinate system coordinates” when a robot marker is detected for the first time. Also, the robot control program helps implement an augmented reality application in which the part for causing the control device 11 to execute this robot coordinate calculation process is ported to NyARtoolkit (ARToolKit is a trademark of Java (trademark of Sun Microsystem, Inc., USA)). The program is created using a class library (see Non-Patent Document 1).

  The image segmentation process periodically executed by the control device 11 is characterized by an image represented by image data from the camera 12 / an image displayed on the touch screen 14 (hereinafter referred to as a camera image). This is a process of dividing into a plurality of segments that can be regarded as different.

  In this image segmentation process, for example, when the camera image is the one shown in FIG. 3A, the camera image is divided into segment groups as shown in FIG. 3B. . That is, the image segmentation process divides an image relating to each object in the camera image into a plurality of segments. Note that the image segmentation process actually executed by the control device 11 according to the present embodiment is a process of segmenting the camera image by an algorithm developed by P. F. Felzenszwalb et al. (See Non-Patent Document 2).

When the robot control program is activated, the control device 11 also starts a process of monitoring whether the information calculation by the robot coordinate calculation process described above is successful (the robot marker on the robot 15 is captured by the camera 12). Then, when the information calculation by the robot coordinate calculation process is successful, the control device 11 starts the main process of the procedure shown in FIG. 4 and first performs the conveyance instruction reception process (the process of steps S101 to S106). Do.

  In this conveyance instruction reception process, the range designation operation and the drag-and-drop operation are performed to allow the user to perform an object (hereinafter referred to as a conveyance object) to be a conveyance work target by the robot 15 and the conveyance thereof. This is a process for designating a place (conveyance destination).

  As illustrated, the control device 11 that has started the transport instruction receiving process first executes a range specifying operation response process (step S101).

  This range designation operation response process is a process for displaying on the touch screen 14 a rectangular frame indicating the range being designated by the range designation operation / the designated range in response to the range designation operation by the user. The range designation operation is a movement of a finger on the touch screen 14 from one vertex position to a diagonal vertex position of a rectangular area (rectangular area including the whole / most portion of the image of the object to be conveyed) to be designated as a range. It is an operation to be performed.

  When the range specifying operation by the user is completed (when the user's finger is separated from the touch screen 14), the control device 11 stores the range specified by the range specifying operation (hereinafter referred to as a specified range). From this, the range designation operation response process (step S101) is terminated.

  And the control apparatus 11 searched and searched all the segments in which the part more than the predetermined ratio was contained in the designated range based on the image data of the camera image at that time, the latest image segmentation processing result, and the designated range. A process (step S102) is performed to specify an image formed of the segment as an image at the current time of the conveyance target designated by the user (hereinafter referred to as a conveyance target image). The latest image segmentation processing result is the processing result of the most recent image segmentation processing. Further, the predetermined ratio is a ratio set in the robot control program.

  Thereafter, the control device 11 performs a process (step S103) for highlighting the conveyance target object image in the image displayed on the touch screen 14.

  As already described, the conveyance object image specified at the time of the processing of step S102 is an image composed of segments in which a portion of a predetermined ratio or more is included in the specified range (that is, only a portion less than the predetermined ratio is in the specified range). The image does not include segments that are not included in the image. Therefore, even if a part of the object other than the conveyance target is displayed within the range specified by the user during the process of step S101, only the conveyance target image is specified as the conveyance target image during the process of step S102. Will be. As a result, during the process of step S103, as illustrated in FIG. 5, only the conveyance object (in this case, the trash can) is highlighted.

  The control device 11 that has finished the process of step S103 (FIG. 4) performs a process (step S104) of generating a color histogram for the conveyance object image and storing it as a reference histogram (the application will be described later). The color histogram for the transport object image is information including a normalized histogram for each of the R, G, and B values of each pixel of the transport object image.

Thereafter, the control device 11 starts a D & D operation response process (step S105). First, an operation for designating a conveyance object image on the touch screen 14 (on the touch screen 14,
It is in a state where it is monitored that the operation of placing a finger on the portion where the conveyance object image is displayed is performed. Then, when an operation for designating a transport object image is performed, the control device 11 displays a transport location designation icon, which is an icon of the transport object image, as illustrated in FIG. 6A. After displaying on the object image, a process of actually responding to the drag-and-drop operation on the transport location designation icon (a process of changing the display position of the transport location designation icon according to the user's icon operation) is started.

  When the drag-and-drop operation on the transport location designation icon is completed (when the finger is released from the transport location designation icon / touch screen 14), the control device 11 is shown in FIG. As described above, after the highlight display of the conveyance object image is stopped, the D & D operation response process (step S105) is ended.

  The control device 11 that has finished the D & D operation response process uses the coordinate transformation matrix calculated by the robot coordinate calculation process to display the current position of the transport location designation icon (the image of the center position of the transport location designation icon at that time). The control coordinate system coordinates of the corresponding robot travel surface position are calculated from the coordinate system coordinates), and the calculation result is stored as the transport location coordinates (step S106).

  And the control apparatus 11 which finished the process of step S106 complete | finishes a conveyance instruction | indication reception process (process of step S101-S106). Although notation in the flowchart (FIG. 4) is omitted, the conveyance instruction reception process actually executed by the control device 11 is performed on the touch screen 14 before the user designates the conveyance object image. When an operation for changing the shape of the displayed rectangular frame is performed, the process after step S102 is executed again. In other words, as a result of specifying a range that is too large / too small, the conveyance instruction reception process highlights a part other than the conveyance target or only a part of the conveyance target. In this case, the user can redo the range specification.

  In addition, the conveyance instruction reception process is performed within a predetermined time after the completion of the drag-and-drop operation with respect to the transfer location designation icon. It is also processing to accept. In addition, even after the completion of the conveyance instruction reception process, the main process is an instruction to change the conveyance location coordinates (drag and drop operation with respect to the conveyance location designation icon) before the completion of the robot control process (step S110; details will be described later). It is processing to accept.

  Control device 11 that has completed the above-described transport instruction acceptance process (control that detects that a predetermined time has passed without the drag-and-drop operation being resumed after the drag-and-drop operation on the transport location designation icon is completed) In step S110, the apparatus 11) executes the robot control process according to the procedure shown in FIG.

  That is, the control device 11 is in a state of periodically executing the transport object coordinate calculation process (step S201) while appropriately performing other processes (image segmentation process or the like).

  The transport object coordinate calculation process periodically executed by the control device 11 searches for an image most similar to the transport object image (hereinafter referred to as the most similar image) from the camera images, This is a process of calculating control coordinate system coordinates corresponding to the image coordinate system coordinates of the center point as conveyance object coordinates.

This transport object coordinate calculation process searches all the “images that are arranged on the camera image and are composed of almost the same number of segments as the transport object image” based on the latest image segmentation process result, Processing for searching for the most similar image by calculating the similarity between the color histogram of each searched image and the standard histogram (see step S104 in FIG. 4) by histogram intersection (see Non-Patent Document 3). It has become.

  Note that “an image composed of almost the same number of segments as the conveyance object image arranged on the camera image” means “a segment composed of a plurality of segments (or a single segment) arranged on the camera image”. The number of images in a predetermined range (in this embodiment, the range of the number of segments of the conveyance target image minus the number of segments of the conveyance target image + 1 in this embodiment) based on the number of segments of the conveyance target image. That is.

  The control device 11 that has completed the transport object coordinate calculation process (step S201) sets the similarity of the most similar image searched by the transport object coordinate calculation process to a similarity threshold (set in the robot control program). It is determined whether or not it is less than (a certain value) (step S202).

  When the similarity of the most similar image is equal to or greater than the similarity threshold (step S202; NO), the control device 11 indicates that the transport location coordinates indicate the position and the transport object coordinates indicate the position. It is determined whether or not the distance between the points (hereinafter referred to as the inter-coordinate distance) is less than a distance threshold (a value set in the robot control program) (step S203).

  When the inter-coordinate distance is equal to or greater than the distance threshold (step S203; NO), the control device 11 determines the robot based on the transport location coordinates, the robot coordinates at that time, the traveling direction vector, and the transport object coordinates. 15 is performed (step S204) to determine the content of the control to be executed. Although detailed description is omitted, the process of step S204 is performed for a short time such as “move a certain amount (˜several centimeters) in a certain direction” or “turn a certain direction”. In this process, the control to be completed is determined as the content of the control executed on the robot 15. The process of step S204 is a process of determining the traveling direction of the robot 15 without particularly considering the position of an object other than the conveyance target on the robot travel surface.

  After completing the process of step S204, the control device 11 performs control of the content determined by the process on the robot 15 (step S205), and then restarts the processes after step S201.

  If the control device 11 detects that the inter-coordinate distance has become less than the distance threshold while repeating the processing of the above procedure (step S203; YES), the robot control processing has ended normally. After storing (not shown) that the object transfer operation has been completed, this robot control process (the process of FIG. 7) is terminated.

  Further, the control device 11 abnormally ends the robot control process when the similarity of the most similar image becomes less than the similarity threshold before the inter-coordinate distance becomes less than the distance threshold (step S202; YES). After storing (not shown) that the conveyance object has been lost, the robot control process is terminated.

  Returning to FIG. 4, the operation of the control device 11 after the end of the robot control process will be described.

  The control device 11 that has finished the robot control process (step S110) deletes the transfer location designation icon from the touch screen 14 when the robot control process has been normally completed (step S111; normal completion) ( Step S112). And the control apparatus 11 starts the process after step S101 again.

  On the other hand, when the robot control process has ended abnormally (step S111; abnormal end), the control device 11 performs a transfer object re-designation request process (step S113). In this conveyance object re-designation request process, in order to request the user to re-designate the conveyance object, a warning icon (in this embodiment, an exclamation mark is shown in the triangle on the touch screen 14). This is a process of outputting a predetermined warning sound to the speaker of the touch screen display 13 after displaying the shape icon).

  After that, the control device 11 causes the user to respecify the conveyance object by performing the conveyance object redesignation operation acceptance process (step S114) having essentially the same content as the above-described processing of steps S101 to S104. (The range surrounding the object to be transported is specified again.) And the control apparatus 11 restarts the process after step S110, when re-designation of the conveyance target object by the user is completed.

  As described above, in the robot control system 10 according to the present embodiment, when an operation for designating a certain range on the touch screen 14 on which the camera image is displayed is performed, the image segmentation processing result for the camera image is displayed. Is used to perform a process of searching for an image composed of a segment group that includes a portion of a predetermined ratio or more in the range specified by the user in the camera image. Is specified as an image at the present time of the conveyance object designated by the user.

  Therefore, the robot control system 10 always displays the transfer object image (even when the user designates an object displayed on the touch screen 14 in a form overlapping with another object as the transfer object). In addition to a system that can accurately specify a range, the user can accurately specify a transport target image even if the user does not specify the range correctly (even if the range that completely includes the transport target is not specified). Will be.

  Further, the robot control system 10 determines the color histogram (reference histogram) of the conveyance object image specified as described above and the color of each image composed of approximately the same number of continuous segments as the conveyance object image in the camera image. By comparing with the histogram, an image most similar to the conveyance object image in the camera image is searched, and the position of the conveyance object being conveyed by the robot 15 is obtained based on the search result. It also has a function.

  As schematically shown in FIGS. 8A to 8D, the color histogram is originally information whose contents vary greatly depending on the object, and an image to be compared with the color histogram is rectangular. If it is not an image but an image composed of approximately the same number of continuous segments as the conveyance object image, an image similar to the conveyance object image can be searched without being affected by the background component. Therefore, the robot control system 10 according to the present embodiment is a system that can accurately grasp the position of the object to be transported (a system that can accurately control the robot 15), compared to a system that employs another method for image search. Will be.

  In addition, an existing system that calculates object coordinates from camera images needs to perform a process (what is called a camera calibration process or the like) for calculating an object equivalent to the coordinate transformation matrix described above during setup. On the other hand, the robot control system 10 calculates a coordinate transformation matrix from the position and orientation of the robot marker attached to the upper surface of the robot 15 in order to detect the position of the robot 15. It is a system. Therefore, it can be said that the robot control system 10 is a system that is easier to set up than an existing system (a system that can be operated as long as the camera 12 is fixed).

<< Second Embodiment >>
Hereinafter, the configuration and operation of the robot control system 10 according to the second embodiment of the present invention will be described using the same reference numerals as those used when describing the robot control system 10 according to the first embodiment.

  The robot control system 10 according to the present embodiment is configured so that the second D & D operation response process of the procedure shown in FIG. 9 is performed in step S105 of the main process (FIG. 4). 10 is an improved system.

  That is, the control device 11 (hereinafter referred to as the second control device 11) in the robot control system 10 according to the second embodiment completes the process of calculating and storing the reference histogram (see step S104 in FIG. 4). The second D & D operation response process is started.

  Then, the second control device 11 that has started the second D & D operation response process first monitors a state in which an operation for designating a conveyance object image on the touch screen 14 is performed (the processing / determination in step S301 is performed). Repeated state).

  When an operation for designating a transport object image is performed (step S301; YES), the second control device 11 displays a transport location designation icon, which is an icon of the transport object image, on the transport object image ( Step S302). Thereafter, the control device 11 calculates a depression angle of the camera 12 (an angle formed by the optical axis of the lens of the camera 12 and the robot traveling surface) from the coordinate transformation matrix, and the calculated depression angle is a preset depression angle threshold (this In the embodiment, a process of determining whether the angle is 50 degrees or less (step S303) is performed.

  Then, when the depression angle of the camera 12 is not less than or equal to the depression angle threshold (step S303; NO), the second control device 11 responds to the drag-and-drop operation with respect to the conveyance location designation icon and sets the conveyance location designation icon. An icon display position changing process (step S304) for changing the display position is performed. The second control device 11 ends the icon display position changing process when the drag-and-drop operation on the transport location designation icon is completed. Then, the second control device 11 stops the highlight display of the conveyance object image (step S306), and then ends the second D & D operation response process (the process of FIG. 9).

  On the other hand, when the depression angle of the camera 12 is equal to or smaller than the depression angle threshold (step S303; YES), the second control device 11 performs an icon display position / display size change process (step S305).

  This icon display position / display size changing process is performed when the transport location designation icon is dragged to the new display position, using the coordinate transformation matrix, the initial position of the transport object image, etc. Obtain the image size of the transport object when the transport object actually exists at a position on the surface, and display the transfer location designation icon that has been enlarged or reduced to the same size as the calculated image size at the new display position It is processing to do.

  The second control device 11 ends the icon display position / display size changing process when the drag-and-drop operation on the transport location designation icon is completed. Then, the second control device 11 stops the highlight display of the conveyance object image (step S306), and then ends the second D & D operation response process.

In short, when the depression angle of the camera 12 is relatively small, an object / region existing on the front side of the robot traveling surface (side closer to the camera 12) exists on the touch screen 14 on the far side of the robot traveling surface. It is displayed much larger than the same size object / area. Therefore, in the robot control system 10 according to the first embodiment that always displays the transfer location designation icon of the same size, “although the transfer location designation icon is dropped at a position not overlapping the image of another object. In addition, there is no room where the transport object can be placed at the designated transport location ”,“ Despite being a place where the transport target can actually be placed, the transport location designation icon is at a position corresponding to the location. If the icon is dropped, the transfer location designation icon may overlap with the image of another object.

  On the other hand, when the depression angle of the camera 12 is equal to or smaller than the depression angle threshold, the robot control system 10 according to the present embodiment determines the size of the transport location designation icon displayed on the touch screen 14 according to the display position. (The image size of the object to be conveyed when the display position is conveyed to the designated position).

  Therefore, the robot control system 10 is a system that does not cause the above-described problems that may occur in the robot control system 10 according to the first embodiment.

  The robot control system 10 according to the second embodiment is configured as a system that does not change the size of the transfer location designation icon when the depression angle of the camera 12 is relatively large. Is relatively large, the amount of change in the image size due to the position on the robot traveling surface is small, so that the above-described problems hardly occur and the depression angle of the camera 12 is relatively large. In addition, if the size of the transport location designation icon is changed, the user may feel a sense of incongruity due to a subtle change in the size of the transport location designation icon during the drag and drop operation of the transport location designation icon. This is because it was understood.

<Deformation>
The robot control system 10 according to the first and second embodiments described above can perform various modifications. For example, the robot control system 10 according to each embodiment described above calculates a coordinate transformation matrix, robot coordinates, and the like by analyzing a camera image using marker graphic information indicating the shape of the marker graphic. However, the robot control system 10 according to each embodiment can be transformed into a system that calculates a coordinate transformation matrix or the like based on information indicating the shape (appearance) of the robot 15 itself.

  Further, although the setup work becomes a more complicated system, the robot control system 10 according to each embodiment does not calculate a coordinate transformation matrix from the position and orientation of the robot marker (or the robot 15 itself) (setup). Sometimes, the work for calculating the coordinate transformation matrix can be transformed into a system).

  The robot control system 10 according to each embodiment tracks the object to be transported by processing different from that described above (searches for an image most similar to the object image to be transported from the camera image, and based on the search result) It is also possible to modify the system to specify the position of the conveyance object. For example, the robot control system 10 according to each embodiment can be transformed into a system that calculates the similarity between the color histogram and the reference histogram by a method other than the histogram intersection. In addition, the robot control system 10 according to each embodiment can be modified to a system that tracks a conveyance object by comparing a color histogram of a rectangular image at various positions in a camera image with a reference histogram. However, when such deformation is performed, the tracking performance of the conveyance target object deteriorates due to the influence of the background component. Therefore, when the robot control system 10 according to each embodiment is modified, the conveyance object is tracked by comparing the color histogram of the image obtained by combining several consecutive segments with the reference histogram. It is desirable.

  The robot control system 10 according to each embodiment includes a system in which the robot 15 transports a designated transport object to a specific location (a system that does not require designation of a transport location), and a robot 15 that can carry an object. It can also be transformed into a system. In addition, the robot control system 10 according to each embodiment is a system that performs segmentation of a camera image by processing of contents (algorithms) different from those described above. It can also be transformed into a system that displays an icon) that is not an icon of the image.

  After searching for the most similar image, the robot control system 10 according to each embodiment changes the transport object image to the searched most similar image, or when searching for the most similar image, It is also possible to transform the system to search only for “images that are arranged on the camera image and are composed of almost the same number of segments as the conveyance target images” located in the vicinity of the similar images. In addition, the robot control system 10 according to each embodiment includes a system that performs a range designation operation and a drag-and-drop operation with a mouse, a system in which the robot 15 is not a component (a robot that has not been developed for its own system) It goes without saying that the system may be transformed into a control system.

DESCRIPTION OF SYMBOLS 10 ... Robot control system 11 ... Control apparatus 12 ... Camera 13 ... Touch screen display 14 ... Touch screen 15 ... Robot

Claims (5)

Photographing means for repeatedly generating image data about a robot use space, which is a space for functioning a robot capable of performing an object transfer operation;
Display means for displaying an image showing the current state of the robot use space on the screen based on each image data generated by the photographing means;
Segmentation means for dividing the image indicated by the image data into a plurality of segments by performing image segmentation processing on the image data generated by the photographing means;
User interface means for causing a user to designate a transport object that is an object to be transported by the robot, and processing performed by the segmentation means when a range designation operation for designating a certain range on the screen is performed Using the result, an image composed of a segment group in which a portion of a predetermined ratio or more is included in the range specified by the range specifying operation from the images displayed on the screen as a transport target image. User interface means for performing a search process and determining that an object whose image at that time is the conveyance object image is designated as the conveyance object;
Robot control means for causing the robot to perform a transport operation on the transport object using the transport object image searched by the user interface means;
The user interface means comprises:
After searching for the transport object image, an iconized version of the transport object image is displayed on the screen as a transport location designation icon that can be dragged and dropped, and dragged and dropped on the transport location designation icon. when the drop operation is performed by the user, Ru means der to determine that the location of the robotic space corresponding to the drop position of the conveying locations specified icon is designated as the transport where to transport the object to be transported With
During the drag-and-drop operation of the transfer location designation icon, the size of the transfer location designation icon is set so that the transfer object exists at a position in the robot use space corresponding to the display position of the transfer location designation icon. A means having a size changing function for changing to the size of the image of the conveyance object in the case of
The robot control means is
A robot control system, characterized in that it is means for causing the robot to perform a transport operation for transporting the transport object to the transport location using the transport object image. The robot control means is
Using the processing result of the segmentation means, a plurality of comparison images consisting of several consecutive segments included in the image displayed on the screen are identified, and the plurality of identified comparison controls A comparison image that is most similar to the conveyance object image is searched from the image, and the conveyance in the robot use space is used for controlling the robot from the position of the searched comparison image on the screen. 2. The robot control system according to claim 1, wherein the robot control system is means for calculating current position information of the object. The user interface means comprises:
3. The robot control system according to claim 1, wherein the size change function is turned on / off according to a posture of the photographing unit with respect to the robot use space. Use of the robot corresponding to each pixel from the position of each pixel in the image indicated by the image data based on the shape information indicating the shape of the specific part of the robot and the image data generated by the photographing unit Conversion information calculating means for calculating conversion information for obtaining a coordinate system coordinate having a coordinate plane parallel to the traveling surface of the robot with respect to a position in space;
The robot control means is
By analyzing the image data generated by the photographing unit using the conversion information calculated by the conversion information calculation unit, the current position information of the robot in the robot use space, which is used for controlling the robot, and The robot control system according to any one of claims 1 to 3 , wherein the robot control system is means for calculating current position information of the conveyance object. Computer
Display means for displaying an image showing the current state of the robot use space on a screen based on image data from a digital camera directed to the robot use space, which is a space for functioning a robot capable of carrying objects; ,
Segmentation means for dividing an image indicated by the image data into a plurality of segments by performing an image segmentation process on the image data from the digital camera;
User interface means for causing a user to designate a transport object that is an object to be transported by the robot, and processing performed by the segmentation means when a range designation operation for designating a certain range on the screen is performed Using the result, an image composed of a segment group in which a portion of a predetermined ratio or more is included in the range specified by the range specifying operation from the images displayed on the screen as a transport target image. User interface means for performing a search process and determining that an object whose image at that time is the conveyance object image is designated as the conveyance object;
A robot control program that functions as an apparatus including a robot control unit that causes the robot to perform a transport operation on the transport target using the transport target object image searched by the user interface unit,
The user interface means comprises:
After searching for the transport object image, an iconized version of the transport object image is displayed on the screen as a transport location designation icon that can be dragged and dropped, and dragged and dropped on the transport location designation icon. when the drop operation is performed by the user, Ru means der to determine that the location of the robotic space corresponding to the drop position of the conveying locations specified icon is designated as the transport where to transport the object to be transported And
Also ,
During the drag-and-drop operation of the transfer location designation icon, the size of the transfer location designation icon is set so that the transfer object exists at a position in the robot use space corresponding to the display position of the transfer location designation icon. A means having a size changing function for changing to the size of the image of the conveyance object in the case of
The robot control means is
A robot control program, comprising: means for causing the robot to perform a transport operation for transporting the transport object to the transport location using the transport object image.

Images