JP2019063953A - Work system, method for controlling work system and program - Google Patents

Abstract

To provide a work system that can reduce possibility for an operator to select a wrong movement destination for an object article by presenting a movement destination to the operator.SOLUTION: A work system includes: an imaging part capable of imaging a movement destination for an object article; a feature point specification part for specifying a feature point for specifying the movement destination; and a synthesis part for synthesizing an image including the movement destination that is imaged by the imaging part and an image that indicates a position of the movement destination based on the feature point.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a work system, a control method of the work system, and a program.

  Patent Document 1 discloses a technique for detecting an anthropomorphic motion by an operator and converting the detected motion into a plurality of signals to drive a robot device.

JP, 2015-168056, A

When manually operating a manipulator such as a robot device, the operator may not know where the target article is to be moved.
An object of the present invention is to provide a working system, a working system control method, and a program that solve the problems described above.

  According to a first aspect of the present invention, a work system includes an imaging unit capable of photographing a destination of a target article, a feature point identifying unit for identifying a feature point for identifying the destination, and the imaging unit. And a combining unit configured to combine the captured image including the moving destination and the image indicating the position of the moving destination based on the feature points.

  According to a second aspect of the present invention, the work system according to the first aspect may include a video signal output unit that outputs an image synthesized by the synthesizing unit.

  According to a third aspect of the present invention, the working system according to the first aspect or the second aspect includes a display unit for receiving the image synthesized by the synthesizing unit, and the display unit projects the received image. It may be

  According to a fourth aspect of the present invention, a control method of a work system is a control method of a work system including an imaging unit capable of photographing a movement destination of a target article, and the feature point for specifying the movement destination And combining the image including the moving destination captured by the imaging unit and the image indicating the position of the moving destination based on the feature point.

  According to a fifth aspect of the present invention, the program specifies a feature point for specifying the movement destination on a computer of a work system including an imaging unit capable of photographing the movement destination of the target article. Combining an image including the moving destination photographed by an imaging unit and an image indicating the position of the moving destination based on the feature point.

  According to each aspect of the present invention, the work system can reduce the possibility of the operator mistaking the target article's destination by presenting the destination to the operator.

It is a schematic diagram showing composition of a work system by a 1st embodiment. It is a schematic block diagram showing composition of a controller by a 1st embodiment. It is a figure which shows the example of the mark image by 1st Embodiment. It is a schematic block diagram showing composition of a controller. It is a flowchart of the work system by 1st Embodiment. It is a schematic diagram showing composition of a work system by a 2nd embodiment. It is a 1st figure which shows the example of the mark image by 2nd Embodiment. It is a 2nd figure which shows the example of the mark image by 2nd Embodiment. It is a schematic block diagram showing composition of a computer concerning at least one embodiment.

First Embodiment
Hereinafter, embodiments will be described in detail with reference to the drawings.
FIG. 1 is a schematic view showing the configuration of a work system according to the first embodiment.
The work system 1 includes an input / output glove 10, a head mount display 20 (an example of a display unit), a master controller 30, a slave controller 40, and a work robot 50. The input / output glove 10 and the head mounted display 20 are attached to the operator O. The work robot 50 is installed remotely to the operator O. The master controller 30 is electrically connected to the input / output glove 10 and the head mounted display 20. The slave controller 40 is electrically connected to the work robot 50. The master controller 30 and the slave controller 40 are configured to be able to communicate via the network N.

  A work system 1 according to a first embodiment of the present invention is applied to a picking system for transporting articles from a warehouse. The picking system is a system for transporting a bin for storing articles to a work space where the work robot 50 is installed, by a conveyor or a transport robot. In the picking system according to the first embodiment of the present invention, the operator O remotely operates the work robot 50 to place the target articles stored in the bin at other positions such as the sorting box Q used for sorting before shipping. When transferring, the destination of the move is presented to the operator O. Sorting box Q means, for example, before packing into a large-size cardboard used for actual delivery, when the target item for which an order has been received does not fit in the standard-size cardboard used when sending it to the purchaser. It is a temporary box used to collect target articles in one place. In other embodiments, work system 1 may be applied to systems other than a picking system. For example, the work system 1 includes other remote work such as winding, sorting, unpacking, packing, inspection, and assembly (including not only in the warehouse but also in remote offices, supermarkets, hospitals, libraries, laboratories, etc.) It may be used.

  The input / output glove 10 is a glove-like input / output device attached to the hand of the operator O. The input / output glove 10 includes an attitude sensor that detects the attitude of the operator O, and a feedback device that feeds back the operation of the work robot 50. The input / output glove 10 outputs a signal generated by the posture sensor to the master controller 30 as an operation signal. The posture sensor is configured to be able to calculate at least the joint angle of the operator O. The attitude sensor may detect the movement of the operator O by motion capture. The feedback device is an element (such as an actuator or an electrode) that can reproduce the tactile sensation of the object touched by the manipulator 51 of the working robot 50. The feedback device operates in accordance with the feedback signal input from the master controller 30.

The work robot 50 includes a manipulator 51 and a camera 52 (an example of an imaging unit).
The manipulator 51 has a plurality of links connected to one another via a plurality of joints. The manipulator 51 adjusts the angle of each joint by driving an actuator provided for each joint in accordance with a control signal input from the slave controller 40. The joints of the manipulator 51 and the joints of the operator O do not necessarily correspond one to one. For example, the manipulator 51 may include less than five fingers (combination of links and joints corresponding to the fingers). Also, for example, the manipulator 51 may include two or more elbows (joints corresponding to elbows). The manipulator 51 is provided with a tactile sensor and a posture sensor. The tactile sensor is provided on the surface of the manipulator 51 (in particular, at the tip of the link corresponding to the finger), and detects the tactile sensation of the touched object. A posture sensor is provided for each joint of the manipulator 51, and detects the angle and angular velocity of each joint. Examples of the posture sensor include a sensor (such as a rotation number sensor or a stroke sensor) that measures the amount of movement of the actuator.
The camera 52 is connected to the manipulator 51 via a joint. The camera 52 outputs video information representing a captured image to the slave controller 40. The attitude of the camera 52 is adjusted in accordance with the control signal input from the slave controller 40. That is, when the actuator connected to the camera 52 and the manipulator 51 is driven according to the control signal, the attitude of the camera 52 is adjusted.

The head mounted display 20 is a device for presenting the image taken by the camera 52 to the operator O by being attached to the head of the operator O. As a method of presenting an image, there is a method of displaying an image on a display facing the eye of the operator, and a method of projecting an image by the projector to the eye of the operator. The head mounted display 20 presents a video according to the video signal input from the master controller 30. The head mounted display 20 includes an attitude sensor (for example, an acceleration sensor, a gyroscope, etc.) that detects the attitude of the head mounted display 20. The head mounted display 20 outputs an attitude signal indicating the detection result of the attitude sensor to the master controller 30.
The coordinates in the space photographed by the camera 52 and the coordinates in the space projected by the head mounted display 20 are standardized by performing calibration before the operation of the work system 1 starts. The shooting direction of the camera 52 is changed according to the orientation of the head mounted display 20. That is, on the head mounted display 20, an image seen when the operator O wearing the head mounted display 20 changes the face direction at the position of the camera 52 is projected.

  The master controller 30 generates a control signal of the manipulator 51 of the working robot 50 based on the operation signal input from the input / output glove 10. Further, the master controller 30 generates a control signal for changing the position and the photographing direction of the camera 52 in accordance with the position and the orientation of the head mounted display 20. Further, the master controller 30 generates a feedback signal for causing the feedback device of the input / output glove 10 to reproduce the tactile sense based on the tactile sensation signal indicating the tactile sensation detected by the manipulator 51 from the working robot 50. Further, the master controller 30 receives a video signal of an image captured by the camera 52 from the slave controller 40, combines this video signal with a video signal of an image indicating a movement destination, and outputs the video signal to the head mounted display 20. That is, the master controller 30 is an example of a presentation device. The master controller 30 also generates a control signal of the camera 52 of the work robot 50 based on the attitude signal input from the head mounted display 20 and transmits the control signal to the slave controller 40.

  The slave controller 40 receives a control signal from the master controller 30, and drives the manipulator 51 and the camera 52 accordingly. The slave controller 40 also obtains signals from the tactile sensor, the attitude sensor, and the camera 52 and transmits the signals to the master controller 30.

FIG. 2 is a schematic block diagram showing the configuration of the master controller according to the first embodiment of the present invention.
The master controller 30 includes a signal reception unit 301, a video signal output unit 302, a tactile sensation signal conversion unit 303, a feedback signal output unit 304, a storage unit 305, an instruction input unit 306, a feature point identification unit 307, a synthesis unit 308, and an operation signal input. The unit 309 includes an operation signal converter 310 and a control signal transmitter 311.

The signal receiving unit 301 receives a video signal, a tactile sensation signal, and an attitude signal from the working robot 50 via the slave controller 40.
The video signal output unit 302 outputs the received video signal to the head mounted display 20. The video signal output unit 302 outputs a video signal of an image captured by the camera 52 combined by the combining unit 308 and a video signal of an image indicating a movement destination.
The tactile sensation signal conversion unit 303 converts the received tactile sensation signal into a feedback signal. That is, the tactile sensation signal conversion unit 303 generates a feedback signal for causing the feedback device to reproduce the tactile sensation of the object touched by the manipulator 51 of the working robot 50. The conversion from the tactile sensation signal to the feedback signal can be realized, for example, by inputting the tactile sensation signal into a previously obtained function.
The feedback signal output unit 304 outputs a feedback signal to the input / output glove 10.

  The storage unit 305 stores the posture of the work robot 50. The posture of the work robot 50 includes the position, angle, and angular velocity of each joint. In addition, the storage unit 305 associates and stores the target article I and the shape of the target article I in advance.

  The instruction input unit 306 receives an input of a work instruction from the picking system or the like. The work instruction includes information indicating the target item I to be the work target and information indicating the movement destination of the target item I. Examples of information capable of specifying a shape include three-dimensional shape data indicating a three-dimensional shape, two-dimensional image data captured from a plurality of viewpoints, feature amount information extracted from three-dimensional shape data and two-dimensional image data, etc. It can be mentioned.

  The feature point identification unit 307 identifies the coordinates of the corner of the sorting box Q. Specifically, the feature point identifying unit 307 sets a window in the image captured by the camera 52, and identifies a point at which the edge intersects within a range of angles within the window. Here, the window is a window indicating the range in which the sorting box Q may be displayed. Further, a point at which an edge in a window intersects within a certain angle range is a point at which an image of a portion of the range of the window is extracted in the image captured by the camera 52 and the edges intersect.

The synthesizing unit 308 synthesizes the video signal including the sorting box Q photographed by the camera 52 and the video signal including the image indicating the moving destination of the target article I to generate a composite video signal. Specifically, the combining unit 308 combines the video signal in which the sorting box Q among the plurality of video signals is captured with the video signal including the mark image indicating the moving destination of the target article I. As an example of the mark image, as indicated by a broken line in FIG. 3, the target article I is determined based on the coordinates of the angle P specified by the feature point specifying unit 307, for example, at each position at a fixed distance. What colored the image K of the shape similar to the outline of (highlight) is mentioned. Moreover, as an example of a mark image, it may be, for example, an image in the shape of a rectangular parallelepiped surrounding the target article I, or an arbitrary image showing a position determined based on the coordinates of the corner P of the sorting box Q It is also good.
The combining unit 308 specifies the target item I according to the order information. Specifically, when the purchaser accesses the website that can purchase the target item I through the terminal device and performs an operation of purchasing the target item I on the terminal device, the terminal device is the buyer In accordance with the operation of (1), information including the information specifying the target item I (for example, ID, name, etc.), the number of purchases of the target item I, the delivery destination of the target item I, etc. is transmitted to the management server of the website. When the management server receives information including the information specifying the target item I, the number of purchases of the target item I, the delivery destination of the target item I, etc. from the terminal device, it stores the date and time when the information was received and the received information. Do. The combining unit 308 acquires the target item I stored by the management server from the management server. The combining unit 308 identifies, in the storage unit 305, a target article similar to the target article I acquired from the management server, and acquires the shape of the target article I associated with the identified target article I. Thereby, the combining unit 308 can acquire the shape of the target article I, and can specify the contour of the target article I and the shape of a rectangular solid surrounding the target article I.

The operation signal input unit 309 receives an input of an operation signal from the input / output glove 10. The operation signal input unit 309 also receives an input of an attitude signal from the head mounted display 20.
The operation signal converter 310 converts the operation signal and the attitude signal into control signals. That is, the operation signal converter 310 generates a control signal for causing the work robot 50 to execute the same operation as the operator O. The conversion from the operation signal and the attitude signal to the control signal can be realized, for example, by inputting the operation signal and the attitude signal to a function obtained in advance. An operation signal of the input / output glove 10 is converted into a control signal of the manipulator 51, and an attitude signal of the head mounted display 20 is converted into a control signal of the camera 52.

  The control signal transmission unit 311 transmits the control signal of the manipulator 51 and the camera 52 converted by the operation signal conversion unit 310 to the slave controller 40.

FIG. 4 is a flowchart of the working system 1 according to the first embodiment of the present invention.
First, calibration between the coordinate system of the input / output glove 10 and the manipulator 51 of the working robot 50 is performed. At this time, calibration between the coordinate system of the head mounted display 20 and the coordinate system of the space in the movement destination of the target article I is simultaneously performed.
When the picking system transports the object to the work space, the picking system outputs a work instruction to the master controller 30.
The instruction input unit 306 of the master controller 30 receives an input of a work instruction from the picking system (step S1). Next, the signal receiving unit 301 receives, from the working robot 50 via the slave controller 40, the video signal captured by the camera 52 and the tactile sensation signal detected by the tactile sensor (step S2). The picking system moves the sorting box Q for sorting the target articles I stored in the bin M to the position of the work robot 50 according to the order information, and stops the sorting box Q. When the sorting box Q is stopped at a predetermined position, a signal indicating a stop is received from the control system of the picking system which has moved the sorting box Q to the position of the work robot 50, not shown. It is informed by

When the feature point identifying unit 307 receives a signal indicating stop from the control device of the picking system, the feature point identifying unit 307 identifies a point P indicating a corner (one of the feature points) of the sorting box Q (step S3). For example, the feature point identifying unit 307 sets a window in the image captured by the camera 52 including the sorting box Q, and identifies a point P where the edge of the image in the window intersects at an angle within a certain range. The point P is a point around the movement destination of the target item I.
The synthesizing unit 308 synthesizes the video signal including the sorting box Q photographed by the camera 52 and the video signal including the image indicating the moving destination of the target article I to generate a composite video signal (step S4). Specifically, the combining unit 308 sets a video signal including the sorting box Q among the plurality of video signals, and a video signal including a mark image indicating the movement destination of the target article I determined based on the coordinates of the corner P. Synthesize

The video signal output unit 302 outputs the video signal combined by the combining unit 308 to the head mounted display 20 (step S5). When a video signal is input, the head mounted display 20 presents an image to the operator O based on the video signal. At this time, the image displayed on the head mounted display 20 is, for example, an image shown in FIG.
The operator O performs an operation on the input / output glove 10 to move the target article I to the movement destination in the sorting box Q based on the mark image displayed by the head mounted display 20. When the operator O operates the input / output glove 10, the operation signal input unit 309 receives an input of an operation signal from the input / output glove 10 (step S6). The operation signal converter 310 converts the operation signal into a control signal of the manipulator 51 (step S7). Then, the control signal transmission unit 311 transmits the control signal of the manipulator 51 converted at step S7 to the slave controller 40 (step S8). Thereby, the slave controller 40 operates the manipulator 51 according to the control signal generated based on the operation of the operator O. Further, the attitude information of the head mounted display is transmitted to the slave controller 40. Thus, the slave controller 40 controls the attitude of the camera 52 such that the movement of the line of sight of the operator O coincides with the movement of the line of sight of the camera 52.
Therefore, the operator O can confirm the positional relationship between the manipulator 51 of the work robot 50 and the target item I, and the movement destination of the target item I. In addition, the operator O can feel the tactile feedback fed back by the tactile sensation signal converter 303. As a result, the operator O can accurately move the target item I to the destination without mistaking the destination.

Second Embodiment
A work system according to a second embodiment of the present invention will be described.
FIG. 5 is a schematic view showing the configuration of a working system 1 according to a second embodiment of the present invention.
Similar to the working system 1 according to the first embodiment of the present invention, the working system 1 includes an input / output glove 10, a head mounted display 20, a master controller 30, a slave controller 40, and a working robot 50.
The work system 1 according to the second embodiment of the present invention is a sorting box Q as a transfer destination of the target article I, and a sheet used when sending the target article I to the purchaser carried by the shipping belt conveyor, The target article I can be packaged directly without using the backing sheet L (hereinafter, simply referred to as the "base sheet L") capable of wrapping and sending the target article I and the sorting box Q, and sends it directly to the purchaser There are a plurality of systems, such as cardboard B (hereinafter simply referred to as "cardboard B") that can be used. In the second embodiment of the present invention, the sorting box Q, the mount L, the cardboard B and the like of the transfer destination to be moved when the target article I stored in the bin M is sent to the purchaser We call it a container.

When the management server receives information including the information specifying the target item I, the number of purchases of the target item I, the destination of the target item I, etc. from the terminal device of the purchaser, the date and time when the information was received, and the received information And remember. The picking system, according to the order information of the target article I stored by the management server, identification information pertaining to each purchaser, and a container of the target article I (ie, sorting box Q, mount L, or cardboard B) Associate and store. Then, in the picking system, the container is moved to the position of the work robot 50 in order to move the target item I ordered by each purchaser to the respective container.
Note that the feature point identifying unit 307 sets a window in the image captured by the camera 52 for each of the mount L and the cardboard B, for example, and the point at which the edge intersects within the range of an angle within the window The feature point P can be identified by identifying. The size of the window applied to each of the sorting box Q, the mount L, and the cardboard B may be different.
The working system 1 according to the second embodiment of the present invention has the same processing as the working system 1 according to the first embodiment of the present invention for each of the containers (ie, the sorting box Q, the mount L and the cardboard B). You can do
When the container of the target article I is the mount L, the mark image projected by the head mount display 20 is, for example, colored with an image K having the same shape as the outline of the target article I as shown in FIG. ). Further, when the destination of the target article I is the cardboard B, the mark image projected by the head mounted display 20 is, for example, colored an image K having the same shape as the outline of the target article I as shown in FIG. Light).
In this manner, the work system 1 can reduce the possibility that the operator O misplaces the target article's destination by presenting the destination to the operator O.

<Other Embodiments>
Although the first and second embodiments of the present invention have been described with reference to the drawings, the specific configuration is not limited to the above, and various design changes and the like may be made. It is possible.
For example, in the first and second embodiments of the present invention described above, the master controller 30 is provided in the vicinity of the operator O, but is not limited thereto. For example, in another embodiment, the above-described control may be performed in a server apparatus on the network N, or the positional relationship between the master controller 30 and the slave controller 40 may be interchanged. That is, in another embodiment, the slave controller 40 provided in the vicinity of the operator O may only transfer the signal, and the master controller 30 provided in the vicinity of the work robot 50 may perform the above processing. .

  Further, according to the first embodiment and the second embodiment of the present invention described above, the master controller 30 combines the mark image of the moving destination of the target article I with the captured image, but the present invention is not limited thereto. For example, in another embodiment of the present invention, character information may be combined and presented in addition to the mark image.

<Computer configuration>
Although the embodiment of the present invention has been described, the above-mentioned master controller 30, slave controller 40 and other control devices may have a computer system inside. The process of the process described above is stored in the form of a program in a computer readable recording medium, and the process is performed by the computer reading and executing the program. An example of a computer is shown below.
FIG. 8 is a schematic block diagram showing the configuration of a computer according to at least one embodiment.
The computer 5 includes a CPU 6, a main memory 7, a storage 8 and an interface 9, as shown in FIG.
For example, each of the above-described master controller 30, slave controller 40, and other control devices is implemented in the computer 5. The operation of each processing unit described above is stored in the storage 8 in the form of a program. The CPU 6 reads a program from the storage 8 and develops it in the main memory 7 and executes the above processing according to the program. Further, the CPU 6 secures a storage area corresponding to each storage unit described above in the main memory 7 in accordance with a program.

  Examples of the storage 8 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magnetooptical disk, a compact disc read only memory (CD-ROM), and a digital versatile disc read only memory (DVD-ROM). , Semiconductor memory and the like. The storage 8 may be internal media directly connected to the bus of the computer 5 or may be external media connected to the computer 5 via the interface 9 or a communication line. When the program is distributed to the computer 5 by a communication line, the computer 5 that has received the distribution may expand the program in the main memory 7 and execute the above processing. In at least one embodiment, storage 8 is a non-transitory tangible storage medium.

  Also, the program may realize part of the functions described above. Furthermore, the program may be a file capable of realizing the above-described functions in combination with a program already recorded in a computer system, a so-called difference file (difference program).

  In each embodiment of the present invention, the image captured by the camera 52 is, for example, displayed on a screen provided in a virtual reality displayed by the head mounted display 20, that is, displayed as part of the virtual reality. It may be one. In addition, the image captured by the camera 52 may be such that the image itself is displayed on, for example, a smart glass, that is, displayed as Augmented Reality. In addition, the image captured by the camera 52 may be such that the image itself is projected on, for example, a holo lens, that is, it is projected as mixed reality.

  In the first embodiment of the present invention, the master controller 30 has been described as generating the control signal of the manipulator 51 of the work robot 50 based on the operation signal input from the input / output glove 10. However, in another embodiment of the present invention, the master controller 30 reads the motion of the operator's hand using a technique such as, for example, motion capture, and controls the manipulator 51 of the work robot 50 based on the read motion. It may generate a signal.

  While several embodiments of the present invention have been described, these embodiments are examples and do not limit the scope of the invention. These embodiments may be subjected to various additions, various omissions, various substitutions, and various modifications without departing from the scope of the invention.

1: Work system 5: Computer 6: CPU
7 Main memory 8 Storage 9 Interface 10 Input / output glove 20 Head mounted display 30 Master controller 40 Slave controller 50 Work robot 51 · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · Input unit 307 ··· Feature point identification unit 308 ··· Combining unit 309 ··· Operation signal input unit 310 ··· Operation signal conversion unit 311 ··· Control signal transmission unit B ··· Cardboard I ··· Target Object L ... Mount M ... Bin N ... Network O ... Operator Q ... Sorting box

Claims (5)

An imaging unit capable of photographing the destination of the target article;
A feature point identification unit that identifies a feature point for identifying the movement destination;
A combining unit configured to combine an image including the moving destination photographed by the imaging unit and an image indicating the position of the moving destination based on the feature point;
Working system equipped with A video signal output unit that outputs an image synthesized by the synthesis unit;
The working system according to claim 1, comprising: A display unit for receiving an image synthesized by the synthesis unit;
The display unit projects the received image.
The work system according to claim 2. A control method of a work system including an imaging unit capable of photographing a movement destination of a target article, the control method comprising:
Specifying a feature point for specifying the movement destination;
Combining the image including the moving destination captured by the imaging unit and the image indicating the position of the moving destination based on the feature point;
How to control the working system including: In a computer of a work system including an imaging unit capable of photographing a destination of a target article,
Specifying a feature point for specifying the movement destination;
Combining the image including the moving destination captured by the imaging unit and the image indicating the position of the moving destination based on the feature point;
A program that runs

Images