JP2021110573A - Sensor device and cargo handling system - Google Patents

Abstract

To improve recognition accuracy of an article being present in a placement area.SOLUTION: A sensor device according to an embodiment includes a first transmitting element and a second receiving element. The first transmitting element is disposed in a first region having a first area corresponding to a placement area of a placement region in which an article can be handled by a cargo handling apparatus, the first region being located above the placement region and facing the placement region, and transmits an ultrasonic wave or a radio wave as a transmission wave toward the placement region. The second receiving element is disposed in the first region, and receives a reflected wave of the transmission wave. At least one of the first transmitting element and a first receiving element is disposed in the vicinity of an outer edge of the first region.SELECTED DRAWING: Figure 4

Description

Embodiments of the present invention relate to sensor devices and cargo handling systems.

Conventionally, a cargo handling system that is installed in a distribution warehouse or the like and grips an article and moves the article based on a sensing result of an ultrasonic sensor device, a camera, or the like is known. The cargo handling system is provided with, for example, a sensor device having a transmitting element that transmits a transmitted wave such as an ultrasonic wave and a receiving element that receives a reflected wave of the transmitted wave.

Japanese Patent No. 3849514 JP-A-2017-75915

Provided are a sensor device and a cargo handling system that improve the recognition accuracy of an article when the article is present in the loading area of the article defined by the cargo handling system.

The sensor device of the embodiment includes a first transmitting element and a second receiving element. The first transmitting element is a region having a first area corresponding to the loading area of the loading region on which the article can be handled by the cargo handling device, and is above the loading region and facing the loading region. It is arranged in a first area existing at a position, and transmits ultrasonic waves or radio waves as a transmission wave toward the mounting area. The second receiving element is arranged in the first region and receives the reflected wave of the transmitted wave. At least one or more of the first transmitting element and the first receiving element are arranged near the outer edge of the first region.

FIG. 1 is a diagram showing an example of the configuration of the cargo handling system of the embodiment. FIG. 2 is a diagram showing an example of arrangement of a sensor head and a camera in a region on the upper surface of the frame of the cargo handling system of the embodiment. FIG. 3 is a diagram showing an arrangement example of an ultrasonic transmitting element and an ultrasonic receiving element of the sensor head of the sensor device of the embodiment. FIG. 4 is a side view of a part of the cargo handling system of the embodiment. FIG. 5 is a diagram showing an example of a propagation path of ultrasonic waves transmitted from the sensor device of the embodiment. FIG. 6 is a diagram showing an example of a propagation path of ultrasonic waves transmitted from the sensor device of the embodiment. FIG. 7 is a diagram showing an example of a specular reflection region of an article of ultrasonic waves transmitted from a sensor head according to an embodiment, in which the reflected wave reaches each sensor head. FIG. 8 is a diagram showing an example of a specular reflection region of an article of ultrasonic waves transmitted from a sensor head according to an embodiment, in which the reflected wave reaches each sensor head. FIG. 9 is a diagram showing an example of a specular reflection region in an article of ultrasonic waves transmitted from all the sensor heads of the embodiment, and the reflected wave reaches each sensor head. FIG. 10 is a diagram showing an example of the hardware configuration of the control device of the embodiment. FIG. 11 is a diagram showing an example of the functional configuration of the control device of the embodiment. FIG. 12 is a diagram for explaining a method of detecting an article by sensing the sensor device of the embodiment. FIG. 13 is a diagram for explaining a method of detecting an article by sensing the sensor device of the embodiment. FIG. 14 is a diagram for explaining an article group of an embodiment, a sensing result, and an example of a captured image. FIG. 15 is a diagram for explaining an article group, a sensing result, and an example of a captured image of the embodiment. FIG. 16 is a diagram for explaining an article group of an embodiment, a sensing result, and an example of a captured image. FIG. 17 is a diagram for explaining an article group of an embodiment, a sensing result, and an example of a captured image. FIG. 18 is a flowchart of an article detection process executed by the sensor device of the embodiment. FIG. 19 is a flowchart of an article recognition process executed by the control device of the embodiment.

Hereinafter, exemplary embodiments of the present invention will be disclosed. The configurations of the embodiments shown below, as well as the actions, results, and effects produced by such configurations, are examples. The present invention can be realized by a configuration other than the configurations disclosed in the following embodiments, and at least one of various effects based on the basic configuration and derivative effects can be obtained. ..

FIG. 1 is a diagram showing an example of the configuration of the cargo handling system of the embodiment. As shown in FIG. 1, the cargo handling system 1 includes a cargo handling device 10, a sensor device 20, a camera 30, and a frame 40. Further, the cargo handling device 10 includes a moving device 50 and a control device 60. Here, the control device 60, the camera 30, the sensor device 20, and the mobile device 50 are connected via, for example, a serial cable or a LAN (Local Area Network).

In the following description, the vertical direction in the cargo handling system 1 of the present embodiment includes, for example, a downward direction including a direction vector toward which gravity is directed, and an upward direction is a direction opposite to the downward direction, for example. The cargo handling device 10 is in the direction of lifting the article W1.

The frame 40 has a square frame-shaped lower frame member 41, a square frame-shaped upper frame member 42, and four (plural) pillar members 43. The lower frame member 41 is fixed to the floor surface. The upper frame member 42 is provided above the lower frame member. The four pillar members 43 connect the four corners of the lower frame member 41 and the four corners of the upper frame member 42.

In the present embodiment, the pallet 2 is arranged on the floor surface of the lower frame member 41. The frame 40 of the present embodiment describes a case where the upper frame member 42 and the lower frame member 41 form a square frame shape. The present embodiment is not limited to the case where the upper frame member 42 and the lower frame member 41 have a square frame shape, and may have a polygonal frame shape other than the square shape or a circular frame shape. ..

The cargo handling device 10 is a device for cargo handling (moving) the article W1 placed on the upper surface 2a of the pallet 2 to a destination (not shown). The destination of the present embodiment may be a destination of the article W1, for example, a conveyor. The pallet 2 is arranged on the floor surface on which the frame 40 is installed. That is, the pallet 2 is positioned at a specified position with respect to the frame 40, and the entire area of the upper surface 2a of the pallet 2 located at the specified position is the placement area R1 of the article W1. The mounting area R1 is not limited to this. For example, the mounting area R1 may be provided on the floor surface or the like. A plurality of articles W1 may be placed in the mounting area R1. The article W1 may be any article, such as a PET bottle or a box. In the present embodiment, when referring to a plurality of articles including one or more articles W1, it is referred to as an article group W. The pallet 2 is moved by a forklift or the like. The article W1 is also referred to as an object, a moving object, a gripping object, a luggage, or the like.

The area R2 in the upper frame member 42 of the present embodiment is an area having substantially the same area as the loading area R1 on which the article W1 can be handled by the cargo handling device 10, and is above the loading area R1 and the mounting. It exists at a position facing the area R1.

In the present embodiment, by providing the sensor head 21 at each corner of the region R2, all the articles W1 existing in the mounting region R1 can be detected. The arrangement of the ultrasonic wave receiving element and the ultrasonic wave transmitting element in the sensor head 21 for detecting all the articles W1 existing in the mounting area R1 will be described later.

The shape of the region R2 is not limited to a quadrangle. The region R2 may have a shape corresponding to the mounting region R1. Even if the region R2 is a polygon other than a quadrangle, it is common in that a sensor head 21 is provided at each corner. Further, the region R2 may have a circular shape. When the region R2 has a circular shape, for example, the sensor head 21 may also have a circular shape and may include three sensor heads. In such a case, it is conceivable that three sensor heads 21 are provided at equal intervals so as to be in contact with the circumference of the region R2.

Further, a camera 30 is provided between the four sensor heads 21 in the area R2. The sensor head 21 and the camera 30 are fixed to the region R2 in the upper frame member 42 of the frame 40 via a mounting member (not shown).

As shown in FIGS. 1 and 2, the camera 30 is arranged at the center of the frame of the upper frame member 42. The camera 30 is, for example, a 3D camera. The camera 30 is an example of an imaging device.

The camera 30 images the article group W placed on the mounting area R1 of the pallet 2 from above, and outputs the captured image obtained by the imaging to the control device 60. The imaging range of the camera 30 is set to include at least the article group W.

The sensor device 20 shown in FIG. 1 detects an article by using ultrasonic waves transmitted from the sensor head 21, for example, by echolocation technology or the like. The sensor device 20 senses the mounting area R1 in which the article group W may exist from above the article group W mounted on the pallet 2 by using the sensor head 21, and controls the sensing result. Output to 60. For example, the sensor device 20 uses the sensor head 21 to transmit ultrasonic waves to the mounting region R1 of the pallet 2 as transmitted waves, receive the reflected waves of the transmitted waves, and based on the reflected waves, the sensor device 20 transmits the ultrasonic waves. Each of the articles W1 existing in the mounting area R1 is detected. The imaging range of the camera 30 and the detection range of the sensor device 20 may be the same range, or one of them may be wider than the other.

FIG. 2 is a diagram showing an arrangement example of the sensor head 21 and the camera 30 of the cargo handling system 1 of the embodiment. As shown in FIG. 2, the region R2 is provided with four (plurality) sensor heads 21 and a camera 30. The region R2 of the present embodiment is a quadrangle composed of two sides L1 and two sides L2. In the direction of the side L1, an interval of L1 / 5 is provided between the sensor heads 21, and in the direction of the side L2, an interval of L2 / 5 is provided between the sensor heads 21.

As shown in FIG. 2, by providing the ultrasonic wave transmitting elements 23A at the four corners of the sensor head 21, four ultrasonic wave transmitting elements 23A are provided on the side L1 and 4 on the side L2 in the region R2. The ultrasonic wave transmitting element 23A will be provided. By arranging a total of 16 ultrasonic transmission elements 23A at intervals, it is possible to apply ultrasonic waves to the entire mounting region R1. That is, even if an interval of L1 / 5 is provided between the sensor heads 21 in the direction of the side L1 and an interval of L2 / 5 is provided between the sensor heads 21 in the direction of the side L2, the mounting area R1 It is possible to apply ultrasonic waves to the entire surface.

Further, a plurality of ultrasonic receiving elements 24A are arranged uniformly and evenly in each of the four sensor heads 21 in the region R2. That is, the plurality of ultrasonic wave receiving elements 24A are provided so as to be able to receive the reflected wave mirror-reflected in the mounting region R1. As a result, the present embodiment can detect the article W1 existing in any region of the mounting region R1.

FIG. 3 is a diagram showing an arrangement example of an ultrasonic transmitting element and an ultrasonic receiving element of the sensor head of the sensor device of the embodiment. As shown in FIG. 3, the sensor head 21 includes a housing 22, a plurality of ultrasonic wave transmitting elements 23A, and a plurality of ultrasonic wave receiving elements 24A. The housing 22 is flat in the vertical direction and formed in a quadrangular shape in a plan view (bottom view). In this embodiment, the housing 22 is not limited to a quadrangular shape, and may be a polygonal shape or a circular shape.

As an example, the ultrasonic wave transmitting elements 23A are arranged one by one at the four corners of the housing 22. The ultrasonic transmission element 23A transmits ultrasonic waves as a transmission wave from above the article W1 mounted on the mounting region R1 toward the mounting region R1, that is, downward. That is, the ultrasonic wave transmitting element 23A transmits ultrasonic waves toward the article W1 placed on the mounting area R1. The ultrasonic wave transmitting element 23A is composed of an antenna element. The ultrasonic wave transmitting element 23A is an example of the first transmitting element.

The number of ultrasonic receiving elements 24A is larger than the number of ultrasonic transmitting elements 23A. The ultrasonic wave receiving element 24A receives the reflected wave of the ultrasonic wave transmitted from the ultrasonic wave transmitting element 23A. Specifically, the ultrasonic wave receiving element 24A receives the ultrasonic waves (reflected waves) transmitted from the ultrasonic wave transmitting element 23A and reflected by the article W1 above the article W1 placed on the mounting region R1. That is, the ultrasonic wave receiving element 24A can receive ultrasonic waves (reflected waves) from below. The ultrasonic wave receiving element 24A is composed of an antenna element. The ultrasonic receiving element 24A is an example of the first receiving element.

The plurality of ultrasonic receiving elements 24A are irregularly distributed in a plane region having substantially the same size as the mounting area R1. As a result, the virtual image obtained by the sensor device 20 at the distance of the article W1 with respect to the reference position is reduced.

As shown in FIG. 2, the sensor head 21 in which the ultrasonic transmission element 23A is arranged at the four corners (in other words, the corners of each corner of the quadrangle) is attached to the four corners of the region R2 (in other words, the corners of each corner of the quadrangle). ), The ultrasonic transmission element 23A is arranged at a predetermined interval in the vicinity of the edge portion R2a (in other words, the edge portion R2a near the outer edge) inside a predetermined distance from the outer edge of the region R2. In the present embodiment, since the ultrasonic wave transmitting element 23A is provided along the edge portion R2a near the outer edge of the region R2, the article W1 such as a box is horizontally arranged near the outer edge of the mounting region R1. If so, the received wave mirror-reflected by the article W1 can be received by the ultrasonic receiving element 24A of the sensor head 21. As a result, it becomes possible to detect the article W1 existing in the vicinity of the outer edge in the mounting region R1 without omission.

In this embodiment, an example in which the ultrasonic wave transmitting element 23A is provided along the edge portion R2a near the outer edge of the region R2 will be described, but the present embodiment is not limited to such an embodiment, and for example, the region R2 The ultrasonic wave receiving element 24A may be provided along the edge portion R2a in the vicinity of the outer edge of the above.

In the example shown in FIG. 2, not only the ultrasonic wave transmitting element 23 but also the ultrasonic wave receiving element 24 is arranged near the edge portion R2a, in other words, near the outer edge. As described above, at least one or more of the ultrasonic transmitting element 23 and the ultrasonic receiving element 24 are arranged near the outer edge of the region R2. With this arrangement, the ultrasonic receiving element 24 can receive the reflected wave mirror-reflected by the article W1 in which the transmitted wave of the ultrasonic transmitting element 23 exists in the vicinity of the outer edge of the mounting region R1. Thereby, the recognition accuracy of the article can be improved.

Since the article W1 facing the mounting area R1 is gathered by a transparent sheet or the like, it may have an inclined surface due to the transparent sheet. In such a case, the sensor head 21 and the camera 30 provided in the region R2 may not be able to accurately detect the article W1. Therefore, in the present embodiment, the ultrasonic wave transmitting element 23 is also provided on the side of the frame 40.

FIG. 4 is a side view of a part of the cargo handling system of the embodiment. As shown in FIG. 4, in addition to the sensor head 21, it has an ultrasonic transmission element 23B and an ultrasonic transmission element 23C provided on the side of the frame 40. The ultrasonic transmission element 23B and the ultrasonic transmission element 23C are connected to the sensor device 20. Then, the ultrasonic wave transmitting element 23B and the ultrasonic wave transmitting element 23C transmit the transmitted wave according to the control from the sensor device 20. In addition, in FIG. 1 and the like, the illustration of the ultrasonic wave transmitting element 23B and the ultrasonic wave transmitting element 23C is omitted.

As shown in FIG. 4, for example, when the article W1 is covered with a transparent sheet E such as a vinyl sheet and an inclined surface Ea is formed on the sheet E, the unevenness of the surface of the sheet E is small, so that ultrasonic waves are generated. Reflects substantially specularly on the surface of the sheet E. Therefore, the ultrasonic waves transmitted from the ultrasonic wave transmitting element 23B toward the article W1 (sheet E) are specularly reflected by, for example, the inclined surface Ea of the sheet E, and reach the ultrasonic wave receiving element 24A of the sensor head 21. .. Sheet E is also referred to as a wrapping sheet or film.

The ultrasonic wave transmitting element 23B and the ultrasonic wave transmitting element 23C can transmit a transmitted wave from the side direction 401 to a plurality of articles W1 grouped by the sheet E existing between the mounting region R1 and the region R2. It is placed in a suitable position. Therefore, the ultrasonic transmission element 23B and the ultrasonic transmission element 23C are supported by the pillar member 43 of the frame 40. The lateral direction 401 is a direction substantially perpendicular to the vertical direction connecting the mounting area R1 and the area R2.

Then, the transmitted wave transmitted from the ultrasonic transmitting element 23B in the lateral direction 401 is reflected by the inclined surface Ea of the sheet E, and the reflected wave is output in the upward direction 402. The ultrasonic wave receiving element 24A of the sensor head 21 can receive the reflected wave. As a result, the sensor device 20 can recognize the plurality of articles W1 grouped on the sheet E.

The ultrasonic wave transmitting element 23B and the ultrasonic wave transmitting element 23C may be supported by the pillar member 43 so that the vertical position can be adjusted, or the vertical position may not be adjustable. The ultrasonic transmission elements 23B and 23C are examples of the second transmission element.

The ultrasonic transmitting elements 23B and 23C are located above the mounting region R1 and below the element group 25, and emit ultrasonic waves as transmitted waves from the side of the article W1 mounted on the mounting region R1. Send to W1. Hereinafter, the ultrasonic transmission element 23 may be used as a general term for the ultrasonic transmission elements 23A, 23B, and 23C.

The ultrasonic wave transmitting element 23B and the ultrasonic wave transmitting element 23C of the present embodiment are provided at different heights of the pillar member 43. As a result, the sensor device 20 can recognize the articles W1 having different heights by the ultrasonic wave transmitting element 23B and the ultrasonic wave transmitting element 23C. In this embodiment, the number of ultrasonic wave transmitting elements 23 provided on the pillar member 43 is not limited to two, and may be one or three or more. That is, the ultrasonic wave transmitting element 23 may be provided at a height corresponding to the type of the article W1 that can be arranged on the pallet 2.

FIG. 5 shows an example in which the ultrasonic wave transmitting element 23B is arranged above the ultrasonic wave transmitting element 23B, but the present invention is not limited to this. The ultrasonic transmission element 23B may be arranged below the ultrasonic transmission element 23B, or the positions of the ultrasonic transmission element 23B and the ultrasonic transmission element 23B in the vertical direction may be the same.

Next, the propagation path of ultrasonic waves in the sensor device 20 will be described. When distinguishing a plurality of sensor heads 21, they may be referred to as sensor heads 21a to 21d (FIG. 2). Further, when distinguishing a plurality of ultrasonic wave transmitting elements 23A in one sensor head 21, they may be referred to as ultrasonic wave transmitting elements 23Aa to 23Ad (FIG. 3).

5 and 6 are diagrams showing an example of a propagation path of ultrasonic waves transmitted from the sensor device of the embodiment. As shown in FIGS. 5 and 6, for example, for ultrasonic waves having a wavelength of several tens of kHz, the surface (upper surface) of the upper surface (upper surface) of the article W1 such as corrugated cardboard has relatively small irregularities, so that the ultrasonic waves are the article W1. Specular reflection, that is, normal reflection is performed on the surface (upper surface) of (article group W).

Specifically, FIG. 5 shows an example of the propagation path of ultrasonic waves transmitted from the ultrasonic wave transmitting element 23Ab of the sensor head 21a. As shown in FIG. 5, when the transmission wave (for example, transmission wave 501, 502) transmitted from the ultrasonic transmission element 23Ab is reflected in the region A1 on the upper surface of the article W1, the ultrasonic wave in the sensor head 21a The sound wave receiving element 24 can receive reflected waves (for example, reflected waves 511 and 512). On the other hand, when the transmitted wave (for example, transmitted waves 503 and 504) transmitted from the ultrasonic transmitting element 23Ab is reflected in the region A2 on the upper surface of the article W1, the ultrasonic receiving element 24 in the sensor head 21c receives the ultrasonic wave. The reflected wave (for example, the reflected wave 513, 514) can be received. In other words, the sensor device 20 can set the inside of the regions A1 and A2 as detection targets by transmitting the transmitted wave by the ultrasonic transmitting element 23Ab.

FIG. 7 is a diagram illustrating regions A1 and A2 that can be detected by the transmitted ultrasonic waves by the ultrasonic wave transmitting element 23Ab of the sensor head 21a shown in FIG. As shown in FIG. 7, the regions A1 and A2 include a region near the outer edge of the mounting region R1. In other words, the ultrasonic wave transmitting element 23Ab provided near the outer edge of the region R2 transmits the transmitted wave, so that the regions A1 and A2 including the region near the outer edge of the mounting region R1 can be detected.

FIG. 6 shows an example of the propagation path of the ultrasonic wave transmitted from the ultrasonic wave transmitting element 23Ac of the sensor head 21a. As shown in FIG. 6, the ultrasonic transmission element 23Ac is provided near the center of the region R2 as compared with the ultrasonic transmission element 23Ab shown in FIG. Then, when the transmitted wave (for example, transmitted waves 601 and 602) transmitted from the ultrasonic transmitting element 23Ac is reflected in the region B1 on the upper surface of the article W1, the ultrasonic receiving element 24 in the sensor head 21a receives the ultrasonic wave. Reflected waves (eg, reflected waves 611, 612) can be received. On the other hand, when the transmitted wave (for example, transmitted waves 603 and 604) transmitted from the ultrasonic transmitting element 23Ac is reflected in the region B2 on the upper surface of the article W1, the ultrasonic receiving element 24 in the sensor head 21c receives the ultrasonic wave. The reflected wave (for example, the reflected wave 613, 614) can be received. In other words, the sensor device 20 can set the inside of the regions B1 and B2 as detection targets by transmitting the transmitted wave by the ultrasonic transmitting element 23Ac.

FIG. 8 is a diagram illustrating regions B1 and B2 that can be detected by the transmitted ultrasonic waves by the ultrasonic wave transmitting element 23Ac of the sensor head 21a shown in FIG. As shown in FIG. 8, the regions B1 and B2 include a region near the center of the mounting region R1. In other words, the ultrasonic wave transmitting element 23Ac provided near the center of the region R2 transmits the transmitted wave, so that the regions B1 and B2 including the region near the center of the mounting region R1 can be detected.

FIG. 9 is a diagram illustrating a detectable region of the mounting region R1 when each ultrasonic wave transmitting element 23A of the four (plural) sensor heads 21 transmits ultrasonic waves. In the example shown in FIG. 9, the specular reflection region of the ultrasonic wave transmitted from one ultrasonic wave transmitting element 23A is shown by a white square. The area where the specular reflection areas of the ultrasonic waves transmitted from the plurality of ultrasonic wave transmitting elements 23A overlap is shown by hatching or blackening. The higher the density, the more specular reflection regions overlap. That is, in the example shown in FIG. 9, by transmitting ultrasonic waves from each ultrasonic wave transmitting element 23A of the four (plural) sensor heads 21, the reflected wave mirror-reflected on the horizontal plane in the mounting region R1 is generated. All can be received by the sensor head 21.

Returning to FIG. 1, the cargo handling device 10 includes a moving device 50 and a control device 60. The moving device 50 is arranged around the pallet 2 and the conveyor.

The moving device 50 includes a base 111, an arm portion 112 supported by the base 111, and a grip portion 113 provided at the tip of the arm portion 112.

The base 111 is a housing fixed to the floor surface G.

The arm portion 112 is, for example, an articulated robot arm. The arm portion 112 is configured to be able to perform an operation of moving the article W1 held by the grip portion 113 in the vertical direction and an operation of moving the article W1 in the horizontal direction.

For example, the arm portion 112 is composed of three movable portions 112a to 112c connected by a plurality of joints that can rotate around one or two axes.

A grip portion 113 is attached to the tip of the arm portion 112 (movable portion 112c). The grip portion 113 sandwiches and grips one article W1. The grip portion 113 may include a pair of claw portions. The gripping of the article W1 is not limited to the gripping method by pinching. For example, the grip portion 113 may suck and grip the article W1 by a negative pressure. Further, the grip portion 113 may grip the two articles W1 at the same time. The grip portion 113 is also referred to as a holding portion.

The control device 60 is a device that controls the mobile device 50, the sensor device 20, and the camera 30. That is, the control device 60 is a device that controls the entire cargo handling system 1. The control device 60 controls the operation of the mobile device 50 based on the image pickup result of the camera 30 and the sensing result of the sensor device 20. The control device 60 may be provided integrally with the moving device 50, or may be provided separately from the moving device 50.

In the present embodiment, the moving device 50 and the control device 60 are used as cargo handling devices, but the present invention is not limited to this, and the control device 60 may be used as a cargo handling device.

Next, the hardware configuration of the control device 60 described above will be described. FIG. 10 is a diagram showing an example of the hardware configuration of the control device 60. Note that FIG. 10 also shows an example of the configuration of the sensor device 20 among the various devices connected to the control device 60.

As shown in FIG. 10, the control device 60 includes a processor 121, a display unit 122, an operation unit 123, a camera interface 124, a mobile device interface 125, a sensor interface 126, a communication unit 127, and a storage unit 128.

The processor 121 is a control device such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an ASIC (Application Specific Integrated Circuit). The display unit 122 is realized by, for example, a display device (output device) such as a liquid crystal display or a touch panel display. The operation unit 123 is realized by an input device (input device) such as a keyboard or a pointing device. The operation unit 123 may be a touch panel provided on the screen of the display unit 122.

The camera interface 124 is an interface that is communicably connected to the camera 30. The processor 121 controls the camera 30 connected to the camera interface 124. The mobile device interface 125 is an interface that is communicably connected to the mobile device 50. The processor 121 controls the mobile device 50 connected to the mobile device interface 125. The sensor interface 126 is an interface that is communicably connected to the sensor device 20. The processor 121 controls the sensor device 20 connected to the sensor interface 126.

Here, as shown in FIG. 10, the sensor device 20 includes, for example, a transmission control unit 301, an operational amplifier 302, an ultrasonic transmission element 23, an ultrasonic reception element 24, an operational amplifier 305, a mixer 306, a processing unit 307, and the like. ..

The transmission control unit 301 generates and transmits a frequency signal corresponding to the frequency (ultrasonic waveform) of the ultrasonic wave to be used. The frequency signal generated by the transmission control unit 301 is amplified via the operational amplifier 302 and then input to the ultrasonic transmission element 23. Further, the frequency signal generated by the transmission control unit 301 is also input to the mixer 306. More specifically, the transmission control unit 301 sequentially transmits a burst signal or a chirp signal to the plurality of ultrasonic transmission elements 23. As a result, the ultrasonic waves transmitted from the ultrasonic wave transmitting element 23 become burst waves or chirp waves. More specifically, the transmission control unit 301 drives one or a plurality of ultrasonic wave transmission elements 23 in order at regular intervals. The transmission control unit 301 is also referred to as a signal generation unit.

The ultrasonic wave transmitting element 23 and the ultrasonic wave receiving element 24 are each composed of an antenna element as described above. The ultrasonic transmitting element 23 transmits as ultrasonic waves (ultrasonic waves) corresponding to a frequency signal, and the reflected wave is received by the ultrasonic receiving element 24. The reflected wave received by the ultrasonic wave receiving element 24 is amplified via the operational amplifier 302 and then input to the mixer 306 as a reflected wave signal. In FIG. 10, one ultrasonic wave transmitting element 23 and one ultrasonic wave receiving element 24 are shown, but in reality, a plurality of ultrasonic wave transmitting elements 23 and one ultrasonic wave receiving element 24 are provided. There is. Further, a plurality of operational amplifiers 302 and 305 are also provided corresponding to the ultrasonic wave transmitting element 23 and the ultrasonic wave receiving element 24.

The mixer 306 performs multiplication operations on both the input frequency signal and the reflected wave signal, and outputs a signal indicating the difference (or sum) of the signals (frequency) to the processing unit 307.

The processing unit 307 can detect the signal input from the mixer 306 for each ultrasonic transmitting element 23 (ultrasonic receiving element 24) and then output the signal to the sensor interface 126 as a detection signal. Further, the processing unit 307 may be realized by the CPU executing a program, or may be configured by dedicated hardware.

With the above configuration, the sensor device 20 senses the article W1 existing in the sensing range, and outputs the sensing result as a detection signal to the control device 60 (sensor interface 126). Such a detection signal represents the shapes of the upper surface and the side surface of the article W1 existing in the sensing range by measured values such as ultrasonic intensity (radio wave intensity).

The communication unit 127 of the control device 60 is a communication interface for connecting to an external device. The communication unit 127 exchanges various data with and from the external device under the control of the processor 121. As the external device, for example, a monitoring device for monitoring the cargo handling system 1 can be considered. Then, when an abnormality occurs, the communication unit 127 notifies the monitoring device that the abnormality has occurred.

The storage unit 128 is realized by, for example, a main storage device such as a RAM (Random Access Memory), a semiconductor memory element such as a flash memory, or an auxiliary storage device such as a hard disk. The storage unit 128 stores programs, setting information, and the like related to the operation of the cargo handling device 10. Further, the storage unit 128 shows limit information indicating the vertical and horizontal movement limit positions of the moving device 50, the movable range of the arm portion 112, the upper limit and the lower limit of the dimensions of the grippable article W1 of the grip portion 113, and the like. Remember. In addition, the storage unit 128 stores baggage information indicating the size (dimensions), shape, and the like of the article W1.

Next, with reference to FIG. 11, the configuration of the functional configuration of the control device 60 and the sensor device 20 will be described. Here, FIG. 11 is a diagram showing an example of the functional configuration of the control device 60 and the sensor device 20.

As shown in FIG. 11, in the control device 60, when the processor 121 executes the program stored in the storage unit 128, the recognition unit 1211, the detection unit 1212, the planning unit 1213, the planning execution unit 1214, and the output unit 1215 is realized as a functional unit.

The recognition unit 1211 acquires the captured image captured by the camera 30. The recognition unit 1211 may directly acquire the captured image via the camera interface 124, or may acquire the captured image temporarily stored in the storage unit 128 via the camera interface 124.

The recognition unit 1211 recognizes the article W1 placed on the pallet 2 based on the acquired captured image. For example, the recognition unit 1211 of the present embodiment extracts the contour of the imaged article by performing image processing on the captured image, and based on the contour, the article W1 placed on the pallet 2 is used. recognize. The recognition unit 1211 of the present embodiment further compares the size (dimensions) and shape of the article W1 indicated by the baggage information stored in the storage unit 128 with the extracted contour. The article W1 placed on the pallet 2 may be recognized. Further, the recognition unit 1211 is the article W1 placed on the pallet 2 in the actual three-dimensional space based on the position where the camera 30 is provided and the position of the article W1 in the captured image. Identify the location information of.

The processing unit 307 of the sensor device 20 may be realized by, for example, a combination of a CPU and a memory such as a ROM or RAM. The processing unit 307 not only detects the signal input from the mixer 306 for each ultrasonic transmitting element 23 (ultrasonic receiving element 24), but also the CPU reads and executes a program stored in the ROM. As a result, the distance measuring unit 307a and the position recognition unit 307b are realized.

The distance measuring unit 307a acquires a signal input from the mixer 306. The distance measuring unit 307a calculates the distance from the sensor head 21 to the article W1 based on the acquired signal.

The position recognition unit 307b recognizes the shape and position of the article W1 using the calculated distance. The position recognition unit 307b transmits the recognized shape and position of the article W1 to the control device 60. The position recognition unit 307b of the present embodiment outputs the recognized shape and position of the article W1 as a distance image.

The recognition unit 1211 sends the recognition result of the article W1 to the planning unit 1213. As an example, the recognition result of the article W1 is the number of articles W1 included in the article group W, and the dimensions, shape, design, packing features, loading features, and positions of each of the articles W1. Further, each position of the article W1 is a position in the horizontal direction and a position in the height direction on the pallet 2. Further, each position of the article W1 may be represented by a relative distance from the grip portion 113.

In addition, the recognition unit 1211 sends the captured image used for recognizing the article W1 to the planning unit 1213.

The detection unit 1212 detects the article W1 based on the position and shape of the article W1 recognized by the position recognition unit 307b of the sensor device 20. The detection result by the detection unit 1212 shall include, for example, the number, dimensions, or shapes of the articles W1 included in the article group W.

The planning unit 1213 derives (calculates) an action plan for operating the drive mechanism of the arm unit 112 and the grip unit 113 of the moving device 50 based on the image recognition result of the recognition unit 1211 and the detection result of the detection unit 1212. .. The planning unit 1213 of the present embodiment derives an action plan based on at least one of the number, dimensions, or shapes of the articles W1 detected by the detection unit 1212.

More specifically, the planning unit 1213 compares the image recognition result acquired from the recognition unit 1211 with the number, dimensions, and shapes of the articles W1 included in the detection result acquired from the detection unit 1212. As described above, since the number, dimensions, and shape of the articles W1 included in the detection result are more accurate than the number, dimensions, and shape of the articles W1 included in the image recognition result, the planning unit 1213 If the image recognition result and the detection result do not match, an action plan is derived based on the detection result.

The planning unit 1213 sets one of the recognized or detected articles W1 as a movement target candidate. Further, the planning unit 1213 determines whether or not the gripping unit 113 can grip the article W1 which is a candidate to be moved based on the image recognition result or the detection result. For example, in the planning unit 1213, the dimension of the article W1 is included in the range from the upper limit to the lower limit of the dimension of the grippable article W1 of the grip portion 113 defined in the limit information of the moving device 50 stored in the storage unit 128. If not, it is determined that the gripping of the article W1 is avoided. In this case, the planning unit 1213 sends a message to the output unit 1215 to avoid gripping without deriving the action plan. Further, the planning unit 1213 sends an captured image in which the article W1 to be avoided to be gripped is reflected to the output unit 1215. Further, the planning unit 1213 may send the detection result of detecting the article W1 to be avoided by the detection unit 1212 to the output unit 1215.

When the planning unit 1213 determines that the moving target candidate article W1 is to be gripped by the gripping unit 113, the planning unit 1213 derives an action plan and sends the derived action plan to the planning execution unit 1214. For example, the planning unit 1213 derives an action plan showing a route (shortest route) for moving the article W1 set as the movement target to the destination. The planning unit 1213 may derive an action plan for simultaneously grasping and moving two or more articles W1. A known technique can be adopted as the method for deriving the action plan.

Further, the criteria for avoiding gripping is not limited to the above-mentioned example, and the planning unit 1213 may determine whether or not to avoid gripping based on other conditions.

The plan execution unit 1214 controls the operation (drive mechanism) of the mobile device 50 based on the action plan derived by the planning unit 1213. For example, the plan execution unit 1214 controls the operation of the moving device 50 based on the action plan to move the object W1 to be moved to the destination by the route shown in the action plan. Further, since the action plan is derived by the planning unit 1213 based on the detection result by the detection unit 1212 and the recognition result by the recognition unit 1211, in other words, the plan execution unit 1214 has the detection result and the recognition result. Controls to grip the article W1 based on. The control target of the plan execution unit 1214 is, for example, the drive mechanism of the grip unit 113.

The output unit 1215 outputs the notification information of the grip avoidance to the external device when the planning unit 1213 notifies that the gripping of the article W1 is to be avoided. For example, the output unit 1215 outputs the reason for avoiding gripping and the captured image in which the article W1 to be avoided to grip is reflected to an external device. Further, the planning unit 1213 may output the detection result of the article W1 to be avoided to be gripped by the output unit 1215 to an external device. Further, the planning unit 1213 may display the reason for avoiding gripping and the captured image in which the article W1 to be avoided from gripping is reflected on the display unit 122 of the control device 60.

With the above-described functional configuration, the cargo handling device 10 can handle (move) the article W1 placed on the pallet 2 to the destination.

Next, a method of detecting the article W1 by sensing the sensor device 20 will be described in detail. 12 and 13 are diagrams illustrating a coordinate system for explaining the detection of an object by the sensor device 20 of the embodiment.

As shown in FIGS. 12 and 13, the sensor device 20 uses the XYZ coordinate system (3-axis Cartesian coordinate system) to determine the position of a certain point (detection point P) of the article W1. Here, the coordinates of the i (i is a positive number of 1 to n) th ultrasonic wave transmitting element 23 are (Xi, Yi, Zi), and the j (j is a positive number of 1 to m) th ultrasonic wave receiving element. Let the coordinates of 24 be (Xj, Yj, Zj), and the coordinates of the detection point P that reflects the ultrasonic waves of the article W1 be (Zptanθ, Zptanφ, Zp). Here, θ is the angle between the straight line obtained by projecting the straight line from the origin to the detection point P onto the XZ plane and the Z axis. φ is the angle between the straight line obtained by projecting the straight line from the origin to the detection point P onto the YZ plane and the Z axis. The coordinates of the ultrasonic wave transmitting element 23 for calculating the position of a certain point (detection point P) of the article W1 are not limited to the above-mentioned coordinates. For example, the coordinates of the i'(i'is a positive number of 1 to n) th ultrasonic transmission element 23 provided on the side surface are (Xi', Yi', Zi'), and j (j is 1 to m). The coordinates of the detection point P that reflects the ultrasonic waves of the article W1 may be obtained as (Zptanθ, Zptanφ, Zp), where the coordinates of the (positive number) th ultrasonic wave receiving element 24 are (Xj, Yj, Zj).

The position recognition unit 307b of the processing unit 307 generates a distance image. At that time, the position recognition unit 307b calculates the position of the ultrasonic wave reflection point based on the received signal (reflected wave) received by each of the ultrasonic wave receiving elements 24. At that time, the delay time for each ultrasonic wave receiving element 24 is calculated.

If the distance Lop from the origin to the detection point P is considered to be sufficiently larger than the coordinates (Xj, Yj) of the ultrasonic receiving element 24 from the origin (far field), the position recognition unit 307b is at the detection point P. The difference Δj between the time it takes for the reflected ultrasonic waves to reach the origin and the time it takes for the reflected ultrasonic waves to reach the j-th ultrasonic wave receiving element 24 is calculated from the angles θ and φ. The difference Δj is also referred to as a delay time. The difference Δj is expressed by the following equation (1). The speed of sound C of ultrasonic waves is used.

The difference Δj (delay time) of each ultrasonic receiving element 24 is calculated, and all the signals delayed by the obtained difference Δj (delay time) for each received signal are added.

Assuming that the signal (ultrasonic wave) transmitted by the i-th ultrasonic transmitting element 23 and received by the j-th ultrasonic receiving element 24 is Rij (t), the signals of all the ultrasonic receiving elements 24 are totaled, θ, The signal composite value Sfar (θ, φ, t) in the φ direction is expressed by the following equation (2).

The position recognition unit 307b includes a plurality of adjustment parameters (combination of θ, φ, and t). Then, the position recognition unit 307b reflects the signal composite value Sfar (θ, φ, t) of the reflected wave received by the plurality of ultrasonic receiving elements 24 by adjusting the signal composite value Sfar (θ, φ, t) by using a plurality of adjustment parameters. A signal showing a signal composite value Sfar (θ, φ, t) that strongly reacts to a point can be obtained. Then, the position recognition unit 307b can recognize the direction in which the reflection point exists from the angles (θ, φ) applied when it reacts strongly to the reflection point.

At this time, the position recognition unit 307b obtains the timing t = Tmax (θ, φ) at which the signal synthesis value Sfar (θ, φ, t) is maximized. Therefore, the position recognition unit 307b can calculate the distance Dfar (θ, φ) to the article W1 in the θ, φ direction. The formula (3) for calculating the distance Dfar (θ, φ) is shown below.
Dfar (θ, φ) = Tmax (θ, φ) × C ・ ・ ・ (3)

In this way, the direction of the detection point P from the origin can be calculated based on the two-dimensional distance Dfar (θ, φ) from the origin to the detection point P and the angles θ, φ.

In this way, the processing unit 307 can calculate the distance and direction from the origin to the detection point P, in other words, the position of the detection point P, based on the reception signal of each ultrasonic wave receiving element 24.

Further, the processing unit 307 obtains the distance as follows using the same coordinate system as described above. If it is considered that the distance Lop from the origin to the detection point P is not sufficiently larger than the coordinates (Xj, Yj) of the ultrasonic receiving element 24 from the origin (near field), the i-th ultrasonic transmitting element 23 is used. The time Tij (θ, φ, Zp) until the transmitted ultrasonic wave is reflected from the origin at the detection point P in the θ and φ directions and reaches the j-th ultrasonic wave receiving element 24 is calculated by the following equation (4). Will be done.
Tij (θ, φ, Zp) = (Lip + Lpj) / C ... (4)

Further, the distance measuring unit 307a calculates the distance Lip from the i-th ultrasonic wave transmitting element 23 to the detection point P by using the following equation (5).

Further, the distance measuring unit 307a calculates the distance Lpj from the detection point P to the j-th ultrasonic wave receiving element 24 by using the following equation (6).

Assuming that the signal transmitted by the i-th ultrasonic transmitting element 23 and received by the j-th ultrasonic receiving element 24 is Rij (t), the signals of all the ultrasonic receiving elements 24 are totaled in the θ and φ directions from the origin. The signal synthesis value Snear (θ, φ, Zp) at the detection point P position of is expressed by the following equation (7).

The position recognition unit 307b is provided with adjustment parameters (combination of θ, φ, Zp) for a range in which the article W1 is considered to exist. Then, the position recognition unit 307b reflects the signal composite value Snear (θ, φ, Zp) of the reflected wave received by the plurality of ultrasonic wave receiving elements 24 by adjusting it using a plurality of adjustment parameters. A signal showing a signal composite value Snear (θ, φ, Zp) that strongly reacts to a point can be obtained. Then, the position recognition unit 307b can recognize the direction in which the reflection point exists from the angles (θ, φ) applied when it reacts strongly to the reflection point.

At this time, the position recognition unit 307b obtains Zpmax (θ, φ) at which the signal synthesis value Snear (θ, φ, Zp) is maximized. Therefore, the position recognition unit 307b can calculate the distance Dnea (θ, φ) to the article W1 in the θ, φ direction. The formula (8) for calculating the distance Dnear (θ, φ) is shown below.
Dnear (θ, φ) = Zpmax (θ, φ) / cosθ / cosφ ・ ・ ・ (8)

In this way, the direction of the detection point P from the origin can be calculated based on the two-dimensional distance Dnea (θ, φ) from the origin to the detection point P and the angles θ, φ.

In this way, the position recognition unit 307b of the processing unit 307 can calculate the distance and direction from the origin to the detection point P, in other words, the position of the detection point P, based on the reception signal of each ultrasonic wave receiving element 24.

As described above, the sensor device 20 acquires the shape and position of the article W1 existing in the sensing range as a sensing result by sensing from above. More specifically, the sensor device 20 transmits ultrasonic waves and receives reflected waves reflected at each position (detection point) in the sensing range. Then, the sensor device 20 acquires the shape of the surface (upper surface, side surface) of the article W1 existing in the sensing range as a physical quantity such as ultrasonic intensity from the difference between the ultrasonic wave and the reflected wave. In the present embodiment, since a plurality of ultrasonic wave transmitting elements 23 and a plurality of ultrasonic wave receiving elements 24 are used to transmit and receive ultrasonic waves and reflected waves, the ultrasonic wave intensity at each detection point is captured three-dimensionally. This makes it possible to accurately obtain the shape and position of the surface of the article W1.

The sensor device 20 creates a two-dimensional distance image that summarizes the calculated distances of the plurality of detection points P. The distance image is, for example, a distance image image. The distance image image has an image area corresponding to the mounting area R1. Then, the distance of the detection point P included in the mounting area R1 is indicated by the pixel value (for example, the brightness value) of the pixel corresponding to the detection point P.

Next, the sensing result and the captured image of the article group W will be described with reference to FIGS. 14 to 17. 14 to 17 are diagrams for explaining an article group of an embodiment, a sensing result, and an example of a captured image. In the examples shown in FIGS. 14 to 17, the difference in luminance value, in other words, the distance to the detection point P is shown by the shade of color. For example, the darker the color, the closer it is to the sensor head 21, and the lighter the color, the farther it is from the sensor head 21.

In FIG. 14, the article W1 constituting the article group W is an example of corrugated cardboard. In this example, five articles W1 are stacked in two stages, the lower article W1 is four, and the upper article W1 is one. In the sensing result, the dot hatched portion 1401 shows that the distance from the sensor head 21 is longer than that of the black-painted portion 1402. In the example shown in FIG. 14, since the distance from the origin to the upper surface of the lower article W1 and the distance from the origin to the upper surface of the upper article W1 are different, they can be regarded as two groups. Therefore, the position recognition unit 307b of the processing unit 307 recognizes them as separate objects (article W1). Further, at this time, although not shown in FIG. 14, the boundary portion between the plurality of articles W1 in the lower row is calculated to have a longer distance from the origin than the upper surface of the article W1 due to the shape of the corner portion of the article W1. Will be done. As a result, after the upper article W1 is moved, the distance measuring unit 307a and the position recognition unit 307b perform processing, so that it can be recognized that there are four lower articles W1.

FIG. 15 is different from the example of FIG. 14 in that the article group W is covered with the sheet E. In the sensing results, the dot-hatched portions 1502 to 1505 show that the calculated distance is longer than that of the black-painted portion 1501. Further, the square frame-shaped portions 1502 to 1505 that surround the black-painted portion 1501 and are provided with different types of dot hatching indicate that the calculated distance is long from the inside to the outside. In this example, since the calculated distance changes continuously in the portion of the inclined surface Ea of the sheet E, the position recognition unit 307b of the processing unit 307 recognizes the article group W as one cohesive object.

On the other hand, as can be seen from FIGS. 14 and 15, an image taken when the article group W is not covered with the sheet E (FIG. 14) and an image taken when the article group W is covered with the sheet E (FIG. 14). 15) is substantially the same. This is because the transparent sheet E is not imaged. Therefore, it is difficult to determine whether or not the article group W is covered with the sheet E only from the captured image. However, in the present embodiment, since the sensing result from the sensor device 20 is used, it is possible to determine whether or not the article group W is covered with the sheet E. As a result, the moving device 50 can perform appropriate gripping of the article W1 in consideration of whether or not it is covered with the seat E.

In FIG. 16, the article W1 constituting the article group W is an example of a PET bottle. In the example shown in FIG. 16, six articles W1 are arranged in a row and in a matrix. In the sensing result, the dot hatched portion 1602 shows that the distance from the sensor head 21 is longer than that of the black-painted portion 1601. Further, the white background region 1603 has a longer distance than the dot hatched portion 1603, and can be recognized as a distance equivalent to that of the pallet 2. From this result, the position recognition unit 307b of the processing unit 307 recognizes the six articles W1 as separate objects from each other. In this case, when the grip portion 113 is, for example, a suction type, the suction target portion (grip target portion) is one black-painted portion (PET bottle cap).

FIG. 17 is different from the example of FIG. 16 in that the article group W is covered with the sheet E. In the sensing results, the two types of dot hatched portions 1702 and 1703 show that the distance from the sensor head 21 is longer than that of the black-painted portion 1701. Further, of the two types of dot hatching 1702 and 1703, the dot hatching 1703 indicates that the distance from the sensor head 21 is shorter than that of the other dot hatching 1702. In the example shown in FIG. 17, a point cloud having a distance close to the distance of the black-painted portion 1701 is also obtained in the portion between the black-painted portion (cap) 1701 (the inclined surface Ea of the sheet E). That is, in the example shown in Example 17, the distance by the dot hatching 1703 is calculated for the region covered with the sheet E between the black-painted portions 1701 (PET bottle caps). Since the distance calculated from the dot hatching 1703 is closer to the sensor head 21 than the distance calculated from the dot hatching 1702 showing the outer surface of the PET bottle, the position recognition unit 307b of the processing unit 307 sets the article group W to one. Recognize as a cohesive object. When the grip portion 113 is, for example, a suction type, the suction target portion (grasping target portion) is six black-painted portions (PET bottle caps). In this way, the gripping target portion of the gripping portion 113 changes between the example of FIG. 16 and the example of FIG.

On the other hand, as can be seen from FIGS. 16 and 17, an image taken when the article group W is not covered with the sheet E (FIG. 16) and an image taken when the article group W is covered with the sheet E (FIG. 16). 17) is substantially the same. This is because the transparent sheet E is not imaged. Therefore, it is difficult to determine whether or not the article group W is covered with the sheet E only from the captured image. However, in the present embodiment, since the sensing result from the sensor device 20 is used, it is possible to determine whether or not the article group W is covered with the sheet E. As a result, the moving device 50 can appropriately grip the article group W in consideration of whether or not it is covered with the seat E.

Next, the flow of the article detection process executed by the sensor device 20 will be described. FIG. 18 is a flowchart of an article detection process executed by the sensor device of the embodiment.

As shown in FIG. 18, the sensor device 20 transmits and receives ultrasonic waves to obtain a signal Rij (t) (S101). Specifically, the sensor device 20 transmits ultrasonic waves one by one or in groups of a plurality of (1st to nth) ultrasonic wave transmitting elements 23 in order, and a plurality of (1st to mth) ultrasonic waves are transmitted. The signal Rij (t) received by the sound wave receiving element 24 is obtained.

Next, the sensor device 20 obtains the distance Dfar (θ, φ) or the distance Dnea (θ, φ) according to the equations (1) to (3) or the equations (4) to (7) (S102). Hereinafter, the distance D (θ, φ) may be used as a general term for the distance Dfar (θ, φ) and the distance Dnea (θ, φ).

Next, the sensor device 20 combines a group of detection points P (hereinafter, also referred to as a detection point group) in which the distance D, the angles θ, and φ are close to each other within a predetermined range from the distance D (θ, φ) as an article. By summarizing, a plurality of detection points P are grouped (S103). Then, the sensor device 20 assigns k (k is a positive number from 1 to p) as identification information to the grouped articles (detection point group). That is, the sensor device 20 (processing unit 307) uses a group of detection points P corresponding to pixels whose distance value difference is within a specified range among a plurality of pixels of the distance image as an article, and sets the reference point (reference point) of the article (the article). Recognize the distance and position from the origin). At this time, for example, in the example of FIG. 14, the distance from the origin to the upper surface of the lower article W1 and the distance from the origin to the upper surface of the upper article W1 are two different groups, so that they are separate. It is recognized as an object (article W1). Further, in the example of FIG. 15, since the calculated distance changes continuously in the portion of the inclined surface Ea of the sheet E, one group of the calculated distances is longer than the distance from the origin to the mounting region R1. Recognized as a cohesive object.

Next, the sensor device 20 obtains the distance from the origin to the article k from the distance D of the detection point group belonging to the grouped articles (k), and from the angles θ, φ and the distance D, the position of the article (k). ((X, Y) = (Dsinθ, Y = Dsinφ)) is obtained (S104).

Next, the recognition process of the article W1 executed by the control device 60 will be described. FIG. 19 is a flowchart of an article recognition process executed by the control device of the embodiment.

As shown in FIG. 19, the control device 60 acquires an image captured by the camera 30 (S201).

Next, the control device 60 recognizes the article W1 from the captured image and the sensing result (S202). Specifically, the control device 60 recognizes the article W1 by obtaining the shading edge or texture of the portion corresponding to θ and φ of the article obtained in S104 of FIG. 18 in the captured image. If there is a shading edge or a texture, it can be seen that the article W1 can be captured in the captured image. Further, by registering and collating the texture features in advance, it is possible to determine the type of the article W1. That is, the control device 60 can recognize the position and type of the article W1 from the color and texture of the article W1 in the captured image.

As described above, in the present embodiment, since the distance of the article W1 is obtained by using ultrasonic waves, a plurality of black boxes, transparent boxes, and transparent sheets (wrapping sheets, shrink films (films)) are put together. The position of the article W1 such as a cardboard box or a PET bottle can be recognized.

Further, as can be seen from the above, the cargo handling system 1 of the present embodiment detects the article W1 based on the sensing result by the ultrasonic wave output to the article W1 from above the article group W including one or more articles W1. And, based on the recognition result of the article W1 based on the image captured from above the article group W, the article W1 is grasped.

As described above, in the present embodiment and the modified example, the sensor device 20 grips the article W1 and moves the article W1 based on the sensing result for the loading area R1 on which the article W1 is placed. Is provided in. The sensor device 20 is arranged in a region R2 located above the mounting region R1 and at a position facing the mounting region R1, and is one or more that transmits ultrasonic waves or radio waves as a transmission wave toward the mounting region R1. It includes an ultrasonic wave transmitting element 23 (first transmitting element) and one or more ultrasonic wave receiving elements 24 (first receiving element) arranged in the region R2 and receiving the reflected wave of the transmitted wave. At least one or more of the ultrasonic transmitting element 23 and the ultrasonic receiving element 24 are arranged near the outer edge of the region R2. Therefore, since the article arranged in the mounting area R1 can be detected regardless of whether or not it exists in the vicinity of the outer edge, the recognition accuracy of the article is improved.

In the above-described embodiment and modification, the sensor device 20 includes a plurality of sensor heads 21 having an ultrasonic wave transmitting element 23A and an ultrasonic wave receiving element 24A. A plurality of sensor heads 21 are provided at each corner of the region R2. Therefore, according to the present embodiment, since a plurality of sensor heads 21 are provided, the accuracy of the sensing result is improved.

Further, in the present embodiment and the modified example, the sensor head 21 has a plurality of ultrasonic wave transmitting elements 23A and a plurality of ultrasonic wave receiving elements 24A. Therefore, according to the present embodiment, the accuracy of the sensing result is improved.

Further, in the present embodiment and the modified example, the sensor device 20 is located above the mounting region R1 and below the region R2, and transmits a transmitted wave from the side of the article W1 mounted on the mounting region R1. The ultrasonic wave transmitting elements 23B and 23C (second transmitting elements) for transmitting toward the article W1 are provided. Therefore, according to the present embodiment, the accuracy of the sensing result is improved.

Further, in the modified example, the sensor device 20 is located above the mounting region R1 and below the region R2, and receives reflected waves on the side of the article W1 mounted on the mounting region R1. It includes receiving elements 24B and 24C (second receiving elements). Therefore, according to the present embodiment, the accuracy of the sensing result is improved.

Further, in the present embodiment, the cargo handling system 1 grips the sensor device 20 that senses the article W1 as a transmission wave with respect to the loading area R1 on which the article W1 is placed, and the article W1. A moving device 50 to be moved and a control device 60 for controlling the moving device 50 based on the sensing result of the sensor device 20 are provided. Therefore, since the article arranged in the mounting area R1 can be detected regardless of whether or not it exists in the vicinity of the outer edge, the recognition accuracy of the article is improved.

Further, in the present embodiment, the cargo handling system 1 is provided above the loading area R1 and includes a camera 30 (imaging device) that images the article W1 mounted on the mounting area R1 from above. The element group 25 is arranged around the camera 30. Therefore, it is easy to reduce the number of cameras 30 to one, for example.

(Modification example 1)
In the above-described embodiment, as shown in FIG. 4, an example in which a plurality of ultrasonic wave transmitting elements 23B and ultrasonic wave transmitting elements 23C are provided on the side portion of the frame 40 has been described. However, the above-described embodiment is not limited to the example in which a plurality of ultrasonic wave transmitting elements 23B and ultrasonic wave transmitting elements 23C are provided on the side portion of the frame 40. Therefore, in this modified example, an example in which the ultrasonic wave receiving element 24B and the ultrasonic wave receiving element 24C are provided on the side portion of the frame 40 will be described.

The ultrasonic wave receiving element 24B and the ultrasonic wave receiving element 24C are mounted in the first direction connecting the mounting area R1 and the area R2, similarly to the ultrasonic wave transmitting element 23B and the ultrasonic wave transmitting element 23C shown in FIG. It is located between the placement area R1 and the area R2. The ultrasonic wave receiving element 24B and the ultrasonic wave receiving element 24C are provided on the side portion of the frame 40. That is, the ultrasonic receiving element 24B and the ultrasonic receiving element 24C are supported outside the mounting region R1 and the region R2 in the second direction intersecting the first direction. The ultrasonic wave receiving element 24B and the ultrasonic wave receiving element 24C are arranged so as to face a direction in which the reflected wave reflected by the article existing between the mounting region R1 and the region R2 can be received. As a result, the ultrasonic wave receiving element 24B and the ultrasonic wave receiving element 24C of the present modification can receive the reflected wave even when the transmitted wave transmitted from the ultrasonic wave transmitting element 23 is reflected by the inclined surface of the article W1.

In this modification, the inclination of the article W1 collected by the sheet E existing between the mounting region R1 and the region R2 of the transmitted wave transmitted from the ultrasonic transmitting element 23A provided on the sensor head 21. Reflects on the surface Ea. As a result, the reflected wave is generated in the lateral direction. The lateral direction is a direction substantially perpendicular to the vertical direction connecting the mounting area R1 and the area R2.

Then, the ultrasonic wave receiving element 24B or the ultrasonic wave receiving element 24C receives the reflected wave from the side direction. The sensor device 20 can recognize the article W1 collected on the sheet E by the signal indicating the reflected wave from the ultrasonic wave receiving element 24B or the ultrasonic wave receiving element 24C.

Further, the present invention is not limited to the example in which the ultrasonic wave transmitting element 23 or the ultrasonic wave receiving element 24 is provided on the side portion of the frame 40, and the ultrasonic wave transmitting element 23 and the ultrasonic wave receiving element 24 may be provided. Further, the sensor head 21 may be provided on the side of the frame 40.

(Modification 2)
In the above-described embodiment, as shown in FIG. 4, an example in which a plurality of ultrasonic wave transmitting elements 23B and ultrasonic wave transmitting elements 23C are provided on the side portion of the frame 40 has been described. However, the above-described embodiment is not limited to the method in which a plurality of ultrasonic wave transmitting elements 23B and ultrasonic wave transmitting elements 23C are provided on the side portion of the frame 40. Therefore, in the first modification, one ultrasonic wave transmitting element 23B is provided for each pillar member 43, and the ultrasonic wave transmitting element 23B can move in the vertical direction along the pillar member 43.

The sensor device 20 of this modified example moves the ultrasonic transmission element 23B in the vertical direction by transmitting a control signal to the ultrasonic transmission element 23B. In the sensor device 20 of the present embodiment, when a plurality of articles W1 grouped by the sheet E are present in the mounting area R1, the ultrasonic transmission element 23B is moved up and down in order to measure the height of the articles W1. Move in the direction. The plurality of articles W1 grouped by the sheet E have an inclined surface Ea as in the above-described embodiment. Therefore, the ultrasonic wave receiving element 24A of the sensor head 21 receives the reflected wave of the transmitted wave transmitted from the ultrasonic wave transmitting element 23B.

For example, the sensor device 20 determines whether or not the ultrasonic wave receiving element 24A of the sensor head 21 has received the reflected wave of the transmitted wave after the ultrasonic wave transmitting element 23B transmits the transmitted wave. When the reflected wave is received, the ultrasonic wave transmitting element 23B is moved and controlled upward. By repeating the control, the sensor device 20 can recognize the heights of the plurality of articles W1 grouped by the transparent film.

In this modification, an example of moving the ultrasonic transmitting element 23B in the vertical direction has been described, but the present invention is not limited to an example of moving the ultrasonic transmitting element 23B in the vertical direction, and the ultrasonic receiving element 24B is moved up and down. It may be moved in the direction.

In this modification, the heights of the plurality of articles W1 grouped on the sheet E can be detected by performing the above-mentioned control, so that the moving device 50 appropriately grips the plurality of articles W1 grouped on the sheet E. It becomes possible to do.

Further, in this modification, a case where the ultrasonic wave transmitting element 23B provided on the side portion of the frame 40 is moved and controlled has been described. However, the movement control of the ultrasonic wave transmitting element 23B is not limited, and the ultrasonic wave receiving element 24B is provided when the ultrasonic wave receiving element 24B is provided on the side of the frame 40 as in the first modification. Movement control may be performed.

In the above-described embodiment and modification, an example in which the sensor head 21 and the camera 30 are provided in the region R2 having substantially the same area as the mounting region R1 will be described. However, the area R2 and the mounting area R1 in the upper frame member 42 are not limited to substantially the same area, and are arranged in the mounting area R1 when the transmitted wave transmitted from the area R2 is transmitted. It suffices that the area of the region R2 and the area of the mounting region R1 correspond to each other so that the reflected wave reflected by the mirror surface can be received by the article.

As described above, according to the above-described embodiment and modification, the ultrasonic wave transmitting element 23A is arranged along the vicinity of the outer edge of the region R2 by providing the sensor head 21 at each corner of the region R2. As a result, the detection accuracy of the article W1 near the outer edge of the mounting region R1 is improved. As a result, the cargo handling system 1 can easily grip the article W1 existing in the loading region R1 regardless of whether or not it exists in the vicinity of the outer edge.

The program executed by the control device 60 and the program executed by the sensor device 20 of each of the above-described embodiments are files in an installable format or an executable format, such as a CD-ROM, a flexible disk (FD), and a CD-R. , DVD (Digital Versailles Disk) and the like, which are recorded and provided on a computer-readable recording medium.

Further, the program executed by the control device 60 and the program executed by the sensor device 20 of each of the above-described embodiments are provided by storing them on a computer connected to a network such as the Internet and downloading them via the network. It may be configured as follows. Further, the program executed by the control device 60 and the program executed by the sensor device 20 of each of the above-described embodiments may be provided or distributed via a network such as the Internet. Further, the program of each of the above-described embodiments may be configured to be provided by incorporating it into a ROM or the like in advance.

The program executed by the control device 60 of each of the above-described embodiments has a module configuration including each of the above-mentioned units (recognition unit, detection unit, planning unit, planning execution unit, output unit), and is used as actual hardware. When the CPU (processor) reads a program from the storage medium and executes it, each of the above units is loaded on the main storage device, and the recognition unit, the detection unit, the planning unit, the planning execution unit, and the output unit are on the main storage device. It is supposed to be generated.

The program executed by the sensor device 20 of each of the above-described embodiments has a module configuration including the above-mentioned parts (distance measuring unit, position recognition unit), and the CPU (processor) stores the above as the actual hardware. By reading the program from the medium and executing it, each of the above-mentioned parts is loaded on the main storage device, and the distance measuring unit and the position recognition unit are generated on the main storage device.

In the above embodiment, the sensor device 20 may detect the article W1 by transmitting and receiving radio waves (millimeter waves) instead of ultrasonic waves. In this case, the sensor device 20 has a radio wave transmitting element and a radio wave receiving element instead of the ultrasonic wave transmitting element 23A and the ultrasonic wave receiving element 24A.

Further, in the above embodiment, an example in which a plurality of sensor heads are arranged at intervals is shown, but the present invention is not limited to this. For example, the sensor head may have an endless shape (annular shape) along the outer edge R1a of the mounting region R1 in the line of sight from above the sensor head.

Further, in the above embodiment, the ultrasonic wave transmitting element 23A and the ultrasonic wave receiving element 24A are provided, respectively, but the present invention is not limited thereto. For example, instead of the ultrasonic transmitting element 23A and the ultrasonic receiving element 24A, an ultrasonic element having an ultrasonic transmitting / receiving function of the ultrasonic transmitting element 23A and the ultrasonic receiving element 24A may be provided.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

1 ... Cargo handling system 20 ... Sensor device (sensor device)
21 ... Sensor head 23 ... Ultrasonic transmission element (transmission element)
23A ... Ultrasonic transmission element (first transmission element)
23B, 23C ... Ultrasonic transmission element (second transmission element)
24 ... Ultrasonic receiving element (receiving element)
24A ... Ultrasonic receiving element (first receiving element)
24B, 24C ... Ultrasonic receiving element (second receiving element)
25 ... Element group 30 ... Camera (imaging device)
50 ... Mobile device R1 ... Placement area R2 ... Area W1 ... Article

Claims (12)

A first area having a first area corresponding to a loading area of a loading area on which an article can be handled by a cargo handling device, and existing at a position above the loading area and facing the loading area. A first transmitting element that is located in a region and transmits ultrasonic waves or radio waves as a transmission wave toward the previously described region.
A first receiving element, which is arranged in the first region and receives a reflected wave of the transmitted wave, is provided.
At least one or more of the first transmitting element and the first receiving element are arranged near the outer edge of the first region.
Sensor device. At least one or more of the first transmitting element and the first receiving element are arranged near the outer edge of the first region.
The sensor device according to claim 1. The first region is a polygon,
A sensor head having the first transmitting element and the first receiving element is provided at each corner of the first region.
The sensor device according to claim 1 or 2. The sensor head is polygonal
The first transmitting element is arranged at each corner of the sensor head.
The sensor device according to claim 3. With respect to the article existing between the pre-described area and the first area, the transmitted wave is arranged at a position where the transmitted wave can be transmitted from a direction substantially perpendicular to the direction connecting the pre-described area and the first area. The second transmitting element is further provided.
The sensor device according to any one of claims 1 to 4. A plurality of the second transmitting elements are provided in a direction connecting the previously described pre-existing region and the first region.
The sensor device according to claim 5. The second transmitting element is movably arranged in a direction connecting the previously described pre-arranged region and the first region.
The sensor device according to claim 5. In the first direction connecting the pre-described area and the first area, in the second direction located between the pre-described area and the first area and intersecting the first direction. A second receiving element that is supported outside the pre-described area and is arranged so as to face a direction in which the reflected wave reflected by the article existing between the pre-described area and the first area can be received. , Further prepare,
The sensor device according to any one of claims 1 to 5. A plurality of the second receiving elements are provided in the direction connecting the previously described pre-existing region and the first region.
The sensor device according to claim 8. The second receiving element is movably arranged in a direction connecting the previously described pre-arranged region and the first region.
The sensor device according to claim 8. The sensor head is square and
The first transmitting element is provided at each of the four corners of the sensor head.
The first area is a quadrangle.
The sensor heads are provided at each of the four corners of the first region.
The sensor device according to claim 4. A first area having a first area corresponding to a loading area of a loading area on which an article can be handled by a cargo handling device, and existing at a position above the loading area and facing the loading area. A first transmitting element that is located in a region and transmits ultrasonic waves or radio waves as a transmission wave toward the previously described region.
A first receiving element arranged in the first region and receiving the reflected wave of the transmitted wave, and
An imaging device arranged near the center of the first region and capable of capturing the direction of the previously described region,
A moving device that moves the first article by grasping the first article recognized based on the reflected wave received by the first receiving element and the captured image captured by the imaging device. Prepare,
At least one or more of the first transmitting element and the first receiving element are arranged near the outer edge of the first region.
Cargo handling system.

Images