JP7259877B2 - DETECTION METHOD, DETECTION DEVICE, DETECTION SYSTEM AND COMPUTER PROGRAM - Google Patents

Description

TECHNICAL FIELD The present invention relates to a detection method, a detection device, a detection system, and a computer program for detecting objects in a sorting machine with higher speed and accuracy.

2. Description of the Related Art In order to transport articles, a sorter that sorts articles by destination is used in a distribution center that is responsible for shipping a large number of articles. By reading the item identification code, the sorting machine identifies the input item and discharges the identified item to a container for packing or a shooter section in which a box is prepared.

The discharge of the articles to the chute section is realized by tilting a tray on which the articles are placed at a position corresponding to the sorting destination frontage of the chute section by means of a tilting mechanism, and by causing the articles to slide out, or by sliding the articles out by means of a shoe. Realized by extrusion.

There are cases in which articles are left in the conveying section of the sorting machine due to insufficient discharge to the chute section. Events can occur where the article is light and does not slide out, or rotates and is not extruded. Therefore, the sorting machine is provided with a detection device for detecting remaining articles that have not been discharged.

Patent Literature 1 discloses the use of a determination device that includes a camera and determines whether or not a transported object is mounted on a transport belt based on reflected light. Japanese Patent Laid-Open No. 2003-100000 describes that the judgment device may make an erroneous judgment because it cannot distinguish between reflected light from the conveying belt and reflected light from the remaining items.

JP-A-11-314720

Apparatuses that detect the presence or absence of articles by image processing can cause sorter malfunctions even at a low probability, as presented in US Pat. Sorting machines used in distribution centers have high transport speeds and carry a wide variety of items, such as lightweight, compact, and transparent packages. For this reason, a detection device using a sensor that measures the displacement of the reflected light from the light-emitting element or whether or not the light projected from the light-emitting element arrives is also used without using image processing. However, since these sensors measure the reflection or transmission of the beam-like light emitted from the light-emitting element at a point, even if a plurality of sensors are used, an undetectable area is generated. Also, sensors that measure the displacement of reflected light may not be able to detect the presence or absence of thin articles such as magazines and shirts.

An object of the present invention is to provide a detection method, a detection device, a detection system, and a computer program for detecting objects in a sorting machine with higher speed and accuracy.

A detection method according to an embodiment of the present disclosure acquires a captured image of a location where an article to be conveyed is placed and moves, and detects a detection target range including the location based on a reference point in the acquired captured image. to acquire at least one of the hue, saturation, and brightness of pixels included in the detection target range in the captured image, and the hue, saturation, and brightness of any pixel in the detection target range Judging whether any one of them is out of the stored range for each of hue, saturation, and lightness, and if it is judged that it is out of the range, it includes a process of detecting the presence of an article at the location. .

A detection device according to an embodiment of the present disclosure includes an image acquisition unit that acquires a captured image of a location where an article being transported moves, and a detection target range that includes the location based on a reference point in the acquired captured image. a range determination unit to determine; a data acquisition unit that acquires at least one of hue, saturation, and brightness of pixels included in the detection target range in the captured image; hue of any pixel within the detection target range; a judgment unit for judging whether any of the saturation and lightness is out of the range stored for each of the hue, saturation, and lightness; A detection unit is provided for detecting the existence of the object.

A detection system according to an embodiment of the present disclosure is installed at a location on a transport path of a sorting machine that sorts input items by destination while transporting them, and illuminates the location with light of different colors from different angles. an imaging device that captures an image of a location illuminated by the lighting device; and a detection device that acquires an image captured by the imaging device and detects whether or not an article exists in the location, The detection device determines a detection target range including the location based on a reference point in the acquired captured image, and determines at least one of hue, saturation, and brightness of pixels included in the detection target range in the captured image. and determine whether any of the hue, saturation, and lightness of any pixel within the detection target range is outside the range stored for each of the hue, saturation, and lightness, and If it is determined that the location is outside, it detects that an article exists at the location.

A computer program according to an embodiment of the present disclosure acquires, in a computer connected to a sorting machine that sorts input articles by destination while conveying them, a captured image of a transportation path of the articles in the sorting machine. determining a detection target range including the location based on a reference point in the acquired captured image, and acquiring at least one of hue, saturation, and lightness of pixels included in the detection target range in the captured image. Then, it is determined whether any of the hue, saturation, and lightness of an arbitrary pixel within the detection target range is out of the range stored for each of the hue, saturation, and lightness. When it is determined that an article exists at the location, a process for detecting the existence of the article is executed.

In the detection method, the detection device, the detection system, and the computer program of the present disclosure, an image obtained from an imaging device installed so as to capture a place where articles such as trays and slats moving on a conveying path in a sorting machine are placed. Detection is performed on the image. A detection target range is determined as a reference point in the captured image, and any of hue, saturation, or lightness of pixels within the detection target range has a value outside the range of values stored in advance. , it is detected that the article is in the picture.

In the detection method according to an embodiment of the present disclosure, the captured image is an image captured by irradiating the spot with light of different colors from different angles.

In the detection method of the present disclosure, light of different colors is irradiated from different angles to the place where the article is placed, and shadows of different colors are projected. Areas with shadows of different colors can be detected by the difference in hue. Even transparent articles can be detected if they are partially shadowed by different light from different angles.

A detection method according to an embodiment of the present disclosure determines, as the reference point, a specific point in a range in which light projected in a specific color or a specific shape is captured in the captured image.

In the detection method of the present disclosure, the captured image includes light of a specific color or a specific shape that is projected onto a portion of the location. A specific point in the range in which light of a specific color or shape is captured is used as a reference point, and the detection target range can be determined by reducing the effects of transport speed, imaging speed, and the like.

In the detection method of an embodiment of the present disclosure, the specific color of light is complementary to at least one of the different colors.

In the detection method of the present disclosure, one of the lights emitted from different angles and the light of a specific color or shape are in a complementary color relationship, so that the light of a specific color or shape is captured. The range is likely to be extracted from the captured image.

A detection method according to an embodiment of the present disclosure pre-stores ranges of hue, saturation, and brightness in an image captured by irradiating light of different colors from different angles to the location where no article exists. Including processing to keep

In the detection method of the present disclosure, data on the range of hue, saturation, and lightness in a state where light of different colors is irradiated from different angles in a captured image of a location such as a tray or slat captured when there is no article. Pre-stored, detection is performed by comparison with this data. It is possible to compare not only the state before the sorting machine is used, but also empty trays and slats whose color may change due to deterioration over time after the start of use.

In the detection method according to an embodiment of the present disclosure, the captured image is a captured image of a tray on which articles conveyed by the sorting machine are placed.

In the detection method of the present disclosure, detection is performed by imaging a tray of articles. As a result of comparison with the hue, saturation, and brightness of the color of the tray, it is possible to detect articles with exteriors of various materials and colors by assuming that an article is detected when there is a difference in any one of them. .

According to the present disclosure, the presence/absence of an object to be conveyed can be detected on the surface using an image, and various objects such as transparent packages and thin objects can be detected with high accuracy.

It is a schematic diagram of the detection method of this embodiment. It is a schematic diagram of the imaging method by the camera with respect to a tray. It is a block diagram which shows the structure of a detection apparatus. 4 is a flowchart showing an example of a detection processing procedure by a processing unit of the detection device; It is a figure which shows an example of the captured image of a tray. It is a figure which shows an example of a detection object range and a mask object range. FIG. 4 is a diagram showing an example of a captured image of a tray on which articles remain; 4 is a distribution map of extracted hues; FIG. 4 is a distribution map of extracted chroma. FIG. 4 is a distribution map of extracted brightness. FIG. FIG. 10 is a diagram showing another example of a captured image of a tray on which articles remain; 4 is a distribution map of extracted hues; FIG. 4 is a distribution map of extracted chroma. FIG. 4 is a distribution map of extracted brightness. FIG. 9 is a flow chart showing an example of a processing procedure of the detection device in modification 1; 10 is a flow chart showing an example of a processing procedure of a detection device in modification 2;

The present disclosure will be specifically described with reference to the drawings showing the embodiments thereof.

FIG. 1 is a schematic diagram of the detection method of this embodiment. FIG. 1 shows a sorting machine 100 using a detection method, a camera 101 mounted to photograph the articles conveyed by the sorting machine 100, and a detection device 2 connected to the camera 101 and the control unit 10 of the sorting machine 100. including. The camera 101 is attached so as to photograph articles conveyed upstream and/or downstream of the chute section 13 sorted by the sorting machine 100 . Based on the image data captured by the camera 101, the detection device 2 notifies the sorting machine 100 of the detection result of the presence or absence of articles to be conveyed.

The sorting machine 100 is divided into an input section 11 , a conveying section 12 and a chute section 13 . In the loading unit 11, an operator (or a robot) causes a reader to read the identification code attached to the article and places the article on an empty tray 121. FIG.

The conveying unit 12 includes a plurality of trays 121 that are connected and run along rails 122 that are provided as, for example, endless annular tracks, as shown in FIG. As shown in FIG. 1, the conveying section 12 is provided with a plurality of trays 121 so as to circulate in parallel with the horizontal plane. Of course, the structure of the conveying unit 12 is not limited to this, and may be a structure in which the plurality of trays 121 travel linearly, or a structure in which the plurality of trays 121 travel so as to spirally circulate. good. The running speed of the tray 121 is, for example, 120 m/min. The running speed is determined by the setting of the sorting machine 100 and controlled by the control section 10, which will be described later.

Identification data is attached to each of the plurality of trays 121 . The transport section 12 is provided with a detection mechanism that detects where at least a specific tray 121 among the plurality of trays 121 is present in the transport section 12 . The transport unit 12 can detect the positions of the trays in the transport unit 12 for each of the plurality of trays according to the order in which the trays 121 are connected. The detection mechanism is, for example, a mechanism including an encoder attached to a motor of a drive unit that makes the tray 121 run, and a detection unit that receives pulse signals output from the encoder and detects the position. Alternatively, the detection mechanism may detect the position using a reader that reads an identification tag attached to the tray 121 at a specific location.

The tilting mechanism for tilting the tray 121 is realized, for example, by a configuration in which the supporting portion of the rail 122 that supports the tray 121 is bendable. The tilting mechanism may be a mechanism that pushes up and tilts a part of the lower surface of the tray 121 .

The chute section 13 is provided parallel to a part of the rails 122 of the tray 121 of the transport section 12 and receives articles discharged from the tray due to the inclination of the tray 121 . On the chute 13, as shown in FIG. 1, distribution materials P, which are small containers or cardboard boxes used for transportation, are placed. The chute section 13 is a sectioned workbench, and a packing worker may pack the articles discharged onto the workbench into the physical distribution material P. As shown in FIG.

The input section 11 , the conveying section 12 and the chute section 13 of the sorting machine 100 are connected to the control section 10 by signal lines and controlled by the control section 10 . The control unit 10 detects that the identification code has been read and the identification code at the timing when the sorting operator causes the reader 112 to read the identification code of the article in the input unit 11 . Based on the detected identification code, the control unit 10 identifies the tray 121 into which the articles have been placed by the loading unit 11 by means of a device that detects whether or not the tray 121 is empty using, for example, a weight sensor. Instead of the weight sensor, the detecting device 2 may detect from an image captured by a camera 101 provided immediately downstream of the loading section 11 . Thereby, the control unit 10 can specify which tray 121 has which article with which identification code is placed. The control unit 10 determines the trays 121 to be tilted by the chute unit 13 according to a preset sorting schedule or control from an external device, and instructs the transport unit 12 to tilt the determined trays 121 .

The detection device 2 of the present embodiment detects the remaining articles by copying the tray 121 traveling downstream of the chute section 13 . If an article remains in the tray 121 which should have been tilted by the control from the control section 10 and the article was discharged, the article has not been discharged to the chute section 13 . The detection device 2 notifies the control unit 10 when detecting the existence of the remaining product. Based on the detection notification from the detection device 2, the control unit 10 instructs the conveying unit 12 to attempt to eject again if there is a remaining item, or attempts to eject to another error buffer unit. good too.

The detection device 2 may be used as a sensor for detecting whether or not an article is placed on the tray 121 immediately downstream of the input section 11, as described above.

In the sorting machine 100 of this embodiment, as described above, the tray 121 travels at a speed of 120 m/min. Since detection of the tray 121 moving at a speed of 120 m/min (=2000 mm/sec=2 mm/msec) is performed by image processing, the processing of the detection device 2 should be as fast as possible or simple to speed up. Something is required.

FIG. 2 is a schematic diagram of a method of photographing the tray 121 with the camera 101. As shown in FIG. The tray 121, which is an object for detecting the presence or absence of articles, has side walls provided along both short sides of a substantially rectangular bottom plate. Both side walls are formed so that height becomes low, so that one long side is approached. Threaded holes in the bottom plate are connected to rails 122 and a tilt mechanism to allow articles to slide down one open long side.

The camera 101 is fixed at a distance such that the entire tray 121 is included in the imaging range so that the camera 101 can be viewed from above in a direction substantially perpendicular to the bottom plate of the tray 121 . In the present embodiment, the camera 101 is configured so that the traveling direction of the tray 121 corresponds to the horizontal direction in the captured image, and the direction orthogonal to the traveling direction of the tray 121 corresponds to the vertical direction in the captured image. installed to correspond to It may be installed so that the running direction corresponds to the vertical direction. The camera 101 is not limited to a configuration in which the photographing direction (depth) thereof is orthogonal to the bottom plate, and may be provided with a slight inclination. The imaging range of the camera 101 includes the entire bottom plate of the tray 121 and is preferably oriented as orthogonally as possible to capture the shadows cast by the articles.

A lighting device 103 is installed for the camera 101 . The lighting device 103 includes a first lighting device 31 , a second lighting device 32 and a third lighting device 33 .

The first lighting device 31 is a lighting device that emits light having a width in the running direction of the tray 121 . The first lighting device 31 is, for example, a so-called line laser that emits light in an elongated rectangular shape having a length in the running direction of the tray 121 . In addition, it may be a lighting device that emits light in a specific shape such as a circle or rectangle having a length in the running direction. The first illumination device 31 is used to identify the border portion of the tray 121 in the image. In the first lighting device 31, among the emitted light, the light reflected by the edge of the tray 121 is not reflected by the gap between the trays 121 or by the receiving plate provided below the tray 121. The detecting device 2 detects the discontinuous point between the light.

The second lighting device 32 is a lighting device that irradiates the tray 121 with light obliquely from above. The second lighting device 32 of the present embodiment is installed on the side of the camera 101 and, as shown in FIG. 2, irradiates the bottom plate of the tray 121 with light from two directions at about 45 degrees. The light emitted from the second lighting device 32 is installed so that the shadow of the article placed on the tray 121 is projected in a different direction from the shadow cast by the light from the third lighting device 33 . As shown in FIG. 2, the second lighting device 32 is installed on a vertical plane along the running direction of the tray 121, with normals on the vertical plane at 45 degrees and 135 degrees with respect to the running direction. It has a radiating surface.

The third lighting device 33 is a lighting device that irradiates the plate portion of the tray 121 with substantially parallel light. The third lighting device 33 of this embodiment is installed in a direction parallel to the running direction of the tray 121 and perpendicular to the bottom plate. The object placed on the tray 121 casts a large shadow. As shown in FIG. 2, a plurality of third lighting devices 33 are provided on both sides of the tray 121 in the running direction.

The installation angles and positions of the first lighting device 31, the second lighting device 32, and the third lighting device 33 are examples, and needless to say, are not limited to the forms described above and shown in FIG. As described above, the first lighting device 31 is installed so that the range and boundary where the light projected from the first lighting device 31 is captured can be specified from the captured image. The second lighting device 32 and the third lighting device 33 are installed at angles different from each other. It should be installed so that the shadow cast by the light is clear.

For example, the colors of light from the first lighting device 31, the second lighting device 32, and the third lighting device 33 are green, red, and blue (specifically, cyan), respectively. In the present embodiment, the colors of the light from the second lighting device 32 and the third lighting device 33 that irradiate the tray 121 are set to colors that are close to the color of the tray 121 and complementary colors and that are easy to distinguish from each other. It is preferable to select The color of the light from the first lighting device 31 should be a distinguishable, preferably complementary color, to the color of the light from the second lighting device 32 or the third lighting device 33 for detecting discontinuities as described above. Preferably. Also, the light from the first lighting device 31 needs to be strongly condensed like a laser beam in order to distinguish it from the irradiation light from the second lighting device 32 and the third lighting device 33 . In this embodiment, assuming that the tray 121 has a yellowish color, the colors of light from the second lighting device 32 and the third lighting device 33 are selected to be red and blue, respectively. The color of the light was chosen to be green. The color scheme of the light is not limited to these colors, but as a result of trial and error, the accuracy of detection by the process described later was good with this color scheme.

With the light from the first lighting device 31, the second lighting device 32, and the third lighting device 33 constantly irradiated, the camera 101 monitors the image signal to the detection device 2 with the area where the tray 121 travels as the imaging range. Output.

A sensor 104 may be used to detect when the tray 121 enters the imaging range of the camera 101 . The sensor 104 uses, for example, a distance measuring sensor that measures the reflected displacement of emitted light. The sensor 104 continues to output a signal to the detection device 2 according to the measured distance.

The detection device 2 captures (obtains) an image from the camera 101 triggered by a change in signal level from the sensor 104 or periodically (at a predetermined capture rate).

The configuration and processing of the detection device 2 for detecting whether or not there is an article in the tray 121 from an image photographed by the photographing method shown in FIG. 2 will be described in detail.

FIG. 3 is a block diagram showing the configuration of the detection device 2. As shown in FIG. The detection device 2 includes a processing section 20 and a storage section 21 . The processing unit 20 is a CPU (Central Processing Unit). The processing unit 20 may be a microcontroller. The processing unit 20 reads out and executes the detection program 2P stored in the storage unit 21 to detect the presence or absence of an article by image processing.

The processing unit 20 has general-purpose input/output pins, and inputs/outputs signals to/from the control unit 10 using a PLC (Programmable Logic Controller) of the sorting machine 100 . The processing unit 20 incorporates a temporary memory used for CPU calculations. The processing unit 20 can obtain data indicating the position of a specific tray 121 from the control unit 10 of the sorting machine 100 . For example, the processing unit 20 can acquire the position of the tray 121 by inputting the output from the encoder 124 of the motor that drives the tray 121 and by counting the pulses from the encoder 124 . The processing unit 20 receives an image signal from the camera 101 and receives an output signal from the sensor 104 .

The storage unit 21 is a non-volatile storage medium such as an SSD (Solid State Drive) or hard disk. In order to improve the processing speed, it is preferable to adopt an SSD with a high read speed. The storage unit 21 stores a detection program 2P. The detection program 2P stored in the storage unit 21 may be a copy of the detection program 9P stored in the storage medium 9 read by the processing unit 20 .

The storage unit 21 stores color information of the tray 121 previously stored before detection. The color information may be stored once at the time of setting, or may be periodically stored using an image signal from the camera 101 so as to match the deterioration of the tray 121 over time.

FIG. 4 is a flowchart showing an example of a detection processing procedure by the processing unit 20 of the detection device 2. As shown in FIG. As described above, for one camera 101, the processing unit 20 performs the following processing at a predetermined capture rate (for example, 30 times/second), triggered by the rise of the signal level of the input signal from the corresponding sensor 104. to run. When multiple cameras 101 are connected, the process is executed for each of the multiple cameras 101 .

The processing unit 20 acquires an image from the image signal from the camera 101 (step S1).

The processing unit 20 determines a reference point from a discontinuous portion of the projection shape from the first illumination device 31 in the acquired image (step S2).

The processing unit 20 determines a detection target range in the acquired image based on the reference point determined in step S2 (step S3). The processing unit 20 determines a mask target range in the detection target range determined in step S3 (step S4).

The storage unit 21 stores in advance the coordinate data of the relative position within the image of the detection target range from the reference point determined in step S2. The storage unit 21 stores the coordinate data of the mask target range used in step S4 as a relative position from the detection target range or a relative position from the reference point. The coordinate data may be rewritten during inspection. The area to be masked is a portion corresponding to a concave portion, a screw, or the like existing in the tray 121 . It is set to eliminate the effects of shadows cast by recesses or screws. In the detection process for the tray 121 that does not need to be masked, the process of step S4 can be omitted.

The processing unit 20 acquires data (numerical values) of the hue, saturation, and lightness of each pixel in the detection target range other than the range determined as the mask target range in step S4 (step S5).

In step S5, it takes time to acquire the hue, saturation, and lightness data of all pixels included in the detection target range. may be obtained from

The processing unit 20 selects any range of hue, saturation, or lightness included in the color information stored in advance in the storage unit 21 from the value group of hue, saturation, or lightness acquired in step S5. It is determined whether or not there is a value outside of (step S6).

In step S6, the processing unit 20 determines that there is an outlier value when the numerical value of only the hue is out of the hue range of the pre-stored color information. The processing unit 20 determines that there is an outlier value when the numerical value of only the saturation is out of the pre-stored range of the saturation of the color information. Similarly, the processing unit 20 determines that there is an outlier value when the numerical value of only the lightness is out of the lightness range of the pre-stored color information. That is, in step S6, the processing unit 20 determines whether or not there is a value outside the range of the color information by ORing the hue, saturation, or lightness regardless of the position of the pixel within the detection target range. .

In step S6, the processing unit 20 determines that there is an outlier value if even one image having a value outside the range of color information exists among the pixels in the detection target range excluding the mask target range. . However, the processing unit 20 may determine whether a predetermined number of pixels or more, such as 5 pixels or more, or 30 pixels or more, instead of 1, have values outside the range.

If it is determined that there is a deviating value in step S6 (S6: YES), the processing section 20 detects that there is an article in the tray 121 corresponding to the image acquired in step S1 (step S7). The processing unit 20 may specify the identification data of the tray 121 that has entered the imaging range of the camera 101 at the timing when the image is acquired in step S1. The processing unit 20 notifies the control unit 10 of the detection result (step S8), and ends one detection process.

In step S8, the processing section 20 keeps outputting a signal that can identify the presence or absence of an article to the control section 10 at the signal level, and the control section 10 detects this at the signal level. In step S<b>8 , the processing unit 20 may transmit to the control unit 10 a message indicating that there is an article together with the identification data of the specified tray 121 .

If it is determined in step S6 that there is no deviating value (S6: NO), the processing unit 20 detects that there is no article in the tray 121 corresponding to the image acquired in step S1 (step S9), and executes the process. Proceed to step S8.

In the processing procedure shown in the flowchart of FIG. 4, the processing unit 20 obtains almost all the data of the pixels within the detection range in step S5, and then compares them with the hue, saturation, and lightness data of the color information. bottom. However, the present invention is not limited to this, and the processing unit 20 sequentially acquires the data of the pixels within the detection range on a pixel-by-pixel basis starting from the pixels at a specific location in the detection range. Hue, saturation, and lightness ranges may be compared. All pixels within the detection range when one or more than a predetermined number of pixels starting from a pixel at a specific location are determined to be data outside the range of hue, saturation, and lightness of color information The presence of the article may be detected (S7) without acquiring the .

The processing procedure shown in the flowchart of FIG. 4 will be described with a specific example. FIG. 5 is a diagram showing an example of a captured image of the tray 121. As shown in FIG. In FIG. 5, the captured image is shown diagrammatically. The example of the captured image shown in FIG. 5 is an image of a rectangular tray 121 photographed from above, and the tray 121 is photographed in the shape of a parallelogram. In FIG. 5, four low-brightness regions corresponding to the recesses of the screw holes provided in the tray 121 as shown in FIG. 2 are shown. In the example of FIG. 5, the captured image is captured with the traveling direction of the tray 121 as the horizontal direction. The tray 121 may appear as a parallelogram as shown. This is because the processing unit 20 of the detection device 2 scans the imaging range line by line in the horizontal direction while the tray 121 is running. If the capture speed from the image signal output by the camera 101 or the imaging speed of the camera 101 is faster than the running speed of the tray 121, the tray 121 is imaged in a rectangular shape. Also, the shape in which the tray 121 is captured is not limited to the shape of the captured image in FIG. When images are captured so that the running direction of the tray 121 is the vertical direction, the captured shape of the tray 121 may be distorted depending on the capture rate.

In the example of FIG. 5, the captured image is captured under the condition that the first lighting device 31 is installed so as to project light onto the upper left boundary portion of the imaging range of the camera 101 . Reference A in FIG. 5 indicates the range in which the reflection of the light projected from the first illumination device 31 is photographed. The processing unit 20 determines the range A from the captured image based on the color information of the hue, saturation, and brightness corresponding to the color of the light from the first lighting device 31 . The left end indicated by symbol B in FIG. 5 in range A is the discontinuity of reflection. The processing unit 20 identifies one end corresponding to the travel destination of the tray 121 in the range A as a discontinuous point indicated by symbol B, and determines the coordinates in the image of the discontinuous point as a reference point (step S2). .

FIG. 6 is a diagram illustrating an example of a detection target range and a mask target range. FIG. 6 shows the detection target range and the mask target range determined based on the reference point B with respect to the captured image shown in the diagram of the same example as the example of FIG. It is. In the example of FIG. 6, the solid-line range denoted by symbol C is the detection target range. As described with reference to FIG. 5, the detection target range C is set in the shape of a parallelogram because the tray 121 scans each line in the horizontal direction while it is running. When the tray 121 is captured in a rectangular shape due to the running speed of the tray 121 and the capture speed from the camera 101, the detection target range C is set to be rectangular. The condition may be set according to the state in which the tray 121 is imaged. The detection target range C is represented by data of relative coordinates (indicated by arrows in FIG. 6) of four corners with respect to the reference point B, and is stored in the storage unit 21 . The detection target range C is based on the reference point B, and is based on the relative coordinate data of the upper left corner in the image, the data of the length in the horizontal direction within the image (for example, the number of pixels), and the relative coordinate data of the lower left corner. may be represented.

In the example of FIG. 6, the hatched range denoted by D is the mask target range. As shown in FIG. 6, the mask target range D is represented by data of relative coordinates of mask target pixels in the image with reference point B as a reference. In the case of the example of FIG. 6, the mask target range D includes data of coordinates relative to the reference point B of the center of the circle corresponding to the positions of the four screw holes, and data indicating the size (radius or diameter) of each circle. is represented by

Coordinate data for determining the detection target range C and the mask target range D shown in FIG. The coordinate data of the detection target range C and the mask target range D may be set and stored by a maintenance person or the like based on an image captured by the camera 101 when the sorting machine 100 is installed, or may be stored periodically. May be updated.

As described with reference to FIGS. 5 and 6, the first lighting device 31 casts a different color of focused light than the light from the second lighting device 32 and the third lighting device 33, thereby detecting A target range C is determined. As a result, the detection device 2 can accurately identify the detection target range C even if the area in which the traveling tray 121 is captured in the imaging range of the camera 101 shifts vertically and horizontally.

Next, an example of detection by the detection device 2 according to the present embodiment will be described. FIG. 7 is a diagram showing an example of a captured image of the tray 121 on which articles remain. The diagram of FIG. 7 shows an image of a plastic bag-like package of articles left on the tray 121 . In the diagram of FIG. 7, the image from the camera 101 shows that the light from the first lighting device 31 is distinguishable in the upper left corner of the image of the tray 121 .

8 to 10 are distribution diagrams of extracted hue, saturation and lightness, respectively. In FIGS. 8-10, the monochrome shading corresponding to the hue, saturation, and lightness values extracted from the image of FIG. 7 are respectively shown at positions within the image for explanation. Note that the processing unit 20 does not create these images. The processing unit 20 does not need to determine which position in the image the extracted hue, saturation, or lightness is. In FIGS. 8 and 9, hatching indicates locations in the image where there are pixels having values outside the range of hue, saturation, or lightness values outside the range of pre-stored color information. 8 to 10 also show the reference point B, the detection target range C, and the mask target range D determined for the image.

In FIG. 8, the positions of pixels corresponding to values outside the hue range of the stored color information in the image shown in FIG. 7 are indicated by hatching. As for the hue, a portion with a color different from the color of the tray 121 is extracted within the range of the exterior of the article. The exterior of the article shown in the image shown in FIG. 7 is made of plastic and has a high reflectance, and the portion that reflects a lot of light from the second lighting device 32 and the third lighting device 33 tends to lose color. Most of the package containing this part is not extracted as shown in FIG. Also, a portion having the same color as the tray 121 is not extracted as shown in FIG. However, it is possible to extract a portion that has little reflection and is different from the color tone of the tray 121, and even if the saturation and brightness are the same as those of the tray 121, it is more likely that only a portion can be extracted as shown in FIG. .

FIG. 9 shows hatched positions in the image shown in FIG. 7 that correspond to values outside the saturation range of the stored color information. As for the saturation, a portion different from the saturation of the surface of the tray 121 is extracted from the range of the exterior of the article. Assuming that the exterior of the article shown in the image shown in FIG. is not extracted. However, unlike the tray 121 for industrial use, the package design is likely to select at least part of a color with high saturation, and even a part of it is stored in advance as shown in FIG. If there is a portion different from the saturation of the color information, it can be extracted.

FIG. 10 shows the brightness distribution of the image shown in FIG. 7 (no out-of-range values are extracted). Even if there is no discernable difference in brightness between the exterior of the article and the tray 121, as shown in FIG. , is likely to be extracted.

In this way, when it is determined that any of hue, saturation, and lightness has a value outside the range of color information stored in advance, the presence of an article is detected. is possible.

Another example of detection by the detection device 2 in this embodiment will be described. FIG. 11 is a diagram showing another example of the captured image of the tray 121 on which articles remain. The diagram of FIG. 11 shows an image of a substantially transparent plastic box-like article left on the tray 121 . The diagram of FIG. 11 also shows that the image from the camera 101 shows the light from the first illumination device 31 distinguishably in the upper left corner of the image of the tray 121 .

12 to 14 are distribution diagrams of extracted hue, saturation and lightness. In FIGS. 12-14, the monochrome shades corresponding to the hue, saturation, and lightness values extracted from the image of FIG. 11 are respectively shown at positions within the image for explanation. In other examples, the processing unit 20 does not create these images, but rather the positions in the images of pixels having hue, saturation, or lightness values outside the range of pre-stored color information. need not be determined. In FIGS. 12 and 13, hatching indicates locations in the image where there are pixels having values outside the range of hue, saturation, or lightness values outside the range of pre-stored color information. 12 to 14 show the reference point B, the detection target range C, and the mask target range D determined for the image.

In FIG. 12, the positions of pixels corresponding to values outside the hue range of the stored color information in the image shown in FIG. 11 are indicated by hatching. A portion having a different hue from that of the tray 121 in which the article is photographed is extracted. However, since the article is transparent, as shown in FIG. Very few. The light from the second lighting device 32 that emits red light from above the tray 121 and the light from the third lighting device 33 that emits blue (cyan) light from the side of the tray 121 even though they are transparent. may cast different colored shadows on different parts. A camera 101 that captures this image from above can capture a portion with a different hue from the tray 121 when there is no article, and the processing unit 20 can extract the article based on the difference in hue.

FIG. 13 shows hatched positions of pixels in the image shown in FIG. 11 that correspond to values outside the saturation range of the stored color information. A portion with a color saturation different from that of the tray 121 in the range of the article is extracted, but since the article is transparent, as shown in FIG. 13, it corresponds to a value outside the saturation range of the color information. Few pixels are extracted.

FIG. 14 shows the lightness distribution of the image shown in FIG. 11 (no out-of-range values are extracted). When the article is transparent, as shown in FIG. 14, shadows from different illumination lights projected on the article by the illumination device 103 are of similar brightness, and the tray 121 without the article is illuminated by the illumination device 103. There is no difference from the case, and it is difficult for the processing unit 20 to extract a pixel having a value outside the lightness range.

In this way, the presence of an article is detected when it is determined that any of the hue, saturation and lightness values fall outside the range of pre-stored color information, even if only a few. It was confirmed that the detecting device 2 can detect even a transparent article from the shadow portion produced by the illumination device 103 .

(Modification 1)
The processing unit 20 of the detection device 2 acquires the hue, saturation, and brightness data of the pixels in the detection target range and compares them with the color information. A normalized correlation process may be performed between the image and the acquired image of the tray 121 to be detected.

The tray 121 travels, on which articles are placed, and is moved by a tilting mechanism, so there is a possibility that scratches and stains that cannot be removed individually may adhere and these scratches and stains may appear in the image captured by the camera 101. There is Therefore, pre-registration of an image of an empty tray 121 is periodically performed, and detection from a picked-up image is possible even if a tray 121 with scratches or stains is used according to the procedure described below.

The storage unit 21 of the detection device 2 stores the detection target range (excluding the mask target range) in the captured image of each empty tray 121 in association with the identification data of the tray 121 . The data to be stored may be color information (hue, saturation, brightness) acquired from the detection target range after mask processing. This pre-stored processing is performed by a maintenance person or an administrator of the sorting machine 100 operating the sorting machine 100, the upper control device of the sorting machine 100, or an operation unit provided in the detection device 2. It may be performed in the detection device 2 .

In this way, the detection device 2 executes the following detection processing in a state in which the color information of each tray 121 affected by scratches and stains is stored. FIG. 15 is a flow chart showing an example of the processing procedure of the detection device 2 in Modification 1. As shown in FIG. Among the processing procedures shown in the flowchart of FIG. 15, the procedures that are common to the procedure shown in the flowchart of FIG. 4 are assigned the same step numbers, and detailed description thereof will be omitted.

The processing unit 20 acquires the captured image of the tray 121 (S1), determines the reference point, the detection target range, and the masking target range (S2-S4), and the hue, saturation, and Brightness data is acquired (step S101). The processing unit 20 transfers the acquired data to the comparison area M1 of the built-in memory (step S102).

The processing unit 20 acquires the identification data of the tray 121 shown in the captured image (step S103). The host controller notifies the processing unit 20 of the identification data (tray number) of the tray 121 for which the captured image is to be acquired in step S1, and the processing unit 20 acquires it. The process of step S103 may be executed before the processes of steps S101 and S102. For example, the tray number is No. 20 and so on.

The processing unit 20 selects the hue and color obtained by performing mask processing on an image captured in an empty state in advance for the tray 121 of the acquired identification data from among all the trays (for example, No. 1 to No. 100). The intensity and brightness data are acquired from the storage unit 21 and transferred to the comparison area M2 of the built-in memory (step S104).

The processing unit 20 performs normalized correlation processing on the hue, saturation, and lightness data of the comparison area M1 and the comparison area M2 (step S105). In step S105, the processing unit 20 executes normalized correlation processing only for hue, normalized correlation processing only for saturation, and normalized correlation processing only for brightness.

The processing unit 20 determines whether or not the degree of matching is high in all of the normalized correlation processes for hue, saturation, and lightness (step S106). In step S106, the processing unit 20 determines whether or not the degree of matching is higher than a predetermined value for all three of hue, saturation and lightness.

If it is determined in step S106 that the degree of matching of any one of hue, saturation, and brightness is equal to or less than a predetermined value (S106: NO), processing unit 20 detects that an article is present in tray 121. (S7). If it is determined in step S106 that the degree of matching is high for all three items (S106: YES), the processing section 20 detects that there is no article in the tray 121 (S9).

Processing using normalized correlation processing may be performed only when there is an instruction from the control unit 10 or a higher-level control device. In other words, it may be performed only for trays 121 that are scratched or soiled. For example, when trays 121 with partially different colors are used, as shown in Modified Example 2 to be described later, the same illumination device 103 is used for all trays 121. Therefore, only when trays 121 with different colors are to be detected, A distinction may be made such as performing normalized correlation processing.

In Modified Example 1, an image (detection target range) in which each tray 121 has a scratch or stain is stored, and normalized correlation processing is performed. As another example, all the trays 121 are periodically imaged, and a predetermined range including the average of hue, saturation, and brightness obtained from those images is stored and used as color information each time, and the detector 2 may be referred to in step S6. The range of hue, saturation, and lightness indicated by the color information shifts due to aging of the tray 121 . As a result, the color information reflects the effects of scratches and stains.

(Modification 2)
The storage unit 21 stores color information of the trays 121 of different colors or shapes, data of the detection target range and masking target range, and controls the control unit 10 of the sorting machine 100 or the entire material handling system including the sorting machine 100. The detection process may be executed after an instruction is received from the host controller as to which tray 121 to detect.

FIG. 16 is a flow chart showing an example of the processing procedure of the detection device 2 in Modification 2. As shown in FIG. Among the processing procedures shown in the flowchart of FIG. 16, the procedures that are common to the procedure shown in the flowchart of FIG. 4 are assigned the same step numbers, and detailed description thereof will be omitted.

The processing unit 20 determines whether or not a detection instruction has been received (step S201). If it is determined that the detection instruction has not been received (S201: NO), the processing unit 20 returns the process to step S201 and waits until it determines that the detection instruction has been received.

If it is determined in step S201 that a detection instruction has been received (S201: YES), the processing unit 20 acquires an image (S1), and specifies data on the type of tray 121 to be detected included in the detection instruction (step S202).

The processing unit 20 determines a reference point for the acquired image (S2), and determines a detection target range based on the reference point from the detection target range data stored in association with the type identified in step S202. (step S203). The processing unit 20 determines the masking range based on the reference point from the masking range data stored in association with the type identified in step S202 (step S204). The processing unit 20 acquires data on the hue, saturation, and lightness of the portion of the detection target range excluding the masking target range (S5).

The processing unit 20 reads out the color information stored corresponding to the type specified in step S202 (step S205), and the data acquired in step S5 is the hue, saturation, and brightness included in the read color information. It is determined whether it is out of any of the ranges (S6).

The processing unit 20 detects the presence or absence of the article based on the determination in S6 (S7, S9), and notifies the detection result (S8).

As described above, even if trays 121 of different colors or shapes are mixed, the type of tray 121 can be identified, and detection is possible if color information or the like is stored in advance.

The detection device 2 shown in the embodiment is configured to detect the presence/absence of remaining items for the sorting machine 100 that conveys the items on the trays 121 . However, the detection device 2 is not limited to this, and the detection device 2 may detect the presence or absence of an article by performing image processing on a captured image of the conveyor belt and the conveyor slat.

The embodiments disclosed as described above are illustrative in all respects and are not restrictive. The scope of the present invention is indicated by the scope of claims, and includes all modifications within the meaning and scope of equivalence to the scope of claims.

REFERENCE SIGNS LIST 100 sorting machine 10 control unit 2 detection device 20 processing unit 21 storage unit 2P detection program

Claims (8)

Acquiring captured images of locations where an article to be conveyed is placed and moved and where different colors of light are irradiated from different angles;
projecting the light of the different color and another light onto the location;
Determining a reference point from the discontinuous point of the different light in the acquired captured image,
determining a detection target range including the location based on the reference point in the captured image;
obtaining at least one of hue, saturation, and brightness of pixels included in the detection target range in the captured image;
Any of the hue, saturation, and lightness of pixels in the range of shadows of different colors projected by the light of different colors within the detection target range is out of the ranges for which the hue, saturation, and lightness are stored, respectively. determine whether there is
A detection method for detecting the existence of an article at the location when it is determined that it is out of range. Said another light is light cast in a particular color or in a particular shape
The detection method according to claim 1. 3. A detection method according to claim 1 or 2 , wherein said further light color is complementary to at least one of said different colors. 4. A range of hue, saturation, and lightness in an image captured by irradiating light of different colors from different angles to said location where no article exists is stored in advance. or the detection method according to item 1. The detection method according to any one of claims 1 to 4, wherein the captured image is a captured image of a tray on which articles conveyed by a sorting machine are placed. an image acquisition unit that acquires captured images of a location where an article to be conveyed moves and is illuminated with light of different colors from different angles;
A reference point determination unit that determines a reference point from a discontinuous portion of an area in which the light of the different color and the light of another color are projected onto the portion in the acquired captured image;
a range determination unit that determines a detection target range including the location based on the reference point in the captured image;
a data acquisition unit that acquires at least one of hue, saturation, and brightness of pixels included in the detection target range in the captured image;
Any of the hue, saturation, and lightness of pixels in the range of shadows of different colors projected by the light of different colors within the detection target range is out of the ranges for which the hue, saturation, and lightness are stored, respectively. a judgment unit for judging whether or not there is, and
A detection device comprising: a detection unit that detects that an article exists at the location when it is determined that the location is out of range. a second lighting device and a third lighting device , which are installed at a portion of a transport path of a sorting machine that sorts input articles by destination while transporting them, and illuminates the portion with light of different colors from different angles;
a first lighting device that projects light of the different color and another light onto the location;
an imaging device that captures an image of a portion illuminated by the first lighting device, the second lighting device, and the third lighting device;
a detection device that acquires an image captured by the imaging device and detects whether or not an article exists at the location;
The detection device is
determining a reference point from the discontinuity of the different light in the captured image;
determining a detection target range including the location based on the reference point in the captured image;
obtaining at least one of hue, saturation, and brightness of pixels included in the detection target range in the captured image;
Any of the hue, saturation, and lightness of pixels in the range of shadows of different colors projected by the light of different colors within the detection target range is out of the ranges for which the hue, saturation, and lightness are stored, respectively. determine whether there is
A detection system for detecting the presence of an article at said location if it determines that it is out of range. A computer connected to a sorting machine that sorts by destination while transporting incoming items,
Acquiring captured images of locations on the transport path of the articles in the sorting machine that are irradiated with light of different colors from different angles,
Determining a reference point from a discontinuous portion of an area in which the light of the different color and the other light are projected onto the portion in the acquired captured image;
determining a detection target range including the location based on the reference point in the captured image;
obtaining at least one of hue, saturation, and brightness of pixels included in the detection target range in the captured image;
Any of the hue, saturation, and lightness of pixels in the range of shadows of different colors projected by the light of different colors within the detection target range is out of the ranges for which the hue, saturation, and lightness are stored, respectively. determine whether there is
A computer program for executing a process of detecting the presence of an article at the location when it is determined that it is out of range.

Images