WO2024048346A1

WO2024048346A1 - Object recognition system, object recognition device, and object recognition method

Info

Publication number: WO2024048346A1
Application number: PCT/JP2023/029980
Authority: WO
Inventors: 勇生熊谷
Original assignee: ソニーセミコンダクタソリューションズ株式会社
Priority date: 2022-08-30
Filing date: 2023-08-21
Publication date: 2024-03-07

Abstract

An object recognition system according to one embodiment of the present disclosure comprises: a plurality of light sources that irradiate an object with light including infrared rays; a control unit that turns on the plurality of light sources at mutually different timings; and an imaging unit that acquires a distance image and an infrared image of the object at each timing. The control unit superimposes the distance images from each timing to generate a composite distance image and superimposes the infrared images from each timing to generate a composite infrared image, and recognizes the object on the basis of the composite distance image and the composite infrared image.

Description

Object recognition system, object recognition device, and object recognition method

The present disclosure relates to an object recognition system, an object recognition device, and an object recognition method.

For example, in a distribution warehouse or manufacturing factory, a box, which is an example of an object, is transported by a belt conveyor, or is lifted from a belt conveyor and transported by a robot such as a robot arm. At this time, a plurality of boxes may be transported side by side, or a plurality of boxes may be loaded on a pallet and transported together with the pallet.

The transport system that realizes this type of transport needs to recognize the boxes and control the transport route switching device, robot, etc. in order to switch the transport route of the box depending on the size and lift it using a robot. Therefore, even when multiple boxes are transported side by side, the size and shape of the boxes can be recognized by detecting the gaps (grooves) between the boxes and the contours (edges) of the boxes. .

Specifically, in order to recognize the size and shape of the boxes, the outline of a group of boxes is detected from the difference in color between the conveyor belt (background) and the boxes (foreground), and the dark straight parts are identified as the breaks (grooves) between the boxes. A technique has been proposed that detects an outline and recognizes a portion surrounded by the outline as a box (for example, see Patent Document 1). Additionally, a technology has been proposed that uses a pair of parallel edges (contours) as clues to generate an object edge set by combining pairs of intersecting edges, and then recognizes the area surrounded by the object edge set as a single box. (For example, see Patent Document 2).

JP 2021-15616 Publication JP 2021-22383 Publication

However, with the above-mentioned technology, features such as black tape or stickers affixed to the box may be detected as outlines (edges), so for example, one box may be recognized as two boxes. It is difficult to recognize objects with high accuracy. In addition, the light reflected from characteristic parts such as glossy black tape or stickers attached to boxes is strong, and these areas become saturated areas in the image, making it difficult to detect characteristic parts and accurately detecting objects such as boxes. It is difficult to recognize well.

Therefore, the present disclosure provides an object recognition system, an object recognition device, and an object recognition method that make it possible to improve object recognition accuracy.

An object recognition system according to an embodiment of the present disclosure includes a plurality of light sources that irradiate an object with light including infrared rays, a control unit that turns on the plurality of light sources at different timings, and a control unit that lights up the plurality of light sources at different timings. an imaging unit that acquires a range image and an infrared image, the control unit superimposing the range images for each of the timings to generate a composite range image, and superimposing the infrared images for each of the timings to generate a composite infrared image. and recognize the object based on the composite distance image and the composite infrared image.

An object recognition device according to an embodiment of the present disclosure includes a plurality of light sources that irradiate an object with light including infrared rays, a control unit that turns on the plurality of light sources at different timings, and a control unit that lights up the plurality of light sources at different timings. an imaging unit that acquires a range image and an infrared image, the control unit superimposing the range images for each of the timings to generate a composite range image, and superimposing the infrared images for each of the timings to generate a composite infrared image. and recognize the object based on the composite distance image and the composite infrared image.

An object recognition method according to an embodiment of the present disclosure includes an object recognition system that turns on a plurality of light sources that irradiate an object with light including infrared rays at different timings, and a distance image of the object at each of the timings. and obtaining an infrared image, superimposing the distance images for each timing to generate a composite distance image, superimposing the infrared images for each timing to generate a composite infrared image, and generating the composite distance image and the composite distance image. and recognizing the object based on an infrared image.

1 is a diagram illustrating a configuration example of an object recognition system according to an embodiment of the present disclosure. FIG. 3 is a diagram for explaining a first arrangement example of a plurality of light sources according to an embodiment of the present disclosure. FIG. 7 is a diagram for explaining a second arrangement example of a plurality of light sources according to an embodiment of the present disclosure. FIG. 7 is a diagram for explaining a third arrangement example of a plurality of light sources according to an embodiment of the present disclosure. FIG. 3 is a diagram for explaining generation of a composite image according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating a first configuration example regarding light emission control according to an embodiment of the present disclosure. FIG. 3 is a diagram illustrating a first timing chart example regarding light emission control according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating a second configuration example regarding light emission control according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating a third configuration example regarding light emission control according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating a second timing chart example regarding light emission control according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating a fourth configuration example regarding light emission control according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating a third timing chart example regarding light emission control according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating a fourth timing chart example regarding light emission control according to an embodiment of the present disclosure. FIG. 2 is a diagram showing an example of a schematic configuration of hardware.

Below, embodiments of the present disclosure will be described in detail based on the drawings. Note that this embodiment does not limit the system, device, method, etc. according to the present disclosure. Moreover, in each of the following embodiments, basically the same portions are given the same reference numerals and redundant explanations will be omitted.

One or more embodiments (including examples and modifications) described below can each be implemented independently. On the other hand, at least a portion of the plurality of embodiments described below may be implemented in combination with at least a portion of other embodiments as appropriate. These multiple embodiments may include novel features that are different from each other. Therefore, these multiple embodiments may contribute to solving mutually different objectives or problems, and may produce mutually different effects.

The present disclosure will be described according to the order of items shown below.
1. Embodiment 1-1. Configuration example of object recognition system 1-2. Example of arrangement of multiple light sources 1-2-1. First arrangement example 1-2-2. Second arrangement example 1-2-3. Third arrangement example 1-3. Example of generation of composite image 1-4. Configuration example and processing example related to light emission control 1-4-1. First configuration example and first processing example 1-4-2. Second configuration example 1-4-3. Third configuration example and second processing example 1-4-4. Fourth configuration example 1-4-5. Third processing example 1-4-6. Fourth processing example 1-5. Action/Effect 2. Hardware configuration example 3. Other embodiments 4. Additional notes

<1. Embodiment>
<1-1. Configuration example of object recognition system>
A configuration example of an object recognition system 1 according to this embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing a configuration example of an object recognition system 1 according to the present embodiment.

As shown in FIG. 1, the object recognition system 1 according to the present embodiment includes a transport section 10, an imaging section 20, a plurality of light sources 30, a robot 40, and a control section 50.

The transport unit 10 transports an object A1 such as a box. This conveyance section 10 is realized by, for example, a belt conveyor. The transport unit 10 is controlled by the control unit 50, for example, but may be controlled by another system such as a transport system. For example, boxes may be transported individually or side by side.

The imaging unit 20 images the object A1 on the transport unit 10. This imaging unit 20 is realized, for example, by an image sensor (distance image sensor) that photographs the object A1. As the image sensor, an active type image sensor such as an iToF (indirect time of flight) type, dToF (direct time of flight) type, or structured light type image sensor is used. Note that the imaging unit 20 is realized by an image sensor capable of acquiring a distance image (depth image having depth information for each pixel) and an infrared image (IR image); It may be realized by two sensors, including an image sensor capable of acquiring an image.

Each light source 30 irradiates light containing infrared rays (for example, laser light) toward the object A1 on the transport unit 10. These light sources 30 are provided around the imaging unit 20 so as not to prevent the imaging unit 20 from imaging the object A1 on the transport unit 10. For example, each light source 30 is arranged radially around the imaging unit 20. The number of light sources 30 is two or more. Among these light sources 30, by changing the light source 30 that emits light, it is possible to arbitrarily change the light emitting position and adjust the angle of the light beam irradiated to the object A1, and also the amount of light can be adjusted. It is possible. The arrangement of each light source 30 will be described in detail later.

The robot 40 is a device that lifts the object A1 from the transport section 10 and transports it to another location. This robot 40 is realized by, for example, a robot arm. For example, the robot 40 is controlled by the control unit 50, but may be controlled by another system such as a transport system.

The control unit 50 is a control device that controls each unit such as the transport unit 10, the imaging unit 20, and each light source 30. The control section 50 includes a light source driver 51, an image processing section 52, and the like. The light source driver 51 is provided commonly to each light source 30 and controls the driving (turning on and off) of each light source 30. The image processing unit 52 performs various image processing. Examples of this image processing include object recognition processing and composite image generation processing. These object recognition processes and composite image generation processes will be described in detail later.

The control unit 50 is realized by, for example, a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). For example, the control unit 50 executes various programs using a RAM (Random Access Memory) or the like as a work area, but it may also be realized by an integrated circuit such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). . CPUs, MPUs, ASICs, and FPGAs can all be considered processors. Further, the control unit 50 may be realized by a GPU (Graphics Processing Unit) in addition to or instead of the CPU. Further, the control unit 50 may be realized by specific software instead of specific hardware.

Further, the image processing unit 52 executes, for example, an application that performs object recognition or image generation. Examples of applications include various applications such as general-purpose applications and dedicated applications. Further, the application may be realized by, for example, AI (artificial intelligence). For example, the application may be executed based on a model trained by a neural network (for example, CNN: convolutional neural network), which is an example of machine learning, or may be executed based on other techniques.

Here, in the object recognition process, the features of the object A1 are recognized by image processing. The features of the object A1 include, for example, a contour and a non-contour portion (non-contour region). The contour also includes, for example, a groove line A1b, which is a gap between the boxes. The non-contour portion includes, for example, black tape A1a. This black tape A1a has gloss and is attached to the surface of the object A1. Note that, in addition to the black tape A1a, there are glossy stickers and the like as members pasted on the surface of the object A1. Features such as the outline of the object A1 (including the groove line A1b) and the black tape A1a are detected, and the object A1 is recognized. Recognizing the object A1 means, for example, understanding the size and shape of the object A1.

Note that the control unit 50 may be connected to each unit such as the transport unit 10, the imaging unit 20, and each light source 30 via a network. The network is, for example, a communication network (communication network) such as a LAN (Local Area Network), a WAN (Wide Area Network), a cellular network, a fixed telephone network, a local IP (Internet Protocol) network, or the Internet. The network may include a wired network or a wireless network. Further, the network may include a core network. The core network is, for example, EPC (Evolved Packet Core) or 5GC (5G Core network). Further, the network may include a data network other than the core network. For example, the data network may be a carrier's service network, for example an IMS (IP Multimedia Subsystem) network. Further, the data network may be a private network such as an in-house network.

Further, as the radio access technology (RAT), LTE (Long Term Evolution), NR (New Radio), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc. can be used. These several types of radio access technologies may be used, for example, NR and Wi-Fi may be used, or LTE and NR may be used. LTE and NR are types of cellular communication technologies that enable mobile communication by arranging multiple areas covered by base stations in the form of cells.

<1-2. Example of arrangement of multiple light sources>
An example of the arrangement of the plurality of light sources 30 according to this embodiment will be described with reference to FIGS. 2 to 4. 2 to 4 are diagrams for explaining arrangement examples (first to third arrangement examples) of the light sources 30 according to this embodiment. In the examples of FIGS. 2 to 4, one area of the transport section 10, the imaging section 20, and each light source 30 are shown in plan view.

<1-2-1. First arrangement example>
As shown in FIG. 2, the number of light sources 30 according to the first arrangement example is two. The two light sources 30 are provided so as to face each other with the imaging unit 20 in between in a plan view, for example, to face each other with the imaging unit 20 at the center in a plan view. That is, the two light sources 30 are provided point-symmetrically with respect to the imaging unit 20. Note that the two light sources 30 may be arranged in a direction parallel to the transport direction of the object A1 in plan view, or may be arranged in a direction perpendicular to the transport direction of the object A1.

<1-2-2. Second arrangement example>
As shown in FIG. 3, the number of light sources 30 according to the second arrangement example is four. The four light sources 30 are provided radially so as to surround the imaging section 20 in a plan view, for example, to surround the imaging section 20 in a plan view. These light sources 30 are arranged, for example, on the same circle centered on the imaging unit 20 at equal intervals. Note that the four light sources 30 are each provided so that the two light sources 30 that form a pair in plan view are point symmetrical with respect to the imaging unit 20 . In order to achieve a uniform light amount distribution, it is desirable to use a symmetrical arrangement with the imaging unit 20 at the center in plan view.

<1-2-3. Third arrangement example>
As shown in FIG. 4, the number of light sources 30 according to the third arrangement example is eight. Similarly to the second arrangement example, the eight light sources 30 are provided radially so as to surround the imaging unit 20 in a plan view, for example, to surround the imaging unit 20 in a plan view. These light sources 30 are arranged, for example, on the same circle centered on the imaging unit 20 at equal intervals. Note that the eight light sources 30 are provided so that the two light sources 30 that form a pair in plan view are point symmetrical with respect to the imaging unit 20 . In order to achieve a uniform light amount distribution, it is desirable to use a symmetrical arrangement with the imaging unit 20 at the center in plan view. Moreover, it is desirable that the number of light sources 30 to be arranged is large. This is because it is possible to increase the number of light emitting patterns, and it becomes easier to set conditions for suppressing saturation depending on reflected light intensity.

Note that the first arrangement example, second arrangement example, third arrangement example, and the number of light sources 30 as described above are merely examples, and other arrangements and numbers may be adopted. The number of light sources 30 may be an even number or an odd number as long as it is two or more. Furthermore, although the light sources 30 are arranged on the same circle at equal intervals, they do not have to be on the same circle or at equal intervals. However, in order to realize a uniform light amount distribution, it is desirable to adopt the first arrangement example, the second arrangement example, the third arrangement example, or the like.

<1-3. Example of generating a composite image>
An example of generating a composite image (image generation processing) according to this embodiment will be described with reference to FIG. 5. FIG. 5 is a diagram for explaining an example of generating a composite image according to this embodiment. In the example of FIG. 5, the number of light sources 30 is two (see the above-mentioned first arrangement example: FIG. 2).

As shown in FIG. 5, the light emitting pattern A is a pattern in which one of the two light sources 30 emits light, and the light emitting pattern B is a pattern in which the other of the two light sources 30 emits light. In both the light emission patterns A and B, the light reflected by the glossy black tape A1a becomes stronger.

According to the light emission pattern A, one of the light sources 30 is turned on for a certain period of time (for example, several seconds), and both the distance image and the infrared image G1 are acquired by the imaging unit 20 (step S1). The black tape A1a in both images G1 includes a saturated region R1 (for example, blown out highlights) that depends on the intensity of reflected light.

According to the light emission pattern B, the other light source 30 is turned on for a certain period of time (for example, several seconds), and both the distance image and the infrared image G2 are acquired by the imaging unit 20 (step S2). The black tape A1a in both images G2 includes a saturated region R1 (for example, blown-out highlights) that depends on the intensity of reflected light.

Note that the position of the saturated region R1 according to the light emission pattern B is different from the position of the saturated region R1 according to the light emission pattern A. This is because the light emitting pattern A and the light emitting pattern B have different light sources 30 (light emitting positions) that emit light, and have different angles of light rays irradiated onto the object A1. That is, by changing the light emitting position, it is possible to prevent the saturated region R1 from being at the same position in different images.

Furthermore, in the light emission pattern A and the light emission pattern B, the light emission time (lighting time) during which the light source 30 emits light may be the same or different. Depending on the installation environment of the light source 30, the light emission time may be intentionally made different. Further, although the individual lighting timings of each light source 30 are different, the individual lighting timings may be different or the same. However, considering the ease of adjusting the beam angle and light amount, it is desirable to vary the timing of extinguishing the lights.

Both images G1 based on the light emission pattern A and both images G2 based on the light emission pattern B are superimposed on each other as distance images and infrared images, and a composite image G3, which is a composite image of both, is generated by the image processing unit 52 (step S3 ). That is, a distance image based on emission pattern A and a distance image based on emission pattern B are superimposed to generate a composite distance image, and an infrared image based on emission pattern A and an infrared image based on emission pattern B are superimposed to generate a composite infrared image. .

The composite range image and composite infrared image are then used for object recognition. The black tape A1a (black tape area) included in both the composite distance image and the composite infrared image does not include a saturated area R1 (for example, blown-out highlights) that depends on the reflected light intensity. That is, the image processing unit 52 superimposes the distance image based on the emission pattern A and the distance image based on the emission pattern B so as to remove the saturated region R1 in the distance image and the infrared image, and the infrared image based on the emission pattern A and the emission pattern By superimposing the infrared image obtained by B, a composite distance image and a composite infrared image are generated.

As a result, the saturated region R1 is eliminated from the composite distance image and the composite infrared image, so features of the object A1 such as a box, such as the black tape A1a and the groove line A1b, can be detected with high accuracy. For example, it is possible to accurately identify the black tape A1a and the groove line A1b from the distance information (depth information) included in the composite distance image, so it is possible to accurately recognize the object A1 such as a box. . This makes it possible to prevent erroneously detecting the black tape A1a as the groove line A1b and recognizing one box as two boxes. In addition, it is possible to accurately detect the outline of object A1 other than groove line A1b through image processing from the distance information included in the composite distance image and the image information included in the composite infrared image, so it is possible to accurately detect the outline of object A1 other than the groove line A1b. A1 can be recognized with high accuracy. Furthermore, since there is no saturated region R1 in the composite distance image and the composite infrared image, it becomes easy to detect characteristic parts, and the object A1 can be recognized with high accuracy.

Note that when recognizing the object A1 only from an RGB image (color image), it is difficult to detect the groove line A1b as one continuous line through image processing, and it is difficult to accurately identify the black tape A1a and the groove line A1b. difficult to do. Further, even when recognizing the object A1 only from an infrared image, it is difficult to accurately distinguish between the black tape A1a and the groove line A1b. Therefore, according to the present embodiment, an infrared image and a distance image are used, and the saturated region R1 is removed from the infrared image and the distance image by composite image processing, and the object A1 is removed from the composite infrared image and the composite distance image. is recognized. As a result, as described above, it is possible to accurately identify the black tape A1a and the groove line A1b, and furthermore, it is possible to detect the groove line A1b as one continuous line, so object recognition is possible. Accuracy can be improved.

In the above description, the images obtained based on the two light emitting patterns A and B have been described, but the number of images obtained by increasing the number of light emitting patterns may be increased. By increasing the number of images, it becomes possible to remove the saturated region R1 more reliably, so that the features of the object A1 such as a box, for example, the black tape A1a, can be recognized with higher accuracy.

<1-4. Configuration example and processing example regarding light emission control>
A configuration example and a processing example regarding light emission control (lighting and extinguishing) according to this embodiment will be described with reference to FIGS. 6 to 13. 6, FIG. 8, FIG. 9, and FIG. 11 are diagrams showing configuration examples (first to fourth configuration examples) regarding light emission control according to the present embodiment. 7, FIG. 10, FIG. 12, and FIG. 13 are diagrams showing timing chart examples (first to fourth timing chart examples) regarding light emission control according to the present embodiment.

<1-4-1. First configuration example and first processing example>
As shown in FIG. 6, in the first configuration example, the number of light sources 30 is four (see the aforementioned second arrangement example: FIG. 3), and the number of light source drivers 51 is one. In the example of FIG. 6, light source A, light source B, light source C, and light source D are shown as four light sources 30. The light source driver 51 is provided in common to the four light sources 30. The light source driver 51 turns on at least one of the light sources 30 for a certain period of time based on the synchronization timing issued from the imaging section 20.

As shown in FIG. 7, in the first processing example, light emission pulses (drive signals) are input to light source A, light source B, light source C, and light source D (four light sources 30) at different timings. In the example of FIG. 7, the light emission pulse is input to each of the light sources A, B, C, and D in this order in synchronization with the read pulse from the imaging unit 20. When this light emitting pulse becomes a high level (amplitude 1), each light source A, B, C, and D becomes an ON state (light emitting state = lighting state), and when it becomes a low level (amplitude 0), each light source A, B, C , D are in the OFF state (light out state).

The imaging unit 20 issues read pulses, that is, synchronization timing (read timing) at predetermined intervals. The light source driver 51 controls the light emission timing (lighting timing) and extinguishing timing of each of the light sources A, B, C, and D based on the synchronization timing issued from the imaging unit 20. For example, the light source driver 51 repeatedly turns on and off (lights light for a certain period of time) light source A, light source B, light source C, and light source D in the order based on the synchronization timing issued from the imaging unit 20.

According to such a first configuration example, a simple configuration regarding light emission control can be realized. In other words, since it is possible to realize light emission control with a simple configuration, object recognition accuracy can be improved with a simple configuration.

<1-4-2. Second configuration example>
As shown in FIG. 8, in the second configuration example, the number of light sources 30 is four (see the aforementioned second arrangement example: FIG. 3), and the number of light source drivers 51 is four. In the example of FIG. 8, light source A, light source B, light source C, and light source D are shown as the four light sources 30, as in the other figures. A light source driver 51 is provided for each light source 30. Each of these light source drivers 51 turns on and off the corresponding light source 30 based on the synchronization timing issued from the imaging unit 20. Note that the imaging unit 20 controls the synchronization timing (readout timing) issued to each light source driver 51, respectively.

According to such a second configuration example, by providing the light source driver 51 for each light source 30, fine control over each light source 30 is made possible. In other words, since the light source driver 51 exists for each light source 30, detailed control of each light source 30 is possible, so that object recognition accuracy can be reliably improved.

<1-4-3. Third configuration example and second processing example>
As shown in FIG. 9, in the third configuration example, the number of light sources 30 is four (see the aforementioned second arrangement example: FIG. 3), and the number of light source drivers 51 is four. In the example of FIG. 9, light source A, light source B, light source C, and light source D are shown as the four light sources 30, as in the other figures. A light source driver 51 is provided for each light source 30. Additionally, a timing distribution section 53 is present. This timing distribution section 53 is realized by, for example, a timing distribution circuit, and is included in the control section 50. The timing distribution unit 53 issues light emission timing (lighting timing) and light extinguishing timing to each light source driver 51 based on the synchronization timing issued from the imaging unit 20. Each of these light source drivers 51 turns on and off the corresponding light source 30 (light emission for a certain period of time) based on the light emission timing and light extinction timing issued from the timing distribution unit 53. Note that the imaging unit 20 controls the synchronization timing (read timing) issued to the timing distribution unit 53.

As shown in FIG. 10, in the second processing example, light emission pulses (drive signals) are input to light source A, light source B, light source C, and light source D (four light sources 30) at different timings. In the example of FIG. 10 as well, similarly to FIG. 7, the light emission pulse is input to each of the light sources A, B, C, and D in this order in synchronization with the readout pulse from the imaging unit 20. When this light emission pulse becomes a high level (amplitude 1), each light source A, B, C, and D becomes an ON state, and when it becomes a low level (amplitude 0), each light source A, B, C, and D becomes an OFF state. .

The imaging unit 20 issues read pulses, that is, synchronization timing (read timing) at predetermined intervals. The timing distribution unit 53 issues light emission timing (lighting timing) and light extinguishing timing to each light source driver 51 based on the synchronization timing issued from the imaging unit 20. The light source driver 51 controls the individual light emission timing and light extinction timing of each of the light sources A, B, C, and D based on the light emission timing and light extinction timing issued from the timing distributor 53. For example, the light source driver 51 repeats turning on and off the light source A, the light source B, the light source C, and the light source D in the order based on the light emission timing and the light extinction timing issued from the timing distribution unit 53 (light emission for a certain period of time).

According to such a third configuration example, even if the imaging unit 20 is of a type in which there is only one output terminal for outputting synchronization timing, each light source 30 can emit light at different timings. Recognition accuracy can be improved.

<1-4-4. Fourth configuration example>
As shown in FIG. 11, in the fourth configuration example, the number of light sources 30 is four (see the aforementioned second arrangement example: FIG. 3), and the number of light source drivers 51 is four. In the example of FIG. 11, light source A, light source B, light source C, and light source D are shown as four light sources 30, as in the other figures. A light source driver 51 is provided for each light source 30. Additionally, a synchronization control section 54 is present. The synchronization control section 54 is realized by, for example, a synchronization control circuit (for example, an FPGA, a microcomputer, etc.), and is included in the control section 50. The synchronization control unit 54 issues a read timing to the imaging unit 20, and also issues light emission timing (lighting timing) and light extinguishing timing to each light source driver 51 in synchronization with the read timing. Each of the light source drivers 51 turns on and off the corresponding light source 30 (light emission for a certain period of time) based on the light emission timing and light extinction timing issued from the synchronous control unit 54, and also causes the corresponding light source 30 to turn on and off (light emission for a certain period of time). The image is read out based on the readout timing.

According to such a fourth configuration example, even if the imaging unit 20 is of a type that does not have an output terminal that outputs synchronization timing, each light source 30 can emit light at different timings while being synchronized with the readout timing. Therefore, object recognition accuracy can be improved. Furthermore, it becomes possible to use various types of imaging sections 20, and it becomes easy to replace the imaging section 20.

<1-4-5. Third processing example>
As shown in FIG. 12, the timing chart according to the third processing example is a timing chart when the imaging unit 20 is iToF. The timing chart according to the third processing example is the same as the timing chart according to the first processing example (see FIG. 7), but the actual light emission pulse and the readout pulse of the imaging unit 20 are short-cycle pulses. When the imaging unit 20 is iToF, the light emission is in the form of short pulses, and a plurality of variations (for example, four phases) in the phase relationship between light emission and imaging are required. For example, in the light emission pulse and the readout pulse, four phases (0deg, 90deg, 180deg, 270deg) are read out by multiple taps (a tap, b tap) of the imaging unit 20. Note that although the example in FIG. 12 shows a type in which 4 phases are read out with each tap (a tap, b tap), similar readout is performed in other variations (for example, 2 phases, 8 phases, etc.). .

According to such a third processing example, it is possible to use an iToF image sensor as the imaging unit 20. This makes it possible to obtain accurate distance images, thereby reliably improving object recognition accuracy.

<1-4-6. Fourth processing example>
As shown in FIG. 13, in the fourth processing example, light source A and light source B emit light at the same time, and light source C and light source D emit light at the same time. For example, the light source driver 51 turns on and off at least two of the light sources A, B, C, and D simultaneously based on the synchronization timing issued from the imaging unit 20 (light emission for a certain period of time).

According to such a fourth processing example, the plurality of light sources 30 emit light at the same time, so that the time required for all frames to be aligned can be shortened. Furthermore, since it becomes possible to increase the amount of light, distance measurement performance can be improved.

In the example of FIG. 13, the light source A and the light source B are turned on (light-emitting light) at the same time, and the light source C and the light source D are turned on (light-emitting light) at the same time, but the light source A and the light source Two light sources C may be turned on at the same time, two light sources A and D may be turned on at the same time, or three light sources A, B, and C may be turned on at the same time.

Further, in the example of FIG. 13, the number of light sources 30 is four, but the number is not particularly limited as long as it is two or more. Moreover, the number of light sources 30 to be turned on at the same time is not particularly limited, as long as it is two or more. Furthermore, various combinations of the number of light sources 30 to be turned on simultaneously are possible, for example, first the number of light sources 30 to be turned on at the same time is two, and then the number of light sources 30 to be turned on at the same time is three. be.

<1-5. Action/Effect>
As described above, according to the present embodiment, the object recognition system 1 includes a plurality of light sources 30 that irradiate the object A1 with light including infrared rays, and a control unit that turns on the plurality of light sources 30 at different timings. 50, and an imaging unit 20 that acquires a distance image and an infrared image of the object A1 at each timing, and the control unit 50 acquires a range image at each timing (for example, one of both images G1 and one of both images G2). are superimposed to generate a composite distance image (for example, one of both composite images G3), and infrared images for each timing (for example, the other of both images G1 and the other of both images G2) are superimposed to generate a composite infrared image (for example, , the other of both composite images G3), and recognizes the object A1 based on the composite distance image and the composite infrared image. As a result, the saturated region R1 is eliminated from the composite distance image and the composite infrared image, and the object A1 can be recognized with high accuracy, so that the object recognition accuracy can be improved.

The control unit 50 also superimposes the distance images for each timing and superimposes the infrared images for each timing so as to remove the saturated region R1 that depends on the intensity of reflected light in the distance images and infrared images for each timing. Then, a composite distance image and a composite infrared image may be generated. This makes it possible to reliably eliminate the saturated region R1 from the composite distance image and the composite infrared image, thereby reliably improving object recognition accuracy.

Furthermore, the control unit 50 may turn off each light source 30 at different timings. Thereby, by controlling both the timing of turning on and turning off the lights, it is possible to reliably eliminate the saturated region R1 from the composite distance image and the composite infrared image, so that object recognition accuracy can be reliably improved.

Furthermore, each light source 30 may be provided with the imaging unit 20 at the center in plan view (see FIGS. 2 to 4). This makes it possible to obtain distance images and infrared images with different beam angles of irradiation light, thereby reliably improving object recognition accuracy.

Further, each light source 30 may be provided so as to face each other with the imaging unit 20 in between in plan view (see FIGS. 2 to 4). This makes it possible to obtain distance images and infrared images with different beam angles of irradiation light, thereby reliably improving object recognition accuracy.

Further, each light source 30 may be provided so as to surround the imaging unit 20 in plan view (see FIGS. 3 and 4). This makes it possible to obtain distance images and infrared images with different beam angles of irradiation light, thereby reliably improving object recognition accuracy.

Further, each light source 30 may be provided point-symmetrically with respect to the imaging unit 20 in plan view (see FIGS. 2 to 4). This makes it possible to obtain distance images and infrared images with different ray angles of irradiation light, and also to realize a uniform light amount distribution, so that object recognition accuracy can be reliably improved.

Further, the control unit 50 has a common light source driver 51 for each light source 30, and the light source driver 51 may turn on at least one of the light sources 30 based on the synchronization timing issued from the imaging unit 20. Good (see Figure 6). This makes it possible to implement control with a simple configuration, so that object recognition accuracy can be improved with a simple configuration.

Further, the control unit 50 may include a light source driver 51 for each light source 30, and each of the light source drivers 51 for each light source 30 may turn on the light source 30 based on the synchronization timing issued from the imaging unit 20 ( (See Figure 8). Accordingly, since the light source driver 51 is provided for each light source 30, detailed control of each light source 30 is possible, so that object recognition accuracy can be reliably improved.

Further, the control unit 50 includes a timing distribution unit 53, and the timing distribution unit 53 issues light emission timing (lighting timing) to the light source driver 51 of each light source 30 based on the synchronization timing issued from the imaging unit 20. , each of the light source drivers 51 for each light source 30 may turn on the light source 30 based on the light emission timing issued from the timing distribution unit 53 (see FIG. 9). Thereby, even if the imaging unit 20 is of a type in which there is only one output terminal for outputting synchronization timing, object recognition accuracy can be improved.

Further, the control unit 50 has a light source driver 51 and a synchronization control unit 54 for each light source 30, and the synchronization control unit 54 issues a readout timing to the imaging unit 20, and synchronizes the readout timing with the readout timing for each light source 30. A light emission timing (lighting timing) is issued to the light source driver 51, each of the light source drivers 51 for each light source 30 lights up the light source 30 based on the light emission timing issued from the synchronization control section 54, and the imaging section 20 performs synchronization. The distance image and the infrared image may be acquired based on the readout timing issued by the control unit 54 (see FIG. 11). Thereby, even if the imaging unit 20 is of a type that does not have an output terminal for outputting synchronization timing, object recognition accuracy can be improved. Furthermore, it becomes possible to use various types of imaging sections 20.

Further, the imaging unit 20 may be an iToF (indirect time of flight) image sensor (see FIG. 12). This makes it possible to obtain accurate distance images, thereby reliably improving object recognition accuracy.

Furthermore, the control unit 50 may turn on some of the light sources 30 at the same timing (see FIG. 13). This makes it possible to adjust the amount of light, thereby reliably improving object recognition accuracy.

Furthermore, the object A1 may be a box, and the features of the box may include a contour and a non-contour part. Object recognition accuracy can be improved for such object A1.

Furthermore, the contour may include the groove line A1b, and the non-contour portion may include the black tape A1a. Object recognition accuracy can be improved for such object A1.

Additionally, the non-contour portion may include a sticker in addition to the black tape A1a. Object recognition accuracy can be improved for such object A1.

<2. Hardware configuration example>
A specific hardware configuration example of information equipment such as a part of the object recognition system 1 and the control unit 50 according to the above-described embodiment (or modification) will be described. A part of the object recognition system 1 and information devices such as the control unit 50 according to the embodiment (or modification) may be realized by, for example, a computer 500 having a configuration as shown in FIG. 14. FIG. 14 is a diagram illustrating an example of a schematic configuration of hardware that implements the functions of an information device.

As shown in FIG. 14, the computer 500 includes a CPU 510, a RAM 520, a ROM (Read Only Memory) 530, an HDD (Hard Disk Drive) 540, a communication interface 550, and an input/output interface 560. Each part of computer 500 is connected by a bus 570.

The CPU 510 operates based on a program stored in the ROM 530 or HDD 540 and controls each part. For example, the CPU 510 loads programs stored in the ROM 530 or HDD 540 into the RAM 520, and executes processes corresponding to various programs.

The ROM 530 stores boot programs such as BIOS (Basic Input Output System) that are executed by the CPU 510 when the computer 500 is started, programs that depend on the hardware of the computer 500, and the like.

The HDD 540 is a recording medium readable by the computer 500 that non-temporarily records programs executed by the CPU 510 and data used by the programs. Specifically, the HDD 540 is a recording medium that records program data 541.

The communication interface 550 is an interface for connecting the computer 500 to an external network 580 (the Internet as an example). For example, the CPU 510 receives data from other devices or transmits data generated by the CPU 510 to other devices via the communication interface 550.

The input/output interface 560 is an interface for connecting the input/output device 590 and the computer 500. For example, CPU 510 receives data from an input device such as a keyboard or mouse via input/output interface 560. Further, the CPU 510 transmits data to an output device such as a display, speaker, or printer via the input/output interface 560.

Note that the input/output interface 560 may function as a media interface that reads a program recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVD (Digital Versatile Disc) and PD (Phase change rewritable disk), magneto-optical recording media such as MO (Magneto-Optical disk), tape media, magnetic recording media, or semiconductors. Memory etc. are used.

Here, for example, when the computer 500 functions as a part of the object recognition system 1 or an information device such as the control unit 50 according to each embodiment (or modification), the CPU 510 of the computer 500 functions as a By executing the information processing program, all or part of the functions of each section according to each embodiment (or modification) are realized. Further, the HDD 540 stores information processing programs and data according to each embodiment. Although the CPU 510 reads the program data 541 from the HDD 540 and executes it, as another example, these programs may be acquired from another device via the external network 580.

<3. Other embodiments>
The processing according to the embodiment (or modification example) described above may be implemented in various different forms (modification examples) other than the embodiment described above. For example, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or the processes described as being performed manually can be performed manually. All or part of the process can also be performed automatically using known methods. In addition, information including the processing procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed arbitrarily, unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

Furthermore, each component of each device shown in the drawings is functionally conceptual, and does not necessarily need to be physically configured as shown in the drawings. In other words, the specific form of distributing and integrating each device is not limited to what is shown in the diagram, and all or part of the devices can be functionally or physically distributed or integrated in arbitrary units depending on various loads and usage conditions. Can be integrated and configured.

Furthermore, the above-described embodiments (or modified examples) can be combined as appropriate within a range that does not conflict with the processing contents. Furthermore, the effects described in this specification are merely examples and are not limited, and other effects may also be present.

<4. Additional notes>
Note that the present technology can also have the following configuration.
(1)
a plurality of light sources that irradiate an object with light including infrared radiation;
a control unit that turns on each of the plurality of light sources at different timings;
an imaging unit that acquires a distance image and an infrared image of the object at each timing;
Equipped with
The control unit generates a composite distance image by overlapping the distance images for each of the timings, generates a composite infrared image by overlapping the infrared images for each of the timings, and generates a composite infrared image based on the composite distance image and the composite infrared image. recognize the object by
Object recognition system.
(2)
The control unit superimposes the distance images for each timing and superimposes the infrared images for each timing so as to remove a saturated region that depends on the intensity of reflected light in the distance images and infrared images for each timing, generating the composite range image and the composite infrared image;
The object recognition system according to (1) above.
(3)
The control unit turns off the plurality of light sources at different timings, respectively.
The object recognition system according to (1) or (2) above.
(4)
The plurality of light sources are provided around the imaging unit in plan view,
The object recognition system according to any one of (1) to (3) above.
(5)
The plurality of light sources are provided so as to face each other with the imaging unit in between in plan view.
The object recognition system according to any one of (1) to (4) above.
(6)
The plurality of light sources are provided so as to surround the imaging unit in a plan view,
The object recognition system according to any one of (1) to (5) above.
(7)
The plurality of light sources are provided so as to be point symmetrical with respect to the imaging unit in a plan view,
The object recognition system according to any one of (1) to (6) above.
(8)
The control unit has a light source driver common to the plurality of light sources,
The light source driver turns on at least one of the plurality of light sources based on synchronization timing issued from the imaging unit.
The object recognition system according to any one of (1) to (7) above.
(9)
The control unit has a light source driver for each of the light sources,
Each of the light source drivers for each light source turns on the light source based on synchronization timing issued from the imaging unit.
The object recognition system according to any one of (1) to (7) above.
(10)
The control unit includes a timing distribution unit,
The timing distribution unit issues light emission timing to the light source driver for each light source based on the synchronization timing issued from the imaging unit,
Each of the light source drivers for each light source lights up the light source based on the light emission timing issued from the timing distribution unit.
The object recognition system according to (9) above.
(11)
The control unit includes a light source driver and a synchronization control unit for each of the light sources,
The synchronization control unit issues readout timing to the imaging unit, and issues light emission timing to the light source driver for each light source in synchronization with the readout timing,
Each of the light source drivers for each light source lights up the light source based on the light emission timing issued from the synchronization control unit,
The imaging unit acquires the distance image and the infrared image based on the readout timing issued from the synchronization control unit.
The object recognition system according to any one of (1) to (7) above.
(12)
The imaging unit is an iToF (indirect time of flight) image sensor,
The object recognition system according to any one of (1) to (11) above.
(13)
The control unit turns on some light sources among the plurality of light sources at the same timing.
The object recognition system according to any one of (1) to (12) above.
(14)
the object is a box;
The box features include contours and non-contours;
The object recognition system according to any one of (1) to (13) above.
(15)
the contour includes a groove line;
the non-contour portion includes black tape;
The object recognition system according to (14) above.
(16)
The non-contour portion includes a seal in addition to the black tape,
The object recognition system according to (15) above.
(17)
a plurality of light sources that irradiate an object with light including infrared radiation;
a control unit that turns on each of the plurality of light sources at different timings;
an imaging unit that acquires a distance image and an infrared image of the object at each timing;
Equipped with
The control unit generates a composite distance image by overlapping the distance images for each of the timings, generates a composite infrared image by overlapping the infrared images for each of the timings, and generates a composite infrared image based on the composite distance image and the composite infrared image. recognizing the object by
Object recognition device.
(18)
The object recognition system
Turning on multiple light sources that irradiate an object with light including infrared rays at different times,
acquiring a distance image and an infrared image of the object at each of the timings;
Generating a composite distance image by overlapping the distance images for each timing, generating a composite infrared image by overlapping the infrared images for each timing, and recognizing the object based on the composite distance image and the composite infrared image. to do and
Object recognition methods, including:
(19)
An object recognition device having the configuration according to the object recognition system according to any one of (1) to (16) above.
(20)
An object recognition method, wherein the object recognition system according to any one of (1) to (16) above recognizes an object.

1 Object recognition system 10 Transport unit 20 Imaging unit 30 Light source 40 Robot 50 Control unit 51 Light source driver 52 Image processing unit 53 Timing distribution unit 54 Synchronization control unit A1 Object A1a Black tape A1b Groove line G1 Both images G2 Both images G3 Both composite images R1 saturation region

Claims

a plurality of light sources that irradiate an object with light including infrared radiation;
a control unit that turns on each of the plurality of light sources at different timings;
an imaging unit that acquires a distance image and an infrared image of the object at each timing;
Equipped with
The control unit generates a composite distance image by overlapping the distance images for each of the timings, generates a composite infrared image by overlapping the infrared images for each of the timings, and generates a composite infrared image based on the composite distance image and the composite infrared image. recognize the object by
Object recognition system.
The control unit superimposes the distance images for each timing and superimposes the infrared images for each timing so as to remove a saturated region depending on the intensity of reflected light in the distance image and infrared image for each timing, generating the composite range image and the composite infrared image;
The object recognition system according to claim 1.
The control unit turns off the plurality of light sources at different timings, respectively.
The object recognition system according to claim 1.
The plurality of light sources are provided around the imaging unit in plan view,
The object recognition system according to claim 1.
The plurality of light sources are provided so as to face each other with the imaging unit in between in plan view.
The object recognition system according to claim 1.
The plurality of light sources are provided so as to surround the imaging unit in a plan view,
The object recognition system according to claim 1.
The plurality of light sources are provided so as to be point symmetrical with respect to the imaging unit in a plan view,
The object recognition system according to claim 1.
The control unit has a light source driver common to the plurality of light sources,
The light source driver turns on at least one of the plurality of light sources based on synchronization timing issued from the imaging unit.
The object recognition system according to claim 1.
The control unit has a light source driver for each of the light sources,
Each of the light source drivers for each light source turns on the light source based on synchronization timing issued from the imaging unit.
The object recognition system according to claim 1.
The control unit includes a timing distribution unit,
The timing distribution unit issues light emission timing to the light source driver for each light source based on the synchronization timing issued from the imaging unit,
Each of the light source drivers for each light source lights up the light source based on the light emission timing issued from the timing distribution unit.
The object recognition system according to claim 9.
The control unit includes a light source driver and a synchronization control unit for each of the light sources,
The synchronization control unit issues readout timing to the imaging unit, and issues light emission timing to the light source driver for each light source in synchronization with the readout timing,
Each of the light source drivers for each light source lights up the light source based on the light emission timing issued from the synchronization control unit,
The imaging unit acquires the distance image and the infrared image based on the readout timing issued from the synchronization control unit.
The object recognition system according to claim 1.
The imaging unit is an iToF (indirect time of flight) image sensor,
The object recognition system according to claim 1.
The control unit turns on some light sources among the plurality of light sources at the same timing.
The object recognition system according to claim 1.
the object is a box;
The box features include contours and non-contours;
The object recognition system according to claim 1.
the contour includes a groove line;
the non-contour portion includes black tape;
The object recognition system according to claim 14.
The non-contour portion includes a seal in addition to the black tape,
The object recognition system according to claim 15.
a plurality of light sources that irradiate an object with light including infrared radiation;
a control unit that turns on each of the plurality of light sources at different timings;
an imaging unit that acquires a distance image and an infrared image of the object at each timing;
Equipped with
The control unit generates a composite distance image by overlapping the distance images for each of the timings, generates a composite infrared image by overlapping the infrared images for each of the timings, and generates a composite infrared image based on the composite distance image and the composite infrared image. recognize the object by
Object recognition device.
The object recognition system
Turning on multiple light sources that irradiate an object with light including infrared rays at different times,
acquiring a distance image and an infrared image of the object at each of the timings;
Generating a composite distance image by overlapping the distance images for each timing, generating a composite infrared image by overlapping the infrared images for each timing, and recognizing the object based on the composite distance image and the composite infrared image. to do and
Object recognition methods, including: