JP2020190437A - Foreign body detection system and method - Google Patents

Abstract

To provide a foreign body detection system and a method capable of detecting a foreign body with high accuracy by a structure as simple as possible.SOLUTION: A foreign body detection system comprises an imaging device 5 that acquires an image in a target area A, an image processing section that stores a program for detecting objects in the image acquired by the imaging device 5 using deep learning, and a boundary determination section 6 that determines a positional relationship between a transparent section 4 allowing viewing the outside from the inside of the target area A and a peripheral object. The system is configured so as to implement mask processing in an area occupied by the transparent section 4 with respect to the image acquired by the imaging device 5 and detect an object with the program using deep learning on the basis of the image on which mask processing has been implemented.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a system and a method for detecting a foreign substance in a target area such as a parking lot.

In a space where a vehicle enters and stops, for example, in a space for entering and exiting a mechanical parking lot, when opening and closing the door of the doorway and carrying a pallet on which the vehicle is placed, the driver in addition to the vehicle It is necessary to confirm that there are no foreign objects such as passengers, occupants, and baggage. Such confirmation work of the absence of foreign matter can be performed visually by the driver, facility staff, etc., but if only human power is relied on, the confirmation may not be thorough due to carelessness or the like. Therefore, in recent years, there has been a demand for a technique for automatically detecting a foreign substance in a target area without relying only on human power so as to reliably confirm the absence of the foreign substance.

In addition, if the automated driving technology currently being developed in each country is put into practical use in earnest, not only driving the vehicle but also various things such as opening and closing doors and transporting pallets as described above will be performed around the vehicle. It can be assumed that automation will be promoted for the operation of. For example, there may be a case where the vehicle delivers the occupant to the destination by automatic driving, then automatically moves to a nearby mechanical parking lot and is stored in the storage space. In this case, the vehicle moves in the absence of the occupants and stops in the entry / exit space. At this time, operations such as opening and closing the door and starting the movement of the pallet may be executed by the person in charge visually confirming the absence of foreign matter and inputting the operation button. However, it will be smoother if there is a mechanism to automatically confirm the absence of foreign matter and start the operation.

Such a mechanism for detecting a foreign substance may be widely required not only in a mechanical parking lot but also in a place where some operation is performed in the vicinity of a stopped vehicle such as a power supply facility for an electric vehicle. Further, the same mechanism can be widely applied not only to a vehicle but also to a situation where foreign matter is desired to be detected around some object.

As a technique for automatically detecting foreign matter in the target area, various methods have been proposed in which an image in the target area is acquired by using a camera or the like and the foreign matter reflected in the image is detected. Examples of technical documents describing such techniques include the following Patent Documents 1 to 6.

JP-A-2016-3493 JP-A-2015-55073 Japanese Unexamined Patent Publication No. 2014-80735 Japanese Unexamined Patent Publication No. 08-42184 Japanese Unexamined Patent Publication No. 62-194366 Japanese Unexamined Patent Publication No. 2012-158868

In the above-mentioned technology for detecting foreign matter, it goes without saying that the minimum accuracy is required for detection, but in addition, simplifying the device configuration as much as possible increases the cost of construction or maintenance. It is important in terms of aspects. However, all of the techniques described in Patent Documents 1 to 6 have not always sufficiently satisfied both accuracy and cost.

Therefore, in the present disclosure, a foreign matter detection system and a method capable of detecting a foreign matter with high accuracy with a structure as simple as possible will be described.

The present disclosure includes an image pickup device for acquiring an image in a target area, an image processing unit for storing a program for detecting an object in an image acquired by the image pickup device by using deep learning, and an image processing unit from the inside of the target area. It is provided with a boundary determination unit that determines the positional relationship between the transparent unit that can visually recognize the outside and surrounding objects, and masks the area occupied by the transparent unit on the image acquired by the imaging device. This applies to a foreign matter detection system configured to detect an object by a program using deep learning based on the image that has passed through.

In the above-mentioned foreign matter detection system, the mask processing may be configured to be performed by the image pickup apparatus.

In the above-mentioned foreign matter detection system, the mask processing may be configured to be performed by the image processing unit.

In the above-mentioned foreign matter detection system, the boundary determination unit can be an object sensor configured to be able to detect the presence or absence of an object in the boundary area with the external space in the target area.

In the above-mentioned foreign matter detection system, the object sensor can be configured to detect an irradiation wave irradiated along a surface formed by the transmission portion.

In the above-mentioned foreign matter detection system, the boundary determination unit is acquired by the image pickup apparatus based on the positional relationship between the position of the object in the image acquired by the image pickup apparatus and the region occupied by the transmission portion in the image. It can be configured to determine whether the object detected in the image is inside or outside the target area.

In the above-mentioned foreign matter detection system, the target area can be an entry / exit space for a mechanical parking lot.

Further, in the present disclosure, an image acquisition step of acquiring an image in the target area and a masking process are applied to an area occupied by a transparent portion that can visually recognize the outside from the inside of the target area with respect to the image acquired in the image acquisition step. The mask processing step to be applied, the first object detection step of detecting an object by a program using deep learning based on the image obtained through the mask processing step, and the second detection of the presence or absence of an object in the boundary region of the target region. It relates to a foreign matter detection method including the object detection step of the above.

Further, the present disclosure includes an image acquisition step of acquiring an image in a target region, an object detection step of detecting an object by a program using deep learning based on the image acquired in the image acquisition step, and the inside of the image. The object detected in the image is inside or outside the target area based on the positional relationship between the position of the object in the image and the area occupied by the transmissive portion that can visually recognize the outside from the inside of the target area. It relates to a foreign matter detection method including a position determination step of determining whether or not the object is present.

In the above-mentioned foreign matter detection method, the target area can be an entry / exit space for a mechanical parking lot.

According to the foreign matter detection system and method of the present invention, it is possible to obtain an excellent effect that foreign matter can be detected with high accuracy by the simplest possible configuration.

It is a schematic plan view explaining the arrangement of the apparatus in the foreign matter detection system according to 1st Embodiment of this disclosure. It is a block diagram explaining the system structure of the foreign matter detection system by 1st Embodiment of this disclosure. It is a figure which shows an example of the image acquired in the 1st Example of this disclosure. It is a figure which shows an example of the image which was acquired in the 1st Example of this disclosure and was partially masked. It is a figure which shows another example of the image which was acquired in 1st Example of this disclosure and was partially masked. It is a flowchart explaining an example of the procedure of the foreign matter detection method by 1st Example of this disclosure. It is a schematic plan view explaining the arrangement of the apparatus in the foreign matter detection system according to 2nd Embodiment of this disclosure. It is a block diagram explaining the system structure of the foreign matter detection system according to the 2nd Example of this disclosure. It is a figure which shows an example of the image acquired in the 2nd Example of this disclosure. It is a figure which shows an example of the image which was acquired in the 2nd Example of this disclosure and was subjected to the object detection processing by a program. It is a flowchart explaining an example of the procedure of the foreign matter detection method by 2nd Example of this disclosure.

Hereinafter, embodiments of the embodiments in the present disclosure will be described with reference to the accompanying drawings. In the foreign matter detection system of the first embodiment shown in FIG. 1, the entry / exit space of the mechanical parking facility is assumed as the target area A. Then, when an object (foreign matter F) other than the vehicle 1 exists in the target area A, the foreign matter F is detected. The target area A, which is an entry / exit space, is a space surrounded by walls on all sides, and an entry / exit port 4 as a transparent portion is provided on the front side (lower side in the figure) so that the vehicle 1 can enter and exit from here. It has become.

In the present specification, the "foreign substance" is an object other than the target object (in this case, the vehicle 1 which is the target object of transportation) existing in the target area, and is not an object originally arranged in the target area. It refers to the object of the vehicle, and is typically a person such as a vehicle driver, a occupant, or an equipment clerk. In addition, baggage and the like brought into the target area A and placed outside the vehicle 1 are also examples of "foreign substances". FIG. 1 illustrates two foreign objects F inside the target region A as an example. Further, one object O is illustrated outside the target area A.

The “transmissive portion” refers to an optical opening in which the outside can be visually recognized from the inside of the target area. That is, the transmissive portion in the first embodiment is the entrance / exit 4 that optically and materially communicates with the external space, but in addition to this, for example, between a glass window through which light transmits and an adjacent space. A part separated by a fence can also be assumed as a "transparent part".

The pallet 2 that conveys the vehicle 1 enters and exits the target area A, which is a warehousing / delivery space, from a storage space (not shown) outside the target area A. The fixed position of the pallet 2 in the target area A is the central portion of the target area A. At the time of warehousing, the vehicle 1 enters the target area A through the warehousing / delivery port 4, stops on the pallet 2, is transported to the storage space together with the pallet 2, and is stored. At the time of delivery, the vehicle 1 is transported from the storage space into the target area A together with the pallet 2, and then exits from the target area A through the entry / exit port 4.

In the target area A, an image pickup device 5 such as a camera that acquires an image in the target area A is arranged. In the case of the first embodiment, as the image pickup device 5, as shown in FIG. 1, a total of two image pickup devices 5a and 5b are installed around the stop position of the vehicle 1. Of the two image pickup devices 5, the image pickup device 5a is installed on the left front side when viewed from the stop position of the vehicle 1, and the image pickup device 5b is installed on the right rear side. In this way, by installing the image pickup devices 5a and 5b diagonally forward and diagonally rearward so as to sandwich the stop position of the vehicle 1 with each other, it is possible to acquire an image of the entire circumference of the vehicle 1 in the target area A. There is.

An object sensor 6 as a boundary determination unit is provided on the inner wall near the entrance / exit 4 which is a transmission unit. The "boundary determination unit" in the present specification is a device for determining the positional relationship between a transmission unit (entrance / exit port 4), which is an optical opening, and a surrounding object (foreign matter F or object O), or a device thereof. Refers to the part. In the case of the first embodiment, when the object sensor 6 which is the boundary determination unit is installed in the vicinity of the entry / exit port 4, and some object is located on the boundary line inside and outside the target area A defined by the entry / exit port 4. In addition, the object is detected. As described above, if the object sensor 6 is provided as the boundary determination unit, the device configuration can be simplified when the foreign matter detection system is implemented.

The object sensor 6 is, for example, a transmissive photoelectric sensor. The object sensor 6 includes an irradiation unit 6a provided on one left and right sides of the warehousing / delivery port 4 and a light receiving unit 6b provided on the other left and right sides, and is along a surface formed by the irradiation unit 6a to the warehousing / delivery port 4. The irradiation light as an irradiation wave is irradiated and detected by the light receiving unit 6b. If any object exists between the irradiation unit 6a and the light receiving unit 6b, the amount of irradiation light detected by the light receiving unit 6b changes. In this way, the object sensor 6 can detect an object existing between the irradiation unit 6a and the light receiving unit 6b.

Here, the "boundary area" refers to an area near the boundary between the target area and the external space defined by the transparent portion. More specifically, it refers to a region of the target region A in which the target foreign matter is less likely to be detected when the region mask processing corresponding to the transparent portion of the image is performed in the foreign matter detecting step described later. In the example shown in FIG. 1, the object sensor 6 is arranged so as to irradiate the irradiation wave along the boundary line defined by the entry / exit port 4, but instead of or in addition to this, the target area A is more. The foreign matter F in the boundary region near the inside of the object F may be detected by the object sensor 6. This is to more reliably prevent detection omission due to mask processing (the arrangement of such an object sensor 6 will be described again later). As the object sensor 6, in addition to the transmissive photoelectric sensor, any device capable of detecting the presence or absence of an object in a specific region can be appropriately used. For example, a photoelectric sensor of a type other than the transmission type may be used, or an ultrasonic sensor or the like may be used.

An operation unit 7, a display unit 8, and an alarm unit 9 are provided on the outer wall in the vicinity of the entry / exit port 4. The operation unit 7 is an input device for inputting operations of each unit such as opening / closing the warehousing / delivery port 4 and transporting the pallet 2. The display unit 8 is a display configured to display the image in the target area A acquired by the image pickup apparatus 5 and other information as needed. The alarm unit 9 is a device that issues an alarm when some situation that should call attention to personnel occurs in the target area A or somewhere in the equipment including the target area A. “Some situation that should call attention to personnel” is, for example, a case where a foreign matter F other than the vehicle 1 is detected in the target area A when the vehicle 1 is transported to a storage space outside the target area A. As for the content of the alarm, an appropriate format can be selected as long as it can call attention to personnel such as an alarm sound, lighting of a warning light, and display of a warning message.

The positions where the operation unit 7, the display unit 8, and the alarm unit 9 are installed are not limited to the positions shown here. For example, devices corresponding to these may be provided in a management room or the like (not shown), or any one of these devices may be provided in a plurality of different places. Further, although the operation unit 7, the display unit 8, and the alarm unit 9 are shown separately here for convenience of explanation, for example, a touch panel display having the functions of the operation unit 7 and the display unit 8 may be provided. Good. Further, the display unit 8 may also serve as the function of the alarm unit 9 (that is, the information corresponding to the alarm is displayed on the display unit 8). In addition, the operation unit 7, the display unit 8, and the alarm unit 9 can be appropriately changed according to the layout of the target area A and other conditions.

FIG. 2 shows an example of a system configuration related to the control of the image pickup device 5, the object sensor 6, the operation unit 7, the display unit 8, and the alarm unit 9. The image pickup device 5, the object sensor 6, the operation unit 7, and the alarm unit 9 are informationally connected to the control device 10 via a communication bus such as Ethernet (registered trademark). The control device 10 is a device that monitors the entire mechanical parking lot including the target area A, which is the warehousing / delivery space, and operates each part, and opens / closes the warehousing / delivery port 4, transports the pallet 2, the image pickup device 5, and the object. It controls the operation of the sensor 6.

Further, the display unit 8 is connected to the communication bus via the image processing unit 11. The image processing unit 11 is an information processing device that appropriately processes the image of the target area A acquired by the image pickup device 5, and transmits image data to the display unit 8 as necessary to send the image to the display unit 8. Is also displayed.

As the image processing performed by the image processing unit 11, the detection of the foreign matter F (see FIG. 1) is particularly assumed in the first embodiment. That is, the image processing unit 11 analyzes the image of the target region A acquired by the image pickup apparatus 5 and determines whether or not an object such as a person exists in the image. The control device 10 detects the foreign matter F when the image processing unit 11 detects an object from the image to be processed, or when the object sensor 6 detects an object in the boundary area of the target area A. It is determined that the alarm is issued through the alarm unit 9.

The image processing unit 11 stores a program P for detecting an object reflected in the image. This program P is configured to detect a specific type of object by a deep learning method using a multi-layer neural network. As a deep learning algorithm, various algorithms such as R-CNN (Regional-Convolutional Neural Network), Fast R-CNN, Faster R-CNN, YOLO (You Only Look Once), and SSD (Single Shot Multibox Detector) are used. However, other suitable algorithms may be used.

Program P can use such an algorithm to identify the type of object reflected in the image and output the probability that the object is that type of object, the coordinates of the center of gravity of the object, the width and height of the object, and the like. It has become like. Since the type of foreign matter F assumed in the target area A, which is the entry / exit space of the mechanical parking lot, is firstly a person, the program P should be configured so that at least a person can be identified from the image. .. In addition, various objects such as animals such as pets, bags, strollers, etc. are assumed as objects that may enter the target area A together with the vehicle 1. Therefore, the program P sets these objects in addition to the person. May also be detected. Further, if the type of the target area to which the foreign matter detection system is installed is different, the type of the object detected as the foreign matter may be different. The program P may be configured to be capable of detecting various objects assumed according to the type of the target area, not limited to the person.

When warehousing the vehicle 1, personnel such as the driver, occupants, and facility staff of the vehicle 1 operate the operation unit 7 to instruct the start of warehousing. The control device 10 opens the entry / exit port 4 in response to an instruction. At the start of warehousing, the warehousing / delivery port 4 is opened with the empty pallet 2 deployed in the central portion, which is a fixed position in the target area A.

When the entry / exit port 4 is opened, the driver moves the vehicle 1 into the target area A and stops it on the pallet 2. After the stop, the driver and other occupants get off and exit the target area A. When all the personnel such as the driver have left, the driver or the like visually inspects the target area A, confirms whether or not a foreign object F such as a person or baggage exists in the target area A, and then operates the operation unit 7. , Input that the vehicle 1 has stopped on the pallet 2. The control device 10 closes the entry / exit port 4 and conveys the pallet 2 loaded with the vehicle 1 to the storage space outside the target area A.

When leaving the vehicle 1, personnel such as the driver, crew, and facility staff of the vehicle 1 operate the operation unit 7 to instruct the start of the delivery. The control device 10 conveys the corresponding pallet 2 into the target area A in response to an instruction, and opens the warehousing / delivery port 4. When the entry / exit port 4 is opened, the driver gets into the vehicle 1 on the pallet 2 and moves out of the target area A. When the vehicle 1 goes out, the driver, the staff, etc. of the vehicle 1 operate the operation unit 7 again, and input that the delivery is completed. The control device 10 closes the entry / exit port 4 in response to an instruction.

Although the above series of operations has been described by taking the case where the driver drives the vehicle 1 as an example, the vehicle 1 may be an autonomous driving vehicle. In that case, a part or all of the above-mentioned operation of the vehicle 1 is performed without the intervention of the driver.

The image processing unit 11 (see FIG. 2) executes foreign matter detection at an appropriate timing. For example, when the vehicle 1 stops on the pallet 2 at the time of warehousing and the driver inputs the fact to the operation unit 7, the foreign matter is detected. At this time, the driver visually confirms the absence of the foreign matter F in the target area A, but at the same time, if the object sensor 6, the imaging device 5, and the image processing unit 11 are used to mechanically detect the foreign matter, the confirmation can be confirmed. You can do it thoroughly. In addition, it is necessary to check the state in the target area A, for example, when an instruction to start shipping is input, when the completion of shipping is input and the loading / unloading port 4 is closed, or in normal times when no particular operation is performed. Foreign matter can be detected at any time when it is considered that there is.

When detecting a foreign object, the control device 10 operates each image pickup device 5 to acquire an image in the target area A. The image processing unit 11 executes object detection processing on the images acquired by each image pickup device 5 by the program P. Further, the control device 10 operates the object sensor 6 and confirms whether or not an object exists in the boundary area of the target area A.

As a result of image processing, when an object (for example, a person) of a type assumed to be a foreign matter F is detected in at least one of the acquired images, the control device 10 activates the alarm unit 9. And issue an alarm. At the same time, the operation of the pallet 2, the entrance / exit 4 or other parts is prohibited. Further, the display unit 8 may indicate that the foreign matter F has been detected. Further, when an object is detected by the object sensor 6, the control device 10 performs the same operation.

As described above, when the foreign matter is detected by using the program P, if the object detection by the object sensor 6 is also executed, the accuracy of the foreign matter detection can be improved. This will be described in detail below.

When it is desired to detect a foreign matter F in the target area A as shown in FIG. 1 based on the image in the target area A acquired by the image pickup apparatus 5, it is important not to create a blind spot first. On the other hand, it is desirable that the number of devices constituting the system is as small as possible, and in particular, the number of image pickup devices 5 should be as small as possible within the range in which the accuracy of foreign matter detection can be ensured. When image processing is performed by the program P using deep learning as described above, a considerable amount of calculation load is generated on the image processing unit 11. In FIG. 2, the image processing unit 11 is displayed as one block, but in reality, for example, one information processing device may be required for each image pickup device 5. As described above, it is very important to reduce the number of image pickup devices 5 in terms of cost, especially when the foreign matter is detected by using the program P using deep learning. The arrangement of the two image pickup devices 5 shown in FIG. 1 is an idea for acquiring an image in the target area A without creating a blind spot by a small number of image pickup devices 5 (note that, here, two devices). Although the foreign matter detection system provided with the image pickup device 5 of the above is illustrated, the number of these two image pickup devices 5 is the minimum necessary for eliminating the blind spot, and the image pickup device is further for the purpose of further improving the detection accuracy. It does not prevent the addition of etc.).

Furthermore, when detecting foreign matter, it is necessary to reduce both detection omission and over-detection as much as possible. Here, when the image pickup apparatus 5 is arranged as shown in FIG. 1, the entrance / exit port 4 is largely inserted into the angle of view of each image pickup apparatus 5. As a result, for example, in the image acquired by the image pickup apparatus 5b, as shown in FIG. 3, there is a possibility that the object O outside the target area A is reflected in the image through the entrance / exit 4 which is a transparent portion. is there. If the object O is a type of object (here, a person) that can be detected by the program P, overdetection may occur.

In order to prevent over-detection, for example, as shown in FIG. 4, a method of performing foreign matter detection by program P after masking the area corresponding to the transparent portion (entrance / exit port 4) reflected in the image is used. Conceivable. However, in this case, a part of the foreign matter F in the vicinity of the warehousing / delivery port 4 is also masked. If the masked amount or position is as shown in FIG. 4, the masked object F can be detected by the program P, but if the position of the foreign matter F is further outward, FIG. 5 As shown in the above, most of the foreign matter F may be masked. In the example shown in FIG. 5, the foreign matter F in the vicinity of the warehousing / delivery port 4 is located inside the target area A, but since most of the foreign matter F has been masked by the masking process, it may not be detected by the program P. is there. Therefore, by activating the object sensor 6 arranged as shown in FIG. 1, the foreign matter F in the boundary region that may not be detected by the mask processing is reliably detected. That is, when the foreign matter is detected by using the program P, the accuracy of foreign matter detection is improved by preventing over-detection by mask processing by the image processing unit 11 and detection omission by the object sensor 6 which is the boundary determination unit. It is.

By the way, when the object sensor 6 is arranged so as to irradiate the irradiation wave along the boundary line inside and outside the target region A as shown in FIG. 1, the foreign matter F on the boundary line can be suitably detected by the object sensor 6. On the other hand, the foreign matter F located just inside the entrance / exit port is not exposed to the irradiation wave of the object sensor 6, and most of the image acquired by the image pickup device 5 is masked through the masking process. A case is also assumed. In such a case, the possibility of omission of detection cannot be denied. Therefore, instead of or in addition to the object sensor 6 at the position shown in FIG. 1, the object sensor 6 may be installed so that the foreign matter F can be detected even in the inner boundary region. In this way, when the object sensor 6 is installed in the boundary area inside the boundary line defined by the entry / exit port 4, it is optimal to set the arrangement of the object sensor 6 as follows. ..
-If the foreign matter F is located inside the region that can be the target of object detection by the object sensor 6, the object sensor 6 is arranged so that the object sensor 6 can be detected by the program P even from the image that has undergone mask processing.
That is,
-The object sensor 6 is arranged so that the boundary area is the target of foreign matter detection by the object sensor 6.

It should be noted that the "region that can be the target of foreign matter detection by the object sensor 6" and the "region where the foreign matter F can be detected by the program P even from the image that has undergone mask processing" may overlap with each other.

An example of the procedure of the foreign matter detection process by the image pickup apparatus 5, the image processing unit 11, and the object sensor 6 as described above will be described with reference to the flowchart of FIG. The foreign matter detection step described below includes an image acquisition step (step S1), a mask processing step (step S2), a first object detection step (step S3), and a second object detection step (step S4). Includes.

First, the image pickup apparatus 5 acquires an image in the target area A (step S1). Subsequently, the acquired image is masked in the area occupied by the transparent portion (entrance / exit port 4) (step S2). The mask processing may be performed by the image pickup apparatus 5 or the image processing unit 11. Further, the object detection process is executed by the program P stored in the image processing unit 11 on the image that has undergone the mask processing (step S3).

On the other hand, the object sensor 6 is operated to detect the presence or absence of an object in the irradiation region of the irradiation wave (step S4). In step S5, it is determined whether the program P has detected an object of a specific type in the image, or the object sensor 6 has detected any object in the irradiation area. If the object is detected in either one or both, it is determined that the foreign matter F exists in the target area A (step S6). If it is determined that the foreign matter F is present, the control device 10 performs various operations to be performed when the foreign matter F is detected. For example, the display unit 8 displays that a foreign substance has been detected. At this time, a message in characters or voice is played, or the background color of the display unit 8 is changed to call attention to the operator (the driver of the vehicle 1 or the staff of the facility). Further, if necessary, the operation of the equipment (for example, opening / closing the storage port 4, transporting the pallet 2 and the like) may be prohibited.

If no object is detected by either the program P or the object sensor 6, it is assumed that no foreign matter exists, and the foreign matter detection step is terminated (step S7).

The procedure described here is an example, and changes can be made such as changing the order of each step as appropriate or adding another step. For example, steps S1 to S3 for detecting an object by the image processing unit 11 and the program P and steps S4 for detecting an object by the object sensor 6 may be ordered or executed in parallel. Further, although the step of determining the object detection result is collectively described as step S5, the determination of the object detection result by the image processing unit 11 and the program P and the determination of the object detection result by the object sensor 6 are separately described here. You may go to.

As described above, the foreign matter detection system of the first embodiment detects the image pickup device 5 that acquires the image in the target area A and the object in the image acquired by the image pickup device 5 by using deep learning. An image processing unit 11 for storing the program P, a boundary determination unit 6 for determining the positional relationship between the transparent unit 4 capable of visually recognizing the outside from the inside of the target area A and surrounding objects, and acquired by the image pickup apparatus 5. The image is masked in the area occupied by the transparent portion 4, and the object is detected by a program using deep learning based on the image that has undergone the masking process. In this way, when the foreign matter is detected by using the program P, the accuracy of the foreign matter detection can be improved by also executing the object detection by the boundary determination unit 6.

In the above-mentioned foreign matter detection system, the mask processing can be performed by the image pickup apparatus.

In the above-mentioned foreign matter detection system, the mask processing may be performed by the image processing unit.

In the above-mentioned foreign matter detection system, the boundary determination unit 6 can be an object sensor configured to be able to detect the presence or absence of an object in the boundary area with the external space in the target area A. In this way, the foreign matter detection system can be implemented with a simple device configuration.

In the above-mentioned foreign matter detection system, the object sensor 6 can be configured to detect the irradiation wave irradiated along the surface formed by the transmission unit 4. In this way, the foreign matter F on the boundary line can be suitably detected by the object sensor 6.

In the above-mentioned foreign matter detection system and method, the target area A can be the entry / exit space of the mechanical parking lot, and in this way, the foreign matter detection can be suitably performed in the entry / exit space of the mechanical parking lot. Can be done.

Further, in the foreign matter detection method of the first embodiment, the image acquisition step (step S1) of acquiring the image in the target area A and the image acquired in the image acquisition step are external to the inside of the target area A. A first object detection step of detecting an object by a program P using deep learning based on an image that has undergone the mask processing step (step S2) and a mask processing step (step S2) in which the area occupied by the transparent portion 4 is visible. (Step S3) and a second object detection step (step S4) of detecting the presence or absence of an object in the boundary region of the target region A are included. In this way, when the foreign matter is detected by using the program P, the accuracy of foreign matter detection can be improved by preventing over-detection by mask processing and detection omission by the boundary object sensor 6.

Therefore, according to the first embodiment described above, foreign matter can be detected with high accuracy with a configuration as simple as possible.

7 and 8 show the configuration of the foreign matter detection system according to the second embodiment of the present disclosure. The foreign matter detection system of the second embodiment has the same basic configuration as that of the first embodiment (see FIGS. 1 and 2) described above, but instead of the object sensor 6, it is bounded by the image processing unit 11. A determination unit 12 is provided. The boundary determination unit 12 determines the positional relationship between the transmission unit (entrance / exit port 4) and the surrounding object (foreign matter F or object O) based on the image acquired by the image pickup device 5. .. More specifically, in the boundary determination unit 12, the object is inside or outside the target area A based on the positional relationship of the object (foreign matter F or object O) reflected in the image with respect to the transmission unit (entrance / exit port 4). It is designed to determine whether it is in one of them.

When an image is acquired by the image pickup device 5 in the foreign matter detection system shown in FIG. 7, for example, the image pickup device 5a acquires an image as shown in FIG. When the object detection process is executed by the program P of the image processing unit 11 for such an image, two foreign objects F in the target area A and an object O outside the entry / exit port 4 are detected.

Program P identifies the type of each object as well as its position in the image. The type and position of the object can be displayed superimposed on the image, for example, as shown in FIG. Here, around each of the detected foreign objects F and object O, the range they occupy is illustrated by a rectangular frame, the type of object identified (here, a person), and each object is an object of that type. A certain probability is displayed together.

In the boundary determination unit 12, whether the foreign matter F and the object O are inside the target area A depending on the position of these objects with respect to the entrance / exit 4 which is the boundary between the target area A and the external space. , Determine if it is on the outside.

The inside / outside determination can be made, for example, by the hierarchical relationship between each frame displaying the range occupied by each object and the area occupied by the warehousing / delivery port 4. In FIG. 10, the area occupied by the warehousing / delivery port 4 is a substantially quadrangular portion shown by a broken line in the figure. It is determined whether each object is inside or outside the entrance / exit 4 depending on how much the line segment corresponding to the lower side of the frame indicating the position of each object overlaps the pixel area of this substantially quadrangle. be able to.

In determining the inside and outside, for example, among the line segments corresponding to the lower side of the frame displayed around each foreign matter F or object O, the portion of a certain threshold value (for example, 80%) or more is the pixel area occupied by the warehousing / delivery port 4. If it is inside, it is determined that the object is outside the target area A. If the ratio of the portion inside the pixel region in the frame is less than the threshold value, it is determined that the object is in the target region A. In the example shown in FIG. 10, it can be determined that the two foreign objects F located on the front side are located inside the target area A and the object O on the back side is located outside. The determination method described here is merely an example, and any other appropriate method can be adopted as the method for determining the inside and outside of each object based on the image.

An example of the procedure of the foreign matter detection step by the program P of the image processing unit 11 and the boundary determination unit 12 as described above will be described with reference to the flowchart of FIG. The foreign matter detection step described below includes an image acquisition step (step S11), an object detection step (step S12), and a position determination step (step S15).

First, the image pickup apparatus 5 acquires an image in the target area A (step S11). Subsequently, the object detection process is executed by the program P on the acquired image (step S12).

In step S13, as a result of the object detection process, it is determined whether or not any object (foreign matter F or object O) that can be regarded as the foreign matter F is detected in the image by the program P. If a specific type of object does not exist, it is assumed that no foreign matter exists, and the foreign matter detection step ends (step S14).

If any object is detected in the image, the process proceeds to step S15. In step S15, the boundary determination unit 12 determines the position of each detected object with respect to the target area A. In step S16, it is determined whether or not there is an object determined to be located in the target area A among the objects detected in step S12. When it is determined that all of the objects are outside the target area A, it is considered that there is no foreign matter, and the foreign matter detection step is terminated (step S14). When it is determined that at least a part of the object is in the target region A, it is determined that the foreign matter F is present, and the foreign matter detection step is terminated (step S17). It should be noted that the procedure described here is an example, and changes can be made such as changing the order of each step as appropriate or adding another step, as described in the first embodiment.

As described above, in the second embodiment, when the foreign matter detection using the program P is executed, the position determination by the boundary determination unit 12 is also executed to improve the accuracy of the foreign matter detection.

In the first embodiment, the object sensor 6 is described as the boundary determination unit, and in the second embodiment, the boundary determination unit 12 stored in the image processing unit 11 is described. However, these are appropriately combined to describe a foreign matter detection system. It is also possible to carry out. In addition, other sensors and the like may be used in combination as appropriate for the purpose of improving the accuracy of foreign matter detection.

Further, in the first and second embodiments, the entry / exit space of the mechanical parking lot is assumed as the target area A, but the same foreign matter detection system can be of any type as long as it is a space where foreign matter detection is required. Can be applied without. For example, it can be assumed that a power supply facility for an electric vehicle or the like is used, and it can also be applied to security in a warehouse with a window, for example.

As described above, in the foreign matter detection system of the second embodiment, the boundary determination unit 12 determines the position of the object in the image acquired by the image pickup apparatus 5 and the area occupied by the transmission unit 4 in the image. Based on the positional relationship, it is configured to determine whether the object detected in the image acquired by the image pickup apparatus 5 is inside or outside the target area A. In this way, when the foreign matter is detected by using the program P, the accuracy of the foreign matter detection can be improved by also executing the position determination by the boundary determination unit 12.

Further, the foreign matter detection method of the second embodiment is performed by an image acquisition step (step S11) for acquiring an image of A in the target region and a program P using deep learning based on the image acquired in the image acquisition step. In the image, based on the object detection step (step S12) of detecting the object, the position of the object in the image, and the area occupied by the transparent portion 4 in which the outside can be visually recognized from the inside of the target area A in the image. Includes a position determination step (step S15) for determining whether the detected object is inside or outside the target area. In this way, when the foreign matter is detected by using the program P, the accuracy of the foreign matter detection can be improved by also executing the position determination of the detected object.

Since other configurations, actions and effects, etc. are the same as those in the first embodiment, the description thereof will be omitted. However, even in the second embodiment, foreign matter can be detected with high accuracy by the simplest configuration possible.

It should be noted that the foreign matter detection system and method of the present invention described in the present disclosure are not limited to the above-described examples, and it goes without saying that various modifications can be made without departing from the gist.

1 Shield (vehicle)
2 Pallet 4 Transparency (entrance / exit)
5 Imaging device 5a Imaging device 5b Imaging device 6 Boundary determination unit (object sensor)
7 Operation unit 8 Display unit 9 Alarm unit 10 Control device 11 Image processing unit 12 Boundary judgment unit A Target area F Object (foreign matter)
O object P program

Claims (10)

An image pickup device that acquires an image in the target area,
An image processing unit that stores a program that detects an object in an image acquired by the image pickup device using deep learning, and an image processing unit.
It is provided with a boundary determination unit that determines the positional relationship between a transparent unit that can visually recognize the outside from the inside of the target area and surrounding objects.
The image acquired by the imaging device is masked in the area occupied by the transmissive portion.
A foreign matter detection system configured to detect an object by a program using deep learning based on the image that has undergone the mask processing. The foreign matter detection system according to claim 1, wherein the mask processing is performed by the imaging device. The foreign matter detection system according to claim 1, wherein the mask processing is performed by the image processing unit. The foreign matter detection system according to any one of claims 1 to 3, wherein the boundary determination unit is an object sensor configured to be able to detect the presence or absence of an object in a boundary area with an external space in the target area. The foreign matter detection system according to claim 4, wherein the object sensor is configured to detect an irradiation wave emitted along a surface formed by the transmission portion. The boundary determination unit is detected in the image acquired by the imaging device based on the positional relationship between the position of the object in the image acquired by the imaging device and the area occupied by the transmissive portion in the image. The foreign matter detection system according to any one of claims 1 to 3, wherein it is configured to determine whether the object is inside or outside the target area. The foreign matter detection system according to any one of claims 1 to 6, wherein the target area is an entry / exit space for a mechanical parking lot. The image acquisition process to acquire the image in the target area and
A mask processing step of masking an area occupied by a transparent portion that allows the outside to be visually recognized from the inside of the target area with respect to the image acquired in the image acquisition step.
The first object detection step of detecting an object by a program using deep learning based on the image that has undergone the mask processing step, and
A foreign matter detection method including a second object detection step of detecting the presence or absence of an object in the boundary area of the target region. The image acquisition process to acquire the image in the target area and
An object detection step of detecting an object by a program using deep learning based on the image acquired in the image acquisition step, and an object detection step.
An object detected in the image is inside the target area based on the positional relationship between the position of the object in the image and the area occupied by the transparent portion that can be visually recognized from the inside of the target area in the image. A foreign matter detection method including a position determination step of determining whether the object is present or outside. The foreign matter detection method according to claim 8 or 9, wherein the target area is an entry / exit space for a mechanical parking lot.

Images