JP7292102B2 - Foreign object detection system and method - Google Patents

Description

The present disclosure relates to systems and methods for detecting foreign objects in areas of interest, such as parking lots.

In a space where a vehicle enters and stops, such as an entry/exit space in a mechanical parking lot, the operator must be present in addition to the vehicle when performing operations such as opening and closing the entrance door and transporting a pallet on which the vehicle is placed. It is necessary to confirm that there are no foreign objects such as passengers, passengers, or baggage. Such confirmation work for the absence of foreign matter can be performed visually by the driver or facility staff. Therefore, in recent years, there has been a demand for a technique for automatically detecting foreign matter in a target area without relying only on human power, so as to reliably confirm the absence of foreign matter.

In addition, if the autonomous driving technology that is currently being developed in various countries is put into practical use, it will not only be possible to drive the vehicle, but also various things around the vehicle such as opening and closing the door and transporting pallets as described above. It can be assumed that automation will also be promoted for the operation of For example, after the vehicle delivers the passengers to the destination by automatic driving, it may automatically move to a nearby mechanical parking lot and be stored in the storage space. In this case, the vehicle will move without a passenger and stop in the loading/unloading space. At this time, operations such as opening and closing the door and starting movement of the pallet may be performed by the staff visually confirming the absence of the foreign object and pressing the operation button. However, it would be smoother if there was a mechanism to automatically confirm the presence of a foreign object and start the operation.

Such a mechanism for detecting a foreign object may be widely required not only in a mechanical parking lot but also in a place where some kind of operation is performed around a stopped vehicle, such as a power feeding facility for an electric vehicle. Moreover, a similar mechanism can be widely applied to situations in which it is desired to detect a foreign object not only in a vehicle but also in the vicinity of some object.

2. Description of the Related Art As techniques for automatically detecting a foreign object in a target area, various methods have been proposed in which an image within the target area is acquired using a camera or the like, and the foreign material reflected in the image is detected. Technical documents describing such techniques include, for example, Patent Documents 1 to 6 below.

JP 2016-3493 A JP 2015-55073 A JP 2014-80735 A JP-A-08-42184 JP-A-62-194366 JP 2012-158968 A

In the technology for foreign matter detection as described above, the minimum accuracy is of course required for detection. important in terms of However, none of the techniques described in Patent Literatures 1 to 6 has necessarily come to fully satisfy both the accuracy and the cost.

Therefore, in the present disclosure, a foreign object detection system and method capable of detecting foreign objects with high accuracy with a simple configuration as much as possible will be described.

The present disclosure includes an imaging device that acquires an image within a target region, an image processing unit that stores a program for detecting an object in the image acquired by the imaging device using deep learning, and from inside the target region A transparent part for visually recognizing the outside and a boundary determination part for judging a positional relationship between a surrounding object, masking the area occupied by the transparent part in the image acquired by the imaging device, and performing the masking process. an object sensor configured to detect an object by a program using deep learning based on an image that has undergone or an image acquired by the imaging device based on the positional relationship within the same image between the position of the object in the image acquired by the imaging device and the area occupied by the transparent portion in the image A foreign object detection system configured to determine whether an object detected within is inside or outside said region of interest .

In the foreign matter detection system described above, the mask processing may be performed by the imaging device.

In the foreign matter detection system described above, the mask processing may be performed by the image processing section.

In the foreign object detection system described above, the object sensor can be configured to detect an irradiation wave irradiated along a plane formed by the transmission section.

In the foreign matter detection system described above, the target area can be an entry/exit space of a mechanical parking lot.

In addition, the present disclosure includes an image acquisition step of acquiring an image within a target region, and masking an area occupied by a transparent portion in which the outside of the target region can be visually recognized from the inside of the image acquired in the image acquisition step. a first object detection step of detecting an object by a program using deep learning based on the image that has undergone the mask processing step; and detecting the presence or absence of an object in the boundary area of the target region by an object sensor. and a second object detection step for detecting a foreign object.

Further, the present disclosure includes an image acquisition step of acquiring an image in a target region, an object detection step of detecting an object by a program using deep learning based on the image acquired in the image acquisition step, and and the position of the object detected in the image based on the positional relationship in the same image between the position of the object in the image and the area occupied in the image by the transparent portion where the outside can be visually recognized from the inside of the target area. It relates to a foreign matter detection method including a position determination step of determining whether it is on or outside.

In the foreign object detection method described above, the target area may be an entry/exit space of a mechanical parking lot.

According to the foreign matter detection system and method of the present invention, an excellent effect can be obtained in that a foreign matter can be detected with high accuracy with a configuration that is as simple as possible.

1 is a schematic plan view for explaining the arrangement of devices in the foreign object detection system according to the first embodiment of the present disclosure; FIG. 1 is a block diagram illustrating the system configuration of a foreign object detection system according to a first embodiment of the present disclosure; FIG. FIG. 4 is a diagram showing an example of an image acquired in the first embodiment of the present disclosure; FIG. FIG. 2 is a diagram showing an example of an image acquired in the first embodiment of the present disclosure and partially subjected to mask processing; FIG. FIG. 4 is a diagram showing another example of an image obtained in the first embodiment of the present disclosure and partially subjected to mask processing; 4 is a flow chart illustrating an example of the procedure of a foreign object detection method according to the first embodiment of the present disclosure; FIG. 5 is a schematic plan view for explaining the arrangement of devices in the foreign object detection system according to the second embodiment of the present disclosure; FIG. 3 is a block diagram illustrating the system configuration of a foreign object detection system according to a second embodiment of the present disclosure; FIG. FIG. 4 is a diagram showing an example of an image acquired in the second embodiment of the present disclosure; FIG. FIG. 10 is a diagram showing an example of an image acquired in the second embodiment of the present disclosure and subjected to object detection processing by a program; 7 is a flow chart explaining an example of the procedure of a foreign object detection method according to the second embodiment of the present disclosure;

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In the foreign object detection system of the first embodiment shown in FIG. 1, the target area A is assumed to be a parking space for mechanical parking equipment. When an object (foreign object F) other than the vehicle 1 exists within the target area A, the foreign object F is detected. The target area A, which is a parking space, is a space surrounded by walls on all four sides, and is provided with a parking opening 4 as a transparent part on the front side (lower side in the drawing), through which the vehicle 1 can enter and exit. It has become.

In this specification, the term "foreign matter" refers to an object other than the object (in this case, the vehicle 1, which is the object of transportation) existing within the target area, and an object other than the object originally placed in the target area. It refers to an object, typically a person such as a vehicle driver, crew member, or facility staff. Baggage or the like that is brought into the target area A and placed outside the vehicle 1 is also an example of the "foreign matter." FIG. 1 shows two foreign substances F inside the target area A as an example. Also, one object O is illustrated outside the target area A. FIG.

A “transmissive portion” refers to an optical opening through which the outside can be seen from inside the target area. In other words, the transmissive part in the first embodiment is the entrance/exit opening 4 that communicates with the external space both optically and physically. A part partitioned by a fence can also be assumed as a "transmission part".

A pallet 2 for transporting a vehicle 1 moves in and out of a storage space (not shown) outside the target area A into and out of the target area A, which is a loading/unloading space. The home position of the pallet 2 in the target area A is the central part of the target area A. FIG. At the time of warehousing, the vehicle 1 enters the object area A through the entrance/exit 4, stops on the pallet 2, is conveyed to the storage space together with the pallet 2, and is stored. When leaving the warehouse, the vehicle 1 is transported from the storage space into the target area A together with the pallet 2, and then leaves the target area A through the entrance/exit 4.例文帳に追加

In the target area A, an imaging device 5 such as a camera for acquiring an image within the target area A is arranged. In the case of the first embodiment, a total of two imaging devices 5a and 5b are installed around the stopping position of the vehicle 1 as the imaging device 5 as shown in FIG. Of the two imaging devices 5, the imaging device 5a is installed on the left front and the imaging device 5b is installed on the right rear when viewed from the stop position of the vehicle 1. As shown in FIG. In this way, by installing the imaging devices 5a and 5b obliquely forward and obliquely rearward so as to sandwich the stop position of the vehicle 1, images of the entire surroundings of the vehicle 1 within the target area A can be acquired. there is

An object sensor 6 as a boundary determination section is provided on the inner wall near the entrance/exit 4, which is a transmission section. In this specification, the term “boundary determination unit” refers to a device that determines the positional relationship between the transmission unit (the entrance/exit port 4), which is an optical opening, and the surrounding object (foreign object F or object O). point to the part In the case of the first embodiment, when the object sensor 6, which is the boundary determination unit, is installed near the entrance/exit 4, and some object is located on the boundary between the inside and outside of the target area A defined by the entrance/exit 4. , the object is detected. By providing the object sensor 6 as the boundary determination section in this manner, the device configuration can be simplified when implementing the foreign object detection system.

The object sensor 6 is, for example, a transmissive photoelectric sensor. The object sensor 6 includes an irradiation unit 6a provided on one of the left and right sides of the entrance/exit 4 and a light receiving unit 6b provided on the other left and right side. irradiating light as an irradiation wave, and detected by the light receiving part 6b. If any object exists between the irradiation unit 6a and the light receiving unit 6b, the amount of irradiation light detected by the light receiving unit 6b changes. Thus, the object sensor 6 can detect an object existing between the irradiation section 6a and the light receiving section 6b.

Here, the “boundary region” refers to a region near the boundary between the target region defined by the transmissive portion and the external space. More specifically, it refers to an area in the target area A in which the target foreign matter is less likely to be detected when the area mask processing corresponding to the transmissive portion of the image is performed in the foreign matter detection step described later. In the example shown in FIG. 1, the object sensor 6 is arranged so as to irradiate the irradiation wave along the boundary line defined by the entrance/exit 4. The object sensor 6 may detect a foreign object F in the boundary area on the inner side of the . This is to more reliably prevent omissions in detection due to mask processing (such arrangement of the object sensors 6 will be described again later). As the object sensor 6, other than a transmissive photoelectric sensor, any device capable of detecting the presence or absence of an object within a specific area can be appropriately used. For example, a photoelectric sensor of a type other than a transmissive type may be used, or an ultrasonic sensor or the like may be used.

An operation unit 7 , a display unit 8 , and an alarm unit 9 are provided on the outer wall near the entrance/exit 4 . The operation unit 7 is an input device for inputting operations of each unit such as opening/closing of the entrance/exit 4 and transportation of the pallet 2 . The display unit 8 is a display configured to display an image within the target area A acquired by the imaging device 5 and other information as necessary. The alarm unit 9 is a device that issues an alarm when a situation requiring the attention of personnel occurs in the target area A or somewhere in the equipment including the target area A. "Some situation that calls attention to personnel" is, for example, a case where a foreign object F other than the vehicle 1 is detected in the target area A when the vehicle 1 is transported to a storage space outside the target area A. The content of the alarm may be an alarm sound, lighting of a warning light, display of a warning message, etc., as long as it can call attention to personnel, and an appropriate format can be selected.

The positions where the operation unit 7, the display unit 8, and the alarm unit 9 are installed are not limited to the positions shown here. For example, devices corresponding to these may be provided in a management room (not shown), or a plurality of these devices may be provided in separate locations. For convenience of explanation, the operation unit 7, the display unit 8, and the alarm unit 9 are shown separately, but for example, a touch panel display integrally having the functions of the operation unit 7 and the display unit 8 may be provided. good. Further, the display unit 8 may also serve as the function of the alarm unit 9 (that is, the information corresponding to the alarm may be displayed on the display unit 8). In addition, the operation unit 7, the display unit 8, and the alarm unit 9 can be appropriately changed according to the layout of the target area A and other conditions.

FIG. 2 shows an example of a system configuration related to control of the imaging device 5, the object sensor 6, the operation section 7, the display section 8, and the alarm section 9. As shown in FIG. The imaging device 5, the object sensor 6, the operation unit 7, and the alarm unit 9 are informationally connected to the control device 10 via a communication bus such as Ethernet (registered trademark). The control device 10 is a device that monitors the entire mechanical parking lot including the target area A, which is the parking space for entering and leaving, and operates each part. It controls the operation of the sensor 6 and the like.

Also, the display unit 8 is connected to the communication bus via the image processing unit 11 . The image processing unit 11 is an information processing device that performs appropriate processing on the image of the target region A acquired by the imaging device 5, and transmits image data to the display unit 8 as necessary to display the image on the display unit 8. is also displayed.

As image processing performed in the image processing unit 11, detection of a foreign object F (see FIG. 1) is particularly assumed in the first embodiment. That is, the image processing unit 11 analyzes the image of the target area A acquired by the imaging device 5 and determines whether or not an object such as a person exists in the image. The control device 10 detects the foreign object F when the image processing unit 11 detects an object from the image to be processed, or when the object sensor 6 detects an object in the boundary area of the target area A. , and issue an alarm through the alarm unit 9 .

The image processing unit 11 stores a program P for detecting an object reflected in an image. This program P is configured to detect a specific type of object by a deep learning method using a multi-layered neural network. As deep learning algorithms, various algorithms such as R-CNN (Regional-Convolutional Neural Network), Fast R-CNN, Faster R-CNN, YOLO (You Only Look Once), and SSD (Single Shot Multibox Detector) are used. can be used, but other suitable algorithms may be used.

Using such an algorithm, the program P can identify the type of object in the image and output the probability that the object is that type of object, the coordinates of the center of gravity of the object, the width and height of the object, etc. It's like Since the first type of foreign matter F assumed in the target area A, which is the entering/exiting space of the mechanical parking lot, is a person, the program P should be configured so that at least a person can be identified from the image. . In addition, as objects that may enter the target area A together with the vehicle 1, various objects such as animals such as pets, bags, and strollers are assumed. can also be detected. In addition, if the type of target area where the foreign object detection system is installed is different, it is conceivable that the type of object detected as a foreign object is also different. The program P may be configured to be capable of detecting not only a person but also various objects assumed according to the type of target area.

When the vehicle 1 is to be parked, personnel such as the driver of the vehicle 1, the crew members, and the staff of the facility operate the operation unit 7 to instruct the start of the warehousing. The controller 10 opens the entrance/exit 4 according to the instruction. At the start of warehousing, the entry/exit port 4 is opened with an empty pallet 2 arranged at the central portion, which is a fixed position within the target area A.

When the entrance/exit 4 is opened, the driver moves the vehicle 1 into the target area A and stops it on the pallet 2 . After stopping, the driver and other passengers get off the vehicle and exit the target area A. After all the personnel such as the driver have left, the driver or the like visually checks the target area A to confirm whether or not foreign matter F such as a person or baggage exists in the target area A, and then operates the operation unit 7. , that the vehicle 1 has stopped on the pallet 2 is input. The control device 10 closes the entrance/exit 4 and transports the pallet 2 loaded with the vehicle 1 to a storage space outside the target area A.

When the vehicle 1 is to leave the garage, personnel such as the driver of the vehicle 1, the crew members, and the staff of the facility operate the operation unit 7 to instruct the start of leaving the garage. The control device 10 conveys the corresponding pallet 2 into the target area A according to the instruction, and opens the entrance/exit 4 . When the entrance/exit 4 is opened, the driver gets into the vehicle 1 on the pallet 2 and moves out of the target area A. When the vehicle 1 has left the vehicle, the driver of the vehicle 1, a staff member, or the like operates the operation unit 7 again to input that the vehicle has left the garage. The control device 10 closes the entrance/exit 4 according to the instruction.

Although the above series of operations has been described as an example in which the driver drives the vehicle 1, the vehicle 1 may be an automatic driving vehicle. In that case, some or all of the above-described operations of the vehicle 1 are performed without intervention of the driver.

The image processing unit 11 (see FIG. 2) executes foreign matter detection at appropriate timing. For example, when the vehicle 1 stops on the pallet 2 at the time of warehousing, foreign matter detection is performed when the driver inputs that fact to the operation unit 7 . At this time, the driver visually confirms the absence of the foreign object F in the target area A. At the same time, the object sensor 6, the imaging device 5, and the image processing unit 11 can be used to mechanically detect the foreign object. can be thorough. In addition, it is necessary to confirm the state of the target area A, for example, when an instruction to start unloading is input, when the completion of unloading is input and the entrance/exit door 4 is closed, or during normal times when no particular operation is performed. A foreign object can be detected at any timing when it is thought that there is.

When performing foreign object detection, the control device 10 operates each imaging device 5 to acquire an image within the target region A. FIG. The image processing unit 11 executes object detection processing on the image acquired by each imaging device 5 by the program P. FIG. Further, the control device 10 activates the object sensor 6 to check whether or not an object exists in the boundary area of the target area A.

As a result of the image processing, when an object of a type assumed to be a foreign object F (for example, a person) is detected in at least one of the acquired images, the control device 10 activates the alarm unit 9. and issue an alarm. At the same time, the operation of the pallet 2, the entrance/exit 4, or other parts is prohibited. Further, the display unit 8 may display that the foreign object F has been detected. Also, when an object is detected by the object sensor 6, the control device 10 performs the same operation.

In this way, when foreign object detection is performed using the program P, if object detection by the object sensor 6 is also executed, the accuracy of foreign object detection can be improved. A detailed description is given below.

When it is desired to detect a foreign object F within the target area A as shown in FIG. 1 based on the image within the target area A acquired by the imaging device 5, it is first important not to create a blind spot. On the other hand, it is desirable that the number of devices constituting the system is as small as possible, and in particular, the number of imaging devices 5 should be as small as possible within the range where the foreign matter detection accuracy can be ensured. When image processing is performed by the program P using deep learning as described above, the image processing unit 11 is subjected to a considerable amount of computational load. Although the image processing unit 11 is shown as one block in FIG. 2, in practice, for example, one imaging device 5 may require about one information processing device. Reducing the number of imaging devices 5 in this way is very important in terms of cost, especially when foreign matter detection is performed using the program P using deep learning. The arrangement of the two imaging devices 5 shown in FIG. 1 is one proposal for acquiring an image in the target area A using a small number of imaging devices 5 without creating a blind spot (here, two imaging devices 5 are arranged). However, these two imaging devices 5 are the minimum number necessary to eliminate blind spots, and for the purpose of further increasing the detection accuracy, further imaging devices etc.).

Furthermore, in foreign matter detection, it is necessary to reduce both detection omissions and over-detections as much as possible. Here, when the imaging devices 5 are arranged as shown in FIG. As a result, for example, in the image acquired by the imaging device 5b, as shown in FIG. be. If the object O is of a type detectable by the program P (here, a person), overdetection may occur.

In order to prevent over-detection, for example, as shown in FIG. 4, there is a method in which a region corresponding to the transmissive portion (entry/exit door 4) reflected in the image is masked, and then foreign matter detection is executed by the program P. Conceivable. However, in this case, part of the foreign matter F in the vicinity of the entrance/exit 4 is also masked. If the masked amount or position is as shown in FIG. 4, the masked object F can be detected by the program P. Most of the foreign matter F may be masked as shown in FIG. In the example shown in FIG. 5, the foreign matter F near the entrance/exit 4 is located inside the target area A, but most of it is masked by the masking process, so there is a possibility that it cannot be detected by the program P. be. Therefore, by activating the object sensor 6 arranged as shown in FIG. 1, the foreign matter F in the boundary area, which may not be detected by the mask processing, can be reliably detected. In other words, when foreign matter is detected using the program P, the mask processing by the image processing unit 11 prevents over-detection, and the object sensor 6, which is the boundary determination unit, prevents detection omissions, thereby increasing the accuracy of foreign matter detection. of.

By the way, when the object sensor 6 is arranged so as to irradiate the irradiation wave along the boundary between the inside and outside of the target area A as shown in FIG. On the other hand, the foreign matter F located just inside the entrance/exit is not hit by the irradiation wave of the object sensor 6, and moreover, most of it is masked in the image acquired by the imaging device 5 through the masking process. case is also assumed. In such a case, it cannot be denied that there is a possibility that detection omissions may occur. Therefore, instead of or in addition to the object sensor 6 at the position shown in FIG. 1, the object sensor 6 may be installed so that the foreign matter F can be detected even in the inner boundary area. In this way, when the object sensor 6 is installed in the boundary area inside the boundary line defined by the entrance/exit 4, the placement of the object sensor 6 is optimally set as follows. .
The object sensor 6 is arranged so that the program P can detect the foreign matter F located inside the area that can be detected by the object sensor 6, even from the masked image.
i.e.
- The object sensor 6 is arranged so that the boundary area is the target of foreign matter detection by the object sensor 6 .

It should be noted that there is no problem even if there is an overlap between "the area that can be detected by the object sensor 6" and "the area where the program P can detect the foreign matter F even from the masked image".

An example of the procedure of the foreign object detection process by the imaging device 5, the image processing unit 11 and the object sensor 6 as described above will be described with reference to the flow chart of FIG. The foreign matter detection process described below includes an image acquisition process (step S1), a mask processing process (step S2), a first object detection process (step S3), and a second object detection process (step S4). contains.

First, the imaging device 5 acquires an image within the target area A (step S1). Subsequently, mask processing is applied to the area occupied by the transmissive portion (the entrance/exit 4) of the acquired image (step S2). The mask processing may be performed by the imaging device 5 or may be performed by the image processing section 11 . Furthermore, an object detection process is performed on the image that has undergone the mask process by the program P stored in the image processing section 11 (step S3).

On the other hand, the object sensor 6 is operated to detect the presence or absence of an object in the irradiation area of the irradiation wave (step S4). In step S5, it is determined whether the program P has detected a particular type of object in the image, or whether the object sensor 6 has detected any object in the illuminated area. If an object is detected in either one or both, it is determined that a foreign object F exists within the target area A (step S6). When it is determined that the foreign object F exists, the control device 10 performs various operations that should be performed when the foreign object F is detected. For example, the display unit 8 displays that a foreign object has been detected. At this time, a text or voice message is played or the background color of the display unit 8 is changed to call attention to the operator (the driver of the vehicle 1 or the staff of the facility, etc.). Further, if necessary, the operation of equipment (for example, opening and closing the entrance 4, transporting the pallet 2, etc.) may be prohibited.

If neither the program P nor the object sensor 6 detects an object, it is determined that no foreign object exists, and the foreign object detection process ends (step S7).

It should be noted that the procedure described here is just an example, and changes can be made such as changing the order of each step or adding another step. For example, steps S1 to S3 for object detection by the image processing unit 11 and the program P and step S4 for object detection by the object sensor 6 may be switched in order or executed in parallel. Also, although the step of determining the result of object detection has been collectively described here as step S5, the determination of the result of object detection by the image processing unit 11 and the program P and the determination of the result of object detection by the object sensor 6 are performed separately. You can go to

As described above, the foreign matter detection system of the first embodiment uses the imaging device 5 that acquires the image within the target area A and the object in the image acquired by the imaging device 5 using deep learning. An image processing unit 11 that stores a program P, a boundary determination unit 6 that determines the positional relationship between a transmission unit 4 that allows the outside to be visually recognized from the inside of the target area A, and surrounding objects. The area occupied by the transmissive portion 4 is masked in the image, and the object is detected by a program using deep learning based on the masked image. In this way, when foreign matter is detected using the program P, the accuracy of foreign matter detection can be improved by also executing object detection by the boundary determination unit 6 .

In the foreign matter detection system described above, mask processing can be performed by the imaging device.

In the foreign matter detection system described above, mask processing may be performed in the image processing section.

In the foreign matter detection system described above, the boundary determination unit 6 can be an object sensor configured to detect the presence or absence of an object in the boundary area between the target area A and the external space. In this way, a foreign object detection system can be implemented with a simple device configuration.

In the foreign matter detection system described above, the object sensor 6 can be configured to detect an irradiation wave irradiated along the plane formed by the transmission section 4 . In this way, the object sensor 6 can suitably detect the foreign matter F on the boundary line.

In the foreign object detection system and method described above, the target area A can be the entering/exiting space of the mechanical parking lot. can be done.

Further, the foreign matter detection method of the first embodiment includes an image acquisition step (step S1) for acquiring an image within the target region A, and an image acquired in the image acquisition step from the inside of the target region A to the outside. A masking process (step S2) for masking the area occupied by the transparent part 4 that can be visually recognized, and a first object detection process for detecting an object by a program P using deep learning based on the image that has undergone the masking process. (Step S3), and a second object detection step (Step S4) of detecting the presence or absence of an object in the boundary area of the target area A. In this way, when foreign matter is detected using the program P, the accuracy of foreign matter detection can be improved by preventing over-detection by the mask processing and omission of detection by the boundary object sensor 6 .

Therefore, according to the first embodiment described above, a foreign object can be detected with high accuracy with the simplest possible configuration.

7 and 8 show the configuration of a foreign object detection system according to a second embodiment of the present disclosure. The foreign object detection system of the second embodiment has the same basic configuration as that of the first embodiment described above (see FIGS. 1 and 2). A determination unit 12 is provided. This boundary determination unit 12 determines the positional relationship between the transmissive portion (entry/exit door 4) and surrounding objects (foreign matter F or object O) based on the image acquired by the imaging device 5. . More specifically, the boundary determining unit 12 determines whether the object (foreign object F or object O) reflected in the image is inside or outside the target area A based on the positional relationship of the object (foreign object F or object O) with respect to the transmissive part (the entrance/exit door 4). It is determined whether there is any one of them.

In the foreign object detection system shown in FIG. 7, when an image is acquired by the imaging device 5, for example, an image as shown in FIG. 9 is acquired by the imaging device 5a. When the program P of the image processing unit 11 executes object detection processing on such an image, two foreign objects F within the target area A and an object O outside the entrance/exit 4 are detected.

The program P identifies the position that each object occupies in the image along with its type. The type and position of the object can be superimposed on the image, as shown in FIG. 10, for example. Here, the range occupied by each detected foreign object F and object O is illustrated by a square frame, and the type of the identified object (here, a person) and each object is an object of that type. Certain probabilities are displayed together.

The boundary determining unit 12 determines whether each foreign object F and the object O are inside the target area A based on the positions of these objects with respect to the entrance/exit 4, which is the boundary between the target area A and the external space. , is outside.

The inside/outside determination can be made, for example, by the vertical relationship between each frame displaying the range occupied by each object and the area occupied by the entrance/exit 4 . In FIG. 10, the area occupied by the entrance/exit 4 is a substantially rectangular portion indicated by broken lines in the drawing. It is determined whether each object is inside or outside the entrance/exit 4 depending on how much the line segment corresponding to the lower side of the frame indicating the position of each object overlaps with this substantially rectangular pixel area. be able to.

In determining the inside/outside, for example, of the line segment corresponding to the lower side of the frame displayed around each foreign object F or object O, a portion exceeding a certain threshold value (for example, 80%) is the pixel area occupied by the entrance/exit 4. If it is inside, the object is determined to be outside the target region A. It is determined that the object is within the target region A if the ratio of the portion of the frame that is inside the pixel region is less than the threshold value. In the example shown in FIG. 10, it can be determined that the two foreign objects F located on the front side are located inside the target area A, and the object O on the back side is located outside. It should be noted that the determination method described here is merely an example, and other appropriate methods can be adopted as the method for determining the inside and outside of each object based on the image.

An example of the procedure of the foreign matter detection process by the program P of the image processing section 11 and the boundary determination section 12 as described above will be described with reference to the flow chart of FIG. The foreign matter detection process described below includes an image acquisition process (step S11), an object detection process (step S12), and a position determination process (step S15).

First, the imaging device 5 acquires an image within the target area A (step S11). Subsequently, object detection processing is performed on the acquired image by the program P (step S12).

In step S13, as a result of the object detection process, it is determined whether or not the program P has detected any object (foreign object F or object O) that can be regarded as the foreign object F in the image. If the specific type of object does not exist, it is determined that no foreign object exists, and the foreign object detection process ends (step S14).

If any object is detected in the image, go to step S15. In step S15, the position of each detected object with respect to the target area A is determined by the boundary determination unit 12 . In step S16, it is determined whether or not there is an object determined to be located within the target area A among the objects detected in step S12. If any of the objects is determined to be outside the target area A, the foreign object detection process is terminated assuming that no foreign object exists (step S14). If it is determined that at least part of the object is within the target region A, it is determined that the foreign matter F is present, and the foreign matter detection step is terminated (step S17). It should be noted that the procedure described here is only an example, and as described in the first embodiment, the order of each step can be changed as appropriate, or other steps can be added.

As described above, in the second embodiment, when executing the foreign object detection using the program P, the position determination by the boundary determination unit 12 is also executed, thereby increasing the accuracy of the foreign object detection.

In the first embodiment, the object sensor 6 is used as the boundary determination unit, and in the second embodiment, the boundary determination unit 12 stored in the image processing unit 11 is described. It is also possible to implement Further, for the purpose of improving the accuracy of foreign matter detection, etc., other sensors may be used together as appropriate.

In addition, in the first and second embodiments, the entry/exit space of a mechanical parking lot was assumed as the target area A, but a similar foreign matter detection system can be applied to any space in which detection of foreign matter is required. can be applied without For example, it is possible to envision a power supply facility for an electric vehicle, and it can also be applied to security in a warehouse with windows, for example.

As described above, in the foreign object detection system of the second embodiment, the boundary determination unit 12 determines the position of the object in the image acquired by the imaging device 5 and the area occupied by the transmission unit 4 in the image. It is configured to determine whether the object detected in the image acquired by the imaging device 5 is inside or outside the target area A based on the positional relationship. In this way, when foreign matter is detected using the program P, the position determination by the boundary determination unit 12 is also performed, thereby improving the accuracy of foreign matter detection.

Further, the foreign matter detection method of the second embodiment includes an image acquisition step (step S11) for acquiring an image of the target region A, and a program P using deep learning based on the image acquired in the image acquisition step. Based on the positional relationship between the object detection step (step S12) of detecting an object and the position of the object in the image and the area occupied in the image by the transparent part 4 where the outside can be visually recognized from the inside of the target area A, and a position determination step (step S15) for determining whether the detected object is inside or outside the target area. In this way, when foreign matter is detected using the program P, the accuracy of foreign matter detection can be improved by also executing the position determination of the detected object.

Other configurations, functions and effects are the same as those of the first embodiment, so description thereof will be omitted. However, according to the second embodiment as well, a foreign object can be detected with a high degree of accuracy with a configuration that is as simple as possible.

The foreign matter detection system and method of the present invention described in the present disclosure are not limited to the above-described embodiments, and can of course be modified in various ways without departing from the scope of the invention.

1 Shield (vehicle)
2 pallet 4 transparent part (entry/exit port)
5 Imaging Device 5a Imaging Device 5b Imaging Device 6 Boundary Determination Unit (Object Sensor)
7 operation unit 8 display unit 9 alarm unit 10 control device 11 image processing unit 12 boundary determination unit A target area F object (foreign matter)
O object P program

Claims (9)

an imaging device that acquires an image within a target region;
an image processing unit that stores a program for detecting an object in an image acquired by the imaging device using deep learning;
A boundary determination unit that determines a positional relationship between a transmission unit that can visually recognize the outside from the inside of the target area and a surrounding object,
performing mask processing on the area occupied by the transparent portion of the image acquired by the imaging device;
Configured to detect an object by a program using deep learning based on the masked image ,
The foreign matter detection system, wherein the boundary determination unit is an object sensor capable of detecting the presence or absence of an object in a boundary area between the target area and the external space. 2. The foreign matter detection system according to claim 1, wherein said object sensor is configured to detect an irradiation wave irradiated along a plane formed by said transmission section. an imaging device that acquires an image within a target region;
an image processing unit that stores a program for detecting an object in an image acquired by the imaging device using deep learning;
A boundary determination unit that determines the positional relationship between a transmission unit that can visually recognize the outside from the inside of the target area and a surrounding object,
performing mask processing on the area occupied by the transparent portion of the image acquired by the imaging device;
Configured to detect an object by a program using deep learning based on the masked image,
The boundary determining unit is configured to determine the image obtained by the imaging device based on the positional relationship within the same image between the position of the object in the image obtained by the imaging device and the area occupied by the transparent portion in the image. A foreign object detection system configured to determine whether an object detected therein is inside or outside the region of interest. 4. The foreign matter detection system according to claim 1, wherein said mask processing is performed by said imaging device. 4. The foreign matter detection system according to claim 1, wherein said mask processing is performed by said image processing unit. The foreign object detection system according to any one of claims 1 to 5 , wherein the target area is an entry/exit space of a mechanical parking lot. an image acquisition step of acquiring an image within the region of interest;
a masking step of masking an area occupied by a transmissive portion that allows the outside to be seen from inside the target area, on the image obtained in the image obtaining step;
A first object detection step of detecting an object by a program using deep learning based on the image that has undergone the mask processing step;
and a second object detection step of detecting presence/absence of an object in the boundary area of the target area with an object sensor . an image acquisition step of acquiring an image within the region of interest;
An object detection step of detecting an object by a program using deep learning based on the image acquired in the image acquisition step;
Based on the positional relationship in the same image between the position of the object in the image and the area occupied in the image by the transparent portion where the outside can be visually recognized from the inside of the target area, the object detected in the image is the target. and a position determining step for determining whether the foreign object is inside or outside the area. 9. The foreign object detection method according to claim 7 , wherein said target area is an entry/exit space of a mechanical parking lot.

Images