JP7420879B2 - Device and method for extracting human presence area - Google Patents

Description

The present invention relates to an apparatus and method for extracting a human presence area used in a garbage pit fall warning system for warning of the danger of a person falling from a platform of a garbage disposal facility into a garbage pit.

Garbage brought into the garbage processing facility is thrown into the ``garbage pit'' through a large door called the ``garbage input door.'' When cleaning work is carried out with the garbage disposal door open, there is a risk that the worker may accidentally fall into the garbage pit. A large crane is operating inside the garbage pit, and there is a possibility that toxic gas may accumulate inside the garbage pit, so if a worker falls, the crane etc. must be stopped in an emergency and rescued as soon as possible. . Conventionally, a crane operator visually detected whether or not a crane had fallen.

Patent Document 1 discloses that the past image data obtained by capturing images of a garbage pit where waste is stored is labeled with the type of waste using a trained identification algorithm. This device is equipped with a type identification unit that inputs new image data of the garbage pit and identifies the type of waste stored in the garbage pit. Examples of types of objects to be identified other than waste include a carry-in door, a worker, and a carry-in vehicle, and furthermore, it is described that falling of a worker or a transport vehicle into a garbage pit is detected.

Patent Document 2 is a system that acquires a garbage image taken of garbage that is being transported to a garbage incineration facility and is being dumped into a garbage pit, and detects items unsuitable for incineration from the garbage image. For example, the system detects garbage that is not identified as garbage bags as unsuitable for incineration, and solves the problem of illegal importation of unsuitable for incineration.

JP2020-038058 JP2019-190805

In Patent Document 1, it is determined whether the worker has fallen from the image stored in the garbage pit, and therefore, the structure is not designed to predict and warn the danger of falling before the worker falls.
Further, Patent Document 2 does not mention anything about a worker falling into a garbage pit.
In addition, in recent years, automatic operation of cranes has been implemented to reduce the number of workers, and there is an increasing demand for measures to prevent workers from falling in case there is no crane operator who can visually see inside the garbage pit.

In view of the above circumstances, the present invention provides an apparatus and method for extracting a human presence area used in a garbage pit fall warning device that can warn of the danger of a person falling from a platform of a garbage disposal facility into a garbage pit. The purpose is to provide.

The garbage pit fall warning device of the present invention detects a dumping area where garbage is dumped into a garbage pit (also referred to as a receiving pit) from a loading area image (for example, a still image or a video) taken from above a platform. It has an image processing unit that extracts an area where a person exists.

The image processing unit includes:
a first overlap amount calculation unit that calculates a first amount of overlap between a first image area preset in the input area image and the human presence area; and/or
It has a second image area preset in a location different from the first image area of the input area image, and a second overlap amount calculation unit that calculates a second overlap amount with the person presence area. You can.

The garbage pit fall warning device is
a first alarm output unit that outputs a first alarm if the first overlap amount is equal to or greater than a first predetermined amount;
a second alarm output unit that outputs a second alarm if the first overlap amount is greater than or equal to a second predetermined amount that is larger than the first predetermined amount; and/or
The device may further include a third alarm output unit that outputs a third alarm if the second amount of overlap is equal to or greater than a third predetermined amount.
The third predetermined amount may be smaller than the first predetermined amount.

The garbage pit fall warning device is
When the platform is viewed from above, it overlaps in the vertical direction (e.g., vertically from the floor surface) with a delivery vehicle (e.g., a garbage truck, a transportation vehicle within the facility, etc.) placed in the input area. You may have an imaging device which images the input area image and is installed so that it does not match.

The garbage pit fall warning device is
First teacher data in which a human presence area is labeled in an image of a person taken from above at a location different from the platform, and second teacher data in which a person presence area is labeled in an image of a person taken from above in the platform. It may include a storage unit that stores an image processing program (person presence area extraction program) generated by intelligent information processing technology using at least one of the data.

The image processing program (person presence area extraction program) may be generated by intelligent information processing technology using both the first teacher data and the second teacher data.

Another invention of the garbage pit fall warning method is as follows:
The method includes an image processing step of extracting a human presence area where a person is present from a loading area image taken from above of a loading area where garbage is loaded from a platform to a garbage pit.

The image processing step includes:
a first overlap amount calculation substep of calculating a first overlap amount between a first image region preset in the input area image and the human presence area; and/or
a substep of calculating a second overlap amount between a second image area preset in a location different from the first image area of the input area image and the human presence area; Good too.
The image processing step includes:
First teacher data in which a human presence area is labeled on an image of a person taken from information at a location different from the platform, and second teacher data in which a person presence area is labeled in an image of a person taken from above within the platform. The human presence area may be extracted from the input area image by executing an image processing program (human presence area extraction program) generated by intelligent information processing technology using at least one of the data.
In the first overlap amount calculation substep, the first overlap amount may be calculated by executing an image processing program (person presence area extraction program).
In the second double deviation amount calculation substep, the second double deviation amount may be calculated by executing an image processing program (person presence area extraction program).

The image processing program (person presence area extraction program) may be generated by intelligent information processing technology using both the first teacher data and the second teacher data.

The garbage pit fall warning method is as follows:
a first alarm output step of outputting a first alarm if the first overlap amount is equal to or greater than a first predetermined amount;
a second alarm step of outputting a second alarm if the first overlap amount is greater than or equal to a second predetermined amount that is larger than the first predetermined amount; and/or
The method may further include a third alarm step of outputting a third alarm if the second amount is equal to or greater than a third predetermined amount.
The third predetermined amount may be smaller than the first predetermined amount.

The garbage pit fall warning method is as follows:
Imaging that captures an image of the loading area with an imaging device installed so as not to overlap in the vertical direction (vertical direction) with a loading vehicle placed in the loading area when the platform is viewed from above May include steps.

Another invention of the garbage pit fall warning program is
This is a program that implements the above garbage pit fall warning method using at least one processor.
A computer-readable recording medium storing computer instructions according to another aspect of the invention is a computer-readable recording medium that implements the steps of the fall warning program described above when the computer instructions are executed by a processor.

Separately from the storage unit, the garbage pit fall warning device has a device for accessing and executing an external device (garbage treatment facility server, cloud server, database, etc.) that stores the image processing program (person presence area extraction program). It may include hardware and programs.
The image processing program (person presence area extraction program) may be retrained.

Examples of "intelligent information processing technology" include machine learning, deep learning, reinforcement learning, and deep reinforcement learning.
Algorithms for machine learning, deep learning, reinforcement learning, and deep reinforcement learning are not particularly limited, and conventional algorithms may be used. As supervised learning, various algorithms such as linear regression, generalized linear model, support vector regression, Gaussian process regression, ensemble method, decision tree, neural network, support vector machine, discriminant analysis, naive Bayes, and nearest neighbor method can be used. May be adopted.

Each element of the garbage pit warning device may include a memory, a processor, an information processing device (for example, a computer, a server) having a software program, a dedicated circuit, firmware, and the like. The information processing device may be on-premises or in the cloud, or a combination of both.

It is a figure showing an example of the constituent elements of a garbage pit fall warning device. It is an example of a top view of a platform. It is an example of a top view showing an imaging area. It is an example of a top view showing the first image area of another embodiment. It is an example of the top view which shows the 1st and 2nd image area|region of another embodiment.

(Embodiment 1)
FIG. 1 shows an example of the components of the garbage pit fall warning device 1 of the first embodiment. FIG. 2 shows an example of a plan view of the platform 60. FIG. 3 shows an example of a plan view of the imaging area.

The imaging device 30 (31, 32, 33, 34) is vertically (vertically) connected to the loading vehicle 5 (51, 52, 53) arranged in the loading area when the platform 60 is viewed from above. are installed so that they do not overlap.
In FIG. 2, imaging devices 31, 32, 33, and 34 each capture input area images (E31, E32, E33, and E34). Each imaging device may be installed, for example, to take an image from the ceiling of the platform 60 toward the floor, and is higher than the top surface of the body of the delivery vehicle 5 (51, 52, 53), It may be installed at a position where images are taken toward the input port from a position below the height. By installing each imaging device so that it does not overlap (does not shade) each other with the delivery vehicle 5 when viewed from above, blind spots where the person 7 may be hidden behind the delivery vehicle 5 are reduced, and the loading vehicle 5 is A person falling through the gap between the vehicle 65 and the delivery vehicle 5 can be reliably imaged.
One or more imaging devices 30 may be provided per input port. In the case of a plurality of images, at least one may take a still image or a moving image. If there are more than one, blind spots can be further reduced.
The imaging device 30 may be installed so as to capture an image in a viewing angle range of 35 degrees to 60 degrees from the upper front to the lower rear of the delivery vehicle 5.
The carry-in vehicle 5 may be either a push-out type in which the garbage inside the body is pushed out by a pushing means or a dump-type in which the garbage is discharged by tilting the body. Each imaging device is installed so that a blind spot does not occur even if the body is tilted during garbage discharge.

In FIG. 3, an imaging area E32 of the imaging device 32 is shown by a chain line, and an imaging area E33 of the imaging device 33 is shown by a chain double-dashed line.
The imaging device 32 images the person 7 (person presence areas 721, 722, 723), but the person 731 behind the person 731 is not imaged by the delivery vehicle 52. On the other hand, the person 731 is imaged by the imaging device 33, and the person 7 (person presence areas 732 and 733) is also imaged.

The image processing unit 10 extracts a human presence area (721, 722, 723, 731, 732, 733). The image processing unit 10 sets (draws) a rectangular frame in the extracted human presence area. The image processing unit 10 may simultaneously display the numerical value of the reliability of extraction in a rectangular frame.
The storage unit 40 stores first teacher data in which an image of a person taken from above at a location different from the platform 60 is labeled with a person presence area, and a person presence area is labeled in an image of a person taken from above within the platform 60. An image processing program (person presence area extraction program) generated by intelligent information processing technology using at least one of the labeled second teacher data is stored.
The first teacher data is, for example, an image captured from the front of a person in a standing position, or an image that can be collected from the Internet (images that are not captured by the same imaging device may be used).
The second training data is, for example, present in an image of a person present on the platform 60, an image of a person present in the input area, a first image area and/or a second image area in the input area image (overlapping ) is an image of a person.
The image processing unit 10 extracts a human presence area using an image processing program (human presence area extraction program).

The image processing unit 10 may display a frame (for example, a rectangular frame) around the extracted human presence area and a numerical value of the reliability of extraction on the monitor.
The image processing unit 10 determines whether the extraction reliability (0 to 100%) of the human presence area is a person in the extraction judgment of the human presence area by the image processing program (person presence area extraction program) (in determining whether it is a person or not). It has a person presence determination unit 101 that determines that the area is a person presence area (person) if it is equal to or higher than a determination criterion threshold value (for example, 60% or more, 75% or more, 80% or more, etc.).
If the size (for example, rectangle, circular, etc.) of the human presence area extracted by the image processing program (human presence area extraction program) is outside the size threshold range (upper limit, lower limit) However, it may be determined that the person is not a person. Since the area size of a person is smaller than the delivery vehicle and larger than equipment such as a shovel, misrecognition can be reduced by setting a size threshold range (upper limit, lower limit). For example, if the rectangular size of the extracted area is larger or smaller than the size threshold range, it is determined that the area is not a human presence area.
The image processing unit 10 includes a threshold setting unit 102 that variably sets the range (upper limit, lower limit) of the person judgment criterion threshold and/or the size threshold. If there are many false positives, you can change the threshold.
The image processing unit 10 includes a first detection unit 103 that determines that a human presence area has been detected when a human presence area (person) is extracted in one frame in each of continuous still images (frames) forming a moving image. has.
The image processing unit 10 calculates the number of judgment criteria (for example, the above-mentioned The second detection unit 104 determines that a human presence area has been detected when a person presence area (person) is extracted from frames (a number smaller than a predetermined number). Since "human movement" occurs continuously over a certain period of time, it is preferable to detect a set of a predetermined number of frames (frame block units) rather than one frame unit because it is more efficient.
The image processing unit 10 includes a determination criterion setting unit 105 that variably sets a predetermined number of frames of a frame block and a determination criterion number smaller than the predetermined number of frames. This allows for variable settings depending on, for example, the situation of the garbage pit facility, the platform, the delivery vehicle, the clothing of the worker, etc.

Moreover, as another embodiment, the image processing program (person presence area extraction program) may be generated by intelligent information processing technology using both the first teacher data and the second teacher data.

The image processing section 10 includes a first overlap amount calculation section 11 and a second overlap amount calculation section 12.
The first overlap amount calculation unit 11 calculates the first overlap amount, which is the overlap between the first image area E01 preset in the input area image and the human presence area (the area surrounded by the rectangular frame). .
The second double deflection amount calculation unit 12 calculates a second image area E02 that is set in advance at a location different from the first image area E01 of the input area image (a location distal from the input port), and a rectangular frame of the human presence area. Calculate the second double overlap amount, which is the overlap with the area (area surrounded by ).
In FIG. 3, the first image area E01 may be set to include a garbage pit from the garbage input port. The first image area E01 may include an image of the rear of the loading vehicle 5, a wheel stopper 61, and an image of the input door 65 of the input slot. The first image area E01 may include an image of the side or front of the delivery vehicle 5. Further, the second image area E02 may be set to be adjacent to the first image area E01 and include the platform 60 side from the input port.

The first alarm output unit 21 outputs a first alarm if the first overlap amount is greater than or equal to the first predetermined amount. The second alarm output unit 22 outputs a second alarm if the first overlap amount is greater than or equal to a second predetermined amount that is larger than the first predetermined amount. The third alarm output unit 23 outputs a third alarm if the second double amount is equal to or greater than a third predetermined amount.
The output may be audio from the speaker 25.
The first predetermined amount may be, for example, a value in a range of 25% or more and less than 35% of the area of the human presence area. For example, the area 732 where a person is present on the wheel stopper 61 corresponds to this area. The first warning may be a direct warning (voice, sound, light, etc.) such as "Be careful of falling", for example.
The second predetermined amount may be, for example, a value in a range of 35% or more and 100% or less of the area of the human presence area. For example, this applies to areas 722 and 733 where people are about to fall into the slot. The second alarm may be, for example, a highly urgent alarm, an automatic stop of the crane, or a notification to the central control room.
The third predetermined amount may be, for example, a value of 5% or more and less than 25% of the area of the human presence area. For example, this applies to the human presence areas 721 and 723. The third warning may be, for example, a simple warning such as "Wear your safety belt." The third predetermined amount may be smaller than the first predetermined amount, for example.

(Example)
In FIG. 4A, the first image area E01 is set in the direction from the starting point of the input slope 621 to the garbage pit 62.
When the recognition rectangular frame of the person presence area 750 overlaps the first image area E01 by 30% to 49%, a first alarm is output. When the recognition rectangular frame of the person presence area 751 overlaps the first image area E01 by 50% to 100%, a second alarm (emergency stop) is output.

In FIG. 4B, the first image area E01 is set in the direction from the starting point of the input slope 621 to the garbage pit 62, and the second image area E02 is set in the direction from the start point of the input slope 621 to the platform 60.
When the recognition rectangular frame of the person presence area 760 overlaps with the second image area E02 by 50% to 100%, a third warning (weak warning) is output.
When the recognition rectangular frame of the human presence area overlaps the first image area E01 by 30% to 49%, a first warning (strong warning) is output. When the recognition rectangular frame of the person presence area 761 overlaps the first image area E01 by 50% to 100%, a second alarm (emergency stop) is output.

In another embodiment, the image processing unit 10 performs edge processing on the image as a process for extracting a person presence area, and performs edge processing on the image to extract the person 7 (e.g. head, shoulders, arms, helmet, hat, By extracting and analyzing the human presence area (surrounded by a rectangular frame) and other structures (carrying vehicle 5, wheel chocks 61, garbage pit 62, input slope 621, input door 65, (such as the floor of the platform 60).
Further, the alarm output means may be, for example, a "display device", and examples thereof include a liquid crystal monitor, an organic EL monitor, a CRT monitor, a smartphone, a tablet, a general-purpose personal computer monitor, and the like.
Furthermore, the alarm output means may be, for example, a voice output means, a sound output means, a lighting means, or the like.

(Embodiment 2)
The garbage pit fall warning method of the second embodiment includes an image processing step of extracting a person presence area where the person 7 is present from a loading area image taken from above of the loading area where garbage is loaded from the platform 60 to the garbage pit 62. include.
The image processing step includes:
a first overlap amount calculation substep of calculating a first overlap amount between a first image region preset in the input area image and the human presence area; and/or
a substep of calculating a second overlap amount between a second image area preset in a location different from the first image area of the input area image and the human presence area; Good too.
The image processing step includes:
First teacher data in which a human presence area is labeled on an image of a person taken from information at a location different from the platform, and second teacher data in which a person presence area is labeled in an image of a person taken from above within the platform. The method may include the step of executing an image processing program (person presence area extraction program) generated by intelligent information processing technology using at least one of the data to extract the person presence area from the input area image. good.

In the step of extracting the human presence region, the reliability of extraction of the human presence region (0 to 100%) is set to a human judgment criterion threshold (for example, 60%) in the extraction judgment of the human presence region (in determining whether or not it is a person). % or more, 75% or more, 80% or more), it may include a person presence determination step of determining that the area is a person presence area (person).
The human presence determination step is performed when the size (e.g., rectangular, circular, etc.) of the human presence area extracted by the image processing program (human presence area extraction program) is outside a size threshold range (upper limit, lower limit). However, it may be determined that the person is not a person.
The image processing step includes a first detection step of determining that a human presence area has been detected when a human presence area (person) is extracted in one frame in each of continuous still images (frames) forming a moving image. May contain.
In the image processing step, the number of judgment criteria (for example, the above-mentioned The method may include a second detection step of determining that the human presence area has been detected when the human presence area (person) is extracted from frames (a number smaller than a predetermined number).
The image processing step may include a threshold setting step of variably setting a range (upper limit, lower limit) of a person judgment criterion threshold and/or a size threshold.
The image processing step may include a determination criterion setting step of variably setting a predetermined number of frames of the frame block and a determination criterion number smaller than the predetermined number of frames.

In the first overlap amount calculation substep, the first overlap amount may be calculated by executing an image processing program (person presence area extraction program).
In the second double deviation amount calculation substep, the second double deviation amount may be calculated by executing an image processing program (person presence area extraction program).
The garbage pit fall warning method is as follows:
a first alarm output step of outputting a first alarm if the first overlap amount is equal to or greater than a first predetermined amount;
a second alarm step of outputting a second alarm if the first overlap amount is greater than or equal to a second predetermined amount that is larger than the first predetermined amount; and/or
The method may further include a third alarm step of outputting a third alarm if the second amount is equal to or greater than a third predetermined amount.
The garbage pit fall warning method is as follows:
an imaging step of capturing an image of the loading area, which is installed so as not to overlap in the vertical direction (vertical direction) with a loading vehicle placed in the loading area when the platform is viewed from above; It's okay to stay.
The image processing program (person presence area extraction program) may be generated by intelligent information processing technology using both first teacher data and second teacher data.

(Embodiment 3)
The garbage pit fall warning program of Embodiment 3 is as follows:
This is a program that implements the above garbage pit fall warning method using at least one processor.

(Embodiment 4)
The computer-readable recording medium in which the computer instructions of the fourth embodiment are stored is a computer-readable recording medium that implements the steps of the fall warning program described above when the computer instructions are executed by a processor.

1 Garbage pit fall alarm 10 Image processing device 101 Person presence determination section 102 Threshold value setting section 103 First detection section 104 Second detection section 105 Judgment standard setting section 11 First overlap amount calculation section 12 Second overlap amount calculation section 21 First Alarm output section 22 Second alarm output section 23 Third alarm output section 25 Speakers 30, 31, 32, 33, 34 Imaging device 40 Storage sections 5, 51, 52, 53 Carrying vehicle 60 Platform 61 Wheel chock 62 Garbage pit 621 Injection Opening inclined surface 65 Loading door 7 Person E01 First image area E02 Second image area

Claims (4)

A device for extracting a human presence area in order to warn of the danger of a person falling into a garbage pit,
First training data in which a human presence area is labeled in an image of a person taken from above at a location different from the platform, and second training data in which a person presence area is labeled in an image of a person taken from above within the platform. Using a human presence area extraction program generated by intelligent information processing technology using both , it is possible to determine whether a person is present in the input area where garbage is thrown into the garbage pit from an image of the input area taken from above the platform. A device that includes an image processing unit that extracts a human presence area. A method for extracting a human presence area to warn of the danger of a person falling into a garbage pit, the method comprising:
First training data in which a human presence area is labeled in an image of a person taken from above at a location different from the platform, and second training data in which a person presence area is labeled in an image of a person taken from above within the platform. Using a human presence area extraction program generated by intelligent information processing technology using both , it is possible to determine whether a person is present in the input area where garbage is thrown into the garbage pit from an image of the input area taken from above the platform. A method comprising an image processing step of extracting a human presence area. A program for implementing the method of claim 2 by means of at least one processor. A computer-readable storage medium having computer instructions stored thereon, the computer-readable storage medium comprising:
A computer-readable recording medium, wherein the computer instructions are executed by a processor to implement the steps of the program according to claim 3.