JP6940907B1 - Peripheral monitoring device for work machines - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a peripheral monitoring device capable of monitoring the surroundings of a work machine over a wide range with a low cost and as few cameras as possible, and capable of suppressing contact between the work machine and a person around the work machine. A peripheral monitoring device 100 for a work machine according to the present disclosure is determined based on a photographing device 200 including one or more hemispherical cameras 220 and photographed image data representing a photographed peripheral image of the working machine 10. It includes a processor 303 that executes processing, and an alarm device 400 that outputs a predetermined alarm signal and / and a predetermined alarm. Then, the processor 303 extracts a person image from the photographed image data, and measures the distance between the work machine 10 and the person around it based on the person image and the installation parameters of the photographing device 200. Then, based on the measured distance, the output of the alarm signal and / and the alarm by the alarm device 400 is controlled. [Selection diagram] Fig. 1

Description

The present invention relates to a peripheral monitoring device for a work machine.

At sites where work machines such as excavators and forklifts are used, contact accidents between the work machines and people around them are prevented. For example, a work machine equipped with a mirror or a camera has been put into practical use so that an operator can visually recognize an obstacle existing around the vehicle body of the work machine. Then, there is known a system that displays an image taken by a camera on a display in the driver's cab of a work machine.

Further, as a technique for measuring the distance to an object existing around the work machine based on an image taken by a camera, Patent Document 1 describes an upper swivel body in a shovel having a lower traveling body and an upper swivel body. A technique for arranging a plurality of stereo cameras is disclosed. In this technology, a stereo camera acquires distance information of an object existing around the excavator.

On the other hand, there is known a technique of installing a wide viewing angle camera in a work machine so as to obtain a field of view in a range that is a blind spot for an operator in the driver's cab of the work machine. For example, Patent Document 2 discloses a work machine surrounding monitoring method and system using a fisheye lens that can be attached to a work machine to immediately grasp the state of the entire blind spot. Further, Patent Document 3 discloses a construction machine capable of easily and appropriately giving guidance at the time of reverse movement by using only one camera. In this construction machine, a wide viewing angle camera such as an omnidirectional camera or a fisheye camera is provided on the upper swing body of the hydraulic excavator.

Japanese Unexamined Patent Publication No. 2020-127217 Japanese Unexamined Patent Publication No. 2012-172350 Japanese Unexamined Patent Publication No. 2008-240362

In order to suppress contact between work machines such as excavators and forklifts and people around them, it is necessary to monitor the surroundings of the work machines over a wide area. Here, in the work machines described in Patent Documents 2 and 3, it is possible to acquire an image of the surroundings of the work machine by using a wide viewing angle camera. In other words, these are intended to allow the operator to visually recognize obstacles existing around the vehicle body of the work machine, but the operator can see the work machine while checking the image displayed on the display or the like in the driver's cab. It must be operated so that it does not come into contact with the people around it.

On the other hand, according to the technique described in Patent Document 1, although the distance information of the objects existing around the excavator can be acquired based on the image taken by the camera, the camera installed in the excavator is a stereo camera. Therefore, in order to monitor the surroundings of the work machine over a wide area, it is necessary to arrange a plurality of stereo cameras. Therefore, cost and space can be an issue.

An object of the present disclosure is to provide a peripheral monitoring device capable of monitoring the surroundings of a work machine over a wide range at low cost and with as few cameras as possible, and capable of suppressing contact between the work machine and people around it. There is.

The peripheral monitoring device of the work machine of the present disclosure is installed in the work machine and acquires a photographing device for photographing the surroundings of the work machine and photographed image data representing the surrounding image of the work machine photographed by the photographing device. A processor that executes a predetermined process based on the captured image data, and an alarm device that outputs a predetermined alarm signal and / or a predetermined alarm are provided. Here, the photographing device includes one or more hemispherical cameras, and is configured such that the optical axis of the hemispherical cameras is arranged at a predetermined angle with respect to the ground. Then, the processor extracts the person image from the captured image data, and based on the extracted person image and the installation parameters of the photographing device, the working machine and the surrounding person The alarm signal and / and the output of the alarm are controlled by the alarm device based on the distance measurement process for measuring the distance and the distance between the work machine and a person around it measured by the distance measurement process. The alert control process to be executed and the execution are performed.

Here, the above-mentioned work machine is a construction machine such as an excavator or a machine equipped with a work machine (forklift or the like). Then, in the peripheral monitoring device of the present disclosure installed on such a work machine, a photographing device is arranged on the work machine. This photographing device is arranged so that the optical axis of the hemispherical camera forms a predetermined angle with respect to the ground. Here, the above-mentioned predetermined angle is defined as an angle at which a person near the ground can be photographed by the photographing device, and is, for example, an angle oriented substantially perpendicular to the ground. The peripheral monitoring device for the work machine of the present disclosure is realized by a simple configuration such as a photographing device, an image processing device, and an alarm device. Can be easily installed. Further, since the photographing device is composed of a hemispherical camera capable of photographing the surroundings of the work machine over a wide range, for example, the periphery of the work machine is compared with the case where the photographing device is composed of a plurality of stereo cameras. The cost of monitoring the camera can be reduced. Further, in such a peripheral monitoring device, distance measurement processing is executed based on the person image extracted by the extraction processing and the installation parameters of the photographing device, and the alarm device is issued based on the result of the distance measurement processing. The output of the alarm signal and / and the alarm is controlled. The alarm output by the alarm device is, for example, a warning light or a warning sound. As a result, contact between the work machine and a person around it can be suppressed. In the alarm control process, when the distance between the work machine and a person around it becomes equal to or less than a predetermined value, the processor causes the work machine and the people around it to move from the alarm device. A signal for instructing a predetermined avoidance operation for avoiding contact may be output. Here, the above-mentioned avoidance operation is an operation in which the operation of the work machine is decelerated, an operation in which the operation of the work machine is stopped, and the like. In such an alarm control process, the signal from the alarm device is input to the control device that controls the operation of the work machine, so that the work machine automatically performs an avoidance operation regardless of the operator's operation. It will be. According to this, it is possible to perform the avoidance operation of the work machine without the operation operation by the operator, and it is possible to more preferably avoid the contact between the work machine and the person around it.

Then, in the peripheral monitoring device of the work machine, in the distance measurement process, the processor acquires the foot coordinates, which are the two-dimensional coordinates of the foot of the person image in the captured image data, and is based on the foot coordinates. Then, a foot angle, which is a predetermined angle between the feet of a person around the work machine and the photographing device, is calculated, and based on the calculated foot angle and the installation height of the photographing device, The distance between the work machine and a person around it may be measured. According to this, the hemispherical camera enables accurate distance measurement processing at low cost and with as few cameras as possible. According to the distance measurement based on the foot coordinates, the foot coordinates can be obtained relatively accurately, so that the measurement accuracy of the distance can be made as high as possible. Further, in the distance measuring process, the processor acquires an image of the head of the person image in the captured image data, and based on the size of the image of the head, the working machine and the person around it The distance may be measured. Even with this, the hemispherical camera can perform distance measurement processing accurately at low cost and with as few cameras as possible. According to the distance measurement based on the size of the head image, the photographed image data does not include the feet of surrounding persons, or the person is climbing to a high place. Even when not on the ground, the distance between the work machine and the people around it can be measured.

Further, in the peripheral monitoring device of the work machine, in the extraction process, the processor has an input layer that accepts input of predetermined image data and an intermediate layer that extracts a feature amount representing a person from the input image data. The pre-learning model, which is a neural network model having an output layer that outputs an identification result based on the feature amount and is constructed by performing training using data including an image representing a person, is described above. The person image may be extracted by inputting the captured image data. According to this, by using the pre-learning model, it is possible to accurately extract a person image from the captured image data.

In the peripheral monitoring device of the work machine described above, the processor generates a captured image histogram showing the frequency of pixel values included in the captured image data, and is based on the smoothed histogram obtained by smoothing the captured image histogram. An image sharpening process that sharpens the captured image data by synthesizing the acquired smoothed image data and the captured image data at a predetermined ratio may be further executed. According to this, it is possible to extract a person image based on the photographed image data which is sharpened according to the environment. Therefore, the accuracy of extracting a person image from the captured image data can be made as high as possible.

Further, in the peripheral monitoring device of the work machine described above, the photographing device may be a device that automatically controls the exposure of the hemispherical camera to a predetermined reference value according to the brightness of the subject. In this case, the photographing apparatus further acquires the brightness of the ground around the work machine, and when the brightness of the ground is equal to or higher than the predetermined first brightness, the exposure of the hemispherical camera is more than the reference value. When the brightness of the ground is lower than the first brightness and is equal to or less than the predetermined second brightness, the exposure of the hemispherical camera is corrected to be smaller than the reference value. May be good. On the other hand, when the brightness of the ground is higher than the second brightness and lower than the first brightness, the photographing device does not have to correct the exposure of the hemispherical camera. According to this, since the person around the work machine can be photographed with an appropriate brightness, the accuracy of extracting the person image from the photographed image data can be made as high as possible.

According to the present disclosure, there is provided a peripheral monitoring device capable of monitoring the surroundings of a work machine over a wide range at low cost and with as few cameras as possible, and capable of suppressing contact between the work machine and people around it. be able to.

It is a figure which shows the schematic structure of the peripheral monitoring device of the work machine in 1st Embodiment. It is a flowchart which shows the processing flow which the control part performs in order to avoid the contact between a work machine and a person around it in the peripheral monitoring apparatus which concerns on 1st Embodiment. It is a figure for demonstrating the area excluded from photographed image data. It is a figure for demonstrating the identification result obtained from the input to the pre-learning model in 1st Embodiment, and the neural network which constitutes the pre-learning model. It is a figure for demonstrating photographed image data and a person image displayed on a display device. It is a figure for demonstrating the relationship between the foot angle, the installation height of the photographing apparatus, and the distance with the surrounding person. It is a flowchart which shows the processing flow performed by the distance measurement processing unit in the foot distance measurement which is one of the distance measurement processing which concerns on 1st Embodiment. It is a figure for demonstrating the subject projected on the sensor of the camera together with the internal parameters of a camera. It is a 1st flowchart which shows the processing flow which the control part performs in order to avoid the contact between a work machine and a person around it in the peripheral monitoring apparatus which concerns on modification 2 of 1st Embodiment. 2 is a second flowchart showing a processing flow performed by a control unit in order to avoid contact between a work machine and a person around the work machine in the peripheral monitoring device according to the second modification of the first embodiment. It is a flowchart which shows the process flow which a control part performs in the image sharpening process which concerns on 2nd Embodiment. It is a flowchart which shows the flow of the exposure compensation of the hemisphere camera which concerns on 3rd Embodiment. It is a figure which shows the modification of the schematic structure of the peripheral monitoring device of a work machine in this disclosure.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The configurations of the following embodiments are exemplary, and the present disclosure is not limited to the configurations of the embodiments.

<First Embodiment>
The outline of the peripheral monitoring device for the work machine according to the first embodiment will be described with reference to FIG. FIG. 1 is a diagram showing a schematic configuration of a peripheral monitoring device for a work machine according to the present embodiment. The peripheral monitoring device 100 for a work machine according to the present embodiment is a device installed on the work machine 10 for monitoring the periphery of the work machine 10. Here, the work machine 10 is a construction machine such as an excavator or a machine equipped with a work machine (forklift or the like). The peripheral monitoring device 100 includes a photographing device 200, an image processing device 300, and a reporting device 400.

The photographing device 200 is a device installed on the work machine 10 and photographs the surroundings of the work machine 10, and has a function of accepting an input of an image such as a still image or a moving image. Specifically, the charged-coupled devices It is realized by a camera using an image sensor such as (CCD), Metal-oxide-semiconductor (MOS) or Complementary Metal-Oxide-Semiconductor (CMOS). The photographing apparatus 200 of the present disclosure includes one or more hemispherical cameras 220, and is configured such that the optical axis of the hemispherical camera 220 is arranged at a predetermined angle with respect to the ground. Here, in the present embodiment, as shown in FIG. 1A, the hemispherical camera 220 is arranged on the upper surface of the upper swivel body 11 of the work machine 10 (for example, a mini excavator), and the hemispherical camera 220 is arranged. The optical axis L2 is directed downward at a predetermined angle with respect to the parallel line L1 with the ground. This angle is defined as an angle at which a person near the ground can be photographed by the photographing device 200. For example, as shown in FIG. 1A, the hemispherical camera 220 is arranged on the upper surface of the upper swivel body 11. In this case, the optical axis L2 of the hemispherical camera 220 is oriented substantially perpendicular to the ground. However, there is no intention of limiting the mode in which the hemispherical camera 220 is oriented substantially perpendicular to the ground in this way, and the hemispherical camera 220 may be tilted in any direction of front, back, left, and right. Then, in the present embodiment, as shown in FIG. 1B, one hemispherical camera 220 is arranged on the upper surface of the driver's cab of the work machine 10. By installing the hemispherical camera 220 on the work machine 10 in this way, it is possible to take a picture of the periphery of the work machine 10 over a wide area without losing the field of view with as few cameras as possible. By arranging the hemispherical camera 220 on the upper surface of the driver's cab, the influence of the blind spot that may occur due to the presence of the driver's cab can be suppressed as much as possible. It should be noted that there is no intention of limiting the photographing apparatus of the present disclosure to a configuration including one hemispherical camera. For example, when the work machine 10 is a large construction machine, one hemispherical camera may be arranged in front of and behind the work machine 10.

The image processing device 300 acquires photographed image data representing the surrounding image of the work machine 10 photographed by the photographing device 200, and executes a predetermined process based on the photographed image data.

Here, the image processing device 300 may be any electronic device as long as it has processing capacity for arithmetic processing such as data acquisition, generation, and update, and processing processing, and is, for example, a computer. That is, the image processing device 300 can be configured as a computer having a processor such as a CPU or GPU, a main storage device such as RAM or ROM, and an auxiliary storage device such as EPROM, a hard disk drive, or removable media. The removable media may be, for example, a USB memory or a disc recording medium such as a CD or DVD. The auxiliary storage device stores an operating system (OS), various programs, various tables, and the like.

The alarm device 400 is a device that outputs a predetermined alarm signal and / and a predetermined alarm. Here, the predetermined alarm is a warning light, a warning sound, or the like. In the present embodiment, as shown in FIGS. 1 (a) and 1 (b), as the alarm device 400, for example, a patrol light (registered trademark) may be arranged on the upper surface of the upper swing body 11. Then, the peripheral monitoring device 100 can give a warning to a person around the work machine 10 by, for example, a warning light emitted by Patrite (registered trademark). In addition, the operator of the work machine 10 can be made aware of the approach of a person to the work machine 10. The alarm device 400 may include, for example, a patrol light (registered trademark) and a speaker. In this case, the peripheral monitoring device 100 can give a warning by the warning light and the warning sound.

Further, the peripheral monitoring device 100 may further include a display device 500. Here, the display device 500 is a device configured to be able to display the captured image data taken by the photographing device 200 and / or predetermined data acquired by processing based on the captured image data. Such a display device 500 is provided, for example, in the cab of the work machine 10. As a result, the operator of the work machine 10 can visually recognize the obstacles existing around the vehicle body of the work machine 10 through the display device 500.

As described above, the peripheral monitoring device 100 of the work machine of the present disclosure is realized by a simple configuration such as a photographing device 200, an image processing device 300, and an alarm device 400, and these are arranged in the existing work machine 10. The peripheral monitoring device 100 can be easily installed on the work machine 10 simply by doing so. Further, as described above, since the photographing device 200 is composed of the hemispherical camera 220, the surroundings of the work machine 10 can be monitored over a wide range at low cost and with as few cameras as possible.

Next, the components of the image processing apparatus 300 will be described in detail based on FIG. 1 (c). FIG. 1C is a diagram showing in more detail the components of the image processing device 300 included in the peripheral monitoring device 100 in the first embodiment.

The image processing device 300 has a storage unit 302 and a control unit 303 as functional units, loads the program stored in the auxiliary storage device into the work area of the main storage device, executes the program, and executes each function through the execution of the program. By controlling the units and the like, it is possible to realize each function that meets a predetermined purpose in each functional unit. However, some or all of the functions may be realized by hardware circuits such as ASICs and FPGAs.

The storage unit 302 includes a main storage device and an auxiliary storage device. The main storage device is a memory in which a program executed by the control unit 303 and data used by the control program are expanded. The auxiliary storage device is a device that stores a program executed by the control unit 303 and data used by the control program.

Further, the storage unit 302 may store the pre-learning model described later. This pre-learning model can be used for the extraction process described later. Here, when the function of the image processing device 300 is realized by a hardware circuit such as FPGA, the pre-learning model may be stored in the memory built in the FPGA.

The control unit 303 is a functional unit that controls the processing performed by the image processing device 300. The control unit 303 can be realized by an arithmetic processing unit such as a CPU. The control unit 303 is further composed of four functional units: an acquisition unit 3031, an extraction processing unit 3032, a distance measuring processing unit 3033, and a notification control processing unit 3034. Each functional unit may be realized by executing the stored program by the CPU. The control unit 303 functions as the processor according to the present disclosure by executing the processes of the acquisition unit 3031, the extraction processing unit 3032, the distance measuring processing unit 3033, and the alarm control processing unit 3034. Then, by mounting this processor on the integrated circuit constituting the image processing device 300, the processor that executes the processing of the functional unit is installed in the work machine 10.

Here, the processing flow performed by the control unit 303 will be described with reference to FIG. FIG. 2 is a flowchart showing a processing flow performed by the control unit 303 in order to avoid contact between the work machine 10 and a person around the work machine 10 in the peripheral monitoring device 100 according to the present embodiment. In the present embodiment, when the power of the peripheral monitoring device 100 is turned on, the execution of this flow is started, and the execution is repeatedly executed at a predetermined calculation cycle while the peripheral monitoring device 100 is operating.

In this flow, first, in S101, the acquisition unit 3031 acquires photographed image data representing the surrounding image of the work machine 10 photographed by the photographing device 200. Here, the photographing device 200 and the image processing device 300 have a communication unit, and by connecting these communication units via a network, the acquisition unit 3031 can acquire the above-mentioned captured image data. become. The network may be wireless, wired, or a combination of wireless and wired.

Here, the acquisition unit 3031 may acquire the captured image data by excluding the image of a certain area predetermined from the surrounding images of the work machine 10 captured by the photographing device 200. FIG. 3 is a diagram for explaining an area excluded from the captured image data. As shown in FIG. 3, the acquisition unit 3031 can acquire captured image data by excluding the area α, which is an area where the work machine 10 can be photographed. According to this, it is possible to acquire the captured image data without including the operator in the driver's cab of the work machine 10, and it is possible to appropriately extract only the people around the work machine 10 in the extraction process described later. .. Further, as shown in FIG. 3, the acquisition unit 3031 can acquire the captured image data by excluding the area β, which is an area where the upper part of the work machine 10 (such as the surrounding sky) can be photographed. According to this, it is possible to acquire the captured image data without including the area where a person cannot exist around the work machine 10, and it is possible to reduce the processing load in the extraction process described later. In this way, the acquisition unit 3031 may acquire only the image of a predetermined area (for example, the acquisition area shown in FIG. 3) as the captured image data among the surrounding images of the work machine 10 captured by the imaging device 200. good. The exclusion area as described above can be adjusted in advance according to the model of the work machine 10 and the installation parameters of the photographing device 200 in each work machine 10. For example, as shown in FIG. 3A, the optical axis L2 of the hemispherical camera 220 is oriented substantially perpendicular to the ground, whereas as shown in FIG. 3B, the hemispherical camera 220 When the optical axis L2 is tilted backward, the area α and the area β can be adjusted in advance according to the tilt. In FIG. 3, the case where the work machine 10 is a mini excavator has been described as an example, but the exclusion area can be set in the same manner when the work machine 10 is a forklift or the like.

Next, in S102, the extraction processing unit 3032 executes an extraction process for extracting a person image from the captured image data. In this extraction process, when a person exists around the work machine 10 and the person is photographed by the photographing apparatus 200, a person image can be extracted from the photographed image data. Here, the extraction processing unit 3032 extracts a person image by excluding the operator in the driver's cab of the work machine 10. For example, as described above, when the acquisition unit 3031 acquires the photographed image data by excluding the area where the work machine 10 can be photographed, the photographed image data includes the operator in the driver's cab of the work machine 10. Therefore, the extraction processing unit 3032 can extract a person image by excluding the operator in the driver's cab of the work machine 10. On the other hand, when the acquisition unit 3031 acquires the surrounding image of the work machine 10 photographed by the photographing apparatus 200 as it is as the photographed image data, the extraction processing unit 3032 is in the cab of the work machine 10 in the photographed image data. The extraction process is executed excluding the operator of. The details of this extraction process will be described later. Then, the extraction processing unit 3032 determines in S103 whether or not a person exists around the work machine 10 based on the result of the extraction processing executed in S102. If an affirmative determination is made in S103, the control unit 303 proceeds to the process of S104, and if a negative determination is made in S103, the execution of this flow is terminated.

If an affirmative determination is made in S103, then in S104, the distance measuring processing unit 3033 executes a distance measuring process for measuring the distance between the work machine 10 and a person around it. In this distance measuring process, the distance between the work machine 10 and the people around it is measured based on the person image extracted by the extraction process and the installation parameters of the photographing device 200. The details of this distance measurement process will be described later. Further, in S104, in addition to the distance measuring process described above, the output of the extracted image data, which is the image data of the person image extracted by the extraction process of S102, is executed. At this time, as will be described later, the display device 500 displays the captured image data captured by the photographing device 200 and the person image extracted by the extraction process together with the rectangle.

Next, in S105, the alarm control processing unit 3034 determines whether or not the above distance measured by the distance measuring process is equal to or less than a predetermined value. Here, the above-mentioned predetermined value is a value of a distance at which contact between the work machine 10 and a person around it is feared due to the operation of the work machine 10, and is based on, for example, the size and the operating range of the work machine 10. It is decided in advance. Then, if an affirmative determination is made in S105, the control unit 303 proceeds to the process of S106, and if a negative determination is made in S105, the execution of this flow is terminated.

If a positive determination is made in S105, the alarm control processing unit 3034 then controls the output of the alarm device 400 in S106. In the present embodiment, the alarm control processing unit 3034 outputs a warning light from the alarm device 400 (for example, Patrite (registered trademark)). When the alarm device 400 is composed of, for example, a patrol light (registered trademark) and a speaker, the alarm control processing unit 3034 outputs a warning light from the patrol light (registered trademark) and emits a warning sound from the speaker. It may be output. Then, after the processing of S106, the execution of this flow is completed.

According to the process described above, the distance to the person around the work machine 10 is measured based on the person image extracted by the extraction process, and the alarm control process is performed based on the result of the distance measurement distance. Will be executed. Then, by outputting a predetermined alarm from the alarm device 400, it is possible to give a warning to a person around the work machine 10 in order to avoid contact with the work machine 10. Further, when the operator of the work machine 10 recognizes the output of the alarm from the alarm device 400, the operator can perform an operation operation for avoiding contact between the work machine 10 and a person around it. As a result, contact between the work machine 10 and a person around it can be suppressed.

(Extraction process)
Next, the details of the extraction process executed by the extraction process unit 3032 will be described. In the extraction process of the present embodiment, a person image is extracted by inputting captured image data into the pre-learning model. Then, the pre-learning model is constructed by learning using data including an image representing a person.

Here, FIG. 4 is a diagram for explaining the identification result obtained from the input to the pre-learning model in the present embodiment and the neural network constituting the pre-learning model. In this embodiment, a neural network model generated by deep learning is used as the pre-learning model. The pre-learning model 30 in the present embodiment includes an input layer 31 that accepts input of predetermined image data, and an intermediate layer (hidden layer) 32 that extracts a feature amount representing a person from the image data input to the input layer 31. It has an output layer 33 that outputs an identification result based on a feature amount. In the example of FIG. 4, the pre-learning model 30 has one intermediate layer 32, the output of the input layer 31 is input to the intermediate layer 32, and the output of the intermediate layer 32 is input to the output layer 33. Has been done. However, the number of intermediate layers 32 is not limited to one, and the pre-learning model 30 may have two or more intermediate layers 32.

Also, according to FIG. 4, each layer 31-33 comprises one or more neurons. For example, the number of neurons in the input layer 31 can be set according to the input image data. Further, the number of neurons in the output layer 33 can be set according to the person image which is the identification result.

Then, neurons in adjacent layers are appropriately connected to each other, and a weight (connection load) is set for each connection based on the result of machine learning. In the example of FIG. 4, each neuron is connected to all neurons in the adjacent layer, but the connection of neurons does not have to be limited to such an example and can be appropriately set.

Such a pre-learning model 30 is constructed by performing supervised learning using, for example, supervised learning that is a set of image data including an image representing a person and a label of an image representing a person. Specifically, a set of a feature amount and a label is given to the neural network, and the weight of the connection between neurons is tuned so that the output of the neural network becomes the same as the label. In this way, a pre-learning model for learning the characteristics of the teacher data and estimating the result from the input is inductively acquired.

As shown in FIG. 1, when the hemispherical camera 220 is arranged on the upper surface of the upper swivel body 11 of the work machine 10, the peripheral image of the work machine 10 taken by the photographing device 200 is displayed. There is a tendency for the work machine 10 to be a bird's-eye view image from above. Therefore, in the present embodiment, a bird's-eye view image may be used as the image data to be learned in order to build the pre-learning model. In this way, by using the bird's-eye view image as the image data learned to build the pre-learning model, it is possible to more accurately extract a person image from the captured image data.

Further, the pre-learning model 30 may be constructed by performing unsupervised learning. For unsupervised learning, for example, domain adaptation, which is a type of transfer learning, can be used. According to this, the pre-learning model can be acquired without preparing a large amount of labeled teacher data.

Then, in the present embodiment, in addition to the captured image data captured by the photographing device 200, the person image extracted by the extraction process is displayed together with the rectangle on the display device 500 described above. Here, FIG. 5 is a diagram for explaining photographed image data and a person image displayed on the display device 500. The operator of the work machine 10 can visually recognize the surroundings of the work machine 10 through the screen SC1 of the display device 500.

FIG. 5A is a diagram showing captured image data displayed on the display device 500. As shown in FIG. 5A, the captured image data SC11 is displayed on the screen SC1 of the display device 500. FIG. 5B is a diagram showing an example in which a person image extracted by the extraction process is displayed on the display device 500 together with the captured image data. As shown in FIG. 5B, the captured image data SC11 and the person image SC12 are displayed on the screen SC1 of the display device 500. In this way, the person image SC12 extracted by the extraction process is displayed together with the rectangle. According to this, the operator of the work machine 10 can easily visually recognize the person image SC12 displayed on the screen SC1 of the display device 500. By extracting the person image using the above-mentioned pre-learning model, the extraction accuracy of the person image from the photographed image data, specifically, the person image SC12 included in the photographed image data SC11 and displayed is represented. The position accuracy of the rectangle is improved.

Here, in the image processing device 300 of the present embodiment, the distortion correction value of the lens of the hemispherical camera 220 is used to correct the distortion of the captured image data. Then, the captured image data SC11 with distortion correction is displayed on the screen SC1 of the display device 500. In addition, a person image is extracted from the captured image data that has been subjected to distortion correction. Therefore, in constructing the pre-learning model, a distortion-corrected image is used as image data including an image representing a person used for learning. However, there is no intention of limiting the present disclosure to this, and the above-mentioned distortion correction may not be executed in the peripheral monitoring device of the present disclosure. In this case, the captured image data including the distortion is displayed on the display device 500, but if the person image can be accurately extracted from the captured image data including the distortion, the working machine 10 can display the captured image data. The operator can visually recognize the person image displayed together with the rectangle. Therefore, in such a case, in constructing the pre-learning model, an image including distortion is used as image data including an image representing a person used for learning. Then, the person image can be accurately extracted from the captured image data including the distortion. According to this, the image processing device 300 does not need to correct the distortion of the captured image data acquired from the photographing device 200, and thus the processing load of the image processing in the peripheral monitoring device is reduced, so that the work can be performed more quickly. The surroundings of the machine 10 can be monitored.

Further, FIG. 5C is a diagram showing variations of a rectangle representing the person image SC12. As shown in FIG. 5C, the rectangle representing the person image SC12 may be a rectangle SC121 that surrounds the whole body of a person, a rectangle SC122 that surrounds the upper body of a person, or a rectangle SC122 that surrounds the upper body of a person. It may be a rectangle SC123 that surrounds. The operator of the work machine 10 can recognize a person image regardless of the rectangle, but for example, in the case of the rectangle SC121 that surrounds the whole body of a person, it becomes easier to recognize the size of the person. Further, for example, when a plurality of person images are included in the captured image data, the number of people can be easily recognized by using the rectangular SC123 that surrounds the neck of the person. That is, when the captured image data includes a plurality of person images, a rectangular SC123 that surrounds the neck of the person can be preferably used.

In the present embodiment, as described above, the function of the image processing device 300 can be realized by a hardware circuit such as an FPGA. In this case, the pre-learning model can be stored in the memory built into the FPGA. According to this, the extraction process using the pre-learning model can be executed as quickly as possible, and thus the periphery of the work machine 10 can be monitored more quickly.

As described above, according to the extraction process of the present embodiment, by using the pre-learning model, a person image can be accurately extracted from the captured image data, and the extracted person image is suitable for the operator of the work machine 10. Can be visually recognized.

(Distance measurement processing)
Next, the details of the distance measuring process executed by the distance measuring processing unit 3033 will be described. In the distance measuring process of the present embodiment, the distance between the work machine 10 and the person around it is measured based on the person image extracted by the extraction process and the installation parameters of the photographing device 200. Regarding such distance measurement processing, in the present embodiment, foot distance measurement based on the foot image of the person image and head distance measurement based on the head image of the person image are used. Used. In the present embodiment, any one of these distance measuring methods may be used, or both of these distance measuring methods may be used. Foot distance measurement and head distance measurement will be described in detail below.

First, foot distance measurement will be described. In foot distance measurement, the distance measurement processing unit 3033 acquires the foot coordinates, which are the two-dimensional coordinates of the foot of the person image in the captured image data. Then, based on the foot coordinates, the foot angle, which is a predetermined angle between the feet of a person around the work machine 10 and the photographing device 200, is calculated. Then, the distance measuring processing unit 3033 can measure the distance between the work machine 10 and the person around it based on the calculated foot angle and the installation height of the photographing device 200.

Ｄ：周囲人物との距離（ｍｍ）
ｈ：撮影装置２００の設置高さ（ｍｍ）
θ：足元角度（ｄｅｇｒｅｅ） FIG. 6 is a diagram for explaining the relationship between the foot angle, the installation height of the photographing device 200, and the distance to the surrounding people. Here, the surrounding person is a person around the work machine 10, and the distance between the surrounding person and the surrounding person, which is the distance between the surrounding person and the work machine 10, is represented as the distance D in FIG. Then, the installation height of the photographing device 200 from the ground is represented as the height h in FIG. Further, the above-mentioned foot angle between the feet of the surrounding person and the photographing device 200 is represented as the angle θ in FIG. Then, the distance D to the surrounding people can be calculated by the following equation 1.

D: Distance to surrounding people (mm)
h: Installation height (mm) of the photographing device 200
θ: Foot angle (degree)

Then, the distance measuring processing unit 3033 executes the processing flow shown in FIG. 7 in order to calculate the distance D with the surrounding person based on the above equation 1. FIG. 7 is a flowchart showing a processing flow performed by the distance measuring processing unit 3033 in the foot distance measuring, which is one of the distance measuring processes according to the present embodiment. This flow is executed as an internal process of the distance measurement process in S104 shown in FIG. 2 above.

In this flow, first, in S1041, the foot coordinates (two-dimensional coordinates of the foot of the person image in the captured image data) are acquired. This will be described with reference to FIG. FIG. 8 is a diagram for explaining a subject projected on the sensor of the camera together with the internal parameters of the camera. In the example shown in FIG. 8, a surrounding person exists at a position deviated from the optical axis L2 of the camera in the Y-axis direction. In this case, the distance measuring processing unit 3033 acquires the coordinates y of the feet of the surrounding people projected on the sensor as the feet coordinates. As described in the above description of the extraction process, the person image can be displayed together with the rectangle. Therefore, for example, the lower end of the rectangle surrounding the whole body of a person may be acquired as the foot coordinates. Further, when the captured image data for which the foot coordinates are acquired is not distortion-corrected, the acquired foot coordinates can be subjected to the distortion correction processing.

ｄｉｓｔ：足元座標の光軸からの距離（ｐｉｘｅｌ）
ｘ：足元座標のｘ座標（ｐｉｘｅｌ）
ｙ：足元座標のｙ座標（ｐｉｘｅｌ）
ｘｃｅｎ：センサ上の光軸のｘ座標（ｐｉｘｅｌ）
ｙｃｅｎ：センサ上の光軸のｙ座標（ｐｉｘｅｌ）
なお、センサ上の光軸の座標（ｘｃｅｎ、ｙｃｅｎ）は、カメラの内部パラメータとして予め定められる。 Then, returning to FIG. 7, next, in S1042, the foot angle (the angle between the foot of a person around the work machine 10 and the photographing device 200) is calculated. Here, assuming that the surrounding person exists at a position deviated from the optical axis L2 of the camera in the X-axis and Y-axis directions, in the above-mentioned processing of S1041, the distance measuring processing unit 3033 is placed on the sensor. The x-coordinate and y-coordinate of the projected foot of the surrounding person are acquired. Then, in the process of S1042, first, the distances of these coordinates from the optical axis L2 are calculated by the following equation 2.

dist: Distance from the optical axis of the foot coordinates (pixel)
x: x-coordinate (pixel) of the foot coordinate
y: y-coordinate (pixel) of the foot coordinate
xcent: x-coordinate (pixel) of the optical axis on the sensor
yfen: y-coordinate (pixel) of the optical axis on the sensor
The coordinates of the optical axis on the sensor (xcen, ycen) are predetermined as internal parameters of the camera.

θ：足元角度（ｄｅｇｒｅｅ）
ｆ：レンズの焦点距離（ｍｍ）
ｄｉｓｔ：足元座標の光軸からの距離（ｐｉｘｅｌ）
Ｎ：カメラの画素サイズ（ｍｍ／ｐｉｘｅｌ） Then, the foot angle θ can be calculated by the following equation 3 based on FIG. 8 above.

θ: Foot angle (degree)
f: Focal length of the lens (mm)
dist: Distance from the optical axis of the foot coordinates (pixel)
N: Camera pixel size (mm / pixel)

Then, returning to FIG. 7, next, in S1043, the installation height of the photographing apparatus 200 is acquired. The installation height of the photographing device 200 is predetermined according to the work machine 10 in which the peripheral monitoring device 100 is installed. The distance measuring processing unit 3033 may perform the processing of S1043 before the processing of S1041.

Next, in S1044, the distance between the work machine 10 and the people around it is measured based on the foot angle calculated by the process of S1042 and the installation height of the photographing device 200 acquired by the process of S1043. NS. Specifically, the distance measuring processing unit 3033 calculates the distance between the work machine 10 and a person around it based on the above equation 1. Then, after the processing of S1044, the control unit 303 proceeds to the processing of S105. As described above, in S104, the output of the extracted image data is executed in addition to the distance measuring process.

Based on the above, the hemispherical camera 220 can perform distance measurement processing accurately at low cost and with as few cameras as possible. In addition, according to such foot distance measurement, since the foot coordinates can be acquired relatively accurately, the above-mentioned distance calculation accuracy can be made as high as possible.

Next, head distance measurement will be described. In the head distance measurement, the distance measurement processing unit 3033 acquires the head image of the person image in the captured image data, and based on the size of the head image, determines the distance between the work machine 10 and the people around it. taking measurement.

Ｄ´：周囲人物との距離（ｍｍ）
Ｄｒｅｆ：基準距離（ｍｍ）
Ｎｒｅｆ：基準距離での縦画素数（ｐｉｘｅｌ）
Ｎｈｅａｄ：頭の画像の縦画素数（ｐｉｘｅｌ）
なお、基準距離Ｄｒｅｆおよび該基準距離での縦画素数Ｎｒｅｆは、予め定められる。そして、このように予め定められた基準と、撮影画像データから取得したの頭の画像の縦画素数Ｎｈｅａｄと、を比較することで、周囲人物との距離Ｄ´が算出される。 Specifically, the distance measuring processing unit 3033 acquires the number of vertical pixels of the head image of the person image in the captured image data. Here, the number of vertical pixels of the head image can be obtained from, for example, the number of vertical pixels of a rectangle surrounding the neck of a person. Then, the present disclosure person has found that the distance can be measured based on the above-mentioned number of vertical pixels representing the size of the head by utilizing the fact that the size of the head of a person has little individual difference. Therefore, the distance measuring processing unit 3033 calculates the distance between the work machine 10 and a person around it based on the following equation 4.

D': Distance to surrounding people (mm)
Dref: Reference distance (mm)
Nref: Number of vertical pixels at reference distance (pixel)
Nhead: Number of vertical pixels (pixel) of the head image
The reference distance Dref and the number of vertical pixels Nref at the reference distance are predetermined. Then, the distance D'with the surrounding person is calculated by comparing the predetermined standard with the number of vertical pixels Nhead of the head image acquired from the captured image data.

Based on the above, the hemispherical camera 220 can perform distance measurement processing accurately at low cost and with as few cameras as possible. According to such head distance measurement, if the captured image data does not include the feet of surrounding people, or if the person is climbing to a high place and the person is not on the ground. Even if there is, the distance between the work machine 10 and the people around it can be calculated.

Ｄｏｕｔ：出力される周囲人物との距離（ｍｍ）
Ｄ：足元測距により算出される周囲人物との距離（ｍｍ）
Ｄ´：頭測距により算出される周囲人物との距離（ｍｍ）
ｗ：ブレンド係数
ここで、ブレンド係数ｗは、下記式６および下記式７に基づいて算出される。

Then, both the foot distance measurement and the head distance measurement described above may be used to calculate the distance between the work machine 10 and a person around it. For example, when the distance D to the surrounding person based on the above formula 1 is calculated by the foot distance measurement and the distance D'to the surrounding person based on the above formula 4 is calculated by the head distance measurement, the distance measurement processing unit 3033. Outputs the distance Dout to the surrounding person calculated based on the following equation 5 as the distance between the work machine 10 and the surrounding person.

Dout: Distance to the output surrounding person (mm)
D: Distance to surrounding people calculated by foot distance measurement (mm)
D': Distance to surrounding people calculated by head distance measurement (mm)
w: Blend coefficient Here, the blend coefficient w is calculated based on the following formula 6 and the following formula 7.

According to this, for example, when the distance D and the distance D'are deviated by 500 mm, the blend coefficient w becomes 0.5, and the distance D to the surrounding person calculated by the foot distance measurement and the head distance measurement are used. The calculated average value of the distance D'with the surrounding person is output as the distance Dout between the work machine 10 and the surrounding person. Further, for example, when the distance D and the distance D'are deviated by 1000 mm or more, the blend coefficient w becomes 1.0, and the distance D'with the surrounding person calculated by the head distance measurement is the same as that of the work machine 10. It is output as the distance Dout with the surrounding people. When the distance D and the distance D'are equal, the blend coefficient w becomes 0, and the distance D with the surrounding person calculated by the foot distance measurement is set as the distance Dout between the work machine 10 and the surrounding person. It is output.

As described above, in the present embodiment, the person image can be accurately extracted from the captured image data by the extraction process, and the distance to the surrounding person can be accurately measured by the distance measurement process. According to the present embodiment, a peripheral monitoring device capable of monitoring the surroundings of the work machine over a wide range at low cost and with as few cameras as possible, and suppressing contact between the work machine and people around it. Can be provided.

The peripheral monitoring device 100 of the present embodiment may include a storage device that stores the installation parameters and the like of the photographing device 200 described above. In this case, the storage device can store the installation parameters of the photographing device 200 for each model of the work machine 10. Then, the above-mentioned installation parameters stored in the storage device can be displayed on the display device 500 together with the model of the work machine 10. Then, the user can easily install the peripheral monitoring device 100 according to the model of the work machine 10.

Further, in the above embodiment, the configuration in which the peripheral monitoring device 100 includes the display device 500 has been described as an example, but the peripheral monitoring device of the present disclosure is not intended to be limited to this. The peripheral monitoring device of the present disclosure does not have to include the above-mentioned display device, and even in this case, the distance measuring process is executed based on the person image extracted by the extraction process and the installation parameters of the photographing device. By executing the alarm control process based on the result of the distance measurement process, it is possible to suppress contact between the work machine and a person around it.

<Modification 1 of the first embodiment>
A modification 1 of the first embodiment will be described. In the first embodiment described above, when the distance between the work machine 10 and a person around it becomes a predetermined value or less, a warning light or a warning sound is output from the alarm device 400. As a result, it is possible to give a warning to a person around the work machine 10 in order to avoid contact with the work machine 10. Further, when the operator of the work machine 10 recognizes the output of the alarm from the alarm device 400, the operator can perform an operation operation for avoiding contact between the work machine 10 and a person around it. Here, if the avoidance operation of the work machine 10 can be performed without the operation operation by the operator, the contact between the work machine 10 and a person around the work machine 10 can be more preferably avoided.

Therefore, in the present modification, when the alarm control processing unit 3034 described in the description of the first embodiment has a distance between the work machine 10 and a person around it of a predetermined value or less, the alarm device 400 Therefore, a signal for instructing a predetermined avoidance operation for avoiding contact between the work machine 10 and a person around the work machine 10 is output.

Here, the alarm device 400 of this modification is configured to be able to output a predetermined alarm signal. The predetermined alarm signal is, for example, a signal for instructing the avoidance operation, and the signal is transmitted to the control controller of the work machine 10 by the Controller Area Network (CAN) or the like. Then, the work machine 10 automatically performs the avoidance operation regardless of the operation of the operator. Here, the avoidance operation is an operation in which the operation of the work machine 10 is decelerated, an operation in which the operation of the work machine 10 is stopped, and the like. According to this, it is possible to perform the avoidance operation of the work machine 10 without the operation operation by the operator, and it is possible to more preferably avoid the contact between the work machine 10 and the person around it. The alarm device 400 may be configured to be capable of outputting electrical or mechanical contacts or input / output signals.

The alarm device 400 may have only a function of outputting a signal for instructing such an avoidance operation, and in addition to the function, a warning light or a warning described in the above description of the first embodiment may be provided. It may have a function of outputting sound. Further, as an alarm signal, information indicating that the distance between the work machine 10 and a person around it is equal to or less than a predetermined value may be notified to a predetermined terminal via a predetermined communication means (for example, Ethernet). ..

<Modification 2 of the first embodiment>
The second modification of the first embodiment will be described. In the first embodiment described above, when the power of the peripheral monitoring device 100 is turned on, the flow shown in FIG. 2 is executed. On the other hand, in this modification, when the power of the peripheral monitoring device 100 is turned on, the flow shown in FIG. 9 described below is executed.

In this modification, the processing flow performed by the control unit 303 will be described with reference to FIG. FIG. 9 is a first flowchart showing a processing flow performed by the control unit 303 in order to avoid contact between the work machine 10 and a person around the work machine 10 in the peripheral monitoring device 100 according to the present modification. In this modification, substantially the same processing as that shown in FIG. 2 of the first embodiment described above will be designated by the same reference numerals and detailed description thereof will be omitted.

In the flow shown in FIG. 9, after the processing of S101, it is determined in S201 whether or not a predetermined time has elapsed since the power of the peripheral monitoring device 100 was turned on. Here, the predetermined time is, for example, the time until the pre-learning model or program used for the extraction process is loaded into the work area of the image processing apparatus 300, and is predetermined according to the scale of the model. Then, when an affirmative determination is made in S201, in this case, the extraction process can be executed immediately, and the control unit 303 proceeds to the process in S102. On the other hand, if a negative determination is made in S201, the control unit 303 proceeds to the process of S202.

If a negative determination is made in S201, then in S202, only the captured image data acquired in the process of S101 is output to the display device 500. Then, after the processing of S202, the execution of this flow is completed.

Here, the control unit 303 cannot immediately execute the extraction process until a predetermined time elapses after the power of the peripheral monitoring device 100 is turned on. In such a case, if the process of the control unit 303 is waited until the extraction process can be executed and the operator of the work machine 10 cannot visually recognize the surroundings via the display device 500, the work machine during this period. There is a risk that contact between 10 and a person around it may occur. On the other hand, according to this flow, the data that can be acquired by the extraction process and the distance measurement process is not output to the display device 500 until a predetermined time elapses after the power of the peripheral monitoring device 100 is turned on. Only the captured image data is output to the display device 500. That is, the surrounding image of the work machine 10 is output to the display device 500 as soon as possible after the power of the peripheral monitoring device 100 is turned on. Therefore, the operator of the work machine 10 can visually recognize the surroundings via the display device 500 as soon as possible after the power of the peripheral monitoring device 100 is turned on. According to this, it is possible to more preferably avoid contact between the work machine 10 and a person around it.

When only the captured image data is output to the display device 500 in this way, the rectangle is not displayed on the person included in the captured image data. Then, the operator of the work machine 10 can recognize that the extraction process and the distance measurement process have not been executed yet.

Further, in this modification, when the power of the peripheral monitoring device 100 is turned on, the flow shown in FIG. 10 described below may be executed. FIG. 10 is a second flowchart showing a processing flow performed by the control unit 303 in order to avoid contact between the work machine 10 and a person around the work machine 10 in the peripheral monitoring device 100 according to the present modification. The processes substantially the same as those shown in FIG. 2 of the first embodiment described above are designated by the same reference numerals, and detailed description thereof will be omitted.

In the flow shown in FIG. 10, after the processing of S102, it is determined in S203 whether or not the result of the extraction processing in the processing of S102 can be acquired. Here, when the result of the extraction process cannot be obtained, for example, the pre-learning model or program used for the extraction process has not yet been loaded into the work area of the image processing device 300 (loading to the work area has been completed). Not) if. In this case, in the flow shown in FIG. 10, the process of S102 is executed regardless of whether or not the above loading is completed. If the extraction of surrounding people cannot be performed because the above loading is not completed in S102, the control unit 303 immediately terminates the processing of S102, assuming that the extraction processing cannot be executed. At this time, the control unit 303 may set a flag for determining whether or not the extraction process has been executed. Then, in S203, the above can be determined based on this flag. Then, if an affirmative determination is made in S203, the control unit 303 proceeds to the process of S103, and if a negative determination is made in S203, the control unit 303 proceeds to the process of S204.

If a negative determination is made in S203, then in S204, only the captured image data acquired in the process of S101 is output to the display device 500. Then, after the processing of S204, the execution of this flow is completed.

Even with the above-described processing, the operator of the work machine 10 can visually recognize the surroundings via the display device 500 as soon as possible after the power of the peripheral monitoring device 100 is turned on. It is possible to more preferably avoid contact between the machine 10 and a person around it.

<Second Embodiment>
The second embodiment will be described. In the present embodiment, the control unit 303 further executes the image sharpening process. Specifically, the control unit 303 generates a captured image histogram showing the frequency of pixel values included in the captured image data. Then, the captured image data is sharpened by synthesizing the smoothed image data acquired based on the smoothed smoothed image histogram and the captured image data at a predetermined ratio. This will be described with reference to FIG.

The control unit 303 of the present embodiment executes the processing flow shown in FIG. 11 in order to execute the image sharpening process. FIG. 11 is a flowchart showing a processing flow performed by the control unit 303 in the image sharpening process according to the present embodiment. This flow is executed after the processing of S101 of the flow shown in FIG. 2 above.

In this flow, after the processing of S101, a captured image histogram is generated in S301. As described above, the captured image histogram is a histogram showing the frequency of pixel values included in the captured image data, and is generated based on a well-known technique.

Next, in S302, smoothed image data is generated. In the process of S302, the control unit 303 first generates a smoothed histogram obtained by smoothing the captured image histogram, and acquires smoothed image data based on the smoothed histogram. Here, the smoothing histogram is generated by stretching the captured image histogram over the entire range of pixel values (256 gradations). For example, a histogram in which the frequency is concentrated in a certain pixel value range is a pixel value. A high-contrast image can be obtained by stretching over the entire range (256 gradations). That is, the smoothed image data is image data in which the contrast of the captured image data is enhanced.

According to such smoothed image data, it is possible to secure the contrast of the image even at night or in foggy weather. However, since the work machine 10 is used in various environments, it is necessary to adjust the contrast according to the environment.

Therefore, in the present embodiment, the sharpening intensity is acquired in the processing of S303. Then, in the process of S304, the smoothed image data is combined with the captured image data at a predetermined ratio. This ratio is a ratio based on the sharpening intensity, and the intensity classification can be set in, for example, three stages of weak, medium, and strong. For example, when the sharpening intensity classification is weak, the smoothed image data and the captured image data are combined at a ratio of 1: 3, and when the sharpening intensity classification is medium, the smoothed image data and the captured image data are combined. When the data and the data are combined at a ratio of 1: 1 and the sharpening intensity classification is strong, the smoothed image data and the captured image data are combined at a ratio of 3: 1. In the processing of S303, the strength value corresponding to such a classification, for example, when the strength classification is weak, the strength value may be acquired as 0.25 (in this case, it is clarified). Twenty-five percent of the captured image data is from the smoothed image data, 75% is from the original captured image data), and any value regardless of classification (value greater than 0 and less than 1) May be obtained. Further, the sharpening intensity may be manually set by the operator of the work machine 10, or may be automatically set according to the environment. Then, after the processing of S304, the control unit 303 proceeds to the processing of S102.

In this way, when the captured image data is sharpened by the processes of S301 to S304 and the extraction process is executed by the next process of S102, it is based on the captured image data that has been sharpened according to the environment. A person image will be extracted. According to this, the accuracy of extracting a person image from the captured image data can be made as high as possible. Further, the operator of the work machine 10 can easily visually recognize an obstacle existing around the vehicle body of the work machine 10 via the display device 500.

Further, the peripheral monitoring device described above can also suitably suppress the contact between the work machine and the person around it.

<Third Embodiment>
A third embodiment will be described. The photographing device 200 of the present embodiment is a device that automatically controls the exposure of the hemispherical camera 220 to a predetermined reference value according to the brightness of the subject. When automatically controlling the exposure of the hemisphere camera 220, if the sky or streetlights appear around the hemisphere camera 220, the brightness will not affect the people or obstacles around the vehicle body. The purpose is to control. Then, such a photographing device 200 further acquires the brightness of the ground around the work machine 10, and when the brightness of the ground is equal to or higher than the predetermined first brightness, the exposure of the hemispherical camera 220 is described above. Correct so that it is larger than the reference value. On the other hand, when the brightness of the ground is lower than the above-mentioned first brightness and is equal to or less than the predetermined second brightness, the exposure of the hemispherical camera 220 is corrected so as to be smaller than the above-mentioned reference value.

Here, in a photographing apparatus in which the exposure is automatically controlled, the exposure tends to be focused on the side light near the optical axis of the camera. However, in the photographing apparatus 200 of the present embodiment, as described in the description of the first embodiment, for example, the optical axis L2 of the hemispherical camera 220 is oriented substantially perpendicular to the ground. Then, the work machine 10 exists in the vicinity of the optical axis L2 of the hemispherical camera 220. In such a case, if the vicinity of the optical axis L2 of the hemispherical camera 220 is left with the exposure focused on the side light, there is a possibility that the person around the work machine 10 cannot be photographed with an appropriate brightness. .. Further, the area where such a work machine 10 can be photographed can be excluded from the photographed image data by the acquisition unit 3031 described above.

Therefore, as a result of diligent studies, the present disclosure person corrects the exposure of the hemispherical camera 220 based on the brightness of the ground around the work machine 10, so that the person around the work machine 10 has an appropriate brightness. I found out that I can shoot. This will be described with reference to FIG.

In the photographing apparatus 200 of the present embodiment, the exposure of the hemispherical camera 220 is corrected according to the processing flow shown in FIG. FIG. 12 is a flowchart showing a flow of exposure compensation of the hemispherical camera 220 according to the present embodiment. This flow is executed by, for example, a predetermined processor included in the photographing apparatus 200 before the processing of S101 of the flow shown in FIG.

In this flow, first, in S401, the exposure of the hemispherical camera 220 is automatically controlled to a predetermined reference value. In the process of S401, the exposure is automatically controlled according to the brightness of the subject photographed by the hemispherical camera 220 based on a well-known technique. The brightness of the subject can be acquired by, for example, a well-known side light sensor. Here, as described above, since the hemispherical camera 220 is generally directed toward the ground, the ground occupies most of the subject photographed by the hemispherical camera 220. Then, in this flow, first, in the processing of S401, there is a tendency that the exposure is automatically controlled mainly according to the brightness of the ground.

Next, in S402, the brightness of the ground around the work machine 10 is acquired. In the process of S402, for example, a region of the ground is determined based on a predetermined rule, and the brightness of the region is acquired by a side light sensor or the like. In this case, a predetermined shooting area determined in advance based on the installation parameters of the shooting device 200 may be determined as the ground area, or the ground may be determined based on the occupancy rate of the brightness in the subject shot by the hemispherical camera 220. The area of may be determined.

Next, in S403, it is determined whether or not the brightness of the ground acquired by the process of S402 is equal to or higher than the predetermined first brightness. Here, the above-mentioned first brightness is determined based on the material of the ground. Specifically, in the present embodiment, the first brightness is predetermined as the brightness corresponding to the case where the ground material is white concrete. Then, if an affirmative determination is made in S403, the process proceeds to S404, and if a negative determination is made in S403, the process proceeds to S405.

If an affirmative judgment is made in S403, then in S404, the exposure of the hemispherical camera 220 is corrected so as to be larger than the above reference value. Here, when an affirmative judgment is made in S403, in this case, the ground is made of a bright material such as white concrete, and in the processing of S401, it is used as a reference value for the exposure of the hemispherical camera 220. , The shutter speed of the camera tends to be high or the aperture value tends to be large. In such a case, if the exposure of the hemispherical camera 220 is left at the reference value, there is a risk that the person around the work machine 10 will be photographed extremely dark. On the other hand, when the correction by the process of S404 is performed, the exposure of the hemispherical camera 220 is made larger than the above reference value, so that the person around the work machine 10 can be photographed with an appropriate brightness. Then, after the processing of S404, the execution of this flow is completed.

On the other hand, if a negative determination is made in S403, then in S405, it is determined whether or not the brightness of the ground acquired by the process of S402 is equal to or less than a predetermined second brightness. Here, the second brightness is lower than the first brightness and is determined based on the material of the ground. Specifically, in the present embodiment, the second brightness is predetermined as the brightness corresponding to the case where the ground material is black asphalt. Then, if an affirmative determination is made in S405, the process proceeds to S406, and if a negative determination is made in S405, the execution of this flow is terminated.

If an affirmative judgment is made in S405, then in S406, the exposure of the hemispherical camera 220 is corrected so as to be smaller than the above reference value. Here, when an affirmative judgment is made in S405, in this case, the ground is made of a dark material such as black asphalt, and in the processing of S401, it is used as a reference value for the exposure of the hemispherical camera 220. , The shutter speed of the camera tends to be slow or the aperture value tends to be small. In such a case, if the exposure of the hemispherical camera 220 is left at the reference value, there is a risk that a person around the work machine 10 will be photographed extremely brightly. On the other hand, when the correction by the processing of S406 is performed, the exposure of the hemispherical camera 220 is made smaller than the above reference value, so that a person around the work machine 10 can be photographed with an appropriate brightness. Then, after the processing of S406, the execution of this flow is completed.

If a negative determination is made in S405, in this case, the brightness of the ground acquired in the process of S402 is higher than the second brightness and lower than the first brightness, and the exposure of the hemispherical camera 220 is corrected. Not done. That is, the exposure of the hemispherical camera 220 is left at the reference value.

When the exposure of the hemispherical camera 220 is corrected in this way, the person around the work machine 10 can be photographed with an appropriate brightness, so that the accuracy of extracting the person image from the photographed image data can be achieved. Can be raised. Further, the operator of the work machine 10 can easily visually recognize the people around the work machine 10 via the display device 500.

Further, the peripheral monitoring device described above can also suitably suppress the contact between the work machine and the person around it.

<Other variants>
The above embodiment is merely an example, and the present disclosure may be appropriately modified and implemented without departing from the gist thereof. For example, the processes and means described in the present disclosure can be freely combined and carried out as long as there is no technical contradiction.

In the above embodiment, an example in which the photographing device 200 is configured to include one hemispherical camera 220 has been described, but the photographing device of the present disclosure is not limited to this. FIG. 13 is a diagram showing a modified example of the schematic configuration of the peripheral monitoring device for the work machine in the present disclosure. As shown in FIG. 13, the photographing apparatus of the present disclosure may be configured to include two hemispherical cameras 220. In this case, one hemispherical camera 220 may be arranged before and after the work machine 10.

Further, the processing described as being performed by one device may be shared and executed by a plurality of devices. For example, the extraction processing unit 3032 may be formed in an integrated circuit different from the image processing device 300. At this time, the other integrated circuit is configured to be suitably cooperative with the image processing device 300. Further, the processing described as being performed by different devices may be executed by one device. In an integrated circuit, it is possible to flexibly change what kind of hardware configuration is used to realize each function.

The present disclosure can also be realized by supplying a computer program having the functions described in the above-described embodiment to a computer, and having one or more processors of the computer read and execute the program. Such computer programs may be provided to the computer by a non-temporary computer-readable storage medium that can be connected to the computer's system bus. Non-temporary computer-readable storage media include, for example, any type of disc such as a magnetic disc (floppy (registered trademark) disc, hard disk drive (HDD), etc.), an optical disc (CD-ROM, DVD disc, Blu-ray disc, etc.). Includes read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic cards, flash memory, optical cards, and any type of medium suitable for storing electronic instructions.

10 ... Work machine 100 ... Peripheral monitoring device 200 ... Imaging device 220 ... Hemispherical camera 300 ... Image processing device 303 ... Control unit 400 ... Alarm device

Claims (9)

An imaging device installed on a work machine that photographs the surroundings of the work machine,
A processor that acquires photographed image data representing the surrounding image of the work machine photographed by the photographing apparatus and executes a predetermined process based on the photographed image data.
An alarm device that outputs a predetermined alarm signal and / and a predetermined alarm, and
With
The photographing device includes one or more hemispherical cameras, and is configured such that the optical axis of the hemispherical camera is arranged at a predetermined angle with respect to the ground.
The processor
An extraction process for extracting a person image from the captured image data, and
A distance measuring process for measuring the distance between the work machine and a person around it based on the extracted person image and the installation parameters of the photographing device.
Based on the distance between the work machine and a person around it measured by the distance measuring process, the alarm signal and / and the alarm control process for controlling the output of the alarm by the alarm device are executed. death,
In the distance measuring process, the processor
The foot coordinates, which are the two-dimensional coordinates of the feet of the person image in the captured image data, are acquired, the distance of the foot coordinates from the optical axis of the hemisphere camera, and the focal length of the lens of the hemisphere camera. Based on, the foot angle, which is a predetermined angle between the feet of a person around the work machine and the imaging device, is calculated.
The distance between the work machine and a person around it is measured based on the calculated foot angle and the installation height of the photographing device.
Peripheral monitoring device for work machines. An imaging device installed on a work machine that photographs the surroundings of the work machine,
A processor that acquires photographed image data representing the surrounding image of the work machine photographed by the photographing apparatus and executes a predetermined process based on the photographed image data.
An alarm device that outputs a predetermined alarm signal and / and a predetermined alarm, and
With
The photographing device includes one or more hemispherical cameras, and is configured such that the optical axis of the hemispherical camera is arranged at a predetermined angle with respect to the ground.
The processor
An extraction process for extracting a person image from the captured image data, and
A distance measuring process for measuring the distance between the work machine and a person around it based on the extracted person image and the installation parameters of the photographing device.
Based on the distance between the work machine and a person around it measured by the distance measuring process, the alarm signal and / and the alarm control process for controlling the output of the alarm by the alarm device are executed. death,
In the distance measuring process, the processor
The head image of the person image in the captured image data is acquired, and the distance between the work machine and the surrounding person is measured based on the number of pixels of the head image.
Peripheral monitoring device for work machines. In the alarm control process, the processor
When the distance between the work machine and a person around it becomes equal to or less than a predetermined value, the alarm device commands a predetermined avoidance operation for avoiding contact between the work machine and a person around it. Output a signal,
The peripheral monitoring device for a work machine according to claim 1 or 2. In the extraction process, the processor
A neural network model having an input layer that accepts input of predetermined image data, an intermediate layer that extracts a feature amount representing a person from the input image data, and an output layer that outputs an identification result based on the feature amount. The person image is extracted by inputting the photographed image data into the pre-learning model constructed by performing the training using the data including the image representing the person.
The peripheral monitoring device for a work machine according to any one of claims 1 to 3. A display device configured to be able to display the captured image data and / or predetermined data acquired by processing based on the captured image data taken by the photographing device is further provided.
The peripheral monitoring device for a work machine according to any one of claims 1 to 4. The processor
Until a predetermined time elapses after the power of the peripheral monitoring device is turned on, the data that can be acquired by the extraction process and the distance measurement process is not output to the display device, and only the captured image data is displayed on the display device. Execute the output process,
The peripheral monitoring device for a work machine according to claim 5. The processor
A captured image histogram showing the frequency of pixel values included in the captured image data is generated.
Image sharpening that sharpens the captured image data by synthesizing the smoothed image data acquired based on the smoothed smoothed histogram of the captured image histogram and the captured image data at a predetermined ratio. Further execute the conversion process,
The peripheral monitoring device for a work machine according to any one of claims 1 to 6. The processor is mounted on an integrated circuit installed in the working machine.
The peripheral monitoring device for a work machine according to any one of claims 1 to 7. The photographing device is
A device that automatically controls the exposure of the hemispherical camera to a predetermined reference value according to the brightness of the subject.
In addition
The brightness of the ground around the work machine is acquired, and when the brightness of the ground is equal to or higher than the predetermined first brightness, the exposure of the hemispherical camera is corrected so as to be larger than the reference value, and the ground is corrected. When the brightness of is lower than the first brightness and is equal to or less than the predetermined second brightness, the exposure of the hemispherical camera is corrected so as to be smaller than the reference value, and the brightness of the ground is higher than the second brightness. If the brightness is higher than the first brightness, the exposure of the hemispherical camera is not corrected.
The peripheral monitoring device for a work machine according to any one of claims 1 to 8.

Images