JP7504015B2 - Vehicle - Google Patents

Description

The present invention relates to a traveling vehicle that monitors the surrounding situation based on images captured by an onboard camera.

Patent Document 1 discloses a harvester that inputs images captured by an on-board camera, recognizes objects present in a field, such as people, fallen stalks, weeds, and ridges, and generates object position information indicating the position of the recognized objects on a map. Based on the generated object position information, preset driving operation control and warning notifications are performed.

JP 2019-004772 A

In Patent Document 1, the image recognition module that recognizes people, fallen culms, weeds, and ridges is constructed using machine learning techniques such as a deep learning type neural network. With appropriate learning, this image recognition module can recognize people, fallen culms, weeds, and ridges in a field with high reliability, but since the recognition targets are not physical objects, it cannot recognize state changes such as sinking of the ground or the collapse of stone walls or slopes.

In view of the above-mentioned circumstances, the object of the present invention is to provide a traveling vehicle that detects changes in conditions, such as subsidence of the ground or the collapse of stone walls or slopes, in the area surrounding the vehicle when monitoring the area around the vehicle.

The traveling vehicle according to the present invention, which monitors the surrounding conditions, comprises a vehicle body position calculation unit which calculates the vehicle body position, a camera unit which photographs the area around the vehicle, a surrounding condition detection unit which detects condition changes in the area around the vehicle based on multiple images taken over time from the camera unit and outputs condition change information, an abnormality determination unit which determines whether the condition change is the occurrence of an abnormality based on the condition change information, and a notification information output unit which outputs notification information indicating the occurrence of the abnormality , and the camera unit takes the images when work starts and ends at a specific work site .

In this configuration, a plurality of images captured over time by a camera unit mounted on a vehicle are compared to detect a state change that occurs during a time interval between the captured images. Based on information indicating the state change (such as the capture time of the captured image that is the detection target of the state change, the difference information between the two captured images at each capture interval, and the spatial position of the detected state change), it is determined whether the state change is considered to be an abnormal situation. For example, even if color information differs between the captured images captured over time, if the color change is based on a color change due to a time change, the state change is not an abnormal situation. A state change in which at least a part of a specific object disappears or appears due to a subtle difference in the shooting direction, along with the appearance or disappearance of the object with a slight positional shift, is not considered to be an abnormal situation. If at least a part of a specific object disappears or appears in an image captured at a later time based on the difference information or color information, it is considered to be an abnormal situation. A state change that is considered to be an abnormal situation by the abnormality determination unit is notified as an abnormal situation.
In addition, when a traveling vehicle is used repeatedly on a daily basis in a specific work area such as a farm or orchard, it is possible to detect changes in the state between the end of one work and the start of the next work. If night falls between the end of a work and the start of the next work, the interval is long and there is a high possibility that many changes in the state will occur. In order to quickly detect changes in the state that occur between the end of a work and the start of the next work, it is preferable to capture images at the end of the work and the start of the work and compare these captured images.

In order to detect small changes in a specific area from multiple images taken over time, it is preferable to cover the area around the vehicle with high-resolution images with a small field of view rather than images with a large field of view. To achieve this, multiple images taken over time must be acquired over time and compared, divided by field of view. For this reason, in a preferred embodiment of the present invention, the surrounding condition detection unit includes a field of view calculation unit that calculates the field of view of the captured image based on the vehicle body position at the time of capture and the shooting direction of the camera unit, an image management storage unit that stores the captured image by adding the capture time and the field of view, and a time-dependent change detection unit that detects the state change by comparing the captured images with the same or nearly the same field of view over time.

Furthermore, state changes often occur instantaneously. State changes that disappear immediately after the change can often be ignored. Therefore, it is preferable to capture images at a timing that matches the state change that is particularly required. For this reason, it is preferable for the camera unit to capture multiple images with different fields of view as a panoramic image at a specified timing.

When a traveling vehicle is used repeatedly on a daily basis in a specific work site such as a farm or orchard, it is possible to detect changes in the state between the end of a work task and the start of the next work task. If night falls between the end of a work task and the start of the next work task, the interval is long and there is a high possibility that many changes in the state will occur. In order to quickly detect changes in the state that occur between the end of a work task and the start of the work task, it is preferable that images are taken at the end of the work task and the start of the work task, and these taken images are compared. For this reason, in a preferred embodiment , the predetermined timing is the start of work at the specific work site and the end of work.

When a vehicle that monitors the surrounding conditions is deployed in a forest, orchard, or farm, it is important to consider people or animals that appear around the vehicle as a change in condition that requires caution. For this reason, in a preferred embodiment, the surrounding condition detection unit includes an image recognition section that detects people and animals from the captured image to detect the change in condition.

A person who appears near a moving vehicle poses a risk of collision with the moving vehicle, so it is necessary to notify the person of the approaching vehicle. In addition, an animal that appears near a moving vehicle poses a risk of not only colliding with the vehicle, but also harming the person near the moving vehicle, so a warning is required to scare off the animal. For this reason, when the abnormal situation is based on the detection of a person or animal, it is preferable that a warning sound is output.

When it is determined that an abnormality has occurred, it is necessary to deal with the abnormality appropriately. To do this, it is necessary to identify the location where the abnormality has occurred, and to include the identified location in the notification information. For this reason, in a preferred embodiment, an abnormality location identification unit is provided that identifies the location where the abnormality has occurred based on the vehicle body position and the shooting direction of the camera unit at the time when the abnormality occurred, and the location is included in the notification information.

FIG. 1 is a schematic diagram showing a work vehicle traveling for surveillance in an orchard. FIG. FIG. FIG. 2 is a functional block diagram showing a control system of the work vehicle. FIG. 2 is a schematic diagram illustrating a data flow in monitoring control.

In the following, as an embodiment of a traveling vehicle according to the present invention, a work vehicle that travels to monitor specific work areas such as farms, orchards, and forests will be taken up. In the following explanation, unless otherwise specified, the direction of arrow F shown in Figures 2 and 3 will be referred to as "forward" and the direction of arrow B as "rear." Furthermore, the direction of arrow L shown in Figure 3 will be referred to as "left" and the direction of arrow R as "right." Furthermore, the direction of arrow U shown in Figure 2 will be referred to as "up" and the direction of arrow D as "down."

As shown in FIG. 1, the work vehicle 1 travels while monitoring the condition of the orchard and its surrounding area. The objects of monitoring are changes in condition that occur in the orchard and its surrounding area. For example, changes in condition that are monitored include people or animals entering the orchard, fallen fruit trees, the state of fruit trees, damage to fences and hedges, and even subsidence of the ground and collapse of stone walls and slopes. These changes in condition are detected based on captured images. The work vehicle 1 may travel simply for the purpose of monitoring. Of course, during monitoring driving, it may be equipped with a work device (not shown) to perform tasks such as weeding, spraying pesticides, and transporting luggage.

[Overall configuration of the work vehicle]
As shown in Figures 2 and 3, the work vehicle 1 in this embodiment includes a vehicle body 10, left and right front leg devices 2, and left and right rear leg devices 3. The vehicle body 10 has a box-shaped external shape. The front leg devices 2 are provided so as to extend downward and forward from the lower end at the front end of the vehicle body 10. The rear leg devices 3 are provided so as to extend downward and rear from the lower end at the rear end of the vehicle body 10. The front leg devices 2 support the front part of the vehicle body 10. The rear leg devices 3 support the rear part of the vehicle body 10.

A satellite positioning module 8 for measuring the position of the work vehicle 1 is provided in the center of the top surface of the vehicle body 10. The satellite positioning module 8 includes a satellite antenna for receiving GNSS (global navigation satellite system) signals (including GPS signals).
A camera unit 7 that captures images of the area around the vehicle is provided at the front end of the top surface of the vehicle body 10 .

The front leg device 2 has a front arm 21, a front wheel 22, and a front motor 23. The front arm 21 is a long member that extends downward and forward from the lower end at the front end of the vehicle body 10. The front wheel 22 and the front motor 23 are provided at the lower end of the front arm 21. In other words, the front wheel 22 and the front motor 23 are provided at the bottom of the front leg device 2. In this embodiment, the front motor 23 is a hydraulic motor.

The front wheels 22 are positioned inward in the left-right direction of the vehicle body from the front arm 21. The front motor 23 is positioned outward in the left-right direction of the vehicle body from the front arm 21. The driving force of the front motor 23 is transmitted to the front wheels 22, thereby driving the front wheels 22.

The rear leg device 3 has a rear arm 31, a rear wheel 32, and a rear motor 33. The rear arm 31 is a long member that extends rearward and downward from the lower end at the rear end of the vehicle body 10. The rear wheel 32 and the rear motor 33 are provided at the lower end of the rear arm 31. In other words, the rear wheel 32 and the rear motor 33 are provided at the bottom of the rear leg device 3. In this embodiment, the rear motor 33 is a hydraulic motor. Hydraulic pressure to the hydraulic motor that is the front motor 23 and the hydraulic motor that is the rear motor 33 is supplied by a hydraulic pump driven by an engine (not shown).

The rear wheels 32 are positioned inward in the left-right direction of the vehicle body from the rear arms 31. The rear motor 33 is positioned outward in the left-right direction of the vehicle body from the rear arms 31. The driving force of the rear motor 33 is transmitted to the rear wheels 32, thereby driving the rear wheels 32.

As shown in Figures 2 and 3, the upper end of the front arm 21 is connected to the vehicle body 10. That is, the upper part of the front landing gear 2 is connected to the vehicle body 10. The front arm 21 can swing relative to the vehicle body 10 around a first axis P1 along the left-right direction of the vehicle body. In addition, the front wheel 22 and the front motor 23 swing integrally with the front arm 21 around the first axis P1.

The upper end of the rear arm 31 is connected to the vehicle body 10. That is, the upper part of the rear leg device 3 is connected to the vehicle body 10. The rear arm 31 can swing relative to the vehicle body 10 around a second axis P2 that runs along the left-right direction of the vehicle body. In addition, the rear wheel 32 and the rear motor 33 swing integrally with the rear arm 31 around the second axis P2.

In addition, in this embodiment, both the front motor 23 and the rear motor 33 can be driven in both forward and reverse rotation. When the front motor 23 and the rear motor 33 are driven in forward rotation, the work vehicle 1 travels forward. When the front motor 23 and the rear motor 33 are driven in reverse, the work vehicle 1 travels backward. When the drive of the front motor 23 and the rear motor 33 is stopped, the work vehicle 1 stops.

[Control system configuration]
Fig. 4 is a control function block diagram showing the control function sections related to the present invention. The control unit 6, which is the core element of this control system, is composed of electronic circuits and a computer system, and has an input processing section 6A and an output processing section 6B. The input processing section 6A processes signals input from a group of traveling state detection sensors 41 consisting of sensors and switches arranged in various parts of the work vehicle 1, a group of attitude detection sensors 42 that detects the attitude of the vehicle body 10, and the like, and transfers the signals to each functional section of the control unit 6. The output processing section 6B processes signals generated in each functional section of the control unit 6, and outputs the signals to a traveling control section 6C and an alarm device 44. Fig. 5 shows the flow of data in this control system.

The control unit 6 includes a vehicle body position calculation unit 61, a surrounding condition detection unit 5, an abnormality determination unit 62, an abnormality location identification unit 63, a notification information output unit 64, and a driving command generation unit 65. The vehicle body position calculation unit 61 calculates the vehicle body position, which is the map coordinates (or position coordinates in the orchard coordinate system) of the vehicle body 10, based on the positioning data successively sent from the satellite positioning module 8.

The surrounding condition detection unit 5 detects changes in conditions in the area surrounding the vehicle based on multiple images captured over time by the camera unit 7, and outputs condition change information indicating the content of the change in condition. For this purpose, the surrounding condition detection unit 5 includes an imaging field of view calculation unit 51, an image management storage unit 52, a time-varying change detection unit 53, and an image recognition unit 54. In this embodiment, since the attitude of the vehicle body 10 varies depending on the swinging positions of the front leg unit 2 and the rear leg unit 3, the surrounding condition detection unit 5 includes a vehicle body attitude calculation unit 55 that calculates the attitude of the vehicle body 10 based on signals from the attitude detection sensor group 42.

The field of view calculation unit 51 calculates the field of view of the captured image based on the vehicle position at the time of capture calculated by the vehicle position calculation unit 61, the capture direction of the camera unit 7, and the vehicle attitude calculated by the vehicle attitude calculation unit 55. Of course, if the vehicle attitude at the time of capture is always held in the default attitude, the vehicle attitude is ignored in calculating the field of view. The image management storage unit 52 stores the captured image with the capture time and field of view.

The time-dependent change detection unit 53 reads out captured images with the same or nearly the same field of view from the image management storage unit 52, compares these captured images over time, and detects changes in the state of the area around the vehicle captured in the captured images. Various methods can be used to detect state changes by comparing captured images. For example, there is a method of calculating the difference between two captured images taken with a time lag and estimating state changes from the difference data, or a method of using a neural network that is trained to input two captured images taken with a time lag and output state changes.

The image recognition unit 54 has a machine-learned neural network recognition function, and detects people and animals from the captured images acquired by the camera unit 7. The surrounding condition detection unit 5 also treats the recognition results by the image recognition unit 54 as detection of state changes that have occurred in the area around the vehicle. Therefore, the surrounding condition detection unit 5 outputs information indicating the state changes detected by the time-change detection unit 53 and the people and animals recognized by the image recognition unit 54 as state change information.

As shown in FIG. 5, the status change information output from the surrounding status detection unit 5 is sent to the abnormality determination unit 62. The abnormality determination unit 62 determines whether the status change indicated in the status change information is an abnormal situation based on the status change information. When data such as the time lapse of the status change, the size and color of the area where the status change has occurred, etc. are input, a rule-based algorithm (rule-based AI) determines whether an abnormal situation has occurred. If it is determined that an abnormal situation has occurred, information indicating the content of the abnormal situation is output. Examples of abnormal situations include man-made disasters and natural disasters such as fallen trees, scattered fruit, abnormal growth of weeds, and collapse of fences and hedges. If the image recognition unit 54 recognizes a person or a large animal such as a bear or wild boar that is present in the direction of travel, it is immediately determined that an abnormal situation has occurred. Of course, the same process is performed when the surrounding status detection unit 5 detects the appearance of a person or animal as a status change.

When the abnormality determination unit 62 outputs information indicating the content of the abnormality, the notification information output unit 64 outputs notification information indicating the occurrence of an abnormality. The destination of the notification information from the notification information output unit 64 differs depending on the abnormality information. For example, if the abnormality is the appearance of a person or animal, a warning sound is issued through the notification device 44, such as a buzzer or speaker.

If the abnormality is a man-made disaster or a natural disaster, the notification information output unit 64 generates notification information including the location where the abnormality occurred identified by the abnormality location identification unit 63 in addition to the information indicating the abnormal state, and records it in internal memory. Furthermore, the notification information is sent to the situation management unit 101 of the work management center computer 100 by data communication via the external communication unit 43 in real-time processing or batch processing, and is managed there. The abnormality location identification unit 63 identifies the location where the abnormality occurred based on the vehicle body position when the abnormality occurred and the shooting direction of the camera unit 7 when the abnormality occurred.

Furthermore, if the abnormality is the appearance of a person or animal ahead in the direction of travel, the work vehicle 1 needs to stop or slow down, so the notification information output unit 64 or the abnormality determination unit 62 sends an emergency command requesting such driving control to the driving command generation unit 65. In response to the emergency command, the driving command generation unit 65 generates a driving command to stop or slow down the work vehicle 1, and sends it to the driving control unit 6C. The driving control unit 6C controls the vehicle driving equipment group 7A based on the driving command.

The driving control unit 6C can independently control the left front motor 23a, the right front motor 23b, the left rear motor 33a, and the right rear motor 33b based on a driving command. The work vehicle 1 can be steered by creating a difference between the speed of the left front motor 23a and the left rear motor 33a and the speed of the right front motor 23b and the right rear motor 33b. Of course, it is also possible to configure the front wheels 22 and the rear wheels 32 as steering wheels.

This work vehicle 1 has an automatic driving mode in which it drives automatically based on a preset target driving route, its own vehicle position, and signals from the driving state detection sensor group 41, and a remote manual driving mode using a remote control. In the automatic driving mode, the driving command generation unit 65 generates driving commands based on the automatic driving program and provides them to the driving control unit 6C. In the remote manual driving mode, it generates driving commands based on a remote control operation signal received from the worker and provides them to the driving control unit 6C.

If the field of view of the camera unit 7 is insufficient for monitoring the surrounding area, the camera unit 7 may be equipped with multiple cameras or may be equipped with a camera head that changes the shooting direction of a single camera. In this case, the camera unit 7 can capture multiple images with different fields of view as a full-surrounding image at a specified timing, expanding the monitoring range of the surrounding area.

When the work vehicle 1 finishes monitoring the orchard and before the next monitoring operation is started, the camera unit 7 takes images at the end and start of the operation to quickly detect any changes in the condition of the orchard, and the images taken at the end and start of the monitoring operation are compared. Since it is highly likely that various changes in condition will occur in various places between the end and start of the monitoring operation, the camera unit 7 takes all-around images at the start and end of the operation.

Other embodiments
(1) In the above embodiment, the work vehicle 1 is given as an example of a traveling vehicle that monitors the surrounding conditions. However, the vehicle may be one that only monitors without performing any special work.

(2) The front motor 23, which is the drive device for the front leg unit 2, may be an electric motor instead of a hydraulic motor. Also, the rear motor 33, which is the drive device for the rear leg unit 3, may be an electric motor instead of a hydraulic motor. An engine-type private generator or a battery is used to supply power to the electric motor.

(3) Only the camera unit 7 may be mounted on the vehicle body 10, and the surrounding condition detection unit 5 and the abnormality determination unit 62 may be provided on an external computer capable of exchanging data with the vehicle body 10 via communication.

The configurations disclosed in the above-mentioned embodiments (including other embodiments, the same applies below) can be applied in combination with configurations disclosed in other embodiments, so long as no contradiction occurs. Furthermore, the embodiments disclosed in this specification are merely examples, and the present invention is not limited to these embodiments, and can be modified as appropriate within the scope of the purpose of the present invention.

This invention can be applied to all moving vehicles equipped with cameras.

1: Work vehicle 5: Surrounding condition detection unit 6: Control unit 7: Camera unit 8: Satellite positioning module 10: Vehicle body 41: Group of driving condition detection sensors 42: Group of attitude detection sensors 43: External communication unit 51: Shooting field of view calculation unit 52: Image management storage unit 53: Time-dependent change detection unit 54: Image recognition unit 55: Vehicle body attitude calculation unit 60: Driving control unit 61: Vehicle body position calculation unit 62: Abnormality determination unit 63: Abnormal location identification unit 64: Notification information output unit 65: Driving command generation unit 91: Notification device 100: Work management center computer 101: Situation management unit

Claims (7)

A traveling vehicle that monitors the surrounding situation,
A vehicle body position calculation unit that calculates a vehicle body position;
A camera unit for capturing images of the area around the vehicle;
a surrounding condition detection unit that detects a change in a state in the surrounding area of the vehicle based on a plurality of images captured by the camera unit over time, and outputs information on the change in state;
an abnormality determination unit that determines whether the state change is an occurrence of an abnormal situation based on the state change information;
a notification information output unit that outputs notification information indicating the occurrence of the abnormal situation ,
A traveling vehicle in which the camera unit captures the images at the start and end of work in a specific work area . The traveling vehicle according to claim 1 , wherein the camera unit captures a plurality of images having different photographing fields of view as a whole surrounding image at a predetermined timing. The traveling vehicle according to claim 2 , wherein the predetermined timing is a start time and an end time of work at the specific work site. 4. A traveling vehicle as claimed in any one of claims 1 to 3, wherein the surrounding condition detection unit includes a field of view calculation unit that calculates a field of view of the captured image based on the vehicle body position at the time of shooting and the shooting direction of the camera unit, an image management storage unit that stores the captured image by assigning the shooting time and the field of view, and a time-change detection unit that detects the change in the condition by comparing the captured images having the same or approximately the same field of view over time. The traveling vehicle according to any one of claims 1 to 4, wherein the surrounding condition detection unit includes an image recognition unit that detects people and animals from the captured image as the detection of the change in condition. The traveling vehicle according to claim 5, wherein an alarm sound is output when the abnormal situation is based on the detection of a person or an animal. A traveling vehicle according to any one of claims 1 to 6, further comprising an abnormality location identification unit that identifies the location of the abnormality based on the vehicle body position and the shooting direction of the camera unit at the time of the abnormality occurrence, and the occurrence location is included in the notification information.

Images