JP2020162438A - Monitoring system and flying robot - Google Patents

Abstract

To enable tracking of an invading object to be continued by estimating a moving direction of the invading object based on a photographed image before the failure of tracking even if tracking fails when capturing and tracking an invading object by sequentially acquired photographed images.SOLUTION: A monitoring system includes a flying robot having an image acquisition part for sequentially acquiring photographed images, which autonomously flies when a detection sensor detects an invading object. The flying robot determines whether or not an object animal is captured in the photographed images acquired while flying in the vicinity of an invasion position. Upon determination that the object animal is not captured in the photographed images after the object animal is captured in the photographed images, the flying robot estimates a moving direction of the object animal based on the object image showing the object animal in the photographed image immediately before the photographed image ceases to capture the object animal, and changes the direction of the flying robot so that the photographing direction of the image acquisition part is changed to the estimated moving direction.SELECTED DRAWING: Figure 1

Description

The present invention relates to a surveillance system and a flying robot that monitor an intruding object that has invaded the surveillance area.

In recent years, crop damage such as wild animals such as wild deer or bears invading a field and damaging the crops cultivated in the field has been increasing. As a countermeasure against such crop damage, measures have been taken such as installing an enclosure to prevent wild animals from invading the field and installing a device to expel wild animals that have invaded the field to the outside of the field. There is.

For example, Patent Document 1 describes a technique in which an unmanned aerial vehicle such as a drone drives a wild animal from a field to a specific place and captures it by spraying a repellent while automatically flying on a predetermined route. ing.

Further, Patent Document 2 describes a technique for monitoring the inside of a parking lot by mounting an imaging device such as an optical camera on an unmanned flying robot and sequentially acquiring captured images including an image showing an object in the parking lot. ing.

JP-A-2018-99044 Japanese Unexamined Patent Publication No. 2014-1198227

In the conventional system in which the flying robot drives the wild animal out of the surveillance area such as the field, when the wild animal becomes accustomed to the flying robot, the wild animal will fly unless the flying robot is brought closer to the wild animal than before. In some cases, they stayed in the surveillance area without escaping from the robot.

When a flying robot that flies while acquiring captured images in sequence is used as a countermeasure against crop damage caused by wild animals, when the flying robot approaches the wild animal, the wild animal is included in the captured image due to the sudden movement of the wild animal. I sometimes escaped to a position where I couldn't. In tracking a person, a car, or the like, even if it is not included in the captured image due to a change in speed or the like, it may be possible to predict from the immediately preceding movement direction or the like and continuously track the person. However, wild animals may make peculiar movements that seem to be irregular, and flying robots cannot predict the movements of wild animals, causing the problem that they cannot capture the appearance of wild animals. Was there. In addition, since the movement of wild animals is often irregular as described above, once the flying robot fails to catch the wild animals, it becomes difficult to track the wild animals thereafter. Sometimes I lost sight of wild animals.

The present invention has been made to solve such a problem, and in the case where an invading object such as a wild animal is captured and tracked by sequentially acquired captured images, even if the tracking fails, the tracking fails. It is an object of the present invention to provide a monitoring system and a flying robot that enable continuous tracking of an intruding object by estimating the moving direction of the invading object based on a previously captured image.

Further, the monitoring system and the flying robot of the present invention improve the estimation accuracy of the moving direction of the invading object based on the moving direction estimated as described above and the direction in which the invading object actually moved, and are peculiar to the invading object. Allows tracking of intruding objects in response to movement.

The monitoring system according to the present invention has a detection sensor that detects an intruding object that has invaded the monitoring area and an image acquisition unit that sequentially acquires captured images, and a flying robot that autonomously flies when the detection sensor detects an intruding object. The detection sensor outputs intrusion position information indicating the intrusion position of the intruding object in response to the detection of the intruding object, and the flight robot controls the flight of the flying robot. The unit, the position acquisition unit that acquires the intrusion position information, the image recognition unit that determines whether or not the target animal has been captured in the captured image acquired during flight near the intrusion position indicated by the intrusion position information, and the captured image. When it is determined that the target animal has been captured, the captured image acquired thereafter has a main control unit that instructs the flight control unit to fly to a position where the target animal can be captured, and the image recognition unit captures the image. If it is determined that the target animal is not captured in the captured image after capturing the target animal in the image, the movement direction of the target animal is estimated based on the target image showing the target animal in the captured image immediately before the target animal cannot be captured. Then, the main control unit instructs the flight control unit to change the direction of the flight robot so that the imaging direction of the image acquisition unit is changed to the estimated movement direction.

Further, in the monitoring system according to the present invention, the image recognition unit uses an animal image indicating an animal and a learning model generated by machine learning using information indicating the type and orientation of the animal as teacher data to generate an captured image. It is preferable to estimate the type and orientation of the target animal estimated from the target image showing the target animal captured in the captured image immediately before the target animal becomes uncaptured, and use the estimated orientation as the movement direction of the target animal. ..

Further, in the monitoring system according to the present invention, the flying robot further has a storage unit, and the image recognition unit captures the target animal in the captured image after the imaging direction is changed by the instruction from the main control unit. The result of the determination is stored in the storage unit as detection probability information, and when the main control unit determines that the target animal is not captured in the captured image after the imaging direction is changed, the imaging direction is changed to another direction. It is preferable to re-instruct the flight control unit to change the direction of the flying robot so as to be changed, estimate the moving direction of the target animal based on the detection probability information, and determine the imaging direction.

Further, in the monitoring system according to the present invention, the flying robot can autonomously fly toward the exit position for expelling the invading object into the monitoring area from the monitoring area, and the image recognition unit invades the monitoring area. When there are a plurality of target animals, it is preferable to specify the target animal having the longest distance from the exit position as the target to be expelled.

Further, in the monitoring system according to the present invention, the flying robot can autonomously fly toward the exit position for expelling the invading object into the monitoring area from the monitoring area, and the image recognition unit invades the monitoring area. When there are a plurality of target animals, the movement directions of the plurality of target animals are estimated, and among the estimated movement directions of the target animals, the target animal having the maximum angle with the direction to the exit position is expelled. It is preferable to specify it as a target.

Further, in the monitoring system according to the present invention, the image recognition unit is an object based on an action value function that defines a value for an action when the estimated moving direction is set as a state and a specific direction of an imaging direction is set as an action. Estimating the moving direction of the animal, the flying robot can capture the target animal in the captured image by changing the direction of the flying robot in the estimated moving direction. It is preferable to further have an update unit that sets a reward for the action and updates the action value function.

The flight robot according to the present invention is a flight robot that has an image acquisition unit that sequentially acquires captured images and can fly autonomously, and has a flight control unit that controls the flight of the flight robot and is acquired during flight. An image recognition unit that determines whether or not the target animal has been captured in the captured image, and if it is determined that the target animal has been captured in the captured image, the flight to a position where the target animal can be captured in the captured image acquired thereafter is performed. It has a main control unit that instructs the flight control unit, and if the image recognition unit determines that the target animal is not captured in the captured image after capturing the target animal in the captured image, it is immediately before the target animal cannot be captured. The moving direction of the target animal is estimated based on the target image showing the target animal in the captured image of the flying robot, and the main control unit changes the imaging direction of the image acquisition unit to the estimated moving direction. Instruct the flight control unit to change the direction.

In the case where the surveillance system and the flying robot according to the present invention capture and track an intruding object by sequentially acquired captured images, even if the tracking fails, the invading object moves based on the captured image before the tracking failure. By estimating the direction, it is possible to continue tracking the intruding object.

It is a figure which shows an example of the schematic structure of a monitoring system. It is a figure which shows an example of the perspective view of a flying robot. It is a figure which shows an example of the flowchart of the main process. It is a schematic diagram for demonstrating an example of a eviction flight. It is a schematic diagram for demonstrating an example of a classification class. It is a schematic diagram which shows an example of the operation of changing the orientation.

Hereinafter, various embodiments of the present invention will be described with reference to the drawings. However, it should be noted that the technical scope of the present invention is not limited to those embodiments, but extends to the inventions described in the claims and their equivalents.

(Monitoring system 1)
FIG. 1 is a diagram showing an example of a schematic configuration of the monitoring system 1. The monitoring system 1 includes a detection sensor 2, a controller 3, and a flight robot 4. The controller 3 connects to the detection sensor 2 and the flight robot 4 via a predetermined communication network 5 (5a, 5b). The predetermined communication network 5 (5a, 5b) is an IP (Internet Protocol) network, a general public line network, a mobile communication telephone network, or the like.

(Detection sensor 2)
The detection sensor 2 is arranged in or around the monitoring area. The detection sensor 2 is, for example, a laser sensor. The detection sensor 2 may be a microwave sensor, an ultrasonic sensor, an image sensor, or the like. The monitoring system 1 may include a plurality of detection sensors 2. For example, the plurality of detection sensors 2 may be arranged in the monitoring area and in the vicinity of the monitoring area, respectively. Further, the plurality of detection sensors 2 may be arranged in at least one of the monitoring area and the periphery of the monitoring area. Further, the detection sensor 2 may be mounted on the flying robot 4, and the detection sensor 2 such as an image sensor mounted on the flying robot 4 invades the monitoring area while the flying robot 4 is patrolling the monitoring area. It may be determined whether or not an object has invaded.

When the detection sensor 2 determines that the invading object invading the monitoring area may be a target animal, the detection sensor 2 transmits a detection signal to the controller 3 via a predetermined communication network 5a. The target animal is a target animal monitored by the monitoring system 1, and is, for example, a wild deer, a wild boar, a bear, or the like. At least one of the size and velocity of the invading object is used to determine the target animal. An index other than size and speed (for example, movement pattern, etc.) may be used to determine the target animal. The detection signal includes intrusion position data of an intruding object. The invasion position data of the invading object is data indicating the three-dimensional coordinates or the plane two-dimensional coordinates in which the invading object is located when the invading object is determined to be the target animal.

For example, when the detection sensor 2 is a laser sensor, the laser irradiation portion of the detection sensor 2 radially transmits a search signal (laser light) to a preset detection range in the monitoring area at a predetermined cycle. Receives the exploration signal that is reflected back by the intruding object within the detection range. The laser irradiation portion includes a laser transmission / reception sensor that rotates within a detection angle on a plane parallel to the ground plane. The detection angle may be 180 degrees or 360 degrees. The laser irradiation portion may include a plurality of laser transmission / reception sensors so that a predetermined range of the vertical viewing angle can be measured. For example, when the laser irradiation portion includes 12 laser transmission / reception sensors within a vertical viewing angle of about 15 degrees, the laser irradiation portion rotates by the detection angle, so that 12 exploration signals (laser light) are simultaneously irradiated. Then, the search signal reflected from the object within the detection range and the vertical viewing angle of about 15 degrees is received.

The detection sensor 2 calculates the distance to the object based on the time difference between the transmission and reception of the search signal. The detection sensor 2 calculates the three-dimensional coordinates of the reflection position corresponding to each search signal based on the position of the detection sensor 2, the direction in which the search signal is transmitted, and the calculated distance. Then, when the calculated three-dimensional coordinates are different from the three-dimensional coordinates of a fixed object such as the ground surface, the detection sensor 2 uses the same invading object to set the coordinates within the specific distance range of the three-dimensional coordinates. Is estimated, and the size of the invading object (for example, the length of the invading object) is calculated. Further, the detection sensor 2 calculates the moving speed when it is estimated that the invading object is moving based on the calculated three-dimensional coordinates and the transmission time of the search signal corresponding to each three-dimensional coordinate.

When the detection sensor 2 determines that the invading object may be a target animal based on at least one of the calculated size and moving speed of the invading object, the detection sensor 2 sends a detection signal to the controller 3 via a predetermined communication network 5a. To send. The intrusion position data of the intruding object included in the detection signal is, for example, data indicating three-dimensional coordinates estimated to be the intruding object. The process of calculating the size and moving speed of the invading object may be executed by the controller 3 described later. In this case, the detection sensor 2 associates the calculated three-dimensional coordinates with the transmission time and transmits the calculated three-dimensional coordinates to the controller 3 via a predetermined network. Further, the detection sensor 2 may be provided by the flight robot 4. In this case, the detection sensor 2 provided in the flight robot 4 irradiates the search signal to detect the invading object while the flight robot 4 automatically flies in the preset flight path in the monitoring area.

(Controller 3)
The controller 3 is an information processing device provided in a monitoring center or the like installed in or around the monitoring area. The controller 3 may be installed near the storage location of the flight robot 4 or may be installed far away from the monitoring area. The controller 3 includes a detection sensor 2 and a communication circuit (not shown) that communicates by wire or wirelessly, and also includes a communication circuit (not shown) that communicates wirelessly with the flight robot 4.

The controller 3 includes a control storage unit (not shown). The control storage unit stores standby position data indicating the standby position of the flying robot 4, exit position data indicating the exit position for expelling an intruding object out of the monitoring area, boundary vector data indicating the boundary of the monitoring area, and the like. .. The exit position is, for example, a position indicating an entrance / exit on the mountain side, not an entrance / exit on the urban side. The boundary vector data is a group of point data including a plurality of latitudes and longitudes on the boundary of the monitored area. Each point indicated by multiple latitudes and longitudes included in the point data group is connected by a line segment in order, and the last point and the first point are connected by a line segment, thereby defining a closed area indicating the monitoring area. To. As the point data group, the latitude and longitude and the corresponding elevation value may be stored. The flight storage unit 46 may store flight space data or obstacle data of the flight robot 4. Further, the exit position may be simply information indicating the direction in which the intruding object is expelled from the monitored area. For example, the information indicating the direction is angle information from magnetic north or the like. In this case, the range in a specific direction is stored as the exit position based on the latitude and longitude information.

When the controller 3 receives the detection signal from the detection sensor 2, the controller 3 acquires the intrusion position data of the intruding object included in the detection signal, and the flight robot is based on the acquired position data and the standby position data read from the flight storage unit 46. The flight path from the standby position of No. 4 to the intrusion position of the intruding object is calculated using a known calculation method (see, for example, Japanese Patent Application Laid-Open No. 2016-181177). The controller 3 transmits a flight instruction to the flight robot 4 via a predetermined communication network 5b, and transmits the intrusion position data, the exit position data, and the route data indicating the flight path of the intruding object to the predetermined communication network 5b. It is transmitted to the flying robot 4 via.

The flight instruction creation process by the controller 3 may be executed by the flight robot 4. In this case, the flight storage unit 46 of the flight robot 4 stores various information stored by the control storage unit, and the controller 3 sends the intrusion position data of the intruding object included in the detection signal to the flight robot 4 together with the flight instruction. Send. Then, the main control unit 43 of the flight robot 4 calculates the flight path from the standby position of the flight robot 4 to the intrusion position of the intruding object, and the exit position data read from the flight storage unit 46 and the intrusion position of the received intruding object. The flight control unit 44 is controlled based on the data and the route data indicating the calculated flight route.

The flight instruction creation process by the controller 3 may be executed by the detection sensor 2. In this case, the detection sensor 2 stores various information stored by the flight storage unit 46. The detection sensor 2 calculates the flight path from the standby position of the flying robot 4 to the intrusion position of the intruding object, and creates a flight instruction including the intrusion position data of the intruding object, the exit position data, and the route data indicating the flight path. , The created flight instruction is transmitted to the flight robot 4 via the controller 3 or directly by wireless communication.

(Flying robot 4)
The flight robot 4 is an unmanned small flying object capable of autonomously flying. The flight robot 4 may be a small aircraft capable of autonomously flying and quasi-autonomously capable of being controlled by operation by a user's wireless controller device or the like. The flight robot 4 is provided with a plurality of rotary wings (rotors) and the like as flight means, and is, for example, a drone, a multicopter, a UAV (Unmanned Aerial Vehicle), and the like.

As shown in FIG. 2, the flying robot 4 has four rotors (propellers) 401 (401a to 401d) on one plane. Each rotor 401 rotates according to the rotation of the motors 402 (402a to 402d) driven by a battery (secondary battery: not shown). Generally, in a single rotor type helicopter, the azimuth is maintained by canceling the anti-torque generated by the main rotor with the moment generated by the tail rotor. On the other hand, in a quad rotor type helicopter such as the flight robot 4, a rotor 401 that rotates in different directions in the front right and the rear left is used around the main body 403, and a rotor 401 that rotates in different directions in the front left and the rear right is used. The anti-torque is offset by. Then, by controlling the rotation speeds (fa to fd) of each rotor 401, it is possible to move various aircraft and adjust their postures.

The flight robot 4 includes an image acquisition unit 41, an image recognition unit 42, a main control unit 43, a flight control unit 44, a position acquisition unit 45, a flight storage unit 46, an update unit 47, and the like, and these are stored in the main body 403. Will be done.

The image acquisition unit 41 is an imaging device provided with a thermal image camera that acquires a thermal image. The thermal image camera detects, for example, the radiant energy of two kinds of wavelengths of electromagnetic radiation from an object, and outputs an captured image created based on the temperature value obtained by the two kinds of radiant energy ratios. The image acquisition unit 41 may be an imaging device including a visible light camera or an infrared thermography camera that outputs an captured image based on visible light. The image acquisition unit 41 sequentially acquires captured images taken at a predetermined frame cycle, and passes the captured images to the image recognition unit 42. The imaging direction of the image acquisition unit 41 is a direction of photographing the front of the flight robot 4 from the front. That is, the image pickup direction of the image acquisition unit 41 is substantially the same as the forward direction of the flight robot 4. The image pickup direction of the image acquisition unit 41 may be a direction tilted vertically downward by a predetermined angle from the forward direction.

Each time the image recognition unit 42 acquires the captured image from the image acquisition unit 41, the image recognition unit 42 determines whether or not the target animal is captured in the captured image. For example, the image recognition unit 42 determines whether or not the captured image includes an object image indicating an intruding object. Then, the image recognition unit 42 determines whether or not the object image is a target image showing the target animal by using the learning model 461 described later. The image recognition unit 42 passes the determination result to the main control unit 43. The learning model 461 is a learning model learned by using machine learning described later.

The main control unit 43 controls the overall operation of the flight robot 4 in an integrated manner, and is, for example, a CPU (Central Processing Unit). The main control unit 43 acquires the detection signal transmitted from the controller 3 and controls the flight so that the image showing the target animal is captured in the captured image, and expels the target animal (invading object) out of the monitoring area. The flight control unit 44, which will be described later, is controlled so as to advance the flight robot 4 to the exit position of.

Upon receiving the flight instruction from the controller 3, the main control unit 43 reads out the application program for flight control and various data stored in the flight storage unit 46.

The flight control unit 44 has four rotors 401 (401a to 401d) so that the flight robot 4 makes flights such as ascending, descending, changing direction (rotation), and advancing in accordance with the instruction from the main control unit 43. ) Is a circuit device for driving and controlling, and is configured based on, for example, an FPGA (Field-Programmable Gate Array). The flight control unit 44 may be configured by the same circuit device as the main control unit 43. The main body 403 stores various sensors such as an orientation sensor, an acceleration sensor, and a gyro sensor (not shown), and the flight control unit 44 is output from the orientation sensor in front of the flying robot and the acceleration sensor output from the orientation sensor. The orientation of the flying robot 4 is controlled based on the acceleration applied to the flying robot 4 (for example, each acceleration in the three-axis directions) and the rotational angular velocity output from the gyro sensor (for example, the rotational angular velocity centered on each of the three axes). To do.

The position acquisition unit 45 receives a signal from a GPS (Global Positioning System) satellite or a quasi-zenith satellite (not shown). The position acquisition unit 45 decodes the signal and acquires time information and the like. Next, the position acquisition unit 45 calculates the pseudo distance from each satellite to the flying robot 4 based on the time information and the like, and solves the simultaneous equations obtained by substituting the pseudo distance to obtain the flying robot 4. Detects self-position (latitude, longitude and altitude). Then, the position acquisition unit 45 associates the position information indicating the detected position with the acquired position time information and periodically outputs the information to the main control unit 43.

The flight storage unit 46 has, for example, a semiconductor memory. The flight storage unit 46 stores a driver program, an operating system program, an application program, data, and the like used for processing in the terminal processing unit 29. The flight storage unit 46 stores data indicating the learning model 461 as data.

The update unit 47 reads the learning model 461 stored in the flight storage unit 46 and executes an update process for updating the learning model 461. The details of the update process will be described later.

(Main processing)
FIG. 3 is a diagram showing an example of a flowchart of the main process executed by the image recognition unit 42 and the main control unit 43 of the flight robot 4.

Upon receiving the flight instruction from the controller 3, the main control unit 43 shifts to the flight operation mode (step S101). In the flight operation mode, the flying robot 4 in the standby mode waiting at the standby position is made to fly to the invasion position of the invading object, and when the target animal is captured in the captured image during the flight near the invasion position, the flight is performed. This mode causes the robot 4 to perform an operation of expelling the target animal to the exit position. The flight instruction is transmitted by the controller 3 that has received the detection signal from the detection sensor 2 that has detected the target animal.

The main control unit 43 receives the flight instruction from the controller 3, the entry position data of the intruding object, the exit position data, and the route data indicating the flight path (step S102). The main control unit 43 instructs the flight control unit 44 to take off the flight robot 4 that has entered the flight mode. Then, the main control unit 43 instructs the flight control unit 44 to fly while referring to the position information output by the position acquisition unit 45 so that the flight robot 4 flies on the flight path indicated by the route data. The flight control unit 44 drives the motor 402 to rotate the roller 401 so that the flight robot 4 performs the flight in response to the instruction of the main control unit 43.

Based on the position information output by the position acquisition unit 45, the main control unit 43 determines whether or not the flight robot 4 has arrived at the intrusion position indicated by the intrusion position data of the intruding object received in step S102 ( Step S103). When the main control unit 43 determines that the flight robot 4 has not arrived at the intrusion position (step S103-No), the main control unit 43 tells the flight control unit 44 to continue the flight until the flight robot 4 arrives at the intrusion position. Instruct.

When the main control unit 43 determines that the flight robot 4 has arrived at the intrusion position (step S103-Yes), the image recognition unit 42 sequentially acquires the captured images from the image acquisition unit 41 in the peripheral region including the intrusion position. Whether or not the target animal is captured is determined by using the learning model 461 described later (step S104). Details of the determination process using the learning model 461 will be described later.

When the invading object detected by the detection sensor 2 in the captured images sequentially acquired by the image recognition unit 42 is not the target animal, or the invading object is not captured, the main control unit 43 means that the target animal is not captured. After making a determination (step S104-No), the process proceeds to step S113. Further, when the main control unit 43 determines that the target animal is captured in the captured images sequentially acquired by the image recognition unit 42 (step S104-Yes), the main control unit 43 determines the target animal as the target animal to be expelled, and expels the target animal. Is started (step S105). Hereinafter, the animal to be expelled may be referred to as a tracked animal. When the main control unit 43 determines the tracking target animal, as shown in FIG. 4, the flying robot 4 moves the tracking target animal out of the monitoring area from the exit position based on the exit position and the position of the tracking target animal. Start a eviction flight to evict.

For example, when the image recognition unit 42 determines that a plurality of target animals are captured in the captured image, the main control unit 43 determines the direction of the exit position in the captured image and the distance from the exit position. The target animal having the longest is may be determined as the track target animal. Further, when it is determined that a plurality of target animals are captured in the captured image, the image recognition unit 42 estimates the movement direction of each of the plurality of target animals among the plurality of animals, and each estimated target. Of the moving directions of the animals, the target animal having the largest angle with the direction to the exit position may be specified as the tracking target animal. The movement direction of the target animal is estimated by the movement direction estimation process described later.

When the eviction flight is started, the image recognition unit 42 executes a labeling process on the target image indicating the tracking target animal captured in the captured image in step S104. In the labeling process, for pixels with high temperature or high brightness in the captured image (for example, pixels having a predetermined value or more compared to the average value of the pixel values in the image), a set of adjacent pixels has a certain size or more. This is a process of extracting a region to be changed as a change region and assigning a unique label in the captured image to each change region. Since the labeling process is a well-known method, detailed description thereof will be omitted. After that, the image recognition unit 42 executes the labeling process on the sequentially acquired captured images, and the main control unit 43 tracks the change region including the tracked animal to which the label is assigned.

FIG. 4 is a schematic diagram for explaining an example of the eviction flight. The flying robot 4a is the flying robot 4 when it is determined that the tracked animal is captured in the captured image (step S104-Yes). During the eviction flight, the main control unit 43 is at a position where the sequentially acquired captured images include (capture) a change region indicating the tracked animal to which the label is assigned, and the imaging direction is in the direction of the exit position. The flight control unit 44 is instructed to fly toward the position. The position where the acquired captured image includes (captures) a change region indicating the tracked animal to which the label is assigned is, for example, an captured image in which the tracked animal to which the label is assigned is located substantially in the center. Is the position of the flying robot 4 to be acquired.

As shown in FIG. 4, the flying robot 4a is in a position where the tracked animal to which the label is assigned is included in the captured image acquired at the position of the flying robot 4a. However, the imaging direction of the flying robot 4a does not face the direction of the exit position. Therefore, the main control unit 43 refers to the exit position indicated by the position information and the exit position data output by the position acquisition unit 45, and instructs the flight control unit 44 to move the flight robot 4a to the left side.

As shown in FIG. 4, the flight robot 4b is a flight robot 4 at a position moved to the left side of the position of the flight robot 4a. The flying robot 4b is in a position where the captured image includes the tracked animal to which the label is assigned. However, the imaging direction of the flight robot 4b has not yet been directed to the exit position. Therefore, the main control unit 43 refers to the exit position indicated by the position information and the exit position data output by the position acquisition unit 45, and instructs the flight control unit 44 to move the flight robot 4b further to the left. ..

As shown in FIG. 4, the flight robot 4c is a flight robot 4 at a position moved to the left side of the position of the flight robot 4b. The flying robot 4c is in a position where the captured image includes the tracked animal to which the label is assigned. Further, the imaging direction of the flight robot 4b is the direction of the exit position. In this way, when it is determined that the tracked animal is captured in the captured image (step S104-Yes), the flight robot 4 moves from the position shown in 4a to the position shown in 4c in response to the instruction from the main control unit 43. To fly.

Next, the main control unit 43 instructs the flight control unit 44 to further move the flight robot 4c to a position within a predetermined distance range from the tracked animal. By bringing the flying robot 4c closer to the tracking target animal in this way, the tracking target animal that dislikes the flying robot 4c can be moved in the direction of the exit position.

When the position in the captured image of the tracked animal to which the label was assigned moves according to the movement of the wild animal, the main control unit 43 assigns the label to the captured images sequentially acquired as described above. The flight control unit 44 is instructed to fly toward a position in which the change region indicating the tracked animal is included (captured) and the imaging direction is the direction of the exit position.

In this way, in the eviction flight, the flight toward the position where the change area indicating the tracked animal to which the label is assigned is included (captured) and the imaging direction is the direction of the exit position, and the tracked animal. The flight to move further to a position within a predetermined distance range is repeatedly carried out.

Whether or not the image recognition unit 42 captures the tracking target animal in the captured images sequentially acquired from the image acquisition unit 41 during the eviction flight (whether or not the change region indicating the tracking target animal to which the label is assigned can be tracked). (Step S106).

When the image recognition unit 42 determines that the tracked animal is captured in the captured images sequentially acquired from the image acquisition unit 41 (step S106-Yes), the main control unit 43 continues the expelling flight (step S106-Yes). Step S107).

When the image recognition unit 42 determines that the tracked animal once captured is not captured in the captured images sequentially acquired from the image acquisition unit 41 (step S106-No), the image recognition unit 42 estimates the moving direction of the tracked animal (step S106-No). S108). In estimating the movement direction, the image recognition unit 42 executes the movement direction estimation process based on the orientation of the tracking target animal captured in the captured image immediately before the tracking target animal cannot be captured in the captured image. The details of the movement direction estimation process will be described later.

The main control unit 43 gives an instruction to the flight control unit 44 to change the direction and / or position of the flight robot 4 so that the image acquisition unit 41 can acquire the captured image of the movement direction estimated by the movement direction estimation process. (Step S109). When the movement direction estimated by the movement direction estimation process is "right", the main control unit 43 rotates the image pickup direction of the flight robot 4 to the right by a predetermined angle (for example, 30 degrees). Instruct. The change of the direction of the flight robot 4 is not limited to the change of the direction corresponding to the movement direction estimated by the movement direction estimation process, and a plurality of predetermined changes after the change of the direction corresponding to the estimated movement direction. The orientation may be changed (for example, the imaging direction of the flying robot 4 is rotated 30 degrees to the right and then 60 degrees to the left).

The image recognition unit 42 determines whether or not the tracking target animal can be tracked after the direction of the flying robot 4 is changed (step S110). For example, the image recognition unit 42 determines that the tracking target animal can be tracked when the tracking target animal is captured in the captured image acquired by the image acquisition unit 41 after the direction of the flight robot 4 is changed. .. The moving direction estimation process may be used to determine whether or not the tracked animal has been captured in the captured image. Further, the image recognition unit 42 determines that the tracking target animal cannot be tracked if the tracking target animal cannot be captured in the captured image acquired by the image acquisition unit 41 after the direction of the flight robot 4 is changed. To do.

When the main control unit 43 determines that the tracking target animal can be tracked (step S110-Yes), the main control unit 43 resumes the expulsion flight to the tracking target animal and returns the process to step S106. When the main control unit 43 determines that the tracking target animal cannot be tracked (step S110-No), the process proceeds to step S112.

During the eviction flight (step S107), the main control unit 43 periodically determines whether or not the tracked animal has been successfully evacuated out of the surveillance area (step S111). For example, the main control unit 43 determines that the flight robot 4 has reached the exit position based on the position information output by the position acquisition unit 45 and the exit position indicated by the exit position data while the expelling flight is continuing. If so, it is determined that the eviction was successful.

When the main control unit 43 determines that the evacuation is successful (step S111-Yes), the main control unit 43 determines whether or not there is another target animal in the area (step S112). For example, when the image recognition unit 42 determines that the tracking target animal has been captured in the captured image after the successful removal, the main control unit 43 determines that there is another target animal. The determination of the presence or absence of other target animals is not limited to the above-mentioned determination method. For example, when the intruder's intrusion position data, exit position data, and route data indicating the flight route are received from the controller 3 together with the flight instruction, as in steps S101 and S102, during the eviction flight or after the eviction is successful, the main control The unit 43 controls the flight control unit 44 so as to fly the flight robot 4 to the intrusion position, and when the image recognition unit 42 determines that the target animal is captured in the captured images sequentially acquired, the unit 43 is within the area. It may be determined that there are other target animals in.

When the main control unit 43 determines that the eviction has not been successful (step S111-No), the process returns to step S106.

When the main control unit 43 determines that there is another target animal in the area (step S112-No), the process returns to step S105.

When the main control unit 43 determines that there are no other target animals in the area (step S112-Yes), the main control unit 43 shifts to the return operation (step S113), and when the flight robot 4 arrives at the standby position, it enters the standby mode. Change and end the main process.

(Movement direction estimation process)
The image recognition unit 42 sequentially acquires captured images from the image acquisition unit 41, and executes a moving direction estimation process on the acquired captured images. The image recognition unit 42 estimates the moving direction using, for example, a learning model 461 generated by machine learning based on CNN (Convolutional Neural Network) such as YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector). Execute the process. The moving direction estimation process is executed in steps S104 and S108 of the main process shown in FIG. In addition, the object recognition method by SSD is described in W.I. Liu, D. Anguelov, D.I. Erhan, C.I. Szegedy, and S. E. Reed, "SSD: Single Shot Multibox Detector", December 29, 2016, [online], <https: // arxiv. org. / Pdf / 1512.02325. Please refer to pdf> etc.

YOLO is one of the object recognition methods using an algorithm that performs image recognition in real time, and is implemented on a machine learning framework such as DarkNET, for example. Hereinafter, an example of the processing of the image recognition unit 42 that realizes the object recognition method based on YOLO in step S108 will be described. First, the image recognition unit 42 divides the captured image immediately before the image acquired from the image acquisition unit 41 into a plurality of grids (for example, one grid divides the captured image into 7 vertical and 7 horizontal regions). A bounding box circumscribing the outer shape of the target animal is set on the grid on which the target animal exists. The bounding box is a rectangular area showing the target animal. The image recognition unit 42 calculates the class probability of the target animal in each bounding box. The class probability is an index showing the probability of both the type of animal and the orientation of the animal.

The image recognition unit 42 uses a learning model generated by machine learning using an animal image showing an animal and information indicating a class corresponding to the animal image as teacher data. The learning model has a multi-layered structure in which a model learning device (not shown) acquires an animal image showing an animal and information indicating a class corresponding to the animal image as teacher data and executes known deep learning learning. It is a learning model generated by learning the weight of each neuron in the neural network. The learning model generation process may be executed by the update unit 47 of the flight robot 4.

As shown in FIG. 5 (a), the classes are, for example, "deer facing forward", "deer facing right front", "deer facing left front", "deer facing right", "deer facing left". They are "deer facing", "deer facing right rear", "deer facing left rear" and "deer facing rear". The target animal may include a human, in which case the class is "human facing forward" or the like. Classes for a plurality of target animals may be provided, in which case the class may be a "right-facing deer herd" or the like. The class may include the posture of the target animal (standing, prone, etc.), in which case the class may be "deer facing forward", "deer standing facing forward", etc. Is. FIG. 5B is an example of an animal image corresponding to each class shown in FIG. 5A.

The image recognition unit 42 uses the learning model to calculate the class probability estimated using the image of the target animal in the bounding box set in the immediately preceding captured image as input information. For example, the image recognition unit 42 has a class probability of 8% for "deer facing forward", 23% for "deer facing right front", 0.3% for "deer facing left front", and "facing right". When "deer" outputs 67%, "deer facing left" outputs 2%, etc., the class "deer facing right" corresponding to the maximum class probability is output, and the movement direction is "right". Judge that there is.

The movement direction estimation process is also used in steps S104 and S110. For example, the image recognition unit 42 calculates the class probability estimated by using the image of the target animal in the bounding box set in the sequentially acquired captured images as input information by using the learned learning model 461. Then, the image recognition unit 42 outputs the class corresponding to the class probability of the highest value. For example, when the image recognition unit 42 outputs the class "deer facing left", the image recognition unit 42 captures the "deer facing left" in the captured image. Then, when the type "deer" is the target animal, in step S104, the image recognition unit 42 determines that the target animal has been captured in the acquired captured image. If the type of animal included in the class is not the target animal, in step S104, the image recognition unit 42 determines that the target animal is not captured in the acquired captured image.

(Modification example 1 of moving direction estimation processing)
Further, the moving direction estimation process is not limited to the above-mentioned process. For example, in step S109, even if the direction of the flight robot 4 is changed so that the captured image of the moving direction estimated by the moving direction estimation process can be acquired by the image acquisition unit 41, the captured image of the moving direction is the tracked animal. If is not captured, the main control unit 43 instructs the flight control unit 44 to rotate the flight robot 4 in a certain direction (counterclockwise, clockwise) until the target animal can be captured.

The update unit 47 stores the result direction in which the target animal can be actually captured, and stores the estimated class, the direction, and the result direction in the flight storage unit 46 in association with each other. Then, the update unit 47 calculates the detection probability in the result direction for each class, and stores the information indicating the calculated detection probability in the flight storage unit 46. For example, if the class immediately before the tracked animal is no longer captured in the captured image is "deer facing left front", the associated result directions are "left front" 5 times, "right front" 3 times, and so on. When "rear" is twice, as shown in FIG. 6, the detection probabilities of "left front" are "0.5" and "rear" in association with the class "deer facing left front" or the movement direction "left front". The detection probability "0.3" for "right front" and the detection probability "0.2" for "rear" are calculated. The information indicating the detection probability is an example of the detection probability information.

By using such a detection probability, when the image recognition unit 42 outputs the class "deer facing left front" and determines that the moving direction is "left front", the initial stage (result direction is stored). In the period when it is not done), the main control unit 43 always instructs the flight control unit 44 so that the captured image on the left front side can be acquired. Then, when learning by the update unit 47 proceeds and the result direction as shown in FIG. 6 is stored, the image recognition unit 42 outputs the class "deer facing left front" and the movement direction is "left front". If this is determined, the main control unit 43 must first instruct the flight control unit 44 to acquire the captured image of the "left front" having the highest detection probability, and must be able to capture the tracked animal. For example, if the flight control unit 44 is re-instructed so that the captured image of the "right front" having the next highest detection probability can be acquired, and if the tracked animal cannot be captured, the detection probability is the next highest. The flight control unit 44 is re-instructed so that the captured image of "rear" can be acquired.

In the example of FIG. 6, the probability that the tracked animal can be detected by acquiring the captured image in the same movement direction as the direction immediately before the tracked animal cannot be captured is the highest, but the detection probability is the next highest. Rather, it is an example in the opposite direction. Although wildlife movements are seemingly irregular and difficult to predict, they may have their own rules. By using the detection probability as described above, it is possible to learn the rules peculiar to wild animals and reflect them in the prediction logic.

An exception rule may be set in the control instruction by the main control unit 43. For example, when the detection probability is less than a certain value (for example, less than "0.2"), or when the number of possible directions is more than a predetermined number (for example, three or more directions), the main control unit 43 flies once. Instruct the flight control unit 44 to raise the robot 4. Then, when the image recognition unit 42 catches the tracking target animal by expanding the field of view of the image acquisition unit 41, the main control unit 43 instructs the flight control unit 44 to lower the flight robot 4. In general, when the wild animals move in random directions, it is more likely that the tracked animals will be caught faster if they fly according to the exception rules described above. When the image recognition unit 42 catches the tracking target animal by raising the flying robot 4 according to the exception rule, the learning is performed in association with the class (movement direction) estimated to be the result direction "rise", and thereafter. It may be reflected in the movement direction estimation process.

(Modification 2 of movement direction estimation processing)
Further, in the movement direction estimation process, the learning model 461 that has been learned by reinforcement learning may be used. For example, in step S109, even if the direction of the flying robot 4 is changed so that the captured image of the moving direction estimated by the moving direction estimation process can be acquired by the image acquisition unit 41, the captured image of the moving direction is the tracking target animal. In the subsequent movement direction estimation process, the image recognition unit 42 uses the learning model 461 that has been learned by reinforcement learning.

Reinforcement learning is, for example, known Q-learning. The action value function Q is used as the learning model 461. In Q-learning, the action value function Q (s, a) representing the value of the action when the action a is selected in the state s is set with the state s of the environment and the action a selected in the state s as independent variables. Be learned. In Q-learning, learning is started in a state where the correlation between the state s and the action a is unknown, and the action value function Q is repeated by repeating trial and error of selecting various actions a in an arbitrary state s. Will be updated. Further, in Q-learning, the action value function Q is configured so that the reward r is obtained when the action a in a certain state s is selected, and the action a in which a higher reward r is obtained is selected. Is learned.

The update formula of the action value function Q is expressed as the following formula (1).

In the formula (1), s _t and a _t is a state and behavior at each time t, the state by action a _t changes from s _t in s _{t + 1.} r t _{+ 1} is a reward obtained by the state changes from s _t in s _{t + 1.} The term of maxQ means the Q when the action a which becomes the maximum value Q at the time t + 1 (which is considered at the time t) is performed. α and γ are learning coefficients and discount rates, respectively, and are arbitrarily set with 0 <α ≦ 1 and 0 <γ ≦ 1.

The learning model 461 is a behavioral value function Q that determines the value as the search direction when the tracked animal is not captured in the captured image in the moving direction. The initial state of the action value function Q is learned in advance using video data or the like acquired while the flight robot 4 is flying in advance. Updating unit 47, in the past, when the tracked animals became not captured in the moving direction of the captured image, to identify the orientation of the target animals in the immediately preceding captured image as a state s _t. Updating unit 47, with respect to the orientation of the target animal of time t, the movement in a specific direction and action a _t. The update unit 47 sets the reward r based on the movement status in a specific direction. The orientation of the target animal is input, for example, by the user visually recognizing the video data.

Updating unit 47, with respect to the orientation of the subject animal at time t, if you change the imaging direction of flying robots 4 in a specific direction, if the subject animal were captured by this change, from 0 as compensation r action a _t set a large value, the target animal by this change in the case that has not been captured, 0 is set as a reward r of the action a _t. It should be noted that the update unit 47 changes the direction of the target animal at time t until the target animal can be captured by changing the imaging direction of the flying robot 4, or until the target animal can be captured. The smaller the time, the larger the reward r may be set. The updating unit 47, a specific direction, the raising or lowering of the flying robots 4, may be included as an action a _t. In this case, the ascending of the flying robot 4 widens the field of view and makes it easier to find the target animal, but the threatening effect on the target animal is reduced. You may.

As a result, in step S108, the image recognition unit 42 obtains a high reward from the captured image immediately before the tracked animal cannot be captured in the captured image by using the learning model 461 using the learned behavioral value function Q. It is possible to output the change direction (rotation direction and / or position of the flying robot 4).

As described in detail above, in the case where the surveillance system and the flying robot according to the present invention are tracking an intruding object by sequentially acquired captured images, even if the tracking fails, the captured image before the tracking failure is obtained. By estimating the moving direction of the invading object based on the above, it is possible to continue tracking the invading object.

It will be appreciated by those skilled in the art that various changes, substitutions and modifications can be made to this without departing from the spirit and scope of the invention.

1: Monitoring system, 2: Detection sensor, 3: Controller, 4: Flight robot, 41: Image acquisition unit, 42: Image recognition unit, 43: Main control unit, 44: Flight control unit, 45: Position acquisition unit, 46 : Flight memory, 461: Learning model, 47: Update

Claims (7)

Surveillance equipped with a detection sensor that detects an intruding object that has invaded the surveillance area, and a flying robot that has an image acquisition unit that sequentially acquires captured images and autonomously flies when the detection sensor detects the invading object. It ’s a system
The detection sensor outputs intrusion position information indicating the intrusion position of the intruding object in response to the detection of the intruding object.
The flying robot
A flight control unit that controls the flight of the flight robot,
The position acquisition unit that acquires the intrusion position information and
An image recognition unit that determines whether or not the target animal has been captured in the captured image acquired during flight near the intrusion position indicated by the intrusion position information.
When it is determined that the target animal has been captured in the captured image, the captured image acquired thereafter has a main control unit that instructs the flight control unit to fly to a position where the target animal can be captured.
When the image recognition unit determines that the target animal is not captured in the captured image after capturing the target animal in the captured image, the target indicating the target animal in the captured image immediately before the target animal cannot be captured. Based on the image, the moving direction of the target animal is estimated, and
The main control unit instructs the flight control unit to change the direction of the flight robot so that the imaging direction of the image acquisition unit is changed to the estimated movement direction.
A monitoring system characterized by that. Immediately before the target animal cannot be captured in the captured image by the learning model generated by machine learning using the animal image showing the animal and the information indicating the type and orientation of the animal as teacher data. The method according to claim 1, wherein the type and orientation of the target animal estimated from the target image showing the target animal captured in the captured image of the above are estimated, and the estimated orientation is set as the movement direction of the target animal. Surveillance system. The flying robot further has a storage unit and has a storage unit.
The image recognition unit stores in the storage unit as detection probability information the result of determining whether or not the target animal has been captured in the captured image after the imaging direction has been changed by the instruction from the main control unit.
When the main control unit determines that the target animal is not captured in the captured image after the imaging direction is changed, the flight robot can change the imaging direction to another direction. Re-instruct the flight control unit to change the direction,
The monitoring system according to claim 1 or 2, wherein the moving direction of the target animal is estimated based on the detection probability information, and the imaging direction is determined. The flying robot can autonomously fly toward an exit position for expelling an object invading the surveillance area out of the surveillance area, and the image recognition unit allows the target animal invading the surveillance area to fly. The monitoring system according to any one of claims 1 to 3, wherein in the case of a plurality of target animals, the target animal having the longest distance from the exit position is specified as the target to be expelled. The flying robot can autonomously fly toward an exit position for expelling an intruding object into the surveillance area out of the surveillance area.
The image recognition unit
When there are a plurality of the target animals that have invaded the monitoring area, the movement directions of the plurality of target animals are estimated.
The monitoring system according to any one of claims 1 to 3, wherein among the estimated movement directions of each target animal, the target animal having the maximum angle with the direction to the exit position is specified as an expulsion target. .. The image recognition unit determines the moving direction of the target animal based on the behavioral value function that determines the value for the behavior when the estimated moving direction is set as the state and the specific direction of the imaging direction is set as the action. Estimate and
The flying robot rewards each action based on the direction in which the imaging direction is changed so that the target animal can be captured in the captured image by changing the direction of the flying robot in the estimated movement direction. The monitoring system according to claim 1, further comprising an update unit for updating the action value function. A flying robot that has an image acquisition unit that sequentially acquires captured images and can fly autonomously.
A flight control unit that controls the flight of the flight robot,
An image recognition unit that determines whether or not the target animal has been captured in the captured image acquired during flight.
When it is determined that the target animal has been captured in the captured image, the captured image acquired thereafter includes a main control unit that instructs the flight control unit to fly toward a position where the target animal can be captured.
When the image recognition unit determines that the target animal is not captured in the captured image after capturing the target animal in the captured image, the target indicating the target animal in the captured image immediately before the target animal cannot be captured. Based on the image, the moving direction of the target animal is estimated, and
The main control unit instructs the flight control unit to change the direction of the flight robot so that the imaging direction of the image acquisition unit is changed to the estimated movement direction.
A flying robot characterized by that.