JP7155062B2 - Surveillance system and flying robot - Google Patents

Description

The present invention relates to a monitoring system and a flying robot that monitor an intruding object that has entered a monitoring area.

In recent years, there has been an increase in crop damage such as wild animals such as wild deer and bears invading fields and destroying crops grown in the fields. As a countermeasure against such damage to crops, measures have been taken such as installing fences to prevent wild animals from entering fields and installing devices to drive out wild animals that have invaded fields from the fields. there is

For example, Patent Literature 1 describes a technology in which an unmanned flying robot such as a drone sprays a repellent while automatically flying along a predetermined route, thereby chasing and capturing wild animals from a field to a specific location. ing.

Further, Patent Document 2 describes a technique for monitoring the inside of a parking lot by having an unmanned flying robot equipped with an imaging device such as an optical camera and sequentially acquiring captured images including images showing objects in the parking lot. ing.

JP 2018-99044 A JP 2014-119827 A

In a conventional system in which a flying robot drives wild animals out of a monitoring area such as a farm field, if the wild animals become accustomed to the flying robots, the wild animals must be brought closer to the flying robots than before. There were cases where the robot stayed in the surveillance area without escaping.

When a flying robot that flies while sequentially acquiring captured images is used to prevent crop damage caused by wild animals, when the flying robot approaches a wild animal, the sudden movement of the wild animal causes the wild animal to be included in the captured image. Sometimes I ran away to a position where I couldn't. In tracking a person or a car, even if it is not included in a photographed image due to a change in speed or the like, there are cases in which it is possible to continue tracking by predicting from the previous movement direction or the like. However, wild animals sometimes make unique movements that seem to have no regularity at first glance. was In addition, since the movement of wild animals is often irregular as described above, once a flying robot fails to capture a wild animal, it becomes difficult to track the wild animal. I even lost track of wild animals.

SUMMARY OF THE INVENTION The present invention has been made to solve such problems. It is an object of the present invention to provide a monitoring system and a flying robot that enable continuous tracking of an intruding object by estimating the moving direction of the intruding object based on the previous captured image.

Further, the monitoring system and flying robot of the present invention improve the accuracy of estimating the direction of movement of the intruding object based on the direction of movement estimated as described above and the direction in which the intruding object actually moved, and Responsive to movement, it enables tracking of intruding objects.

A surveillance system according to the present invention includes a detection sensor that detects an intruding object that has entered a surveillance area and an image acquisition unit that sequentially acquires captured images, and a flying robot that autonomously flies when the detection sensor detects an intruding object. and wherein the detection sensor outputs intrusion position information indicating the intrusion position of the intruding object in response to detection of the intruding object, and the flying robot performs flight control for controlling the flight of the flying robot. a position acquisition unit that acquires intrusion position information; an image recognition unit that determines whether or not a target animal is captured in a captured image acquired during flight in the vicinity of the intrusion position indicated by the intrusion position information; a main control unit that instructs the flight control unit to fly to a position where the target animal can be captured in the captured image acquired thereafter when it is determined that the target animal is captured in the image recognition unit; After capturing the target animal in the image, if it is determined that the target animal is not captured in the captured image, the moving direction of the target animal is estimated based on the target image showing the target animal in the captured image immediately before the target animal is no longer captured. Then, the main control unit instructs the flight control unit to change the direction of the flying robot so that the imaging direction of the image acquisition unit is changed to the estimated movement direction.

Further, in the monitoring system according to the present invention, the image recognition unit uses a learning model generated by machine learning using an animal image representing an animal and information representing the type and orientation of the animal as teacher data to convert the captured image into It is preferable to estimate the type and direction of the target animal estimated from the target image showing the target animal captured in the captured image immediately before the target animal is no longer captured, and set the estimated direction as the moving direction of the target animal. .

Further, in the monitoring system according to the present invention, the flying robot further has a storage unit, and the image recognition unit determines whether or not the target animal has been captured in the captured image after the imaging direction has been changed according to the instruction from the main control unit. is stored in the storage unit as detection probability information, and when it is determined that the target animal is not captured in the captured image after the imaging direction is changed, the main control unit changes the imaging direction to another direction. It is preferable to re-instruct the flight control unit to change the direction of the flying robot so as to change the orientation, estimate the moving direction of the target animal based on the detection probability information, and determine the imaging direction.

Further, in the surveillance system according to the present invention, the flying robot can autonomously fly toward an exit position for expelling an intruding object out of the surveillance area, and the image recognition unit is capable of intruding into the surveillance area. If there are a plurality of target animals, it is preferable to specify the target animal that is the longest from the exit position as the target to be kicked out.

Further, in the surveillance system according to the present invention, the flying robot can autonomously fly toward an exit position for expelling an intruding object out of the surveillance area, and the image recognition unit is capable of intruding into the surveillance area. If there are multiple target animals, the direction of movement of each of the target animals is estimated, and the target animal that forms the largest angle with the direction to the exit position among the estimated moving directions of each target animal is ejected. It is preferable to specify it as a target.

Further, in the monitoring system according to the present invention, the image recognition unit uses the estimated moving direction as the state and the specific direction of the imaging direction as the action, based on the action value function that defines the value for the action, the object The moving direction of the animal is estimated, and the flying robot changes the orientation of the flying robot in the estimated moving direction so that the target animal can be captured in the captured image. It is preferable to further include an updating unit that sets a reward for the action and updates the action-value function.

A flying robot according to the present invention has an image acquisition unit that sequentially acquires captured images and is capable of flying autonomously. An image recognition unit that determines whether or not the target animal has been captured in the captured image, and if it is determined that the target animal has been captured in the captured image, flight toward a position where the target animal can be captured in the captured image acquired thereafter. a main control unit that instructs the flight control unit, and the image recognition unit determines that the target animal is not captured in the captured image after capturing the target animal in the captured image, immediately before the target animal cannot be captured. based on the target image showing the target animal in the captured image of , the main control unit controls the flying robot so that the imaging direction of the image acquisition unit is changed to the estimated moving direction. Instruct the flight controller to change orientation.

In the monitoring system and flying robot according to the present invention, when an intruding object is captured and tracked by captured images sequentially acquired, even if tracking fails, the intruding object moves based on the captured images before the tracking failure. Estimating the direction allows continued tracking of the intruding object.

It is a figure which shows an example of schematic structure of a monitoring system. It is a figure which shows an example of the perspective view of a flying robot. FIG. 7 is a diagram showing an example of a flowchart of main processing; It is a schematic diagram for demonstrating an example of a drive-out flight. FIG. 4 is a schematic diagram for explaining an example of classification classes; FIG. 11 is a schematic diagram showing an example of an orientation change operation;

Various embodiments of the present invention will now be described with reference to the drawings. However, it should be noted that the technical scope of the present invention is not limited to those embodiments, but extends to the invention described in the claims and equivalents thereof.

(Monitoring system 1)
FIG. 1 is a diagram showing an example of a schematic configuration of a monitoring system 1. As shown in FIG. A monitoring system 1 comprises a detection sensor 2 , a controller 3 and a flying robot 4 . The controller 3 connects with the detection sensor 2 and the flying robot 4 via a predetermined communication network 5 (5a, 5b). The predetermined communication network 5 (5a, 5b) is an IP (Internet Protocol) network, a general public line network, a mobile communication telephone network, or the like.

(Detection sensor 2)
The detection sensors 2 are arranged in or around the monitored area. The detection sensor 2 is, for example, a laser sensor. The detection sensor 2 may be a microwave sensor, an ultrasonic sensor, an image sensor, or the like. The monitoring system 1 may comprise multiple detection sensors 2 . For example, multiple detection sensors 2 may be placed in and around the monitored area, respectively. Also, the plurality of detection sensors 2 may be arranged in at least one of the monitored area and the periphery of the monitored area. Furthermore, the detection sensor 2 may be mounted on the flying robot 4, and while the flying robot 4 is patrolling the monitoring area, the detection sensor 2, such as an image sensor mounted on the flying robot 4, may enter the monitoring area. It may be determined whether or not an object has entered.

The detection sensor 2 transmits a detection signal to the controller 3 via the predetermined communication network 5a when it determines that the intruding object that has entered the monitoring area may be the target animal. The target animal is an animal to be monitored by the monitoring system 1, such as wild deer, wild boar, and bear. At least one of the size and speed of the intruding object is used to determine the target animal. Indices other than size and speed (for example, movement patterns, etc.) may be used to determine the target animal. The detection signal includes intrusion position data of the intruding object. The intruding position data of the intruding object is data indicating the three-dimensional coordinates or two-dimensional coordinates of the position of the intruding object when the intruding object is determined to be the target animal.

For example, when the detection sensor 2 is a laser sensor, the laser irradiation part of the detection sensor 2 radially transmits a search signal (laser light) at a predetermined cycle to a preset detection range within the monitoring area, It receives the search signal that has been reflected back by an intruding object or the like within the detection range. The laser irradiation part has a laser transmitting/receiving sensor that rotates within a detection angle on a plane parallel to the ground plane. The detection angle may be 180 degrees or 360 degrees. The laser irradiation portion may include a plurality of laser transmitting/receiving sensors so as to measure a predetermined range of vertical viewing angles. For example, if the laser irradiation part has 12 laser transmitting/receiving sensors within a vertical viewing angle of about 15 degrees, 12 search signals (laser beams) are simultaneously irradiated by rotating the laser irradiation part by the detection angle. and receive probe signals reflected from objects within the detection range and within a vertical viewing angle of approximately 15 degrees.

The detection sensor 2 calculates the distance to the object based on the time difference between the transmission and reception of the search signal. The detection sensor 2 calculates the three-dimensional coordinates of the reflection position corresponding to each search signal based on the position of the detection sensor 2, the direction in which the search signal was transmitted, and the calculated distance. Then, when the calculated three-dimensional coordinates are different from the three-dimensional coordinates of a fixed feature such as the ground surface, the detection sensor 2 detects coordinates within a specific distance range from each other among the three-dimensional coordinates as those of the same intruding object. and calculate the size of the intruding object (for example, the length of the intruding object). Further, when the intruding object is estimated to be moving based on the calculated three-dimensional coordinates and the transmission time of the search signal corresponding to each three-dimensional coordinate, the detection sensor 2 calculates the moving speed.

When the detection sensor 2 determines that the intruding object may be the target animal based on at least one of the calculated size and moving speed of the intruding object, the detection sensor 2 sends a detection signal to the controller 3 via the predetermined communication network 5a. to send. The intrusion position data of the intruding object included in the detection signal is, for example, data indicating the three-dimensional coordinates estimated to be the intruding object. The process of calculating the size and moving speed of the intruding object may be executed by the controller 3, which will be described later. In this case, the detection sensor 2 associates the calculated three-dimensional coordinates with the transmission time and transmits them to the controller 3 via a predetermined network. Alternatively, the detection sensor 2 may be provided by the flying robot 4 . In this case, while the flying robot 4 is automatically flying along a preset flight route within the surveillance area, the detection sensor 2 provided in the flying robot 4 detects an intruding object by emitting a search signal.

(Controller 3)
The controller 3 is an information processing device installed in a monitoring center or the like installed in or around a monitored area. The controller 3 may be installed near the storage location of the flying robot 4, or may be installed far from the surveillance area. The controller 3 includes a communication circuit (not shown) for performing wired or wireless communication with the detection sensor 2 and a communication circuit (not shown) for performing wireless communication with the flying robot 4 .

The controller 3 has a control storage section (not shown). The control storage unit stores standby position data indicating the standby position of the flying robot 4, exit position data indicating the exit position for driving the intruding object out of the monitoring area, boundary vector data defining the boundary of the monitoring area, and the like. . The exit position is, for example, a position indicating a doorway on the mountain side instead of a doorway on the city side. Boundary vector data is a group of point data containing multiple latitudes and longitudes on the boundary of the monitored area. Each point indicated by a plurality of latitudes and longitudes included in the point data group is connected by a line segment in order, and the last point and the first point are connected by a line segment, thereby defining a closed area indicating a monitoring area. be. Latitudes, longitudes, and corresponding elevation values may be stored as point data groups. The flight storage unit 46 may store flightable space data or obstacle data for the flying robot 4 . Alternatively, the exit position may simply be information indicating the direction in which the intruding object is driven out of the monitored area. For example, the information indicating the direction is angle information from magnetic north. In this case, a specific directional range is stored as the exit position based on the latitude and longitude information.

Upon receiving the detection signal from the detection sensor 2, the controller 3 acquires the intrusion position data of the intruding object included in the detection signal, and based on the acquired position data and the standby position data read from the flight storage unit 46, the flying robot 4 is calculated using a known calculation method (see, for example, JP-A-2016-181177) from the standby position to the intrusion position of the intruding object. The controller 3 transmits flight instructions to the flying robot 4 via the predetermined communication network 5b, and transmits the entry position data of the intruding object, the exit position data, and the route data indicating the flight route through the predetermined communication network 5b. to the flying robot 4 via.

The process of creating flight instructions by the controller 3 may be executed by the flying robot 4 . In this case, the flight storage unit 46 of the flying robot 4 stores various information stored by the control storage unit, and the controller 3 sends the intrusion position data of the intruding object included in the detection signal to the flying robot 4 together with the flight instruction. Send. Then, the main control unit 43 of the flying robot 4 calculates the flight path from the standby position of the flying robot 4 to the entry position of the intruding object, and outputs the exit position data read from the flight storage unit 46 and the received entry position of the intruding object. The flight control unit 44 is controlled based on the data and the route data indicating the calculated flight route.

The process of creating flight instructions by the controller 3 may be executed by the detection sensor 2 . In this case, the detection sensor 2 stores various information stored by the flight storage unit 46 . The detection sensor 2 calculates the flight path from the standby position of the flying robot 4 to the entry position of the intruding object, and creates flight instructions including entry position data of the intruding object, exit position data, and route data indicating the flight route. , the created flight instructions are sent to the flying robot 4 via the controller 3 or directly by wireless communication.

(flying robot 4)
The flying robot 4 is an unmanned small flying object capable of autonomous flight. The flying robot 4 may be a small flying object that can fly autonomously and that can be controlled by a user's operation using a wireless controller device or the like. The flying robot 4 has a plurality of rotors as a means of flight, and is, for example, a drone, a multicopter, or a UAV (Unmanned Aerial Vehicle).

As shown in FIG. 2, the flying robot 4 has four rotors (propellers) 401 (401a to 401d) on one plane. Each rotor 401 rotates according to rotation of a motor 402 (402a to 402d) driven by a battery (secondary battery: not shown). Generally, in a single-rotor helicopter, the azimuth angle is maintained by canceling out the reaction torque generated by the main rotor with the moment generated by the tail rotor. On the other hand, in a quad-rotor helicopter such as the flying robot 4, a rotor 401 that rotates in different directions on the front right and rear left about the main body 403 and rotates in different directions on the front left and rear right is used. is used to cancel the anti-torque. By controlling the number of rotations (fa to fd) of each rotor 401, it is possible to adjust various movements and attitudes of the body.

The flying robot 4 includes an image acquisition unit 41 , an image recognition unit 42 , a main control unit 43 , a flight control unit 44 , a position acquisition unit 45 , a flight storage unit 46 , an update unit 47 and the like, which are stored in the main body 403 . be done.

The image acquisition unit 41 is an imaging device equipped with a thermal imaging camera that acquires thermal images. A thermal imaging camera detects, for example, radiant energies of two wavelengths of electromagnetic radiation from an object, and outputs captured images created based on temperature values obtained from the ratio of the two radiant energies. The image acquisition unit 41 may be an imaging device including a visible light camera or an infrared thermography camera that outputs a captured image based on visible light. The image acquisition unit 41 sequentially acquires captured images captured at predetermined frame intervals, and passes the captured images to the image recognition unit 42 . The imaging direction of the image acquisition unit 41 is the direction in which the front of the flying robot 4 is captured from the front. That is, the imaging direction of the image acquisition unit 41 is substantially the same as the forward direction of the flying robot 4 . The imaging direction of the image acquisition unit 41 may be a direction tilted vertically downward by a predetermined angle from the forward direction.

Each time the image recognition unit 42 acquires a captured image from the image acquisition unit 41, the image recognition unit 42 determines whether or not the target animal is captured in the captured image. For example, the image recognition unit 42 determines whether an object image representing an intruding object is included in the captured image. Then, the image recognition unit 42 uses a learning model 461, which will be described later, to determine whether or not the object image is a target image representing a target animal. The image recognition section 42 passes the determination result to the main control section 43 . The learning model 461 is a learning model learned using machine learning, which will be described later.

The main control unit 43 comprehensively controls the overall operation of the flying robot 4, and is, for example, a CPU (Central Processing Unit). The main control unit 43 acquires a detection signal transmitted from the controller 3, controls flight so that an image showing the target animal is captured in the captured image, and drives the target animal (intruding object) out of the monitoring area. A flight control unit 44, which will be described later, is controlled so as to advance the flying robot 4 to the exit position of .

Upon receiving a flight instruction from the controller 3 , the main control unit 43 reads an application program and various data for flight control stored in the flight storage unit 46 .

The flight control unit 44 controls the four rotors 401 (401a to 401d) so that the flying robot 4 can ascend, descend, change direction (rotate), and move forward in accordance with instructions from the main control unit 43. ), and is constructed based on, for example, an FPGA (Field-Programmable Gate Array). The flight control section 44 may be configured with the same circuit device as the main control section 43 . The main body 403 stores various sensors such as an azimuth sensor, an acceleration sensor, and a gyro sensor (not shown). The orientation of the flying robot 4 is controlled based on the acceleration applied to the flying robot 4 (e.g., acceleration in each of the three-axis directions) and the rotational angular velocity output from the gyro sensor (e.g., rotational angular velocity about each of the three axes). do.

The position acquisition unit 45 receives signals from GPS (Global Positioning System) satellites or quasi-zenith satellites (not shown). The position acquisition unit 45 decodes the signal and acquires time information and the like. Next, the position acquisition unit 45 calculates the pseudo-ranges from each satellite to the flying robot 4 based on the time information, etc., and solves the simultaneous equations obtained by substituting the pseudo-ranges to obtain the position of the flying robot 4. Detect self-location (latitude, longitude and altitude). Then, the position acquisition unit 45 associates the position information indicating the detected position with the acquired position time information, and periodically outputs the information to the main control unit 43 .

The flight storage unit 46 has, for example, a semiconductor memory. The flight storage unit 46 stores driver programs, operating system programs, application programs, data, and the like used for processing in the terminal processing unit 29 . The flight storage unit 46 stores data representing the learning model 461 as data.

The update unit 47 reads out the learning model 461 stored in the flight storage unit 46 and executes update processing for updating the learning model 461 . Details of the update processing will be described later.

(main processing)
FIG. 3 is a diagram showing an example of a flowchart of main processing executed by the image recognition unit 42 and the main control unit 43 of the flying robot 4. As shown in FIG.

When the main control unit 43 receives a flight instruction from the controller 3, it shifts to the flight operation mode (step S101). In the flight operation mode, the flying robot 4 in the standby mode, which is on standby at the standby position, flies to the intrusion position of the intruding object. This is a mode in which the robot 4 performs an action of expelling the target animal to the exit position. A flight instruction is transmitted by the controller 3 that has received a detection signal from the detection sensor 2 that has detected the target animal.

The main control unit 43 receives the entry position data of the intruding object, the exit position data, and the route data indicating the flight route together with the flight instruction from the controller 3 (step S102). The main control unit 43 instructs the flight control unit 44 to take off the flying robot 4 that has shifted to the flight mode. Then, the main control unit 43 refers to the position information output by the position acquisition unit 45 and instructs the flight control unit 44 to fly so that the flying robot 4 flies along the flight route indicated by the route data. The flight control unit 44 drives the motor 402 to rotate the rollers 401 so that the flying robot 4 flies according to the instruction from the main control unit 43 .

Based on the position information output by the position acquisition unit 45, the main control unit 43 determines whether or not the flying robot 4 has arrived at the intrusion position indicated by the intrusion position data of the intruding object received in step S102 ( step S103). When the main control unit 43 determines that the flying robot 4 has not arrived at the entry position (step S103-No), the main control unit 43 instructs the flight control unit 44 to continue flying until the flying robot 4 reaches the entry position. instruct.

When the main control unit 43 determines that the flying robot 4 has arrived at the entry position (step S103—Yes), the image recognition unit 42 sequentially acquires captured images from the image acquisition unit 41 in the surrounding area including the entry position. A learning model 461, which will be described later, is used to determine whether or not the target animal is captured in (step S104). Details of the determination process using the learning model 461 will be described later.

When the intruding object detected by the detection sensor 2 in the captured images sequentially acquired by the image recognition unit 42 is not the target animal, or the intruding object cannot be captured, the main control unit 43 determines that the target animal is not captured. After determination (step S104-No), the process proceeds to step S113. Further, when the main control unit 43 determines that the target animal is captured in the captured images sequentially acquired by the image recognition unit 42 (step S104-Yes), the main control unit 43 determines the target animal as an animal to be chased out, and performs the chase flight. is started (step S105). Hereinafter, the animal to be kicked out may be referred to as a tracked animal. When the main control unit 43 determines the tracked animal, the flying robot 4 moves the tracked animal out of the monitoring area from the exit position based on the exit position and the position of the tracked animal, as shown in FIG. Initiate an ejection flight to eject.

For example, when the image recognition unit 42 determines that a plurality of target animals are captured in the captured image, the main control unit 43 determines the direction of the exit position in the captured image, and determines the distance from the exit position. may be determined as the tracked animal. Further, when it is determined that a plurality of target animals are captured in the captured image, the image recognition unit 42 estimates the movement direction of each of the plurality of target animals among the plurality of animals. Among the movement directions of the animal, the target animal that forms the largest angle with the direction to the exit position may be identified as the tracked animal. The moving direction of the target animal is estimated by moving direction estimation processing, which will be described later.

When the chase flight is started, the image recognition unit 42 performs labeling processing on the target image showing the tracked target animal captured in the captured image in step S104. In the labeling process, for pixels with high temperature or high brightness in the captured image (for example, pixels with a predetermined value or more compared to the average value of the pixel values in the image), a group of adjacent pixels is determined to have a certain size or more. This is a process of extracting different regions as changed regions and assigning a unique label in the captured image to each changed region. Since the labeling process is a well-known method, detailed description thereof will be omitted. After that, the image recognition unit 42 performs labeling processing on the captured images that are sequentially acquired, and the main control unit 43 tracks the changed area including the tracked animal to which the label is assigned.

FIG. 4 is a schematic diagram for explaining an example of a chase flight. The flying robot 4a is the flying robot 4 when it is determined that the tracked animal is captured in the captured image (step S104-Yes). During the evacuation flight, the main control unit 43 controls the position in which the sequentially acquired captured images include (capture) a changed area indicating the tracked animal to which the label has been assigned, and the imaging direction is in the direction of the exit position. The flight control unit 44 is instructed to fly toward a certain position. The position where the changed region indicating the tracked animal assigned with the label is included (captured) in the acquired captured image is, for example, the captured image where the tracked animal assigned with the label is positioned substantially at the center. is the position of the flying robot 4 to be obtained.

As shown in FIG. 4, the flying robot 4a is at a position where the tracked animal with the assigned label is included in the captured image acquired at the position of the flying robot 4a. However, the imaging direction of the flying robot 4a does not face the direction of the exit position. Therefore, the main control unit 43 refers to the exit position indicated by the position information and the exit position data output by the position acquisition unit 45, and instructs the flight control unit 44 to move the flying robot 4a to the left.

As shown in FIG. 4, the flying robot 4b is the flying robot 4 at a position moved leftward from the position of the flying robot 4a. The flying robot 4b is at a position where the tracked animal to which the label is assigned is included in the acquired captured image. However, the imaging direction of the flying robot 4b is not yet directed toward the exit position. Therefore, the main control unit 43 refers to the exit position indicated by the position information and the exit position data output by the position acquisition unit 45, and instructs the flight control unit 44 to move the flying robot 4b further to the left. .

As shown in FIG. 4, the flying robot 4c is the flying robot 4 at a position moved leftward from the position of the flying robot 4b. The flying robot 4c is at a position where the tracked animal to which the label is assigned is included in the acquired captured image. Furthermore, the imaging direction of the flying robot 4b faces the direction of the exit position. Thus, when it is determined that the tracked animal is captured in the captured image (step S104-Yes), the flying robot 4 moves from the position 4a to the position 4c in accordance with the instruction from the main control unit 43. fly.

Next, the main control unit 43 instructs the flight control unit 44 to further move the flying robot 4c to a position within a predetermined distance range from the tracked animal. In this way, by causing the flying robot 4c to approach the tracked animal, it becomes possible to move the tracked animal, which dislikes the flying robot 4c, toward the exit position.

When the position in the captured image of the tracked animal to which the label has been assigned moves in accordance with the movement of the wild animal, the main control unit 43 causes the sequentially acquired captured images to have the label assigned to them, as described above. The flight control unit 44 is instructed to fly toward a position where the change area indicating the tracked animal is included (captured) and where the imaging direction is in the direction of the exit position.

In this way, in the ejection flight, there is flight toward a location that includes (captures) a change area indicating the tracked animal assigned a label, and the imaging direction is in the direction of the exit location. and a flight to move further to a position within a predetermined distance range from .

During the chase flight, the image recognition unit 42 determines whether the tracked animal is captured (whether the changed region indicating the tracked animal assigned with the label is tracked) in the captured images sequentially obtained from the image obtaining unit 41. (step S106).

When the image recognition unit 42 determines that the tracked animal is captured in the captured images sequentially acquired from the image acquisition unit 41 (step S106-Yes), the main control unit 43 continues the chase flight ( step S107).

When the image recognition unit 42 determines that the once-captured tracked animal is not captured in the captured images sequentially acquired from the image acquisition unit 41 (step S106-No), the image recognition unit 42 estimates the moving direction of the tracked animal (step S108). In estimating the moving direction, the image recognizing unit 42 executes moving direction estimation processing based on the orientation of the tracked animal captured in the captured image immediately before the tracked animal is no longer captured in the captured image. Details of the movement direction estimation processing will be described later.

The main control unit 43 instructs the flight control unit 44 to change the orientation and/or position of the flying robot 4 so that the image acquisition unit 41 can acquire the captured image in the movement direction estimated by the movement direction estimation process. (Step S109). When the moving direction estimated by the moving direction estimation process is "right", the main control unit 43 causes the flight control unit 44 to rotate the imaging direction of the flying robot 4 rightward by a predetermined angle (for example, 30 degrees). direct to. Note that the change in the orientation of the flying robot 4 is not limited to the change in the orientation corresponding to the movement direction estimated by the movement direction estimation process. The direction may be changed (for example, the imaging direction of the flying robot 4 is rotated rightward by 30 degrees and then leftward by 60 degrees).

After the orientation of the flying robot 4 is changed, the image recognition unit 42 determines whether or not the tracked animal can be tracked (step S110). For example, the image recognition unit 42 determines that the tracked animal can be tracked when the tracked animal is caught in the captured image acquired by the image acquisition unit 41 after the orientation of the flying robot 4 is changed. . Note that moving direction estimation processing may be used to determine whether or not the tracked animal is captured in the captured image. Further, when the tracked animal is not captured in the captured image acquired by the image acquisition unit 41 after the orientation of the flying robot 4 is changed, the image recognition unit 42 determines that the tracked animal cannot be tracked. do.

When determining that the tracked animal can be tracked (step S110-Yes), the main control unit 43 restarts the chase flight for the tracked animal, and returns the process to step S106. When the main control unit 43 determines that the tracked animal cannot be tracked (step S110-No), the process proceeds to step S112.

During the chase flight (step S107), the main controller 43 periodically determines whether or not the tracked animal has been successfully chased out of the monitoring area (step S111). For example, the main control unit 43 determines that the flying robot 4 has reached the exit position based on the position information output by the position acquisition unit 45 and the exit position indicated by the exit position data while the chase flight is continuing. If so, it is determined that the eviction was successful.

When the main control unit 43 determines that the ejection was successful (step S111-Yes), it determines whether or not there is another target animal within the area (step S112). For example, when the image recognition unit 42 determines that the tracked animal has been captured in the captured image after the chase has succeeded, the main control unit 43 determines that there is another target animal. Note that the determination of the presence or absence of other target animals is not limited to the determination method described above. For example, during the evacuation flight or after successful evacuation, similarly to steps S101 and S102, when receiving flight instructions from the controller 3 as well as the entry position data of the intruding object, the exit position data, and the route data indicating the flight route, the main control The unit 43 controls the flight control unit 44 so that the flying robot 4 flies to the entry position. It may be determined that there is another target animal in

When the main control unit 43 determines that the ejection has not succeeded (step S111-No), the process returns to step S106.

When the main control unit 43 determines that there is another target animal within the region (step S112-No), the process returns to step S105.

When the main control unit 43 determines that there is no other target animal within the area (step S112-Yes), it shifts to the return operation (step S113), and when the flying robot 4 reaches the standby position, it enters the standby mode. Change and exit the main process.

(Moving direction estimation process)
The image recognition unit 42 sequentially acquires the captured images from the image acquisition unit 41 and executes moving direction estimation processing on the acquired captured images. The image recognition unit 42 estimates the movement direction using a learning model 461 generated by machine learning based on CNN (Convolutional Neural Network) such as YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector). Execute the process. The movement direction estimation process is performed in steps S104 and S108 of the main process shown in FIG. An object recognition method using SSD is described in W. Liu, D. Anguelov, D.; Erhan, C.; Szegedy, and S. E. Reed, “SSD: Single Shot Multibox Detector”, December 29, 2016, [online], <https://arxiv. org. /pdf/1512.02325. pdf> etc.

YOLO is one of object recognition methods using an algorithm that performs image recognition in real time, and is implemented on a machine learning framework such as DarkNET, for example. An example of the processing of the image recognition unit 42 that implements the YOLO-based object recognition method in step S108 will be described below. The image recognition unit 42 first divides the previous captured image acquired from the image acquisition unit 41 into a plurality of grids (for example, one grid is obtained by dividing the captured image into 7 vertical and 7 horizontal regions), A bounding box that circumscribes the outline shape of the target animal is set in the grid on which the target animal exists. A bounding box is a rectangular area that represents the target animal. The image recognition unit 42 calculates the class probability of the target animal within each bounding box. Class probability is a measure of the accuracy of both the type of animal and the orientation of that animal.

The image recognition unit 42 uses a learning model generated by machine learning using animal images representing animals and information representing classes corresponding to the animal images as teacher data. A learning model is obtained by a model learning device (not shown) acquiring animal images representing animals and information representing classes corresponding to the animal images as teacher data, and executing known deep learning learning to obtain a multi-layered structure. It is a learning model generated by learning the weight of each neuron in the neural network. Note that the learning model generation process may be executed by the updating unit 47 of the flying robot 4 .

As shown in FIG. 5A, the classes include, for example, "deer facing forward", "deer facing right front", "deer facing left front", "deer facing right", and "deer facing left". Deer Facing Right, Deer Facing Right Rear, Deer Facing Left Rear, and Deer Facing Backward. Target animals may include humans, in which case the class may be "forward-facing humans," or the like. A class may be provided for a plurality of target animals, in which case the class might be "a herd of deer facing right". The class may include the posture of the target animal (standing, prone, etc.). is. FIG. 5(b) is an example of an animal image corresponding to each class shown in FIG. 5(a).

The image recognition unit 42 uses the learning model to calculate the estimated class probability using the image of the target animal within the bounding box set in the immediately preceding captured image as input information. For example, the image recognition unit 42 determines that the class probabilities of "deer facing forward" are 8%, "deer facing right front" is 23%, "deer facing left front" is 0.3%, and "deer facing right" is 0.3%. Deer" is 67%, and "deer facing left" is 2%. judge there is.

It should be noted that the movement direction estimation process is also used in steps S104 and S110. For example, the image recognition unit 42 uses the learned learning model 461 to calculate the estimated class probability using the image of the target animal within the bounding box set in the sequentially acquired captured images as input information. Then, the image recognition unit 42 outputs the class corresponding to the class probability with the highest value. For example, when the image recognition unit 42 outputs the class "deer facing left", it captures "deer facing left" in the captured image. Then, if the type "deer" is the target animal, in step S104, the image recognition unit 42 determines that the target animal has been captured in the acquired captured image. If the type of animal included in the class is not the target animal, the image recognition unit 42 determines in step S104 that the captured image does not capture the target animal.

(Modified example 1 of movement direction estimation processing)
Also, the moving direction estimation process is not limited to the above process. For example, in step S109, even if the orientation of the flying robot 4 is changed, the captured image in the moving direction of the animal to be tracked can be acquired by the image acquisition unit 41 so that the captured image in the moving direction estimated by the moving direction estimation process can be acquired by the image acquisition unit 41. If the target animal cannot be caught, the main control unit 43 instructs the flight control unit 44 to rotate the flying robot 4 in a certain direction (counterclockwise or clockwise) until the target animal can be caught.

The updating unit 47 stores the resulting direction in which the target animal was actually captured, associates the estimated class with the direction, and the resulting direction, and stores them in the flight storage unit 46 . Then, the update unit 47 calculates the resulting direction detection probability for each class, and stores information indicating the calculated detection probability in the flight storage unit 46 . For example, if the class immediately before the tracked animal is no longer captured in the captured image is "deer facing left front", the associated result directions are "left front" five times, "right front" three times, When "rear" is two times, as shown in FIG. A detection probability of "0.3" for "right front" and a detection probability of "0.2" for "back" are calculated. Information indicating the detection probability is an example of detection probability information.

By using such a detection probability, when the image recognition unit 42 outputs the class "deer facing left front" and determines that the movement direction is "left front", the initial stage (result direction is stored In the period when it is not performed, the main control unit 43 instructs the flight control unit 44 to always acquire the captured image of the left front. Then, when the learning by the update unit 47 progresses and the result direction as shown in FIG. 6 is stored, the image recognition unit 42 outputs the class "deer facing left front" and the moving direction is "left front". When the main control unit 43 determines that the target animal is to be tracked, the main control unit 43 first instructs the flight control unit 44 to acquire the captured image of the “front left” area, which has the highest probability of detection, and must be able to capture the animal to be tracked. For example, the flight control unit 44 is instructed again so that the captured image of the “front right” with the next highest probability of detection can be acquired. The flight control unit 44 is instructed again so that the captured image of the “rear” can be acquired.

In the example of FIG. 6, the probability that the tracked animal can be detected is the highest by acquiring a captured image in the same moving direction as the direction immediately before the tracked animal is no longer captured. Rather, it is an example of the opposite direction. The movements of wild animals are seemingly irregular and difficult to predict, but there are cases where the movements have specific regularities. By using the detection probability as described above, it becomes possible to learn the regularities peculiar to wild animals and reflect them in the prediction logic.

Note that an exception rule may be set for the control instruction by the main control unit 43 . For example, when the detection probability is less than a certain value (for example, less than “0.2”), or when the number of possible detection directions is a predetermined number or more (for example, 3 directions or more), the main control unit 43 performs a flight once. The flight controller 44 is instructed so that the robot 4 ascends. When the image recognition unit 42 captures the tracked animal by widening the visual field range of the image acquisition unit 41, the main control unit 43 instructs the flight control unit 44 to cause the flying robot 4 to descend. In general, when wild animals move in random directions, flying according to the exception rule as described above increases the possibility of catching the tracked animal more quickly. Note that when the image recognition unit 42 catches the tracked animal by raising the flying robot 4 according to the exceptional rule, the resulting direction “rising” is associated with the estimated class (moving direction) for learning. It may be reflected in the movement direction estimation process.

(Modified example 2 of movement direction estimation processing)
Also, in the movement direction estimation process, the learning model 461 that has been learned by reinforcement learning may be used. For example, in step S109, even if the orientation of the flying robot 4 is changed, the captured image in the moving direction of the animal to be tracked can be acquired by the image acquisition unit 41 so that the captured image in the moving direction estimated by the moving direction estimation process can be acquired by the image acquisition unit 41. is not captured, the image recognition unit 42 uses the learning model 461 that has been learned by reinforcement learning in the subsequent movement direction estimation processing.

Reinforcement learning is, for example, known Q-learning. An action-value function Q is used as the learning model 461 . In Q-learning, the action value function Q(s, a) representing the action value when the action a is selected in the state s is obtained by using the environmental state s and the action a selected in the state s as independent variables. be learned. In Q-learning, learning is started in a state where the correlation between state s and action a is unknown, and by repeating trial and error to select various actions a in arbitrary state s, action-value function Q repeats Updated. In addition, in Q-learning, when an action a in a certain state s is selected, a reward r is obtained, and an action value function Q is learned.

An update formula for the action-value function Q is expressed as the following formula (1).

式（１）において、ｓ_t及びａ_tはそれぞれ時刻ｔにおける状態及び行動であり、行動ａ_tにより状態はｓ_tからｓ_t+1に変化する。ｒ_t+1は、状態がｓ_tからｓ_t+1に変化したことで得られる報酬である。ｍａｘＱの項は、時刻ｔ＋１で最大の価値Ｑになる（と時刻ｔで考えられている）行動ａを行ったときのＱを意味する。α及びγはそれぞれ学習係数及び割引率であり、０＜α≦１、０＜γ≦１で任意に設定される。

In equation (1), s _t and a _t are the state and action at time t, respectively, and the state changes from s _t to s _t+1 by action a _t . r _t+1 is the reward obtained by changing the state from s _t to s _t+1 . The term maxQ means the Q when the action a that becomes the maximum value Q at the time t+1 (considered at the time t) is performed. α and γ are a learning coefficient and a discount rate, respectively, and are arbitrarily set within 0<α≦1 and 0<γ≦1.

The learning model 461 is an action value function Q that determines the value of the search direction when the tracked animal is not caught in the captured image in the movement direction. The initial state of the action-value function Q is learned in advance using image data or the like obtained while the flying robot 4 is flying in advance. In the past, when the tracked animal was not caught in the captured image in the movement direction, the updating unit 47 identifies the orientation of the target animal in the immediately previous captured image as the state s _t . The updating unit 47 sets movement in a specific direction with respect to the orientation of the target animal at time _t as action at. The updating unit 47 sets the reward r based on the state of movement in the specific direction. Note that the direction of the target animal is input by, for example, the user viewing video data.

When the imaging direction of the flying robot 4 is changed to a specific direction with respect to the direction of the target animal at time _t , the update unit 47 sets the reward r for the action at from 0 to If a large value is set and the target animal is not captured by this change, 0 is set as the reward _r for the action at. Note that the update unit 47 updates the orientation of the target animal at time t by changing the imaging direction of the flying robot 4 until the target animal can be captured, or the number of changes until the target animal can be captured. The reward r may be set to a larger value as the time of is shorter. Further, the updating unit 47 may include the _ascending or descending of the flying robot 4 as the action at as the specific direction. In this case, as the flying robot 4 rises, the visual field range is widened, making it easier to find the target animal, but the effect of intimidating the target animal is reduced. may

Accordingly, in step S108, the image recognition unit 42 uses the learning model 461 using the learned action-value function Q to obtain a high reward from the captured image immediately before the tracked animal is no longer captured in the captured image. It is possible to output the changed direction (rotational direction and/or position of the flying robot 4) to be changed.

As described in detail above, the monitoring system and the flying robot according to the present invention track an intruding object using captured images that are sequentially acquired. By estimating the moving direction of the intruding object based on the above, it is possible to continue tracking the intruding object.

It should be understood by those skilled in the art that various changes, substitutions and modifications can be made thereto without departing from the spirit and scope of the present invention.

1: monitoring system, 2: detection sensor, 3: controller, 4: flying robot, 41: image acquisition unit, 42: image recognition unit, 43: main control unit, 44: flight control unit, 45: position acquisition unit, 46 : flight memory unit, 461: learning model, 47: update unit

Claims (7)

A surveillance comprising: a detection sensor that detects an intruding object that has entered a surveillance area; and a flying robot that has an image acquisition unit that sequentially acquires captured images, and that autonomously flies when the detection sensor detects the intruding object. a system,
The detection sensor outputs intrusion position information indicating an intrusion position of the intruding object in response to detection of the intruding object,
The flying robot is
a flight control unit that controls the flight of the flying robot;
a position acquisition unit that acquires the intrusion position information;
an image recognition unit that determines whether or not the target animal is captured in the captured image acquired while flying in the vicinity of the intrusion position indicated by the intrusion position information;
a main control unit that instructs the flight control unit to fly to a position where the target animal can be captured in the subsequently acquired captured image when it is determined that the target animal is captured in the captured image;
When the image recognition unit determines that the target animal is not captured in the captured image after capturing the target animal in the captured image, the image recognition unit indicates the target animal in the captured image immediately before the target animal cannot be captured. estimating the moving direction of the target animal based on the image;
The main control unit instructs the flight control unit to change the orientation of the flying robot so that the imaging direction of the image acquisition unit is changed to the estimated movement direction.
A surveillance system characterized by: Immediately before the target animal is no longer captured in the captured image, the image recognition unit uses a learning model generated by machine learning using an animal image representing the animal and information representing the type and orientation of the animal as teacher data. 2. The method according to claim 1, wherein the type and direction of the target animal estimated from the target image showing the target animal captured in the captured image of the target animal are estimated, and the estimated direction is used as the moving direction of the target animal. monitoring system. The flying robot further has a storage unit,
The image recognition unit stores in the storage unit, as detection probability information, a result of determining whether or not the target animal is captured in the captured image after the imaging direction has been changed according to an instruction from the main control unit,
When it is determined that the target animal is not captured in the captured image after the imaging direction is changed, the main control unit controls the flying robot to change the imaging direction to another direction. reinstructing the flight control unit to change orientation;
3. The monitoring system according to claim 1, wherein the moving direction of the target animal is estimated based on the detection probability information, and the imaging direction is determined. The flying robot is capable of autonomously flying toward an exit position for expelling an intruding object out of the surveillance area. 4. The monitoring system according to any one of claims 1 to 3, wherein when there are a plurality of target animals, the target animal with the longest distance from the exit position is specified as the target to be ejected. The flying robot is capable of autonomously flying toward an exit position for expelling objects that enter the surveillance area out of the surveillance area;
The image recognition unit
when a plurality of the target animals have entered the surveillance area, estimating the movement direction of each of the plurality of target animals;
4. The monitoring system according to any one of claims 1 to 3, wherein, among the estimated moving directions of each target animal, the target animal that forms the largest angle with the direction to the exit position is specified as the target to be kicked out. . The image recognition unit determines the moving direction of the target animal based on an action value function that determines the value of the action when the estimated moving direction is the state and the specific direction of the imaging direction is the action. presume,
The flying robot rewards each action based on the changed imaging direction in which the target animal can be captured in the captured image by changing the direction of the flying robot in the estimated moving direction. 2. The monitoring system according to claim 1, further comprising an updating unit that updates the action-value function by setting . A flying robot having an image acquisition unit that sequentially acquires captured images and capable of autonomous flight,
a flight control unit that controls the flight of the flying robot;
an image recognition unit that determines whether or not the target animal is captured in the captured image acquired during flight;
a main control unit that, when it is determined that the target animal is captured in the captured image, instructs the flight control unit to fly toward a position where the target animal can be captured in the subsequently acquired captured image;
When the image recognition unit determines that the target animal is not captured in the captured image after capturing the target animal in the captured image, the image recognition unit indicates the target animal in the captured image immediately before the target animal cannot be captured. estimating the moving direction of the target animal based on the image;
The main control unit instructs the flight control unit to change the orientation of the flying robot so that the imaging direction of the image acquisition unit is changed to the estimated movement direction.
A flying robot characterized by:

Images