JP2019018664A - Imaging control system - Google Patents

Abstract

To enable an operator of an unmanned air vehicle to operate the unmanned air vehicle by recognizing a situation around the unmanned air vehicle and inside of a visually confirmable range.SOLUTION: A control system of an unmanned air vehicle with a camera for photographing an object is provided that comprises: a first imaging apparatus for photographing a photographing object; a second imaging apparatus for photographing an operator of the unmanned air vehicle; unmanned air vehicle control means controlling the operation and photographing of the unmanned air vehicle; video acquisition means acquiring image data including the operator by the second imaging apparatus; user registration information acquisition means holding information on subject identification of the face and body features of the operator; and subject recognition means detecting the operator present in the video acquisition means using the information for subject identification held by the user registration information acquisition means. When the subject recognition means cannot recognize the operator, the control system of the unmanned air vehicle notifies the operator about the impossibility of recognition of the operator.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an imaging control system including an unmanned aerial vehicle attached to an imaging apparatus, and a system that appropriately controls the flight range of the unmanned aerial vehicle based on an image of the attached imaging apparatus.

  In recent years, it has become possible to mount an image pickup device on a small-sized and lightweight aircraft or helicopter that can be remotely operated, and to perform still image / moving image shooting from the sky unattended. This unmanned aerial vehicle is generally called a drone, and is capable of autonomous flight according to a set flight schedule using a position measuring device such as a GPS.

  With the spread of unmanned aerial vehicles as described above, it is considered that the situation where a stable flight cannot be performed for some reason and the vehicle collides with an obstacle or a person increases. In order to prevent such a collision in advance, the technique of Patent Document 1 shown below has been proposed.

  Patent Document 1 makes full use of sensors indicating the flight state of various unmanned air vehicles such as an engine speed sensor, a power supply sensor, and a position sensor to determine whether or not a predetermined standard is always satisfied. It is characterized by returning.

JP 2006-082774 A

  While drone photography is convenient, in the Aviation Law, when flying without the permission of the Minister of Land, Infrastructure, Transport and Tourism, there is a rule that "unmanned aircraft and their surroundings should always be monitored and flying within the visual (direct visual) range". It has been written. Therefore, in patent document 1, when the unmanned air vehicle cannot be confirmed by the operator, the surroundings of the unmanned air vehicle cannot be visually confirmed.

  Therefore, an object of the present invention is to perform control so that an operator of an unmanned air vehicle can recognize and operate the situation around the unmanned air vehicle and a range that can be visually confirmed.

In order to achieve the above object, an imaging control system according to the present invention includes:
A control system for an unmanned air vehicle including a camera for photographing an object, the first imaging device for photographing an object to be photographed, and the second imaging for photographing an operator of the unmanned air vehicle Apparatus, unmanned air vehicle control means for controlling the operation and shooting of the unmanned air vehicle, video acquisition means for obtaining image data including the operator by the second imaging device, and the face and body of the operator User registration information acquisition means for holding characteristic subject identification information; and subject recognition means for detecting the operator existing in the video acquisition means using the subject identification information held by the user registration information acquisition means; And when the operator cannot recognize the operator by the subject recognition means, the operator is notified accordingly.

  According to the imaging control system according to the present invention, it is possible to provide a function that allows an operator of an unmanned air vehicle to recognize and operate the surroundings of the unmanned air vehicle and a range that can be visually confirmed.

It is a schematic diagram of an unmanned flight control system in an embodiment of the present invention. It is a block diagram which shows the structure of the drone in the Example of this invention. It is a block diagram of the imaging device mounted in the drone in the Example of this invention. It is a block diagram of the operation terminal which operates the drone in the Example of this invention and the mounted imaging device. It is a sequence diagram which shows the flight control system of the drone in a 1st Example. It is a sequence diagram which shows the flight control system of the drone in a 2nd Example.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a schematic diagram of an unmanned air vehicle control system (drone control system) according to the present invention. This drone control system is an imaging control system.

  As shown in FIG. 1, the operator 1003 uses the operation terminal 400 to control the drone 100 via the network terminal 1000 including wireless communication. The drone 100 has two main imaging devices 200 and 301 for monitoring the operator 1003 and a main imaging device 200 for aerial photography. The main image pickup device 200 and the sub image pickup devices 300 and 301 are attached to the pan heads 105, 110, and 111 that can be electrically operated for pan operation (rotation by about 180 degrees), tilt operation (rotation by about 90 degrees), and the like. It is possible to operate by subject recognition processing of each imaging device and an instruction from the operation terminal 400. Information such as the drone position and the video imaged by each imaging device can be recorded in the management server 1002 via the network terminal 1000.

  A wireless interface (wireless IF) 1001 is a transmission / reception device compliant with IEEE 802.11b for wireless LAN connection.

  In the present embodiment, the drone 100 and the operation terminal 400 are connected to the management server 1002 via the network terminal 1000. However, the drone 100 and the operation terminal 400 are directly connected to each other by a wireless connection so that the management server function and the drone The present invention can also be implemented in a configuration in which data of information such as the position and video captured by each imaging device is recorded in the drone 100 or a recording device (not shown) inside the operation terminal 400.

   FIG. 2 is a block diagram showing an internal configuration of the drone 100, and FIG. 4 is a block diagram showing an internal configuration of the main imaging device 200 and the sub imaging devices 300, 301 mounted on the drone 100.

  First, the internal configuration of the drone 100 will be described.

  The drone 100 includes a drone driving unit 101, a drone control unit 102, a data transmission / reception unit 103, and a wireless IF 1001. The drone driving unit 101 is a motor that operates the propeller, an ESC that controls the rotation speed of the motor, and the like. The drone driving unit 101 receives an instruction from the operation terminal 400 by the wireless IF 1001 and is controlled by the drone control unit 102.

  The drone control unit 102 is a processor such as a CPU or DSP in which a flight controller or the like for instructing the rotation speed and operation direction of the motor is mounted on the drone driving unit 101, and appropriately reads various programs for operating the drone 100. By executing the above, operation control of each part and processing for the operation by the operation user 1003 are performed.

  The data transmission / reception processing unit 103 collects the captured images of the main imaging device 200 and the sub imaging devices 300 and 301 and the position, altitude, and acceleration information of the drone 100 from the various sensor units 106, and the management server 1002 and the operation terminal 400 by the wireless IF 1001. Send drone data to In addition, an instruction from the operation user 1003 can be received and transmitted to the drone control unit 102.

  The pan head operation units 105, 110, and 111 can perform pan and tilt operations as described above. In addition, a gimbal unit is also provided, so that the impact can be absorbed so that the imaging device is not shaken by the impact of the operation of the drone 100.

  The various sensor units 106 include a GPS that acquires position information of the drone 101, a gyro sensor that acquires inclination information, a compass that acquires orientation information, an atmospheric pressure sensor that acquires altitude information, an acceleration sensor that acquires movement speed, an obstacle, An ultrasonic sensor or the like that acquires the distance to the ground.

  FIG. 3 is a diagram illustrating the internal structure of the main imaging device and the sub imaging device.

  Since the internal configurations of the main imaging device 200 and the sub imaging devices 300 and 301 are the same, description of the internal configuration of the sub imaging device is omitted.

  3, the main imaging apparatus 200 includes a lens unit 201, a lens driving unit 202, an imaging element 203, an image processing unit 204, a subject recognition unit 205, an imaging device control unit 206, a position information acquisition unit 207, and a storage medium 208. Has been.

  The lens unit 201 forms an optical image of the subject on the image sensor 203.

  The lens driving unit 202 controls the diaphragm mechanism that controls the light amount of the optical image, the operation during autofocus (hereinafter referred to as AF) and the operation during IS, and the number of diaphragm stages of the diaphragm mechanism. The control signal is received from the imaging device control unit 206 of the imaging device 200 and operates the lens unit.

  The image processing unit 204 amplifies the image signal obtained by the image sensor 203, performs A / D conversion processing, various correction processing for the image data after A / D conversion, or compression processing for compressing the image data. An image signal processing unit for performing

  The subject recognition unit 205 recognizes a specific object from the image information processed by the image signal processing unit 204 and outputs information such as the range of the object. As specific examples that can be recognized, it is possible to detect previously registered object information, a human body, a face, and the like.

  The imaging device control unit 206 has a microprocessor configuration including a CPU, a RAM, a program storage ROM, and the like. The CPU is connected to each circuit described above via a bus line, and is connected to the program storage ROM. The entire imaging apparatus is controlled by the stored control program.

  The storage medium 208 is not limited to a flash memory or HDD, but may be a memory card or an optical disk as an external memory. The image data output from the image processing unit 204, the recognition object data output from the subject recognition unit 206, and the like are stored. Each data can be transmitted from the data transmission / reception processing unit 103 of the drone 100 to the management server 1002 and the operation terminal 400 via the wireless IF 1001 via the wireless network. In accordance with an instruction from the operation terminal, settings such as recording start, recording end, and recording mode of the imaging apparatus can be temporarily stored and transmitted to the imaging apparatus control unit 206 to perform various controls.

  FIG. 4 shows the internal configuration of the operation terminal 400.

  The operation control unit 401 has a microprocessor configuration including a CPU, a RAM, a program storage ROM, and the like, and controls various drone operation units.

  The movement instructing unit 402 instructs the moving direction of the drone 100. In addition, an instruction for confirming information of the various sensor units 106 can be issued and acquired from the management server 1002 in real time.

  The imaging instruction unit 403 performs recording start, recording end, recording mode setting, and the like of the main imaging device 200 and the sub imaging devices 300 and 301.

  The memory 404 can temporarily hold information obtained from the drone 100 or the management server 1002. Specifically, it is possible to acquire a video from the management server 1002 and perform playback processing based on processing such as fast-forwarding and rewinding the video content.

  In the present embodiment, an unmanned flight system for clarifying a “visible flight range” by monitoring the operator 1003 with a sub imaging device mounted on the drone 100 will be described.

  FIG. 5 is a sequence diagram showing the flight control of the drone in the present embodiment.

  The operating user 1003 acquires the position information of the operator with the position information acquisition unit 405 in the operation terminal 400 before flying the drone (S501). The position information of the operator is transmitted to the management server 1002 via the wireless IF 1001 and the network (S502). The management server 1002 receives the operator position information transmitted from the operation terminal 400 and records data (S701). Based on the received position information, information on whether or not the point where the operator is within the drone's flightable range is acquired and transmitted to the information terminal 400 (S702). The information terminal 400 can send a flight instruction to the drone when it is a flightable area. If it is not in the flightable area, the flight operation function is locked so that the flight instruction cannot be given to the drone, and the user's position information is acquired again (S503). If the flight is possible, the operator 1003 issues a flight instruction to the drone from the operation terminal 400 (S504).

  After receiving the flight instruction, the drone device 100 performs flight control on each function and starts flying (S601). Immediately after the start of flight, the drone's position information is acquired by using the GPS function in the various sensor units 106, and the information is transmitted to the management server 1002 (S602). The management server 1002 acquires drone position information from the drone device 100 (S703).

  The operator 1003 issues a shooting instruction from the operation terminal 400 (S505). Recording processing of the main imaging device 200 and the sub imaging devices 300 and 301 mounted on the drone device 100 is started (S603).

  The sub imaging devices 300 and 301 recognize the operator 1003 using the subject recognition unit 205 (S604). Information on the face and human body of the operator 1003 is registered in the storage medium 208 in advance. It is possible to track the recognized user until the recognition is lost. At this time, if the face of the operator 1003 is outside the specific range of the recorded image, the camera platform operating units 110 and 111 in the drone device 100 are operated so as to automatically control to be at the center of the recorded video. Also good.

[When the operator cannot recognize]
When the operator 1003 cannot recognize the sub imaging devices 300 and 301, the following control is performed (S605). For example, when an obstacle (a building, a large plant, or the like) appears between the operator 1003 and the drone device 100, the operator cannot be recognized. At this time, first, the management server 1002 is requested for the position information of the operator 1003 and the position information of the drone device 100 (S606). The position information of the operator 1003 and the position information of the drone device 100 are received from the management server, and the drone is controlled so as to return the flight route recorded in the recording medium 205 in advance.

  In parallel with S606, an abnormality notification due to the fact that it cannot be recognized by the operator is transmitted to the operation terminal 400 (S607). When the operation terminal 400 receives an abnormality notification from the drone device 100, the operation terminal 400 sounds a warning sound or displays a warning on the operation terminal (S506). In addition, when an abnormality notification is received, a movement recommendation is displayed so as to move to a range that can be seen by the operator. (S507).

  If the operator can recognize, after performing the recording process, the video data is transmitted to the management server 1002 (S609, S706).

  As described above, by monitoring the operator 1003 with the sub imaging devices 300 and 301, even if the drone becomes invisible due to an obstacle, the operator can be warned and the surrounding situation can be visually confirmed. It is urged to operate the drone without fail, and it is possible to fly without harming the drone flight rules.

  The schematic diagram of the unmanned flight control system of the second embodiment, the block diagram showing the internal configuration of the drone device, the block diagram showing the internal configuration of the imaging device, and the block diagram showing the internal configuration of the operating device are the same as those of the first embodiment. Is omitted. In addition, regarding the sequence diagram showing the flight control system of the drone of FIG. 6, the same parts as those in the sequence described in the first embodiment are omitted, and different parts will be described.

  FIG. 6 is a sequence diagram showing the flight control of the drone in the present embodiment.

  When the operator 1003 cannot recognize the sub imaging devices 300 and 301, the following control is performed (S605). For example, let us consider a case where the drone is operated far and the operator cannot confirm from the visual confirmation and the sub imaging devices 300 and 301.

  First, the management server 1002 is requested for the position information of the operator 1003 and the position information of the drone device 100 (S606 ′). The position information of the operator 1003 and the position information of the drone device 100 are received from the management server 1002 (S705 ′), and the sub-imaging device 300 or 301 in the direction of the operator from the position information and information from each sensor of the various sensor units 106. The pan / tilt head is operated so that the lens is directed to zoom the imaging device to the telephoto side (S607 ′).

  It is determined whether the operator can recognize the zoom operation (S608 ′). If it can be recognized, the recording process is continued, and the video data is transmitted to the management server 1002 when the recording ends (S609).

  If the operator cannot be recognized by the operation in S608 ′, the operation terminal 400 is notified to notify the abnormality (S506 ′). In addition, when an abnormality notification is received, a movement recommendation is displayed so as to move to a range that can be seen by the operator. (S507 ').

  As described above, when the operator 1003 is monitored by the sub imaging devices 300 and 301 and the subject cannot be detected halfway, the zoom operation and the pan head operation of the imaging device are performed based on the positional relationship between the drone and the operator. The operation is performed appropriately, and it is determined again whether the drone is in a visible range from the operator. If the subject cannot be recognized yet, a warning is given to the operator. By this operation, the operator is encouraged to operate the drone in a range where the drone can be visually confirmed, and it is possible to fly without harming the drone flight rules.

100 drone, 400 operation terminal, 1000 network terminal

Claims (6)

A control system for an unmanned air vehicle equipped with a camera for photographing an object,
A first imaging device for photographing an object to be photographed;
A second imaging device for photographing the operator of the unmanned air vehicle;
Unmanned air vehicle control means for controlling the operation and shooting of the unmanned air vehicle,
Video acquisition means for acquiring image data including the operator by the second imaging device;
User registration information acquisition means for holding subject identification information of the operator's face and body features;
Subject recognition means for detecting the operator present in the video acquisition means using the subject identification information held by the user registration information acquisition means,
An imaging control system characterized by notifying the operator when the subject recognition means cannot recognize the operator. A control system for an unmanned air vehicle equipped with a camera for photographing an object,
A first imaging device for photographing an object to be photographed;
A second imaging device for photographing the operator of the unmanned air vehicle;
Unmanned air vehicle control means for controlling the operation and shooting of the unmanned air vehicle,
Video acquisition means for acquiring image data including the operator by the second imaging device;
User registration information acquisition means for holding subject identification information of the operator's face and body features;
Subject recognition means for detecting the operator existing in the video acquisition means using subject identification information held by the user registration information acquisition means;
A pan head device for performing pan and tilt operations on the first and second imaging devices, and
An imaging control system, wherein when the operator can no longer recognize the operator by the subject recognition means, the second imaging device performs an enlargement process to enlarge the operator to a range where the operator can be recognized.   The imaging control system according to claim 1, further comprising a warning unit that emits a warning sound when receiving a notification that the operator cannot be recognized.   The imaging control system according to claim 1, further comprising warning means for displaying a warning when receiving a notification that the operator cannot be recognized.   The imaging according to claim 1, further comprising: a flight control unit that acquires flight path information from the flight path recording unit and controls to return to the flight path when the operator cannot recognize. Control system.   3. The imaging control system according to claim 2, wherein the head device operates in a tracking direction when tracking is performed during subject recognition by the subject recognition unit.