JP2014212479A - Control device, control method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a control device making it possible to maneuver a flying body mounted with a camera through an intuitive operation.SOLUTION: A control device comprises: an image display unit 210 that acquires, from a flying body, an image captured by an imaging device 101 provided in the flying body, and displays the image; and a flight instruction generation unit 232 that generates a flight instruction for the flying body on the basis of content of an operation to the image captured by the imaging device and displayed by the image display unit.

Description

  The present disclosure relates to a control device, a control method, and a computer program.

  A technique relating to a method of taking a photograph in which a camera is attached to a flying object that can be operated wirelessly and taken by the camera is disclosed (for example, see Patent Document 1). By attaching a camera to the flying object, it is possible to take a picture from the sky or from a place where the tripod cannot be stood. In addition, by attaching a camera to the aircraft and taking images, the cost can be reduced compared to using a real airplane or helicopter, imaging can be performed safely, imaging can be performed even in low altitudes and narrow places, and the target is approached. Thus, various advantages such as being able to take an image are brought about.

JP 2006-27448 A

  In order to execute the above-described imaging method, it is necessary to further operate the camera after operating the flying object using a proportional system (propo) or the like. Therefore, in order to execute the above-described imaging method, considerable training for operating the flying object freely is necessary, and further training for operating the camera attached to the flying object is indispensable. .

  Therefore, the present disclosure provides a new and improved control device, control method, and computer program capable of maneuvering a flying object equipped with a camera by an intuitive operation.

  According to the present disclosure, an image display unit that acquires and displays an image captured by an imaging device provided in a flying object from the flying object, the image capturing device captures an image, and the image display unit displays the image. There is provided a control device including a flight instruction generation unit that generates a flight instruction of the flying object based on the content of an operation on an image.

  Further, according to the present disclosure, the image captured by the imaging device provided in the flying object is acquired and displayed from the flying object, and the image displayed by the imaging device is captured and displayed. And a step of setting the content of the operation performed on the aircraft as a flight instruction of the flying object.

  Further, according to the present disclosure, the computer displays the image captured by the imaging device included in the flying object, acquired from the flying object, and the imaging device captures and displays the image. And a step of setting the content of the operation performed on the current image as a flight instruction of the flying object is provided.

  As described above, according to the present disclosure, it is possible to provide a new and improved control device, control method, and computer program capable of maneuvering a flying object equipped with a camera by an intuitive operation.

1 is an explanatory diagram illustrating an external appearance example of an imaging system according to an embodiment of the present disclosure. FIG. 1 is an explanatory diagram illustrating an external appearance example of an imaging system according to an embodiment of the present disclosure. FIG. 1 is an explanatory diagram illustrating an external appearance example of an imaging system according to an embodiment of the present disclosure. FIG. FIG. 3 is an explanatory diagram illustrating an appearance example of a flying device 100 included in an imaging system 10 according to an embodiment of the present disclosure. FIG. 3 is an explanatory diagram illustrating an appearance example of a flying device 100 included in an imaging system 10 according to an embodiment of the present disclosure. It is explanatory drawing which shows the modification of the flight apparatus of one Embodiment of this indication. It is explanatory drawing which shows the modification of the flight apparatus of one Embodiment of this indication. It is explanatory drawing which shows the structural example of the support | pillar 111 which supports the rotor covers 102a-102d. It is explanatory drawing which shows a mode that the rotor cover 102a rotates along the slit part 112 of the support | pillar 111. FIG. It is explanatory drawing which shows a mode that the rotor cover 102a rotates along the slit part 112 of the support | pillar 111. FIG. 4 is an explanatory diagram showing an example of a state in which a user controls the operation of the flying device 100 using a controller 200. FIG. 3 is an explanatory diagram illustrating a functional configuration example of a flying device 100 and a controller 200 according to an embodiment of the present disclosure. FIG. 5 is a flowchart illustrating an operation example of the imaging system 10 according to an embodiment of the present disclosure. 4 is an explanatory diagram illustrating an example of information displayed on a display unit 210 of a controller 200 according to an embodiment of the present disclosure. FIG. 4 is an explanatory diagram illustrating an example of information displayed on a display unit 210 of a controller 200 according to an embodiment of the present disclosure. FIG. 4 is an explanatory diagram illustrating an example of information displayed on a display unit 210 of a controller 200 according to an embodiment of the present disclosure. FIG. It is an example of a display of information on the display unit 210 when the user wants to set the position specified by the tap at the center of the image. It is an example of a display of information on the display unit 210 when the user wants to set the position specified by the tap at the center of the image. It is explanatory drawing which shows the state which performed operation which the user dragged on the display part. FIG. 10 is an explanatory diagram showing a state after the user performs a drag operation on the display unit 210. It is explanatory drawing which shows the state which the user performed pinch operation on the display part. It is explanatory drawing shown about control of the flight of the flying device 100 by image feedback. FIG. 11 is an explanatory diagram illustrating an example of a correspondence relationship between a user operation on an image captured by the imaging device 101 and movement of the flying device 100; FIG. 11 is an explanatory diagram illustrating an example of a correspondence relationship between a user operation on an image captured by the imaging device 101 and movement of the flying device 100; FIG. 11 is an explanatory diagram illustrating an example of a correspondence relationship between a user operation on an image captured by the imaging device 101 and movement of the flying device 100; FIG. 11 is an explanatory diagram illustrating an example of a correspondence relationship between a user operation on an image captured by the imaging device 101 and movement of the flying device 100; 4 is an explanatory diagram illustrating an example of information displayed on a display unit 210 of a controller 200. FIG. FIG. 11 is an explanatory diagram illustrating an example of a correspondence relationship between a user operation on an image captured by the imaging device 101 and movement of the flying device 100; FIG. 11 is an explanatory diagram illustrating an example of a correspondence relationship between a user operation on an image captured by the imaging device 101 and movement of the flying device 100; It is explanatory drawing which shows the state which the flight apparatus 100 is located inside the predetermined flight range 21 centering on the initial position 20. FIG. It is explanatory drawing which shows the state which the flight apparatus 100 is going to fly beyond the predetermined flight range 21 centering on the initial position 20. FIG. It is explanatory drawing which shows the state which the flying apparatus 100 is going to land automatically. 4 is an explanatory diagram showing an outline of the operation of the flying device 100. FIG. 5 is a flowchart illustrating an operation example of the imaging system 10 according to an embodiment of the present disclosure. It is explanatory drawing which shows the example of control of the position and attitude | position of the flying apparatus 100 by image recognition. It is explanatory drawing which shows the example of control of the position and attitude | position of the flying apparatus 100 by image recognition. It is explanatory drawing which shows the example of control of the position and attitude | position of the flying apparatus 100 by image recognition. It is explanatory drawing which shows the example of control of the position and attitude | position of the flying apparatus 100 by image recognition.

  Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

The description will be made in the following order.
<1. One Embodiment of the Present Disclosure>
[External appearance of imaging system]
[Example of functional configuration of imaging system]
[Operation example of flight device and controller]
<2. Summary>

<1. One Embodiment of the Present Disclosure>
[External appearance of imaging system]
First, an appearance example of an imaging system according to an embodiment of the present disclosure will be described with reference to the drawings. 1 to 3 are explanatory diagrams illustrating an appearance example of an imaging system according to an embodiment of the present disclosure. Hereinafter, an appearance example of an imaging system according to an embodiment of the present disclosure will be described with reference to FIGS.

  FIG. 1 is an explanatory diagram illustrating an external appearance example of an imaging system 10 according to an embodiment of the present disclosure in a perspective view, and FIG. 2 illustrates an imaging system according to an embodiment of the present disclosure. 10 is an explanatory view showing an external appearance example of FIG. 10 in a perspective view from the opposite side of the perspective view shown in FIG. 1. Also, FIG. 3 is an explanatory diagram showing an external appearance example of the imaging system 10 according to an embodiment of the present disclosure in a perspective view from the bottom surface side.

  An imaging system 10 according to an embodiment of the present disclosure includes a flying device 100 and a controller 200 that controls the flying device 100. The flying device 100 can fly by rotating the rotor under the control of the controller 200. However, when the flying device 100 is not caused to fly, the controller 200 has a configuration that allows the rotor to be accommodated as shown in FIGS.

  The flying device 100 illustrated in FIGS. 1 to 3 can fly by four rotors. 1 to 3 show rotor covers 102a to 102d that protect the rotors. The imaging system 10 according to an embodiment of the present disclosure can accommodate the rotor covers 102a to 102d inside the controller 200 as shown in FIGS.

  The imaging system 10 according to an embodiment of the present disclosure has a configuration in which the controller 200 can slide in the direction of A in FIG. The imaging system 10 according to an embodiment of the present disclosure has a configuration in which the controller 200 is removed from the flying device 100 when the controller 200 is slid in the direction of A in FIG.

  The flying device 100 includes an imaging device 101 that captures a still image or a moving image. The imaging device 101 includes an imaging element such as a lens, a charge coupled device (CCD) image sensor, and a complementary metal oxide semiconductor (CMOS) image sensor. The imaging device 101 provided in the flying device 100 performs imaging of a still image or a moving image under the control of the controller 200, and provides the captured image to the controller 200.

  The controller 200 controls the flight state of the flying device 100 and the imaging of an image by the imaging device 101 provided in the flying device 100. The controller 200 controls such a flight state and imaging by wireless communication. As shown in FIG. 1, the controller 200 includes a display unit 210. The functional configuration of the controller 200 for controlling the flight state of the flying device 100 and the imaging of an image by the imaging device 101 provided in the flying device 100 will be described in detail later.

  The display unit 210 includes a flat panel display device such as a liquid crystal display device or an organic EL display device, and can display an image captured by the imaging device 101 and information for controlling the operation of the flying device 100. The display unit 210 is provided with a touch panel, and the user can directly operate the information displayed on the display unit 210 by touching the display unit 210 with a finger or the like.

  So far, the appearance example of the imaging system 10 according to an embodiment of the present disclosure has been described with reference to FIGS. Next, an example of the appearance of the flying device 100 will be described in more detail.

  FIG. 4 is an explanatory diagram illustrating an appearance example of the flying device 100 included in the imaging system 10 according to an embodiment of the present disclosure. Hereinafter, an appearance example of the flying device 100 will be described with reference to FIG.

  FIG. 4 illustrates a state where the controller 200 is removed from the imaging system 10 according to the embodiment of the present disclosure illustrated in FIG. 1. FIG. 4 shows spring portions 103a to 103d in addition to the imaging device 101 and the rotor covers 102a to 102d shown in FIGS. The spring portions 103a to 103d are respectively wound around the pillars at the four corners of the flying device 100, and rotate the rotor covers 102a to 102d around the pillars.

  FIG. 4 illustrates a state in which the rotor covers 102 a to 102 d are accommodated inside the flying device 100 by the user. With the configuration of the flying device 100 shown in FIG. 4, the controller 200 is put on the flying device 100, so that the imaging system 10 according to an embodiment of the present disclosure has the configuration shown in FIGS. 1 to 3. Become. When the controller 200 is put on the flying device 100, the rotation of the rotor covers 102 a to 102 d is suppressed by the controller 200.

  In FIG. 4, a support column 105 provided in the center of the flying device 100, a rotor 106 provided in the support column 105, and a plurality of vent holes 107 are illustrated. The rotor 106 performs a rotational motion with the support column 105 as an axis under the control of the controller 200. The vent hole 107 can be provided corresponding to the range of rotational movement of the rotor 106. Since the flying device 100 includes the rotor 106, the rotational motion of the rotor 106 can be achieved even when sufficient lift cannot be obtained only by the rotational motion of each rotor protected by the rotor covers 102a to 102d. The lift can be increased.

  FIG. 5 is an explanatory diagram illustrating an appearance example of the flying device 100 included in the imaging system 10 according to an embodiment of the present disclosure. FIG. 5 shows a state in which the controller 200 is removed from the state shown in FIG. 1 by sliding the controller 200 in the direction of A in FIG. When the controller 200 is slid in the direction of A in FIG. 1 from the state shown in FIG. 1 and the controller 200 is removed, the rotor covers 102a to 102d whose rotation has been suppressed by the controller 200 are removed from the spring portions 103a to 103d. By the action, it rotates as shown in FIG.

  FIG. 5 shows the rotors 104a to 104d protected by the rotor covers 102a to 102d. Each of the rotors 104a to 104d independently performs a rotational motion under the control of the controller 200, and causes the flying device 100 to float.

  As shown in FIGS. 4 and 5, the flying device 100 is provided with rotors 104 a to 104 d protected by the rotor covers 102 a to 102 d with predetermined steps. Since the rotors 104a to 104d are thus provided with a predetermined step, the flying device 100 can be accommodated in a compact manner when accommodating the rotors 104a to 104d inside.

  The existing flying device has lost its portability because the unit for outputting thrust is fixed. As shown in FIGS. 4 and 5, the flying device 100 configuring the imaging system 10 according to an embodiment of the present disclosure has a structure that can accommodate the rotors 104 a to 104 d on the inside, thereby greatly increasing portability. It is possible to increase it.

  Further, the flying device 100 rotates so that the rotor covers 102a to 102d jump out of the flying device 100 in a state where the controller 200 is removed, so that the center of gravity is compared to the state in which the rotor covers 102a to 102d are accommodated. And the center of the rotors 104a to 104d can be extended, and the attitude of the flying device 100 during flight can be stabilized.

  Here, the flight principle of the flying device 100 will be described. The flying device 100 can perform operations such as levitation, movement, stationary, and landing by the rotors 104a to 104d whose rotation speed can be independently controlled. The rotation direction of the rotors 104a and 104c and the rotation direction of the rotors 104b and 104d are opposite to each other. The rotation direction of the rotors 104a and 104c and the rotation direction of the rotors 104b and 104d are opposite to each other. When rotated at a constant speed, the flying device 100 floats or descends.

  Further, for example, when the rotational speed of the rotors 104a and 104b is made slower than the rotational speed of the rotors 104c and 104d in a state where the flying device 100 is levitated, the flying device 100 is in a state where the rotor 104a and 104b are levitated. It is possible to move in the direction. Further, for example, when the rotational speed of the rotors 104a and 104c is made slower than the rotational speed of the rotors 104b and 104d while the flying device 100 is flying, the flying device 100 is rotated clockwise or counterclockwise while flying. It is possible to rotate horizontally clockwise.

  Thus, the flying device 100 enables operations such as flying, horizontal movement, rotational movement, stationary, and landing by appropriately changing the rotational speeds of the rotors 104a to 104d. In the present embodiment, the control of the operation of the flying device 100 based on the change in the rotational speed of the rotors 104a to 104d is not performed by the operation on the flying device 100 itself, but by the operation on the image captured by the imaging device 101. Is called.

  As described above, the position and posture of the flying device 100 can be freely changed by controlling the rotational speed of the rotors 104a to 104d. However, in order to capture an image using the imaging device 101, the rotation of the rotors 104a to 104d In addition to controlling the speed, it is desirable to know the self position in the environment and the relative position with respect to the imaging target.

  The method of knowing the self position in the environment and the relative position with respect to the imaging target includes, for example, a method using an acceleration sensor, a gyro sensor, or other inertial sensors, or a plurality of target points by recognizing feature points or objects in the environment with an image sensor or the like. There is a method for estimating one's position from the amount of movement.

  For example, the current position / orientation can be obtained from the amount of change in position and orientation by measuring the acceleration and angular velocity of the own device with an inertial sensor such as an acceleration sensor or a gyro sensor. Furthermore, by measuring an absolute amount such as a direction or altitude with a barometric pressure sensor or the like, it is possible to obtain the position / orientation of the aircraft while correcting an error associated with the integration of the change amount.

  In addition, for example, an image sensor or the like can recognize a feature point or an object in the environment and can estimate its own position from the movement amounts of a plurality of target points. This technique is called SLAM (Simultaneous Localization And Mapping). When this SLAM is used, the position recognition accuracy can be improved by using it together with the movement amount from the inertial sensor.

  With such control, the flying device 100 can appropriately know the position of the aircraft in an environment where imaging is to be performed, and can automatically move or stop to an appropriate position by feedback control.

  In the above example, four rotors are provided except for the rotor 106 provided in the center of the flying device 100, but the number of rotors is not limited to this example. For example, the flying device may include eight rotors.

  6 and 7 are explanatory diagrams illustrating modifications of the flying device according to the embodiment of the present disclosure. 6 and 7 are explanatory views showing an external appearance example of a flying device 100 ′ having eight rotors. FIGS. 6 and 7 show a flying device 100 ′ including rotor covers 102 a ′ to 102 h ′. As shown in FIGS. 6 and 7, a flying device 100 ′ having eight rotors can be provided as a modification of the embodiment of the present disclosure.

  In the above-described example, the rotor covers 102a to 102d are rotated by the action of the spring portions 103a to 103d, but the present disclosure is not limited to such examples. For example, a slit may be provided in each support column on which the rotor covers 102a to 102d are provided, and the rotor covers 102a to 102d may be rotated along the slits.

  FIG. 8 is an explanatory diagram showing an example of the structure of the support column 111 that supports the rotor covers 102a to 102d. The column 111 shown in FIG. 8 is provided with a slit portion 112 that rotates the rotor covers 102 a to 102 d around the column 111. By rotating the rotor covers 102a to 102d along the slits 112, when the rotor covers 102a to 102d are opened to the outside of the flying device 100, all the rotor covers 102a to 102d are positioned at the same height. Is possible.

  FIG. 9 and FIG. 10 are explanatory diagrams showing a state in which the rotor cover 102a rotates along the slit portion 112 of the support column 111. FIG.

  The appearance example of the flying device 100 has been described above with reference to FIGS. Next, the controller 200 that controls the operation of the flying device 100 will be described.

  FIG. 11 is an explanatory diagram showing an example of how the user controls the operation of the flying device 100 using the controller 200. When the controller 200 is detached from the flying device 100, it functions as a device capable of remotely operating the flying device 100 as shown in FIG. The user can control take-off, flight, and landing of the flying device 100 by using the controller 200.

  And the imaging system 10 which concerns on one Embodiment of this indication transmits the image which the imaging device 101 provided in the flying device 100 imaged to the controller 200 in real time, and the imaging part which the display part 210 of the controller 200 displays is displayed. The operation of the flying device 100 can be controlled by accepting an operation on an image captured by the device 101. That is, the user can control the operation of the flying device 100 not by operating the flying device 100 itself but by operating the image captured by the imaging device 101.

  Heretofore, the appearance example of the imaging system according to an embodiment of the present disclosure has been described. Next, a functional configuration example of the imaging system according to an embodiment of the present disclosure will be described.

[Example of functional configuration of imaging system]
FIG. 12 is an explanatory diagram illustrating a functional configuration example of the flying device 100 and the controller 200 according to an embodiment of the present disclosure. Hereinafter, functional configuration examples of the flying device 100 and the controller 200 according to an embodiment of the present disclosure will be described with reference to FIG.

  As illustrated in FIG. 12, the flying device 100 according to an embodiment of the present disclosure includes an imaging device 101, rotors 104a to 104d, motors 108a to 108d, a control unit 110, a communication unit 120, and a sensor unit. 130, a position information acquisition unit 132, an alert generation unit 140, and a battery 150.

  12, the controller 200 according to an embodiment of the present disclosure includes a display unit 210, a communication unit 220, a control unit 230, a position information acquisition unit 240, a battery 242, and a power supply unit. 244. The control unit 230 includes a flight instruction generation unit 232 and a display control unit 234.

  The control unit 110 controls the operation of the flying device 100. For example, the control unit 110 adjusts the rotational speeds of the rotors 104a to 104d by adjusting the rotational speeds of the motors 108a to 108d, the imaging processing by the imaging device 101, and other devices (for example, the controller 200) via the communication unit 120. Information transmission / reception processing, alert generation processing for the alert generation unit 140, and the like can be controlled.

  As described above, the imaging apparatus 101 includes a lens, an imaging element such as a CCD image sensor, a CMOS image sensor, and the like. The imaging device 101 provided in the flying device 100 executes still image or moving image imaging under the control of the controller 200. An image captured by the imaging apparatus 101 is transmitted from the communication unit 120 to the controller 200.

  The rotors 104a to 104d cause the flying device 100 to fly by generating lift by rotation. The rotors 104a to 104d are rotated by the rotation of the motors 108a to 108d. The motors 108a to 108d rotate the rotors 104a to 104d. The rotation of the motors 108 a to 108 d can be controlled by the control unit 110.

  The communication unit 120 performs transmission / reception processing of information by wireless communication with the controller 200. The flying device 100 transmits an image captured by the imaging device 101 from the communication unit 120 to the controller 200. In addition, the flying device 100 receives an instruction regarding flight from the controller 200 by the communication unit 120.

  The sensor unit 130 is a device group that acquires the state of the flying device 100, and may be configured of, for example, an acceleration sensor, a gyro sensor, an ultrasonic sensor, an atmospheric pressure sensor, or the like. The sensor unit 130 may convert the acquired state of the flying device 100 into a predetermined signal and provide it to the control unit 110 as necessary. The position information acquisition unit 132 acquires information on the current position of the flying device 100 using, for example, GPS (Global Positioning System). The position information acquisition unit 132 can provide the acquired information on the current position of the flying device 100 to the control unit 110 as necessary.

  The alert generation unit 140 generates an alert such as sound or light under the control of the control unit 110 when the flying device 100 tries to fly beyond a preset flight range.

  The battery 150 stores electric power for operating the flying device 100. The battery 150 may be a primary battery that can only be discharged, or a secondary battery that can be charged. When the battery 150 is a secondary battery, the battery 150 can be supplied with electric power from the controller 200 when the flying device 100 and the controller 200 are combined as shown in FIG.

  As described above, the display unit 210 includes a flat panel display device such as a liquid crystal display device or an organic EL display device. The display unit 210 can display, for example, an image captured by the imaging device 101 and information for controlling the operation of the flying device 100. The display unit 210 is provided with a touch panel, and the user can directly operate the information displayed on the display unit 210 by touching the display unit 210 with a finger or the like.

  The communication unit 220 transmits and receives information to and from the flying device 100 by wireless communication. The controller 200 receives an image captured by the imaging device 101 from the flying device 100 by the communication unit 220. Further, the controller 200 transmits an instruction regarding the flight of the flying device 100 from the communication unit 220 to the flying device 100. A command regarding the flight of the flying device 100 may be generated by the control unit 230.

  The control unit 230 controls the operation of the controller 200. For example, the control unit 230 displays characters, graphics, images, and other information on the display unit 210, information transmission / reception processing with other devices (for example, the flying device 100) via the communication unit 220, and a power supply unit The power supply processing to the flying device 100 for 244 can be controlled.

  The flight instruction generation unit 232 generates an instruction regarding the flight of the flying device 100. In the present embodiment, the flight instruction generation unit 232 generates an instruction related to the flight of the flying device 100 based on an operation on an image captured by the imaging device 101. The flight instruction generation unit 232 generates an instruction related to the flight of the flying device 100 based on an operation on the image captured by the imaging device 101, so that the controller 200 can deal with a user who is not familiar with the operation of the flying device 100. Thus, the flying device 100 can be easily operated. In addition, the flight instruction generation unit 232 generates an instruction related to the flight of the flying device 100 based on an operation on the image captured by the imaging device 101, thereby causing the flying device 100 to fly in a composition desired by the user. Flight instructions can be generated. A specific example of the instruction generation processing related to the flight of the flying device 100 by the flight instruction generation unit 232 will be described in detail later.

  The display control unit 234 controls display of characters, graphics, images, and other information on the display unit 210. The display control unit 234 controls the display of characters, figures, symbols, images, and other information displayed on the display unit 210 in the drawings referred to in the following description.

  The position information acquisition unit 240 acquires information on the current position of the controller 200 using, for example, GPS (Global Positioning System). The position information acquisition unit 240 can provide the acquired information on the current position of the controller 200 to the control unit 232 as necessary.

  The battery 242 stores electric power for operating the controller 200. The battery 242 may be a primary battery that can only be discharged or a secondary battery that can be charged. The battery 242 can execute power supply to the flying device 100 via the power supply unit 244 when the flying device 100 and the controller 200 are combined as shown in FIG. The power supply unit 244 supplies the power stored in the battery 242 to the flying device 100 under the control of the control unit 230 when the flying device 100 and the controller 200 are combined as shown in FIG. .

  The flying device 100 and the controller 200 that configure the imaging system 10 according to an embodiment of the present disclosure have the configuration illustrated in FIG. 12, so that the imaging device 101 displays an image displayed on the display unit 210 of the controller 200. The flying device 100 can be maneuvered based on the operation on the image being performed. That is, the user can control the operation of the flying device 100 not by operating the flying device 100 itself but by operating the image captured by the imaging device 101.

  For example, information is transmitted between the flying device 100 and the controller 200 by wireless communication using the 2.4 GHz band, the 5 GHz band, or other frequency bands according to the IEEE 802.11 standard, the IEEE 802.15.1 standard, or other standards. It can be sent and received.

  The functional configuration examples of the flying device 100 and the controller 200 according to the embodiment of the present disclosure have been described above with reference to FIG. Next, an operation example of the imaging system 10 according to an embodiment of the present disclosure will be described.

[Operation example of imaging system]
FIG. 13 is a flowchart illustrating an operation example of the imaging system 10 according to an embodiment of the present disclosure, and in particular, a flowchart illustrating an operation example of the controller 200 configuring the imaging system 10. FIG. 13 shows an operation example when the user removes the controller 200 from the flying device 100 and steers the flying device 100 using the controller 200. Hereinafter, an operation example of the imaging system 10 according to an embodiment of the present disclosure will be described with reference to FIG.

  First, the controller 200 transmits a take-off instruction to the flying device 100 placed in a stationary state on the floor or the palm of the user's hand based on a user operation (step S101). When the flight instruction generation unit 232 detects that the predetermined operation corresponding to the take-off instruction has been performed on the display unit 210 having the touch panel, the controller 200 issues a take-off instruction to the flying device 100 by the flight instruction generation unit 232. The generated takeoff instruction is transmitted to the flying device 100 by wireless communication.

  When the flying device 100 receives the take-off instruction from the controller 200, the flying device 100 rotates the motors 108 a to 108 d from the control unit 110. Then, the flying device 100 is levitated by the lift generated by the rotation of the rotors 104a to 104d due to the rotation of the motors 108a to 108d.

  Subsequently, the controller 200 receives an image captured by the imaging device 101 provided in the flying device 100 by wireless communication and displays the image on the display unit 210 (step S102). While the image captured by the imaging apparatus 101 is displayed on the display unit 210, the controller 200 stands by until an operation is performed on the image displayed on the display unit 210 (step S103).

  For example, when the user performs an operation on an image displayed on the display unit 210 by touching the display unit 210, the controller 200 subsequently operates on the image displayed on the display unit 210. Is detected by the flight instruction generator 232 (step S104). When the operation content is detected with respect to the image displayed on the display unit 210, the controller 200 subsequently converts the operated content into a flight instruction of the flying device 100 by the flight instruction generation unit 232 (step S100). S105).

  Although a specific example will be described in detail later, for example, when the content of the operation detected in step S104 is an operation for positioning the subject specified by the user in the center of the image captured by the imaging device 101, The flight instruction generation unit 232 converts an operation performed by the user into a flight instruction of the flying device 100.

  When the operated content is converted into the flight instruction of the flying device 100 in step S105, the controller 200 subsequently transmits the flight instruction obtained in step S105 to the flying device 100 by wireless communication (step S106). . In the flying device 100, the control unit 110 controls the rotation of the motors 108 a to 108 d based on the flight instruction transmitted from the controller 200.

  The imaging system 10 according to an embodiment of the present disclosure is based on an operation on an image captured by the imaging apparatus 101 displayed on the display unit 210 of the controller 200 by the operation of the controller 200 as described above. The flying device 100 can be operated. That is, the user can control the operation of the flying device 100 not by operating the flying device 100 itself but by operating the image captured by the imaging device 101.

  The user views an image captured by the imaging device 101 displayed on the display unit 210 and, if a desired composition is obtained, sends an imaging instruction to the imaging device 101 using the controller 200. . The flying device 100 sends an imaging instruction from the control unit 110 to the imaging device 101 based on the imaging instruction transmitted from the controller 200. The imaging apparatus 101 executes an imaging process based on the imaging instruction sent from the control unit 110. Then, the flying device 100 transmits an image obtained by the imaging process in the imaging device 101 to the controller 200.

  The operation example of the imaging system 10 according to an embodiment of the present disclosure has been described above. Next, an example of information displayed on the display unit 210 of the controller 200 configuring the imaging system 10 according to an embodiment of the present disclosure will be described.

  FIG. 14 is an explanatory diagram illustrating an example of information displayed on the display unit 210 of the controller 200 according to an embodiment of the present disclosure. Hereinafter, an example of information displayed on the display unit 210 of the controller 200 will be described with reference to FIG.

  FIG. 14 shows a state in which the takeoff / landing button v11 and the imaging button v12 are displayed so as to be superimposed on the image captured by the imaging device 101. The takeoff / landing button v11 is a button for taking off or landing the flying device 100. The user can take off or land the flying device 100 by touching the takeoff / landing button v11. If the rotors 104a to 104d are not rotating, the takeoff / landing button v11 functions as a button for taking off the flying device 100. If at least one of the rotors 104a to 104d is rotating, the takeoff / landing button v11 is the flying device. Functions as a button for landing 100.

  The imaging button v12 is a button for causing the imaging apparatus 101 to execute an imaging process. The user can cause the imaging device 101 to execute an imaging process by touching the imaging button v12 and cause the imaging device 101 to capture a still image or a moving image.

  FIG. 14 shows a state in which the target frame v13, the target position v14, and the movement arrow v15 are displayed so as to be superimposed on the image captured by the imaging device 101. The target frame v13 has a function indicating an area designated by a user operation. The target position v14 has a function of indicating the target position in the captured image of the area indicated by the target frame v13. The movement arrow v15 has a function of indicating a straight path until the target frame v13 reaches the target position v14.

  When the user performs an operation on the image captured by the imaging device 101 such that the region surrounded by the target frame v13 reaches the target position v14, the flying device 100 is surrounded by the target frame v13. The position and orientation are controlled so that the region reaches the target position v14. In addition, until the area surrounded by the target frame v13 reaches the target position v14, the controller 200 displays the movement arrow v15 on the display unit 210, so that the flying device 100 changes the position and posture. Tell the user that it is inside.

  A display example of specific information on the display unit 210 will be described in more detail. First, an example of displaying information on the display unit 210 when the user-specified position is to be set at the center of the image will be described.

  15 and 16 are explanatory diagrams illustrating examples of information displayed on the display unit 210 of the controller 200 according to an embodiment of the present disclosure. FIGS. 15 and 16 show examples of displaying information on the display unit 210 when it is desired to place the position designated by the user in the center of the image. Hereinafter, an example of information displayed on the display unit 210 of the controller 200 will be described with reference to FIGS.

  When it is desired to place the position designated by the user at the center of the image, the controller 200 causes the user to perform an operation of tapping the designated position once, for example. FIG. 15 illustrates a state in which the user has once performed an operation of tapping a place where the display unit 210 is present. When the user taps a place on the display unit 210 once, the controller 200 causes the display unit 210 to display the target frame v13 and the target position v14. The display control unit 234 executes display processing on the display unit 210.

  When the user taps once on a place on the display unit 210 and the flight instruction generation unit 232 detects the tap, the flight instruction generation unit 232 flies so that the region surrounded by the target frame v13 is located at the target position v14. A flight instruction for flying the device 100 is generated. When the user once taps a place on the display unit 210 and the flight instruction generation unit 232 detects the tap, the display control unit 234 displays a movement arrow v15 connecting the target frame v13 and the target position v14 on the display unit 210. Execute display control to display.

  FIG. 16 illustrates a state after the user has performed an operation of tapping a place where the display unit 210 is once. As shown in FIG. 16, when the user performs an operation of tapping once on a place on the display unit 210, the display control unit 234 displays a movement arrow v <b> 15 connecting the target frame v <b> 13 and the target position v <b> 14 on the display unit 210. Is done. This movement arrow v15 is displayed on the display unit 210 until the target frame v13 reaches the target position v14.

  FIGS. 17 and 18 are display examples of information on the display unit 210 when it is desired to place the position designated by the user with the tap in the center of the image. As shown in FIG. 17, once the user taps a certain place on the display unit 210, the target frame v <b> 13, the target position v <b> 14, and the movement arrow v <b> 15 are displayed on the display unit 210. Then, when the flying device 100 reaches a desired position, as shown in FIG. 18, a centering process is performed in which the position designated by the tap by the user is placed at the center of the image.

  When the position designated by the user is to be set at a desired position in the image, the controller 200 causes the user to perform an operation of dragging the designated position on the display unit 210, for example. FIG. 19 is an explanatory diagram illustrating a state where the user has performed an operation of dragging on the display unit 210. When the user touches a place on the display unit 210 and performs a drag operation to move the display unit 210 while touching the place, the controller 200 causes the display unit 210 to display the target frame v13 and the target position v14. . The display control unit 234 executes display processing on the display unit 210.

  When the flight instruction generation unit 232 detects the user's drag operation, the flight instruction generation unit 232 generates a flight instruction for causing the flying device 100 to fly so that the region surrounded by the target frame v13 is located at the target position v14. . When the flight instruction generation unit 232 detects the user's drag operation, the display control unit 234 executes display control for causing the display unit 210 to display a movement arrow v15 that connects the target frame v13 and the target position v14.

  FIG. 20 is an explanatory diagram illustrating a state after the user performs a drag operation on the display unit 210. As illustrated in FIG. 20, when the user performs a drag operation on the display unit 210, the display control unit 234 displays a movement arrow v <b> 15 that connects the target frame v <b> 13 and the target position v <b> 14 on the display unit 210. This movement arrow v15 is displayed on the display unit 210 until the target frame v13 reaches the target position v14.

  When the user wants to change the magnification of the image, the controller 200 causes the user to perform, for example, a so-called pinch operation that opens or squeezes the display unit 200 with two fingers. FIG. 21 is an explanatory diagram showing a state where the user has performed a pinch operation on the display unit 210. When the user touches a place with the display unit 210 with two fingers and performs an operation to open or squeeze the finger on the display unit 210 while touching the place, a so-called pinch operation is performed, the controller 200 The target frame v13 and the target position v14 are displayed on the unit 210. The display control unit 234 executes display processing on the display unit 210.

  When the user performs a pinch operation on the display unit 210, the display control unit 234 determines the target frame v13 centered on the middle point of two fingers and causes the display unit 210 to display the frame. When the user performs a pinch operation on the display unit 210, the display control unit 234 determines the target position v14 from the amount of movement to a position away from the position where the user's finger contacts, and displays the target position v14 on the display unit 210. Display.

  As described above, the controller 200 that has detected the user operation on the image captured by the captured image 101 and displayed on the display unit 210 generates a flight instruction corresponding to the user operation and sends it to the flying device 100. Send. Here, generation of a flight instruction by the controller 200 will be described in more detail.

  The flight of the flying device 100 is controlled from the controller 200 by performing image feedback so that the image captured by the imaging device 101 is in a state desired by the user. FIG. 22 is an explanatory diagram showing control of the flight of the flying device 100 by image feedback.

  As shown on the left side of FIG. 22, the flight instruction generation unit 232 detects a moment when the user puts a finger on the display unit 210 equipped with a touch panel, and a predetermined centering on the position where the user puts the finger. The size area is registered as the reference image 251. In the above-described centering process associated with the screen tap operation, the target position is the center of the image, so the flight instruction generation unit 232 calculates the amount of image movement from the position where the user places the finger to the target position. Then, the flight instruction generation unit 232 calculates a flight control command based on the calculated image movement amount and transmits it to the flying device 100.

  In the next frame, the image captured by the imaging device 101 also changes due to a change in the attitude of the flying device 100. Therefore, the flight instruction generation unit 232 searches for an image by template matching or the like in the vicinity of the position of the reference image 251 in the previous frame as shown in the upper right of FIG. Find the position. Then, the flight instruction generation unit 232 calculates the amount of movement from the obtained position of the region to the target position, calculates the flight control command again, and transmits it to the flying device 100. This process is repeated until the region of the reference image 251 (or the region most similar to the region of the reference image 251) is sufficiently close to the target position.

  During a drag operation on the display unit 210, the flight instruction generation unit 232 detects the moment when the user touches the display unit 2190 with a finger, and refers to an area of a predetermined size centered on the position where the user places the finger. Register as an image 251. The flight instruction generation unit 232 continues to update the position of the user's finger on the latest image as the target position until the user releases the finger from the display unit 210. When the user lifts his / her finger from the display unit 210, the flight instruction generating unit 232 sets the position where the user last put his finger as the target position as it is.

  The flight instruction generation unit 232 can search for an image most similar to the reference image 251 while changing the search range (scale). The flight instruction generation unit 232 executes an image search by changing the scale so that the amount of movement from the current size to the target size is controlled as in the position control even if the size of the area of the reference image 251 changes. Can be calculated and converted into flight control commands.

  The controller 200 can generate a flight control command based on information specified by the user by touching the display unit 210, and can control the flight state of the flying device 100. When the user newly touches a different place on the display unit 210 while the flight state of the flying device 100 is being controlled based on the user's operation, the controller 200 immediately changes the control target to change the flying device. 100 flight states can be controlled.

  A correspondence relationship between the user's operation on the image captured by the imaging device 101 and the movement of the flying device 100 will be described. 23 to 26 are explanatory diagrams illustrating an example of a correspondence relationship between a user operation on an image captured by the imaging device 101 and movement of the flying device 100.

  FIG. 23 illustrates a correspondence relationship between the user's operation and the movement of the flying device 100 when the user performs an operation of moving the image leftward with respect to the image captured by the imaging device 101. . As shown in FIG. 23, when the user performs an operation of moving the image leftward on the image captured by the imaging device 101, the controller 200 causes the flying device 100 to rotate horizontally clockwise. A flight control command is generated and transmitted to the flying device 100.

  When the flight device 100 receives a flight control command that rotates horizontally in the clockwise direction from the controller 200, the flying device 100 controls the rotation of the rotors 104a to 104d so as to rotate horizontally in the clockwise direction as shown in FIG. To fly. Thus, by flying so as to rotate horizontally in the clockwise direction, the flying device 100 can change the position and posture to a position where an image can be taken with a composition desired by the user.

  FIG. 24 illustrates a correspondence relationship between the user's operation and the movement of the flying device 100 when the user performs an operation of moving the image downward on the image captured by the imaging device 101. . As illustrated in FIG. 24, when the user performs an operation of moving the image downward on the image captured by the imaging device 101, the controller 200 issues a flight control command that raises the flying device 100. Generate and transmit to the flying device 100.

  When the flight device 100 receives a flight control command for raising the aircraft from the controller 200, the flying device 100 controls the rotation of the rotors 104a to 104d to fly as shown in FIG. By flying in such a manner, the flying device 100 can change the position and posture to a position where an image can be captured with a composition desired by the user.

  FIG. 25 illustrates a correspondence relationship between the user's operation and the movement of the flying device 100 when the user performs a pinch operation for reducing the image on the image captured by the imaging device 101. As illustrated in FIG. 25, when the user performs a pinch operation for reducing the image on the image captured by the imaging device 101, the controller 200 generates a flight control command that causes the flying device 100 to move backward. To the flying device 100. Here, the side on which the imaging device 101 is provided is described as the front of the flying device 100.

  When the flight device 100 receives a flight control command for retreating from the controller 200, the flying device 100 controls the rotation of the rotors 104a to 104d to fly back as shown in FIG. By flying in such a backward direction, the flying device 100 can change the position and posture to a position where the user can capture an image with a desired size by a pinch operation.

  FIG. 26 illustrates a correspondence relationship between the user's operation and the movement of the flying device 100 when the user performs a pinch operation for enlarging the image captured by the imaging device 101. As illustrated in FIG. 26, when the user performs a pinch operation for enlarging an image captured by the imaging apparatus 101, the controller 200 generates a flight control command that causes the flying apparatus 100 to move backward. To the flying device 100. Here, the side on which the imaging device 101 is provided is described as the front of the flying device 100.

  When the flight device 100 receives a flight control command for moving forward from the controller 200, the flying device 100 controls the rotation of the rotors 104a to 104d to fly forward as shown in FIG. By flying in this way, the flying device 100 can change the position and posture to a position where the user can capture an image with a desired size by a pinch operation.

  As described above, the controller 200 can convert a user operation on an image captured by the imaging device 101 into a flight control command of the flying device 100 and transmit the flight control command to the flying device 100. The flying device 100 can change the position and orientation according to the user's operation on the image captured by the imaging device 101.

  As shown in FIG. 12, the flying device 100 includes a sensor unit 130. The flying device 100 includes the sensor unit 130, so that it can detect surrounding obstacles. For example, by providing the flying device 100 with an ultrasonic sensor as the sensor unit 130, the flying device 100 can estimate the distance to the floor or ceiling during the flight. Further, for example, when the flying device 100 is provided with the image sensor as the sensor unit 130, the flying device 100 can estimate the distance to the surrounding wall and the like by environment recognition using the imaging device 101.

  And if it moves so that it may become the composition designated by the user's operation with respect to the image which the imaging device 101 imaged, the case where it touches the obstacle around the flying device 100 can be considered. In such a case, the controller 200 may display the movement limit on the display unit 210 by providing information from the flying device 100.

  FIG. 27 is an explanatory diagram illustrating an example of information displayed on the display unit 210 of the controller 200 included in the imaging system 10 according to an embodiment of the present disclosure. FIG. 27 shows a state where a limit range v <b> 17 indicating the limit of the slide operation is displayed on the display unit 210. This limit range v17 is for notifying the user that the flying device 100 will collide with an obstacle such as a ceiling if the inside of the limit range v17 is designated and slid. The controller 200 can notify the user that the flying device 100 may collide with an obstacle such as a ceiling by blinking the limit range v17.

  Further, as shown in FIG. 27, the controller 200 causes the display unit to notify the user that the desired composition change is impossible, for example, by blinking the moving arrow v15 in a different color (for example, red). The display of 210 may be controlled.

  The imaging device 101 provided in the flying device 100 can execute optical zoom using movement of a zoom lens, electronic zoom by image conversion, and zoom by actually approaching or moving away from a subject. When the imaging apparatus 101 includes such a zoom function, for example, as shown in FIG. 21, when a composition is designated by a user's pinch operation, which zoom function should the flying apparatus 100 perform zooming on? It becomes difficult to judge.

  Therefore, the flying device 100 can normally operate the optical zoom with priority in order to avoid a risk of collision due to movement. However, since there is always a limit in the zoom range in the case of optical zoom, the flying device 100 first operates the zoom lens of the imaging device 101, and if the composition specified by the user cannot be achieved by optical zoom alone, You can fly closer or away from the subject.

  In addition, there may be a case where the flight apparatus 100 deviates from the target size due to the influence of wind or the like during flight control of the flying device 100 as shown in FIG. In the case of deviation from the target size, the flying device 100 performs flight control so as to correct the deviation of the target size by movement. By performing flight control so as to correct the deviation of the target size, if the image pickup position of the image pickup apparatus 101 is shifted due to factors such as wind, the zoom correction makes a gradual shift from the original position. To prevent that. That is, as shown in FIG. 21, the flying device 100 changes the size using the optical zoom when the size change is explicitly specified, and adjusts the position by movement during other control. May be.

  In order to make flight control of the flying device 100 easier, a flight control command that moves the flying device 100 in only one direction may be generated by the controller 200. For example, in the above-described example, when the user taps the display unit 210 once, the controller 200 controls the flight of the flying device 100 by image feedback so that the tapped position is located at the center of the captured image.

  However, in order to make the flight control of the flying device 100 easier, the following operation may be provided to the user. For example, when the user taps the display unit 210 once and then performs a drag operation from the tapped position, the controller 200 can execute flight control that causes the flying device 100 to rotate the yaw axis.

  FIG. 28 is an explanatory diagram illustrating an example of a correspondence relationship between a user operation on an image captured by the imaging device 101 and movement of the flying device 100. FIG. 28 shows an example where the user taps the display unit 210 once and then performs a drag operation from the tapped position.

  As shown in FIG. 28, based on the three-dimensional positions of the imaging device 101 and the tapped target, the center of the target is the origin, and the direction of the optical axis of the imaging device is orthogonal to the X and X axes. A coordinate system having two axes as the Y axis and the Z axis is obtained. When the user taps the display unit 210 once and then performs a drag operation from the tapped position, the controller 200 causes the flying device 100 to rotate the yaw axis about the Z axis in this coordinate system. Control can be performed. As a result, the operation of the flying device 100 becomes easier, and the flying device 100 is caused to fly while rotating around the object in this manner, and the imaging device 101 captures an image. It is possible to capture an image that provides an effect similar to imaging.

  For example, when the user taps the display unit 210 once and then performs a pinch operation from the tapped position, the controller 200 causes the flying device 100 to fly forward or backward with respect to the subject. Control can be performed.

  FIG. 29 is an explanatory diagram illustrating an example of a correspondence relationship between a user operation on an image captured by the imaging device 101 and movement of the flying device 100. FIG. 29 shows an example in which the user taps the display unit 210 once and then performs a pinch operation from the tapped position.

  As shown in FIG. 29, when the user taps the display unit 210 once and then performs a pinch operation from the tapped position, the controller 200 causes the flight in the forward or backward direction along the X axis. Flight control that causes the flying device 100 to perform may be executed. The coordinate system at this time is determined based on the three-dimensional positions of the imaging device 101 and the tapped target object, as in FIG. By capturing an image with the imaging device 101 while flying the flying device 100 in this manner, the flying device 100 can capture an image with the size of the subject desired by the user.

  As described above, the imaging system 10 according to an embodiment of the present disclosure allows the user to operate the controller 200 with respect to an image captured by the imaging device 101 included in the flying device 100, instead of operating the flying device 100 itself. Let it be done. The controller 200 generates a flight control command for the flying device 100 based on an operation performed on an image captured by the imaging device 101, and controls the flight of the flying device 100 with the generated flight control command.

  As described above, by causing the controller 200 to perform an operation on an image captured by the imaging device 101 provided in the flying device 100, even when the user is not used to maneuvering the flying device 100, the flying device 100. You can enjoy the maneuvering. Then, by taking an image with the imaging device 101 provided in the flying device 100, the user can easily take an image like aerial photography.

  The imaging system 10 according to an embodiment of the present disclosure can allow a user to steer the flying device 100 remarkably easily. On the other hand, when taking an image using the flying device 100, the user does not hold the camera, so the flying device 100 cannot be controlled as the user desires due to an unexpected disturbance such as a sudden gust of wind. The situation is possible. When the flying device 100 is in a state that cannot be controlled as the user desires, there is a risk that the flying device 100 may collide with an environment such as a person, a wall, or a ceiling.

  Considering the occurrence of such a situation, the flying device 100 configuring the imaging system 10 according to an embodiment of the present disclosure includes an alert generation unit 140 as illustrated in FIG. 12. The alert generation unit 140 generates an alert such as sound or light under the control of the control unit 110 when the flying device 100 tries to fly beyond a preset flight range. When the flying device 100 tries to fly beyond the preset flight range, the flying device 100 can automatically alert the user and the surrounding people by generating an alert by the alert generator 140. .

  FIG. 30 is an explanatory diagram showing a state in which the flying device 100 is located inside a predetermined flight range 21 with the initial position 20 as the center. From this state, if an unexpected disturbance such as a sudden gust of wind occurs, the flying device 100 may fly beyond the flight range 21. FIG. 31 is an explanatory diagram showing a state in which the flying device 100 is about to fly beyond a predetermined flight range 21 centered on the initial position 20.

  Here, for example, whether or not the preset flight range is exceeded is determined by the position information acquired by the position information acquisition unit 132 included in the flying device 100 and the position information acquisition unit 240 included in the controller 200, for example. It may be performed by the control unit 110 comparing the positional information thus obtained. When determining by comparing the position information, the position information can be periodically transmitted and received between the flying device 100 and the controller 200.

  Even if the alert generation unit 140 generates an alert, if the instruction is not transmitted from the controller 200 operated by the user for a predetermined period, the flying device 100 controls the rotation of the motors 108a to 108d, thereby the rotor 104a. It is possible to land automatically by slowing the rotation of ~ 104d or stopping the rotation. The control unit 110 can determine whether an instruction from the controller 200 has been transmitted after the alert generation unit 140 generates an alert. FIG. 32 is an explanatory diagram showing a state in which the flying device 100 is about to land automatically when an instruction is not transmitted from the controller 200 even when the alert generation unit 140 generates an alert.

  Thus, even when a situation in which the flying device 100 cannot be controlled as the user desires due to an unexpected disturbance such as a user's misoperation or a sudden gust of wind blows, the flying device 100 is set in advance. Alerts can be issued to alert the surroundings when the flight range is likely to be exceeded. The flying device 100 can automatically land to avoid a collision with a person or the environment if there is no instruction from the controller 200 even if an alert is issued. By operating in this way, the flying device 100 can greatly reduce the possibility of a collision with a person or the environment.

  In the imaging system 10 according to an embodiment of the present disclosure, the flying device 100 includes the imaging device 101, so that the flying device 100 is taken off from the user's hand and the user is imaged by the imaging device 101. Further, an operation in which the flying device 100 automatically returns to the user can be realized. Hereinafter, the operation of such a flying device 100 will be described.

  FIG. 33 is an explanatory diagram illustrating an outline of an operation in which the flying device 100 automatically performs take-off from the user, imaging, and return to the user in the imaging system 10 according to an embodiment of the present disclosure. . As shown in FIG. 33, when the flying device 100 initially in the user's hand starts taking off based on a take-off instruction from the controller 200, for example, the flying device 100 automatically reaches a position where the user's figure can be imaged with a preset composition. To fly. When the flying device 100 captures an image of the user with the set composition, the flying device 100 automatically returns to the user.

  In the imaging system 10 according to an embodiment of the present disclosure, by flying the flying device 100 in this way, for example, even in a place where the tripod cannot be stood, the same image as when the tripod is stood up is used. An image can be captured by the flying device 100.

  There are various techniques for realizing such automatic flight, imaging, and feedback. In the following description, a technique for realizing automatic flight, imaging, and feedback by image recognition will be disclosed.

  FIG. 34 is a flowchart illustrating an operation example of the imaging system 10 according to an embodiment of the present disclosure. FIG. 34 shows an operation example of the imaging system 10 when the flying device 100 automatically performs flight, imaging, and feedback by image recognition. Hereinafter, an operation example of the imaging system 10 according to an embodiment of the present disclosure will be described with reference to FIG.

  First, the user sets a desired composition in the imaging system 10 (step S201). The imaging system 10 stores the composition set in step S201 (step S202). Various methods can be used to set the composition, but one example is as follows. For example, the imaging system 10 may cause the user to set the composition by causing the user to touch the display unit 210 to designate a desired face position and face size.

  Further, the imaging system 10 may store the composition set in step S201 by either the flying device 100 or the controller 200. The imaging system 10 may store the composition set in step S201 in another device that is neither the flying device 100 nor the controller 200, for example, an external server device connected to the Internet.

  When the user sets a desired composition in the imaging system 10, the imaging system 10 is caused to memorize the imaging target face by causing the imaging device 101 to capture an image (step S203). The imaging system 10 creates a face identification dictionary for identifying a face using the face image captured by the imaging apparatus 101 (step S204).

  The imaging system 10 may create the face identification dictionary by either the flying device 100 or the controller 200. Further, the imaging system 10 may store the created face identification dictionary in another device that is neither the flying device 100 nor the controller 200, for example, an external server device connected to the Internet.

  When the user causes the imaging system 10 to remember the face to be imaged, the user subsequently operates the controller 200 to take off the flying device 100 (step S205). The flying device 100 that has taken off flies while taking an image with the imaging device 101. Then, the flying device 100 flies while controlling the position so that the position and size of the stored face in the image captured by the imaging device 101 become the composition set by the user (step S206).

  The position control in step S206 is specified by, for example, whether or not the stored face is included in the image captured by the imaging apparatus 101, and if the stored face is included, the position and size of the face are specified by the user. Depending on the relationship between position and size.

  When the flying device 100 determines that the position and size of the stored face in the image captured by the image capturing device 101 are in the composition set by the user, the image capturing device 101 captures an image (step S207). ).

  When the flying device 100 captures an image in step S207, the flying device 100 returns to the user and performs flight while controlling the position so that the stored face in the image captured by the imaging device 101 is close to the front. (Step S208). The position control in step S208 can be performed in the same manner as the position control in step S206.

  When the flying device 100 is sufficiently close to the user, the user presents a palm under the flying device 100 (step S209). When the flying device 100 detects that the palm of the user has been pushed out by the sensor unit 130, the flying device 100 relaxes the rotation of the rotors 104a to 104d and land on the palm of the pushed user (step S210).

  The imaging system 10 according to an embodiment of the present disclosure can realize the flight, imaging, and return of the flying device 100 automatically by operating the flying device 100 and the controller 200 as described above. In the imaging system 10 according to an embodiment of the present disclosure, by flying the flying device 100 in this way, for example, even in a place where the tripod cannot be stood, the same image as when the tripod is stood up is used. There is an effect that an image can be captured by the flying device 100.

35 to 38 are explanatory diagrams illustrating an example of control of the position and orientation of the flying device 100 based on image recognition. FIG. 35 shows a case where the stored face position v21 in the image captured by the imaging apparatus 101 and the face position v22 set by the user are separated along the X axis. If the coordinate of the face position v21 is Xf and the coordinate of the face position v22 is Xt, the movement amount Vθ per unit time of the flying device 100 is
Vθ = Kθ × (Xf−Xt)
Is required. Kθ is a coefficient.

FIG. 36 shows a case where the stored face position v21 in the image captured by the imaging apparatus 101 and the face position v22 set by the user are separated along the Y axis. When the coordinate of the face position v21 is Yf and the coordinate of the face position v22 is Yt, the movement amount Vθ per unit time of the flying device 100 is
Vy = Ky × (Yf−Yt)
Is required. Ky is a coefficient.

FIG. 37 shows a case where the face orientation at the stored face position v21 in the image captured by the imaging apparatus 101 is different from the face orientation set by the user. Assuming that the face orientation at the face position v21 is θf and the face orientation set by the user is θt, the movement amount Vx of the flying device 100 per unit time is
Vx = Kx × (θf−θt)
Is required. Kx is a coefficient.

FIG. 38 shows a case where the face size at the stored face position v21 in the image captured by the image pickup apparatus 101 is different from the face size at the face position v22 set by the user. Is. If the face orientation at the face position v21 is Syf and the face orientation set by the user is Syt, the movement amount Vz per unit time of the flying device 100 is
Vz = Kz × (Syf-Syt)
Is required. Kz is a coefficient.

  The imaging system 10 according to an embodiment of the present disclosure can control the position and orientation of the flying device 100 by recognizing the image captured by the imaging device 101 in this way.

  In the above-described example, the return of the flying device 100 is automatically performed by image recognition, but the present disclosure is not limited to such an example. For example, the position information acquired by the position information acquisition unit 132 of the flying device 100 and the position information acquired by the position information acquisition unit 240 of the controller 200 are compared, and the flying device 100 flies so as to approach the position of the controller 200. Control may be executed. When the return of the flying device 100 is realized based on the comparison of the position information, the position information can be periodically transmitted and received between the flying device 100 and the controller 200.

<2. Summary>
As described above, according to an embodiment of the present disclosure, the controller 200 capable of maneuvering the flying device 100 that includes a plurality of rotors by a simple operation is provided. The controller 200 displays an image captured by the imaging device 101 provided in the flying device 100, and converts an operation on the image from a user into a command for operating the flying device 100.

  The controller 200 according to an embodiment of the present disclosure converts an operation from a user on an image captured by the imaging device 101 into a command for operating the flying device 100 and transmits the command to the flying device 100. Thereby, the controller 200 which concerns on one Embodiment of this indication enables a user to steer the flight apparatus 100 by intuitive operation.

  In the above-described embodiment, the imaging system 10 in which the controller 200 and the flying device 100 can be integrated has been described, but it goes without saying that the present disclosure is not limited to such an example. For example, even if the controller 200 and the flying device 100 cannot be integrated, for example, the function of the controller 200 may be assigned to a smartphone, a tablet-type portable terminal, or the like.

  Each step in the processing executed by each device in the present specification does not necessarily have to be processed in time series in the order described as a sequence diagram or flowchart. For example, each step in the processing executed by each device may be processed in an order different from the order described as the flowchart, or may be processed in parallel.

  In addition, it is possible to create a computer program for causing hardware such as a CPU, ROM, and RAM incorporated in each device to exhibit functions equivalent to the configuration of each device described above. A storage medium storing the computer program can also be provided. Moreover, a series of processes can also be realized by hardware by configuring each functional block shown in the functional block diagram with hardware.

  The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present disclosure belongs can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present disclosure.

  For example, in the above-described embodiment, the image captured by the imaging device 101 provided in the flying device 100 is displayed, and the user's operation on the image is converted into a command for operating the flying device 100. The present disclosure is not limited to such examples. For example, when the imaging apparatus 101 has a pan function or a tilt function, the controller 200 uses a user's operation on an image captured by the imaging apparatus 101 as a command for a pan operation or a tilt operation of the imaging apparatus 101. It may be converted and transmitted to the flying device 100.

  Then, the controller 200 combines a command for maneuvering the flying device 100 and a command for panning or tilting the imaging device 101, which are generated from an operation by the user with respect to an image captured by the imaging device 101, so that the user The flying device 100 may be controlled so that a desired image can be captured.

In addition, this technique can also take the following structures.
(1)
An image display unit that acquires and displays an image captured by the imaging device provided in the flying object from the flying object;
A flight instruction generation unit that generates a flight instruction of the flying object based on a content of an operation performed on an image captured by the imaging device and displayed by the image display unit;
A control device comprising:
(2)
The said flight instruction production | generation part produces | generates the flight instruction | indication of the said flying body based on the content of operation with respect to the said image display part which is displaying the image which the said imaging device is imaging, The control apparatus as described in said (1) .
(3)
The control device according to (2), wherein the flight instruction generation unit generates a flight instruction of the flying object for causing the imaging device to capture an image with a composition corresponding to an operation on the image display unit.
(4)
The flight instruction generation unit generates the flight instruction of the flying body such that a predetermined range designated by an operation on the image display unit is located at a center of an image captured by the imaging device (3) The control device described in 1.
(5)
The flight instruction generation unit generates a flight instruction of the flying body such that a predetermined range specified by an operation on the image display unit is located at a user's desired position in an image captured by the imaging device. The control device according to (3).
(6)
The flight instruction generation unit generates a flight instruction of the flying object such that a predetermined range specified by an operation on the image display unit is a user's desired size in an image captured by the imaging device, The control device according to (3).
(7)
A display control unit for controlling display on the image display unit;
The display control unit causes the image display unit to display a moving path until the position of the flying object can be captured by the imaging apparatus with a composition corresponding to an operation on the image display unit. The control apparatus in any one of-(6).
(8)
The said flight instruction production | generation part is a control apparatus as described in said (3) which produces | generates the flight instruction | command which advances the said flight body, if operation with respect to the said image display part is operation which expands the said image.
(9)
The control device according to (8), wherein the operation for enlarging the image is a pinch operation with respect to the image display unit.
(10)
The said flight instruction production | generation part is a control apparatus as described in said (3) which produces | generates the flight instruction | indication which reverses the said flight body, if operation with respect to the said image display part is operation which reduces the said image.
(11)
The control device according to (10), wherein the operation of reducing the image is a pinch operation with respect to the image display unit.
(12)
The said flight instruction production | generation part is a control apparatus as described in said (3) which produces | generates the flight instruction | indication which moves the said flight body to a horizontal direction, if operation with respect to the said image display part is operation which slides the said image.
(13)
If the operation on the image display unit is an operation for setting the designated position at the center of the image, the flight instruction generation unit causes the center of the image captured by the imaging device to be the designated position. The control device according to (3), wherein a flight instruction for moving the flying object is generated.
(14)
The control device according to (13), wherein the operation of setting the designated position at the center of the image is a tap operation on the image display unit.
(15)
The said image display part displays the path | route of the said flying body until it reaches a target position, while the said flying body is moving based on the flight instruction | indication which the said flight instruction | indication production | generation part produced | generated, Said (1). Control device.
(16)
The control device according to (1), wherein the image display unit superimposes and displays a limit range in which the flying object can move by an operation on the image display unit.
(17)
Acquiring and displaying an image captured by an imaging device included in the flying object from the flying object; and
The step of setting the contents of the operation performed on the image captured by the imaging device and displayed in the displaying step as a flight instruction of the flying object;
A control method comprising:
(18)
On the computer,
Acquiring and displaying an image captured by an imaging device included in the flying object from the flying object; and
The step of setting the contents of the operation performed on the image captured by the imaging device and displayed in the displaying step as a flight instruction of the flying object;
A computer program that executes

DESCRIPTION OF SYMBOLS 10 Imaging system 100 Flight apparatus 101 Imaging apparatus 102a-102d Rotor cover 103a-103d Spring part 104a-104d, 106 Rotor 105 Support | pillar 107 Vent part 200 Controller

Claims (18)

An image display unit that acquires and displays an image captured by the imaging device provided in the flying object from the flying object;
A flight instruction generation unit that generates a flight instruction of the flying object based on a content of an operation performed on an image captured by the imaging device and displayed by the image display unit;
A control device comprising:   The control device according to claim 1, wherein the flight instruction generation unit generates a flight instruction of the flying object based on a content of an operation performed on the image display unit displaying an image captured by the imaging device.   The control device according to claim 2, wherein the flight instruction generation unit generates a flight instruction of the flying body for causing the imaging device to capture an image with a composition corresponding to an operation on the image display unit.   The flight instruction generation unit generates a flight instruction of the flying object such that a predetermined range designated by an operation on the image display unit is located at a center of an image captured by the imaging device. The control device described.   The flight instruction generation unit generates a flight instruction of the flying body such that a predetermined range specified by an operation on the image display unit is located at a user's desired position in an image captured by the imaging device. The control device according to claim 3.   The flight instruction generation unit generates a flight instruction of the flying body such that a predetermined range specified by an operation on the image display unit is a user's desired size in an image captured by the imaging device. Item 4. The control device according to Item 3. A display control unit for controlling display on the image display unit;
The display control unit causes the image display unit to display a moving path until the position of the flying object can be captured by the imaging apparatus with a composition corresponding to an operation on the image display unit. The control device described.   The control device according to claim 3, wherein the flight instruction generation unit generates a flight instruction for moving the flying object forward if the operation on the image display unit is an operation of enlarging the image.   The control device according to claim 8, wherein the operation for enlarging the image is a pinch operation with respect to the image display unit.   The control device according to claim 3, wherein the flight instruction generation unit generates a flight instruction for moving the flying object backward when the operation on the image display unit is an operation for reducing the image.   The control device according to claim 10, wherein the operation of reducing the image is a pinch operation with respect to the image display unit.   The control device according to claim 3, wherein the flight instruction generation unit generates a flight instruction for moving the flying object in a horizontal direction when the operation on the image display unit is an operation of sliding the image.   If the operation on the image display unit is an operation for setting the designated position at the center of the image, the flight instruction generation unit causes the center of the image captured by the imaging device to be the designated position. The control device according to claim 3, wherein a flight instruction for moving the flying object is generated.   The control device according to claim 13, wherein the operation of setting the designated position at the center of the image is a tap operation on the image display unit.   The control device according to claim 1, wherein the image display unit displays a trajectory to a target position while the flying object is moving based on the flight instruction generated by the flight instruction generation unit.   The control device according to claim 1, wherein the image display unit superimposes and displays a limit range in which the flying object can move by an operation on the image display unit. Acquiring and displaying an image captured by an imaging device included in the flying object from the flying object; and
The step of setting the contents of the operation performed on the image captured by the imaging device and displayed in the displaying step as a flight instruction of the flying object;
A control method comprising: On the computer,
Acquiring and displaying an image captured by an imaging device included in the flying object from the flying object; and
The step of setting the contents of the operation performed on the image captured by the imaging device and displayed in the displaying step as a flight instruction of the flying object;
A computer program that executes

Images