JP6711565B2 - Communication device, control method thereof, and program - Google Patents

Description

The present invention relates to a communication device having a mobile function.

2. Description of the Related Art In recent years, cameras have been downsized, and are now mounted on small moving bodies called so-called drones. As a typical drone, a small helicopter type device is known.

There are two main ways to use drones. One is a method in which a user operates a smartphone, which is a type of a mobile phone, a so-called tablet device, or the like to remotely control flight processing and shooting processing of a drone. For example, Patent Document 1 proposes a configuration in which a rotary-wing drone such as a 4-wing helicopter is operated by a remote control device.

Another way is to let the drone autonomously control flight control and shooting control. In this operation mode, it is premised that flight conditions and shooting conditions are set for the drone in advance.

JP, 2013-144539, A

As the usage of communication devices having a moving function such as a drone is diversified, how to determine the operation mode of the drone becomes a problem. As a simple method, there is a method of manually setting by the user, but there is a problem that setting each time it is used is troublesome. Further, for example, in the case of a small drone, it is possible to reduce the size of the operation unit such as the display unit, the touch panel, and the buttons as much as possible, and thus it is preferable to request a complicated operation in determining the operation mode. Absent.

A communication device according to the present invention includes a moving unit that moves the communication device, a communication unit that communicates with an external device that has a control application that remotely controls the movement of the communication device, and an operation unit that receives an operation from a user. In the case where the communication means receives a control request transmitted from the external device due to the user operating the control application within a predetermined period after the operation start operation is accepted by the operation means, The external device operates in a first mode for remotely controlling the movement of the communication device by the moving means via the control application, and the communication means sends a control request transmitted from the external device within the predetermined period. If not accepted, the communication device automatically operates in a second mode in which the movement of the moving means is autonomously controlled.

According to the present invention, it is possible to determine an operation mode of a communication device having a moving function without requiring a complicated operation.

The figure which shows an example of the hardware constitutions of the drone and tablet device in 1st Embodiment. System configuration diagram in the first embodiment The figure which shows an example of the screen of the control application in 1st Embodiment. The figure which shows the processing flow of the drone 100 in 1st Embodiment.

[First Embodiment]
Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

<Drone hardware configuration>
In this embodiment, the drone 100 will be described as an example of a communication device having a moving function. The drone 100 in this embodiment is a small helicopter type device. Hereinafter, the structure of the drone 100 will be described with reference to FIG.

The control unit 101 controls each unit of the drone 100 based on a program stored in a storage unit 102 or the like described later and an input signal from each unit. The control unit 101 functions as a unit that performs storage control, communication control, imaging control, movement control, other input/output control, etc., but is not necessarily configured with one piece of hardware. For example, a plurality of hardware may share the processing, or a plurality of hardware may cooperate to function as one means.

The storage unit 102 is used as an area for storing a control program executed by the control unit 101 or as a storage area for various data. The storage unit 102 includes a ROM, a RAM, a HDD, a flash memory, and the like.

The moving unit 103 has a configuration for flying the drone 100. Specifically, it is equipped with a rotor (rotary blade), a motor, a sensor for attitude control, and the like, and by controlling the rotation speed and angle of the rotor under the control of the control unit 101, the flight direction, speed, and altitude of the drone 100 are controlled. , And tilt (rotation). By controlling the moving unit 103, it is possible to fly the drone 100 toward a predetermined position and hover (stop flight) at the predetermined position.

The BLE communication unit 104 is a communication unit that performs communication conforming to the LE (Low Energy) standard defined by Bluetooth (registered trademark) version 4.0. Bluetooth Low Energy (BLE) is a communication protocol that enables communication with less power consumption than ordinary Bluetooth and wireless LAN.

The wireless LAN communication unit 105 is a communication unit that performs wireless LAN communication conforming to the IEEE 802.11 standard.

The imaging unit 106 includes an optical lens, a CMOS sensor, a digital image processing unit, and the like, and converts an analog signal input via the optical lens into digital data to acquire image data. The image capturing unit 106 also includes a pan-tilt mechanism, and controls the image capturing direction according to the control of the control unit 101. The image data acquired by the image capturing unit 106 is stored in the storage unit 102 and also input to the detection unit 107 described later.

The detection unit 107 analyzes the image data input from the imaging unit 106 and determines whether a specific subject is included in the image data. The detection unit 107 performs determination processing on a frame-by-frame basis of the input image data, and notifies the control unit 101 of the detection result and, if detected, the coordinate position information in the image data. It is assumed that the information regarding the specific subject to be determined is registered in advance and stored in the storage unit 102.

The position information acquisition unit 108 acquires information on the current position (latitude, longitude) of the drone 100. The position information acquisition unit 108 is composed of, for example, a GPS receiver.

Reference numeral 109 is an altitude information acquisition unit. The altitude information acquisition unit 109 acquires altitude information of the drone 100 from the ground. The altitude information acquisition unit 109 is composed of, for example, a laser emitting unit and a light receiving unit, and measures the distance to the ground by the reflected light from the ground with respect to the laser emission.

The operation unit 110 is for receiving an operation of the drone 100. The operation unit 110 in the present embodiment has buttons for turning on/off the power of the drone 100 and receiving instructions for starting/stopping the operation. In addition, in order to miniaturize the drone 100, it is preferable to keep the operation unit 110 to a minimum.

The hardware configuration of the drone 100 has been described above with reference to FIG.

<Hardware configuration of tablet device>
FIG. 1B is a hardware configuration diagram of a tablet device 150, which is an example of an external device. Although the tablet device is used in this embodiment, a mobile phone such as a smartphone, a personal computer, a game machine, a dedicated controller, or the like may be used.

The control unit 151 controls each unit of the tablet device 150 based on a program stored in the storage unit 152 and the like described later and an input signal from each unit. The control unit 151 functions as a unit that performs storage control, communication control, imaging control, display control, other input/output control, etc., but is not necessarily configured with one piece of hardware. For example, a plurality of hardware may share the processing, or a plurality of hardware may cooperate to function as one means.

The storage unit 152 is used as an area for storing a control program executed by the control unit 151 or as a storage area for various data. The storage unit 152 includes a ROM, a RAM, a HDD, a flash memory, and the like.

The public line communication unit 153 is a communication unit that performs communication using a so-called public line using 3G, LTE (Long Term Evolution), or the like. The public line communication unit 153 allows the tablet device 150 to connect to the Internet.

The BLE communication unit 154 is a communication unit that performs communication conforming to the LE (Low Energy) standard defined by Bluetooth (registered trademark) version 4.0.

The wireless LAN communication unit 155 is a communication unit that performs wireless LAN communication conforming to the IEEE 802.11 standard.

The imaging unit 156 includes an optical lens, a CMOS sensor, a digital image processing unit, and the like, and converts an analog signal input via the optical lens into digital data to acquire image data.

The display unit 157 is a display for displaying various information. A screen for controlling the drone 100, which will be described later, is also displayed on the display unit 157.

The operation unit 158 is an operation unit for receiving an operation from the user. In this embodiment, a touch sensor is used as the operation unit 158, and a so-called touch panel unit is configured by overlapping the display unit 157.

The hardware configuration of the tablet device 150 has been described above with reference to FIG. Note that FIG. 1B is an example, and may be configured to include a processing block (not shown).

<System configuration>
Next, a system configuration showing a usage pattern of the drone 100 in this embodiment will be described with reference to FIG.

It is assumed that the tablet device 150 in the present system is pre-installed with a control function for remotely operating the drone 100 and a control application for receiving an image from the drone 100 to display and record the image. By executing this control application, the tablet device 150 functions as a control device that controls the drone 100. This application is downloaded from a specific site via the internet. Note that the control application does not necessarily have to be installed in advance, and a so-called web application that operates on a web browser may be used, for example.

202 is a tablet user 202 who uses the tablet device 150.

Reference numeral 203 denotes a subject that the drone 100 shoots. The subject 203 and the tablet user 202 may be the same person.

Reference numeral 204 denotes a position information transmitting device attached to the subject 203. The position information transmitting device 204 has a GPS (Global Positioning System) function and a BLE communication function. Here, the BLE communication function has the same function as the BLE communication unit 104. The position information transmitting device 204 can transmit the position information acquired by the GPS function to the drone 100 by BLE communication.

<Drone operation mode>
Next, the operation mode of the drone 100 will be described. The operation modes of the drone 100 of this embodiment include the following two.

The first is a remote control mode. The remote operation mode is an operation mode in which the flight application and the imaging process of the drone 100 are remotely controlled by the control application executed by the tablet device 150. The drone 100 communicates with the tablet device 150 via a wireless LAN, and executes operations such as flight and imaging according to commands transmitted from the tablet device 150 based on the operation of the tablet user 202.

Here, FIG. 3A shows an example of the screen of the control application displayed on the display unit 157 of the tablet device 150.

Reference numeral 301 denotes the entire operation screen of the control application. The operation screen 301 is also an area for displaying a so-called live view, and a captured image streaming-transmitted from the drone 100 by wireless LAN communication is displayed in real time.

302 is a button for executing movement control of the drone 100 in various directions (forward, backward, right, left, ascent, descent). When the move button 302 is pressed, a move control command is transmitted to the drone 100 by wireless LAN communication. The movement control command is repeatedly transmitted while the movement button 302 is continuously pressed. The control unit 101 of the drone 100 receives the movement control command by the wireless LAN communication unit 105, and controls the movement unit 103 based on the content of the movement control command. When the finger is moved away from the move button 302, the transmission of the movement control command is stopped, which causes the drone 100 to stop the moving operation.

303 is a button for instructing rotation of the drone 100 in various rotation directions. While the rotation button 303 is being pressed, the rotation control command is repeatedly transmitted to the drone 100 by wireless LAN communication. The control unit 101 of the drone 100 receives the rotation control command by the wireless LAN communication unit 105 and controls the moving unit 103 based on the content of the rotation control command. When the finger is released from the rotation button 303, the transmission of the rotation control command is stopped, which causes the drone 100 to stop the rotation operation.

Reference numeral 304 denotes a Rec button for controlling start/stop of capturing a moving image. When the Rec button 304 is pressed, a Rec command is transmitted to the drone 100 by wireless LAN communication. When the wireless LAN communication unit 105 receives the Rec command, the control unit 101 of the drone 100 controls the image capturing unit 106 and the storage unit 102 to start capturing a moving image. The moving image data obtained by imaging is stored in the storage unit 102, but may be transmitted to the tablet device 150 by wireless LAN communication and stored in the storage unit 152 of the tablet device 150.

A Photo button 305 is used to capture a still image. When the Photo button 305 is pressed, a Photo command is transmitted to the drone 100 by wireless LAN communication. When the wireless LAN communication unit 105 receives the Photo command, the control unit 101 of the drone 100 controls the image capturing unit 106 and the storage unit 102 to start capturing a still image. Further, although the still image data obtained by imaging is stored in the storage unit 102, it may be transmitted to the tablet device 150 by wireless LAN communication and stored in the storage unit 152 of the tablet device 150.

The above is the outline of the remote operation mode.

Next, the second operation mode will be described. The second operation mode is the autonomous mode. The autonomous mode is an operation mode in which the drone 100 autonomously controls flight processing and shooting processing. In the autonomous mode, the drone 100 controls its own flight so as to track the subject 203. Specifically, from the position information transmitted by the BLE communication from the position information transmission device 204 attached to the subject 203, and the current position information of the drone 100 itself acquired by the position information acquisition unit 108 and the altitude information acquisition unit 109. , Calculates the distance and direction to the subject 203, and controls flight so as to maintain a constant distance with respect to the subject 203.

Further, in the autonomous mode, the drone 100 automatically images the subject 203. For example, the detection unit 107 detects the subject 203 in the captured image by using information for identifying the subject 203 registered in the drone 100 in advance. When it is determined that the subject 203 is included in the captured image, the capturing process is performed. After detecting the subject, the subject 203 is tracked, and flight is controlled so that the subject 203 falls within the angle of view. In the case of a still image, the subject 203 is imaged at regular intervals, and in the case of a moving image, the image pickup is continued while the subject 203 is detected within the angle of view. Furthermore, in order to image the subject 203 from various angles, the flight may be controlled so that the positional relationship with the subject 203 is changed at regular time intervals.

The image data obtained by imaging is stored in the storage unit 102 as in the remote operation mode, but may be transmitted to the tablet device 150 by wireless LAN communication and stored in the storage unit 152 of the tablet device 150. ..

<Operation flow of drone operation>
Next, a flow in which the drone 100 of this embodiment determines an operation mode will be described with reference to FIG. The flowchart shown in FIG. 4 is started when the drone 100 is powered on.

In step S401, the control unit 101 determines whether the operation start button included in the operation unit 110 has been pressed. If it is determined that the button has been pressed, the process proceeds to S402. If it is determined that the button has not been pressed, the process is repeated.

In step S402, the control unit 101 executes hovering. Specifically, the driving of the moving unit 103 is started and the altitude information acquisition unit 109 is controlled to ascend to a position of several tens of centimeters to several meters from the ground, and hovering (stop flight) is executed at that position. This allows the user of the drone 100 to recognize that the drone is ready to fly.

In step S403, the control unit 101 establishes a connection with the tablet device 150 at the wireless LAN level. Here, two methods will be described as connection methods in the wireless LAN. Only one of the methods may be implemented, or both methods may be implemented so that selection can be made by a user operation or the like.

The first method is to enable the wireless LAN access point function of the tablet device 150 and make a connection request to the tablet device 150 using the parameters that the drone 100 becomes a station and holds in advance. The timing at which the tablet device 150 activates the access point function of the wireless LAN is, for example, when the control application is activated or when a predetermined menu in the control application is operated. In this case, it is assumed that the SSID (Service Set IDentifier) of the network and the encryption key information generated on the access point side are stored in the drone 100 in advance.

The second method is a method of acquiring wireless LAN communication parameters by using BLE instead of storing wireless LAN parameters in the drone 100 in advance. In this method, the control unit 101 controls the BLE communication unit 104 to periodically transmit an advertisement packet, and upon receiving a connection request from the tablet device 150 that has received the advertisement packet, connects the BLE communication. Next, the wireless LAN parameters are exchanged by BLE communication. After exchanging the wireless LAN parameters, the wireless LAN access point function is activated. The tablet device 150 controls the wireless LAN communication unit 105 and requests the tablet device 150 for wireless LAN connection using the exchanged parameter information.

In S404, the control unit 101 proceeds to the processing of S405 if the connection is successful as a result of the connection request executed in S403. If the wireless LAN connection cannot be established even after the lapse of a predetermined time, the process proceeds to S410, the autonomous mode is entered, and the operation is started.

In step S405, the control unit 101 controls the wireless LAN communication unit 105 to send a packet for confirming application-level connection to the control application executed by the tablet device 150. The tablet device 150 that has received the packet displays the screen of FIG. 3B and prompts the user to confirm whether or not to perform remote control. Upon accepting a user operation for performing remote control (“YES” in FIG. 3B), the tablet device 150 transmits a connection OK response packet to the drone 100.

In step S406, the control unit 101 receives the response packet from the control application via the wireless LAN communication unit 105, and determines whether the connection at the application level can be confirmed. If it can be confirmed, the process proceeds to S407, and if it cannot be confirmed, the process proceeds to S410.

In step S<b>407, the control unit 101 waits for a predetermined time until a command, which is an example of a control request, is transmitted from the control application via the wireless LAN communication unit 105. Here, the command is a command requesting a transition of the operation mode of the drone 100 to a remote control mode, or a command requesting control related to flight control and shooting control of the drone 100. If the command is received within the predetermined time, the process proceeds to S409, and if not received, the process proceeds to S410.

In step S409, the control unit 101 determines the operation mode as the remote operation mode, shifts to the remote operation mode, and starts the operation. At the same time, the control unit 101 controls the wireless LAN communication unit 105 to start the process of streaming the real-time captured image acquired by the imaging unit 106 to the tablet device 150. Then, the operation start processing flow is ended by this step.

In S410, the control unit 101 determines the operation mode as the autonomous mode, automatically shifts to the autonomous mode, and starts the operation. At the same time, the control unit 101 controls the BLE communication unit 104 to establish the BLE communication connection with the position information transmitting device 204 in order to periodically acquire the position information from the position information transmitting device 204. Then, the operation start processing flow is ended by this step.

In S410, it is assumed that the operation in the autonomous mode is started, but if the control unit 101 does not have sufficient free space for recording the captured image, the operation in the autonomous mode is completely executed. Can not. Therefore, when it is determined that there is not enough free space for recording the captured image in the storage unit 102, an error notification may be issued and the operation may be ended. The error method depends on the hardware provided in the drone 100, but examples thereof include lighting of an LED, a buzzer sound, and an error display on the display.

The operation start flow of the drone 100 has been described above. Next, a case of stopping the drone will be described.

In the remote operation mode, the tablet user 202 operates the control application to move the drone 100 to the hand of the tablet user 202. Then, the tablet user 202 can stop the drone 100 by pressing the operation stop button on the operation unit 110.

In the autonomous mode, the drone 100 first stores the start position by the position information acquisition unit 108 at the timing when the autonomous mode is started. After a lapse of a predetermined time, the operation returns to the starting place and the operation is stopped. Alternatively, in consideration of the case where the subject 203 is the owner of the drone 100, it is possible to approach the position of the position information transmitting device 204 and stop the operation after a predetermined time has elapsed.

The above is the description of the system in the present embodiment. As described above, when the operation start button is pressed, the drone 100 waits for a connection from the tablet device 150 (control application) within a predetermined period, and when the predetermined period elapses, the autonomous mode is automatically set. Works with. By doing so, the user can determine the operation mode without performing a complicated operation on the drone 100. As a result, it is possible to reduce the number of operating members of the drone 100, which can contribute to downsizing.

Specifically, the determination of the operation mode will be described from the viewpoint of the user. When the user wants to operate the drone 100 in the remote control mode, he or she can press the operation start button of the drone 100 and then operate the control application on the tablet within a certain time to remotely control the drone 100. Moreover, when it is desired to operate in the autonomous mode, it suffices to press the operation start button of the drone 100. After that, the drone 100 operates in the autonomous mode after a certain period of time without doing anything. It is necessary to wait for a certain time from the pressing of the operation start button to the start of the autonomous mode, but since the drone 100 is hovering during that time, the user can recognize that the drone 100 is operating normally.

[Other Embodiments]
The configuration of the drone 100 is not limited to the configuration shown in FIG. 1A, and at least a part of the configuration may be a removable device. For example, a detachable camera may be attached to the drone. In this case, the external camera may be configured to include the imaging unit 106, the detection unit 107, and a part of the control unit 101 and a part of the storage unit 102 of FIG. Then, the drone may be provided with an interface for mounting the external camera and a means for controlling the external camera. As a camera to be attached to the drone, a so-called wearable camera, which is small and is used by being attached to a photographer, is preferable.

Further, it may be configured to include a processing block not shown in FIG. For example, an LED indicating the status of the drone 100 may be provided. Moreover, you may provide the display which performs various displays. However, the larger the display, the larger the drone 100 as a whole, so it is preferable to use only a small display that displays the status of the drone 100. As an example of the display on the LED or the display, there is a display that can identify whether the drone 100 is in the connection waiting state from the tablet device 150, the remote control mode, or the autonomous mode.

Further, in the present embodiment, a small helicopter is given as an example of the drone, but the drone is not limited to this. For example, the present invention can be applied to a device that moves on land by wheels or the like, or a humanoid device that moves by walking.

Further, in the present embodiment, the communication between the drone and the tablet device is performed by the wireless LAN, but the communication method is not limited to this. For example, Bluetooth (registered trademark), Zigbee (registered trademark), infrared ray, or the like can be used. Further, the communication between the drone and the position transmitting device is not limited to Bluetooth, and it is possible to use wireless LAN, Zigbee (registered trademark), infrared rays, or the like.

The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads the program. This is the process to be executed.

Claims (16)

A communication device,
Moving means for moving the communication device;
Communication means for communicating with an external device having a control application for remotely controlling the movement of the communication device;
An operation unit that receives an operation from a user,
When the communication unit receives a control request transmitted from the external device by the user operating the control application within a predetermined period after the operation start operation is received by the operation unit, the external device is When operating in the first mode for remotely controlling the movement of the communication device by the moving means via the control application, and the communication means does not accept a control request transmitted from the external device within the predetermined period. The communication device is automatically operated in a second mode in which the communication device autonomously controls movement by the moving means. The communication device according to claim 1, wherein the moving unit is for flying the communication device. The communication device according to claim 1, wherein the communication unit attempts to communicate with the external device when the predetermined operation is accepted. The communication means can perform communication using a first communication protocol and a second communication protocol that consumes less power than the first communication protocol,
4. When the predetermined operation is accepted, the communication unit attempts to communicate with the external device by notifying the external device using the second communication protocol. The communication device described. The communication device according to claim 3 or 4, wherein when the predetermined operation is accepted, the communication device attempts to communicate with the external device and drives the moving device. The communication device according to claim 2, wherein when the predetermined operation is accepted, the communication means attempts communication with the external device and drives the moving means to perform hovering. Has imaging means,
7. The communication device according to claim 1, wherein, in the first mode, the communication unit transmits image data obtained by image pickup by the image pickup unit to the external device. 8. The communication device according to claim 7, wherein in the first mode, the communication unit streams image data obtained by the image pickup by the image pickup unit to the external device in real time. Further comprising movement control means for controlling movement by the movement means,
9. The communication device according to claim 1, wherein in the second mode, the movement control unit controls the movement unit so as to track a specific subject. 9. The communication device according to claim 7, wherein the imaging unit is attachable to and detachable from the communication device. 9. The communication device according to claim 7, wherein the image pickup unit is a wearable camera. The communication device according to any one of claims 1 to 11, wherein the communication device is a helicopter type device. The communication device according to claim 1, wherein the external device is a tablet device. The communication device according to any one of claims 1 to 12, wherein the external device is a smartphone. A method of controlling a communication device, comprising: a moving means; and a communication means for communicating with an external device having a control application for remotely controlling movement by the moving means,
A step of receiving an operation start operation from a user,
When the communication unit receives a control request transmitted from the external device due to the user operating the control application within a predetermined period after the operation start operation is received, the external device is controlled by the external device. Operating in a first mode for remotely controlling movement of the communication device by the moving means via
When the communication unit does not accept a control request transmitted from the external device within the predetermined period, the communication device automatically operates in a second mode in which the movement by the moving unit is autonomously controlled. A method for controlling a communication device having the same. A program that can be executed by a computer that causes a computer to function as each unit of the communication device according to claim 1.

Images