JP6855616B2 - Operating devices, mobile devices, and their control systems - Google Patents

Description

The present invention relates to a movement control system for a mobile device such as a drone having an image pickup device, and more particularly to a system for moving the image pickup device by operating a touch panel or the like.

In recent years, some flight devices called drones that can move according to wireless operations have an image pickup device, and a dedicated operation terminal is generally prepared for the operation. There are some operation terminals that can be operated using a touch panel, but most of them use a UI that simply replaces the operation of the dedicated operation terminal, so intuitive operation is not possible and it is possible to operate freely. Requires proficiency in operation. Further, from the viewpoint of operating the image pickup device by the touch panel, there is a conventional technique for realizing intuitive operation in the field of surveillance cameras in general.

Patent Document 1 discloses an invention of an operation method in which a pinch-in and pinch-out operations, which are common in touch panel operations of smartphones and tablets, correspond to zoom operations of surveillance cameras. This correlates the result that can be intuitively inferred from the operation that the operator normally performs on the touch panel and the process that causes a similar result in the operation of the surveillance camera.

Japanese Unexamined Patent Publication No. 2014-99719

Patent Document 1 discloses a change in the shooting direction and a zoom control by operating a touch panel with respect to the built-in surveillance camera. However, a configuration in which the position of the image pickup device mounted on a drone or the like is moved by touch panel operation has not been considered.

Therefore, an object of the present invention is a movement control system that can intuitively and easily understand the movement of an imaging device mounted on a moving device such as a drone and capable of moving a position when operated by an operation terminal having a touch panel function. Is to provide.

In order to achieve the above object, the movement control system in the present invention is an operation device for externally controlling a mobile device holding the image pickup means, and is a display means for displaying an image captured by the image pickup means. In response to the touch operation means for detecting the touch operation and the touch operation detected by the touch operation means, the moving device is moved based on the operation direction of the move operation when the touch operation is a move operation. When the touch-on state is continuously detected by the touch operation means after the move operation, the instruction means continues the movement based on the move operation. It is characterized by being characterized by.

According to the present invention, when an image pickup device mounted on a drone or the like and capable of moving a position is operated by an operation terminal having a touch panel function, the movement can be intuitively and easily understood.

A block diagram showing a configuration example of an operation system according to the first embodiment of the present invention. A flowchart showing the processing of the operation terminal according to the first embodiment of the present invention. The figure which shows the pinch operation which concerns on 1st Embodiment of this invention. The figure which shows the touch position which concerns on 1st Embodiment of this invention. The figure which shows the determination of the moving direction by the pinch-out position which concerns on 1st Embodiment of this invention. The figure which shows the determination of the moving direction by the position of the pinch-in which concerns on 1st Embodiment of this invention. The figure which shows the operation corresponding to the drag operation which concerns on 1st Embodiment of this invention. The figure which shows the operation of the image pickup apparatus by the pinch-out operation which concerns on 1st Embodiment of this invention. The figure which shows the operation of the image pickup apparatus by the pinch-out operation which concerns on 1st Embodiment of this invention. A flowchart showing the processing of the mobile device according to the first embodiment of the present invention. The figure which shows the UI which makes clear the moving direction which concerns on 1st Embodiment of this invention. The figure which shows the moving coordinate system and the optical coordinate system which concerns on 2nd Embodiment of this invention. A flowchart showing the processing of the operation terminal according to the second embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First Embodiment)
FIG. 1 is a block diagram showing a configuration example of a movement control system according to the first embodiment of the present invention. In FIG. 1, 100 indicates a mobile device, and 200 indicates an operation device (operation terminal). The movement control system includes the movement device 100 and the operation device 200.

Reference numeral 118 denotes an imaging device, which is held by the moving device 100 by a holding member. The image pickup device 118 may have a structure that is removable from the moving device 100, or may have a structure that is integrally fixed.

The image pickup optical system 101 includes a lens group that collects external light on the image pickup element 103, and a mechanical mechanism for operating and fixing the lens group. The image pickup optical system operation unit 102 is an electric mechanism for operating the mechanical mechanism of the image pickup optical system 101, and outputs an electric signal for operating the internal mechanical mechanism of the image pickup optical system 101 by an operation command from the CPU 106. The image sensor 103 forms an image of the light taken in from the image pickup optical system 101, and converts the formed image into electrical information. Further, the image sensor 103 can be configured to acquire information for distance detection by the image plane phase difference method. The image pickup signal processing unit 104 forms the image information obtained from the image pickup element 103, and from the image plane phase difference information obtained from the image pickup element 103, the light source forming an image on each image pickup pixel of the image pickup element 103 from the image pickup device. The distance to the object can be calculated. That is, with respect to an arbitrary local part of the captured image, how far the image was captured from the imaging device is calculated, and the information can be recorded and used as needed. The principle that can be calculated is, for example, an application of the phase difference AF method by pupil division, which has been widely used in the past. Since the image sensor 103 has a structure including two photodiodes that can obtain information for the right eye image and information for the left eye image for each pixel, distance information can be detected by the phase difference AF method. It shall be.

The CPU 105 controls the entire processing of the mobile device 100 including the image pickup device 118. The processing program of the mobile device 100 and information necessary for processing are stored in advance in the ROM 106, and the CPU 105 operates based on the processing program held in the ROM 106. The RAM 107 plays a role of temporarily holding data during processing.

The subject recognition unit 108 can recognize a specific object from the image information processed by the image pickup signal processing unit 104 and output information such as the range of the object. Specifically, it is possible to detect pre-registered object information, for example, a human body, a face, or the like of a person. A specific person can be detected if there is face information registered in advance. The memory slot 109 is an interface for reading and writing recorded data to and from the memory card 110. The memory card 110 records image information through the memory slot 109.

The drive unit 111 is a mechanical part for enabling the autonomous position movement of the moving device 100, and specifically includes four motors and propellers attached to the motor shafts in this embodiment. The four propellers can be appropriately rotationally driven by a movement command comprehensively determined from the CPU 105, and operations such as position movement, posture change, and directional rotation can be performed. The orientation detection sensor 112 can detect whether the mobile device 100 is facing north, south, east, or west. GPS 113 can identify the current position by receiving and processing GPS signals. The acceleration sensor 114 can detect the acceleration in the three-dimensional direction applied to the moving device 100, and can detect the acceleration during acceleration, the falling state, and the like. The gyro 115 can detect the amount of rotation in three dimensions in three axial directions. The mobile device 100 can comprehensively process the information obtained from the configurations of the orientation detection sensors 112 to 115 to calculate the current position, acceleration state, rotation state, and the like. The communication unit 116 can transmit and receive necessary information such as command exchange, image information, and position information with the operation terminal 200 via wireless communication between the operation terminal 200 and the communication unit 206.

The imaging optical system 102 and the imaging signal processing units 104 to 116 can exchange commands and information via the bus 117.

The CPU 201 controls the entire processing of the operation terminal 200. The processing program of the operation terminal 200 and information necessary for processing are stored in advance in the ROM 202, and the CPU 201 operates based on the processing program held in the ROM 202. The RAM 203 plays a role of temporarily holding data during processing.

The LCD display unit 204 can display an image or video image captured by the image pickup device 118 received via the communication unit 206, which will be described later, or can display an operation member for the touch operation unit 205.

The touch operation unit 205 is an interface for receiving operations / instructions from the user. The CPU 201 can detect the following operation patterns / states for the touch operation unit 205.
-Touching the touch operation unit with a finger or pen (hereinafter referred to as touchdown).
-The touch operation unit must be touched with a finger or pen (hereinafter referred to as touch-on).
-Move while touching the touch operation part with a finger or pen (hereinafter referred to as move).
-Release of the finger or pen that was touching the touch operation unit (hereinafter referred to as touch-up).
-A state in which nothing is touched on the touch operation unit (hereinafter referred to as touch-off).

These operations and the touch position coordinates in which the finger or pen is touching the touch operation unit 205 are notified to the CPU 201 through an internal bus (not shown), and the CPU 201 performs what kind of operation to the touch operation unit 205 based on the notified information. Is determined.

Regarding the move, the moving direction of the finger or pen moving the touch operation unit 205 can also be determined for each vertical component and horizontal component based on the change in the position coordinates. Further, it is assumed that a stroke is drawn when the touch operation unit 205 is touched up after a certain move from the touchdown. If it is detected that the move has been performed, it is determined that the drag has been performed. The operation of quickly dragging and finally releasing the finger as if flipping it shall be judged as a flick.

It can also be detected that two simultaneous touches (multi-touch) are performed on the touch operation unit 205. Using multi-touch detection, as shown in Fig. 3 (a), the touch panel is moved so that the distance between the two touched points is separated while the touch panel is touched at two points with a finger or the like. (Hereinafter referred to as pinch-out) can also be discriminated. On the contrary, as shown in FIG. 3B, it can be determined that the movement is made so that the distance between the two touched points becomes closer (hereinafter, referred to as pinch-in). The pinch-in and pinch-out are collectively referred to as a pinch operation.

Further, the touch operation unit 205 can display the operation member on the LCD display unit 204 and then accept the individual operation corresponding to the operation member. The communication unit 206 can transmit and receive necessary information such as command exchange, image information, and position information with the mobile device 100 via communication with the communication unit 116 in the mobile device 100.

FIG. 2 is a flowchart showing the processing of the operation terminal according to the first embodiment of the present invention. The flowchart of FIG. 2 illustrates a processing procedure that is executed by controlling each processing block of the operation terminal 200, and is realized by expanding the program stored in the ROM 202 into the RAM 203 and executing the program by the CPU 201. ..

First, in step S201, it is determined whether or not there is a touch operation on the touch operation unit 205. If there is no touch operation, step S201 is repeated and the process waits until there is a touch operation. At this time, the mobile device 100 stops (hovering) at one point in the air, and the image captured by the image pickup device 118 is displayed on the LCD display unit 204 of the operation terminal 200 in real time via the communication unit 116 and the communication unit 206. It is assumed that it is displayed. Further, in this embodiment, it is assumed that the optical axis direction of the imaging device 118 and the moving axis direction defined by the driving unit 111 of the moving device 100 match as shown in FIG. 12D. If there is a touch operation, the process proceeds to step S202. In step S202, it is determined whether or not the operation pattern of the touch operation performed is a pinch operation. In step S202, if the touch operation performed is a pinch operation, the process proceeds to step S203, otherwise the process proceeds to step S209.

In step S203, it is determined whether or not the touch center point is inside the touch panel center region. This determination will be described in detail with reference to FIG. FIG. 4A is a diagram illustrating a touch panel central region on the touch panel. The touch panel frame 402 indicates the size of the operable area of the touch operation unit 205. For the sake of explanation, the xy coordinates are set based on the center point of the touch panel frame 402. The touch panel center area 401 is set as an area smaller than the touch panel frame 402 with reference to the center of the touch panel frame 402. The branching of the process in step S203 will be described from the two examples of FIGS. 4 (b) and 4 (c). In FIG. 4B, the finger areas 403 and 404 show the ground contact areas (start points of the pinch operation) of the two fingers first touched for the pinch operation, respectively. The positions of the center of gravity of each of the finger regions 403 and 404 are obtained, and the midpoint between the two center of gravity positions is defined as the touch center point 405. In the case of FIG. 4B, since the touch center point 405 is included in the touch panel center region 401, in step S203, the branch proceeds to step S204. In the opposite case, in FIG. 4C, since the similarly obtained touch center point 408 exists outside the touch panel central region 401, in step S203, the branch proceeds to step S205. The method of determining the direction of the moving axis by the moving command to the moving device 100 in steps S204 and S205 will be described in detail with reference to FIGS. 5 and 6.

FIG. 5 is a diagram showing a determination of a moving direction depending on a pinch-out position according to the first embodiment of the present invention. The touch panel center area 401, the touch panel frame 402, the finger areas 406, 407, and the touch center point 408 in FIG. 5A are the same as those in FIG. 4C. In FIG. 5A, in addition to the xy axis of FIG. 4, the z axis is also added for explanation. In FIG. 5A, a plane frame at a position three-dimensionally separated from the touch panel frame 402 by a distance L is virtualized, and this frame is referred to as a virtual front frame 502. The distance L is a preset value and is a value stored in the ROM 202 in the operation terminal 200. As shown by the hand 501, the plane actually touched is the position of the touch panel frame 402. The finger regions 406 and 407 and the touch center point 408 projected onto the virtual front frame 502 are referred to as the finger regions 503 and 504 and the touch center point 505.

In FIG. 5A, a virtual frame is set in front of the touch panel frame 502 at a distance L, whereas in FIG. 6, a virtual frame is set behind the distance L, and this virtual frame is set as a virtual rear frame. It is set to 606. The finger regions 406 and 407 and the touch center point 408 projected onto the virtual rear frame 606 are referred to as the finger regions 601, 602 and the touch center point 603.

The moving axis is determined in the optical axis direction in step S204, and this direction is the z-axis direction in FIG. 5A. If moving forward, the direction defined in the direction 506 of FIG. 5A is the traveling direction of the moving device 100. If the vehicle moves backward, the direction defined in the direction 604 of FIG. 6 is the traveling direction of the moving device 100.

In the case of forward movement, the moving axis determined in step S205 is the moving axis in the direction 507 from the center of the touch panel frame 402 toward the touch center point 505. FIGS. 5 (b) and 5 (c) show this as a projection onto the xz plane and the yz plane. Assuming that the xy coordinates of the touch center points 408 and 505 are (x1, y1), the angle θ1 formed by the straight line drawn from the origin to (x, z) = (x1, L) on the xz plane of FIG. 5 (b). Can be defined. On the yz plane of FIG. 5C, it can be defined by the angle θ2 formed by the straight line drawn to (x, z) = (y1, L).

If the vehicle is moving backward, the movement axis is the direction 605 from the center of the touch panel frame 402 toward the touch center point 603 in FIG. After determining the moving axis of the moving device 100 in this way, the process proceeds to step S206.

In step S206, it is determined whether or not the touch operation is pinched out from the operation direction of the touch operation. If it is a pinch-out, the process proceeds to step S207, and the moving direction is set to the forward direction of the moving axis. In this case, if the process has passed step S204, the direction of direction 506 is determined, and if the process has passed step S205, the direction of direction 507 is determined as the moving direction. If it is a pinch-in, the process proceeds to step S208, and the moving direction is set to the backward direction of the moving axis. In this case, if the step S204 has been passed, the direction of the direction 604 is determined, and if the step S205 has been passed, the direction of the direction 605 is determined as the moving direction. After determining the moving direction, the process proceeds to step S211.

Here, a case where an operation other than the pinch operation is performed in step S202 after returning to step S202 will be described.

If an operation other than the pinch operation is performed in step S202, the process proceeds to step S209, and it is determined whether or not the touch operation performed is a drag operation. If the touch operation performed in step S209 is determined to be a drag operation, the process proceeds to step S210 to determine the moving direction. A method of determining the moving direction when the drag operation is performed will be described in detail with reference to FIG. 7.

FIG. 7 is a diagram showing an operation corresponding to a drag operation according to the first embodiment of the present invention. In step S210, the up / down / left / right direction or the direction of a combination thereof is determined as the moving direction. In FIG. 7B, the finger area 701 is set as the start point of the drag operation, and the operation directions of the drag operation are the direction (1) (upward), the direction (2) (right direction), the direction (3) (downward), and the direction. (4) Consider the case (to the left). At this time, the traveling direction of the moving device corresponding to each direction input by the drag operation is shown in FIG. 7 (c). About the direction (1) (downward), direction (2) (leftward), direction (3) (upward), direction (4) (rightward) and the center of gravity of the finger region 701 in FIG. 7 (c), respectively. A symmetrical direction is set as the moving direction of the moving device 100. FIG. 7 (d) shows a display image of the LCD display unit 204 imaged by the image pickup apparatus 118 when traveling in a certain direction in this direction. When the drag operation is performed in the directions (1), (2), (3), and (4) in FIG. 7 (b), the images (1), (2), (3), and (4) shown in FIG. 7 (d) are displayed. You can get each. In the image viewer of a smartphone or tablet, the same effect as the change of the image when a general drag operation is performed on the image can be obtained. As a result, the operator can more intuitively operate the moving device 100, that is, the imaging device 118 by the drag operation. After the moving direction is determined by the drag operation in this way, the process proceeds to step S211.

After the moving direction is determined by the pinch operation and the drag operation, the determined moving direction (S211) and the moving start command (S212) are transmitted to the moving device 100, and the process proceeds to step S213.

In step S213, it is determined whether the touch-on state continues after the pinch operation and the drag operation are performed, and in this case, whether or not the finger is kept attached to the touch operation unit 205 after the operation. Specifically, even after the touch operation is detected, it is determined whether or not the touch-on state is continued for a certain period of time or longer.

If it is determined in step S213 that the touch operation is continuing, step S213 is repeated and waits until the touch-up is performed. That is, while the touch-on continues, the moving device 100 continues to move in the instructed direction. At that time, for example, as shown in FIG. 11, the moving direction of the moving device 100 in response to the touch operation may be displayed together with the image by an arrow or characters.

When it is determined that the touch operation is completed (touched up), the process proceeds to step S214, a movement stop command is transmitted to the moving device 100, and the process returns to step S201.

Here, a case where an operation other than the drag operation is performed in step S209 after returning to step S209 will be described. If an operation other than the drag operation is performed in step S209, the process proceeds to step S215 to determine whether or not there is a command corresponding to the operation. If there is a corresponding command, the process proceeds to step S216, the corresponding command is transmitted to the mobile device 100, and then the process returns to step S201. If there is no corresponding command, the process returns to step S201.

The processing of the operation terminal 200 according to the first embodiment has been described above, but here, the operation of the mobile device 100 when a command is received from the operation terminal 200 will be described with reference to FIG. FIG. 10 is a flowchart showing the processing of the mobile device according to the present embodiment. The flowchart of FIG. 10 illustrates a processing procedure that is executed by controlling each processing block of the mobile device 100, and is realized by expanding the program stored in the ROM 106 into the RAM 107 and executing the program by the CPU 105. ..

First, in step S1001, it is determined whether or not the movement direction and the movement start command have been received from the operation terminal 200. When the movement start command is received, the process proceeds to step S1002, and the moving device 100 starts moving in the instructed moving direction. If not received, step S1001 is repeated and movement control is not performed until an operation is performed.

After starting the movement control in step S1002, the process proceeds to step S1003, and it is determined whether or not the movement stop command has been received. When the movement stop command is received, the movement device 100 stops the movement (S1004). If the movement stop command has not been received, step S1002 is repeated, and the movement is continued until there is an operation.

8 and 9 show a case where pinch-out is performed as an example of this operation.

FIG. 8 is a diagram showing the operation of the moving device 100 when the pinch-out operation is performed in the central region of the touch panel. FIG. 8A shows a case where the finger is placed in the finger regions 801 and 802 and the pinch-out operation is performed in the directions 803 and 804 when the traveling direction is “advance”. In this case, the moving device 100 moves from the position 808 to the 809 direction as shown in FIG. 8B. At this time, the image displayed on the LCD display unit 204 is as shown in FIG. 8 (c).

FIG. 9 is a diagram showing the operation of the moving device 100 when the pinch-out operation is performed outside the central region of the touch panel. FIG. 9A shows a case where the finger is placed in the finger regions 901 and 903 and the pinch-out operation is performed in the directions 903 and 904, and the moving direction of the moving device 100 is not parallel to the optical axis of the imaging device 118. In this case, the moving device 100 moves diagonally to the upper right and forward, and the image displayed on the LCD display unit 204 at this time is as shown in FIG. 9B.

The moving speed of the moving device 100 may be a predetermined speed, but it may also be configured to transmit from the operation terminal 200. In that case, for example, it may be determined according to the tracing distance at the time of the pinch operation, or it may be accelerated according to the duration of the touch-on. For example, when the two points are moved in opposite directions while continuing the touch-on, such as moving in the direction of the pinch-in while the touch-on is continued after the pinch-out, the speed may be decelerated. In addition, if the pinch operation is repeatedly performed within a short period of time after being touched up, the stop command may not be sent and the movement may be continued, or the movement may be performed stepwise according to the number of times the pinch operation is repeated. You may change the speed. Further, the speed may be controlled according to the elapsed time from the start of movement of the moving device 100.

By the processing of the first embodiment described above, when the image pickup device mounted on the drone or the like and capable of moving the position is operated by the operation terminal having the touch panel function, the movement control of the image pickup device is intuitively easy to understand. Is possible.

(Second embodiment)
Hereinafter, the process according to the second embodiment of the present invention will be described with reference to the drawings. The portion where the same processing as that of the first embodiment is performed is indicated by the same symbol, and the description thereof will be omitted. In the second embodiment, for example, as shown in FIG. 12E, the image pickup device 118 has a configuration in which the pan / tilt angle can be changed with respect to the moving device 100, and a command is transmitted from the operation terminal 200 to the image pickup device 118. Pan-tilt control shall be possible by operation.

FIG. 13 is a flowchart showing the processing of the operation terminal according to the second embodiment of the present invention. The flowchart of FIG. 13 illustrates a processing procedure that is executed by controlling each processing block of the operation terminal 200, and is realized by expanding the program stored in the ROM 202 into the RAM 203 and executing the program by the CPU 201. ..

In step S1301, the operation terminal 200 acquires information regarding the imaging direction of the imaging device 118 with respect to the orientation of the mobile device 100. This information can be acquired, for example, by inquiring from the operation terminal 200 to the mobile device 100 at regular intervals, or by notifying the operation terminal 200 when the mobile device 100 changes the imaging direction of the image pickup device 118.

After acquiring the information regarding the imaging direction of the imaging device 118 with respect to the moving device 100, the process proceeds to step S1302, and whether or not the pan / tilt operation is performed, that is, the normal position of the moving device 100 and the optical axis direction of the imaging device 118 are one. Determine if you are doing it. The matching state means that the moving device 100 and the imaging device 118 are in the same front-rear direction, up-down direction, and left-right direction. If they match, the process proceeds to step S1304.

If they do not match, the process proceeds to step S1303, and a process of matching the moving coordinate system of the moving device 100 and the optical coordinate system of the imaging device 118 at the pan-tilt angle is performed. Here, the moving coordinate system is a coordinate system representing the front-back direction, the up-down direction, and the left-right direction of the moving device 100. The optical coordinate system is a coordinate system representing a coordinate system representing the front-back direction, the up-down direction, and the left-right direction of the image pickup apparatus 118, and matches the coordinate system of the image displayed on the operation terminal 200. The coordinate transformation process in step S1303 will be described with reference to FIG.

12 (a) shows a pinch-out in which a finger is placed in the finger areas 1201 and 1202 in the center of the touch operation unit and the finger is spread in the directions 1203 and 1204, (b) is a pinch-in, and (c) is a drag operation. .. In FIG. 12D, the pan-tilt angle of the image pickup device 118 is not attached to the moving device 100, and the forward direction of the moving coordinate system of the moving device 100 and the optical axis direction 1210 of the image pickup device 118 coincide with each other. If this is the case. In this case, the pinch-out shown in FIG. 12 (a) advances in the direction (1) of FIG. 12 (d), and the pinch-in operation shown in FIG. 12 (b) advances in the direction (2) of FIG. 12 (d). .. 12 (c) When the operations of the directions (3), (4), (5), and (6) are performed by the drag operation, the directions (3), (4), and (5) of FIG. 12 (d) are performed, respectively. , (6).

Here, consider a case where the image pickup device 118 has a tilt angle in the −90 degree direction with respect to the moving device 100 as shown in FIG. 12 (e). In this case, if the operator does not move in the vertical downward direction, that is, in the direction (1) in FIG. 12 (e) in response to the pinch-out in FIG. 12 (a), the operator displays the display on the LCD display unit 204 of the operation terminal 200. Becomes unnatural. Therefore, in this case, the direction of the optical axis direction 1211 is set as the forward direction of the pinch-out operation, and the directions (2), (3), (4), (5), and (5) of FIG. 6) Determine the directional coordinates. Then, the moving direction instructed by the pinch operation is converted based on the relative relationship between the moving device 100 and the imaging device 118, and the moving direction of the moving device 100 is determined. In the example of FIG. 12E, the case where the tilt angle is −90 degrees is shown, but even if the pan / tilt angle is an arbitrary value, the optical axis direction is aligned with the pinch-out direction as described above. The moving coordinate system can be converted based on. In the following processing, it is assumed that the moving direction is determined after performing this coordinate conversion.

Since the subsequent processes of steps S1304 to S1319 are the same as those of steps S201 to S216 of the first embodiment, detailed description thereof will be omitted.

In this embodiment, the operation terminal 200 calculates the movement direction and the movement distance according to the touch operation and transmits the operation information to the movement device 100. However, the operation terminal 200 transmits the operation information and the movement device 100 transmits the operation information. The moving direction and moving speed may be calculated from the received operation information. In that case, the moving device 100 converts the operation information to calculate the moving direction based on the relative relationship between the directions of the moving device 100 and the imaging device 118.

By the process of the second embodiment described above, even in a situation where the forward direction of the drone or the like and the direction of the optical axis of the image pickup device are arbitrarily changed, the movement of the moving device 100 can be intuitively and easily understood by a touch operation. ..

In this embodiment, the moving device 100 moves in the designated direction from the reception of the movement start command to the reception of the stop command, but it is configured to recognize an obstacle or the like during the movement operation and automatically stop. It is also possible to do. Even if the stop command from the operation terminal 200 is not received, if an obstacle or the like is present by the subject recognition unit 108, a collision is performed for safety by using the distance information detection enabled by the configuration of the image sensor 103. Stop at a distance that does not. At this time, the limit of the approaching distance may be changed depending on the type of the obstacle detected by the subject recognition unit. For example, when the obstacle is an object such as a wall, the moving device 100 stops in front of the obstacle by a distance A, and when the obstacle is a person, the moving device 100 is in a position in front of the obstacle by a distance B. Stop at (distance B> distance A).

Further, when the movement is stopped due to the movement restriction of an obstacle or the like, the control may be switched so that the zoom operation is performed in the same direction. In that case, it is possible to change the color of the arrow shown in FIG. 11 and the character string to be displayed so that it is easy to recognize that the movement cannot be performed due to an obstacle or the like and that the zoom operation has been switched. Similarly, the movement of the moving device by the drag operation may be switched to the change of the imaging direction.

Further, in this embodiment, the movement of the moving device 100 is limited to forward / backward movement and vertical / horizontal movement, but a turning movement may be accepted according to the circular tracing operation. For example, an annular UI is prepared, and a right-handed turn command is transmitted to the moving device 100 by tracing clockwise with a finger, and a left-turning command is transmitted to the moving device 100 when tracing counterclockwise.

Further, for example, by touching the switching member displayed on the LCD display unit 204, the operation corresponding to the pinch operation may be configured to switch between the movement in the front-rear direction and the zoom operation. Similarly, it is also possible to switch the operation corresponding to the drag operation between moving in the left-right direction or up-down direction and changing the imaging direction (pan-tilt operation). As described above, in the present embodiment, the movement instruction to the moving device 100 and the zoom instruction to the imaging device 118 can be selectively executed according to a predetermined condition.

In this embodiment, a drone or the like has been described as an example, but the moving device does not have to be able to move in three dimensions, and can be applied to, for example, a surveillance camera whose moving direction is regulated by a rail or the like. ..

The above is a description of a preferred embodiment of the present invention, but the present invention is not limited to the above embodiment within the scope of the technical idea of the present invention, and is appropriately modified and adapted depending on the target circuit form. It should be done.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Move device 101 Imaging optical system 102 Imaging optical system operation unit 103 Image sensor 104 Image pickup signal processing unit 105 CPU
106 ROM
107 RAM
108 Subject recognition unit 109 Memory slot 110 Memory card 111 Drive unit 112 Direction detection sensor 113 GPS
114 Accelerometer 115 Gyro 116 Communication unit 118 Imaging device 200 Operation terminal 201 CPU
202 ROM
203 RAM
204 LCD display 205 Touch operation 206 Communication

Claims (16)

An operating device that externally controls a mobile device that holds an imaging means.
A display means for displaying an image captured by the image pickup means, and a display means.
Touch operation means to detect touch operation and
It has an instruction means for moving the moving device based on the operation direction of the move operation when the touch operation is a move operation in response to the touch operation detected by the touch operation means.
When the touch-on state is continuously detected by the touch operation means after the move operation,
The instruction means is an operation device characterized in that movement based on the move operation is continued. The instruction means responds to an increase in the duration when the touch-on state is continuously detected after the touch operation or the number of repetitions when the same operation is repeated at intervals within a predetermined time. The operating device according to claim 1, wherein the speed of movement is increased. The operation device according to claim 1 or 2, wherein the instruction means moves the moving device so that the image displayed on the display means moves in the operation direction of the move operation. The operating device according to any one of claims 1 to 3, wherein the instruction means further determines a direction of movement based on an installation angle between the moving device and the imaging means. The instruction means according to any one of claims 1 to 4, wherein the moving device is swiveled when the touch operation detected by the touch operation means is an arcuate tracing operation. Operating device. The operating device according to any one of claims 1 to 5, wherein the display means superimposes and displays the moving direction of the moving device on the image. The operating device according to any one of claims 1 to 6.
With the moving device
A control system characterized by having. A control method for controlling a mobile device holding an imaging means from an external operating device.
The operating device includes a display means for displaying an image captured by the imaging means and a detecting means for detecting a touch operation.
In response to the touch operation detected by the detection means,
When the touch operation is a move operation, a step of moving the moving device based on the operation direction of the move operation, and
If the touch-on state is continuously detected by the detection means after the move operation, the step of continuing the movement based on the move operation and
A control method characterized by having. After the detection of the touch operation, the duration when the touch-on state is continuously detected or the number of repetitions when the same operation is repeated at intervals within a predetermined time increases. The control method according to claim 8, further comprising a step of increasing the speed of movement. 8. The aspect of claim 8 or 9, wherein the moving device is moved so that the image displayed on the display means moves in the operating direction of the move operation based on the operating direction of the move operation. Control method. The control method according to any one of claims 8 to 10, further comprising determining the direction of the movement based on the installation angle of the moving device and the imaging means. When the touch operation detected by the detection means is an arcuate tracing operation, a step of rotating the moving device and a step of rotating the moving device.
The control method according to any one of claims 8 to 11, wherein the control method comprises. The control method according to any one of claims 8 to 12, further comprising a step of superimposing the moving direction of the moving device on the image and displaying it on the display means. A communication means that receives operation information including a touch operation pattern and an operation direction detected by the detection means from the operation device.
A control means for executing each step according to any one of claims 8 to 13 based on the operation information, and
A mobile device characterized by having. A computer program for causing a computer to execute each step according to any one of claims 8 to 14. A computer-readable recording medium according to claim 15.

Images