JP2017169170A - Imaging apparatus, moving apparatus, imaging system, imaging method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus, a moving apparatus, an imaging system, an imaging method, and a program which enable an effective coordination even when an operator of the moving device and a photographer of the imaging apparatus respectively operate the devices.SOLUTION: An imaging apparatus 20 comprises an imaging part 21 which is mounted to a moving apparatus 10 and capable of communication with the moving apparatus 10 bidirectionally, and images a subject to generate image data; and a second control part 28 which transmits to the moving apparatus 10 information that can be displayed by a first display part 32 of an operating apparatus 30 for operating the moving apparatus 10.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an imaging device, a moving device, an imaging system, an imaging method, and a program for imaging a subject and generating image data of the subject.

  2. Description of the Related Art In recent years, a technique for preventing collision by analyzing video data transmitted from a camera provided in a mobile robot and detecting an obstacle present in the traveling direction of the mobile robot is known (see Patent Document 1). . In this technique, a control device for manipulating a mobile robot analyzes video data transmitted from a camera, and displays the analysis result and video data on the display screen of the control device.

JP 2007-320024 A

  By the way, in recent years, when an imaging device such as a camera is attached to a mobile device such as an unmanned aerial vehicle including a mobile robot or a drone, the operator and the photographer are separately used for the mobile device and the imaging device using separate operation terminals. In the situation of maneuvering and shooting while cooperating, improvement of usability of multi-person collaboration was insufficient.

  The present invention has been made in view of the above, and an imaging device, a mobile device, and an imaging device capable of effective cooperation even when operated by a driver of the mobile device and a photographer of the imaging device, respectively. It is an object to provide a system, an imaging method, and a program.

  In order to solve the above-described problems and achieve the object, an imaging apparatus according to the present invention is an imaging apparatus that is mounted on a mobile device and capable of bidirectional communication with the mobile device. An image pickup unit that generates data and a control unit that transmits information that can be displayed on a display unit of a control device that controls the mobile device to the mobile device.

  In the imaging device according to the present invention as set forth in the invention described above, the information includes at least the image data generated by the imaging unit.

  The imaging device according to the present invention further includes a distance detection unit that detects a distance from the imaging device to the subject based on the image data, and the information is detected by the distance detection unit. Further, distance information regarding the distance is further included.

  The imaging device according to the present invention is the communication unit according to the above invention, wherein the communication unit receives an instruction signal for instructing the imaging device to perform shooting from the operating device that operates the imaging device, and transmits the image data to the operating device. When the communication unit receives the instruction signal, the control unit causes the imaging unit to perform imaging, and the information includes reception information indicating that the communication unit has received the instruction signal. It is further characterized by including.

  In the imaging device according to the present invention, in the above invention, when the control unit receives a control signal for controlling movement of the moving device according to an operation of the control device from the moving device, the communication unit And transmitting control information related to the control signal to the controller device.

  Moreover, in the imaging device according to the present invention, in the above invention, when the position information related to the position of the moving device is input from the moving device, the control unit transmits the position information to the operating device via the communication unit. It is characterized by transmitting to.

  In the imaging device according to the present invention, in the above invention, when the mobile device is returning to the takeoff start position, the control unit stops shooting by the imaging unit when the instruction signal is received. It is characterized by that.

  In addition, the mobile device according to the present invention is equipped with an imaging device that captures an image of a subject and generates image data, can communicate with the imaging device bidirectionally, and transmits a signal transmitted from a control device having a display unit. The mobile device moves in response, and includes a communication unit that receives the signal transmitted from the control device and transmits information input from the imaging device to the control device.

  In the mobile device according to the present invention as set forth in the invention described above, the information includes at least the image data.

  In the mobile device according to the present invention as set forth in the invention described above, the information further includes distance information related to a distance from the imaging device to the subject.

  In the mobile device according to the present invention, in the above invention, a position detector that detects position information related to a current position of the mobile device, and a control that transmits the position information detected by the position detector to the imaging device. And a section.

  In the mobile device according to the present invention, in the above invention, the information further includes reception information indicating that the imaging device has received an instruction signal instructing the imaging device to perform imaging, and the control unit includes the control unit, When the communication unit receives the signal, the mobile device is stationary when the information is received.

  In the mobile device according to the present invention, in the above invention, the control unit maintains the feedback of the mobile device when the information is received when the mobile device is returned to the takeoff start position. It is characterized by.

  In addition, an imaging system according to the present invention includes an imaging device that captures an image of a subject and generates image data, a mobile device that is equipped with the imaging device and is capable of bidirectional communication with the imaging device, and controls the mobile device. An imaging system including an operating device for operating the imaging device, and the imaging device provides information that can be displayed on a display unit of the operating device for operating the moving device to the moving device. A control unit for transmitting, and the mobile device includes a communication unit that receives the signal transmitted from the control device and transmits the information input from the imaging device to the control device. It is characterized by.

  In addition, an imaging method according to the present invention is an imaging method executed by an imaging device that is mounted on a mobile device and capable of bidirectional communication with the mobile device, the imaging step of imaging a subject and generating image data; And a control step of transmitting to the mobile device information that can be displayed on a display unit of the control device that controls the mobile device.

  A program according to the present invention is a program executed by an imaging device that is mounted on a mobile device and capable of bidirectional communication with the mobile device, the imaging step of imaging a subject and generating image data, And a control step of transmitting information that can be displayed on the display unit of the control device that controls the mobile device to the mobile device.

  According to the present invention, even when the operator of the mobile device and the photographer of the imaging device operate each other, effective cooperation is possible, and it is possible to concentrate on each other's operation.

FIG. 1 is a schematic diagram showing a schematic configuration of an imaging system according to an embodiment of the present invention. FIG. 2 is a block diagram showing a functional configuration of the imaging system according to the embodiment of the present invention. FIG. 3 is a flowchart showing an outline of the operation executed by the imaging apparatus according to the embodiment of the present invention. FIG. 4 is a diagram illustrating an example of an image displayed by the operation device according to the embodiment of the present invention. FIG. 5 is a diagram illustrating an example of an image displayed by the control device according to the embodiment of the present invention. FIG. 6 is a flowchart showing an outline of operations performed by the mobile device according to the embodiment of the present invention. FIG. 7 is a schematic diagram showing a schematic configuration of an imaging system according to a modification of the embodiment of the present invention.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention (hereinafter referred to as “embodiments”) will be described with reference to the drawings. The present invention is not limited to the embodiments described below. Further, in the description of the drawings, the same portions will be described with the same reference numerals.

[Schematic configuration of imaging system]
FIG. 1 is a schematic diagram showing a schematic configuration of an imaging system according to an embodiment of the present invention. FIG. 2 is a block diagram showing a functional configuration of the imaging system according to the embodiment of the present invention. The imaging system 1 shown in FIGS. 1 and 2 has an imaging device mounted on a moving device, and the photographer U2 controls the imaging device using the operating device while the operator U1 moves the moving device using the steering device. This is a system capable of photographing the subject P1.

  As shown in FIGS. 1 and 2, the imaging system 1 includes a moving device 10, an imaging device 20, a control device 30, and an operation device 40. The imaging device 20 is detachably attached to the moving device 10. Of course, the imaging device 20 may be attached to the moving device 10 via a stabilizer, a vibration correction mechanism (for example, a gimbal) and a rig. The mobile device 10 and the imaging device 20 are connected via a cable 2 such as a USB so as to be capable of bidirectional communication. In addition, the mobile device 10 and the control device 30 are both in a first frequency band (for example, 27 MHz, 40 MHz, 72 MHz, 73 MHz, a frequency band according to a radio standard of each country such as 2.4 GHz, 5 GHz, and 5.8 GHz). Are communicably connected to each other. In addition, the imaging device 20 and the operation device 40 have a second frequency band different from the first frequency band (for example, when the first frequency is 5 GHz, a frequency band according to a wireless standard in each country such as 5.8 GHz). It is connected so that it can communicate in both directions. In the present embodiment, the mobile device 10 and the imaging device 20 are connected by the cable 2. However, the present invention is not limited to this, and a configuration capable of wireless communication using the third frequency band is adopted. Also good.

[Configuration of mobile device]
First, the configuration of the moving device 10 will be described. The mobile device 10 has an unmanned aerial vehicle (UAV) according to the present invention. The moving device 10 is configured as a drone of a rotary vane drone having four rotors 10a. The number of rotors 10a is not limited to four, and other numbers may be provided. Furthermore, the moving device 10 is not limited to a drone of a rotary wing drone, and may be constituted by other unmanned aircraft such as a drone of a fixed wing drone. Furthermore, the mobile device 10 may be constituted by a self-propelled device that can be steered wirelessly, such as a self-propelled robot, a car, a model car, a capsule endoscope, and a ship.

  The mobile device 10 includes a propulsion unit 11, a power source 12, a spatial information acquisition unit 13, a position / orientation detection unit 14, an altitude / attitude detection unit 15, a first recording unit 16, a first communication unit 17, and a first communication unit 17. 2 communication unit 18 and first control unit 19.

  The propulsion unit 11 uses a plurality of (four in the example of FIG. 1) rotors 10a and a plurality of motors (not shown) for driving each rotor 10a under the control of the first control unit 19. The mobile device 10 is caused to fly.

  The power supply 12 is configured using a battery, a booster circuit, and the like. The power supply 12 adjusts each part of the mobile device 10 to a predetermined voltage and supplies power to each part.

  The spatial information acquisition unit 13 is configured using a laser radar or the like. The spatial information acquisition unit 13 irradiates the pulse laser beam radially from the moving device 10 and measures the return time of the reflected pulse laser beam to thereby detect each obstacle (direction of each irradiation point) around the moving device 10. Spatial information regarding the distance to each irradiation point) is acquired, and the acquired result is output to the first control unit 19. The spatial information acquisition unit 13 is configured using, for example, a plurality of imaging devices, and extracts the feature amount from the image data from each imaging device, thereby determining the distance and direction of each obstacle around the moving device 10. You may get it.

  The position / orientation detection unit 14 is configured using a GPS (Global Positioning System) receiver, a magnetic direction sensor, or the like. The position / orientation detection unit 14 includes current position information related to the current position of the mobile device 10 and an angle (azimuth) formed by the nose direction (the direction in which the nose faces) of the mobile device 10 with respect to a predetermined reference orientation (for example, north). ) Direction information is detected, and the detection result is output to the first control unit 19. In the present embodiment, the position / orientation detection unit 14 functions as a position detection unit.

  The altitude posture detection unit 15 is configured using an atmospheric pressure sensor, a gyro sensor (angular velocity sensor), an inclination sensor (acceleration sensor), and the like. The altitude posture detection unit 15 includes altitude information regarding the altitude of the imaging device 20, tilt angle information regarding the tilt angles of the image capturing device 20 (tilt angles with respect to a reference posture in which the four rotors 10 a are positioned in the horizontal plane), and rotation of the moving device 10. Rotation angle information relating to the angle (rotation angle centered on a vertical line passing through the center position of the four rotors 10a) is detected, and the detection result is output to the first control unit 19.

  The first recording unit 16 is configured using a recording medium such as a flash memory or an SDRAM (Synchronous Dynamic Random Access Memory). The first recording unit 16 records various programs for driving the mobile device 10 and temporarily records information being processed.

  The first communication unit 17 is configured using a predetermined communication module. The first communication unit 17 performs wireless communication with the control device 30 that remotely controls the flight operation of the imaging device 20 under the control of the first control unit 19. Further, the first communication unit 17 transmits information input from the imaging device 20 to the control device 30 under the control of the first control unit 19.

  The second communication unit 18 is configured using a communication module. The second communication unit 18 communicates with the imaging device 20 via the cable 2 under the control of the first control unit 19.

  The first control unit 19 is configured using a CPU (Central Processing Unit) or the like, and is input via the second communication unit 18, an instruction signal from the control device 30 input via the first communication unit 17. The operation of the propulsion unit 11 (the flight operation of the mobile device 10) and communication are controlled according to the data from the imaging device 20.

  Here, a detailed configuration of the first control unit 19 will be described. The first control unit 19 includes a power source determination unit 191, a movement determination unit 192, a posture determination unit 193, a propulsion control unit 194, a direction control unit 195, a posture control unit 196, and a first communication control unit 197. Have.

  The power source determination unit 191 detects the remaining amount of the power source 12 (remaining power amount), and determines the flight time and the flight distance that the mobile device 10 is capable of based on the detection result.

  The movement determination unit 192 is based on the current position information and azimuth information detected by the position / orientation detection unit 14, the altitude information detected by the altitude posture detection unit 15, and the flight time determined by the power supply determination unit 191. Determine the travel distance.

  The posture determination unit 193 determines the posture of the imaging device 20 based on the tilt angle information and the rotation angle information detected by the altitude posture detection unit 15.

  The propulsion control unit 194 propels the mobile device 10 based on the determination results of the power source determination unit 191, the movement determination unit 192, and the posture determination unit 193. Specifically, the propulsion control unit 194 raises, moves forward, stops, moves backward, and moves down the moving device 10 by independently controlling the rotational speeds of the four rotors 10a. For example, the propulsion control unit 194 changes the moving direction of the moving device 10 to front and rear (the nose direction of the moving device 10 is front) or left and right (right and left when viewed in the nose direction of the moving device 10).

  The direction control unit 195 turns the moving device 10 based on the determination results of the power source determination unit 191, the movement determination unit 192, and the posture determination unit 193. Specifically, the direction control unit 195 controls the rotation speed of the four rotors 10a independently, and the attitude control unit 196 moves by controlling the rotation speeds of the four rotors 10a independently. The tilt angle and rotation angle of the device 10 are changed.

  The posture control unit 196 controls the posture of the mobile device 10 based on the determination result of the posture determination unit 193 and the operation signal of the control device 30.

  The first communication control unit 197 controls communication of the first communication unit 17 and the second communication unit 18. Specifically, the first communication control unit 197 transmits information input from the imaging device 20 via the second communication unit 18 to the control device 30 via the first communication unit 17. In addition, the first communication control unit 187 transmits information including operation information input from the control device 30 via the first communication unit 17 (hereinafter referred to as “mobile device information”) via the second communication unit 18. It transmits to the imaging device 20. Here, the mobile device information refers to the current position of the mobile device 10, control information (operation information) related to a control signal for controlling the movement of the mobile device 10 transmitted from the control device 30, and the moving direction and movement of the mobile device 10. The altitude of the apparatus 10 and the state of the moving apparatus 10 (for example, normal movement, return movement, remaining battery level).

[Configuration of imaging device]
Next, the configuration of the imaging device 20 will be described. The imaging device 20 is fixed so as not to move with respect to the moving device 10 in a state where the shooting direction is aligned with the nose direction of the moving device 10. Note that the imaging device 20 may be fixed to the moving device 10 via a rotation mechanism so that the photographing region can be changed with respect to the moving device 10. Of course, the imaging device 20 may be fixed to the moving device 10 via a stabilizer, a vibration correction mechanism (for example, a gimbal), a rig, or the like.

  As illustrated in FIG. 2, the imaging device 20 includes an imaging unit 21, an elevation angle direction detection unit 22, a clock 23, a first operation unit 24, a third communication unit 25, a fourth communication unit 26, 2 recording unit 27 and second control unit 28.

  The imaging unit 21 captures a subject and generates image data under the control of the second control unit 28. The imaging unit 21 includes an optical system 211 and an imaging element 212.

  The optical system 211 has at least a focus lens, a zoom lens, a shutter, and a diaphragm. The optical system 211 forms a subject image on the light receiving surface of the image sensor 212. The optical system 211 changes each shooting parameter such as zoom magnification, focus position, aperture value, and shutter speed under the control of the second control unit 28.

  The image sensor 212 receives the subject image formed by the optical system 211 and performs photoelectric conversion to generate image data, and outputs the image data to the second control unit 28. The imaging device 212 is configured using a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like. In addition, the imaging element 212 includes phase difference pixels (not shown) for detecting the distance of the subject and pixels for generating image data arranged in a two-dimensional matrix. The number of phase difference pixels can be set as appropriate.

  The elevation angle azimuth detector 22 is configured using an azimuth sensor, a triaxial acceleration sensor, a gyro sensor, and a magnetic azimuth sensor. The elevation angle azimuth detecting unit 22 is elevation angle information related to the inclination angle (elevation angle) of the imaging device 20 (the optical axis of the optical system 211) with respect to the horizontal plane, and the shooting direction of the imaging device 20 with respect to a predetermined reference orientation (for example, north). Direction information regarding the angle (direction) to be formed is detected, and the detection result is output to the second control unit 28.

  In addition to the timekeeping function, the clock 23 generates date / time information related to the date / time taken by the imaging unit 21 and outputs the date / time information to the second control unit 28.

  The first operation unit 24 is provided on the outer surface of the imaging device 20 and is configured by using a button, a switch, or the like that receives an operation by the user of the imaging device 20, and receives an input of an instruction signal corresponding to the operation by the user.

  The third communication unit 25 communicates with the mobile device 10 via a cable under the control of the second control unit 28.

  The fourth communication unit 26 performs wireless communication with the controller device 40 under the control of the second control unit 28. The fourth communication unit 26 is configured using a wireless communication module.

  The second recording unit 27 is configured using a flash memory, an SDRAM, and a memory card. The second recording unit 27 includes an image data recording unit 271 that records image data generated by the imaging unit 21 and a program recording unit 272 that records various programs executed by the imaging device 20.

  The second control unit 28 is configured using a CPU or the like. In accordance with an instruction signal input from the operating device 40 via the fourth communication unit 26, the imaging of the imaging unit 21 is controlled, and image data generated by the imaging unit 21 is transmitted to the operating device 40 or the moving device 10. . In this case, in addition to sending the image data as it is, the second control unit 28 may be temporally and pixel-thinned data (resized image data) or compressed image data (reduced image data or compression processing). Image data). In addition, the second control unit 28 performs processing for increasing the visibility of the image data (for example, contrast enhancement processing and exposure value reduction processing) or results of analyzing the image (the subject by the image feature extraction unit 282 described later). May be transmitted.

  Here, a detailed configuration of the second control unit 28 will be described. The second control unit 28 includes an image processing unit 281, an image feature extraction unit 282, a focus position acquisition unit 283, an angle of view acquisition unit 284, a distance detection unit 285, a trimming unit 286, and a second communication control unit. 287 and a photographing control unit 288.

  The image processing unit 281 performs predetermined image processing on the image data generated by the imaging unit 21. Specifically, the image processing unit 281 performs gain-up processing, white balance processing, gradation processing, synchronization processing, and the like on the image data.

  The image feature extraction unit 282 extracts feature amounts included in the image data subjected to image processing by the image processing unit 281. Specifically, the image feature extraction unit 282 extracts features of the subject, for example, the face and face position and size, gender, luminance value, and feature amount (vector, etc.) based on the image data.

  The focus position acquisition unit 283 acquires focus position information regarding the focus position of the optical system 211 of the imaging unit 21. Specifically, the focus position acquisition unit 283 acquires the position of the optical system 211 that is in focus with the subject as focus position information.

  The view angle acquisition unit 284 acquires the current view angle of the optical system 211 of the imaging unit 21. Specifically, the angle-of-view acquisition unit 284 acquires the position of each of the plurality of zoom lenses constituting the optical system 211, and calculates the angle of view from the acquired position.

  The distance detection unit 285 moves from the imaging device 20 to the subject based on a contrast change between two pieces of image data generated by the imaging unit 21 that are temporally changed or phase difference information by phase difference pixels provided in the imaging element 212. Detect the distance. This information can be used for focusing the optical system 211, and is also important information for the maneuvering side that steers the moving apparatus 10 as distance information between the moving apparatus 10 and the subject and the moving apparatus 10 and the object. Even when shooting with a general imaging device 20 (camera), various distances to a target object, zoom operations, and the like are combined to make various pictures and take special photos, so the mobile device 10 and the imaging device 20 It is important for the pilot U1 to know the distance that the photographer U2 cares about, in addition to the distance information for flight and movement, at the time of cooperative shooting. Not only the distance information but also a change in distance and a moving direction information that can be determined from the information can be used effectively. Even when shooting without changing the distance and changing other conditions, it is more advantageous to take a picture using this information. With this information (information on the distance from the imaging device 20 to the subject), the operator U1 may operate the control device 30 to perform an operation for designating stationary control such as hovering, or the first of the mobile device 10. Based on this information (information on the distance from the imaging device 20 to the subject), the control unit 19 may automatically perform the same stationary control on the moving device 10. Also, when shooting in a predetermined order while changing the distance, this information can be used effectively when shooting from various angles with a constant distance, and can be applied to accurate 3D information acquisition shooting.

  The trimming unit 286 generates trimming image data and thumbnail image data by performing trimming processing on the image data on which the image processing unit 281 has performed image processing.

  The second communication control unit 287 performs communication control of each of the third communication unit 25 and the fourth communication unit 26. Specifically, the second communication control unit 287 can display information that can be displayed on the first display unit 32 of the control device 30 for operating the mobile device 10 via the third communication unit 25 (hereinafter referred to as “imaging device”). Information ”) to the mobile device 10. When the second communication control unit 287 receives an instruction signal for instructing photographing from the operation device 40 via the fourth communication unit 26, the second communication control unit 287 causes the imaging unit 21 to perform photographing and the fourth communication unit 26 operates the operation. The reception information indicating that the instruction signal instructing photographing is received from the device 40 is transmitted to the mobile device 10 via the third communication unit 25. Further, the second communication control unit 287 transmits the mobile device information input from the mobile device 10 via the third communication unit 25 to the operation device 40 via the fourth communication unit 26.

  The imaging control unit 288 controls imaging of the imaging unit 21. Specifically, the imaging control unit 288 causes the imaging unit 21 to perform imaging when an instruction signal instructing imaging is input from the operation device 40 via the fourth communication unit 26.

[Configuration of control device]
Next, the configuration of the control device 30 will be described. The control device 30 includes a fifth communication unit 31, a first display unit 32, a second operation unit 33, a third recording unit 34, and a third control unit 35.

  The fifth communication unit 31 is configured using a communication module. The fifth communication unit 31 wirelessly communicates with the first communication unit 17 of the mobile device 10 under the control of the third control unit 35 to transmit an instruction signal from the control device 30 and the mobile device 10. The state information transmitted from is received and output to the third control unit 35.

  The first display unit 32 displays various information related to the moving device 10 under the control of the third control unit 35. The 1st display part 32 is comprised using display panels, such as a liquid crystal and organic EL (Electro Luminescence).

  The second operation unit 33 is configured by using a button, a switch, a ten-hour key, and the like that accept an operation by the pilot, receives an instruction signal input according to the operation by the pilot, and sends the instruction signal to the third control unit 35. Output to.

  The third recording unit 34 records various programs related to the control device 30. The third recording unit 34 is configured using a Flash memory or SDRAM.

  The third control unit 35 is configured using a CPU and controls each unit of the control device 30. The third control unit 35 controls the communication between the fifth communication unit 31 and the first communication unit 17 of the mobile device 10, and the first display control controls the display on the first display unit 32. Part 352.

[Configuration of operation device]
Next, the configuration of the operation device 40 will be described. The operation device 40 includes a sixth communication unit 41, a second display unit 42, a third operation unit 43, a fourth recording unit 44, and a fourth control unit 45.

  The sixth communication unit 41 is configured using a communication module. The sixth communication unit 41 transmits an instruction signal from the operation device 40 by performing wireless communication with the fourth communication unit 26 of the imaging device 20 under the control of the fourth control unit 45, and at the same time, the imaging device 20. The information including the image data transmitted from is received and output to the fourth control unit 45.

  The second display unit 42 displays an image corresponding to the image data generated by the imaging device 20 and various types of information related to the imaging device 20 under the control of the fourth control unit 45. The 2nd display part 42 is comprised using display panels, such as a liquid crystal and organic EL.

  The third operation unit 43 is configured by using a button, a switch, a touch panel, or the like that accepts an operation by the photographer. The third operation unit 43 accepts an input of an instruction signal according to an operation by the photographer and outputs the instruction signal to the fourth control unit 45. To do.

  The fourth recording unit 44 records various programs related to the operation device 40 and image data generated by the imaging device 20. The fourth recording unit 44 is configured using a flash memory, an SDRAM, a memory card, and the like.

  The fourth control unit 45 is configured using a CPU and controls each unit of the operation device 40. The fourth control unit 45 is configured to control communication between the sixth communication unit 41 and the fourth communication unit 26 of the imaging device 20, and second display control to control display of the second display unit 42. Part 452.

[Operation of imaging system]
Next, the operation of the imaging system 1 described above will be described. Hereinafter, as the operation processing of the imaging system 1, the operation of the imaging device 20 will be described and then the operation of the moving device 10 will be described.

[Operation of imaging device]
FIG. 3 is a flowchart showing the operation of the imaging apparatus 20. As shown in FIG. 3, first, the second control unit 28 determines whether or not communication connection with the mobile device 10 is possible via the cable 2 by the third communication unit 25 (via the cable 2). Whether or not the mobile device 10 and the imaging device 20 are connected) and whether or not the communication state with the controller device 40 is good via the fourth communication unit 26 (step S101). . Specifically, the second control unit 28 determines that the communication connection with the mobile device 10 is not possible (for example, when it is determined that the cable 2 is not connected) or the operation. When it is determined that the communication state with the device 40 is not good, an LED (Light Emitting Diode) (not shown) or a speaker (not shown) provided on the outer surface of the imaging device 20 (the moving device 10) is used. Informing that the communication state is abnormal. In this case, the controller device 40 may cause the second display unit 42 to display that the normal state is abnormal.

  Subsequently, the second control unit 28 is set to a communication mode in which communication is performed according to an instruction signal from the operation device 40 while cooperating with the mobile device 10 via the third communication unit 25 and the fourth communication unit 26. It is judged whether it is (step S102). If the second control unit 28 determines that the communication mode is set (step S102: Yes), the imaging device 20 proceeds to step S103 described later. On the other hand, when the second control unit 28 determines that the communication mode is not set (step S102: No), the imaging device 20 proceeds to step S119 described later.

  In step S <b> 103, the second control unit 28 determines whether or not communication with the mobile device 10 is possible via the third communication unit 25. When the second control unit 28 determines that communication with the mobile device 10 is possible via the third communication unit 25 (step S103: Yes), the imaging device 20 proceeds to step S104 described later. In contrast, when the second control unit 28 determines that communication with the mobile device 10 is not possible via the third communication unit 25 (step S103: No), the imaging device 20 proceeds to step S115 described later.

  In step S104, the imaging control unit 288 causes the imaging unit 21 to perform imaging. Then, the shooting control unit 288 executes shooting preparation processing for performing AF processing and AE processing on the imaging unit 21 (step S105). In this case, the distance detection unit 285 detects the distance from the imaging device 20 to the subject based on the image data generated by the imaging unit 21. Of course, the distance detection unit 285 may detect the distance from the imaging device 20 to the subject based on temporally continuous image data, or based on the AF processing result and the image data. The distance from the subject to the subject may be detected. Further, the trimming unit 286 generates trimming image data by performing a trimming process on the image data generated by the imaging unit 21. The distance detection unit 285 may detect the distance from the imaging device 20 to the subject using another known technique.

  Subsequently, the second communication control unit 287 transmits the imaging device information to the mobile device 10 via the third communication unit 25 (step S106). Here, the imaging device information includes the distance information regarding the distance to the subject detected by the distance detection unit 285, the image data generated by the imaging unit 21, the trimming image data generated by the trimming unit 286, and the fourth communication unit 26. The contents of the instruction signal input from the controller device 40 via the control device 40 (for example, a release signal for instructing shooting, a first release signal for instructing shooting preparation operations for AF processing and AE processing, and an instruction signal for changing the shooting parameters of the imaging unit 21) ). As a result, as shown in FIG. 4, in the first display unit 32 of the control device 30, the distance T <b> 2 to the subject, the image W <b> 1 corresponding to the image data captured by the imaging device 20, and the progress information regarding the traveling direction of the moving device 10. A1, the remaining battery level A2 of the mobile device 10, and the position information G1 regarding the current mobile device 10 on the map M1 can be displayed. As a result, the operator can intuitively understand in what state the photographer is currently photographing the subject. Further, since the operator can grasp the information of the subject desired by the photographer, the operation content of the moving device 10 can be changed in real time. Furthermore, since the operator can grasp the photographing situation of the photographer in real time, the operator can concentrate on the operation of the mobile device 10. Since the desired information varies depending on the user, the situation, and the object, it is preferable that the desired information can be changed. The information can be selected from a DB recorded in any of the recording units described above. The program may be made programmable by a program included in the program. In addition, the idea of summarizing and displaying the obtained information is also a preferred application, and this summarization function can be given to any of the devices, but it was summarized before communication. However, it is possible to reduce the information and increase the speed.

  Thereafter, the second control unit 28 acquires mobile device information from the mobile device 10 via the third communication unit 25 (step S107). Here, the mobile device information is at least current position information, travel direction, turning information, altitude, operation status of the mobile device 10 such as a stationary state, a return state, and a remaining amount of the power supply 12 regarding the mobile device 10.

  Subsequently, the second control unit 28 transmits the image data generated by the imaging unit 21 and the moving device information input from the moving device 10 to the operating device 40 via the fourth communication unit 26 (step S108). Accordingly, as shown in FIG. 5, the second display control unit 252 causes the second display unit 42 of the operating device 40 to display the image W1 corresponding to the image data, the travel information A1 regarding the travel direction of the mobile device 10, the mobile device. Ten remaining battery charges A2, position information G1 and altitude T1 regarding the current mobile device 10 on the map M1 can be displayed. As a result, the photographer can intuitively understand how the operator is currently operating the mobile device 10. Furthermore, since the photographer can take a picture in real time while predicting the position of the desired subject from the traveling direction of the moving device 10, the subject can take a picture with a desired composition. Furthermore, the photographer can grasp the position of the mobile device 10 in real time. Note that the moving device information includes only the traveling direction of the moving device 10, but of course the state of the moving device 10, for example, while the altitude is increasing (altitude decreasing), turning (turning direction), moving speed, and moving device. The second display control unit 452 may further cause the second display unit 42 of the controller device 40 to display the information including the ten movement histories (movement logs) and the like. Thereby, the photographer can grasp the state of the moving device 10 by the operator in more detail, and can concentrate on photographing. Further, when the photographer is watching and notices that it is too fast, or when it is preferable that the photographer is stationary, the request may be displayed as voice assistance when the photographer wants to notify the operator. For example, it may be possible to specify that the mobile device 10 can be set to a specific speed, set to a specific altitude, set to a specific position, or the like. In this case, if the mobile device 10 outputs such information and the photographer can see and set the information, it becomes possible to make a specific and accurate setting according to the situation.

  Thereafter, when the shooting operation signal is received from the controller device 40 via the fourth communication unit 26 (step S109: Yes), when the mobile device 10 is returning (step S110: Yes), the imaging device 20 The process proceeds to step S116 described later. On the other hand, when a photographing operation signal is received from the controller device 40 via the fourth communication unit 26 (step S109: Yes), when the mobile device 10 is not returning (step S110: No), the imaging device 20 Shifts to step S111 described below.

  In step S <b> 111, the second communication control unit 287 transmits a notification that there has been an imaging instruction to the mobile device 10 via the third communication unit 25.

  Thereafter, the imaging control unit 288 causes the imaging unit 21 to perform imaging (step S112), causes the image processing unit 281 to perform image processing on the image data generated by the imaging unit 21, and records the image data in the image data recording unit 271 (step S112). Step S113). In this case, the imaging control unit 288 records the image data and the moving device information in the image data recording unit 271 in association with each other. Thereby, it is possible to grasp the position of the image data at the time of shooting during reproduction. Of course, in addition to the image data and the mobile device information, the movement history (movement log) of the mobile device 10 may be associated and recorded in the image data recording unit 271.

  Subsequently, the second communication control unit 287 transmits the image data generated by the imaging unit 21 to the mobile device 10 via the third communication unit 25 (step S114).

  Thereafter, the second control unit 28 determines whether or not the mobile device 10 is stopped based on information input from the mobile device 10 via the third communication unit 25 (step S115). When it is determined by the second control unit 28 that the moving device 10 is stopped (step S115: Yes), the imaging device 20 ends this operation. On the other hand, when it is judged by the 2nd control part 28 that the moving apparatus 10 has not stopped (step S115: No), the imaging device 20 returns to step S101 mentioned above.

  In step S116, the imaging control unit 288 stops imaging by the imaging unit 21. After step S116, the imaging device 20 proceeds to step S115.

  In step S109, when the imaging operation signal is not received from the operating device 40 via the fourth communication unit 26 (step S109: No), the imaging device 20 proceeds to step S117.

  Subsequently, when a parameter change signal for changing the imaging parameter of the imaging unit 21 is received from the operation device 40 via the fourth communication unit 26 (step S117: Yes), the imaging control unit 288 responds to the parameter change signal. The imaging parameters of the imaging unit 21, such as the zoom magnification and the aperture value, are changed so as to become the imaging parameters (step S118). After step S118, the imaging device 20 proceeds to step S115.

  In step S117, when the parameter change signal for changing the shooting parameter of the imaging unit 21 is not received from the operation device 40 via the fourth communication unit 26 (step S117: No), the imaging device 20 proceeds to step S115. To do.

  In step S <b> 119, the shooting control unit 288 executes normal shooting processing that causes the imaging unit 21 to perform shooting in accordance with an operation performed by the first operation unit 24 and records it in the image data recording unit 271. After step S119, the imaging device 20 ends this operation.

[Operation of mobile device]
Next, the operation of the moving device 10 will be described. FIG. 6 is a flowchart showing the operation of the mobile device 10. As shown in FIG. 6, first, the first controller 19 determines the communication state of the mobile device 10 (step S201). Specifically, the first control unit 19 determines whether or not communication connection with the imaging device 20 is possible via the cable by the second communication unit 18 and via the first communication unit 17. It is determined whether or not the communication state with the control device 30 is good. In this case, if the first control unit 19 determines that the communication state with the imaging device 20 is not possible or the communication state with the control device 30 is not good, the first control unit 19 moves. An LED (not shown) provided on the outer surface of the device 10 notifies that the communication state is abnormal.

  Subsequently, the first control unit 19 acquires imaging device information from the imaging device 20 via the second communication unit 18 (step S202).

  Thereafter, when a control signal is received from the control device 30 via the first communication unit 17 (step S203: Yes), the mobile device 10 proceeds to step S204 described later. On the other hand, when the control signal is not received from the control device 30 via the first communication unit 17 (step S203: No), the mobile device 10 proceeds to step S207 described later.

  In step S204, when there is reception information in the imaging device information acquired from the imaging device 20 (step S204: Yes), the moving device 10 proceeds to step S207 described later. On the other hand, when there is no reception information in the imaging device information acquired from the imaging device 20 (step S204: No), the moving device 10 proceeds to step S205 described later.

  In step S205, when the first control unit 19 determines that there is an obstacle in the traveling direction of the mobile device 10 (step S205: Yes), the mobile device 10 proceeds to step S207 described later. On the other hand, when the first control unit 19 determines that there is no obstacle in the traveling direction of the mobile device 10 (step S205: No), the mobile device 10 proceeds to step S206 described later.

  In step S <b> 206, the first control unit 19 moves the moving device 10 based on the control signal input from the control device 30. After step S206, the mobile device 10 proceeds to step S212 to be described later.

  In step S207, the first control unit 19 determines whether or not the moving device 10 can be stopped (hovering). When it is determined by the first control unit 19 that the moving device 10 can be stopped (step S207: Yes), the propulsion control unit 194 performs a stationary control process for stopping the moving device 10 (step S208). Specifically, the first control unit 19 stops the moving device 10 at that position. After step S208, the mobile device 10 proceeds to step S212 to be described later.

  In step S207, when the first control unit 19 determines that the moving device 10 cannot be stationary (step S207: No), the moving device 10 proceeds to step S209 described later.

  Subsequently, the first control unit 19 determines whether or not to return the mobile device 10 to the return location (step S209). Specifically, the first control unit 19 can return the mobile device 10 to the return location based on the remaining amount of the power supply 12 and the current position information of the mobile device 10 detected by the position and orientation detection unit 14. Determine whether or not. Here, the return place is the takeoff start position (time information of the takeoff start position) of the mobile device 10. When the first control unit 19 determines that the mobile device 10 can return to the return location (step S209: Yes), the propulsion control unit 194 executes a feedback control process for returning the mobile device 10 to the return location. (Step S210). After step S210, the mobile device 10 proceeds to step S212 described later.

  In step S209, when the first control unit 19 determines that the mobile device 10 cannot return to the return place (step S209: No), the mobile device 10 proceeds to step S211 described later.

  Subsequently, the propulsion control unit 194 executes avoidance control processing for moving the moving device 10 to the avoidance location (step S211). Specifically, the propulsion control unit 194 calculates a movable distance from the remaining amount of the power supply 12, and the safe avoidance closest to the calculated distance and the current position of the mobile device 10 detected by the position and orientation detection unit 14. The moving device 10 is moved to a place (a flat place without an obstacle). After step S211, the mobile device 10 proceeds to step S212 described later.

  In step S <b> 212, the first communication control unit 197 transmits the mobile device information regarding the mobile device 10 to the control device 30 via the first communication unit 17.

  Subsequently, when communicating with the imaging device 20 via the second communication unit 18 (step S213: Yes), the first communication control unit 197 receives image data from the imaging device 20 via the second communication unit 18. Is acquired (step S214).

  Subsequently, the first communication control unit 197 transmits necessary information to the control device 30 via the first communication unit 17 (step S215).

  Thereafter, when an end signal is input via the control device 30 (step S216: Yes), the propulsion control unit 194 executes a stop control process for stopping the rotor 10a of the moving device 10 (step S217). After step S217, the mobile device 10 ends this process. On the other hand, when the end signal is not input via the control device 30 (step S216: No), the moving device 10 returns to the above-described step S201.

  In step S213, when communication with the imaging device 20 is not performed via the second communication unit 18 (step S213: No), the mobile device 10 proceeds to step S216.

  According to the embodiment of the present invention described above, even when each of the operator U1 of the mobile device 10 and the photographer U2 of the imaging device 20 operates, it is possible to concentrate on each other's operation.

  Further, according to the embodiment of the present invention, the second control unit 28 transmits the moving device information input from the moving device 10 to the operating device 40, and the second display unit 42 of the operating device 40 converts the image data into image data. Since the corresponding image and information related to the moving device 10 can be displayed, the photographer can intuitively grasp the operation content of the operator.

  Further, according to the embodiment of the present invention, the second control unit 28 transmits the imaging device information to the moving device 10, the moving device 10 transmits to the control device 30, and the first display unit 32 of the control device 30. Since the information regarding the state of the mobile device 10 and the imaging device information of the imaging device 20 are displayed, the operator can grasp the subject and the imaging situation desired by the photographer.

  Further, according to the embodiment of the present invention, the distance information related to the distance from the imaging device 20 to the subject is further included as the imaging device information, and the first display unit 32 of the control device 30 displays the distance from the imaging device 20 to the subject. Since the distance is displayed, the operator can intuitively grasp the distance to the subject, and can reflect this in the operation of the mobile device 10.

  Further, according to an embodiment of the present invention, when the imaging apparatus information further includes a reception signal indicating that the imaging instruction signal has been received, and the mobile apparatus 10 receives the reception signal via the imaging apparatus 20, Since the first control unit 19 stops the moving device 10, the photographer can shoot the subject with a desired composition and reduce blurring caused by the moving device 10.

  In addition, according to the embodiment of the present invention, when the moving device 10 is returning, when the instruction signal instructing the photographing is input from the operation device 40, the second control unit 28 controls the imaging unit 21. Since the return of the mobile device 10 is prioritized by stopping the shooting, power consumption of the mobile device 10 due to shooting can be prevented.

  In addition, according to the embodiment of the present invention, the second control unit 28 transmits the moving device information regarding the position and the traveling direction of the moving device 10 input from the moving device 10 to the operating device 40, thereby Since the current position and the traveling direction of the mobile device 10 are displayed on the second display unit 42 of 40, the photographer can intuitively grasp the operation content of the operator.

(Other embodiments)
In the embodiment of the present invention, the communication unit capable of communicating with the outside is provided in each of the mobile device and the imaging device, but the communication unit capable of communicating with the outside is provided in either the mobile device or the imaging device. The operation of the control device and the operation device may be performed via the communication unit. For example, as in the imaging system 1a illustrated in FIG. 7, a communication unit that can communicate with the outside is provided only in the mobile device 10, and data of the imaging device 20 is transmitted to the control device 30 by the communication unit provided in the mobile device 10. You may do it.

  In addition to the drone and the digital still camera, the imaging system according to the present invention is a medical video imaged by an electronic device such as a digital video camera, a surveillance camera, and a tablet portable device having an imaging function, and an endoscope or a microscope. The present invention can also be applied to a display device that displays an image corresponding to image data in the industrial field. Furthermore, the present invention can also be applied to an apparatus that performs photographing while moving the apparatus by two users. For example, from a telesurgical device or medical device with an operator who removes the affected area in the patient with an electric scalpel while viewing the image, and a photographer who images the inside of the patient for the operator, endoscope and observation scope The present invention can also be applied to an endoscope system.

  The program to be executed by the imaging apparatus according to the present invention is file data in an installable or executable format, and is a CD-ROM, flexible disk (FD), CD-R, DVD (Digital Versatile Disk), USB medium. And recorded on a computer-readable recording medium such as a flash memory.

  The program to be executed by the imaging apparatus according to the present invention may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Furthermore, a program to be executed by the imaging apparatus according to the present invention may be provided or distributed via a network such as the Internet.

  In the description of the flowchart in the present specification, the context of the processing between steps is clearly indicated using expressions such as “first”, “after”, “follow”, etc., in order to implement the present invention. The order of processing required is not uniquely determined by their representation. That is, the order of processing in the flowcharts described in this specification can be changed within a consistent range.

  Although some embodiments of the present application have been described in detail with reference to the drawings, these are merely examples, and various modifications and improvements can be made based on the knowledge of those skilled in the art including the aspects described in the section of the disclosure of the invention. It is possible to implement the present invention in other forms.

  Moreover, the above-mentioned “section (module, unit)” can be read as “means”, “circuit”, and the like. For example, the control unit can be read as control means or a control circuit.

  As described above, the present invention can include various embodiments not described herein, and various design changes can be made within the scope of the technical idea specified by the claims. It is.

  DESCRIPTION OF SYMBOLS 1,1a ... Imaging system; 2 ... Cable ;; 10 ... Moving apparatus; 10a ... Rotor; 11 ... Propulsion part; 12 ... Power supply; 14 ... Position and orientation detection unit; 15 ... Altitude posture detection unit; 16 ... First recording unit; 17 ... First communication unit; 18 ... Second communication unit; First control unit; 20 ... imaging device; 21 ... imaging unit; 22 ... elevation angle azimuth detection unit; 23 ... clock; 24 ... first operation unit; 26 ... 4th communication unit; 27 ... 2nd recording unit; 28 ... 2nd control unit; 30 ... control device; 31 ... 5th communication unit; 1 display section; 33 ... second operation section; 34 ... third recording section; 35 ... third control section; 40 ... operating device; 41 ... sixth communication section;・ Table 2 43; third operation unit; 44 ... fourth recording unit; 45 ... fourth control unit; 191 ... power supply determination unit; 192 ... movement determination unit; Determination unit: 194 ... Propulsion control unit; 195 ... Direction control unit; 196 ... Attitude control unit; 197 ... First communication control unit; 211 ... Optical system; 271, image data recording unit, 272, program recording unit, 281, image processing unit, 282, image feature extraction unit, 283, focus position acquisition unit, 284, angle of view. Acquisition unit; 285 ... Distance detection unit; 286 ... Trimming unit; 287 ... Second communication control unit; 288 ... Shooting control unit; 351 ... Third communication control unit; 1st display control part; 451 ... 4th communication control part; 452 ... 2nd display control part; 1 ... subject; U1 ··· operator; U2 ··· photographer

Claims (16)

An imaging apparatus mounted on a mobile device and capable of bidirectional communication with the mobile device,
An imaging unit that images a subject and generates image data;
A control unit that transmits information that can be displayed on a display unit of a control device that controls the mobile device to the mobile device;
An imaging apparatus comprising:   The imaging apparatus according to claim 1, wherein the information includes at least the image data generated by the imaging unit. A distance detector for detecting a distance from the imaging device to the subject based on the image data;
The imaging apparatus according to claim 2, wherein the information further includes distance information regarding the distance detected by the distance detection unit. The apparatus further includes a communication unit that receives an instruction signal for instructing the imaging device to perform imaging from the operating device that operates the imaging device, and transmits the image data to the operating device.
The control unit causes the imaging unit to perform imaging when the communication unit receives the instruction signal,
The imaging apparatus according to claim 1, wherein the information further includes reception information indicating that the communication unit has received the instruction signal.   When the control unit receives a control signal for controlling movement of the mobile device according to the operation of the control device from the mobile device, the control unit transmits control information related to the control signal to the operation device via the communication unit. The imaging apparatus according to claim 4, wherein:   The said control part transmits the said positional information to the said operating device via the said communication part, when the positional information regarding the position of the said mobile device is input from the said mobile device, The said control apparatus is characterized by the above-mentioned. The imaging device described.   The said control part stops imaging | photography by the said imaging part, when the said mobile device is returning to the takeoff start position, and receiving the said instruction signal, The any one of Claims 4-6 characterized by the above-mentioned. The imaging device described in one. An imaging device that captures an image of a subject and generates image data is mounted, is capable of bidirectional communication with the imaging device, and moves according to a signal transmitted from a control device having a display unit,
A mobile device comprising: a communication unit that receives the signal transmitted from the control device and transmits information input from the imaging device to the control device.   The mobile device according to claim 8, wherein the information includes at least the image data.   The moving device according to claim 8, wherein the information further includes distance information regarding a distance from the imaging device to the subject. A position detector that detects position information relating to the current position of the mobile device;
A control unit that transmits the position information detected by the position detection unit to the imaging device;
The moving apparatus according to claim 8, further comprising: The information further includes reception information indicating that the imaging device has received an instruction signal for instructing the imaging device to perform shooting,
The mobile device according to claim 11, wherein when the communication unit receives the signal, the control unit stops the mobile device when the information is received.   The mobile device according to claim 11 or 12, wherein when the information is received when the mobile device returns the mobile device to a takeoff start position, the control device maintains the feedback of the mobile device. . An imaging device that captures an image of a subject and generates image data, a moving device that is equipped with the imaging device and can communicate with the imaging device bidirectionally, a control device that controls the moving device, and an operation of the imaging device An imaging device comprising:
The imaging device
A control unit that transmits information that can be displayed on a display unit of a control device that controls the mobile device to the mobile device;
The mobile device is
A communication unit that receives the signal transmitted from the control device and transmits the information input from the imaging device to the control device;
An imaging system comprising: An imaging method executed by an imaging device mounted on a mobile device and capable of bidirectional communication with the mobile device,
An imaging step of imaging a subject to generate image data;
A control step of transmitting information that can be displayed on a display unit of a control device that controls the mobile device to the mobile device;
An imaging method comprising: A program executed by an imaging device that is mounted on a mobile device and capable of bidirectional communication with the mobile device,
An imaging step of imaging a subject to generate image data;
A control step of transmitting information that can be displayed on a display unit of a control device that controls the mobile device to the mobile device;
A program characterized by having executed.

Images