JP2018201119A - Mobile platform, flying object, support apparatus, portable terminal, method for assisting in photography, program, and recording medium - Google Patents

Abstract

To attractively photograph a desired image even where a user of a camera to photograph a photographic object is not familiar with photography.SOLUTION: A mobile platform for assisting in photography of a second image by means of a photographic apparatus includes: an acquisition section for acquiring a first image; an information acquisition section for acquiring information about a first photographic object of one or more photographic objects included in the first image, and acquiring information about a first composition of one or more compositions used for specifying positions of the one or more photographic objects, which include the first photographic object, in a second image; and a generation section for generating, according to the first composition, action information related to an action of the photographic apparatus used for photographing the second image.SELECTED DRAWING: Figure 3

Description

  The present disclosure relates to a mobile platform, a flying object, a support device, a portable terminal, an imaging assistance method, a program, and a recording medium that support imaging of images.

  Conventionally, virtual camera systems are known that calculate camera shots using proven imaging techniques to visualize computer-generated stories, play events, and dynamically update camera views. This virtual camera system analyzes a camera arrangement position for displaying a subject at a desired angle and distance and with minimum occlusion.

  This virtual camera system is mainly used for a 3D game camera system, and three-dimensional image data representing an arbitrary virtual three-dimensional space is prepared in advance. This virtual camera system can represent an area viewed from a specific viewpoint in a virtual three-dimensional space so as to have an arbitrary composition. This virtual camera system uses, for example, a limited composition, a composition using a three-division method that takes into account one subject, and a three-division method that takes into account the whole elements from virtual three-dimensional image data prepared in advance. A part of the three-dimensional image data can be displayed on the display according to the composition and the equalized composition (see Non-Patent Document 1).

William Bares, “A Photographic Composition Assistant for Intelligent Virtual 3D Camera Systems”, Millsaps College, Department of Computer Science, Jackson MS 39210, USA, Internet <URL: http://link.springer.com/chapter/10.1007/11795018_16>

  When the virtual camera system described in Non-Patent Document 1 is applied to a camera system that handles images captured in real space, the camera system has an arbitrary composition based on pre-captured three-dimensional image data. Will be processed. For this reason, the camera system cannot determine the composition of an image that has not yet been captured. Therefore, for example, when a user of a camera that captures a subject is unaccustomed to imaging, it is difficult to attractively capture a desired image.

  In one aspect, the mobile platform is a mobile platform that assists imaging of the second image by the imaging device, and includes an image acquisition unit that acquires the first image, and one or more subjects included in the first image. Information of the first subject is acquired, and information of the first composition is acquired from one or more compositions that define the positions of one or more subjects including the first subject in the second image. And an information acquisition unit that generates operation information related to the operation of the imaging apparatus for capturing the second image according to the first composition.

  The information acquisition unit may select and acquire a first subject from a plurality of subjects included in the first image.

  The information acquisition unit may acquire information on the first subject based on the color component of the subject included in the first image.

  The information acquisition unit may acquire information on the first subject based on the spatial frequency of the subject included in the first image.

  The information acquisition unit may acquire position information of the imaging device and acquire information of the first subject based on the position information of the imaging device.

  The information acquisition unit may acquire information on the first subject based on an imaging mode when the second image is captured by the imaging device.

  The information acquisition unit may select and acquire the first composition from a plurality of compositions.

  The mobile platform may further include a recognition unit that recognizes the shape of the first subject. The information acquisition unit may acquire information of the first composition based on the shape of the first subject.

  The mobile platform may further include a recognition unit that recognizes a scene when the second image is captured. The information acquisition unit may acquire information of the first composition based on the scene.

  The generation unit may generate rotation information regarding rotation of a support member that rotatably supports the imaging device as operation information.

  The generation unit may determine the rotation amount and the rotation direction of the support member based on the position of the first subject in the first image and the position of the first subject in the first composition.

  The generation unit may generate movement information regarding movement of the imaging device as the operation information.

  The generation unit may determine the amount of movement of the imaging device along the direction of gravity based on the size of the first subject in the first image and the size of the first subject in the first composition.

  The generation unit is based on a correspondence relationship between the position of the first subject in the first image, the position of the first subject in the first composition, and the movement distance in the first image and the movement distance in the real space. The moving amount and moving direction of the imaging device may be determined.

  You may further provide the presentation part which shows operation | movement information.

  The first image may be an image captured by the imaging device.

  The mobile platform is a flying object including an imaging device and a support member that rotatably supports the imaging device, and may further include a control unit that controls flight of the flying object or rotation of the support member based on operation information. .

  The mobile platform is a support device that includes a support member that is gripped by a user during use and supports the imaging device in a rotatable manner, and may further include a control unit that controls rotation of the support member based on operation information.

  The mobile platform is a mobile terminal, and may further include a communication unit that transmits operation information to the flying object or the support device.

  In one aspect, the flying body includes an imaging device, a support member that rotatably supports the imaging device, a motion information acquisition unit that acquires motion information generated by the mobile platform, and a flying object based on the motion information. A control unit that controls the flight or rotation of the support member.

  In one aspect, the support device controls the rotation of the support member based on the support member that rotatably supports the imaging device, the operation information acquisition unit that acquires the operation information generated by the mobile platform, and the operation information. A control unit.

In one aspect, the imaging assistance method is an imaging assistance method in a mobile platform that assists imaging of the second image by the imaging device, and the step of acquiring the first image;
Of the one or more subjects included in the first image, the step of obtaining information on the first subject, and one that defines the position of the one or more subjects including the first subject in the second image Of the above compositions, the method has a step of acquiring information of the first composition and a step of generating operation information related to the operation of the imaging apparatus for capturing the second image in accordance with the first composition.

  The step of acquiring information about the first subject may include a step of selecting and acquiring the first subject from a plurality of subjects included in the first image.

  The step of acquiring information about the first subject may include a step of acquiring information about the first subject based on the color component of the subject included in the first image.

  The step of acquiring information about the first subject may include the step of acquiring information about the first subject based on the spatial frequency of the subject included in the first image.

  The imaging assistance method may further include a step of acquiring position information of the imaging device. The step of acquiring information on the first subject may include a step of acquiring information on the first subject based on the position information of the imaging device.

  The step of acquiring information about the first subject may include a step of acquiring information about the first subject based on an imaging mode when the second image is captured by the imaging device.

  The step of acquiring information on the first composition may include a step of selecting and acquiring the first composition from a plurality of compositions.

  The imaging assistance method may further include a step of recognizing the shape of the first subject. The step of acquiring information on the first composition may include the step of acquiring information on the first composition based on the shape of the first subject.

  The imaging assistance method may further include a step of recognizing a scene when the second image is captured. The step of acquiring information on the first composition may include the step of acquiring information on the first composition based on the scene.

  The step of generating the motion information may include a step of generating, as the motion information, rotation information related to rotation of a support member that rotatably supports the imaging device.

  The step of generating the motion information includes a step of determining a rotation amount and a rotation direction of the support member based on the position of the first subject in the first image and the position of the first subject in the first composition. It's okay.

  The step of generating the operation information may include a step of generating movement information regarding movement of the imaging device as the operation information.

  The step of generating the motion information is a step of determining an amount of movement of the imaging device along the direction of gravity based on the size of the first subject in the first image and the size of the first subject in the first composition. May be included.

  The step of generating the motion information includes the correspondence between the position of the first subject in the first image, the position of the first subject in the first composition, the movement distance in the first image, and the movement distance in real space. A step of determining a moving amount and a moving direction of the imaging device based on the relationship may be included.

  The imaging assistance method may further include a step of presenting operation information to the presentation unit.

  The first image may be an image captured by the imaging device.

  The mobile platform may be an aircraft including an imaging device and a support member that rotatably supports the imaging device. The imaging assistance method may further include the step of controlling the flight of the flying object or the rotation of the support member based on the operation information.

  The mobile platform may be a support device that includes a support member that is gripped by a user during use and supports the imaging device rotatably. The imaging assistance method may further include a step of controlling the rotation of the support member based on the operation information.

  The mobile platform may be a mobile terminal. The imaging assistance method may further include a step of transmitting the operation information to the flying object or the support device.

  In one aspect, the program acquires a first image to a mobile platform that assists in capturing the second image by the imaging device, and the first of the one or more subjects included in the first image. Obtaining information on the subject, and obtaining information on the first composition out of one or more compositions defining the position of the one or more subjects including the first subject in the second image; And a step of generating operation information relating to the operation of the imaging apparatus for capturing the second image in accordance with the first composition.

  In one aspect, the recording medium has a step of acquiring the first image in a mobile platform that assists the imaging of the second image by the imaging device, and the first of the one or more subjects included in the first image. A step of acquiring information on one subject, and a step of acquiring information on the first composition among one or more compositions defining the positions of one or more subjects including the first subject in the second image. And a step of generating operation information relating to the operation of the imaging apparatus for capturing the second image according to the first composition, and a computer-readable recording medium storing a program for executing the program.

  Note that the summary of the invention described above does not enumerate all the features of the present disclosure. In addition, a sub-combination of these feature groups can also be an invention.

FIG. 2 is a schematic diagram illustrating a configuration example of an imaging assistance system in the first embodiment. The block diagram which shows an example of the hardware constitutions of the unmanned aircraft in 1st Embodiment The block diagram which shows an example of a function structure of the UAV control part in 1st Embodiment The block diagram which shows an example of the hardware constitutions of the portable terminal in 1st Embodiment Diagram for explaining the operation outline of the imaging assist system Diagram showing an example of a live view image The figure which shows an example of the color division image in which the live view image was divided by the color The figure which shows the example of selection of the main subject A diagram showing an example of composition selection The figure which shows the example of rotation of the imaging range for aerial photography with the determined composition Diagram showing an example of moving an unmanned aerial vehicle for aerial photography with a determined composition Illustration for explaining the movement of an unmanned aerial vehicle viewed from the horizontal direction The sequence diagram which shows the operation example of the imaging assistance system in 1st Embodiment Schematic diagram showing a configuration example of an imaging assistance system in the second embodiment The block diagram which shows an example of the hardware constitutions of the unmanned aircraft in 2nd Embodiment The block diagram which shows an example of a function structure of the UAV control part in 2nd Embodiment The block diagram which shows an example of the hardware constitutions of the portable terminal in 2nd Embodiment The block diagram which shows an example of a function structure of the terminal control part in 2nd Embodiment. The sequence diagram which shows the operation example of the imaging assistance system in 2nd Embodiment The perspective view which shows the structural example of the imaging assistance system containing the gimbal apparatus and portable terminal in 3rd Embodiment. Front perspective view showing a configuration example of a gimbal device in a fourth embodiment The rear perspective view which shows the structural example of the imaging assistance system containing the gimbal apparatus and portable terminal in 4th Embodiment.

  Hereinafter, although this indication is explained through an embodiment of the invention, the following embodiment does not limit the invention concerning a claim. Not all combinations of features described in the embodiments are essential for the solution of the invention.

  The claims, the description, the drawings, and the abstract include matters subject to copyright protection. The copyright owner will not object to any number of copies of these documents as they appear in the JPO file or record. However, in other cases, all copyrights are reserved.

  In the following embodiment, an unmanned aerial vehicle (UAV: Unmanned Aerial Vehicle) is illustrated as a flying object. The flying object includes an aircraft moving in the air. In the drawings attached to this specification, the unmanned aerial vehicle is represented as “UAV”. Examples of the mobile platform include a flying object, a portable terminal, a gimbal device, and a gimbal camera device. The mobile platform may be a device other than these, for example, a transmitter, a PC (Personal Computer), or another mobile platform. The imaging assisting method is a method in which operation on a mobile platform is defined. The recording medium stores a program (for example, a program that causes a mobile platform to execute various processes).

(First embodiment)
FIG. 1 is a schematic diagram illustrating a configuration example of an imaging assistance system 10 according to the first embodiment. The imaging assistance system 10 includes an unmanned aircraft 100, a transmitter 50, and a portable terminal 80. The unmanned aircraft 100, the transmitter 50, and the portable terminal 80 can communicate with each other by wired communication or wireless communication (for example, a wireless local area network (LAN)).

  The unmanned aerial vehicle 100 may fly according to a remote operation by the transmitter 50, or may fly according to a preset flight path. The unmanned aerial vehicle 100 determines a composition for aerial photography and generates operation information of the unmanned aerial vehicle 100 so that the determined composition is obtained. Unmanned aerial vehicle 100 controls the operation of unmanned aerial vehicle 100 according to the operation information. The transmitter 50 may instruct control of the flight of the unmanned aircraft 100 by remote control. That is, the transmitter 50 may operate as a remote controller. The portable terminal 80 can be carried by a user who plans to take an aerial photograph using the unmanned aircraft 100 together with the transmitter 50. The portable terminal 80 assists the determination of the composition by the unmanned aircraft 100 and assists in imaging.

  FIG. 2 is a block diagram illustrating an example of a hardware configuration of the unmanned aerial vehicle 100. The unmanned aerial vehicle 100 includes a UAV control unit 110, a communication interface 150, a memory 160, a storage 170, a gimbal 200, a rotary wing mechanism 210, an imaging unit 220, an imaging unit 230, a GPS receiver 240, The configuration includes an inertial measurement unit (IMU) 250, a magnetic compass 260, a barometric altimeter 270, an ultrasonic sensor 280, and a laser measuring device 290. The gimbal 200 is an example of a support member. The imaging unit 220 and the imaging unit 230 are an example of an imaging device.

  The UAV control unit 110 is configured using, for example, a central processing unit (CPU), a micro processing unit (MPU), or a digital signal processor (DSP). The UAV control unit 110 performs signal processing for overall control of operations of each unit of the unmanned aircraft 100, data input / output processing with respect to other units, data calculation processing, and data storage processing.

  The UAV control unit 110 controls the flight of the unmanned aircraft 100 according to a program stored in the memory 160. For example, the UAV control unit 110 may control the flight of the unmanned aircraft 100 so as to realize the composition determined in cooperation with the mobile terminal 80. UAV control unit 110 controls the flight of unmanned aerial vehicle 100 in accordance with instructions received from remote transmitter 50 via communication interface 150. Memory 160 may be removable from unmanned aerial vehicle 100.

  The UAV control unit 110 acquires position information indicating the position of the unmanned aircraft 100. The UAV control unit 110 may acquire position information indicating the latitude, longitude, and altitude at which the unmanned aircraft 100 exists from the GPS receiver 240. The UAV control unit 110 acquires, from the GPS receiver 240, latitude / longitude information indicating the latitude and longitude where the unmanned aircraft 100 exists, and altitude information indicating the altitude where the unmanned aircraft 100 exists from the barometric altimeter 270, as position information. Good.

  The UAV control unit 110 acquires orientation information indicating the orientation of the unmanned aircraft 100 from the magnetic compass 260. In the direction information, for example, a direction corresponding to the nose direction of the unmanned aircraft 100 is indicated.

  The UAV control unit 110 acquires imaging range information indicating the imaging ranges of the imaging unit 220 and the imaging unit 230. The UAV control unit 110 acquires angle-of-view information indicating the angle of view of the imaging unit 220 and the imaging unit 230 from the imaging unit 220 and the imaging unit 230 as parameters for specifying the imaging range. The UAV control unit 110 acquires information indicating the imaging directions of the imaging unit 220 and the imaging unit 230 as a parameter for specifying the imaging range. The UAV control unit 110 acquires posture information indicating the posture state of the imaging unit 220 from the gimbal 200 as information indicating the imaging direction of the imaging unit 220, for example. The UAV control unit 110 acquires information indicating the direction of the unmanned aircraft 100. The information indicating the posture state of the imaging unit 220 indicates a rotation angle from the reference rotation angle of the pitch axis and yaw axis of the gimbal 200. The UAV control unit 110 acquires position information indicating a position where the unmanned aircraft 100 exists as a parameter for specifying the imaging range. The UAV control unit 110 defines an imaging range indicating a geographical range captured by the imaging unit 220 based on the angle of view and the imaging direction of the imaging unit 220 and the imaging unit 230, and the position where the unmanned aircraft 100 exists. Imaging range information may be acquired by generating imaging range information.

  The UAV control unit 110 controls the gimbal 200, the rotary blade mechanism 210, the imaging unit 220, and the imaging unit 230. The UAV control unit 110 controls the imaging range of the imaging unit 220 by changing the imaging direction or angle of view of the imaging unit 220. The UAV control unit 110 controls the imaging range of the imaging unit 220 supported by the gimbal 200 by controlling the rotation mechanism of the gimbal 200.

  The imaging range refers to a geographical range captured by the imaging unit 220 or the imaging unit 230. The imaging range is defined by latitude, longitude, and altitude. The imaging range may be a range in three-dimensional spatial data defined by latitude, longitude, and altitude. The imaging range is specified based on the angle of view and imaging direction of the imaging unit 220 or the imaging unit 230 and the position where the unmanned aircraft 100 is present. The imaging direction of the imaging unit 220 and the imaging unit 230 is defined from the azimuth and the depression angle in which the front surface where the imaging lenses of the imaging unit 220 and the imaging unit 230 are provided is directed. The imaging direction of the imaging unit 220 is a direction specified from the heading direction of the unmanned aerial vehicle 100 and the attitude state of the imaging unit 220 with respect to the gimbal 200. The imaging direction of the imaging unit 230 is a direction specified from the nose direction of the unmanned aircraft 100 and the position where the imaging unit 230 is provided.

  The UAV control unit 110 may add information about the aerial image as additional information (an example of metadata) to the captured image (aerial image) captured by the imaging unit 220 or the imaging unit 230. The additional information includes information (flight information) related to the flight of the unmanned aircraft 100 at the time of aerial photography and information (imaging information) related to imaging by the imaging unit 220 or the imaging unit 230 at the time of aerial photography. The flight information may include at least one of aerial shooting position information, aerial shooting route information, and aerial shooting time information. The imaging information may include at least one of aerial view angle information, aerial shooting direction information, aerial shooting posture information, imaging range information, and subject distance information.

  The aerial shooting position information indicates a position (aerial shooting position) where the aerial image is taken aerial. The aerial shooting position information may be based on the position information acquired by the GPS receiver 240. The aerial shooting route information indicates a route (aerial shooting route) where the aerial image is taken aerial. The aerial shooting route information may be configured by a set of aerial shooting positions in which the aerial shooting positions are continuously connected. The aerial shooting time information indicates the time (aerial shooting time) when the aerial image was taken aerial. The aerial shooting time information may be based on timer time information referred to by the UAV control unit 110.

  The aerial view angle information indicates the view angle information of the imaging unit 220 or the imaging unit 230 when the aerial image is taken aerial. The aerial shooting direction information indicates the imaging direction (aerial shooting direction) of the imaging unit 220 or the imaging unit 230 when the aerial image is aerial. The aerial shooting posture information indicates posture information of the imaging unit 220 or the imaging unit 230 when the aerial image is taken aerial. The imaging range information indicates the imaging range of the imaging unit 220 or the imaging unit 230 when the aerial image is aerial. The subject distance information indicates information on the distance from the imaging unit 220 or the imaging unit 230 to the subject. The subject distance information may be based on information detected by the ultrasonic sensor 280 or the laser measuring device 290. As the subject distance information, a plurality of images including the same subject may be captured, and these images may be used as a stereo image to calculate the distance to the subject. Further, the imaging information may include information on the orientation of the unmanned aircraft 100 during aerial photography.

  The communication interface 150 communicates with the transmitter 50 and the portable terminal 80. The communication interface 150 may transmit the aerial image to the portable terminal 80. The communication interface 150 may transmit at least a part of the additional information to the portable terminal 80 together with the aerial image.

  The communication interface 150 may receive information for determining the composition of the aerial image scheduled to be captured by the imaging unit 220 or the imaging unit 230. The information for determining the composition of the aerial image to be captured may include, for example, selection information for selecting the main subject in the aerial image (live view image) and selection information for selecting the composition. . The communication interface 150 may transmit an aerial image (for example, an aerial moving image such as a live view image or an aerial still image) captured by the imaging unit 220 or the imaging unit 230 to the mobile terminal 80. The communication interface 150 may communicate directly with the portable terminal 80 or may communicate with the portable terminal 80 via the transmitter 50.

  The memory 160 includes a gimbal 200, a rotating blade mechanism 210, an imaging unit 220, an imaging unit 230, a GPS receiver 240, an inertial measurement device 250, a magnetic compass 260, a barometric altimeter 270, an ultrasonic sensor 280, and a laser. A program and the like necessary for controlling the measuring device 290 are stored. The memory 160 may be a computer-readable recording medium, such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), and It may include at least one flash memory such as a USB (Universal Serial Bus) memory.

  The memory 160 holds various information and various data acquired via the communication interface 150. The memory 160 may store sample information of various compositions for capturing an image. Composition sample information may be stored in a table format. The memory 160 may hold information on the composition determined by the UAV control unit 110. The memory 160 may hold operation information relating to the operation of the unmanned aerial vehicle 100 for realizing imaging with the determined composition. The operation information of the unmanned aircraft 100 may be read from the memory 160 at the time of aerial photography, and the unmanned aircraft 100 may operate according to this operation information.

  The gimbal 200 may support the imaging unit 220 rotatably about the yaw axis, pitch axis, and roll axis. The gimbal 200 may change the imaging direction of the imaging unit 220 by rotating the imaging unit 220 around at least one of the yaw axis, the pitch axis, and the roll axis.

  The yaw axis, pitch axis, and roll axis may be determined as follows. For example, assume that the roll axis is defined in the horizontal direction (direction parallel to the ground). In this case, a pitch axis is defined in a direction parallel to the ground and perpendicular to the roll axis, and a yaw axis (see z-axis) is defined in a direction perpendicular to the ground and perpendicular to the roll axis and the pitch axis.

  The imaging unit 220 captures a subject in a desired imaging range and generates captured image data. Image data obtained by imaging by the imaging unit 220 is stored in the memory included in the imaging unit 220 or the memory 160.

  The imaging unit 230 captures the surroundings of the unmanned aircraft 100 and generates captured image data. Image data of the imaging unit 230 is stored in the memory 160.

  The GPS receiver 240 receives a plurality of signals indicating times and positions (coordinates) of each GPS satellite transmitted from a plurality of navigation satellites (that is, GPS satellites). The GPS receiver 240 calculates the position of the GPS receiver 240 (that is, the position of the unmanned aircraft 100) based on the plurality of received signals. The GPS receiver 240 outputs the position information of the unmanned aircraft 100 to the UAV control unit 110. The calculation of the position information of the GPS receiver 240 may be performed by the UAV control unit 110 instead of the GPS receiver 240. In this case, the UAV control unit 110 receives information indicating the time and the position of each GPS satellite included in a plurality of signals received by the GPS receiver 240.

  Inertial measurement device 250 detects the attitude of unmanned aerial vehicle 100 and outputs the detection result to UAV control unit 110. The inertial measurement device IMU 250 detects the acceleration of the unmanned aircraft 100 in the three axial directions of the front, rear, left and right, and the angular velocity in the three axial directions of the pitch axis, the roll axis, and the yaw axis. .

  The magnetic compass 260 detects the heading of the unmanned aircraft 100 and outputs the detection result to the UAV control unit 110.

  Barometric altimeter 270 detects the altitude at which unmanned aircraft 100 flies, and outputs the detection result to UAV control unit 110. The altitude at which the unmanned aircraft 100 flies may be detected by a sensor other than the barometric altimeter 270.

  The ultrasonic sensor 280 emits an ultrasonic wave, detects an ultrasonic wave reflected by the ground or an object, and outputs a detection result to the UAV control unit 110. The detection result may indicate a distance from the unmanned aircraft 100 to the ground, that is, an altitude. The detection result may indicate a distance from the unmanned aircraft 100 to an object (subject).

  Laser measuring device 290 irradiates the object with laser light, receives the reflected light reflected by the object, and measures the distance between unmanned aircraft 100 and the object (subject) using the reflected light. As an example, the distance measurement method using laser light may be a time-of-flight method.

  FIG. 3 is a block diagram illustrating an example of a functional configuration of the UAV control unit 110. The UAV control unit 110 includes an image acquisition unit 111, a main subject determination unit 112, a composition determination unit 113, a motion information generation unit 114, and a motion control unit 115. The main subject determination unit 112 and the composition determination unit 113 are examples of an information acquisition unit. The composition determination unit 113 is an example of a recognition unit. The motion information generation unit 114 is an example of a generation unit. The operation control unit 115 is an example of a control unit.

  The image acquisition unit 111 may acquire an image stored in the memory 160 (for example, an aerial image captured by the imaging unit 220 or the imaging unit 230). The image acquisition unit 111 may acquire an aerial image being aerial captured by the imaging unit 220 or the imaging unit 230. The aerial image may be a moving image or a still image. The aerial video during aerial photography is also referred to as a live view image (an example of a first image). As an aerial image acquired by the image acquisition unit 111, a live view image is mainly exemplified.

  The main subject determination unit 112 determines (specifies) a main subject (an example of a first subject) among one or more subjects included in the live view image acquired by the image acquisition unit 111. The determination of the main subject is an example of acquisition of information on the main subject. The determination of the main subject may be performed manually by the user of the mobile terminal 80, for example, or may be performed automatically by the unmanned aircraft 100. The main subject may be a subject outside the live view image (for example, an arbitrary subject included in the map information corresponding to the aerial shooting range desired by the user). In this case, the composition suitable for the desired subject can be determined beyond the imaging range of the unmanned aircraft 100.

  The composition determination unit 113 determines a composition for imaging the determined main subject. The determination of the composition for imaging the main subject is an example of acquisition of composition information for imaging the main subject. Since this composition is a composition to be imaged yet to be imaged, it is also referred to as a scheduled composition (an example of the first composition). The composition may be information defining the positional relationship of one or more subjects in the image. The composition determination unit 113 may determine the planned composition according to the main subject information with reference to the sample information of the composition held in the memory 160. The determination of the scheduled composition may be performed manually by the user of the portable terminal 80 or automatically by the unmanned aircraft 100, for example.

  Sample composition information includes, for example, at least one of a Rule of Thirds composition, a Two-part composition, a Triangular composition, a Diagonal composition, an Alphabet composition, a Central one-point composition, a Frame composition, a Sandwich composition, and a Tunnel composition. May be included as sample information. The sample information of the composition may include information in which the main subject is arranged at a predetermined intersection or division point (for example, golden division point) of each composition.

  The motion information generation unit 114 generates motion information of the unmanned aerial vehicle 100 for realizing aerial photography according to the determined planned composition. The operation information of the unmanned aircraft 100 includes, for example, movement information related to movement of the unmanned aircraft 100 (for example, movement amount and movement direction of the unmanned aircraft 100), rotation information related to rotation of the unmanned aircraft 100 (for example, rotation amount of the unmanned aircraft 100, (Rotation direction), rotation information related to rotation of the gimbal 200 (for example, rotation amount of the gimbal 200, rotation direction), and other operation information of the unmanned aerial vehicle 100 may be included.

  The operation control unit 115 may control the flight of the unmanned aircraft 100 according to the generated operation information (for example, the movement amount and movement direction of the unmanned aircraft 100). The motion control unit 115 may control the direction of the unmanned aircraft 100 according to the generated motion information (for example, the rotation amount and the rotation direction of the unmanned aircraft 100). As described above, the operation control unit 115 can change the imaging range of the imaging unit 220 by moving the unmanned aircraft 100.

  The operation control unit 115 may control the rotation of the gimbal 200 according to the generated operation information (for example, the rotation amount and rotation direction of the gimbal 200). The operation control unit 115 may control the attitude of the unmanned aircraft 100 according to the operation information, and may control the attitude of the gimbal 200 by rotating the gimbal 200. As described above, the operation control unit 115 can change the imaging range of the imaging unit 220 by rotating the gimbal 200.

  FIG. 4 is a block diagram illustrating an example of a hardware configuration of the mobile terminal 80. The portable terminal 80 may include a terminal control unit 81, an interface unit 82, an operation unit 83, a wireless communication unit 85, a memory 87, and a display unit 88. The operation unit 83 is an example of an information acquisition unit. The wireless communication unit 85 is an example of a communication unit.

  The terminal control unit 81 is configured using, for example, a CPU, MPU, or DSP. The terminal control unit 81 performs signal processing for overall control of operations of each unit of the mobile terminal 80, data input / output processing with other units, data calculation processing, and data storage processing.

  The terminal control unit 81 may acquire data and information from the unmanned aircraft 100 via the wireless communication unit 85. The terminal control unit 81 may acquire data and information from the transmitter 50 via the interface unit 82. The terminal control unit 81 may acquire data and information input via the operation unit 83. The terminal control unit 81 may acquire data and information held in the memory 87. The terminal control unit 81 may send data and information to the display unit 88 and cause the display unit 88 to display display information based on the data and information.

  The terminal control unit 81 may execute an imaging assistance application. The imaging assistance application may be an application that assists the aerial photography with the desired composition by the unmanned aircraft 100. The terminal control unit 81 may generate various data used in the application.

  The interface unit 82 inputs and outputs information and data between the transmitter 50 and the portable terminal 80. The interface unit 82 may input / output via a USB cable, for example. The interface unit 65 may be an interface other than USB.

  The operation unit 83 receives and acquires data and information input by the user of the mobile terminal 80. The operation unit 83 may include buttons, keys, a touch panel, a microphone, and the like. Here, it is exemplified that the operation unit 83 and the display unit 88 are mainly configured by a touch panel. In this case, the operation unit 83 can accept a touch operation, a tap operation, a drag operation, and the like. The operation unit 83 may acquire selection information of the main subject by receiving a selection operation for selecting the main subject. The operation unit 83 may acquire composition selection information by receiving a selection operation for selecting a composition.

  The wireless communication unit 85 performs wireless communication with the unmanned aircraft 100 by various wireless communication methods. This wireless communication method of wireless communication may include, for example, communication via a wireless LAN, Bluetooth (registered trademark), or a public wireless line. The wireless communication unit 55 may transmit main subject selection information and composition selection information to the unmanned aerial vehicle 100.

  The memory 87 includes, for example, a ROM that stores a program that defines the operation of the mobile terminal 80 and set value data, and a RAM that temporarily stores various information and data used during processing by the terminal control unit 81. You can do it. The memory 87 may include memories other than ROM and RAM. The memory 87 may be provided inside the mobile terminal 80. The memory 87 may be provided so as to be removable from the portable terminal 80. The program may include an application program.

  The display unit 88 is configured using, for example, an LCD (Liquid Crystal Display), and displays various information and data output from the terminal control unit 81. The display unit 88 may display various data and information related to the execution of the imaging assistance application. The display unit 88 may display a selection screen for selecting a main subject and a selection screen for selecting a composition.

  Note that the portable terminal 80 may be attached to the transmitter 50 via a holder. The portable terminal 80 and the transmitter 50 may be connected via a wired cable (for example, a USB cable). The portable terminal 80 may not be attached to the transmitter 50, and the portable terminal 80 and the transmitter 50 may be provided independently. The imaging assistance system 10 may not include the transmitter 50.

Next, an outline of the operation of the imaging assistance system 10 will be described.
FIG. 5 is a diagram for explaining an operation outline of the imaging assistance system 10.

  In FIG. 5, there is a road R1 in the mountain M1, and a person H1 is located on the road R1. The unmanned aerial vehicle 100 is flying over the mountain M1 while taking an aerial photograph. Unmanned aerial vehicle 100 captures a live view image of mountain M <b> 1 and transmits it to portable terminal 80. The portable terminal 80 receives the live view image G1 from the unmanned aircraft 100 and displays the live view image G1 on the display unit 88. Thereby, the user of the portable terminal 80 can confirm the live view image G1.

  It is assumed that the user desires composition adjustment in order to capture the subject in which the live view image G1 is reflected more attractively. In this case, the mobile terminal 80 receives an operation for instructing composition adjustment via the operation unit 83 and transmits a composition adjustment command to the unmanned aircraft 100. When the unmanned aircraft 100 receives the composition adjustment command, the unmanned aircraft 100 determines a main subject (for example, a person H1) in the live view image G1, determines a planned composition, and generates operation information of the unmanned aircraft 100.

  The unmanned aerial vehicle 100 moves according to the operation information and notifies the mobile terminal 80 that it has moved to a desired position (movement completed). When the portable terminal 80 receives the movement completion notification, the portable terminal 80 transmits an imaging command to the unmanned aircraft 100 based on a user instruction via the operation unit 83, for example. Note that the portable terminal 80 may acquire a live view image at the position of the unmanned aircraft 100 after movement via the wireless communication unit 85 when receiving the movement completion notification. In this case, the user can confirm the live view image after movement by displaying, and can determine whether or not to take an aerial image at this position. When the unmanned aircraft 100 receives the imaging command, the imaging unit 220 or 230 performs aerial shooting according to the imaging command to obtain an aerial image (an example of a second image).

  Thereby, the unmanned aerial vehicle 100 can acquire an aerial image of a desired planned composition. In FIG. 5, the person H1 as the main subject is arranged at an arbitrary position in the live view image. However, in the aerial photographed image obtained by adjusting the composition, the person H1 as the main subject is 3 in the three-part composition. Located on the intersection of the dividing lines. Thus, the unmanned aerial vehicle 100 operates (moves here) to achieve a desired composition, and can take an aerial photograph.

  Note that the imaging command may be transmitted by the transmitter 50 instead of the portable terminal 80. In this case, the transmitter 50 may transmit information about pressing of an imaging button (not shown) included in the transmitter 50 to the unmanned aircraft 100 in cooperation with the portable terminal 80 using communication or the like.

  Next, an example of determining the main subject will be described.

  FIG. 6A is a diagram illustrating an example of a live view image G <b> 1 captured by the unmanned aircraft 100. FIG. 6B is a diagram illustrating an example of a color segmented image G2 in which the live view image G1 is segmented by color. FIG. 6C is a diagram illustrating an example of selecting a main subject using the color segmented image G2.

  The live view image G1 includes an ocean having a plurality of color components (for example, blue and light blue) and an island in which a forest having a green component exists. In unmanned aerial vehicle 100, communication interface 150 transmits live view image G1 to portable terminal 80. In the portable terminal 80, the wireless communication unit 85 may receive the live view image G1 from the unmanned aircraft 100, and the display unit 88 may display the live view image G1.

  The main subject determining unit 112 may divide the live view image G1 into a plurality of image blocks (for example, 16 × 16 blocks). The main subject determination unit 112 may divide the live view image G1 into one or more regions based on the color components of each image block, and generate a color-segmented image G2. In FIG. 6B, the blue portion of the sea may be partitioned as region A, the light blue portion of the ocean may be partitioned as region B, and the green portion of the island may be partitioned as region C. In the portable terminal 80, the wireless communication unit 85 may receive the color segment image G2 from the unmanned aircraft 100, and the display unit 88 may display the color segment image G2.

  The main subject determining unit 112 may determine any color component region in the color-segmented image G2 as the main subject. As shown in FIG. 6C, the display unit 88 displays the color segmented image G2 according to the imaging assistance application, and which region (here, ZA indicating the region A, ZB indicating the region B, ZC indicating the region C) is displayed. A guide display for determining whether to select the main subject may be performed. FIG. 6C illustrates that the area C is selected as the main subject via the operation unit 83. In this case, in the portable terminal 80, the wireless communication unit 85 transmits main subject selection information obtained via the operation unit 83 to the unmanned aircraft 100. In the unmanned aircraft 100, the communication interface 150 receives main subject selection information. The main subject determination unit 112 determines the main subject based on the main subject selection information. In this case, the main subject is a set of pixels corresponding to the imaging target intended by the user in the captured image.

  When the main subject determination unit 112 lacks information on part of the subject at the end of the live view image G1, in other words, when the subject is divided between the inside and outside of the live view image G1, The pixel information of the subject may be interpolated. For example, the main subject determining unit 112 may interpolate the pixel information of the subject based on pixel information (for example, pixel value) around the subject in the live view image G1. The main subject determination unit 112 may interpolate pixel information (for example, pixel values) around the subject in the live view image G1 as it is as pixel information of the subject. The main subject determination unit 112 may collect a plurality of pieces of pixel information around the subject, generate a new color by weighting or averaging based on the plurality of pieces of pixel information, and may interpolate the pixel information of the subject. . The main subject determination unit 112 may use nearest neighbor interpolation (Nearest neighbor), bilinear interpolation (Bilinear), bicubic interpolation (Bicubic), or the like as an interpolation technique.

  As described above, the main subject determination unit 112 may acquire main subject selection information via the communication interface 150 and may determine the main subject based on the selection information. Thereby, the unmanned aerial vehicle 100 can determine the main subject desired by the user from the subjects included in the color segmented image G2 based on the live view image G1. Thus, the unmanned aerial vehicle 100 can adjust the composition based on the main subject desired by the user.

  The main subject determining unit 112 may determine the main subject without using the subject selection information. A plurality of main subjects may be determined.

  For example, the main subject determining unit 112 may determine a predetermined region as the main subject among the color component regions in the color-segmented image G2. That is, the main subject determination unit 112 may group the live view images G1 by color and recognize a predetermined color group as the main subject. In this case, the main subject determination unit 112 may determine, for example, a region (for example, a region C that is an island region in FIG. 6B) located at the center surrounded by each color component as the main subject. Thereby, the unmanned aerial vehicle 100 can make a subject that stands out by being surrounded by a surrounding region as a main subject.

  In addition, the main subject determining unit 112 may determine a region having a predetermined size or less (for example, the minimum) as a main subject among regions divided by color components. Thereby, the unmanned aerial vehicle 100 can adjust the composition with reference to an area of a small size that is more difficult to discriminate than other areas in the live view image G1. For example, a person who gets lost in the mountains can be determined as the main subject.

  In addition, the main subject determination unit 112 may determine a predetermined color area obtained from the operation unit 83 or the memory 87 as the main subject. Thereby, the unmanned aerial vehicle 100 can determine the subject of the color desired by the user or the subject of the color previously determined as the main imaging target as the main subject.

  In this way, the unmanned aerial vehicle 100 can roughly determine the subject for each color such as a mountain, the sea, or a person's clothes. Therefore, the unmanned aerial vehicle 100 can register a specific color component as a subject to be noticed, for example, and can automatically determine the main subject based on the color.

  The main subject determining unit 112 may divide the live view image G1 into one or more regions based on the spatial frequency and determine a predetermined region as the main subject. That is, the main subject determining unit 112 may group the live view images G1 with the spatial frequency and recognize a group within a predetermined spatial frequency range as the main subject.

  The higher the spatial frequency, the more edges in the image and the clearer the image. On the other hand, the lower the spatial frequency, the fewer the edges in the image and the blurred image. Therefore, the main subject determining unit 112 may determine, as a main subject, a region having a spatial frequency equal to or higher than a predetermined frequency (for example, the highest) among regions divided by the spatial frequency. Thereby, the unmanned aerial vehicle 100 can adjust the composition with reference to a relatively clear area.

  The main subject determining unit 112 may determine a region having a predetermined spatial frequency obtained from the operation unit 83 or the memory 87 as the main subject. Thereby, the unmanned aerial vehicle 100 can determine, as a main subject, a subject having a spatial frequency desired by the user or a subject having a spatial frequency that has been previously determined as a main imaging target.

  The main subject determination unit 112 determines a predetermined subject as a main subject among one or more subjects included in the live view image G1 based on information on the aerial shooting position where the live view image G1 was taken aerial. Good. In this case, the main subject determination unit 112 may acquire map information of the map DB accumulated by an external server or the like via the communication interface 150. The map information may include information on the type of subject (eg, mountain, river, sea, building). The main subject determination unit 112 may determine the main subject based on the type of subject and the size of the subject.

  Thereby, the unmanned aerial vehicle 100 can adjust the composition based on the geographical information and the terrain information of the unmanned aerial vehicle 100.

  In addition, the main subject determination unit 112 may acquire evaluation information related to the aerial image of the subject included in the live view image G1 from an external server or the like via the communication interface 150. The main subject determination unit 112 may determine a subject whose evaluation information is greater than or equal to a predetermined reference (for example, the highest evaluation) as the main subject.

  Thereby, the unmanned aerial vehicle 100 can adjust the composition with reference to the subject highly evaluated by the other person. The information on the aerial shooting position may be obtained from information acquired by the GPS receiver 240.

  In addition, the main subject determination unit 112 may determine the main subject based on an imaging mode that is set when the determined main subject is aerial. Further, when the sunset mode is set, the sun, the vicinity of the horizon where the sun sets, or the vicinity of the horizon may be determined as the main subject.

  Thereby, the unmanned aerial vehicle 100 can determine the main subject in consideration of the imaging mode in which the subject is considered. Therefore, the unmanned aerial vehicle 100 can determine the main subject suitable for the imaging information (camera parameters) set according to the imaging mode, and can expect to obtain a clear aerial image.

  The main subject determination unit 112 may accumulate main subject determination information based on any of the above-described methods, for example, in the memory 160 together with information on the imaging range such as the live view image G1. The main subject determining unit 112 determines a main subject among subjects included in the live view image G1 based on main subject determination information stored in the memory 160 (that is, main subject determination information that has been proven in the past). It's okay. For example, the main subject determining unit 112 may select a subject that has been a main subject more than a predetermined number of times (for example, the most) in the past (for example, the same) or a main frequency more than a predetermined frequency (for example, most frequently) in the past. The subject set as the subject may be determined as the main subject.

  Thereby, the unmanned aerial vehicle 100 can determine the main subject by machine learning according to the past performance, in other words, according to the selection tendency of the user and the determination tendency of the unmanned aircraft 100. The unmanned aerial vehicle 100 can adjust the composition with reference to the main subject determined by machine learning.

Next, an example of composition determination will be described.
FIG. 7 is a diagram illustrating an example of composition selection.

  When the main subject is determined, the composition determination unit 113 acquires information on composition candidates scheduled for aerial photography from the memory 160 based on the main subject. For example, the composition determination unit 113 compares the composition of the live view image with the sample information of the composition held in the memory 160, and sets the sample information of the composition having a degree of coincidence equal to or higher than a predetermined reference to the composition of the live view image. Two or more composition candidates may be determined. The composition determination unit 113 determines the degree of coincidence between the composition of the live view image and the sample information of the composition based on the shape, position, and size of the main subject in both compositions, and the shape, position, and size of a plurality of subjects in both compositions. The determination may be made based on at least one of a positional relationship, etc.

  As an example, as shown in FIG. 7, it is assumed that the main subject determining unit 112 selects a river having an elongated shape as the main subject. In this case, the composition determination unit 113 refers to the sample information of the composition held in the memory 160 and, for example, obtains a diagonal composition, a three-part composition, or other composition suitable for aerial shooting of an elongated shape region. You may acquire as a composition candidate. In the diagonal composition, the river as the main subject may be arranged along the diagonal line in the composition. In the three-divided composition, the river may be arranged along the dividing line for dividing the river into three as the main subject or overlapping the dividing line. The composition determination unit 113 may transmit the acquired composition candidate to the mobile terminal 80 via the communication interface 150. In the portable terminal 80, the wireless communication unit 85 may acquire composition candidate information. As shown in FIG. 7, the display unit 88 may display composition candidate information.

  The display unit 88 may display the composition candidates according to the imaging assistance application, and may perform guide display for determining which composition candidate (here, the diagonal composition, the three-part composition) is selected as the scheduled composition. In FIG. 7, it is illustrated that the diagonal composition is selected as the scheduled composition via the operation unit 83. In this case, in the mobile terminal 80, the wireless communication unit 85 transmits composition selection information obtained via the operation unit 83 to the unmanned aircraft 100. In the unmanned aerial vehicle 100, the communication interface 150 receives composition selection information. The composition determination unit 113 determines a planned composition based on composition selection information.

  The composition candidate image displayed on the display unit 88 may be an image held in the memory 160 and sent to the portable terminal 80 (for example, a composition image as sample information of the composition). The composition candidate image displayed on the display unit 88 may be an image generated by the composition determination unit 113 and sent to the portable terminal 80. In this case, the composition determination unit 113 may generate a composition candidate image to be displayed based on the composition information corresponding to the shape of the main subject and the live view image G1. For example, when there are rivers as main subjects in the live view image G1 and mountains exist as subjects on both sides of the live view image G1, the composition determination unit 113 simplifies the shapes of these subjects, for example, and sets each subject according to the positional relationship of the composition candidates. May be generated. Further, instead of the composition determination unit 113, the terminal control unit 81 of the mobile terminal 80 uses the composition information corresponding to the shape of the main subject acquired from the unmanned aircraft 100 and the live view image G1 to display the composition candidate image. May be generated.

  In this way, the composition determination unit 113 can determine composition candidates in consideration of the shape of the main subject, for example. The display unit 88 of the portable terminal 80 can display the determined composition candidate and can prompt the user to select. The display unit 88 may display the composition candidate as a still image or a moving image preview. The composition determination unit 113 may acquire composition selection information via the communication interface 150 and determine a planned composition based on the selection information.

  Thereby, the unmanned aerial vehicle 100 can determine a composition in which the main subject is arranged at a desired position in the aerial image planned to be aerial, and can attractively take the main subject. Further, the composition determination unit 113 can automatically determine composition candidates, and can present composition candidates in a limited manner from sample information of various compositions. Since the composition determination unit 113 determines the planned composition based on the selection information, the intention of the user can be reflected in the selection of the planned composition.

  The composition determination unit 113 may determine the composition according to the shape of the main subject without presenting composition candidates. Thereby, the unmanned aerial vehicle 100 can take an aerial photograph using a planned composition with an excellent balance in consideration of the shape of the main subject, and the main subject can be aerial photographed attractively.

  The composition determination unit 113 may determine the composition without using the composition selection information.

  For example, the composition determination unit 113 may recognize the scene of the live view image G1 using a scene recognition algorithm. The composition determination unit 113 may determine a composition suitable for the scene or present a composition candidate in accordance with the scene recognition result. For example, when the live view image G1 is recognized as a sunrise (Sun Rise) scene, the composition determination unit 113 determines a composition such as a central one-point composition or a two-part composition suitable for imaging the scene, These composition candidates may be presented. For scene recognition, for example, deep learning may be used, and a convolutional neural network may be used.

  Thereby, the unmanned aerial vehicle 100 can determine a composition capable of attractively photographing the scene in accordance with the scene in which the live view image G1 is aerial.

  The composition determination unit 113 may store, for example, composition determination information based on any of the above-described methods in the memory 160 together with information on the imaging range such as the live view image G1. The composition determination unit 113 may determine the composition in consideration of the main subject based on the composition determination information accumulated in the memory 160 (that is, the composition determination information that has been proven in the past). For example, the composition determination unit 113 may use a composition that has been used more than a predetermined number of times (for example, most frequently) in the same (for example, the same) imaging range, or a composition that has been used more than a predetermined frequency (for example, most frequently) in the past. May be determined as the composition of the aerial photography schedule.

  Thereby, the unmanned aerial vehicle 100 can determine the composition by machine learning according to the past results, in other words, according to the selection tendency of the user and the determination tendency of the unmanned aircraft 100. The unmanned aerial vehicle 100 can attractively take aerial photographs of the main subject with a composition determined by machine learning.

  In addition, the composition determination unit 113 may acquire evaluation information regarding an aerial image using the composition from an external server or the like via the communication interface 150. The main subject determination unit 112 may determine a composition whose evaluation information is greater than or equal to a predetermined reference (for example, the highest evaluation) as a planned composition.

  Thereby, the unmanned aerial vehicle 100 can determine the composition highly evaluated by others as the planned composition. Therefore, the unmanned aerial vehicle 100 can acquire an aerial image in which a subject is arranged with an objectively preferable composition.

  Next, an example of generating operation information will be described.

  The motion information generation unit 114 determines an imaging range in order to perform aerial photography according to the determined composition. The imaging range can be specified by the position of the unmanned aircraft 100, the orientation of the unmanned aircraft 100, the orientation of the imaging unit 220, the angle of view of the imaging unit 220 or the imaging unit 230, and the like. Therefore, the operation information generation unit 114 changes the information for changing from the operation state of the unmanned aircraft 100 in which the live view image G1 is aerial to the operation state of the unmanned aircraft 100 for realizing the determined composition to the operation information. It may be generated as information.

  For example, the motion information generation unit 114 uses the unmanned aerial vehicle after moving from the position of the unmanned aircraft 100 before moving (when the live view image G1 is aerial) as the motion information (for realizing the determined composition). The movement information of the unmanned aerial vehicle 100 for moving to the 100 position may be included.

  The motion information generation unit 114 operates as the motion information of the unmanned aircraft 100 after the change (for realizing the determined composition) from the direction of the unmanned aircraft 100 before the change (when the live view image G1 is aerial). The rotation information of the unmanned aerial vehicle 100 for changing to the direction may be included.

  The motion information generation unit 114 changes the rotation state (for example, the rotation angle) of the gimbal 200 before the change (corresponding to the orientation of the imaging unit 220) as the operation information. Rotation information may be included.

  The motion information generation unit 114 has, as the motion information, the image capturing unit 220 or the image capturing unit 230 for changing the angle of view of the image capturing unit 220 or the image capturing unit 230 before the change to the orientation of the image capturing unit 220 or the image capturing unit 230 after the change. The angle of view change information may be included. The angle of view of the imaging unit 220 or the imaging unit 230 may correspond to the zoom magnification of the imaging unit 220 or the imaging unit 230. That is, the motion information generation unit 114 uses, as motion information, zoom magnification change information for changing from the zoom magnification of the imaging unit 220 or the imaging unit 230 before the change to the zoom magnification of the imaging unit 220 or the imaging unit 230 after the change. May include.

  FIG. 8A is a diagram illustrating an example of rotation of the imaging range for performing aerial photography with the determined composition. FIG. 8B is a diagram illustrating an example of movement of the unmanned aerial vehicle 100 for performing aerial photography with the determined composition. FIG. 8C is a diagram for explaining the movement of the unmanned aerial vehicle 100 as viewed from the horizontal direction.

  FIG. 8A shows a current composition C1 as a composition in which a live view image is simplified and a planned composition C2. In FIG. 8A, as in FIG. 7, mountains M11 and M12 exist on both sides of the river RV11. The planned composition C2 is a diagonal composition.

  The motion information generation unit 114 compares the size of the subject in the current composition C1 with the size of the subject in the planned composition C2. The motion information generation unit 114 may calculate the amount of change in altitude of the unmanned aerial vehicle 100 (that is, the amount of movement in the direction of gravity) based on the size of the subject in the current composition C1 and the size of the subject in the planned composition C2. For example, when the size of the subject in the planned composition C2 is twice the size of the subject in the current composition C1, the motion information generation unit 114 moves and moves so that the altitude of the unmanned aircraft 100 is halved. The amount may be calculated. For example, when the subject size in the planned composition C2 is 1/2 with respect to the subject size in the current composition C1, the motion information generation unit 114 moves and moves so that the altitude of the unmanned aircraft 100 is doubled. The amount may be calculated. The altitude information of the unmanned aerial vehicle 100 in the current composition C1 may be an aerial shooting altitude when the live view image G1 is taken aerial, and may be acquired by the barometric altimeter 270 or the like.

  In FIG. 8A, the size of the river RV11 and the sizes of the mountains M11 and M12 are the same in the current composition C1 and the planned composition C2. That is, the distance (subject distance) to the river RV11 or the mountains M11 and M12 as the subject is not changed. In this case, the motion information generation unit 114 may determine that the altitude of the unmanned aerial vehicle 100 has not been changed and the amount of movement in the direction of gravity is 0.

  Thereby, the unmanned aerial vehicle 100 can easily calculate the amount of change in altitude (that is, the amount of movement in the direction of gravity) by comparing the current composition C1 and the planned composition C2. Accordingly, the unmanned aerial vehicle 100 can be moved not only in a two-dimensional space (horizontal direction) but also in a three-dimensional space.

  Instead of changing the altitude of the unmanned aircraft 100, the zoom magnification of the imaging unit 220 or the imaging unit 230 may be changed.

  The motion information generation unit 114 may compare the positional relationship of each subject in the current composition C1 with the positional relationship of each subject in the planned composition C2. Based on the positional relationship of each subject in the current composition C1 and the positional relationship of each subject in the planned composition C2, the motion information generating unit 114 may rotate the rotational amount and rotational direction of the composition, that is, the rotational amount and rotational direction of the unmanned aircraft 100, or The rotation amount and the rotation direction of the imaging unit 220 or the imaging unit 230 may be calculated. The direction of rotation here may be, for example, a direction along the horizontal direction. Note that the positional relationship information of each subject may be calculated and acquired by mapping through mathematical coordinate transformation.

  In FIG. 8A, each subject in the planned composition C2 is rotated 30 degrees counterclockwise with respect to each subject in the current composition C1. In this case, the motion information generation unit 114 calculates that the rotation direction is counterclockwise and that the rotation amount is 30 degrees around the optical axis of the imaging unit 220 or the imaging unit 230.

  Thereby, the unmanned aerial vehicle 100 can easily calculate the rotation amount of the composition by comparing the current composition C1 and the planned composition C2.

  The motion information generation unit 114 may generate rotation information of the gimbal 200. For example, the motion information generation unit 114 displays the angle of view information of the imaging unit 220, the screen position of the subject on the display unit 88 of the mobile terminal 80 in the live view image G1, and the screen of the same subject on the display unit 88 in the planned composition. Based on the position, rotation information of the gimbal 200 may be calculated. The moving distance of the same subject on the screen is proportional to the change amount (rotation amount) of the rotation angle of the gimbal 200.

  In FIG. 8B, in the current composition C1 and the planned composition C2, the distance w1 (corresponding to the movement distance on the screen) of the same subject M21 is 1/6 of the side w of the screen of the display unit 88. Suppose that Further, it is assumed that the imaging field angle indicated by the field angle information of the imaging unit 220 is 90 degrees. In this case, the rotation angle (angle θ2) of the gimbal 200 for realizing the planned composition C2 is 15 degrees. Further, the motion information generation unit 114 may derive (for example, calculate) the rotation direction of the gimbal 200 based on the correspondence relationship between the movement direction in the real space and the movement direction on the screen of the display unit 88. Note that the moving direction in the real space and the moving direction on the screen of the display unit 88 may be opposite to each other. For example, when the gimbal 200 is rotated in the gravitational direction (downward) while taking an aerial image of FIG. 6A, the position of the island in the aerial image moves upward in FIG. 6A.

  The operation information generation unit 114 may generate movement information of the unmanned aircraft 100. For example, the motion information generation unit 114 instructs the motion control unit 115 to fly the unmanned aircraft 100 for a predetermined distance (for example, a predetermined short distance). The unmanned aerial vehicle 100 flies for a predetermined distance according to the control of the operation control unit 115. The motion information generation unit 114 may determine the correspondence between the movement distance due to the flight in real space and the movement distance on the screen of the display unit 88 in cooperation with the terminal control unit 81 of the portable terminal 80.

  Specifically, the motion information generation unit 114 may notify the mobile terminal 80 of information on a predetermined distance related to the flight via the communication interface 150. The terminal control unit 81 of the portable terminal 80 detects information on the movement distance that the same subject has moved on the screen of the display unit 88 in accordance with the movement of the unmanned aircraft 100 by a predetermined distance in the aerial image during the flight. It's okay. The terminal control unit 81 may transmit information on the travel distance to the unmanned aircraft 100 via the wireless communication unit 85. The motion information generation unit 114 may receive information on the travel distance via the communication interface 150. In this way, the motion information generation unit 114 determines the correspondence between the movement distance due to the flight in the real space and the movement distance on the screen, and holds this correspondence information in the memory 87 or the like. Good. For example, the memory 160 may hold information that the moving distance in the real space is α times the moving distance on the screen.

  In addition, the terminal control unit 81 may also store information on the correspondence relationship between the moving direction in the real space and the moving direction on the screen in the memory 160. In addition, the position before and behind the movement in real space may be acquired by the GPS receiver 240, for example.

  In FIG. 8B, the same subject M21 imaged by the imaging unit 220 has moved from the position p1 to the position p2. Based on the distance d1 (corresponding to the movement distance) between the position p1 and the position p2 and the correspondence information (for example, α times) held in the memory 120d, the movement distance of the unmanned aircraft 100 for realizing the planned composition is calculated. You can do it.

  As a result, the unmanned aircraft 100 can determine the position of the unmanned aircraft 100 that should be positioned to realize the planned composition C2 even when the unmanned aircraft 100 does not include a large number of sensors. Information on the moving direction can be acquired.

  Further, the motion information generation unit 114 may generate movement information by other methods. FIG. 8C is a diagram illustrating an example of the relationship between the change in the angle of view of the imaging unit 220 before and after movement and the horizontal movement distance. The point which shows each position of FIG. 8C has shown the mode seen from the side.

  For example, the motion information generation unit 114 may acquire the angle θ1 with respect to the gravity direction of the gimbal 200 when the live view image G1 is aerial. The angle θ1 may be calculated based on the inclination of the unmanned aerial vehicle 100 with respect to the direction of gravity during the aerial shooting of the live view image G1 and the inclination of the gimbal 200 with respect to the unmanned aircraft 100. Based on the altitude of the unmanned aerial vehicle 100 and the angle θ1 of the gimbal 200, the motion information generating unit 114 is arranged in the horizontal direction where the horizontal position p11 of the unmanned aircraft 100 and the central portion of the imaging range of the unmanned aircraft 100 are located. A distance d11 from the position p12 may be calculated.

  Then, the motion information generation unit 114 is based on the angle of view information of the imaging unit 220, the screen position of the subject in the current composition C1, and the screen position of the same subject on the display unit 88 in the planned composition C2. Rotation information (for example, a rotation angle corresponding to the angle θ2) may be calculated. At this time, the motion information generation unit 114 may refer to the information on the correspondence relationship between the movement distance due to the flight in the real space and the movement distance on the screen, which is stored in the memory 160. Based on the altitude h of the unmanned aircraft 100 and the angle (θ1 + θ2) after rotation of the gimbal 200 with respect to the direction of gravity, the motion information generation unit 114 and the position p11 in the horizontal direction of the unmanned aircraft 100 and after rotation (planned composition C2). The distance d1 + d2 from the horizontal position p13 where the center of the imaging range of the unmanned aerial vehicle 100 (when realized) is located may be calculated. Therefore, the motion information generation unit 114 includes the horizontal position p12 where the center of the imaging range of the unmanned aerial vehicle 100 before rotation (at the time of aerial shooting of the live view image G1), and after rotation (when the planned composition C2 is realized). The difference between the horizontal position p13 where the center of the imaging range of the unmanned aircraft 100 is located, that is, the movement distance d12 corresponding to the angle θ2 can be calculated.

  Thereby, the unmanned aerial vehicle 100 realizes the planned composition even if the information on the correspondence between the movement distance in the real space and the movement distance on the screen is not obtained in advance, and this correspondence relation is unknown. The movement amount and the movement direction can be calculated.

  The motion information generation unit 114 calculates the movement distance d12 in the direction of one axis in the horizontal direction (for example, the x direction shown in FIG. 8C) by the above-described method. Similarly, the motion information generation unit 114 may calculate a movement distance d22 (not shown) in another direction (for example, the y direction) orthogonal to one axis in the horizontal direction. The motion information generation unit 114 may calculate the movement distance in the horizontal direction (xy direction) by combining the movement distance d12 and the movement distance d22. Further, the motion information generation unit 114 may calculate the movement direction by combining the movement distances of the x-direction component and the y-direction component.

  In this manner, the unmanned aerial vehicle 100 can grasp the operation of the unmanned aerial vehicle 100 necessary for generating the operation information from the current composition C1 to the planned composition C2. In addition, the unmanned aircraft 100 can generate the rotation information of the gimbal 200 as the operation information, thereby changing the imaging range by adjusting the gimbal 200 and providing the operation information for obtaining the desired planned composition C2. In this case, since the unmanned aircraft 100 can change the imaging range by the rotation of the gimbal 200, the unmanned aircraft 100 does not need to be moved, power consumption for the unmanned aircraft 100 can be saved, and low cost can be realized. it can.

  The unmanned aerial vehicle 100 can calculate the rotation angle of the gimbal 200 by a relatively simple calculation based on the position of the subject in the current composition C1 and the planned composition C2. Therefore, the unmanned aerial vehicle 100 can suppress the calculation processing load low.

  Further, the unmanned aerial vehicle 100 generates movement information of the unmanned aircraft 100 as operation information, thereby changing the imaging range by adjusting the geographical position of the unmanned aircraft 100 to obtain operation information for obtaining a desired planned composition C2. Can be provided. That is, the unmanned aerial vehicle 100 moves spatially, thereby changing the imaging range and realizing an appropriate scheduled composition C2.

Next, an operation example of the imaging assistance system 10 will be described.
FIG. 9 is a sequence diagram illustrating an operation example of the imaging assistance system 10.

  First, in the unmanned aircraft 100, the imaging unit 220 captures an aerial image. (S101). The communication interface 150 transmits the captured aerial image (for example, live view image) to the portable terminal 80 (S102). Here, as an example, the aerial image in S101 will be described as a live view image.

  In the portable terminal 80, the wireless communication unit 85 receives a live view image (S151). The display unit 88 may display a live view image, for example. The user may confirm the display of the live view image, and may perform an operation for adjusting the composition via the operation unit 83 when desiring to adjust the composition. When the terminal control unit 81 receives a composition adjustment operation through the operation unit 83, the terminal control unit 81 activates the imaging assist application and transmits a composition adjustment command to the unmanned aircraft 100 via the wireless communication unit 85 (S152).

  In the unmanned aircraft 100, when the UAV control unit 110 acquires the composition adjustment command via the communication interface 150 (S103), the UAV control unit 110 activates the imaging assistance application. The main subject determining unit 112 determines the main subject in the live view image (S104). In S <b> 104, the main subject determination unit 112 may provide screen information for selecting the main subject to the mobile terminal 80 via the communication interface 150. In the portable terminal 80, the wireless communication unit 85 receives screen information for selecting the main subject from the unmanned aircraft 100, the operation unit 83 accepts a main subject selection operation, and the wireless communication unit 85 receives the main subject. The selection information based on the selection operation may be transmitted to the unmanned aircraft 100 (S153). The main subject determination unit 112 may acquire main subject selection information via the communication interface 150 and may determine the main subject based on the main subject selection information.

  The composition determination unit 113 determines the composition based on the determined main subject (S105). In S <b> 105, the composition determination unit 113 may provide screen information for selecting the main composition to the mobile terminal 80 via the communication interface 150. In the portable terminal 80, the wireless communication unit 85 receives screen information for selecting a composition from the unmanned aircraft 100, the operation unit 83 receives a composition selection operation, and the wireless communication unit 85 performs a composition selection operation. The selection information based may be transmitted to the unmanned aircraft 100 (S154). The composition determination unit 113 may acquire composition selection information via the communication interface 150 and may determine the composition based on the composition selection information.

  Based on the determined composition, the motion information generation unit 114 generates (for example, calculates) motion information of the unmanned aircraft 100 (for example, information on the moving direction and amount of movement of the unmanned aircraft 100) (S106). The moving direction and moving amount of the unmanned aerial vehicle 100 here may be, for example, the moving direction and moving amount from the aerial shooting position to the aerial shooting position for realizing the planned composition of the live view image in S101.

  The operation control unit 115 performs flight control toward the destination based on the calculated moving direction and moving amount, and moves (S107). This destination is a position moved by the moving direction and moving amount from the position of the live view image before moving. The operation control unit 115 determines whether or not the movement to the destination is completed (S108). If the movement to the destination is not completed, the process proceeds to S107.

  When the movement to the destination is completed, the communication interface 150 transmits a movement completion notification to the portable terminal 80 (S109). In the portable terminal 80, the wireless communication unit 85 receives the movement completion notification from the unmanned aircraft 100 (S155). The operation unit 83 may accept an operation for imaging by the imaging unit 220 or the imaging unit 230 included in the unmanned aircraft 100 after movement. When the terminal control unit 81 receives an imaging operation through the operation unit 83, the terminal control unit 81 transmits an imaging command to the unmanned aircraft 100 via the wireless communication unit 85 (S156).

  In the unmanned aircraft 100, the communication interface 150 receives an imaging command from the portable terminal 80 (S110). The imaging unit 220 or the imaging unit 230 captures an aerial image according to the imaging command (S111). This aerial image is an image taken aerially at the position where the unmanned aircraft 100 has moved in order to realize the planned composition, and is an image according to the planned composition.

  Note that transmission and reception of imaging commands may be omitted. In this case, the unmanned aircraft 100 may take an aerial photograph at the position after the movement when or after the movement based on the operation information is completed.

  Note that the processing of S101 to S106 may be performed while the unmanned aircraft 100 is not moving (when it is located at a predetermined position without moving), or may be performed while the unmanned aircraft 100 is moving.

  Note that the operation information may be rotation information of the gimbal 200 instead of movement information of the unmanned aircraft 100. In this case, in S106, the motion information generation unit 114 generates (for example, calculates) the motion information of the unmanned aerial vehicle 100 (for example, information on the rotation direction and amount of rotation of the gimbal 200) based on the determined composition. Here, the rotation direction and the rotation amount of the gimbal 200 may be, for example, the rotation direction and the rotation amount of the gimbal 200 for realizing the planned composition of the live view image of S101 from the aerial shooting position. In S <b> 107, the operation control unit 115 controls the rotation toward the target rotation position based on the calculated rotation direction and rotation amount, and rotates the gimbal 200. The target rotation position is a position rotated by the rotation direction and the rotation amount from the angle of the gimbal 200 at the time of capturing the live view image before the movement. In S108, the operation control unit 115 determines whether or not the rotation to the target rotation position is completed. If the rotation to the target rotation position is not completed, the process proceeds to S107, and the operation control unit 115 continues the rotation operation.

  Note that the operation control unit 115 may perform either one of the rotation control of the gimbal 200 or the flight control of the unmanned aircraft 100, or both. In addition, since the change of the imaging range by rotation of the gimbal 200 is not accompanied by the movement of the unmanned aircraft 100, the degree of change of the imaging range is small. On the other hand, since the change of the imaging range due to the movement of the unmanned aircraft 100 is accompanied by the movement of the unmanned aircraft 100, the imaging range is greatly changed. Therefore, if the flight control of the unmanned aircraft 100 is performed after the rotation control of the gimbal 200, the flight control of the unmanned aircraft 100 can assist in realizing the desired composition even if the rotation of the gimbal 200 does not reach the desired composition. It is. That is, the unmanned aerial vehicle 100 can achieve a desired composition by the flight control of the unmanned aerial vehicle 100 while saving energy by the rotation control of the gimbal 20.

  Thus, according to the unmanned aerial vehicle 100 and the imaging assisting system 10, it is possible to determine a composition for attractively shooting a desired subject in consideration of the desired subject. In other words, the unmanned aerial vehicle 100 and the imaging assist system 10 can assist in capturing an image in consideration of improving the composition of the captured image as well as simply placing a desired subject in the captured image. Therefore, even when the user does not have sufficient know-how regarding the photography, the determination of the composition can be assisted by the unmanned aerial vehicle 100, and aerial photography of a desired subject can be assisted. Further, since the unmanned aerial vehicle 100 can operate in accordance with the composition (for example, movement of the unmanned aircraft 100 and adjustment of the rotation angle of the gimbal 200), the planned composition can be used in future aerial photography.

  The unmanned aerial vehicle 100 can quickly perform the operation of the unmanned aerial vehicle 100 based on the motion information by determining the main subject, determining the composition, and generating the motion information related to the imaging assistance. The unmanned aerial vehicle 100 can reduce the processing load of the portable terminal 80 by determining the main subject, determining the composition, and generating the operation information related to the imaging assistance, and reduces the communication load with the portable terminal 80. Reduction can also be realized. Therefore, the mobile terminal 80 can contribute to processing related to imaging assistance in cooperation with the unmanned aircraft 100 while reducing the processing load of the mobile terminal 80 itself.

  The unmanned aerial vehicle 100 can generate a scheduled composition based on a desired subject included in the live view image by determining a main subject, determining a composition, and the like based on the live view images of S101 and S102. . Therefore, the unmanned aerial vehicle 100 can aerially photograph a desired subject with a desired composition in a flow of performing a series of imaging. The unmanned aerial vehicle 100 can perform the main imaging with a composition suitable for the main subject included in the subject reflected during the temporary imaging before the main imaging.

  The display unit 88 may display operation information. The display information related to the operation information may be “move 10 m eastward”, “rotate the gimbal 200 20 degrees in the direction of gravity”, and the like. Instead of displaying the operation information on the display unit 88, the operation information may be presented by another presentation unit. For example, an audio output unit (not shown) may output audio information related to the operation information, or a vibration unit (not shown) may perform vibration indicating the operation information.

  The unmanned aircraft 100 presents the operation information, so that the user can confirm the content of the operation information. Therefore, an operation instruction may be sent from the transmitter 50 to the unmanned aircraft 100 by a user who has confirmed the operation information operating the transmitter 50. The unmanned aerial vehicle 100 may acquire an operation instruction via the communication interface 150 and move the unmanned aircraft 100 to an aerial shooting position for realizing a planned composition. In this case, even if the unmanned aircraft 100 does not have the function of the operation control unit 115, the operation of the unmanned aircraft 100 for realizing the planned composition can be performed.

(Second Embodiment)
In the first embodiment, it is exemplified that the unmanned aerial vehicle determines the main subject, determines the composition, and generates the operation information. The second embodiment exemplifies that the mobile terminal determines the main subject, determines the composition, and generates the operation information. Note that in the second embodiment, the description of the same configuration and operation as in the first embodiment will be omitted or simplified.

  FIG. 10 is a schematic diagram illustrating a configuration example of an imaging assistance system 10A according to the second embodiment. The imaging assistance system 10A includes an unmanned aircraft 100A, a transmitter 50, and a portable terminal 80A. Unmanned aerial vehicle 100A, transmitter 50, and portable terminal 80A can communicate with each other by wired communication or wireless communication (for example, wireless LAN (Local Area Network)).

  The portable terminal 80A determines a composition for aerial photography with the unmanned aircraft 100A, and generates operation information of the unmanned aircraft 100A so as to obtain the determined composition. Unmanned aerial vehicle 100A controls the operation of unmanned aerial vehicle 100A in accordance with the operation information. The portable terminal 80 </ b> A can be carried by a user who plans to take an aerial photograph using the unmanned aircraft 100 </ b> A together with the transmitter 50. The portable terminal 80A assists aerial photography with the unmanned aircraft 100A.

  FIG. 11 is a block diagram illustrating an example of a hardware configuration of the unmanned aircraft 100A. The unmanned aerial vehicle 100 </ b> A includes a UAV control unit 110 </ b> A instead of the UAV control unit 110 as compared with the unmanned aircraft 100 in the first embodiment. In the unmanned aerial vehicle 100A of FIG. 11, the same components as those of the unmanned aircraft 100 shown in FIG. 2 are denoted by the same reference numerals, and the description thereof is omitted or simplified. Note that the memory 160 may not hold information related to imaging assistance (for example, sample information of the composition, information on the correspondence between the distance in the real space and the distance on the screen).

  FIG. 12 is a block diagram illustrating an example of a functional configuration of the UAV control unit 110A. The UAV control unit 110A includes an operation control unit 115 and an operation information acquisition unit 116. In the UAV control unit 110A of FIG. 12, the same components as those of the UAV control unit 110 shown in FIG. 3 are denoted by the same reference numerals, and the description thereof is omitted or simplified.

  The operation information acquisition unit 116 acquires operation information of the unmanned aircraft 100A from the portable terminal 80A, for example, via the communication interface 150. The operation control unit 115 controls the operation of the unmanned aircraft 100A according to the acquired operation information. The content of the operation control of the unmanned aircraft 100A may be the same as that of the first embodiment.

  FIG. 13 is a block diagram illustrating an example of a hardware configuration of the mobile terminal 80A. The mobile terminal 80 </ b> A includes a terminal control unit 81 </ b> A instead of the terminal control unit 81 as compared with the mobile terminal 80 in the first embodiment. In the mobile terminal 80A of FIG. 13, the same reference numerals are given to the same components as those of the mobile terminal 80 shown in FIG. 4, and the description thereof is omitted or simplified. Note that the memory 87 is similar to the memory 160 included in the unmanned aircraft 100 in the first embodiment, such as information related to imaging assistance (for example, sample information of composition, correspondence between distance in real space and distance on the screen) And information related to machine learning).

  FIG. 14 is a block diagram illustrating an example of a functional configuration of the terminal control unit 81A. The terminal control unit 81A includes an image acquisition unit 811 that includes a main subject determination unit 812, a composition determination unit 813, and an operation information generation unit 814. The main subject determination unit 812 and the composition determination unit 813 are examples of an information acquisition unit. The motion information generation unit 114 is an example of a generation unit.

  The image acquisition unit 811 may acquire an image held in the memory 87 (for example, an aerial image captured by the imaging unit 220 or the imaging unit 230 of the unmanned aircraft 100A). For example, the image acquisition unit 811 may acquire an aerial image during the aerial imaging by the imaging unit 220 or the imaging unit 230 via the communication interface 150. The aerial image may be a moving image or a still image. Aerial video during aerial photography is also referred to as a live view image. A live view image is mainly exemplified as an aerial image acquired by the image acquisition unit 811.

  The main subject determination unit 812 determines (specifies) the main subject among one or more subjects included in the live view image acquired by the image acquisition unit 811. The determination of the main subject is an example of acquisition of information on the main subject. The main subject determination method by the main subject determination unit 812 may be the same as the main subject determination method by the main subject determination unit 112 included in the unmanned aircraft 100 according to the first embodiment.

  The composition determination unit 813 determines a composition for imaging the determined main subject. The determination of the composition for imaging the main subject is an example of acquisition of composition information for imaging the main subject. The composition determination method by the composition determination unit 813 may be the same as the composition determination method by the composition determination unit 113 included in the unmanned aircraft 100 according to the first embodiment.

  The operation information generation unit 814 generates operation information of the unmanned aircraft 100A for realizing aerial photography according to the determined composition. The operation information generation method by the operation information generation unit 814 may be the same as the operation information generation method by the operation information generation unit 114 included in the unmanned aircraft 100 according to the first embodiment. The generated operation information may be transmitted to the unmanned aircraft 100A by the wireless communication unit 85, for example.

Next, an operation example of the imaging assistance system 10A will be described.
FIG. 15 is a sequence diagram illustrating an operation example of the imaging assistance system 10A.

  First, the unmanned aerial vehicle 100A performs the processes of S101 and S102. The portable terminal 80A performs the process of S151.

  In the portable terminal 80A, the display unit 88 may display a live view image, for example. The user may confirm the display of the live view image, and may perform an operation for adjusting the composition via the operation unit 83 when desiring to adjust the composition. This operation is an example of a composition adjustment start instruction. When the terminal control unit 81 receives a composition adjustment operation through the operation unit 83, the terminal control unit 81 activates the imaging assistance application (S161).

  The main subject determining unit 812 determines the main subject in the live view image (S162). In S162, the main subject determining unit 812 may display a selection screen for selecting the main subject on the display unit 88. The operation unit 83 may acquire main subject selection information by receiving a main subject selection operation. The main subject determination unit 812 may determine the main subject based on the main subject selection information.

  The composition determination unit 813 determines the composition based on the determined main subject (S163). In S163, the composition determination unit 813 may cause the display unit 88 to display a selection screen for selecting a planned composition. The operation unit 83 may acquire planned composition selection information by receiving a composition selection operation. The composition determination unit 813 may determine the composition based on the composition selection information.

  The motion information generation unit 814 generates motion information of the unmanned aircraft 100A based on the determined composition (S164). For example, the wireless communication unit 85 transmits the generated operation information of the unmanned aircraft 100A to the unmanned aircraft 100A (S165).

  In the unmanned aircraft 100A, the communication interface 150 receives the operation information of the unmanned aircraft 100A from the portable terminal 80A (S121).

  Subsequently, the unmanned aircraft 100A performs the processes of S107 to S111, and the portable terminal 80A performs the processes of S155 and S156.

  As described above, according to the mobile terminal 80A and the imaging assistance system 10A, it is possible to determine a composition for attractively shooting a desired subject in consideration of the desired subject. That is, the mobile terminal 80A and the imaging assistance system 10A can assist in capturing an image in consideration of improving the composition of the captured image as well as simply placing a desired subject in the captured image. Therefore, even when the user does not have sufficient know-how regarding the photography, composition determination can be assisted by the portable terminal 80A, and aerial photography of a desired subject can be assisted. In addition, since the unmanned aircraft 100A can operate in accordance with the composition (for example, movement of the unmanned aircraft 100A and adjustment of the rotation angle of the gimbal 200), the planned composition can be used in future aerial photography.

  In addition, the mobile terminal 80A can reduce the processing load of the unmanned aircraft 100A by determining the main subject related to imaging assistance, determining the composition, and generating the operation information. The unmanned aircraft 100A can be concentrated in the processing. Further, since the unmanned aircraft 100A can operate according to the operation information generated by the portable terminal 80 or the like which is another device, even if the processing load on the unmanned aircraft 100 is reduced, a desired operation for realizing a desired composition can be performed. Can be implemented.

  It should be noted that an information processing device other than the mobile terminal 80A (eg, transmitter 50, PC, other information processing device) has an imaging assistance function (eg, main subject determination function, composition determination function, operation information generation function) that the mobile terminal 80A has. You may have.

(Third embodiment)
In the first and second embodiments, assisting aerial photography with an unmanned aerial vehicle is exemplified. In 3rd Embodiment, assisting the imaging by the imaging device attached to the gimbal apparatus is illustrated. Note that in the third embodiment, the description of the same configuration and operation as those in the first and second embodiments is omitted or simplified.

  FIG. 16 is a perspective view illustrating a configuration example of an imaging assistance system 10B according to the third embodiment. The imaging assistance system 10B includes a gimbal device 300 and a portable terminal 80B. The gimbal device 300 and the portable terminal 80B can communicate with each other by wired communication (for example, USB communication) or wireless communication (for example, wireless LAN, Bluetooth (registered trademark), near field communication, public wireless line). The gimbal device 300 is an example of a support device.

  The mobile terminal 80B may determine a composition for imaging by the imaging unit 820 included in the mobile terminal 80B attached to the gimbal apparatus 300, and generate operation information of the gimbal apparatus 300 so that the determined composition is obtained. Alternatively, the gimbal device 300 determines a composition for imaging by the imaging unit 820 included in the mobile terminal 80B attached to the gimbal device 300, and generates operation information of the gimbal device 300 so that the determined composition is obtained. Good. The gimbal device 300 controls the operation of the gimbal device 300 according to the operation information. The gimbal device 300 can be possessed by a user who plans to take an image using the portable terminal 80B. The portable terminal 80B or the gimbal device 300 assists imaging by the portable terminal 80B attached to the gimbal device 300. The imaging unit 820 is an example of an imaging device.

  As shown in FIG. 16, the gimbal device 300 includes a gimbal 310, an attachment part 315, and a grip part 330. The attachment unit 315 attaches the portable terminal 80B to the gimbal device 300 and fixes the position and orientation of the portable terminal 80B with respect to the gimbal device 300.

  The gimbal 310 may support the portable terminal 80B so as to be rotatable around the yaw axis, the pitch axis, and the roll axis. The gimbal 310 may change the imaging direction of the imaging unit 820 included in the mobile terminal 80B by rotating the mobile terminal 80B around at least one of the yaw axis, the pitch axis, and the roll axis. Since the position of the imaging unit 820 in the portable terminal 80B is fixed, it can be said that the rotation of the portable terminal 80B corresponds to the rotation of the imaging unit 820.

  The gripper 330 may be gripped by the user when in use (when imaging). The gripper 330 shown in FIG. 16 is an example, and the shape of the gripper 330 different from that in FIG. 16, the position of the gripper 330 with respect to the gimbal device 300, and the size of the gripper 330 with respect to the gimbal device 300 may be used. In FIG. 16, the dotted line indicating the gimbal 310 is divided in the vicinity of the mobile terminal 80B, which indicates that the gimbal 310 is located on the back side of the mobile terminal 80B.

  Next, configuration examples of the gimbal device 300 and the portable terminal 80B will be described.

  Although not shown, the gimbal device 300 includes at least a part of the hardware configuration of the unmanned aircraft 100 or the unmanned aircraft 100A. Although not shown, the gimbal device 300 has at least a part of the functional configuration of the unmanned aircraft 100 or the unmanned aircraft 100A.

  Although not shown, the mobile terminal 80B may have the same hardware configuration as the mobile terminal 80 or the mobile terminal 80A. Although not shown, the portable terminal 80B may have the same functional configuration as the portable terminal 80 or the portable terminal 80.

  In the imaging assistance system 10B, when the portable terminal 80B has the function of the portable terminal 80 of the first embodiment, the gimbal device 300 may have the function of the unmanned aircraft 100 of the first embodiment. That is, the gimbal device 300 may have a main subject determination function, a composition determination function, and a gimbal device 300 operation information generation function in an image captured by the imaging unit 820 (for example, a live view image). Further, the gimbal device 300 may have an operation control function of the gimbal device 300 based on the operation information.

  In the imaging assistance system 10B, when the mobile terminal 80B has the function of the mobile terminal 80A of the second embodiment, the gimbal device 300 may have the function of the unmanned aircraft 100A of the second embodiment. That is, the mobile terminal 80B may have a main subject determination function, a composition determination function, and a gimbal device 300 operation information generation function in an image (for example, a live view image) captured by the imaging unit 820. The gimbal device 300 may have an operation information acquisition function of the gimbal device 300. Further, the gimbal device 300 may have an operation control function of the gimbal device 300 based on the operation information.

  Note that, unlike the unmanned aerial vehicles 100 and 100A, the gimbal device 300 is not considered to fly. Therefore, the operation information may be rotation information of the gimbal 310. Therefore, the gimbal device 300 may control the rotation of the gimbal 310 based on the operation information.

  Thus, according to the gimbal device 300, the portable terminal 80B, and the imaging assisting system 10B, a composition for attractively shooting a desired subject can be determined in consideration of the desired subject. In other words, the gimbal device 300, the portable terminal 80B, and the imaging assistance system 10B can assist in capturing an image in consideration of improving the composition of the captured image as well as simply placing a desired subject in the captured image. Therefore, even when the user does not have sufficient know-how regarding the photography, the determination of the composition can be assisted by the gimbal device 300 or the portable terminal 80B, and the imaging of a desired subject can be assisted. Further, since the gimbal device 300 can perform an operation in accordance with the composition (for example, adjustment of the rotation angle of the gimbal 310), the planned composition can be used for future imaging.

  In addition, the gimbal device 300 can quickly perform the operation of the gimbal device 300 based on the motion information by determining the main subject, determining the composition, and generating the motion information related to the imaging assistance. In addition, the gimbal device 300 can reduce the processing load of the mobile terminal 80B by determining the main subject related to imaging assistance, determining the composition, and generating the operation information, and can reduce the communication load with the mobile terminal 80B. Reduction can also be realized. Therefore, the mobile terminal 80B can contribute to processing related to imaging assistance in cooperation with the gimbal device 300 while reducing the processing load of the mobile terminal 80B itself. Also, the gimbal device 300 can use the operation information generated by the imaging assistance for the rotation control of the gimbal 310. That is, imaging assistance with imaging assistance can be implemented not only for aerial photography with the unmanned aircraft 100 as in the first and second embodiments, but also for imaging using the gimbal device 300.

  Further, the portable terminal 80B can reduce the processing load of the gimbal device 300 by determining the main subject, determining the composition, and generating the operation information related to the imaging assistance, and can reduce the processing load of the gimbal device 300 for processing the captured image. Can concentrate. In addition, since the gimbal device 300 can operate according to the operation information generated by the mobile terminal 80B, which is another device, even if the processing load on the gimbal device 300 is reduced, a desired operation for realizing a desired composition can be performed. Can be implemented.

(Fourth embodiment)
In 3rd Embodiment, assisting imaging with the imaging device attached to the gimbal apparatus was illustrated. In 4th Embodiment, assisting the imaging by the imaging part with which a gimbal apparatus is provided is illustrated. Note that in the fourth embodiment, the description of the same configuration and operation as those of the first to third embodiments is omitted or simplified.

  FIG. 17A is a front perspective view showing a configuration example of a gimbal apparatus 300C in the fourth embodiment. FIG. 17B is a rear perspective view illustrating a configuration example of an imaging assistance system 10C in the fourth embodiment. The imaging assistance system 10C includes a gimbal device 300C and a portable terminal 80C. The gimbal device 300C and the portable terminal 80C can communicate with each other by wired communication (for example, USB communication) or wireless communication (for example, wireless LAN, Bluetooth (registered trademark), near field communication, public wireless line). The gimbal device 300C is an example of a support device.

  The portable terminal 80C may determine a composition for imaging by the imaging unit 320 built in the gimbal apparatus 300C, and generate operation information of the gimbal apparatus 300C so that the determined composition is obtained. Alternatively, the gimbal device 300C may determine a composition for imaging by the imaging unit 320, and generate operation information of the gimbal device 300C so that the determined composition is obtained. The gimbal device 300C controls the operation of the gimbal device 300C according to the operation information. The gimbal device 300C can be possessed by a user who plans to take an image using the gimbal device 300C. The portable terminal 80C or the gimbal device 300C assists the imaging by the gimbal device 300C.

  As shown in FIGS. 17A and 17B, the gimbal device 300C includes a gimbal 310C, an imaging unit 320, and a gripping unit 330. The imaging unit 320 is an example of an imaging device. In the gimbal device 300C and the imaging assistance system 10C in FIGS. 17A and 17B, the same components as those in the gimbal device 300 and the imaging assistance system 10B shown in FIG. 16 are denoted by the same reference numerals, and the description thereof is omitted or simplified. To do.

  The gimbal 310C may support the imaging unit 320 rotatably around the yaw axis, the pitch axis, and the roll axis. The gimbal 310C may change the imaging direction of the imaging unit 320 by rotating the imaging unit 320 around at least one of the yaw axis, the pitch axis, and the roll axis. For example, the imaging unit 320 may capture the back direction on the paper. The imaging unit 320 may be able to change the imaging direction. The gripping part 330 may be gripped by the user's hand HD1, for example.

  Next, configuration examples of the gimbal device 300 and the portable terminal 80B will be described.

  Although not shown, the gimbal device 300C includes at least a part of the hardware configuration of the unmanned aircraft 100 or the unmanned aircraft 100A. Although not shown, the gimbal device 300C has at least a part of the functional configuration of the unmanned aircraft 100 or the unmanned aircraft 100A.

  Although not shown, the mobile terminal 80C may have the same hardware configuration as the mobile terminal 80 or the mobile terminal 80A. Although not shown, the mobile terminal 80 </ b> C may have the same functional configuration as the mobile terminal 80 or the mobile terminal 80.

  In the imaging assistance system 10C, when the mobile terminal 80C has the function of the mobile terminal 80 of the first embodiment, the gimbal device 300C may have the function of the unmanned aircraft 100 of the first embodiment. That is, the gimbal device 300C may have a main subject determination function, a composition determination function, and a gimbal device 300C operation information generation function in an image captured by the imaging unit 320 (for example, a live view image). Also, the gimbal device 300C may have a function for controlling the operation of the gimbal device 300C based on the operation information.

  In the imaging assistance system 10C, when the mobile terminal 80C has the function of the mobile terminal 80A of the second embodiment, the gimbal device 300C may have the function of the unmanned aircraft 100A of the second embodiment. That is, the mobile terminal 80C may have a main subject determination function, a composition determination function, and a gimbal device 300C operation information generation function in an image (for example, a live view image) captured by the imaging unit 320. The gimbal device 300C may have an operation information acquisition function of the gimbal device 300C. Also, the gimbal device 300C may have an operation control function of the gimbal device 300C based on the operation information.

  Unlike the unmanned aircraft 100, 100A, the gimbal device 300C is not considered to fly. Therefore, the operation information may be rotation information of the gimbal 310C. Therefore, the gimbal device 300C may control the rotation of the gimbal 310C based on the operation information.

  As described above, according to the gimbal device 300C, the portable terminal 80C, and the imaging assistance system 10C, a composition for attractively shooting a desired subject can be determined in consideration of the desired subject. In other words, the gimbal device 300C, the portable terminal 80C, and the imaging assistance system 10C can assist in capturing an image in consideration of improving the composition of the captured image as well as simply placing a desired subject in the captured image. Therefore, even when the user does not have sufficient know-how regarding the photography, the determination of the composition can be assisted by the gimbal device 300C or the portable terminal 80C, and the imaging of a desired subject can be assisted. Further, since the gimbal device 300C can perform an operation in accordance with the composition (for example, adjustment of the rotation angle of the gimbal 310C), the planned composition can be used for future imaging.

  In addition, the gimbal device 300C can quickly perform the operation of the gimbal device 300C based on the motion information by determining the main subject, determining the composition, and generating the motion information related to the imaging assistance. In addition, the gimbal device 300C can reduce the processing load of the mobile terminal 80C by determining the main subject, determining the composition, and generating the operation information related to the imaging assistance, and reduces the communication load with the mobile terminal 80C. Reduction can also be realized. Therefore, the mobile terminal 80C can contribute to processing related to imaging assistance in cooperation with the gimbal device 300C while reducing the processing load of the mobile terminal 80C itself. Also, the gimbal device 300C can use the operation information generated by the imaging assistance for the rotation control of the gimbal 310C. That is, imaging assistance with imaging assistance can be implemented not only for aerial photography with the unmanned aircraft 100 as in the first and second embodiments, but also for imaging using the gimbal device 300C.

  Further, the portable terminal 80C can reduce the processing load of the gimbal device 300C by determining the main subject, determining the composition, and generating the operation information related to the imaging assistance, and can reduce the processing load of the gimbal device 300C for processing the captured image. Can concentrate. In addition, since the gimbal device 300C can operate according to the operation information generated by the mobile terminal 80C, which is another device, even if the processing load of the gimbal device 300C is reduced, a desired operation for realizing a desired composition can be performed. Can be implemented.

  As mentioned above, although this indication was explained using an embodiment, the technical scope of this indication is not limited to the range as described in an embodiment mentioned above. It will be apparent to those skilled in the art that various modifications and improvements can be made to the embodiment described above. It is also apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present disclosure.

  The execution order of each process such as operation, procedure, step, and stage in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. ”And the like, and can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. is not.

10, 10A Imaging auxiliary system 50 Transmitter 80, 80A, 80B, 80C Portable terminal 81, 81A Terminal control unit 82 Interface unit 83 Operation unit 85 Wireless communication unit 87 Memory 88 Display unit 100, 100A Unmanned aerial vehicle 110, 110A UAV control unit 111 Image acquisition unit 112 Main subject determination unit 113 Composition determination unit 114 Motion information generation unit 115 Motion control unit 116 Motion information acquisition unit 150 Communication interface 160 Memory 200 Gimbal 210 Rotor blade mechanism 220, 230 Imaging unit 240 GPS receiver 250 Inertial measurement Device 260 Magnetic compass 270 Barometric altimeter 280 Ultrasonic sensor 290 Laser measuring device 300, 300C Gimbal device 310, 310C Gimbal 315 Mounting unit 320 Imaging unit 330 Grip unit 811 Image acquisition unit 812 Main subject determination unit 8 3 composition determining unit 814 operates the information generating unit 820 imaging unit

Claims (42)

A mobile platform that assists in capturing a second image by an imaging device,
An image acquisition unit for acquiring a first image;
Of the one or more subjects included in the first image, information on the first subject is acquired, and the positions of the one or more subjects including the first subject in the second image are defined 1 An information acquisition unit that acquires information of the first composition among the two or more compositions;
A generating unit that generates operation information related to an operation of the imaging apparatus for capturing the second image according to the first composition;
Mobile platform with The information acquisition unit selects and acquires the first subject from a plurality of subjects included in the first image;
The mobile platform according to claim 1. The information acquisition unit acquires information on the first subject based on a color component of the subject included in the first image;
The mobile platform according to claim 1 or 2. The information acquisition unit acquires information on the first subject based on a spatial frequency of the subject included in the first image;
The mobile platform according to claim 1 or 2. The information acquisition unit acquires position information of the imaging device, and acquires information of the first subject based on the position information of the imaging device;
The mobile platform according to claim 1. The information acquisition unit acquires information on the first subject based on an imaging mode at the time of imaging the second image by the imaging device.
The mobile platform according to claim 1. The information acquisition unit selects and acquires the first composition from a plurality of the compositions.
The mobile platform according to claim 1. A recognition unit for recognizing the shape of the first subject,
The information acquisition unit acquires information of the first composition based on a shape of the first subject;
The mobile platform according to any one of claims 1 to 7. A recognition unit for recognizing a scene when the second image is captured;
The information acquisition unit acquires the information of the first composition based on the scene;
The mobile platform according to any one of claims 1 to 7. The generation unit generates rotation information related to rotation of a support member that rotatably supports the imaging device as the operation information.
The mobile platform according to any one of claims 1 to 9. The generation unit determines a rotation amount and a rotation direction of the support member based on the position of the first subject in the first image and the position of the first subject in the first composition.
The mobile platform according to claim 10. The generation unit generates movement information regarding movement of the imaging device as the operation information.
The mobile platform according to any one of claims 1 to 11. The generation unit determines a movement amount of the imaging device along the direction of gravity based on the size of the first subject in the first image and the size of the first subject in the first composition. To
The mobile platform according to claim 12. The generating unit includes a position of the first subject in the first image, a position of the first subject in the first composition, a moving distance in the first image, and a moving distance in real space. Determining the moving amount and moving direction of the imaging device based on the correspondence relationship of
The mobile platform according to claim 12. A presentation unit for presenting the operation information;
The mobile platform according to any one of claims 1 to 14. The first image is an image captured by the imaging device.
The mobile platform according to any one of claims 1 to 15. The mobile platform is
A flying object including a support member that rotatably supports the imaging device and the imaging device;
A control unit that controls the flight of the flying object or the rotation of the support member based on the operation information;
The mobile platform according to claim 1, 3 to 6, or 8 to 16. The mobile platform is
A support device comprising a support member that is gripped by a user during use and rotatably supports the imaging device;
A control unit for controlling rotation of the support member based on the operation information;
The mobile platform according to claim 1, 3 to 6, or 8 to 16. The mobile platform is
Mobile device,
A communication unit that transmits the operation information to a flying object or a support device;
The mobile platform according to any one of claims 1 to 4 and 7 to 16. A flying object,
An imaging device;
A support member that rotatably supports the imaging device;
An operation information acquisition unit that acquires the operation information generated by the mobile platform according to any one of claims 1 to 4 and 7 to 16.
Based on the operation information, a control unit that controls the flight of the flying object or the rotation of the support member;
Aircraft equipped with. A support device,
A support member that rotatably supports the imaging device;
An operation information acquisition unit that acquires the operation information generated by the mobile platform according to any one of claims 1 to 4 and 7 to 16.
A control unit for controlling rotation of the support member based on the operation information;
A support device comprising: An imaging assistance method in a mobile platform for assisting imaging of a second image by an imaging device,
Obtaining a first image;
Obtaining information of a first subject among one or more subjects included in the first image;
Obtaining information of a first composition among one or more compositions defining the positions of one or more subjects including the first subject in the second image;
Generating operation information relating to an operation of the imaging device for capturing the second image according to the first composition;
An imaging assistance method comprising: The step of acquiring information on the first subject includes the step of selecting and acquiring the first subject from a plurality of subjects included in the first image.
Including steps,
The imaging assistance method according to claim 22. The step of acquiring information about the first subject includes the step of acquiring information about the first subject based on a color component of the subject included in the first image.
The imaging assistance method according to claim 22 or 23. The step of acquiring information on the first subject includes the step of acquiring information on the first subject based on a spatial frequency of the subject included in the first image.
The imaging assistance method according to claim 22 or 23. Further including the step of obtaining positional information of the imaging device,
The step of acquiring information on the first subject includes the step of acquiring information on the first subject based on position information of the imaging device.
The imaging assistance method according to claim 22. The step of acquiring information on the first subject includes the step of acquiring information on the first subject based on an imaging mode at the time of imaging the second image by the imaging device.
The imaging assistance method according to claim 22. The step of acquiring information on the first composition includes the step of selecting and acquiring the first composition from a plurality of the compositions.
The imaging assistance method according to any one of claims 22 to 27. Recognizing the shape of the first subject,
The step of acquiring information on the first composition includes the step of acquiring information on the first composition based on the shape of the first subject.
The imaging assistance method according to claim 28. Recognizing a scene when the second image is captured,
The step of acquiring information on the first composition includes the step of acquiring information on the first composition based on the scene.
The imaging assistance method according to claim 28. The step of generating the operation information includes a step of generating rotation information relating to rotation of a support member that rotatably supports the imaging device as the operation information.
The imaging assistance method according to any one of claims 22 to 30. The step of generating the movement information includes the rotation amount and the rotation direction of the support member based on the position of the first subject in the first image and the position of the first subject in the first composition. Including the step of determining
The imaging assistance method according to claim 31. The step of generating the operation information includes a step of generating movement information relating to movement of the imaging device as the operation information.
The imaging assistance method according to any one of claims 22 to 32. The step of generating the motion information is based on the size of the first subject in the first image and the size of the first subject in the first composition. Including determining the amount of movement,
The imaging assistance method according to claim 33. The step of generating the motion information includes the position of the first subject in the first image, the position of the first subject in the first composition, the moving distance in the first image, and real space. Determining the amount and direction of movement of the imaging device based on the correspondence with the movement distance in
The imaging assistance method according to claim 33. Further comprising the step of presenting the operation information to a presentation unit,
The imaging assistance method according to any one of claims 22 to 35. The first image is an image captured by the imaging device.
The imaging assistance method according to any one of claims 22 to 36. The mobile platform is a flying object including a support member that rotatably supports the imaging device and the imaging device,
Controlling the flight of the aircraft or the rotation of the support member based on the operation information;
The imaging assistance method according to any one of claims 22, 24 to 27, and 29 to 37. The mobile platform is a support device that includes a support member that is gripped by a user during use and supports the imaging device rotatably.
Further comprising controlling rotation of the support member based on the operation information.
The imaging assistance method according to any one of claims 22, 24 to 27, and 29 to 37. The mobile platform is a mobile terminal;
Further comprising transmitting the motion information to the vehicle or support device.
The imaging assistance method according to any one of claims 22 to 37. In a mobile platform that assists in capturing the second image by the imaging device,
Obtaining a first image;
Obtaining information of a first subject among one or more subjects included in the first image;
Obtaining information of a first composition among one or more compositions defining the positions of one or more subjects including the first subject in the second image;
Generating operation information relating to an operation of the imaging device for capturing the second image according to the first composition;
A program for running In a mobile platform that assists in capturing the second image by the imaging device,
Obtaining a first image;
Obtaining information of a first subject among one or more subjects included in the first image;
Obtaining information of a first composition among one or more compositions defining the positions of one or more subjects including the first subject in the second image;
Generating operation information relating to an operation of the imaging device for capturing the second image according to the first composition;
The computer-readable recording medium which recorded the program for performing this.

Images