JP2017081246A - Flight control device, flight control method, multicopter, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a remote control device which, while reducing the size and weight of the airframe of an unmanned flying object, can improve the stability in flight by using plural algorithms.SOLUTION: A multicopter 11 comprises a flight control part 31 and a power device 32. The flight control part 31 controls flight by controlling own speed and attitude, based on a picture imaged by a high resolution and high speed part reading sensor 52 and by activating the power device 32, based on an operation signal from a remote control part 12. Further, the flight control part 31 transmits a picture imaged by a high resolution and low speed imaging sensor 60 to the remote control part 12 and makes the remote control part display the picture. The power device 32, for instance, has power for plural rotors, is controlled by the flight control part 31, and makes the airframe itself of the multicopter 11 fly.SELECTED DRAWING: Figure 1

Description

  The present disclosure relates to a flight control device, a flight control method, a multicopter, and a program, and more particularly, to a flight control device, a flight control method, a multicopter, and a program that can improve the stability of flight by the multicopter.

  A technique for stabilizing the flight of an unmanned air vehicle represented by a multicopter has been developed. A multicopter is a flying object having a plurality of rotor blades, and is controlled by a remote operation by a remote operation device such as a remote controller. Some multicopters are equipped with an imaging camera such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The captured viewing image is transmitted to the remote control device by wireless communication and provided in the remote control device. Some of them are displayed on the display unit.

  As a technology related to such a multicopter, for example, an image of the ground surface is captured using a vertical camera, and a horizontal translation speed is measured using an optical flow type algorithm and a corner detector type algorithm in combination. There has been proposed a technique for controlling flight based on the information on the horizontal translation speed (see Patent Document 1).

JP 2012-006587 A

  By the way, a corner detector type algorithm generally needs to detect and track corners scattered sparsely in an image using a high-resolution image sensor.

  However, for example, when using an image obtained by an image sensor with a low resolution of about 30 × 30 pixels, if a corner detector type algorithm is used, there may be a case where no corner exists in the image.

  On the other hand, in the case of an optical flow type algorithm, a certain number of times of imaging is required per unit time, but the resolution of the image sensor and the number of times of imaging per unit time are in a trade-off relationship, and it is expensive to achieve both. .

  Therefore, when using the above algorithms together, different image sensors suitable for the algorithm should be physically used. However, unmanned air vehicles such as multicopters are required to reduce the size and weight of the entire aircraft. However, if the number of mounted image sensors or the like is physically increased, there is a possibility that the entire body is enlarged and the weight is increased.

  The present disclosure has been made in view of such a situation, and in particular, it is possible to improve stability in flight using a plurality of algorithms while reducing the size and weight of an unmanned air vehicle. It is to make.

  The flight control device according to one aspect of the present disclosure captures an image and reads out a pixel signal of a partial region of the image at high speed, and the pixel signal in the imaging unit based on attitude information of a fuselage Is a flight control device including a posture region conversion unit that determines a partial region of the image.

  An image speed conversion unit that calculates the moving speed of the aircraft based on the pixel signal read by the imaging unit, and an attitude detection unit that detects the attitude information of the aircraft based on the moving speed of the aircraft. And can be further included.

  An altitude measuring unit that measures the distance from the aircraft to the ground surface as altitude information can be further included. The image speed converting unit includes a pixel signal read by the imaging unit, and altitude information. Based on the above, the moving speed of the aircraft can be calculated.

  The altitude measurement unit uses ultrasonic waves to measure the distance from the aircraft to the ground level as altitude information, and the image speed conversion unit includes the pixel signal read by the imaging unit, and The moving speed of the airframe can be calculated based on altitude information.

  The image pickup unit may include an image plane phase difference pixel, and the altitude measurement unit may include a signal from the body to the ground surface based on a pixel signal read from the image plane phase difference pixel. The distance can be measured as altitude information, and the image speed conversion unit can calculate the moving speed of the aircraft based on the pixel signal read by the imaging unit and the altitude information. .

  A light projecting unit that projects the image capturing direction of the image capturing unit may be further included, and the light projecting unit may project the image capturing direction based on the altitude information. it can.

  When the altitude information is lower than a predetermined altitude and can sufficiently project the ground surface, the light projecting unit can project the imaging direction.

  The image speed conversion unit includes either a first type algorithm that prioritizes temporal resolution or a second type algorithm that prioritizes spatial resolution based on the pixel signal read by the imaging unit. The moving speed of the aircraft is calculated, and the moving speed of the aircraft is calculated by the first type algorithm in which the image speed converting unit prioritizes the time resolution, the posture detecting unit detects the posture detection Based on the posture information detected by the unit, the imaging unit reads out the pixel signal at high speed, determines a first region of a part of the image, and the image speed conversion unit gives priority to the spatial resolution When calculating the moving speed of the airframe by the second type algorithm, the posture region conversion unit includes a part of the image in the imaging unit, and the first region It may be so as to determine a wide second region than the region.

  When the image speed conversion unit calculates the moving speed of the aircraft with a first type of algorithm that prioritizes the time resolution, the pixel signal determined by the attitude region conversion unit is read out at high speed. A part of the first area is a normal vector from the imaging viewpoint to the imaging surface in the imaging unit and a vertical direction from the imaging viewpoint, and is an area set in accordance with an intersection with the imaging surface be able to.

  The first region that is set in a vertical direction from the imaging viewpoint and that corresponds to the intersection with the imaging surface is a circle having a predetermined radius on the imaging surface with the intersection as the center, or It can be an elliptical region.

  The first region that is set in a vertical direction from the imaging viewpoint and that corresponds to the intersection with the imaging surface is a circle having a predetermined radius on the imaging surface with the intersection as the center, or It can be a rectangular region surrounding an elliptical range.

  A transmission unit that transmits an image captured by the imaging unit, and a remote operation unit that operates the aircraft remotely and receives and displays an image transmitted from the transmission unit are further included. Can do.

  The transmission unit causes the image captured by the imaging unit to be transmitted in association with the attitude information of the aircraft when the image is captured, and the remote operation unit transmits the image from the transmission unit. Receiving the image associated with the posture information, extracting an image in a predetermined range specified based on the posture information from the image, and displaying the extracted image. it can.

  A flight control method according to an aspect of the present disclosure is a flight control method for a flight control apparatus including an imaging unit that captures an image and reads out pixel signals of a partial region of the image at high speed. The flight control method includes a step of determining a partial region of the image from which the pixel signal is read at high speed.

  A program according to one aspect of the present disclosure includes an imaging unit that captures an image and reads out pixel signals of a partial region of the image at high speed, and a part of the image that reads out the pixel signal at high speed in the imaging unit This is a program that causes a computer to function as a posture region conversion unit that determines the region.

  A multicopter according to one aspect of the present disclosure captures an image and reads out a pixel signal of a partial region of the image at a high speed, and reads out the pixel signal at a high speed in the imaging unit. A multi-copter including a posture area conversion unit that determines the area of the image.

  In one aspect of the present disclosure, an image is captured by the imaging unit, pixel signals of a partial region of the image are read at high speed, and the pixel signal is high-speed in the imaging unit by an attitude region conversion unit. A partial region of the image to be read out is determined.

  According to one aspect of the present disclosure, it is possible to improve stability in flight by using a plurality of algorithms while reducing the size and weight of the entire unmanned aircraft.

It is a figure explaining the structural example of 1st Embodiment of the multicopter and the remote control part to which this indication is applied. It is a figure explaining the structural example of the high-resolution high-speed partial readout sensor in the multicopter of FIG. It is a figure explaining the some algorithm which calculates the moving speed by the flight control part of FIG. It is a flowchart explaining the flight control process by the multicopter of FIG. 1 and a remote control part. It is a figure explaining the process in an optical flow type algorithm. It is a figure explaining the method of determining the range read at high speed in an image. It is a figure explaining the method of determining the range read at high speed in an image. It is a figure explaining the method of determining the range read at high speed in an image. It is a flowchart explaining the movement speed calculation process of FIG. It is a figure explaining the structural example of 2nd Embodiment of the multicopter and the remote control part to which this indication is applied. It is a flowchart explaining the flight control process by the multicopter of FIG. 10, and a remote control part. It is a flowchart explaining the image processing of FIG. It is a figure explaining the image processing of FIG. It is a flowchart explaining the light projection control process by the multicopter of FIG. It is a figure explaining a 1st modification. It is a figure explaining the 2nd modification. It is a figure explaining the 3rd modification. It is a figure explaining the 4th modification. And FIG. 11 is a diagram illustrating a configuration example of a general-purpose personal computer.

  Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

The description will be given in the following order.
1. 1. First embodiment 2. Second embodiment First Modified Example 4. Second modified example 5. Third modification 6. Fourth modification example 7. Application examples

<< 1. First embodiment >>
<Example of Configuration of First Embodiment of Multicopter>
FIG. 1 is a block diagram illustrating a configuration example of a first embodiment of a multicopter to which the present disclosure is applied. The multicopter 11 of FIG. 1 includes a plurality of rotor blades, communicates with the remote control unit 12, and controls flight by remote control.

  More specifically, the multicopter 11 includes a flight control unit 31 and a power unit 32. The flight control unit 31 controls its own speed and posture based on the image captured by the high-resolution high-speed partial readout sensor 52 and operates the power unit 32 based on the operation signal from the remote operation unit 12. Control the flight. In addition, the flight control unit 31 transmits an image captured by the high-resolution low-speed readout sensor 60 to the remote operation unit 12 for display.

  The power unit 32 has power of a plurality of rotor blades, for example, and is controlled by the flight control unit 31 to fly the airframe itself of the multicopter 11.

  More specifically, the flight control unit 31 includes a lens 51, a high-resolution partial high-speed readout sensor 52, an image-speed conversion unit 53, an ultrasonic distance measuring sensor 54, an inertial navigation device 55, a PID control device 56, a remote operation reception unit. 57, a posture-region conversion unit 58, a wide-angle lens 59, a high-resolution low-speed imaging sensor 60, and an image transmission unit 61. In the following, the high-resolution high-speed partial readout sensor 52 and the high-resolution low-speed imaging sensor 60 are also simply referred to as sensors 52 and 60 as necessary.

  The high-resolution partial high-speed readout sensor 52 is composed of a CMOS (Complementary Metal Oxide Semiconductor) image sensor or the like, and is an image constituted by light incident on the body of the multicopter 11 through the lens 51 from the vertical direction, that is, from the ground direction. (Ground image) is captured and supplied to the image-speed conversion unit 53. The high-resolution high-speed readout sensor 52 captures a high-resolution image based on the area information supplied from the posture-area conversion unit 58 and reads out an image of a partial partial area at high speed.

  The ultrasonic distance measuring sensor 54 emits an ultrasonic wave in the vertical direction, that is, the ground direction to the body of the multicopter 11, obtains a round-trip time of the ultrasonic wave reflected from the ground, and determines the distance to the ground as the multicopter. 11 as the flight altitude (altitude information) and output to the image-speed converter 53.

  The image-speed conversion unit 53 obtains the flight speed (movement speed) from the image supplied from the sensor 52 and the flight altitude (altitude information) supplied from the ultrasonic distance measuring sensor 54, and as speed information. In addition to being supplied to the inertial navigation device 55, altitude information is also supplied. The image-speed conversion unit 53 selects an algorithm to be used for obtaining the moving speed from one of two types of algorithms when calculating the next image according to the speed obtained based on the image. The speed is calculated by switching the algorithm for each image. At this time, the image-speed conversion unit 53 supplies the posture-region conversion unit 58 with information on the algorithm used when calculating the moving speed using the next image.

  The inertial navigation device 55 calculates the position of the multicopter 11 on the earth, the traveling direction, and its posture based on the supplied speed information, and supplies the calculated position to the PID control device 56.

  The remote operation receiving unit 57 receives an operation signal transmitted from the remote operation unit 12 by radio waves of a predetermined frequency and supplies the received operation signal to the PID control device 56.

  The posture-region conversion unit 58 obtains the velocity using the posture information which is the posture information of the multicopter 11 supplied from the inertial navigation device 55 and the next imaged image supplied from the image-speed conversion unit 53. Based on the algorithm at the time, information on the area to be read out of the image captured by the sensor 52 is obtained and supplied to the sensor 52.

  The PID (Proportional Integral Derivative) control device 56 is the position, traveling direction, and attitude information of the multicopter 11 supplied from the inertial navigation device 55, and the operation supplied from the remote operation receiving unit 57. Based on the signal, the PID control controls the flight by individually controlling the rotational speed and yaw pitch roll for the plurality of rotor blades constituting the power unit 32.

  The high-resolution low-speed imaging sensor 60 has a high-resolution, low-resolution image of a wide-angle direction that is composed of light incident through the wide-angle lens 59 in a direction that looks around the aircraft of the multicopter 11. A frame rate image is captured and supplied to the image transmission unit 61.

  The image transmission unit 61 transmits the image supplied from the sensor 60 to the remote operation unit 12 using radio waves of a predetermined frequency. That is, the image captured by the sensor 52 is an image for obtaining the moving speed supplied to the inertial navigation device 55 by the image-speed conversion unit 53, and the image captured by the sensor 60 is transmitted to the remote operation unit 12. This is an image for transmission and display as a viewing image.

  The remote operation unit 12 is a so-called remote controller of the multicopter 11, and includes an operation UI (User Interface) unit 71, a remote operation transmission unit 72, an image reception unit 73, and a display unit 74.

  The operation UI unit 71 is a user interface for remotely operating the multicopter 11. The operation UI unit 71 includes, for example, an operation lever and receives an operation input from the user and supplies a corresponding operation signal to the remote operation transmission unit 72.

  The remote operation transmission unit 72 transmits the operation signal supplied from the operation UI unit 71 to the remote operation reception unit 57 of the flight control unit 31 in the multicopter 11 by radio waves of a predetermined frequency.

  The image receiving unit 73 receives an image transmitted from the multicopter 11 by a radio wave having a predetermined frequency, and supplies the image to the display unit 74 for display.

  That is, with such a configuration, a user who operates the remote operation unit 12 can remotely operate the multicopter 11 while confirming the imaging direction and position of the multicopter 11 while viewing the image displayed on the display unit 74. It becomes.

<Configuration example of high-resolution, high-speed partial readout sensor>
Next, a detailed configuration example of the high-resolution high-speed partial readout sensor 52 will be described with reference to FIG.

  The sensor 52 includes an overall control unit 91, a pixel array 92, a vertical control unit 93, comparator columns 94-1 and 94-2, counter columns 95-1 and 95-2, horizontal control units 96-1 and 96-2, And an output control unit 97.

  The overall control unit 91 controls the overall operation of the sensor 52, and based on the region information indicating which region in the captured image from the posture-region conversion unit 58 should be read out, The signal in the necessary area is read out.

  In the pixel array 92, unit pixels are two-dimensionally arranged for m rows and n columns, row control lines are wired for each row, and column signal lines are wired for each column. Yes. Each unit pixel is provided with, for example, an RGB color filter with a Bayer array as shown in the lower left part of the figure, and a pixel signal with a corresponding wavelength is generated.

  The vertical control unit 93 is configured by a shift register or the like, and controls the row address and row scanning of the pixel array 92 via a row control line.

  The comparator columns 94-1 and 94-2 compare the signal voltage of the vertical signal line corresponding to the signal output from each unit pixel in the nth column of the pixel array 92 with the reference voltage Vref of the ramp waveform. For example, when the reference voltage Vref is larger than the signal voltage, the comparator columns 94-1 and 94-2 have the output Vco at "H" level, and when the reference voltage Vref is equal to or lower than the signal voltage Vx, the output Vco is "L". "Become level.

  The counter columns 95-1 and 95-2 are asynchronous up / down counters, which perform comparison by performing down (DOWN) counting or up (UP) counting in synchronization with a clock signal from a clock circuit (not shown). The comparison period from the start of the comparison operation in the instrument columns 94-1 and 94-2 to the end of the comparison operation is measured.

  The comparator columns 94-1 and 94-2 and the counter columns 95-1 and 95-2 receive an N-bit analog signal supplied from each unit pixel of the pixel array 92 via the column signal line for each column. Converted to a digital signal and output.

  The horizontal control units 96-1 and 96-2 are configured by shift registers and the like, and column addresses of ADCs (Analogue Digital Converters) in the comparator columns 94-1 and 94-2 and the counter columns 95-1 and 95-2. And control column scanning.

  The output control unit 97 outputs the pixel signal converted into the digital signal output from the counter columns 95-1 and 95-2.

  Note that the comparator columns 94-1 and 94-2, the counter columns 95-1 and 95-2, and the horizontal control units 96-1 and 96-2 are provided at the upper and lower portions in the figure, respectively, so as to control the entire system. Under the control of the unit 91, for example, the comparator column 94-1, the counter column 95-1, and the horizontal control unit 96-1, the comparator column 94-2, the counter column 95-2, and the horizontal control unit 96-2. By controlling the operation in two systems, two pixel signals can be output. As a result, it is possible to read out a pixel signal in a relatively wide area of the entire pixel array 92 with one system and to read out a pixel signal in a relatively narrow area of a part of the pixel array 92 with the other system. .

  The principle structure of the sensor 60 is the same as that of the sensor 52, but the reading speed is lower than the reading speed of the pixel signal of the pixel array 92. Further, the reading system of the pixel signal is one system. . However, the configuration of the sensor 60 may be the same as that of the sensor 52.

<Difference in image required for each algorithm>
Next, with reference to FIG. 3, the image difference required for each algorithm for obtaining the moving speed in the image-speed converting unit 53 will be described.

  The image-speed conversion unit 53 calculates the moving speed of the multicopter 11 based on the image supplied from the sensor 52. At this time, the image-speed conversion unit 53 calculates the movement speed by switching between the two algorithms according to the movement speed obtained from the immediately preceding image.

  More specifically, the image-velocity conversion unit 53 detects a moving speed by switching between a corner detector type algorithm and an optical flow type algorithm.

  With the corner detector type algorithm, for example, a point where edges intersect by edge detection is detected as a feature point called a corner from an image, and a moving speed is obtained by movement of feature points in a plurality of preceding and following images. In the case of this corner detector type algorithm, there is a possibility that the corner cannot be detected in the image, and therefore it is desirable that the image is as wide as possible. In other words, the corner detector type algorithm gives priority to spatial resolution. However, since an image of a wide area takes a transfer time as much as the number of pixels to be transferred increases, the transfer speed of the image is limited. As a result, if the moving speed is higher than the predetermined speed, the transfer of the image is delayed, and the corner that can be detected in the previous image cannot be detected in the current image, and the tracking error occurs. May cause the movement speed to become unsatisfactory.

  On the other hand, in the optical flow type algorithm, for example, a vector set of velocity fields is obtained for each pixel by a gradient method or a block matching method, and the moving speed of the multicopter 11 is obtained. In the case of an optical flow type algorithm, the calculation process takes time, and the calculation time increases explosively when an attempt is made to process a high-resolution image over a wide range. In addition, as the moving speed increases, a tracking error may occur unless the frame rate is increased. That is, the optical flow type algorithm gives priority to time resolution.

  Therefore, in the flight control unit 31 of the present disclosure, a corner detector type algorithm is employed as shown in the left part of FIG. A rectangular area including a relatively wide area P1 is used as a part of the imaging surface P obtained by imaging the range of the ground surface G1 obtained by imaging from the viewpoint Q1 toward the ground surface G. . By using such a wide range of images, a high-resolution but low-speed read image can be used, corner detection accuracy can be increased, and occurrence of tracking errors can be suppressed.

  On the other hand, when the moving speed is relatively high, as shown in the right part of FIG. 3, an optical flow type algorithm is employed and imaging is performed from the viewpoint Q2 of the sensor 52 toward the ground surface G. A rectangular range including a relatively narrow region P2 that is a part of the imaging surface P obtained by imaging the range of the ground surface G2 obtained in this manner is used. By using an image in such a narrow range, it is possible to use a low-resolution but high-speed read image, to increase the accuracy of optical flow, and to suppress the occurrence of tracking errors.

  Note that the low-speed reading here means that when the range of reading a wide area with high resolution is wide, the time required for reading is increased by the increase in the number of target pixels than when reading an image of a narrow area. It indicates that it is slow because it is necessary. Further, higher resolution means that the image is in a wide range, which means that the resolution is higher than that in a narrow range image and the number of target pixels is large. That is, it is assumed that the reading speed per reading unit of one pixel or one line pixel is the same in any case.

<Flight control processing by multi-copter and remote control unit in FIG. 1>
Next, a flight control process by the multicopter 11 and the remote control unit 12 of FIG. 1 will be described with reference to the flowchart of FIG.

  In step S11, the operation UI unit 71 determines whether or not it is an operation input timing. If it is the operation input timing, the process proceeds to step S12.

  In step S <b> 12, the operation UI unit 71 receives an operation input and supplies a corresponding operation signal to the remote operation transmission unit 72. The remote operation transmission unit 72 transmits the received operation signal to the multicopter 11 with a radio wave having a predetermined frequency.

  On the other hand, in step S <b> 21, the inertial navigation device 55 of the multicopter 11 updates the posture information and supplies the current posture information to the posture-region converting unit 58. In the first process, for example, attitude information of the aircraft that has landed on the ground may be supplied in a default state.

  In step S <b> 22, the posture-region conversion unit 58 generates region information indicating which region of the region imaged by the sensor 52 is read based on the current algorithm, and supplies the region information to the high-resolution / high-speed partial read sensor 52. To do.

  For example, if the current algorithm is a corner detector type algorithm, as described with reference to the left part of FIG. Information is set.

  On the other hand, if the current algorithm is an optical flow type algorithm, the area of the imaging plane P that images the range of the ground plane G2 in the range that can be imaged by the sensor 52 as shown in the right part of FIG. Area information is set so as to read the image of P2. However, in this case, it is necessary to set a region P2 that accurately includes the ground surface G2.

  That is, when the aircraft is flying in a state where the aircraft has landed on a flat ground, as shown in the right part of FIG. 5, the ground surface G and the imaging surface that image the ground surface G direction from the viewpoint Q <b> 2. Since the parallel state with P is maintained, the imaging region P1 is set substantially at the center of the imaging surface P.

  However, the multicopter 11 flies with the aircraft tilted in various directions during flight. Therefore, for example, as shown in the left part of FIG. 5, when the multicopter 11 is inclined by the angle θ with respect to the vertical direction, the normal line N of the imaging surface P makes an angle θ with respect to the vertical direction. Since the imaging surface P is not parallel to the ground surface G, the region on the imaging surface P where the ground surface G2 is imaged is a region P1 ′.

  Therefore, when the current algorithm is an optical flow type algorithm, a region P1 ′ that is an imaging region of the ground surface G2 on the imaging surface P shown in FIG. 5 is obtained, and region information corresponding to this region P1 ′ is obtained. Needs to be supplied to the sensor 52.

  Therefore, as shown in FIG. 6, the posture-region conversion unit 58 defines a normal vector [n] of the imaging plane P starting from the viewpoint Q2, and further, the posture-region conversion unit 58 is in the vertical direction starting from the viewpoint Q2. When the vector [c] whose end point is the intersection with the imaging plane P is set, the vector [c] is assumed to be k times the unit vector [g] in the vertical direction ([c] = k × [ g]).

  In the present specification, [A] represents the vector A. In the drawing, an arrow is added to A to represent the vector A in a general mathematical expression. And

  Then, the posture-region conversion unit 58 defines a vector [c] in which the inner product of the vector ([c]-[n]) and the vector [n] is 0, as shown in the upper part of FIG. The ground surface G1 is imaged in a region P1 ′ that is a circular range with a predetermined radius centering on a position (black circle in the figure) that is the intersection point of the vector [c] on the imaging surface P. I reckon. That is, the intersection indicated by the black circle in the figure is essentially the intersection between the line of sight facing the vertical direction and the imaging surface. Here, when it is easier to specify a part of the area on the sensor 52 by using the coordinates in the horizontal direction and the vertical direction, the posture-area conversion unit 58 further includes, for example, FIG. An x × y rectangular area Z surrounded by a solid line shown in the lower row may be set as area information to be read from the imaging surface P.

  When the lens 51 is a super-wide-angle lens or a fish-eye lens, the higher the image height, which is the distance between the optical axis and the image point on the image plane, the more easily distortion occurs in the imaging area on the ground surface. Therefore, as shown in FIG. 8, if it is near the center position of the imaging plane P, a circular area close to a perfect circle as shown by the area P1a is set as the imaging area on the ground surface. For example, the reading area may be set with reference to an elliptical area as indicated by the areas P1b to P1e as the area moves closer to the end of P.

  Now, the description returns to the flowchart of FIG.

  In step S <b> 23, the sensor 52 performs exposure for a predetermined time in a part of the imaging region set based on the region information supplied from the posture-region converting unit 58 among the pixel signals on the pixel array 92. . However, a part of the imaging region mentioned here includes a case where the imaging region is a relatively wide range close to the entire imaging region, as in the corner detector type algorithm.

  In step S <b> 24, the sensor 52 reads out the pixel signal in the region designated by the posture-region conversion unit 58 and supplies an image composed of the read pixel signal to the image-speed conversion unit 53. As shown in FIG. 2, the comparator columns 94-1 and 94-2, the counter columns 95-1 and 95-2, and the horizontal control units 96-1 and 96-2 each have two processes. For example, in the case of a corner detector type algorithm, the pixel signal of a comparatively wide area, which is a part of the pixel array 92, is formed in the comparator column 94-1. When the counter column 95-1 and the horizontal control unit 96-1 read out and are an optical flow type algorithm, a pixel signal of a relatively narrow area, which is a part of the pixel array 92, is compared with the comparator column 94-. 2, the reading operation may be divided so that the counter column 95-2 and the horizontal control unit 96-2 read. At this time, reading may be performed using different analog gains.

  In step S <b> 25, the ultrasonic sensor 54 measures the distance information using ultrasonic waves, and supplies the measured distance information to the image-speed conversion unit 53 as the flight altitude (altitude information).

  In step S <b> 26, the image-speed conversion unit 53 executes the movement speed calculation process, calculates the movement speed based on the image supplied from the sensor 52, and calculates the movement speed in the next process. Select the algorithm.

<Movement speed calculation process>
Here, the movement speed calculation process will be described with reference to the flowchart of FIG.

  In step S41, the image-speed conversion unit 53 determines whether or not the currently set algorithm is an optical flow type algorithm. In step S31, for example, when an optical flow type algorithm is set, the process proceeds to step S42.

  In step S <b> 42, the image-velocity conversion unit 53 calculates the moving speed based on the image supplied from the sensor 52 using an optical flow type algorithm, and supplies it to the inertial navigation device 55. That is, in this case, only the pixel signal of a specified part and a relatively narrow area is read out at a high frame rate, so that the optical flow type algorithm moves at a relatively high speed. The speed is calculated.

  In step S43, the image-speed conversion unit 53 determines whether the calculated moving speed is lower than a predetermined speed and the texture of the image is stronger than a predetermined value. In step S43, for example, if the calculated moving speed is lower than the predetermined speed or the texture of the image is strong, the process proceeds to step S44.

  In step S44, the image-speed conversion unit 53 sets an algorithm for calculating the moving speed using the next image to a corner detector type algorithm, and sets the information about the set algorithm as the posture-region conversion unit. 58. That is, when the current moving speed is not high and the pixel signal of a relatively wide area of the pixel array 92 can be read and the texture is strong and the corner can be detected relatively easily, the corner detection is performed. It is set to calculate the moving speed by a container type algorithm.

  On the other hand, if the calculated moving speed is higher than the predetermined speed in step S43 or the texture property of the image is weak, the process proceeds to step S47.

  In step S47, the image-velocity conversion unit 53 is set to calculate with the optical flow type algorithm when calculating the movement speed using the next image, and the information on the set algorithm is set to the posture-region. This is supplied to the conversion unit 58. That is, in this case, it is difficult to transfer a pixel signal with a corner detector type algorithm that is a part of the sensor 52 and requires a relatively wide range. Further, when the texture property is low, corner detection is performed. Since it is difficult to detect corners with a container type algorithm, it is an optical flow type algorithm.

  Furthermore, in step S41, when the current algorithm is not an optical flow type algorithm but a corner detector type algorithm, the process proceeds to step S45.

  In step S <b> 45, the image-velocity conversion unit 53 calculates the moving speed using a corner detector type algorithm, and supplies it to the inertial navigation device 55. That is, in this case, pixel signals in a part and a relatively wide area are read out at a low frame rate, so that a moving speed at a relatively low speed is calculated by a corner detector type algorithm. .

  In step S46, the image-speed conversion unit 53 determines whether the calculated moving speed is higher than a predetermined speed or there are few trackers. In step S46, when the calculated moving speed is either higher than the predetermined speed or the tracker is low, an image of a relatively wide area that is a part of the sensor 52 is transferred. Since it is difficult to detect corners because there are few trackers, the process proceeds to step S47, and an optical flow type algorithm is selected. The tracker is a feature point for detecting and tracking corners scattered sparsely in an image obtained in the calculation process of the corner detector type algorithm.

  On the other hand, in step S46, when the calculated moving speed is lower than the predetermined speed and there are many trackers, the process is a part of the sensor 52, and an image of a relatively wide area can be transferred. Since this is possible and corner detection is relatively easy, the process proceeds to step S44 where a corner detector type algorithm is selected.

  Through the above processing, the moving speed is calculated and supplied to the inertial navigation device 55, and an algorithm used for calculating the moving speed using the next image is selected according to the calculated moving speed. The posture-region converting unit 58 is notified.

  Now, the description returns to the flowchart of FIG.

  In step S27, the inertial navigation device 55 recalculates the position, speed, and posture of the multicopter 11 based on the speed information indicating the moving speed, and supplies the posture information to the posture-region converting unit 58. Information on the speed and attitude is supplied to the PID controller 56. The PID control device 56 operates the power device 32 in accordance with the operation signal corresponding to the operation content in the remote operation unit 12 supplied from the remote operation reception unit 57 in addition to the position, speed, and posture. Let's control the flight.

  In step S28, the high-resolution low-speed imaging sensor 60 determines whether or not it is time to start exposure of the entire pixel. For example, if it is determined that it is time to start exposure, the process proceeds to step S28. Proceed to S29.

  In step S29, the sensor 60 starts exposure of all pixels for a predetermined time. In step S28, if it is not time to start exposure, the process of step S29 is skipped.

  In step S30, the sensor 60 determines whether or not the predetermined time has elapsed since the start of exposure and the exposure has ended. If the predetermined time has not elapsed, the process proceeds to step S33.

  In step S33, the remote operation receiving unit 57 determines whether or not the end of the operation is instructed based on the operation signal from the remote operation unit 12, and if it is determined that the end is not instructed, the process is as follows. Return to step S21. If the end is instructed based on the operation signal, the process ends.

  That is, for the sensor 60, as long as there is no end instruction and the exposure time continues, the processing of steps S21 to S30 is repeated and the exposure state is continued. That is, since the sensor 60 is an image whose reading is slower than the sensor 52, the exposure and reading processes in the sensor 52 are repeated a plurality of times during one exposure and reading of the sensor 60.

  If it is determined in step S30 that the exposure of the sensor 60 is complete, the process proceeds to step S31.

  In step S <b> 31, the sensor 60 reads out pixel signals of all its own pixels and supplies them to the image transmission unit 61.

  In step S <b> 32, the image transmission unit 61 transmits an image composed of the read pixel signals to the remote operation unit 12.

  In step S <b> 13, the image receiving unit 73 determines whether an image has been transmitted from the multicopter 11. If it is determined in step S13 that an image has been transmitted, the process proceeds to step S14.

  In step S <b> 14, the image receiving unit 73 receives the transmitted image.

  In step S15, the image receiving unit 73 displays the received transmission image on the display unit 74.

  In step S16, the operation UI unit 71 determines whether or not the accepted operation input is an instruction to end, and if not, the process returns to step S11. If no operation input is accepted in step S11, the process of step S12 is skipped. If no image is transmitted in step S13, the processes of steps S14 and S15 are skipped.

  If the end is instructed in step S16, the process ends.

  With the above processing, the algorithm is switched according to the moving speed. Therefore, when the moving speed is low and corners are easy to detect, the corner detector type algorithm Using a relatively wide area that is part of the image, corners are detected appropriately and the movement speed is calculated. In addition, when the moving speed is high or corner detection is difficult, a low-resolution image of a part of a relatively narrow area where the ground surface is imaged can be quickly It was read out and the movement speed was calculated by an optical flow type algorithm.

  Accordingly, depending on the moving speed, the sensor 52 is a part of the imaging region where the ground surface is imaged, and a relatively wide region or a part of the imaging region where the ground surface is imaged. The moving speed can be detected by reading an image in a relatively narrow area and switching the algorithm. As a result, even with a plurality of algorithms, it is possible to capture images corresponding to each algorithm with the single sensor 52 by adjusting the imaging region, thereby reducing the size and weight of the aircraft. However, it is possible to improve the accuracy of flight control.

  It is also possible to continue displaying the viewing image. In the above description, an optical flow is obtained by imaging the ground surface, and at the same time, a distance from the ground surface is measured to calculate a relative speed with respect to the ground surface. However, the target is not limited to the ground surface, and for example, it is possible to obtain a relative speed with respect to a direction other than the vertical direction such as an indoor ceiling or a side wall. In addition, in detecting the moving speed, an example has been described in which an optical flow type algorithm is used as an algorithm giving priority to temporal resolution, and a corner detector type algorithm is used as an algorithm giving priority to spatial resolution. Other algorithms may be used as long as they give priority to temporal resolution and spatial resolution.

<< 2. Second embodiment >>
<Example of Configuration of Second Embodiment of Multicopter>
In the above, a configuration example has been described in which a sensor that captures an image necessary for calculating the moving speed and a sensor that captures an image necessary for viewing are individually provided. One image may be captured, and a part of the image may be used as an image for calculating the moving speed, and another part may be used for viewing.

  FIG. 10 shows a multi-copter 11 in which a single super wide-angle image is captured, a part of which is used as an image for calculating the moving speed, and another part is used for viewing. The example of a structure with the remote control part 12 is shown.

  In the multicopter 11 and the remote operation unit 12 in FIG. 10, the same reference numerals and the same names are assigned to the configurations having the same functions as those in the multicopter 11 and the remote operation unit 12 in FIG. 1. The description thereof will be omitted as appropriate.

  That is, the multicopter 11 and the remote operation unit 12 in FIG. 10 are different from the multicopter 11 and the remote operation unit 12 in FIG. Instead of the partial readout sensor 52 and the high resolution low speed readout sensor 60, a high resolution high speed partial readout image plane phase difference pixel sensor 132 is provided.

  Further, an image-altitude conversion unit 133 and an image transmission unit 135 are provided instead of the ultrasonic distance measuring sensor 54 and the image transmission unit 61, and a light projecting unit 134 is newly provided.

  Further, the remote operation unit 12 is newly provided with an image processing unit 151.

  The super-wide-angle lens 131 is a wider-angle lens than the wide-angle lens 59 and is provided at the lower part of the body of the multicopter 11. However, the imaging area covers a wide range from the vertical direction to the range close to the horizontal direction in all directions. It is a lens that can be.

  The high-resolution high-speed partial readout image plane phase difference pixel sensor 132 basically has the same function as the high-resolution high-speed partial readout pixel sensor 52, but further includes image plane phase difference pixels. The pixel signal of the phase difference pixel is supplied to the image-altitude conversion unit 133. The image-altitude conversion unit 133 obtains the distance to the ground, which is the subject in the imaging area, as altitude information by using the pixel signal of the image plane phase difference pixel, and supplies it to the inertial navigation device 55 and the light projecting unit 134. .

  For this reason, it becomes possible to obtain its own altitude by imaging the ground surface, and it is not necessary to provide the ultrasonic distance measuring sensor 54. Therefore, the multicopter 11 can be reduced in size and weight. Become. Hereinafter, the high-resolution and high-speed partial readout image plane phase difference pixel sensor 132 is also simply referred to as a sensor 132.

  The sensor 132 is imaged in combination with the super-wide-angle lens 131 to obtain an image for viewing, an image for determining a moving speed, and an altitude that have been captured by the high-resolution low-speed imaging sensor 60 so far. The image plane phase difference signals from the image plane phase difference pixels can be simultaneously imaged.

  Therefore, in FIG. 10, the stream St <b> 1 output from the sensor 132 indicates the flow of the image for obtaining the moving speed, and indicates that it is supplied to the image-speed converting unit 53. A stream St2 output from the sensor 132 indicates the flow of the viewing image and indicates that the stream is supplied to the image transmission unit 135. Furthermore, the stream St3 indicates the flow of the pixel signal of the image plane phase difference pixel and indicates that the stream is supplied to the image-altitude conversion unit 133.

  Based on the altitude information supplied from the image-altitude conversion unit 133, the light projecting unit 134 emits auxiliary light when it can sufficiently project the ground surface. The light emitted from the light projecting unit 134 may be visible light, infrared light, ultraviolet light, or the like. However, in the case of using infrared light, ultraviolet light, or the like, the sensor 132 needs to be configured to receive light with a corresponding wavelength.

  The image transmission unit 135 associates the viewing image supplied from the sensor 132 with the attitude information of the aircraft supplied from the inertial navigation device 55 at the timing of capturing the image to the remote control unit 12. Send.

  The image processing unit 151 of the remote operation unit 12 performs, for example, image processing such as extracting an image of a region in the traveling direction out of the transmitted images from the image associated with the posture information, and the display unit 74.

<Flight control processing by the flight control unit of FIG. 10>
Next, the flight control process by the multicopter 11 and the remote control unit 12 of FIG. 10 will be described with reference to the flowchart of FIG. Note that the processes of steps S51 to S54, S56, and S57 and steps S71 to S74, S76 to S81, S83, and S84 in the flowchart of FIG. 11 are the same as steps S11 to S16 and steps S21 to S24 in the flowchart of FIG. Since it is the same as the process of S26 thru | or S33, the description is abbreviate | omitted.

  That is, in step S <b> 75, the sensor 132 supplies the pixel signal of the image plane phase difference pixel among the read pixel signals to the image-altitude conversion unit 133. The image-altitude conversion unit 133 obtains the distance to the ground surface, which is the subject, as the altitude based on the pixel signal of the image plane phase difference pixel.

  That is, the altitude can be obtained using only the pixel signal imaged by the image plane phase difference pixel in the sensor 132 without using another sensor such as the ultrasonic distance measuring sensor 54.

  In step S82, the image transmission unit 135 stores the image captured by all the pixels by the processing in steps S79 and S81 and the current posture information in the inertial navigation device 55 updated in step S71 in association with each other. In step S83, the image associated with the posture information is transmitted to the remote operation unit 12.

  In response to this, in step S54, the image receiving unit 73 receives the image associated with the posture information and supplies it to the image processing unit 151.

  In step S55, the image processing unit 151 performs image processing, extracts only the image corresponding to the traveling direction from the transmitted images, and supplies the extracted image to the display unit 74.

<Image processing>
Here, the image processing in the image processing unit 151 will be described with reference to the flowchart of FIG.

  In step S101, the image processing unit 151 maps the pixel values of all the supplied pixels to a sphere map (spherical map). That is, for example, the sensor 132 is provided on the lower surface of the multicopter 11 and picks up an image incident through the super-wide-angle lens 131, so that the sensor 132 is substantially an image composed of pixels on the hemispherical surface SH shown in FIG. It will be imaged. However, since the transmitted image is transmitted as a planar image, the image processing unit 151 remaps it to the pixel (sphere map) on the hemispherical surface SH in FIG.

  In step S <b> 102, the image processing unit 151 extracts a window-shaped range constituting a predetermined area on the sphere map, for example, a pixel in a range corresponding to the traveling direction based on the posture information, and generates an image. .

  That is, since the entire image of the lower surface of the multicopter 11 is captured on the hemispherical surface SH in FIG. 13, for example, pixels in the window-shaped range W corresponding to the traveling direction are extracted based on the orientation direction. Then, a plane image is generated.

  For example, in FIG. 13, VD ((horizontal angle θ, elevation angle φ, distance VD) or (x, y, z) with respect to the center) is set as a vector corresponding to the traveling direction. A window-shaped range W corresponding to a plane in contact with the hemispherical surface SH where [VD] is a normal line is set. Here, it is assumed that the window-shaped range W is a rectangular range surrounded by the points p1, p2, p3, and p4 on the hemisphere SH, and the normal line of the plane is the vector [VD]. The image processing unit 151 extracts the pixel values of the pixels in the range W, generates a square image, and supplies the image to the display unit 74 for display as an image in the traveling direction.

  With the above processing, it is possible to always display an image in a direction corresponding to the traveling direction among the supplied omnidirectional images on the display unit 74 of the remote operation unit 12. Further, since the posture information is known, even in a direction other than the traveling direction in FIG. 13, it is possible to display appropriately by designating based on the posture information.

<Light emission control processing>
Next, the light projection control process by the multicopter 11 of FIG. 10 will be described with reference to the flowchart of FIG.

  In step S121, the image-altitude conversion unit 133 measures the distance to the ground, that is, altitude information, based on the pixel signal captured by the image plane phase pixel. In this process, the value obtained most recently by the process of step S75 described with reference to the flowchart of FIG. 11 may be used as it is.

  In step S122, the image-altitude converting unit 133 supplies the measured altitude information to the light projecting unit 134. The light projecting unit 134 determines whether or not the effect as auxiliary light can be sufficiently obtained by projecting light, which is lower than a predetermined altitude. If it is determined in step S122 that an effect as auxiliary light can be obtained, the process proceeds to step S123.

  In step S123, the light projecting unit 134 projects light as auxiliary light.

  On the other hand, when it is determined in step S122 that the effect is not higher than the predetermined altitude and the auxiliary light is not obtained, the light projecting unit 134 is turned off (not projected) in step S124.

  In step S125, the light projecting unit 134 determines whether or not the end of the operation is instructed. If the end is not instructed, the process returns to step S121. That is, until termination is instructed, it is determined whether or not an effect as auxiliary light is obtained based on altitude information supplied from the image-altitude conversion unit 133, and if there is an effect, light is projected. If so, repeat the process of turning off. Then, if it is determined in step S125 that termination has been instructed, the process ends.

  That is, by the above processing, it is possible to always project the light projecting unit 134 as auxiliary light when the altitude is sufficiently low. With such a process, when an optical flow type algorithm is used, an image captured at a high frame rate is often used. Therefore, the light projecting unit 134 projects light as auxiliary light. The accuracy of the calculated moving speed can be improved. Further, the auxiliary light projected by the light projecting unit 134 may be visible light or invisible light such as infrared light or ultraviolet light. However, when using invisible light, the sensor 132 needs to be configured to receive light of a corresponding wavelength. For example, when using infrared light or ultraviolet light, it is necessary to attach a corresponding filter. There is.

<< 3. First Modification >>
In the above, in the optical flow type algorithm, an example has been described in which the latest captured image is used to specify an imaging region necessary for calculating the moving speed using the next image. Since the multicopter 11 has moved before the image is captured, it is expected that the appropriate imaging area is shifted with the movement.

  Therefore, for example, as shown in FIG. 15, the previous image uses a circular region centered on the point c1 on the imaging surface P, and the latest image has a circular shape centered on the point c2. When an area is used, the center position at the time of the next imaging is predicted as a point c3 from these points c1 and c2, and a circular area P1 centered on this point c3 is set as the area information. May be.

  In this case, for example, the following equation (1) may be used.

・・・（１）

... (1)

  Here, the subscripts t + 1, t, t-1, t-2,..., T0 of the vector x are subscripts in the time direction, and the function f is determined from the vector x indicating the center position of the previous area. This indicates that the function predicts the next center position specified by (t + 1). Further, the right term predicts the next center position by adding the difference between the center position of the latest imaging time t and the center position of the latest imaging time t-1 to the center position of the latest imaging time t. It has been shown that you can.

<< 4. Second Modification >>
In the multicopter 11 of FIG. 10 according to the second embodiment, an image for calculating the moving speed and an image for viewing need to be captured and read by one sensor 132. For example, FIG. 16, in addition to the pixel array 171 corresponding to the pixel array 92, a memory array 172 is provided to perform high-speed partial reading, which is a part of a relatively narrow area, For example, when reading out a pixel signal in a range of 1/16 at 960 fps (stream St1 in FIG. 10) and reading out a relatively wide area at a low speed, for example, the entire range is read out at 60 fps (see FIG. 10 streams St2).

  That is, in FIG. 16, the pixel signal from the pixel array 171 is always read as a digital signal by the AD converter 171a at a high speed and stored in the memory array 172, and the adder / synthesizer circuit 172a controls the area and the read timing. When the readout area is a part and a relatively wide area, the pixel signal of the comparatively wide area is read from the memory array 172, and when it is a part and a comparatively narrow area, it is designated. The pixel signal of the partial area thus read is read out from the memory array 172.

  Further, in the case where the memory array 172 is configured with respect to the pixel array 171 as shown in FIG. 16, a stacked structure may be formed by a different process.

  In addition to this, the binned low-resolution area and the non-binned high-resolution area may be read out as a plurality of streams.

  In addition, an area for reading all channels out of the three RGB channels and an area for reading any one of RGB may be set and read as a plurality of streams.

  Furthermore, a sensor module having a hetero configuration using a relatively high resolution low speed image sensor and a relatively low resolution high speed image sensor may be used. At this time, since the low resolution side tends to have a narrow viewing angle, in order to compensate for this, for example, the imaging direction may be adjusted using a mechanical gimbal device. The gimbal device may be controlled based on the posture information obtained by the inertial navigation device 55.

<< 5. Third modification >>
In the above description, an example in which the sensor 132 is configured by a normal image sensor has been described. However, multi-streams may have various structures.

  For example, when an image for obtaining a moving speed is captured with infrared light, and an image for viewing is captured with RGB visible light, various configurations as shown in FIG. 17 may be used. FIG. 17 shows the configuration of four types of image sensors 132. The lower left part in the figure is a top view as viewed from the direction facing the pixel array, and the rest are from the upper part to the lower part in the figure. A side cross-sectional view when incident light enters is shown.

  That is, in the upper left sensor 132 in FIG. 17, the light incident through the super-wide-angle lens 131 reflects visible light in the right direction in the figure and transmits infrared light in the lower direction in the figure. A splitter 191 for separation is provided. Furthermore, an infrared light receiving layer 192 that receives the transmitted infrared light and converts it into a pixel signal is provided under the splitter 191. The pixel signal output from the infrared light receiving layer 192 is red. This is supplied to the external light high-speed partial reading unit 193. The infrared light high-speed partial reading unit 193 partially reads out the infrared light pixel signal as the stream St1. Also, a visible light receiving layer 194 that receives visible light and converts it into a pixel signal composed of RGB is provided on the right side of the splitter 191. The visible light receiving layer 194 has a pixel signal corresponding to the received light amount. Is output to the visible light reading unit 195. The visible light reading unit 195 reads visible light at a relatively low speed as the stream St2.

  In addition, in the sensor 132 at the lower left in FIG. 17, on one side of the pixel array 201 (similar to the pixel array 92), a red pixel signal is read as a stream St1 at a relatively high speed. An external light high-speed partial readout unit 203 is provided, and a visible light readout unit 202 that reads out pixel signals of visible light of all pixels as a stream St2 at a relatively low speed is provided on the opposite side surfaces. The arrangement of the infrared light high-speed partial reading unit 203 and the visible light reading unit 202 with respect to the pixel array may be any other than the lower left part of FIG. 17, and other pixel signals can be transferred as long as the pixel signal can be transferred from the pixel array 201. It may be an arrangement. In this case, the pixel array 201 is, for example, one of the G pixels in the Bayer array as an IR pixel and a pixel for receiving infrared light as shown in the figure. Furthermore, the IR pixels may be sparsely or regularly arranged on the entire imaging surface without sticking to replacing one pixel of the 2 × 2 pixel Bayer array with the IR pixel. In this case, the visible light is captured at a high resolution and the infrared light image is captured at a low resolution.

  Further, in the sensor 132 in the upper right part of FIG. 17, the blue light receiving layer 221 that receives blue light from the top in the figure in order of short wavelength of incident light that has passed through the super-wide-angle lens 131 receives green light. A green light receiving layer 222 and a red light receiving layer 223 that receives red light are provided, and an infrared light receiving layer 224 that receives infrared light is provided in the lowermost layer. Further, an infrared light high-speed partial reading unit 226 that reads the infrared light signal read from the infrared light receiving layer 224 as a stream St1 and partially reads at high speed is provided. Further, a visible light reading unit 225 that reads RGB pixel signals read as visible light from the blue light receiving layer 221, the green light receiving layer 222, and the red light receiving layer 223, respectively, as a stream St2 is provided.

  In addition, in the sensor 132 in the lower right part of FIG. 17, an infrared light absorbing organic sensor 241 that absorbs only infrared light out of incident light transmitted through the super-wide-angle lens 131 and transmits visible light is provided. An RGB light receiving semiconductor sensor 243 for receiving RGB light which is visible light is provided thereunder. In addition, on the right side of the infrared light absorbing organic sensor 241, an infrared light high-speed partial reading unit 242 that partially reads the infrared light signal read from the infrared light absorbing organic sensor 241 as a stream St1 at high speed is provided. It has been. Further, on the right side of the RGB light receiving semiconductor sensor 243, a visible light reading unit 244 that reads all RGB pixel signals read as visible light as a stream St2 is provided.

<< 6. Fourth Modification >>
In the above, the image of the entire region imaged by the sensor 132 is read out and transmitted to the remote operation unit 12 in association with the posture information. In the remote operation unit 12, an image in a specific direction is obtained based on the posture information. Although an example of generating and displaying has been described, since the image necessary for display is only a very small part of the image transmitted from the multicopter 11, the image transmitted from the multicopter 11 is transmitted in a necessary size in advance. By doing so, you may make it reduce the load which concerns on transmission.

  That is, for example, as shown in FIG. 18, only the range of the quadrilateral formed by the points p1, p2, p3, and p4 corresponding to the window-shaped range W in FIG. Then, only the range on the rectangle surrounded by the alternate long and short dash line in the drawing that roughly surrounds the range W may be read and transmitted.

  Through the processing as described above, it is possible to reduce the transmission load and also reduce the load related to the image processing for displaying on the display unit 74.

<< 7. Application example >>
<Example executed by software>
By the way, the series of processes described above can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a recording medium in a general-purpose personal computer or the like.

  FIG. 19 shows a configuration example of a general-purpose personal computer. This personal computer incorporates a CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

  The input / output interface 1005 includes an input unit 1006 including an input device such as a keyboard and a mouse for a user to input an operation command, an output unit 1007 for outputting a processing operation screen and an image of the processing result to a display device, programs, and various types. A storage unit 1008 including a hard disk drive for storing data, a LAN (Local Area Network) adapter, and the like, and a communication unit 1009 for performing communication processing via a network represented by the Internet are connected. Also, a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory), a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor A drive 1010 for reading / writing data from / to a removable medium 1011 such as a memory is connected.

  The CPU 1001 is read from a program stored in the ROM 1002 or a removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, installed in the storage unit 1008, and loaded from the storage unit 1008 to the RAM 1003. Various processes are executed according to the program. The RAM 1003 also appropriately stores data necessary for the CPU 1001 to execute various processes.

  In the computer configured as described above, the CPU 1001 loads the program stored in the storage unit 1008 to the RAM 1003 via the input / output interface 1005 and the bus 1004 and executes the program, for example. Is performed.

  The program executed by the computer (CPU 1001) can be provided by being recorded on the removable medium 1011 as a package medium, for example. The program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  In the computer, the program can be installed in the storage unit 1008 via the input / output interface 1005 by attaching the removable medium 1011 to the drive 1010. Further, the program can be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. In addition, the program can be installed in advance in the ROM 1002 or the storage unit 1008.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  In this specification, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Accordingly, a plurality of devices housed in separate housings and connected via a network and a single device housing a plurality of modules in one housing are all systems. .

  The embodiment of the present disclosure is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present disclosure.

  For example, the present disclosure can take a configuration of cloud computing in which one function is shared by a plurality of apparatuses via a network and is jointly processed.

  In addition, each step described in the above flowchart can be executed by being shared by a plurality of apparatuses in addition to being executed by one apparatus.

  Further, when a plurality of processes are included in one step, the plurality of processes included in the one step can be executed by being shared by a plurality of apparatuses in addition to being executed by one apparatus.

In addition, this indication can also take the following structures.
<1> An imaging unit that captures an image and reads pixel signals of a partial region of the image at high speed;
A flight control apparatus comprising: a posture region conversion unit that determines a partial region of the image, in which the pixel signal is read out at high speed in the imaging unit based on posture information of a body.
<2> An image speed conversion unit that calculates a moving speed of the airframe based on the pixel signal read by the imaging unit;
The flight control device according to <1>, further including an attitude detection unit that detects the attitude information of the aircraft based on the moving speed of the aircraft.
<3> An altitude measuring unit that measures the distance from the aircraft to the ground surface as altitude information,
The flight control device according to <2>, wherein the image speed conversion unit calculates the moving speed of the aircraft based on a pixel signal read by the imaging unit and altitude information.
<4> The altitude measurement unit uses ultrasonic waves to measure the distance from the aircraft to the ground surface as altitude information,
The flight control device according to <3>, wherein the image speed conversion unit calculates the moving speed of the aircraft based on the pixel signal read by the imaging unit and the altitude information.
<5> The imaging unit includes an image plane phase difference pixel,
The altitude measuring unit measures the distance from the aircraft to the ground surface as altitude information based on the pixel signal read from the image plane phase difference pixel,
The flight control device according to <3>, wherein the image speed conversion unit calculates the moving speed of the aircraft based on the pixel signal read by the imaging unit and the altitude information.
<6> Further includes a light projecting unit that projects the image capturing direction of the image capturing unit,
The flight control device according to <3>, wherein the light projecting unit projects the imaging direction based on the altitude information.
<7> The flight control device according to <6>, wherein the light projecting unit projects the imaging direction when the altitude information is lower than a predetermined altitude and can sufficiently project the ground surface.
<8> The image speed conversion unit may be a first type algorithm that prioritizes temporal resolution or a second type algorithm that prioritizes spatial resolution based on the pixel signal read by the imaging unit. Calculate the moving speed of the aircraft with either
When the image speed conversion unit calculates the moving speed of the aircraft with a first type algorithm that prioritizes the time resolution, the attitude region conversion unit is based on the attitude information detected by the attitude detection unit, In the imaging unit, a first region of a part of the image that reads the pixel signal at high speed is determined,
When the image speed conversion unit calculates the moving speed of the aircraft with a second type algorithm that prioritizes the spatial resolution, the posture region conversion unit is a part of the image in the imaging unit, The flight control device according to any one of <1> to <7>, wherein a second region wider than the first region is determined.
<9> When the image speed conversion unit calculates the moving speed of the aircraft by a first type algorithm that prioritizes the time resolution, the pixel signal determined by the attitude region conversion unit is read at high speed. The first region of a part of the image is set according to a normal vector from an imaging viewpoint to an imaging surface in the imaging unit and a vertical direction from the imaging viewpoint and an intersection with the imaging surface. The flight control device according to <8>, which is a region.
<10> The first region that is set in a vertical direction from the imaging viewpoint and that is set according to the intersection with the imaging surface is a circle having a predetermined radius on the imaging surface with the intersection as the center Or the flight control apparatus as described in <9> which is an elliptical area | region.
<11> The first region that is set in a vertical direction from the imaging viewpoint and that is set according to the intersection with the imaging surface is a circle having a predetermined radius on the imaging surface with the intersection as the center. Or the flight control apparatus as described in <10> which is a square area | region surrounding an elliptical range.
<12> a transmission unit that transmits an image captured by the imaging unit;
The flight control device according to any one of <1> to <11>, further including a remote operation unit that operates the aircraft remotely and receives and displays an image transmitted from the transmission unit.
<13> The transmission unit transmits an image captured by the imaging unit in association with posture information of the aircraft when the image is captured,
The remote control unit receives the image associated with the posture information transmitted from the transmission unit, and extracts an image in a predetermined range specified based on the posture information from the image. The flight control device according to <12>, wherein the extracted image is displayed.
<14> A flight control method of a flight control apparatus including an imaging unit that captures an image and reads out pixel signals of a partial region of the image at high speed,
The flight control method includes a step of determining a partial region of the image in which the pixel signal is read at a high speed in the imaging unit.
<15> An imaging unit that captures an image and reads out pixel signals of a partial region of the image at high speed;
A program that causes the computer to function as an attitude region conversion unit that reads out the pixel signal at a high speed and determines a partial region of the image in the imaging unit.
<16> An imaging unit that captures an image and reads pixel signals of a partial region of the image at high speed;
A multi-copter, comprising: an attitude region conversion unit that reads out the pixel signal at high speed and determines a partial region of the image.

  11 Multi-copter, 12 Remote control unit, 31 Flight control unit, 32 Power unit, 51 Lens, 52 High-resolution high-speed partial readout unit, 53 Image-speed conversion unit, 54 Ultrasonic ranging sensor, 55 Inertial navigation device, 56 PID control Device, 57 Remote operation reception unit, 58 Posture-region conversion unit, 59 Wide-angle lens, 60 High-resolution low-speed imaging sensor, 61 Image transmission unit, 71 Operation UI unit, 72 Remote operation transmission unit, 73 Image reception unit, 74 Display unit 91 control unit, 92 pixel array, 93 vertical control unit, 94-1, 94-2 comparator column, 95-1, 95-2 counter column, 96-1, 96-2 horizontal control unit, 97 output control , 131 super wide-angle lens, 132 high-resolution high-speed partial readout image plane phase difference pixel sensor, 133 image- Degree converting unit, 134 light projecting unit, 135 image transmitting section, 151 image processing unit

Claims (16)

An imaging unit that captures an image and reads out pixel signals of a partial region of the image at a high speed;
A flight control apparatus comprising: a posture region conversion unit that determines a partial region of the image, in which the pixel signal is read out at high speed in the imaging unit based on posture information of a body. An image speed conversion unit that calculates a moving speed of the airframe based on the pixel signal read by the imaging unit;
The flight control device according to claim 1, further comprising an attitude detection unit that detects the attitude information of the aircraft based on the moving speed of the aircraft. An altitude measuring unit that measures the distance from the aircraft to the ground as altitude information;
The flight control device according to claim 2, wherein the image speed conversion unit calculates the moving speed of the aircraft based on pixel signals read by the imaging unit and altitude information. The altitude measurement unit uses ultrasonic waves to measure the distance from the aircraft to the ground level as altitude information,
The flight control device according to claim 3, wherein the image speed conversion unit calculates the moving speed of the aircraft based on the pixel signal read by the imaging unit and the altitude information. The imaging unit includes an image plane phase difference pixel,
The altitude measuring unit measures the distance from the aircraft to the ground surface as altitude information based on the pixel signal read from the image plane phase difference pixel,
The flight control device according to claim 3, wherein the image speed conversion unit calculates the moving speed of the aircraft based on the pixel signal read by the imaging unit and the altitude information. A light projecting unit that projects the image capturing direction of the image capturing unit;
The flight control device according to claim 3, wherein the light projecting unit projects the imaging direction based on the altitude information. The flight control device according to claim 6, wherein the light projecting unit projects the imaging direction when the altitude information is lower than a predetermined altitude and can sufficiently project the ground surface. The image speed conversion unit is either a first type algorithm that prioritizes temporal resolution or a second type algorithm that prioritizes spatial resolution based on the pixel signal read by the imaging unit. Calculate the moving speed of the aircraft,
When the image speed conversion unit calculates the moving speed of the aircraft with a first type algorithm that prioritizes the time resolution, the attitude region conversion unit is based on the attitude information detected by the attitude detection unit, In the imaging unit, a first region of a part of the image that reads the pixel signal at high speed is determined,
When the image speed conversion unit calculates the moving speed of the aircraft with a second type algorithm that prioritizes the spatial resolution, the posture region conversion unit is a part of the image in the imaging unit, The flight control apparatus according to claim 1, wherein a second area wider than the first area is determined. When the image speed conversion unit calculates the moving speed of the aircraft with a first type of algorithm that prioritizes the time resolution, the pixel signal determined by the attitude region conversion unit is read out at high speed. The part of the first region is a region that is set in accordance with a normal vector from the imaging viewpoint to the imaging surface in the imaging unit and a vertical direction from the imaging viewpoint and an intersection with the imaging surface. The flight control apparatus according to claim 8. The first region that is set in a vertical direction from the imaging viewpoint and that corresponds to the intersection with the imaging surface is a circle having a predetermined radius on the imaging surface with the intersection as the center, or The flight control device according to claim 9, wherein the flight control device is an elliptical region. The first region that is set in a vertical direction from the imaging viewpoint and that corresponds to the intersection with the imaging surface is a circle having a predetermined radius on the imaging surface with the intersection as the center, or The flight control device according to claim 10, wherein the flight control device is a rectangular region surrounding an elliptical range. A transmission unit that transmits an image captured by the imaging unit;
The flight control device according to claim 1, further comprising: a remote control unit that operates the aircraft remotely and receives and displays an image transmitted from the transmission unit. The transmission unit transmits an image captured by the imaging unit in association with posture information of the aircraft when the image is captured,
The remote control unit receives the image associated with the posture information transmitted from the transmission unit, and extracts an image in a predetermined range specified based on the posture information from the image. The flight control apparatus according to claim 12, wherein the extracted image is displayed. A flight control method for a flight control apparatus including an imaging unit that captures an image and reads out pixel signals of a partial region of the image at high speed,
The flight control method includes a step of determining a partial region of the image in which the pixel signal is read out at a high speed in the imaging unit. An imaging unit that captures an image and reads out pixel signals of a partial region of the image at a high speed;
A program that causes the computer to function as an attitude region conversion unit that reads out the pixel signal at a high speed and determines a partial region of the image in the imaging unit. An imaging unit that captures an image and reads out pixel signals of a partial region of the image at a high speed;
A multi-copter, comprising: an attitude region conversion unit that reads out the pixel signal at high speed and determines a partial region of the image in the imaging unit.

Images