JP6803960B1 - Image processing equipment, image processing methods, programs, and recording media - Google Patents

Abstract

PROBLEM TO BE SOLVED: To sufficiently protect a camera against an impact from the outside of an unmanned aerial vehicle, and to improve the detection accuracy of an image captured by the unmanned aerial vehicle. An image processing device processes an image captured by an imaging unit included in an air vehicle surrounded by a frame body. The image processing unit of the image processing device identifies a first color in which the frame color of the frame body is small in proportion to the entire color of the scene imaged by the imaging unit in a predetermined color state, and the frame color is changed. The first image in which the scene is captured by the image pickup unit in the state of the first color is acquired, and based on the first image and the first color, automatic exposure control, automatic focus control, and automatic white balance control are performed. Calculate the detection value used for at least one of. [Selection diagram] Fig. 6

Description

The present disclosure relates to an image processing apparatus, an image processing method, a program, and a recording medium.

Conventionally, it has been studied to reduce the influence of an unmanned aerial vehicle colliding with an obstacle around it. For example, an unmanned aerial vehicle equipped with an internal frame, an internal flight propulsion system attached to the internal frame, an external frame, a gimbal system, a control system, a power supply, an external frame drive system, and a camera is known. (See Patent Document 1). The gimbal system has two rotary couplings that connect the internal flight propulsion system and the external frame.

European Patent Application Publication No. 3450310

In the unmanned aerial vehicle of Patent Document 1, the camera is arranged outside the outer frame or inside the outer frame so that the outer frame does not overlap the imaging range of the camera. In this case, the external frame does not exist in the imaging range of the camera, and the resistance to collision of obstacles with the camera becomes insufficient. Further, assuming that the entire unmanned aerial vehicle including the camera is surrounded by the external frame, the external frame overlaps the imaging range of the camera, and the image quality of the image based on the captured image may deteriorate. For example, the focal position may be determined, the brightness may be adjusted, or the white balance may be adjusted according to an unintended area in the image. This may be due to, for example, that the presence of the external frame in the imaging range causes the detection of the image by adding the external frame in addition to the desired subject. Therefore, it is desirable that the camera can be sufficiently protected against an impact from the outside of the human aircraft and that the detection accuracy of the image captured by the unmanned aerial vehicle is improved.

In one aspect, the image processing device is an image processing device that processes an image captured by an imaging unit included in an air vehicle surrounded by a frame body, and includes an image processing unit that processes the image, and the image processing unit , The first color, which is included in the entire color of the scene imaged by the image pickup unit in a state where the frame color of the frame body is a predetermined color, is small, and the frame color is imaged in the state of the first color. A first image in which a scene is captured by a unit is acquired, and detection used for at least one of automatic exposure control, automatic focus control, and automatic white balance control based on the first image and the first color. Calculate the value.

The image processing unit may acquire a second image in which the scene is captured before the first image, and specify a first color having a small proportion included in the entire color of the second image.

The image processing unit performs hue detection on the second image, calculates a hue histogram showing the number of pixels of each hue in the second image by hue detection, and sets the number of pixels to a hue equal to or less than the threshold in the hue histogram. A corresponding first color may be specified.

The second image may be an RGB image converted based on the captured image captured by the imaging unit.

The image processing device may further include a first control unit that presents information on the first color.

The frame body may have a light emitting portion capable of emitting light in a plurality of colors. A second control unit that controls the light emission of the light emitting unit and changes the color of the frame of the frame body to the first color may be further provided.

The image processing unit generates a third image obtained by extracting the high frequency component of the first image, and subtracts the first color component in the first image from the third image to generate a fourth image. Then, the detection value used for the automatic focus control may be calculated for the fourth image.

The image processing unit subtracts the first color component in the first image from the first image to generate a fifth image, and performs automatic exposure control or automatic white balance control on the fifth image. The detection value used may be calculated.

The image processing device may further include a third control unit that controls imaging by the image pickup unit based on the detection value.

The third control unit may control at least one of the lens, shutter, aperture, dimming filter, and image processing of the first image by the image processing unit based on the detection value.

The frame body may be rotatable with respect to the imaging unit.

The image processing device may be a flying object.

In one aspect, the image processing method is an image processing method for processing an image captured by an imaging unit included in an air vehicle surrounded by a frame body, and the frame color of the frame body is imaged in a predetermined color state. Acquires the step of identifying the first color in which the proportion of the entire color of the scene captured by the unit is small, and the first image in which the scene is captured by the imaging unit with the frame color being the first color. A step of calculating a detection value used for at least one of automatic exposure control, automatic focus control, and automatic white balance control based on a first image and a first color.

The steps for identifying the first color include a step of acquiring a second image in which the scene is captured before the first image, and a first color in which the proportion of the entire color of the second image is small. May include, and a step of identifying.

The steps for specifying the first color are a step of performing hue detection on the second image, a step of calculating a hue histogram indicating the number of pixels of each hue in the second image by hue detection, and a hue histogram. In, a step of specifying a first color corresponding to a hue in which the number of pixels is equal to or less than a threshold value may be included.

The second image may be an RGB image converted based on the captured image captured by the imaging unit.

The image processing method may further include a step of presenting information on the first color.

The frame body may have a light emitting portion capable of emitting light in a plurality of colors. The image processing method may further include a step of controlling the light emission of the light emitting unit and changing the color of the frame of the frame body to the first color.

The step of calculating the detection value is the step of generating a third image obtained by extracting the high frequency component of the first image, and the step of subtracting the first color component in the first image from the third image. The step of generating the image of 4 and the step of calculating the detection value used for the automatic focus control for the 4th image may be included.

The steps for calculating the detection value include a step of subtracting the first color component in the first image from the first image to generate a fifth image, and automatic exposure control or automatic exposure control for the fifth image. It may include a step of calculating the detection value used for automatic white balance control.

The image processing method may further include a step of controlling imaging by the imaging unit based on the detected value.

The step of controlling the image pickup by the image pickup unit may include a step of controlling at least one of the lens, the shutter, the aperture, the dimming filter, and the image processing for the first image of the image pickup unit based on the detection value.

The frame body may be rotatable with respect to the imaging unit.

The image processing method may be executed by the image processing apparatus. The image processing device may be a flying object.

In one aspect, the program is captured by an image processing device that processes an image captured by an image pickup unit included in an air vehicle surrounded by a frame body, and the image pickup unit captures the frame color of the frame body in a predetermined color state. A step of specifying a first color having a small proportion of the entire color of the scene, and a step of acquiring a first image in which the scene is captured by the imaging unit with the frame color being the first color. It is a program for executing a step of calculating a detection value used for at least one of automatic exposure control, automatic focus control, and automatic white balance control based on a first image and a first color. ..

In one aspect, the recording medium is imaged by an image processing device that processes an image captured by the image pickup unit included in the flying object surrounded by the frame body, and by the image pickup unit in a state where the frame color of the frame body is a predetermined color. A step of specifying a first color having a small proportion in the entire color of the scene to be performed, and a step of acquiring a first image in which the scene is captured by the imaging unit with the frame color being the first color. , A program for executing a step of calculating a detection value used for at least one of automatic exposure control, automatic focus control, and automatic white balance control based on a first image and a first color. It is a computer-readable recording medium that has been recorded.

The outline of the above invention does not list all the features of the present disclosure. Sub-combinations of these feature groups can also be inventions.

Schematic diagram showing a configuration example of an air vehicle system according to an embodiment Diagram showing an example of the concrete appearance of an unmanned aerial vehicle Block diagram showing an example of the hardware configuration of an unmanned aerial vehicle Block diagram showing a configuration example of the imaging unit Block diagram showing an example of terminal hardware configuration Diagram to outline the operation of an unmanned aerial vehicle The figure which shows the specific example of the frame color of the protection frame The figure which shows the detection example for the automatic focus control which added the protection frame. Diagram showing an example of detection for automatic exposure control or automatic white balance control with a protective frame added Flowchart showing an example of operation by an unmanned aerial vehicle

Hereinafter, the present disclosure will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Not all combinations of features described in the embodiments are essential to the solution of the invention.

The claims, description, drawings, and abstracts include matters that are subject to copyright protection. The copyright holder will not object to any person's reproduction of these documents as long as they appear in the Patent Office files or records. However, in other cases, all copyrights are reserved.

In the following embodiment, an unmanned aerial vehicle (UAV) will be illustrated as an air vehicle. The image processing device is, for example, an unmanned aerial vehicle, but may be another device (for example, a terminal, a transmitter, a server, or other image processing device). The image processing method defines the operation of the flying object. Further, the recording medium is a recording medium in which a program (for example, a program for causing an air vehicle to execute various processes) is recorded.

The “part” or “device” in the following embodiments is not limited to a physical configuration realized only by hardware, but also includes a device in which the functions of the configuration are realized by software such as a program. Further, the functions of one configuration may be realized by two or more physical configurations, or the functions of two or more configurations may be realized by, for example, one physical configuration. Further, the "acquisition" referred to in the embodiment is not limited to simply indicating an operation of directly acquiring information, signals, etc. For example, the processing unit acquires, that is, receives the information via the communication unit, and also stores the memory. Includes any acquisition from a unit (eg memory). The understanding and interpretation of these terms are the same for the description of the scope of claims.

FIG. 1 is a schematic view showing a configuration example of the flying object system 10 in the embodiment. The aircraft body system 10 includes an unmanned aerial vehicle 100 and a terminal 80. The unmanned aerial vehicle 100 and the terminal 80 can communicate with each other by wired communication or wireless communication (for example, wireless LAN (Local Area Network)). FIG. 1 illustrates that the terminal 80 is a mobile terminal (for example, a smartphone or a tablet terminal), but other terminals (for example, a PC (Personal Computer) or a control rod can be used to control the unmanned aerial vehicle 100. It may be a machine (propo).

FIG. 2 is a diagram showing an example of a specific appearance of the unmanned aerial vehicle 100. FIG. 2 shows a perspective view of the unmanned aerial vehicle 100 flying in the moving direction STV0.

As shown in FIG. 2, a roll axis (see x-axis) is set in a direction parallel to the ground and along the moving direction STV0. In this case, the pitch axis (see y-axis) is set in the direction parallel to the ground and perpendicular to the roll axis, and the yaw axis (see z-axis) is perpendicular to the ground and perpendicular to the roll axis and pitch axis. Is set.

The unmanned aerial vehicle 100 has a configuration including a UAV main body 102, a gimbal 200, an imaging unit 220, and a protective frame body 300. The protective frame body 300 is not included in the unmanned aerial vehicle 100, and may be treated as a separate body from the unmanned aerial vehicle 100. The protective frame body 300 supported by the gimbal 200 is also referred to as a gimbal ball. The gimbal 200 at least stabilizes the posture of the imaging unit 220.

The UAV main body 102 includes a plurality of rotary blades 211 (propellers). The UAV main body 102 flies the unmanned aerial vehicle 100 by controlling the rotation of the plurality of rotary blades 211. The UAV body 102 flies the unmanned aerial vehicle 100 using, for example, four rotors 211. The number of rotor blades is not limited to four. Further, the unmanned aerial vehicle 100 may be a fixed-wing aircraft having no rotor blades. The UAV main body 102 may accommodate a processor, a memory, an actuator, a battery, and the like.

The gimbal 200 includes a pitch rotation frame 201, a roll rotation frame 202, and a yaw rotation frame 203. The pitch rotation frame 201 is inserted through the connecting body 206. The UAV main body 102 and the imaging unit 220 are connected to the connecting body 206. The pitch rotation frame 201 may be connected to the roll rotation frame 202 via the connecting portion 207. The roll rotation frame 202 has a circular shape here, but is not limited to this. The roll rotation frame 202 may be connected to the yaw rotation frame 203 via the connecting portion 208. The yaw rotation frame 203 has a circular shape here, but is not limited to this. The yaw rotation frame 203 is connected to the protective frame body 300 via the connecting portion 209. The pitch rotation frame 201, the roll rotation frame 202, and the yaw rotation frame 203 may be rotated by, for example, the driving force of the actuator.

The pitch rotation frame 201 is rotatable with respect to the roll rotation frame 202 with reference to the connecting portion 207, that is, is rotatable about the pitch axis. The roll rotation frame 202 is rotatable with respect to the yaw rotation frame 203 with reference to the connecting portion 208, that is, is rotatable about the roll axis. The yaw rotation frame 203 is rotatable with respect to the protective frame body 300 with reference to the connecting portion 209, that is, is rotatable about the yaw axis. The pitch rotation frame 201 may be rotatable with respect to the connecting body 206 or may be integrally rotatable with the connecting body 206.

Further, the roll rotation frame 202 may be rotatable with respect to the pitch rotation frame 201 with reference to the connecting portion 207, that is, may be rotatable about the pitch axis. The yaw rotation frame 203 may be rotatable with respect to the roll rotation frame 202 with reference to the connecting portion 208, that is, may be rotatable about the roll axis. The protective frame body 300 may be rotatable with respect to the yaw rotation frame 203 with reference to the connecting portion 209, that is, may be rotatable about the yaw axis. The protective frame body 300 may be rotatable with respect to the yaw rotating frame 203 or may be integrally rotatable with the yaw rotating frame 203.

Therefore, the imaging unit 220 is rotatably supported by the gimbal 200. The imaging unit 220 is rotatable about at least one of a pitch axis, a roll axis, and a yaw axis as a rotation center. Similarly, the protective frame body 300 is rotatably supported by the gimbal 200. The protective frame body 300 is rotatable about at least one of a pitch axis, a roll axis, and a yaw axis as a rotation center.

The image pickup unit 220 is formed in the shape of a rectangular parallelepiped in FIG. 2, but is not limited to this. The imaging unit 220 is provided with the lens of the imaging unit 220 exposed from the side surface (for example, the front surface) of the housing of the imaging unit 220, and can image in the front direction (front). The front may be in the direction of the nose of the unmanned aerial vehicle 100, and may or may not match the moving direction STV0 in FIG. 2, for example. Since the imaging unit 220 can be rotated in any of the three axial directions by the gimbal 200, it may be possible to image in all directions in the three-dimensional space by controlling the rotation of the gimbal 200.

The protective frame body 300 surrounds the UAV main body 102, the gimbal 200, and the imaging unit 220, and is housed inside the protective frame body 300. As a result, the UAV main body 102, the gimbal 200, and the imaging unit 220 included in the protective frame body 300 have resistance to an impact from the outside of the protective frame body 300. Therefore, for example, even if an obstacle exists around the unmanned aerial vehicle 100, the posture of each part of the unmanned aerial vehicle 100 (for example, the imaging unit 220) can be maintained with high accuracy.

The protective frame body 300 is formed by a combination of a large number of protective frames 310. The protective frame body 300 is formed, for example, in the shape of a substantially polyhedron. In FIG. 2, a substantially equilateral triangle is formed by three protective frames 310, and a large number of substantially equilateral triangles are combined to form a protective frame body 300. The shape of the protective frame body 300 is not limited to this.

In this way, the protective frame body 300 may be connected to the gimbal 200 from the outside of the gimbal 200. As a result, the posture of the imaging unit 220 is stabilized, and it becomes easy to obtain an imaging range in a three-dimensional space in a stable manner.

Further, the protective frame body 300 may surround the rotary blade 211. In this case, the unmanned aerial vehicle 100 can also improve the resistance of the rotary blade 211 to an external impact. Further, the unmanned aerial vehicle 100 can prevent the unmanned aerial vehicle 100 from colliding with the obstacle and falling even when it is necessary to fly in the vicinity of the obstacle, for example.

The imaging unit 220 may include an imaging camera that captures a subject within a desired imaging range. The unmanned aerial vehicle 100 can fly in any place, and may fly in a place where there are obstacles around the unmanned aerial vehicle 100 (for example, a relatively narrow space, indoors). The subject imaged by the imaging unit 220 may include any subject, but may include an object installed indoors, an object existing at a construction site or an inspection site, and the like. The captured image captured by the imaging unit 220 may include at least one of a still image and a moving image. The moving image may include a live view image.

In FIG. 2, the gimbal 200 is a three-axis gimbal that stabilizes the posture in the three-axis directions of the pitch axis, the roll axis, and the yaw axis, but is not limited to this. For example, a biaxial gimbal that stabilizes the posture in the biaxial directions of the pitch axis and the roll axis may be used.

In FIG. 2, the lens of the imaging unit 220 is installed on the front surface of the housing, but the present invention is not limited to this. The imaging unit 220 may be installed facing any direction in the three-dimensional space, up, down, front, back, left, and right. Further, although not shown in FIG. 2, the imaging unit 220 may be rotatable with respect to the coupling 206 in at least one direction of the roll axis, the pitch axis, and the yaw axis.

Note that FIG. 2 illustrates that there is only one imaging unit 220 and that the imaging unit 220 is provided in connection with the connecting body 206, but the present invention is not limited to this. There may be a plurality of imaging units 220. For example, the imaging unit 220 may include a sensing camera that images the surroundings of the unmanned aerial vehicle 100 in order to control the flight of the unmanned aerial vehicle 100. Further, the imaging unit 220 may be provided on the side surface (for example, the front surface) or the bottom surface of the UAV main body 102. The plurality of imaging units 220 may form a pair and function as a so-called stereo camera. The imaging unit 220 may have a single focus lens or a fisheye lens.

The connector 206 may be a part of the UAV main body 102 or the imaging unit 220. Further, the imaging unit 220 may be incorporated in the UAV main body 102.

FIG. 3 is a block diagram showing an example of the hardware configuration of the unmanned aerial vehicle 100. The unmanned aerial vehicle 100 includes a UAV control unit 110, a communication unit 150, a storage unit 160, a gimbal 200, a rotary wing mechanism 210, an imaging unit 220, a GPS receiver 240, and an inertial measurement unit (IMU). The configuration includes a Unit) 250, a magnetic compass 260, a barometric altimeter 270, an ultrasonic sensor 280, a laser measuring device 290, and a protective frame body 300.

The UAV control unit 110 is configured by using a processor (for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a DSP (Digital Signal Processor)). The UAV control unit 110 performs signal processing for controlling the operation of each part of the unmanned aerial vehicle 100 in a centralized manner, data input / output processing with and from other parts, data calculation processing, and data storage processing.

The UAV control unit 110 may control the flight of the unmanned aerial vehicle 100 according to the program stored in the storage unit 160. The UAV control unit 110 may control the flight of the unmanned aerial vehicle 100 according to the flight control instruction by the terminal 80.

The UAV control unit 110 acquires position information indicating the position of the unmanned aerial vehicle 100. The UAV control unit 110 may acquire position information indicating the latitude, longitude, and altitude in which the unmanned aerial vehicle 100 exists from the GPS receiver 240. The UAV control unit 110 acquires latitude / longitude information indicating the latitude and longitude of the unmanned aerial vehicle 100 from the GPS receiver 240 and altitude information indicating the altitude at which the unmanned aerial vehicle 100 exists from the barometric altitude meter 270 as position information. Good. The UAV control unit 110 may acquire the distance between the ultrasonic wave radiation point and the ultrasonic wave reflection point by the ultrasonic sensor 280 as altitude information.

The UAV control unit 110 may acquire orientation information indicating the orientation of the unmanned aerial vehicle 100 from the magnetic compass 260. The orientation information may be indicated, for example, in the orientation corresponding to the orientation of the nose of the unmanned aerial vehicle 100.

The UAV control unit 110 may acquire position information indicating the position where the unmanned aerial vehicle 100 should exist when the imaging unit 220 images the imaging range to be imaged. The UAV control unit 110 may acquire position information indicating the position where the unmanned aerial vehicle 100 should exist from the storage unit 160. The UAV control unit 110 may acquire position information indicating the position where the unmanned aerial vehicle 100 should exist from the terminal 80 via the communication unit 150. The UAV control unit 110 may refer to the three-dimensional map database to specify the position where the unmanned aerial vehicle 100 can exist, and acquire the position as position information indicating the position where the unmanned aerial vehicle 100 should exist.

The UAV control unit 110 may acquire the imaging range of the imaging unit 220. The UAV control unit 110 may acquire the angle of view information indicating the angle of view of the imaging unit 220 from the imaging unit 220 as a parameter for specifying the imaging range. The UAV control unit 110 may acquire information indicating the imaging direction of the imaging unit 220 as a parameter for specifying the imaging range. The UAV control unit 110 may acquire posture information indicating the posture state of the imaging unit 220 from the gimbal 200, for example, as information indicating the imaging direction of the imaging unit 220. The posture information of the imaging unit 220 may indicate the rotation angle of the gimbal 200 from at least one reference rotation angle of the roll axis, pitch axis, and yaw axis.

The UAV control unit 110 may acquire position information indicating the position where the unmanned aerial vehicle 100 exists as a parameter for specifying the imaging range. The UAV control unit 110 may define an imaging range indicating a geographical range to be imaged by the imaging unit 220 based on the angle of view and the imaging direction of the imaging unit 220 and the position where the unmanned aerial vehicle 100 exists.

The UAV control unit 110 may acquire information on the imaging range from the storage unit 160. The UAV control unit 110 may acquire information on the imaging range via the communication unit 150.

The UAV control unit 110 controls the gimbal 200, the rotor blade mechanism 210, and the image pickup unit 220. The UAV control unit 110 may control the imaging range of the imaging unit 220 by changing the imaging direction or the angle of view of the imaging unit 220. The UAV control unit 110 may control the imaging range of the imaging unit 220 supported by the gimbal 200 by controlling the rotation mechanism of the gimbal 200.

The imaging range is defined by latitude, longitude, and altitude. The imaging range may be a range in 3D spatial data defined by latitude, longitude, and altitude. The imaging range may be a range in two-dimensional spatial data defined by latitude and longitude. The imaging range may be specified based on the angle of view and imaging direction of the imaging unit 220, and the position where the unmanned aerial vehicle 100 exists. The imaging direction of the imaging unit 220 may be defined from the direction in which the front surface of the imaging unit 220 provided with the imaging lens faces and the depression angle. The imaging direction of the imaging unit 220 may be a direction specified from the orientation of the nose of the unmanned aerial vehicle 100 and the attitude of the imaging unit 220 with respect to the gimbal 200.

The UAV control unit 110 may identify the environment around the unmanned aerial vehicle 100 by analyzing a plurality of images captured by the plurality of imaging units 220. The UAV control unit 110 may control the flight, for example, avoiding obstacles, based on the environment around the unmanned aerial vehicle 100. The unmanned aerial vehicle 100 may not be able to avoid obstacles. The UAV control unit 110 does not have to have an obstacle avoidance function. Even in this case, the unmanned aerial vehicle 100 can be protected even if a collision or the like occurs due to the presence of the protective frame body 300.

The UAV control unit 110 may acquire three-dimensional information (three-dimensional information) indicating the three-dimensional shape (three-dimensional shape) of an object existing around the unmanned aerial vehicle 100 as information on the environment around the unmanned aerial vehicle 100. This three-dimensional information may be used as information on the surrounding environment of the unmanned aerial vehicle 100. The object may be at least a part of an object, a building, etc. arranged indoors. The three-dimensional information is, for example, three-dimensional spatial data. The UAV control unit 110 may acquire three-dimensional information by generating three-dimensional information indicating the three-dimensional shape of an object existing around the unmanned aerial vehicle 100 from each of the captured images obtained from the plurality of imaging units 220. .. The UAV control unit 110 may acquire three-dimensional information indicating the three-dimensional shape of an object existing around the unmanned aerial vehicle 100 by referring to a three-dimensional map or a three-dimensional design drawing. The three-dimensional map and the three-dimensional design drawing may be stored in the storage unit 160, or may be stored in a server existing on the network. The three-dimensional design drawing may be used at a construction site or an inspection site.

The UAV control unit 110 controls the flight of the unmanned aerial vehicle 100 by controlling the rotary wing mechanism 210. That is, the UAV control unit 110 controls the position including the latitude, longitude, and altitude of the unmanned aerial vehicle 100 by controlling the rotary wing mechanism 210. The UAV control unit 110 may control the imaging range of the imaging unit 220 by controlling the flight of the unmanned aerial vehicle 100. The UAV control unit 110 may control the angle of view of the image pickup unit 220 by controlling the zoom lens included in the image pickup unit 220. The UAV control unit 110 may control the angle of view of the image pickup unit 220 by the digital zoom by utilizing the digital zoom function of the image pickup unit 220.

When the imaging unit 220 is fixed to the unmanned aerial vehicle 100 and the imaging unit 220 cannot be moved, the UAV control unit 110 moves the unmanned aerial vehicle 100 to a specific position at a specific date and time to obtain a desired image in a desired environment. The range may be imaged by the imaging unit 220. Alternatively, even if the imaging unit 220 does not have a zoom function and the angle of view of the imaging unit 220 cannot be changed, the UAV control unit 110 desired by moving the unmanned aerial vehicle 100 to a specific position at a specified date and time. The imaging unit 220 may image a desired imaging range in the above environment.

The UAV control unit 110 may control the orientation of the protective frame body 300 supported by the gimbal 200 by controlling the rotation mechanism of the gimbal 200. In this case, the UAV control unit 110 may rotate the protective frame body 300 with at least one of the pitch axis, the roll axis, and the yaw axis as the center of rotation. Further, the UAV control unit 110 may control the rotation direction of the protective frame body 300, the amount of rotation (rotation speed) per predetermined time, and the like. That is, for example, not only the protective frame body 300 may rotate in contact with an obstacle, but the UAV control unit 110 may voluntarily rotate the protective frame body 300. The UAV control unit 110 does not have to rotate spontaneously, and the protective frame body 300 does not have to rotate spontaneously.

The communication unit 150 communicates with another communication device (for example, the terminal 80). The communication unit 150 may perform wireless communication by any wireless communication method. The communication unit 150 may perform wired communication by any wired communication method. The communication unit 150 may transmit the captured image captured by the imaging unit 220 or an image based on the captured image to the terminal 80. The communication unit 150 may transmit the captured image and additional information (metadata) related to the captured image and the image based on the captured image to the terminal 80. The communication unit 150 may acquire information on flight control instructions from the terminal 80. The flight control instruction information may include information on the flight path and flight position for the unmanned aerial vehicle 100 to fly, and information on the imaging position for imaging by the imaging unit 220.

The storage unit 160 may hold various information, various data, various programs, and various images. The various images may include a captured image or an image based on the captured image. In the program, the UAV control unit 110 has a gimbal 200, a rotary blade mechanism 210, an imaging unit 220, a GPS receiver 240, an inertial measurement unit 250, a magnetic compass 260, a barometric altimeter 270, an ultrasonic sensor 280, and a laser measuring instrument 290, and It may include a program necessary to control the protection frame body 300. The storage unit 160 may be a computer-readable recording medium. The storage unit 160 includes a memory, and may include a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The storage unit 160 may include at least one of an HDD (Hard Disk Drive), an SSD (Solid State Drive), an SD card, a USB (Universal Serial bus) memory, and other storage. At least a portion of the storage unit 160 may be removable from the unmanned aerial vehicle 100.

The gimbal 200 may rotatably support the imaging unit 220 about the yaw axis, the pitch axis, and the roll axis. The gimbal 200 may change the imaging direction of the imaging unit 220 by rotating the imaging unit 220 around at least one of the yaw axis, the pitch axis, and the roll axis.

The gimbal 200 may rotatably support the protective frame body 300 around the yaw axis, the pitch axis, and the roll axis. The gimbal 200 may change the orientation of the protective frame body 300 by rotating the protective frame body 300 around at least one of the yaw axis, the pitch axis, and the roll axis.

The rotary blade mechanism 210 includes a plurality of rotary blades 211 and a plurality of drive motors for rotating the plurality of rotary blades 211. The rotary wing mechanism 210 flies the unmanned aerial vehicle 100 by controlling the rotation by the UAV control unit 110.

The imaging unit 220 images a subject in a desired imaging range and generates data of the captured image. The captured image and the image based on the captured image may be stored in the memory or the storage unit 160 included in the imaging unit 220.

The GPS receiver 240 receives a plurality of signals indicating the time transmitted from the plurality of navigation satellites (that is, GPS satellites) and the position (coordinates) of each GPS satellite. The GPS receiver 240 calculates the position of the GPS receiver 240 (that is, the position of the unmanned aerial vehicle 100) based on the plurality of received signals. The GPS receiver 240 outputs the position information of the unmanned aerial vehicle 100 to the UAV control unit 110. The position information of the GPS receiver 240 may be calculated by the UAV control unit 110 instead of the GPS receiver 240. In this case, information indicating the time included in the plurality of signals received by the GPS receiver 240 and the position of each GPS satellite is input to the UAV control unit 110.

The inertial measurement unit 250 detects the attitude of the unmanned aerial vehicle 100 and outputs the detection result to the UAV control unit 110. The inertial measurement unit 250 detects, as the attitude of the unmanned aerial vehicle 100, the acceleration in the three axial directions of the front-back, left-right, and up-down of the unmanned aerial vehicle 100, and the angular velocity in the three-axis directions of the pitch axis, the roll axis, and the yaw axis. You can.

The magnetic compass 260 detects the direction of the nose of the unmanned aerial vehicle 100 and outputs the detection result to the UAV control unit 110.

The barometric altimeter 270 detects the altitude at which the unmanned aerial vehicle 100 flies, and outputs the detection result to the UAV control unit 110.

The ultrasonic sensor 280 emits ultrasonic waves, detects ultrasonic waves reflected by the ground or an object, and outputs the detection result to the UAV control unit 110. The detection result may indicate the distance or altitude from the unmanned aerial vehicle 100 to the ground. The detection result may indicate the distance from the unmanned aerial vehicle 100 to the object (subject).

The laser measuring device 290 irradiates an object with laser light, receives the reflected light reflected by the object, and measures the distance between the unmanned aircraft 100 and the object (subject) by the reflected light. As an example, the distance measurement method using the laser beam may be the time-of-flight method.

FIG. 4 is a block diagram showing a configuration example of the imaging unit 220 included in the unmanned aerial vehicle 100.

The imaging unit 220 has a housing 220z. The image pickup unit 220 includes a camera processor 11, a shutter 12, an image sensor 13, an image processing unit 14, a memory 15, a shutter drive unit 19, an element drive unit 20, a gain control unit 21, and a flash 18 inside the housing 220z. Have. Further, the image pickup unit 220 has an ND filter 32, an aperture 33, a lens group 34, a lens drive unit 36, an ND drive unit 38, and an aperture drive unit 40 inside the housing 220z. It should be noted that at least a part of each configuration in the imaging unit 220 may not be provided. The camera processor 11 and the image processing unit 14 may each function by different processors, or each function may be realized by the same processor.

The camera processor 11 determines imaging conditions for imaging. The imaging conditions may include exposure time, exposure amount, focal position, and the like. The imaging conditions may be determined according to the imaging mode for capturing an image. The imaging conditions may be determined based on images captured in the past. The camera processor 11 controls each drive unit (for example, element drive unit 20, shutter drive unit 19, aperture drive unit 40, ND drive unit 38, lens drive unit 36) based on the imaging conditions, and each component unit (for example, lens drive unit 36). , The image sensor 13, the shutter 12, the aperture 33, the ND filter, and at least one of the lens group 34) may operate.

The camera processor 11 may perform automatic exposure (AE) control according to, for example, imaging conditions. In the automatic exposure control, the camera processor 11 may control at least one of the shutter 12, the aperture 33, and the ND filter 32 to adjust the exposure amount at the time of imaging. The camera processor 11 may perform automatic focus control (AF) according to, for example, imaging conditions. In the automatic focus control, the camera processor 11 may control the lens of the lens group 34, for example, to control the focal position.

In addition, the camera processor 11 determines the image conditions for the captured image. Image conditions may include white balance, etc. The image conditions may be determined according to the imaging mode for capturing the image. Image conditions may be determined based on previously captured images. The camera processor 11 may control each component (for example, the gain control unit 21 and the image processing unit 14) based on the image conditions.

The camera processor 11 may perform automatic white balance control (AWB: Automatic White Balance) in conjunction with the image processing unit 14, for example, according to image conditions.

The camera processor 11 may send an image pickup instruction to the element drive unit 20 that supplies a timing signal to the image sensor 13.

The shutter 12 is, for example, a focal plane shutter, and is driven by the shutter driving unit 19. The light incident when the shutter 12 is opened is imaged on the image pickup surface of the image pickup element 13. The image sensor 13 photoelectrically converts the optical image formed on the image pickup surface and outputs it as an image signal. A CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor: Complementary MOS) image sensor may be used as the image pickup device 13.

The gain control unit 21 reduces noise with respect to the image signal input from the image sensor 13 and controls the gain (gain) for amplifying the image signal.

The image processing unit (ISP: Image Signal Processor) 14 performs analog-to-digital conversion on the image signal amplified by the gain control unit 21 to generate image data. The image processing unit 14 may perform shading correction, color correction, contour enhancement, noise removal, gamma correction, debayer, compression, and the like. The image processing unit 14 may adjust the white balance. The image processing unit 14 may interlock with the gain control unit 21 and adjust the white balance in consideration of the gain.

Further, the image processing unit 14 may perform conversion from the RAW image to the RGB image. The image processing unit 14 may perform hue detection on a RAW image or an RGB image. The image processing unit 14 may perform 3A detection based on the result of hue detection and calculate the detection value. “3A” related to 3A detection includes automatic exposure control (AE), automatic focus control (AF), and automatic white balance control (AWB). The detection value of 3A detection includes at least one detection value used for automatic exposure control, automatic focus control, and automatic white balance control. The calculation of the detected value may include a statistical value (for example, an average value) of the detected values of a plurality of 3A detections calculated based on a plurality of images.

The memory 15 is a storage medium for storing various data, image data, and various programs. The various programs may include, for example, an AE algorithm, an AF algorithm, and an AWB algorithm.

The shutter drive unit 19 opens and closes the shutter 12 at the shutter speed instructed by the camera processor 11. The exposure amount can be adjusted by the shutter speed.

The element drive unit 20 is a timing generator, and supplies a timing signal to the image sensor 13 in accordance with an image pickup instruction from the camera processor 11, and performs a charge accumulation operation, a read operation, a reset operation, and the like of the image sensor 13.

The flash 18 illuminates the subject by flashing at the time of nighttime imaging or backlight (during backlight correction) according to the instruction of the camera processor 11. For the flash 18, for example, an LED (light Emitting Diode) light is used. The flash 18 may be omitted.

The lens group 34 collects the light from the subject and forms an image on the image sensor 13. The lens group 34 may include a focus lens, a zoom lens, an image shake correction lens, and the like. The lens group 34 is driven by the lens driving unit 36. The lens driving unit 36 has a motor (not shown), and when a control signal from the camera processor 11 is input, the lens group 34 may be moved in the direction of the optical axis up (optical axis direction). When the lens driving unit 36 performs a zooming operation for moving the zoom lens to change the zoom magnification, the lens barrel which is a part of the housing 220z and accommodates the lens group 34 may be expanded and contracted in the front-rear direction. The focal position can be adjusted by moving the lens.

The diaphragm 33 is driven by the diaphragm drive unit 40. The aperture drive unit 40 has a motor (not shown), and when a control signal from the camera processor 11 is input, the aperture of the aperture 33 is expanded or contracted. The amount of exposure can be adjusted by adjusting the opening degree of the aperture 33.

The ND filter 32 is arranged in the direction of the optical axis op (in the direction of the optical axis), for example, in the vicinity of the diaphragm 33, and performs dimming that limits the amount of incident light. The ND drive unit 38 has a motor (not shown), and when a control signal from the camera processor 11 is input, the ND filter 32 may be inserted and removed on the optical axis op. The amount of exposure can be adjusted by the amount of dimming by the ND filter 32.

FIG. 5 is a block diagram showing an example of the hardware configuration of the terminal 80. The terminal 80 includes a terminal control unit 81, an operation unit 83, a communication unit 85, a storage unit 87, and a display unit 88. The terminal 80 may be possessed by a user who desires instructions for flight control of the unmanned aerial vehicle 100.

The terminal control unit 81 is configured by using, for example, a CPU, MPU, or DSP. The terminal control unit 81 performs signal processing for controlling the operation of each unit of the terminal 80 in a centralized manner, data input / output processing with each other unit, data calculation processing, and data storage processing.

The terminal control unit 81 may acquire data and information from the unmanned aerial vehicle 100 via the communication unit 85. The terminal control unit 81 may acquire data or information input via the operation unit 83. The terminal control unit 81 may acquire the data and information stored in the storage unit 87. The terminal control unit 81 may transmit data or information to the unmanned aerial vehicle 100 via the communication unit 85. The terminal control unit 81 may send data or information to the display unit 88 and display the display information based on the data or information on the display unit 88. The information displayed on the display unit 88 and the information transmitted to the unmanned aerial vehicle 100 by the communication unit 85 include information on the flight path and flight position for the unmanned aerial vehicle 100 to fly, and information on the imaging position for imaging by the imaging unit 220. May include.

The operation unit 83 receives and acquires data and information input by the user of the terminal 80. The operation unit 83 may include input devices such as buttons, keys, a touch panel, and a microphone. The touch panel may be composed of an operation unit 83 and a display unit 88. In this case, the operation unit 83 can accept touch operations, tap operations, drag operations, and the like.

The communication unit 85 wirelessly communicates with the unmanned aerial vehicle 100 by various wireless communication methods. The wireless communication method of this wireless communication may include, for example, communication via a wireless LAN or a public wireless line. The communication unit 85 may perform wired communication by any wired communication method.

The storage unit 87 may store various information, various data, various programs, and various images. The various programs may include application programs executed on the terminal 80. The storage unit 87 may be a computer-readable recording medium. The storage unit 87 may include a ROM, a RAM, and the like. The storage unit 87 may include at least one of HDD, SSD, SD card, USB memory, and other storage. At least a part of the storage unit 87 may be removable from the terminal 80.

The storage unit 87 may hold a captured image acquired from the unmanned aerial vehicle 100 or an image based on the captured image. The storage unit 87 may hold the captured image or additional information of the image based on the captured image.

The display unit 88 is configured by using, for example, an LCD (Liquid Crystal Display), and displays various information and data output from the terminal control unit 81. The display unit 88 may display, for example, a captured image or an image based on the captured image. The display unit 88 may display various data and information related to the execution of the application program.

Next, the operation of the unmanned aerial vehicle 100 will be described.
FIG. 6 is a diagram for explaining an outline of operation by the unmanned aerial vehicle 100.

The UAV control unit 110 starts imaging by the imaging unit 220 in response to a predetermined imaging start trigger. At the start of imaging, the color of the protective frame 310 of the protective frame body 300 is the initial value. The initial value of the color of the protective frame 310 may be any color, for example, black. The imaging by the imaging unit 220 is the imaging of a plurality of images, and may be a continuous image capture or a discontinuous image capture, a moving image capture, or a still image capture. The image pickup start trigger may include receiving an image pickup start instruction from the terminal 80 via the communication unit 150, detecting that a predetermined time for starting image pickup has come, and the like. In the case of imaging a moving image, for example, 30 images (corresponding to 30 fps) or 60 images (corresponding to 60 fps) may be obtained per second.

In the unmanned aerial vehicle 100, the imaging unit 220 images a predetermined scene during flight or non-flight. The scene may indicate an imaging range, a subject, or an imaging environment. Further, the scene may be linked to the imaging mode set in the unmanned aerial vehicle 100. For example, the imaging mode may be specified via the operation unit 83 of the terminal 80, and the imaging mode information may be sent to the unmanned aerial vehicle 100 via the communication unit 85 and the communication unit 150. In the same scene, the color components included in the imaging range are similar, that is, the color components of the captured image are similar.

The image processing unit 14 receives the image signal from the image sensor 13 and acquires the RAW image G11 as the captured image. The image processing unit 14 converts the RAW image G11 into an RGB image G12. The image processing unit 14 performs hue detection on the RGB image G12, and identifies a color having a small proportion contained in the entire color of the RGB image G12, that is, a color having a proportion of a predetermined value or less, according to the result of the hue detection. That is, the image processing unit 14 identifies a small number of colors in the captured scene. For example, when a landscape scene is imaged, purple can be a specific color because the proportion of purple is small in nature.

The conversion to the RGB image G12 is not essential and may be omitted. In this case, the image processing unit 14 performs hue detection on the RAW image G11, and identifies a color having a small proportion of the entire color of the RAW image G11 according to the result of the hue detection.

The image processing unit 14 designates this specified color as the color of the protective frame 310 of the protective frame body 300. Therefore, at least one of the RAW image G11 and the RGB image G12 is an image for designating the color of the protection frame 310, and is also referred to as a frame color designation image G1.

The frame color of the protective frame 310 of the protective frame body 300 is changed to the above-mentioned specified (designated) color. For example, the frame color of the protective frame 310 is changed to purple.

In this case, protection frames 310 of various colors may be prepared in advance, and any person or device may replace the protection frames 310 with the protection frames 310 of the specified color. In this case, the UAV control unit 110 (an example of the first control unit) causes the presentation unit (for example, the display unit, the audio output unit, the vibration unit) to present the information of the specified color (an example of the first color). You can. The presentation unit may be provided by, for example, the terminal 80. In this case, the terminal control unit 81 of the terminal 80 may acquire the color information specified from the unmanned aerial vehicle 100 via the communication unit 150 and the communication unit 85 and have the presentation unit present the information. As a result, the color of the protective frame 310 of the 300 protective frames to be changed can be confirmed by the user of the terminal 80, and the protective frame 310 can be easily replaced with the protective frame 310 of the color to be changed. Further, when the protection frame 310 is replaced, the unmanned aerial vehicle 100 may be flight-controlled so as to return to the position of the user who possesses the terminal 80, for example. After replacing the protection frame 310, the unmanned aerial vehicle 100 may resume imaging the scene to be imaged.

Further, the protective frame body 300 may have a light emitting portion capable of emitting light in a plurality of colors. The light emitting unit may be configured to include, for example, an LED. The light emitting unit may be arranged at any position on the protective frame body 300, and may irradiate any position on the protective frame body 300 to emit light. Further, the light emitting unit is embedded in the LED in each protection frame 310, and the position of the embedded protection frame 310 may emit light when the LED emits light. The UAV control unit 110 (an example of the second control unit) may control the light emission of the light emitting unit and change the color of the protection frame 310 of the protection frame body 300 to the above-specified color.

As a result, the unmanned aerial vehicle 100 can automatically and easily change the color of the protective frame 310 of the protective frame body 300 to the specified color. Further, unlike the manual replacement of the protection frame 310, it is not necessary for the unmanned aerial vehicle 100 to move to the replacement location of the protection frame 310 (for example, to return to the user's position), so that the protection frame 310 is flying while the unmanned aerial vehicle 100 is flying. You can change the color of. Therefore, the unmanned aerial vehicle 100 can flexibly change the color of the protective frame 310 according to the scene while continuously capturing images like a moving image.

In the unmanned aerial vehicle 100, the imaging unit 220 captures a predetermined scene during flight or non-flight with the frame color of the protection frame 310 changed. This scene is the same as the scene already captured after the start of imaging (the previously captured scene). Therefore, an image of a scene with a color distribution similar to that previously captured is captured.

The image processing unit 14 receives the image signal from the image sensor 13 and acquires the RAW image G21 as the captured image. The image processing unit 14 performs 3A detection on the RAW image G21. The RAW image G21 is an image for performing 3A detection, and is also referred to as a 3A detection image G2. In this case, the image processing unit 14 masks the color (for example, purple) region of the protection frame 310 in the image based on the RAW image G21 and excludes it from the target of 3A detection. The masked region (masked region MR) is not added to the 3A detection and is not affected by the 3A detection. The mask region MR has the number and shape of pixels occupied by the protection frame 310 in the image based on the RAW image G21. The image processing unit 14 performs 3A detection on the image based on the RAW image G21 in which a part of the region is masked, and calculates the detection value.

The image processing unit 14 converts the RAW image G21 into the RGB image G22 based on the 3A detection result (that is, the detection value) of the RAW image G21. The RGB image G22 may be the target of subsequent image processing, or may be displayed by the display unit 88 or the like of the terminal 80.

Further, the camera processor 11 controls the imaging by the imaging unit 220 based on the detection value of the 3A detection of the RAW image G21. For example, the camera processor 11 may control one or more lenses included in the lens group 34 and perform autofocus control based on the detection value used for the autofocus control. The camera processor 11 may control at least one of the shutter 12 (for example, shutter speed), the ND filter 32, and the aperture 33, and perform automatic exposure control based on the detection value used for the automatic exposure control. The camera processor 11 may control the image processing unit 14 and perform automatic white balance control based on the detection value used for the automatic white balance control. In the control of the lens group 34, the shutter 12, the ND filter 32, and the aperture 33, the driving unit for driving them may be controlled by the camera processor 11.

As described above, the unmanned aerial vehicle 100 (an example of an image processing device) captures an image captured by an imaging unit 220 included in the unmanned aerial vehicle 100 (an example of an air vehicle) surrounded by a protective frame body 300 (an example of a frame body). To process. The unmanned aerial vehicle 100 includes an image processing unit 14 that processes an image. The image processing unit 14 may specify a first color in which the color of the protective frame 310 of the protective frame body 300 is included in the entire color of the scene imaged by the image capturing unit 220 in a predetermined color state. The image processing unit 14 may acquire a 3A detection image G2 (an example of the first image) in which the scene is captured by the image capturing unit 220 with the color of the protection frame 310 being the first color. The image processing unit 14 may calculate the detection value used for the 3A control based on the 3A detection image G2 and the first color. The 3A control includes at least one of automatic exposure control (AF control), automatic focus control (AE control), and automatic white balance control (AWB control).

As a result, the unmanned aerial vehicle 100 captures the color of the protection frame 310 as a first color that is not included in this scene even when a part of the protection frame body 300 is located in the scene imaged by the imaging unit 220. it can. Therefore, the unmanned aerial vehicle 100 can derive a detection value excluding the influence of the protective frame body 300, which is not a desired subject, by calculating the detection value used for 3A detection in consideration of the first color. Therefore, the unmanned aerial vehicle 100 can suppress focusing on an unintended part of the imaging range (for example, a region where the protection frame 310 is located) and adjusting the brightness and white balance with reference to this region.

In addition, the unmanned aerial vehicle 100 can improve the resistance to an external impact on the image pickup unit 220 by surrounding the image pickup unit 220 with the protective frame body 300. Further, since the unmanned aerial vehicle 100 can obtain the detection value of 3A detection with high accuracy, the image pickup control by the image pickup unit 220 can be performed with high accuracy, and it is possible to suppress that the inspection work is hindered. As described above, the imaging unit 220 can be sufficiently protected against an impact from the outside of the unmanned aerial vehicle 100, the detection accuracy of the image based on the captured image can be improved, and the deterioration of the image quality can be suppressed.

Further, the camera processor 11 (an example of the third control unit) may control the image pickup by the image pickup unit 220 based on the detection value of the 3A detection. As a result, the unmanned aerial vehicle 100 uses the detection value excluding the influence of the protection frame 310 to focus on an unintended part of the imaging range of the imaging unit 220, or to use this region as a reference for brightness and white balance. It is possible to suppress the adjustment of.

Further, the camera processor 11 is based on the detection value of 3A detection, and the lens of the lens group 34 of the image pickup unit 220, the shutter 12, the aperture 33, the ND filter 32 (an example of the dimming filter), and the 3A by the image processing unit 14. At least one of the image processing for the detection image G2 may be controlled. As a result, the unmanned aerial vehicle 100 can adjust the exposure amount at the time of imaging, adjust the focal position, and adjust the white balance by using each unit included in the imaging unit 220.

Further, the unmanned aerial vehicle 100 may mainly perform the above image processing and imaging control. In this case, the derivation of the detection value of the 3A detection and the imaging control based on the detection value can be performed by one device, efficient processing can be performed, and the processing time can be shortened. Further, it is not necessary to prepare a device for performing these processes separately from the unmanned aerial vehicle 100. Other devices (for example, terminal 80, transmitter) may mainly perform the above image processing and image pickup control.

Further, in the unmanned aerial vehicle 100, the protective frame body 300 may be rotatable with respect to the imaging unit 220. In this case, the position of the protection frame 310 changes in the imaging range of the imaging unit 220. Even in this case, the unmanned aerial vehicle 100 can derive a highly accurate detection value of 3A detection by excluding the influence of the movable protection frame 310. Therefore, the unmanned aerial vehicle 100 can perform 3A control using the detection value of highly accurate 3A detection. Therefore, in the unmanned aerial vehicle 100, the control quality of the 3A control can be kept constant even if the protection frame 310 moves, and the image quality of the obtained image can be kept constant. For example, the unmanned aerial vehicle 100 can suppress a sudden change in the focal position, the amount of exposure, and the white balance due to the rotation of the protective frame 310.

FIG. 7 is a diagram showing a specific example of the frame color of the protection frame 310.

The image processing unit 14 performs hue detection on the RGB image G12 acquired as the frame color designation image G1, and identifies a color in which the proportion of the entire color of the RGB image G12 is small according to the result of the hue detection. Processing (also called frame color identification processing) is performed. This identified color is designated as the frame color of the protective frame 310. Although it is illustrated here that the frame color designation image G1 is an RGB image G12, it may be a RAW image G11.

In the frame color specifying process, the image processing unit 14 converts the RGB value of each pixel of the RGB image G12 into an H (HUE) value (hue value). That is, the image processing unit 14 converts the RGB value of each pixel of the RGB image G12 into an H value (hue value). In this case, the image processing unit 14 may calculate the H value based on the RGB value by using (Equation 1).

That is, when the R value, the G value, and the B value are the same, the image processing unit 14 is shown as in the first line of (Equation 1), and the H value is Undefined. When the B value is the minimum value among the R value, the G value, and the B value, the H value is shown as in the second line of (Equation 1). When the R value is the minimum value among the R value, the G value, and the B value, the H value is shown as shown in the third line of (Equation 1). When the G value is the minimum value among the R value, the G value, and the B value, the H value is shown as shown in the fourth line of (Equation 1).

The image processing unit 14 may calculate the value of the HSV color space from the value of the RGB color space of the RGB image G12. That is, the image processing unit 14 may calculate not only the H value but also the S (Saturation) value (saturation value) and the V (Value) value (brightness value). Therefore, the color specified by the H value, the S value, and the V value is distributed at any position in the HSV color space indicated by the cylinder in FIG. 7.

The image processing unit 14 accumulates the calculated H values and generates a hue histogram HG which is a histogram of the H values. The hue histogram HG indicates the number of pixels of each hue in the RGB image G12 represented by the HSV color component. In the hue histogram HG, the horizontal axis indicates the H value (Hue), and the vertical axis indicates the number of pixels (Cnt) of the pixel having the H value. The hue histogram HG makes it possible to recognize which hue has a large number of pixels and which hue has a small number of pixels.

The image processing unit 14 refers to the hue histogram HG and identifies the hue whose number of pixels is the threshold th or less. The number of pixels of the threshold value th or less may include that the predetermined number of pixels is equal to or less than the threshold value th regardless of the total number of pixels of the RGB image G12, and is predetermined with respect to the total number of pixels of the RGB image G12. It may include that the ratio of hue is equal to or less than the threshold value th. Therefore, a color (hue) having an absolute small number of pixels in the RGB image G12 and a color having a relatively small number of pixels in the RGB image G12 are specified. That is, few colors are specified in the scene where the image is captured.

The image processing unit 14 specifies, for example, the hue (H_frame) having the minimum number of pixels, as shown in the hue histogram HG2. The image processing unit 14 designates this specified color as a new color of the protective frame 310 of the protective frame body 300.

In this way, the image processing unit 14 may acquire the frame color designation image G1 (an example of the second image) in which the same scene as the 3A detection image G2 is captured before the 3A detection image G2. .. The image processing unit 14 may specify a first color (for example, H_frame) having a small proportion included in the entire color of the frame color designation image G1.

As a result, the unmanned aerial vehicle 100 simply adds a simple operation of acquiring the frame color designation image G1 before the acquisition of the 3A detection image G2 by the imaging unit 220, and obtains a new frame color that is not in the scene. Easy to identify. Further, when the unmanned aerial vehicle 100 obtains a continuous image sequence such as a moving image by the imaging unit 220, the unmanned aerial vehicle 100 uses different image frames in the series of image sequences to specify the color of the protection frame 310 and the protection frame. It is possible to carry out 3A detection in which the influence of 310 is suppressed. The image sequence has a plurality of image frames arranged in chronological order.

Further, the image processing unit 14 may perform hue detection on the frame color designation image G1. The image processing unit 14 may calculate a hue histogram HG indicating the number of pixels of each hue in the frame color designation image G1 by hue detection. The image processing unit 14 may specify the first color corresponding to the hue whose number of pixels is the threshold value th or less in the hue histogram HG.

As a result, the unmanned aerial vehicle 100 can specify a small number of colors in the imaging scene by performing a hue detection and generating a hue histogram HG, and can designate the specified color as a new color of the protection frame 310.

Further, the frame color designation image G1 may be an RGB image G12 converted based on the RAW image G11 captured by the imaging unit 220. As a result, it is possible to identify a color having a small amount of color components contained in the imaging scene with higher accuracy than using the RAW image G11. Further, the conversion from RGB to HSV is highly versatile and easy to carry out. The RAW image G11 may be used as the frame color designation image G1.

Next, 3A detection including the protective frame body 300 will be described.
FIG. 8 is a diagram showing an example of detection for automatic focus control in which the protection frame 310 is added.

The detection for automatic focus control is performed using the image G2 for 3A detection. The 3A detection image G2 is a RAW image G21.

The image processing unit 14 extracts high frequency components from the RAW image G21 and generates an edge image G3. That is, the image processing unit 14 passes the RAW image G21 through the HPF (High Pass Filter) to substantially not attenuate the frequency components higher than the cutoff frequency and reduce the frequency components lower than the cutoff frequency. In the edge image G3, high frequency components (that is, edges, contours) are emphasized. In FIG. 8, in the edge image G3, 〇, Δ, □ as a subject and the outline of the protective frame 310 are shown.

The image processing unit 14 extracts the pixels of the protection frame 310 from each pixel of the RAW image G21. In this case, the image processing unit 14 extracts the color pixels of the protection frame 310 when the RAW image G21 is captured. The area of the color pixels of the protection frame 310 is the mask area MR.

The image processing unit 14 subtracts the mask region MR from the edge image G3 to generate the AF detection image G4 excluding the mask region MR. In FIG. 8, in the AF detection image G4, the contours of 〇, Δ, and □ as the subject are shown. The image processing unit 14 performs detection (AF detection) for performing automatic focus control on the AF detection image G4, and calculates the detection value of the AF detection. That is, in AF detection, high frequency components are detected to facilitate focusing.

In this way, the image processing unit 14 may generate an edge image G3 (an example of a third image) from which the high frequency component of the 3A detection image G2 is extracted. The image processing unit 14 subtracts the component of the color (an example of the first color) of the changed protection frame 310 in the 3A detection image G2 from the edge image G3, and subtracts the AF detection image G4 (fourth image). An example) may be generated. The image processing unit 14 may calculate the detection value used for the automatic focus control for the AF detection image G4.

In the extraction of the high frequency component for AF detection, since the edge is conspicuous, a linear member such as the protection frame 310 is likely to be extracted, which tends to affect the detection value of AF detection. On the other hand, the unmanned aerial vehicle 100 can suppress the influence of the protective frame body 300 on the high frequency component (edge portion of the image) of the 3A detection image G2 based on the color, and can improve the accuracy of the automatic focus control.

FIG. 9 is a diagram showing an example of detection for automatic exposure control or automatic white balance control in which the protection frame 310 is added.

The detection for automatic exposure control or automatic white balance control is performed using the 3A detection image G2. The 3A detection image G2 is a RAW image G21.

The image processing unit 14 extracts the pixels of the protection frame 310 from each pixel of the RAW image G21. In this case, the image processing unit 14 extracts the color pixels of the protection frame 310 when the RAW image G21 is captured. The area of the color pixels of the protection frame 310 is the mask area MR.

The image processing unit 14 subtracts the mask region MR from the RAW image G21 to generate the AE / AWB detection image G5 excluding the mask region MR. In FIG. 9, in the AE / AWB detection image G5, 〇, Δ, and □ as subjects are shown. The image processing unit 14 performs detection (AE detection) for performing automatic exposure control on the AE / AWB detection image G5, and calculates the detection value of the AE detection. The image processing unit 14 performs detection (AWB detection) for performing automatic white balance control on the AE / AWB detection image G5, and calculates the detection value of the AWB detection. AE / AWB detection indicates at least one of AE detection and AWB detection.

In this way, the image processing unit 14 subtracts the component of the color (an example of the first color) of the changed protection frame 310 in the 3A detection image G2 from the 3A detection image G2 for AE / AWB detection. Image G5 (an example of a fifth image) may be generated. The image processing unit 14 may calculate the detection value used for the automatic exposure control or the automatic white balance control for the AE / AWB detection image.

As a result, the unmanned aerial vehicle 100 can suppress the influence of the protective frame body 300 on the 3A detection image G2 based on the color, and can improve the accuracy of at least one of the automatic exposure control and the automatic white balance control.

FIG. 10 is a flowchart showing an operation example by the unmanned aerial vehicle 100.

The image processing unit 14 acquires a RAW image G11 in which a predetermined scene is captured by the imaging unit 220 (S11). The image processing unit 14 generates an RGB image G12 based on the RAW image G11 (S12). The image processing unit 14 performs hue detection on the RGB image G12 and identifies a few colors in the scene (S13). The protective frame 310 of the protective frame body 300 is manually or automatically changed to a frame of the specified color (S14). Note that S12 can be omitted.

The image processing unit 14 acquires the RAW image G21 of the next image frame (S15). The imaging scene of the RAW image G21 is the same as the imaging scene of the RAW image G11 obtained in S11. The image processing unit 14 excludes the region of the protection frame 310 of the protection frame body 300 in the RAW image G21 as the mask region MR, and calculates the detection value of 3A detection (S16). Specifically, the image processing unit 14 calculates at least one of the detection value of AF detection, the detection value of AE detection, and the detection value of AWB detection.

The camera processor 11 controls the imaging unit 220 based on the calculated 3A detection detection value (S17). For example, the camera processor 11 may control at least one of the shutter 12, the aperture 33, and the ND filter 32 of the imaging unit 220 to perform automatic exposure control. The camera processor 11 may control the lens of the lens group 34 of the imaging unit 220 to perform automatic focus control. The camera processor 11 may perform image processing on the 3A detection image G2 via the image processing unit 14 and perform automatic white balance control.

Although the present disclosure has been described above using the embodiments, the technical scope of the present disclosure is not limited to the scope described in the above-described embodiments. It will be apparent to those skilled in the art to make various changes or improvements to the embodiments described above. It is clear from the description of the claims that the form with such changes or improvements may be included in the technical scope of the present disclosure.

The execution order of each process such as operation, procedure, step, and step in the device, system, program, and method shown in the claims, the specification, and the drawing is particularly "before" and "prior to". As long as the output of the previous process is not used in the subsequent process, it can be realized in any order. Even if the scope of claims, the specification, and the operation flow in the drawings are explained using "first", "next", etc. for convenience, it means that it is essential to carry out in this order. is not.

In the above embodiment, the image pickup unit 220 has the functions of image pickup control and image processing, but some functions of the image pickup unit 220 may be distributed and implemented by other devices or other units. For example, the UAV control unit 110 may perform at least a part of the image pickup control and image processing functions of the image pickup unit 220.

10 Aircraft system 80 Terminal 81 Terminal control unit 83 Operation unit 85 Communication unit 87 Storage unit 88 Display unit 100 Unmanned aerial vehicle 110 UAV control unit 150 Communication unit 160 Storage unit 200 Gimbal 210 Rotating wing mechanism 220 Imaging unit 240 GPS receiver 250 Inertial Measuring device 260 Magnetic compass 270 Atmospheric pressure meter 280 Ultrasonic sensor 290 Laser measuring instrument 300 Protective frame body 310 Protective frame HG, HG2 Hua histogram G1 Frame color specification image G11 RAW image G12 RGB image G2 3A Detection image G21 RAW image G22 RGB Image G3 Edge image G4 AF detection image G5 AE / AWB detection image

Claims (26)

An image processing device that processes an image captured by an imaging unit included in an air vehicle surrounded by a frame body.
An image processing unit for processing the image is provided.
The image processing unit
A first color having a small proportion of the total color of the scene imaged by the imaging unit in a state where the frame color of the frame body is a predetermined color is specified.
A first image in which the scene is captured by the imaging unit while the color of the frame is the first color is acquired.
Based on the first image and the first color, a detection value used for at least one of automatic exposure control, automatic focus control, and automatic white balance control is calculated.
Image processing device. The image processing unit
Prior to the first image, a second image in which the scene was captured was acquired.
Identifying the first color, which is less contained in the overall color of the second image.
The image processing apparatus according to claim 1. The image processing unit
Hue detection was performed on the second image,
A hue histogram showing the number of pixels of each hue in the second image is calculated by hue detection.
In the hue histogram, the first color corresponding to the hue whose number of pixels is equal to or less than the threshold value is specified.
The image processing apparatus according to claim 2. The second image is an RGB image converted based on the captured image captured by the imaging unit.
The image processing apparatus according to claim 2 or 3. A first control unit for presenting the information of the first color is further provided.
The image processing apparatus according to any one of claims 1 to 4. The frame body has a light emitting portion capable of emitting light in a plurality of colors.
A second control unit that controls the light emission of the light emitting unit and changes the color of the frame of the frame body to the first color is further provided.
The image processing apparatus according to any one of claims 1 to 4. The image processing unit
A third image obtained by extracting the high frequency component of the first image is generated.
A fourth image is generated by subtracting the first color component in the first image from the third image.
The detection value used for the automatic focus control is calculated for the fourth image.
The image processing apparatus according to any one of claims 1 to 6. The image processing unit
A fifth image is generated by subtracting the first color component in the first image from the first image.
The detection value used for the automatic exposure control or the automatic white balance control is calculated for the fifth image.
The image processing apparatus according to any one of claims 1 to 7. A third control unit that controls imaging by the imaging unit based on the detection value is further provided.
The image processing apparatus according to any one of claims 1 to 8. The third control unit controls at least one of the lens, shutter, aperture, dimming filter, and image processing of the first image by the image processing unit based on the detection value. ,
The image processing apparatus according to claim 9. The frame body is rotatable with respect to the imaging unit.
The image processing apparatus according to any one of claims 1 to 10. The image processing device is the flying object.
The image processing apparatus according to any one of claims 1 to 11. It is an image processing method that processes an image captured by an imaging unit included in an air vehicle surrounded by a frame body.
A step of specifying a first color in which the frame color of the frame body is a predetermined color and the proportion of the entire color of the scene imaged by the imaging unit is small is small.
A step of acquiring a first image in which the scene is captured by the imaging unit while the color of the frame is the first color.
A step of calculating a detection value used for at least one of automatic exposure control, automatic focus control, and automatic white balance control based on the first image and the first color.
Image processing method having. The step of identifying the first color is
Prior to the first image, a step of acquiring a second image in which the scene was captured,
A step of identifying the first color, which is contained in a small proportion of the entire color of the second image, is included.
The image processing method according to claim 13. The step of identifying the first color is
The step of hue detection for the second image and
A step of calculating a hue histogram showing the number of pixels of each hue in the second image by hue detection, and a step of calculating a hue histogram.
Including the step of specifying the first color corresponding to the hue whose number of pixels is equal to or less than the threshold value in the hue histogram.
The image processing method according to claim 14. The second image is an RGB image converted based on the captured image captured by the imaging unit.
The image processing method according to claim 14 or 15. Further comprising the step of presenting the first color information.
The image processing method according to any one of claims 13 to 16. The frame body has a light emitting portion capable of emitting light in a plurality of colors.
A step of controlling the light emission of the light emitting unit and changing the color of the frame of the frame body to the first color is further included.
The image processing method according to any one of claims 13 to 16. The step of calculating the detection value is
A step of generating a third image obtained by extracting a high frequency component of the first image, and
A step of subtracting the first color component in the first image from the third image to generate a fourth image.
A step of calculating the detection value used for the automatic focus control with respect to the fourth image is included.
The image processing method according to any one of claims 13 to 18. The step of calculating the detection value is
A step of subtracting the first color component in the first image from the first image to generate a fifth image.
The fifth image includes a step of calculating the detection value used for the automatic exposure control or the automatic white balance control.
The image processing method according to any one of claims 13 to 19. A step of controlling imaging by the imaging unit based on the detection value is further included.
The image processing method according to any one of claims 13 to 20. The step of controlling the image pickup by the image pickup unit is a step of controlling at least one of the lens, the shutter, the aperture, the dimming filter, and the image processing for the first image of the image pickup unit based on the detection value. Including,
The image processing method according to claim 21. The frame body is rotatable with respect to the imaging unit.
The image processing method according to any one of claims 13 to 22. The image processing method is executed by an image processing apparatus.
The image processing device is the flying object.
The image processing method according to any one of claims 13 to 23. An image processing device that processes the image captured by the image pickup unit of the flying object surrounded by the frame body.
A step of specifying a first color in which the frame color of the frame body is a predetermined color and the proportion of the entire color of the scene imaged by the imaging unit is small is small.
A step of acquiring a first image in which the scene is captured by the imaging unit while the color of the frame is the first color.
A step of calculating a detection value used for at least one of automatic exposure control, automatic focus control, and automatic white balance control based on the first image and the first color.
A program to execute. An image processing device that processes the image captured by the image pickup unit of the flying object surrounded by the frame body.
A step of specifying a first color in which the frame color of the frame body is a predetermined color and the proportion of the entire color of the scene imaged by the imaging unit is small is small.
A step of acquiring a first image in which the scene is captured by the imaging unit while the color of the frame is the first color.
A step of calculating a detection value used for at least one of automatic exposure control, automatic focus control, and automatic white balance control based on the first image and the first color.
A computer-readable recording medium that contains a program for executing the program.