WO2021249245A1 - Device, camera device, camera system, and movable member - Google Patents

Abstract

A device comprising a plurality of light receiving elements including a first light receiving element, wherein the first light receiving element is constructed to receive a light beam which passes through a first lens and is split by a pupil. The device comprises a first image sensor, wherein the first image sensor is configured to deviate from a focal point position of the first lens along an optical axis direction of the first lens by a predetermined distance or more. The device comprises a circuit, wherein the circuit is constructed to generate, on the basis of a signal generated by the first light receiving element, information for the control of a second camera device. The second camera device comprises a second image sensor.

Description

Device, camera device, camera system and moving body Technical field

The invention relates to a device, a camera device, a camera system and a mobile body.

Background technique

Patent Document 1 discloses a camera system that calculates a distance value for each pixel of M×N pixels based on the TOF algorithm.

[Prior Art Literature]

[Patent Literature]

[Patent Document 1] Japanese Patent Document Special Form No. 2019-508717

Summary of the invention

An apparatus according to one aspect includes a plurality of light receiving elements including a first light receiving element, and the first light receiving element is configured to receive a light beam that passes through a first lens and is divided by a pupil. The device includes a first image sensor that is arranged to deviate from the focal position of the first lens by more than a predetermined distance along the optical axis direction of the first lens. The device includes a circuit configured to generate information for control of a second imaging device including a second image sensor based on a signal generated by the first light receiving element.

The circuit may be configured to generate information used for focus adjustment of the second imaging device based on the signal generated by the first light receiving element.

The first image sensor may be disposed closer to the first lens side than the position of the focal point of the first lens.

The first light receiving element may be arranged at a preset interval. The first image sensor may be arranged to deviate from the focal position of the first lens along the optical axis direction of the first lens, so that the expansion width of the image of the infinitely distant subject on the image plane of the first image sensor is greater than or equal to a preset interval.

The first lens may be a fixed focus lens.

The first image sensor may include a first light-receiving element and a second light-receiving element, and the second light-receiving element is configured to receive a light beam belonging to a wavelength range of visible light and generate an image signal. The circuit may be configured to generate information used for exposure control and color adjustment of the second imaging device based on the image signal generated by the second light receiving element.

The first image sensor may include a third light-receiving element configured to receive a light beam in a wavelength range of invisible light and generate an image signal. The circuit may be configured to generate an image in the wavelength range of non-visible light based on the image signal generated by the third light receiving element.

The imaging range of the first imaging device including the first lens and the first image sensor may be larger than the imaging range of the second imaging device.

The imaging range of the first imaging device may be set to include the imaging range of the second imaging device.

The circuit may be configured to generate distance measurement information in the imaging range of the first imaging device based on the signal generated by the first light receiving element. The circuit may be configured such that when the imaging range of the second imaging device is changed, the focus adjustment of the second imaging device is performed based on the distance measurement information in the imaging range of the second imaging device after the change.

An imaging device according to one aspect includes the above-mentioned device and a first lens.

An imaging system according to one aspect includes the above-mentioned imaging device and a second imaging device.

The moving body according to one aspect is equipped with the above-mentioned imaging device and moves.

The moving body may include a second camera device and a support mechanism capable of controlling the posture of the second camera device and supporting it.

According to the above-mentioned one aspect, the information related to the distance to the subject can be acquired more accurately.

In addition, the above content does not enumerate all the necessary features of the present invention. In addition, sub-combinations of these feature groups can also constitute inventions.

Description of the drawings

FIG. 1 shows an example of the appearance of an unmanned aerial vehicle (UAV) 10 and a remote operation device 12.

Fig. 2 shows an example of the functional blocks of UAV10.

FIG. 3 shows an example of functional blocks of the imaging device 60.

FIG. 4 schematically shows the arrangement pattern of the light receiving elements of the image sensor 320.

FIG. 5 schematically shows the positional relationship between the focal point FP of the lens 300 and the image sensor 320.

FIG. 6 schematically shows the MTF characteristics obtained by the lens 300 and the image sensor 320.

FIG. 7 schematically shows the output waveform of the light receiving element 410 assuming that the image surface 322 of the image sensor 320 is located at the position of the focal point FP.

FIG. 8 schematically shows the output waveform of the light receiving element 410 when the image plane 322 of the image sensor 320 is located at a position away from the focal point FP.

FIG. 9 schematically shows the imaging range 910 of the imaging device 60 and the imaging range 920 of the imaging device 100.

Fig. 10 schematically shows a ranging action when a variable focus lens is used.

FIG. 11 shows an example of a computer 1200.

detailed description

Hereinafter, the present invention will be explained through the embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, all the feature combinations described in the embodiments are not necessarily necessary for the solution of the invention. It is obvious to those of ordinary skill in the art that various changes or improvements can be made to the following embodiments. It is obvious from the description of the claims that all such changes or improvements can be included in the technical scope of the present invention.

The claims, the description, the drawings of the description, and the abstract of the description include matters that are the subject of copyright protection. As long as anyone makes copies of these files as indicated in the patent office's documents or records, the copyright owner will not raise an objection. However, in other cases, all copyrights are reserved.

Various embodiments of the present invention can be described with reference to flowcharts and block diagrams. Here, the blocks can represent (1) a stage of a process of performing an operation or (2) a "part" of a device that performs an operation. Specific stages and "sections" can be implemented by programmable circuits and/or processors. Dedicated circuits may include digital and/or analog hardware circuits. May include integrated circuits (ICs) and/or discrete circuits. Programmable circuits may include reconfigurable hardware circuits. Reconfigurable hardware circuits can include logical AND, logical OR, logical exclusive OR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGA), programmable logic arrays (PLA) ) And other memory components.

The computer-readable medium may include any tangible device that can store instructions to be executed by a suitable device. As a result, the computer-readable medium on which instructions are stored includes a product including instructions that can be executed to create means for performing operations specified by the flowchart or block diagram. As an example of a computer-readable medium, it may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. As a more specific example of the computer readable medium, it may include floppy (registered trademark) disk floppy disk, floppy disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) Or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (RTM) disc, memory Sticks, integrated circuit cards, etc.

The computer-readable instructions may include any one of source code or object code described in any combination of one or more programming languages. The source code or object code includes traditional procedural programming languages. Traditional procedural programming languages can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, or Smalltalk (registered trademark), JAVA (registered trademark) , C++ and other object-oriented programming languages and "C" programming language or similar programming languages. The computer-readable instructions may be provided locally or via a wide area network (WAN) such as a local area network (LAN) or the Internet to a processor or programmable circuit of a general-purpose computer, a special-purpose computer, or other programmable data processing device. The processor or programmable circuit can execute computer-readable instructions to create means for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, and so on.

FIG. 1 shows an example of the appearance of an unmanned aerial vehicle (UAV) 10 and a remote operation device 12. The UAV 10 includes a UAV main body 20, a universal joint 50, a plurality of imaging devices 60, and the imaging device 100. The gimbal 50, the camera device 100, and the camera device 60 are an example of a camera system. UAV10, or mobile, refers to the concept including flying objects moving in the air, vehicles moving on the ground, and ships moving on the water. Flying objects moving in the air refer to concepts that include not only UAVs, but also other aircraft, airships, and helicopters that move in the air.

The UAV main body 20 includes a plurality of rotors. Multiple rotors are an example of a propulsion section. The UAV main body 20 makes the UAV 10 fly by controlling the rotation of a plurality of rotors. The UAV main body 20 uses, for example, four rotors to make the UAV 10 fly. The number of rotors is not limited to four. In addition, UAV10 can also be a fixed-wing aircraft without rotors.

The imaging device 100 is an imaging camera that captures a subject included in a desired imaging range. The universal joint 50 rotatably supports the imaging device 100. The gimbal 50 is an example of a support mechanism that can control the posture of the imaging device 100 and support it. For example, the gimbal 50 uses an actuator to rotatably support the imaging device 100 with a pitch axis. The universal joint 50 uses an actuator to further rotatably support the imaging device 100 around the roll axis and the yaw axis, respectively. The universal joint 50 can change the posture of the camera device 100 by rotating the camera device 100 about at least one of the yaw axis, the pitch axis, and the roll axis.

The imaging device 60 is a sensor camera that photographs the surroundings of the UAV 10 in order to control the flight of the UAV 10. The two camera devices 60 can be installed on the nose of the UAV 10, that is, on the front side. In addition, the other two camera devices 60 may be provided on the bottom surface of the UAV 10. The two imaging devices 60 on the front side can function as so-called stereo cameras. The two imaging devices 60 on the bottom side may also be paired to function as a stereo camera. Based on the images captured by the plurality of camera devices 60, three-dimensional spatial data around the UAV 10 can be generated. At least one of the plurality of imaging devices 60 is an imaging device for acquiring information used for control of the imaging device 100 including focus control and exposure control. The number of camera devices 60 included in the UAV 10 is not limited. It is sufficient that the UAV 10 includes at least one camera device 60. The UAV 10 may also include at least one camera 60 on the nose, tail, side, bottom, and top surfaces of the UAV 10, respectively. The viewing angle that can be set in the imaging device 60 may be larger than the viewing angle that can be set in the imaging device 100. At least one of the plurality of imaging devices 60 may have a single focus lens or a fisheye lens.

The remote operation device 12 communicates with the UAV 10 to remotely operate the UAV 10. The remote operation device 12 can communicate with the UAV 10 wirelessly. The remote operation device 12 transmits instruction information indicating various commands related to the movement of the UAV 10 such as ascending, descending, accelerating, decelerating, advancing, retreating, and rotating to the UAV 10. The instruction information includes, for example, instruction information for raising the height of the UAV 10. The indication information may indicate the height at which the UAV10 should be located. The UAV 10 moves to be located at the height indicated by the instruction information received from the remote operation device 12. The instruction information may include an ascending instruction to raise the UAV10. UAV10 rises while receiving the rise command. When the height of UAV10 has reached the upper limit height, even if the ascending instruction is accepted, the ascent of UAV10 can be restricted.

Fig. 2 shows an example of the functional blocks of UAV10. UAV10 includes UAV control unit 30, memory 37, communication interface 36, propulsion unit 40, GPS receiver 41, inertial measurement unit 42 (IMU42), magnetic compass 43, barometric altimeter 44, temperature sensor 45, humidity sensor 46, universal joint 50. The imaging device 60 and the imaging device 100.

The communication interface 36 communicates with other devices such as the remote operation device 12. The communication interface 36 can receive instruction information including various instructions to the UAV control unit 30 from the remote operation device 12. The memory 37 stores the UAV control unit 30 to control the propulsion unit 40, the GPS receiver 41, the IMU 42, the magnetic compass 43, the barometric altimeter 44, the temperature sensor 45, the humidity sensor 46, the universal joint 50, the imaging device 60, and the imaging device 100. Required procedures, etc. The memory 37 may be a computer-readable recording medium, and may include at least one of flash memory such as SRAM, DRAM, EPROM, EEPROM, USB memory, and solid state drive (SSD). The storage 37 may be provided inside the UAV main body 20. It can be configured to be detachable from the UAV main body 20.

The UAV control unit 30 controls the flying and shooting of the UAV 10 in accordance with a program stored in the memory 37. The UAV control unit 30 may be composed of a microprocessor such as a CPU or an MPU, a microcontroller such as an MCU, and the like. The UAV control unit 30 controls the flight and shooting of the UAV 10 in accordance with instructions received from the remote operating device 12 via the communication interface 36. The propulsion unit 40 propels the UAV10. The propulsion unit 40 includes a plurality of rotors and a plurality of drive motors that rotate the plurality of rotors. The propulsion unit 40 rotates a plurality of rotors via a plurality of drive motors in accordance with an instruction from the UAV control unit 30 to cause the UAV 10 to fly.

The GPS receiver 41 receives a plurality of signals indicating the time transmitted from a plurality of GPS satellites. The GPS receiver 41 calculates the position (latitude and longitude) of the GPS receiver 41, that is, the position (latitude and longitude) of the UAV 10 based on the received signals. The IMU42 detects the posture of the UAV10. The IMU 42 detects the acceleration of the UAV 10 in the three-axis directions of front and rear, left and right, and up and down, and the angular velocities of the pitch axis, the roll axis, and the yaw axis as the attitude of the UAV 10. The magnetic compass 43 detects the position of the nose of the UAV10. The barometric altimeter 44 detects the flying altitude of the UAV10. The barometric altimeter 44 detects the air pressure around the UAV 10 and converts the detected air pressure to altitude to detect the altitude. The temperature sensor 45 detects the temperature around the UAV 10. The humidity sensor 46 detects the humidity around the UAV 10.

The imaging device 100 includes an imaging unit 102 and a lens unit 200. The lens part 200 is an example of a lens device. The imaging unit 102 includes an image sensor 120, a control unit 110, a memory 130, and a distance measuring sensor. The image sensor 120 may be composed of CCD or CMOS. The image sensor 120 captures optical images formed through the plurality of lenses 210 and outputs the captured images to the control unit 110. The control unit 110 generates a recording image through image processing based on the pixel information read from the image sensor 120 and stores it in the memory 130. The control unit 110 may be constituted by a microprocessor such as a CPU or an MPU, a microcontroller such as an MCU, or the like. The control unit 110 can control the imaging device 100 in accordance with an operation command of the imaging device 100 from the UAV control unit 30. The control unit 110 may be at least a part of the circuit in the present invention. The memory 130 may be a computer-readable recording medium, and may include at least one of flash memory such as SRAM, DRAM, EPROM, EEPROM, USB memory, and solid state drive (SSD). The memory 130 stores programs and the like necessary for the control unit 110 to control the image sensor 120 and the like. The memory 130 may be provided inside the housing of the imaging device 100. The storage 130 may be configured to be detachable from the housing of the imaging device 100.

The distance measuring sensor measures the distance to the subject. The distance measuring sensor can measure the distance to the specified subject. The ranging sensor may be an infrared sensor, an ultrasonic sensor, a stereo camera, a TOF (Time Of Flight) sensor, etc.

The lens part 200 includes a plurality of lenses 210, a plurality of lens driving parts 212, and a lens control part 220. The multiple lenses 210 can function as zoom lenses, variable focal length lenses, and focus lenses. At least a part or all of the plurality of lenses 210 are configured to be able to move along the optical axis. The lens unit 200 may be an interchangeable lens provided to be detachable from the imaging unit 102. The lens driving unit 212 moves at least a part or all of the plurality of lenses 210 along the optical axis via a mechanism member such as a cam ring. The lens driving part 212 may include an actuator. The actuator may include a stepper motor. The lens control unit 220 drives the lens driving unit 212 in accordance with a lens control instruction from the imaging unit 102 to move one or more lenses 210 in the optical axis direction via a mechanism member. The lens control commands are, for example, zoom control commands and focus control commands.

The lens part 200 further includes a memory 222 and a position sensor 214. The lens control unit 220 controls the movement of the lens 210 in the optical axis direction via the lens drive unit 212 in accordance with the lens operation command from the imaging unit 102. Part or all of the lens 210 moves along the optical axis. The lens control section 220 performs at least one of a zooming action and a focusing action by moving at least one of the lenses 210 along the optical axis. The position sensor 214 detects the position of the lens 210. The position sensor 214 can detect the current zoom position or focus position.

The lens driving part 212 may include a shake correction mechanism. The lens control section 220 may move the lens 210 in a direction along the optical axis or a direction perpendicular to the optical axis via a shake correction mechanism to perform shake correction. The lens driving part 212 may drive a shake correction mechanism by a stepping motor to perform shake correction. In addition, the shake correction mechanism may be driven by a stepping motor to move the image sensor 120 in a direction along the optical axis or a direction perpendicular to the optical axis to perform shake correction.

The memory 222 stores control values of the plurality of lenses 210 moved via the lens driving unit 212. The memory 222 may include at least one of flash memory such as SRAM, DRAM, EPROM, EEPROM, and USB memory.

FIG. 3 shows an example of functional blocks of the imaging device 60. FIG. 3 shows functional blocks of the imaging device 60 that acquires an image for generating information used for control of the imaging device 100. The imaging unit 60 includes a lens 300, an image sensor 320, a control unit 310, and a memory 330. The control unit 310 exchanges information with the control unit 110 through the UAV control unit 30.

The control unit 310 may be constituted by a microprocessor such as a CPU or an MPU, a microcontroller such as an MCU, or the like. The memory 330 may be a computer-readable recording medium, and may include at least one of flash memory such as SRAM, DRAM, EPROM, EEPROM, USB memory, and solid state drive (SSD). The memory 330 stores programs and the like necessary for the control unit 310 to control the image sensor 320 and the like. The control unit 310 included in the imaging device 60, the control unit 110 included in the imaging device 100, and the UAV control unit 30 may all function as at least a part of the circuit.

The image sensor 320 may be composed of CCD or CMOS. The image sensor 320 captures an optical image formed by the light beam passing through the lens 300 and outputs the captured image to the control unit 310. The control unit 310 performs image processing based on the pixel information read from the image sensor 320 to generate an image, which is stored in the memory 330.

In this embodiment, the lens 300 is a fixed focus lens. The positional relationship between the lens 300 and the image sensor 320 is fixed. As described later, the image sensor 320 is provided so as to deviate from the position of the focal point FP of the lens 300 by a predetermined distance or more in the direction of the optical axis AX of the lens 300. For example, the image sensor 320 is provided closer to the lens 300 side than the focal position of the lens 300. That is, the imaging device 60 is designed to capture a subject in a so-called focus backward state. The control section 310 adopts a subject image captured in a state where the focus is moved backward, acquires distance measurement information through a phase difference detection method, and generates information for focus adjustment of the imaging device 100.

FIG. 4 schematically shows the arrangement pattern of the light receiving elements of the image sensor 320. The image sensor 320 includes a plurality of light receiving elements arranged two-dimensionally. In FIG. 4, "G" denotes a light-receiving element that receives light passing through a filter that transmits light in the green wavelength range. "R" denotes a light-receiving element that receives light passing through a filter that transmits light in the red wavelength range. "B" represents a light-receiving element that receives light passing through the green filter. "IR" refers to a light-receiving element that receives light passing through a filter that transmits light in the infrared wavelength range. Each light-receiving element generates information about one pixel in the image.

The image sensor 320 includes a plurality of light-receiving elements, including light-receiving element 410a and light-receiving element 410b, light-receiving element 420G, light-receiving element 420Ga, light-receiving element 420Gb, light-receiving element 420B and light-receiving element 420R, and light-receiving element 430. The image sensor 320 has a pixel arrangement in which the 4×4 pixel arrangement shown in FIG. 4 is arranged in a matrix. In addition, the light-receiving element 420G, the light-receiving element 420Ga, the light-receiving element 420Gb, the light-receiving element 420B, and the light-receiving element 420R are collectively referred to as the "light-receiving element 420" in some cases. The light-receiving element 410a and the light-receiving element 410b are sometimes collectively referred to as the "light-receiving element 410".

The light-receiving element 410 is an example of a first light-receiving element that receives a light beam that has passed through the lens 300 and is divided by the pupil. That is, the light receiving element 410 is a light receiving element for distance measurement based on phase difference detection. The control unit 310 is configured to generate information used for focus adjustment of the imaging device 100 based on the signal generated by the light receiving element 410. The control unit 310 generates information used for control of the imaging device 100 including the image sensor 120 based on the signal generated by the light receiving element 410.

The light receiving element 420 is an example of a second light receiving element that receives a light beam in the wavelength range of visible light and generates an image signal. The light receiving element 430 is an example of a third light receiving element that receives a light beam in a wavelength range of invisible light and generates an image signal. In this embodiment, the light receiving element 430 receives light in the infrared wavelength range.

The control unit 310 generates information used for exposure control and color adjustment of the imaging device 100 based on the image signal generated by the light receiving element 420. Specifically, the control unit 310 generates, based on the image signal generated by the light receiving element 420, a visible light image that is an image formed by light in the wavelength range of visible light. The control unit 310 generates brightness information of each area within the imaging range of the imaging device 60 based on the brightness information of the visible light image. In addition, the control unit 310 generates color information of each area within the imaging range of the imaging device 60 based on the brightness information of each color of the visible light image. The control unit 310 outputs the generated brightness information and color information to the control unit 110. The control unit 110 performs automatic exposure control of the imaging device 100 based on the brightness information output from the control unit 310. The control unit 110 adjusts the white balance of the image generated by the image sensor 120 of the imaging device 100 based on the color information output from the control unit 310. In this way, the control unit 310 is configured to generate information used for exposure control and color adjustment of the imaging device 100 based on the image signal generated by the light receiving element 420.

The light receiving element 430 generates an image signal formed of light in the infrared wavelength range. The control unit 310 generates an infrared image that is an image formed by light in the infrared wavelength range based on the image signal generated by the light receiving element 430. For example, an infrared image is used as an image for analyzing temperature distribution. In addition, infrared images are also used as images for analyzing subjects in dark places. Thus, the control unit 310 is configured to generate an image in the wavelength range of invisible light based on the image signal generated by the light receiving element 430.

The light receiving element 410a is arranged on the right side in one pixel area. The light receiving element 410b is arranged on the left side in one pixel area. Specifically, in the pixel region where the light receiving element 410a and the light receiving element 420Ga are provided, when viewed from the subject side, the light receiving element 410a is provided on the right side, and the light receiving element 420Ga is provided on the left side. In addition, in the pixel area where the light-receiving element 410b and the light-receiving element 420Gb are provided, when viewed from the subject side, the light-receiving element 410b is provided on the left side, and the light-receiving element 420Gb is provided on the right side. In this way, the light-receiving element 410b and the light-receiving element 420Gb are separately provided on the left and right in the horizontal pixel direction in one pixel area. Thus, the light receiving element 410a and the light receiving element 410b mainly receive light beams that pass through different pupils of the exit pupil of the lens 300.

In addition, the plurality of light receiving elements 410 provided on the right side of one pixel region like the light receiving element 410a may be referred to as "first phase difference elements". In addition, the plurality of light receiving elements 410 provided on the left side of one pixel region like the light receiving element 410b may be referred to as "second phase difference elements". As shown in FIG. 4, the light receiving element 410a and the light receiving element 410b are alternately arranged every two pixels in the horizontal pixel direction. Therefore, the interval between the pixel provided with the light receiving element 410a and the pixel provided with the light receiving element 410b is two pixels. In addition, the first phase difference element including the light receiving element 410a is provided every four pixels in the horizontal pixel direction. In addition, the second phase difference element including the light receiving element 410b is provided every four pixels in the horizontal pixel direction.

The control unit 310 uses the correlation between the image generated by the first phase difference element including the light receiving element 410a and the image generated by the second phase difference element including the light receiving element 410b to detect the phase difference. The control unit 310 acquires distance measurement information based on the detected phase difference. In this way, the imaging device 60 acquires the distance information of the subject through the phase difference detection method. In addition, the control unit 310 generates information used for focus adjustment of the imaging device 100 based on the signal generated by the light receiving element 410. In addition, the control unit 310 may generate information for the zoom value or angle of view control of the lens 300 based on the signal generated by the light receiving element for phase difference detection.

FIG. 5 schematically shows the positional relationship between the focal point FP of the lens 300 and the image sensor 320. The focal point FP is a point at which the light rays passing through the lens 300 intersect the optical axis AX when the light rays parallel to the optical axis AX of the lens 300 are incident on the lens 300. As shown in FIG. 5, in the optical axis AX direction, the image surface 322 of the image sensor 320 is located at a position away from the focal point FP. Specifically, the image surface 322 of the image sensor 320 is located closer to the lens 300 side than the focal point FP. Thus, the image sensor 320 is arranged at a position deviated from the position of the focal point FP so that the image plane 322 is located closer to the lens 300 than the focal point FP.

In this way, the lens 300 is fixed so that the focal point FP is located on the back side of the image sensor 320. As a result, the image of the subject at infinity is projected onto the image sensor 320 with the focus blurred. In addition, an image of a short-distance subject is also projected on the image sensor 320 in a state where the focus is blurred.

FIG. 6 schematically shows the MTF characteristics obtained by the lens 300 and the image sensor 320. The solid line 600 represents the MTF characteristic when the image surface 322 of the image sensor 320 is set at a position away from the focus FP position as in the imaging device 60. The dotted line 610 represents the MTF characteristic when the image surface 322 of the image sensor 320 is assumed to be at the position of the focal point FP. As described above, compared with the case where the image plane 322 is assumed to be at the position of the focal point FP, the MTF of the imaging device 60 becomes lower in the high spatial frequency domain.

FIG. 7 schematically shows the output waveform of the light receiving element 410 assuming that the image surface 322 of the image sensor 320 is located at the position of the focal point FP. The graph 710 shows the output waveform of the first phase difference element including the light receiving element 410a. The graph 720 shows the output waveform of the second phase difference element including the light receiving element 410b. In the graph 710 and the graph 720, the horizontal axis represents the position in the horizontal direction, and the vertical axis represents the brightness value. In addition, the graph 710 and the graph 720 show the output waveform of the light receiving element 410 when a pattern with a strong luminance difference in the horizontal direction is used as the subject. As shown in the graph 710 and the graph 720, when the focus state is approached with respect to the subject, the edge becomes steep. That is, the edge is represented by the signal of a few light receiving elements 410. Therefore, the phase difference is calculated by the correlation operation of the edge portion formed by fewer pixels, so the calculation accuracy of the phase difference is reduced. In addition, since the generated phase difference is also reduced, the calculation accuracy of the phase difference is also liable to decrease.

FIG. 8 schematically shows the output waveform of the light receiving element 410 when the image plane 322 of the image sensor 320 is located at a position away from the focal point FP. The graph 810 shows the output waveform of the first phase difference element including the light receiving element 410a. The graph 820 shows the output waveform of the second phase difference element including the light receiving element 410b. In the graph 810 and the graph 820, the horizontal axis represents the position in the horizontal direction, and the vertical axis represents the brightness value. In addition, as in FIG. 7, the graph 810 and the graph 820 show the output waveform of the light receiving element 410 when a pattern having a strong luminance difference in the horizontal direction is used as the subject. When the image plane 322 is located at a position away from the focal point FP, the blur generated in the subject image becomes larger than when the image plane 322 is located at the focal point FP. Therefore, as shown in graph 810 and graph 820, the edges become gentle. In this way, the edge can be represented by the signals of the plurality of light receiving elements 410. Therefore, it is possible to improve the calculation accuracy of the phase difference when the phase difference is calculated by correlation calculation. Furthermore, since the generated phase difference also becomes larger, the calculation accuracy of the phase difference is improved.

In order to improve the calculation accuracy of the phase difference, it is preferable that at least the plurality of light receiving elements 410 can capture an image of an infinitely distant subject formed on the image plane 322 of the image sensor 320 through the lens 300. That is, it is preferable to set the image sensor 320 to be offset from the focal point FP in the direction of the optical axis AX so that the expansion width of the image of the subject at infinity is greater than or equal to the interval of the plurality of light receiving elements 410. As shown in Fig. 4, the light-receiving element 410a and the light-receiving element 410b are alternately arranged every two pixels in the horizontal pixel direction. That is, the interval between the pixel provided with the light receiving element 410a and the pixel provided with the light receiving element 410b is two pixels. Therefore, when the light receiving element 410 is arranged every two pixels, it is preferable that the image sensor 320 is offset from the focal point FP in the direction of the optical axis AX, so that the expansion width of the image of the subject at infinity is two pixels. above.

In addition, in order to be able to use more signals of the light-receiving element 410 to indicate the edge, it is desirable to set the image sensor 320 to deviate from the focal point FP in the direction of the optical axis AX, so that the expansion width of the image of the infinitely distant subject is greater than or equal to the receiving pass designation. The distance between the light receiving element 410 in the pupil area. For example, as shown in FIG. 4, preferably, when the first phase difference element including the light receiving element 410a is arranged every four pixels, and the second phase difference element including the light receiving element 410b is arranged every four pixels, the image sensor 320 It is set to shift from the focal point FP in the optical axis AX direction so that the expansion width of the image of the infinitely distant subject is four pixels or more. In addition, as the expansion width of the image of an infinitely distant subject, the half-spread value of the point spread function, the half-spread value of the line spread function, or the like can be applied.

In this way, when the light-receiving elements 410 in the image sensor 320 are arranged at a preset interval, the image sensor 320 is preferably arranged to deviate from the position of the focal point FP of the lens 300 along the optical axis direction of the lens 300, so that the image surface of the image sensor 320 The expansion width of the image of the infinitely distant subject on 322 is greater than or equal to the preset interval.

As described above, in the imaging device 60, the positional relationship between the lens 300 and the image sensor 320 is fixed so that the focal point FP is located on the back side of the image sensor 320 (the side opposite to the object side). As a result, it is possible to improve the detection accuracy of the phase difference in a large distance range from an infinitely distant subject to a short-distance subject. In addition, the control section 310 acquires brightness information and color information of the subject based on the image information of the visible light image obtained from the image sensor 320. In order to perform exposure control of the imaging device 100, brightness information in pixel units is not necessarily required, and it is sometimes preferable to obtain average brightness information of each partial region of the image. Similarly, in order to adjust the white balance of the imaging device 100, color information in a pixel unit is not necessarily required, and it is sometimes preferable to obtain average color information of each partial region of the image. Therefore, it is possible to acquire brightness information and color information with sufficient accuracy even from an image accompanied by a certain degree of blur.

FIG. 9 schematically shows the imaging range 910 of the imaging device 60 and the imaging range 920 of the imaging device 100. The viewing angle of the imaging device 60 is larger than the viewing angle of the imaging device 100. The imaging range 910 of the imaging device 60 is larger than the imaging range 920 of the imaging device 100. As shown in FIG. 9, the imaging range 910 of the imaging device 60 includes all of the imaging range 920 of the imaging device 100.

In addition, the imaging range of the imaging device 60 is determined by the posture of the UAV 10. On the other hand, the imaging range 920 of the imaging device 100 is determined based on the posture of the imaging device 100 controlled by the gimbal 50 and the angle of view of the imaging device 100 controlled by the control unit 110 in addition to the posture of the UAV 10. When the UAV 10 is in an arbitrary posture, the imaging range 920 of the imaging device 60 may include all the imaging ranges that can be changed by the imaging device 100 under the control of the universal joint 50 and the control unit 110.

As described above, the control unit 310 generates distance measurement information through phase difference detection based on the signal generated by the light receiving element 410 among the signals generated by the image sensor 320. Here, the control in the case where the current imaging range 920 of the imaging device 100 is changed to the imaging range 922 will be described. For example, when the universal joint 50 rotates the imaging device 100 around the yaw axis and the pitch axis to change the imaging range of the imaging device 100 from the imaging range 920 to the imaging range 922, the control unit 110 obtains the distance measurement generated by the control unit 310 The distance measurement information of the area corresponding to the imaging range 922 in the information. The control unit 110 controls the position of the focus lens included in the lens 210 based on the acquired distance measurement information, thereby performing focus control on the subject within the imaging range 922. As a result, the control unit 110 can quickly perform focus control in accordance with the change in the imaging range of the imaging device 100.

In addition, when the imaging range of the imaging device 100 is changed from the imaging range 920 to the imaging range 922, the control unit 110 acquires the brightness information detected by the control unit 310 from the image of the image of the imaging range 910 corresponding to the imaging range 922. And based on the acquired brightness information, the exposure control of the imaging device 100 is performed. As a result, exposure control can be quickly performed in accordance with changes in the imaging range of the imaging device 100. In addition, the control unit 110 acquires the color information detected by the control unit 310 from the image of the region corresponding to the imaging range 922 in the image of the imaging range 910, and determines the whiteness of the image acquired by the image sensor 120 based on the acquired color information. Balance the parameters of processing. As a result, the imaging device 100 can quickly determine appropriate white balance processing parameters to perform white balance adjustment.

As described above, the control unit 310 generates distance measurement information in the imaging range of the imaging device 60 based on the signal generated by the light receiving element 410. In addition, when the imaging range of the imaging device 100 is changed, the control unit 310 performs focus adjustment of the imaging device 100 based on the distance measurement information in the imaging range of the imaging device 100 after the change.

As described above, the imaging device 60 can capture a larger range than the imaging device 100, and acquire distance information, brightness information, and color information of the subject. Thus, the imaging device 60 can be used as a 3A (AE/AF/AWB) camera. Since the imaging range of the imaging device 60 is set to include the imaging range of the imaging device 100, even when the imaging range of the imaging device 100 changes due to the posture of the UAV 10 or the control of the universal joint 50, the imaging device can be used. The distance information, brightness information and color information obtained by 60 can quickly perform 3A-related control. In addition, since the wavelength filter of a partial pixel region of the image sensor 320 is set as an IR filter, an infrared image can be acquired at the same time.

Fig. 10 schematically shows a ranging action when a variable focus lens is used. The manner in which the position of the focal point of the lens 300 is set to be variable will be described with reference to FIG. 10. The control unit 310 calculates the subject distance based on the image information of the light receiving element 410 obtained by changing the focal position of the lens 300 and performing three imaging. For example, the control unit 310 performs imaging in a state where the imaging point of the lens 300 is located at the position 1010, and performs phase difference detection using image information generated by the image sensor 320. In this way, the control unit 110 calculates the distance D1 to the subject. Next, the control unit 310 moves the focal point of the lens 300, performs imaging with the imaging point at the position 1020, and performs phase difference detection using the image information generated by the image sensor 320. Thus, the distance D2 to the subject is calculated. Next, the control unit 310 moves the focal point of the lens 300, performs imaging with the imaging point at the position 1030, and performs phase difference detection using the image information generated by the image sensor 320. Thus, the distance D3 to the subject is calculated.

The control unit 110 determines the distance to the subject based on the values of the distance D1, the distance D2, and the distance D3. For example, the control unit 110 selects one of the distances D1, D2, and D3 as the distance to the subject. As an example, the control unit 110 can calculate the Euclidean distance of any combination of two of D1, D2, and D3, select the two values that provide the shortest Euclidean distance from the values of D1, D2, and D3, and select from the selected Euclidean distance. One of the two values is the distance to the subject. The control unit 110 may select two values that provide the shortest Euclidean distance, or may select an average value from the selected two values as the distance to the subject. For example, depending on the distance to the subject and the spatial frequency component of the subject itself, there may be an optimal focus position for detecting the phase difference with high accuracy. As described in the related description of FIG. 10, there are cases where the phase difference can be detected with higher accuracy by changing the position of the focal point of the lens 300 to perform multiple imaging and performing phase difference detection.

In addition, the control unit 110 may determine the position of the focal point of the lens 300 based on the imaging mode of the imaging device 100. The control section 110 may select a predetermined position corresponding to the imaging mode of the imaging device 100 as the focal position of the lens 300. When the imaging mode of the imaging device 100 is set to the portrait mode, the control unit 110 can set the focal position of the lens 300 closer to the image sensor 320 than when the imaging mode of the imaging device 100 is set to the landscape mode. .

The focal position of the lens 300 can be changed by changing the position of the lens 300 on the optical axis AX. In the case where the lens 300 includes a focusing lens, the position of the focal point of the lens 300 can be changed by changing the position of the focusing lens on the optical axis AX. In addition, in the above modification, the method of changing the focal position of the lens 300 has been described, but a method of changing the positional relationship between the lens 300 and the image sensor 320 on the optical axis AX may also be adopted. For example, instead of changing the focal position of the lens 300, a method of changing the position of the image sensor 320 on the optical axis AX may be adopted. Moreover, in addition to changing the focal position of the lens 300, a method of changing the position of the image sensor 320 on the optical axis AX may also be adopted.

In the above-described embodiment, the imaging device 100 and the imaging device 60 are imaging devices mounted on the UAV 10 as an example of the imaging system. The camera device 100 and the camera device 60 can be configured to be detachable from the UAV 10. At least one of the imaging device 100 and the imaging device 60 can be realized by a camera included in a portable terminal such as a smartphone. At least one of the camera 100 and the camera 60 may be configured to be detachable from the UAV 10. At least one of the imaging device 100 and the imaging device 60 may not be an imaging device mounted on a movable body such as the UAV10. For example, the camera device 100 may be a camera device supported by a handheld gimbal. In addition, the imaging device 100 and the imaging device 60 may be imaging devices that are not supported by the UAV 10 and the handheld gimbal. For example, at least one of the camera device 100 and the camera device 60 may be a camera device that can be held by the user. At least one of the camera device 100 and the camera device 60 may be a fixedly installed camera device such as a surveillance camera.

FIG. 11 shows an example of a computer 1200 that can embody aspects of the present invention in whole or in part. The program installed on the computer 1200 can make the computer 1200 function as an operation associated with the device according to the embodiment of the present invention or one or more "parts" of the device. Alternatively, the program can cause the computer 1200 to perform the operation or the one or more "parts". This program enables the computer 1200 to execute the process or stages of the process involved in the embodiment of the present invention. Such a program may be executed by the CPU 1212, so that the computer 1200 executes a specified operation associated with some or all of the blocks in the flowcharts and block diagrams described in this specification.

The computer 1200 of this embodiment includes a CPU 1212 and a RAM 1214, which are connected to each other through a host controller 1210. The computer 1200 further includes a communication interface 1222, an input/output unit, which is connected to the host controller 1210 through the input/output controller 1220. The computer 1200 also includes a ROM 1230. The CPU 1212 operates in accordance with the programs stored in the ROM 1230 and RAM 1214 to control each unit.

The communication interface 1222 communicates with other electronic devices through a network. The hard disk drive can store programs and data used by the CPU 1212 in the computer 1200. The ROM 1230 stores therein a boot program executed by the computer 1200 during operation, and/or a program dependent on the hardware of the computer 1200. The program is provided via a computer-readable recording medium such as CR-ROM, USB memory, or IC card, or a network. The program is installed in RAM 1214 or ROM 1230, which is also an example of a computer-readable recording medium, and is executed by CPU 1212. The information processing described in these programs is read by the computer 1200 and brings about the cooperation between the programs and the various types of hardware resources described above. The device or method may be constituted by realizing operation or processing of information as the computer 1200 is used.

For example, when performing communication between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214, and instruct the communication interface 1222 to perform communication processing according to the processing described by the communication program. The communication interface 1222 reads the transmission data stored in the transmission buffer provided in a recording medium such as RAM 1214 or USB memory under the control of the CPU 1212, and sends the read transmission data to the network or receives the data from the network The received data is written into the receiving buffer provided in the recording medium, etc.

In addition, the CPU 1212 can make the RAM 1214 read all or required parts of files or databases stored in an external recording medium such as a USB memory, and perform various types of processing on the data on the RAM 1214. Then, the CPU 1212 can write the processed data back to the external recording medium.

It is possible to store various types of information such as various types of programs, data, tables, and databases in the recording medium and receive information processing. For the data read from the RAM 1214, the CPU 1212 can perform various types of operations, information processing, conditional judgment, conditional transition, unconditional transition, and information retrieval/retrieval/information specified by the instruction sequence of the program described in various places in this disclosure. Replace various types of processing, and write the results back to RAM 1214. In addition, the CPU 1212 can search for information in files, databases, and the like in the recording medium. For example, when multiple entries having the attribute value of the first attribute respectively associated with the attribute value of the second attribute are stored in the recording medium, the CPU 1212 may retrieve the attribute value of the specified first attribute from the multiple entries And read the attribute value of the second attribute stored in the entry to obtain the attribute value of the second attribute that is associated with the first attribute that meets the predetermined condition.

The programs or software modules described above may be stored on the computer 1200 or on a computer readable storage medium near the computer 1200. In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium to provide the program to the computer 1200 via the network.

It should be noted that the execution order of the actions, sequences, steps, and stages in the devices, systems, programs, and methods shown in the claims, descriptions and drawings, as long as there is no special indication that "in... "Before", "in advance", etc., and as long as the output of the previous processing is not used in the subsequent processing, they can be implemented in any order. Regarding the operating procedures in the claims, the description, and the drawings, the description is made using "first", "next", etc. for convenience, but it does not mean that it must be implemented in this order.

The present invention has been described above using the embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those of ordinary skill in the art that various changes or improvements can be made to the above-mentioned embodiments. It is obvious from the description of the claims that all such changes or improvements can be included in the technical scope of the present invention.

Symbol Description

10 UAV

12Remote operation device

20 UAV subject

30 UAV Control Department

36 communication interface

37 memory

40 Promotion Department

41 GPS receiver

42 inertial measurement device

43 magnetic compass

44 barometric altimeter

45 temperature sensor

46 humidity sensor

500,000 joint

60 camera

100 camera

102 Camera Department

110 Control Department

120 image sensor

130 memory

200 lens

210 lenses

212 lens drive unit

214 position sensor

220 lens control unit

222 memory

300 lenses

310 Control Department

320 image sensor

322 Image surface

330 memory

410, 420, 430 light receiving element

710, 720, 810, 820 charts

910, 920, 922 camera range

1010, 1020, 1030 position

1200 computers

1210 host controller

1212 CPU

1214 RAM

1220 input/output controller

1222 communication interface

1230 ROM

Claims (14)

1. An apparatus, characterized by comprising: a first image sensor, the first image sensor comprising a plurality of light receiving elements including first light receiving elements, the first light receiving element is configured to receive through a first lens and be pupil Divided light beams, the first image sensor is arranged to deviate from the focal position of the first lens by more than a predetermined distance along the optical axis direction of the first lens; and

   A circuit configured to generate information for control of a second imaging device based on a signal generated by the first light receiving element, the second imaging device including a second image sensor.
2. The device according to claim 1, wherein the circuit is configured to generate information used for focus adjustment of the second imaging device based on a signal generated by the first light receiving element.
3. The device according to claim 1, wherein the first image sensor is arranged closer to the first lens side than the focal position of the first lens.
4. The device according to claim 1 or 2, wherein the first light-receiving element is arranged at a preset interval,

   The first image sensor is arranged to deviate from the focal position of the first lens along the optical axis direction of the first lens, so as to expand the image of the infinitely distant subject on the image plane of the first image sensor The width is greater than or equal to the preset interval.
5. The device according to claim 1 or 2, wherein the first lens is a fixed focus lens.
6. The device according to claim 1 or 2, wherein the first image sensor comprises

   The first light-receiving element and

   A second light receiving element configured to receive a light beam belonging to the wavelength range of visible light and generate an image signal,

   The circuit is configured to generate information used for exposure control and color adjustment of the second imaging device based on the image signal generated by the second light receiving element.
7. 7. The device according to claim 6, wherein the first image sensor further comprises a third light receiving element, and the third light receiving element is configured to receive a light beam in a wavelength range of invisible light and generate an image signal;

   The circuit is configured to generate an image in a wavelength range of invisible light based on the image signal generated by the third light receiving element.
8. The device according to claim 1 or 2, wherein the imaging range of the first imaging device including the first lens and the first image sensor is larger than the imaging range of the second imaging device.
9. The device according to claim 8, wherein the imaging range of the first imaging device is set to include the imaging range of the second imaging device.
10. The device according to claim 8, wherein the circuit is configured as:

    Generating distance measurement information within the imaging range of the first imaging device based on the signal generated by the first light receiving element;

    When the imaging range of the second imaging device is changed, the focus adjustment of the second imaging device is performed based on the distance measurement information in the imaging range of the second imaging device after the change.
11. A camera device, characterized by comprising the device according to claim 1 or 2; and

    The first lens.
12. A camera system, characterized by comprising the camera device according to claim 11; and

    The second camera device.
13. A mobile body characterized in that it mounts the imaging device according to claim 11 and moves.
14. The mobile body according to claim 13, further comprising the second camera device; and

    A supporting mechanism capable of controlling the posture of the second imaging device and supporting it.

Images