JP6972458B1 - Control devices, imaging systems, moving objects, control methods, and programs - Google Patents

Abstract

PROBLEM TO BE SOLVED: To further optimize image runout correction. A control device controls a support mechanism according to a first signal in a first frequency band based on a vibration signal indicating vibration of an image pickup device, and a first drive mechanism according to a second signal in the second frequency band based on the vibration signal. It is provided with a circuit configured to perform image shake correction by controlling. If the circuit moves the optical system or image sensor to the boundary position of the movable range while performing image shake correction, the circuit moves the optical system or image sensor to a predetermined first position within the movable range. After performing the reset operation to move and moving the optical system or image sensor to the boundary position within the movable range, the first frequency band and the second frequency band based on the vibration signal are included until the reset operation is completed. By controlling the support mechanism according to the third signal in the third frequency band, it is configured to perform image shake correction. [Selection diagram] FIG. 5

Description

The present invention relates to a control device, an imaging system, a moving body, a control method, and a program.

Patent Document 1 describes that "one or both of the first lens 3 and the second lens 4 are rotated to correct the shake of the image at the time of camera shake".
[Prior Art Document]
[Patent Document]
[Patent Document 1] Japanese Unexamined Patent Publication No. 9-251127

It is desired to further optimize the image shake correction in an image pickup system including an image pickup device and a support mechanism that rotatably supports the image pickup device.

The control device according to one aspect of the present invention includes an image pickup device having a first drive mechanism for moving the optical system or an image sensor in a direction intersecting the optical axis of the optical system, and a support mechanism for rotatably supporting the image pickup device. It may be a control device for controlling an imaging system. The control device controls the support mechanism according to the first signal in the first frequency band based on the vibration signal indicating the vibration of the image pickup device, and controls the first drive mechanism according to the second signal in the second frequency band based on the vibration signal. A circuit configured to perform image shake correction may be provided. If the circuit moves the optical system or image sensor to the boundary position of the movable range while performing image shake correction, the circuit moves the optical system or image sensor to a predetermined first position within the movable range. After performing the reset operation to move and moving the optical system or image sensor to the boundary position within the movable range, the first frequency band and the second frequency band based on the vibration signal are included until the reset operation is completed. By controlling the support mechanism according to the third signal in the third frequency band, the image shake correction may be performed.

The second frequency band may include a higher frequency than the frequency included in the first frequency band.

The second frequency band may include a part of the frequency included in the first frequency band.

When the circuit moves to the boundary position within the movable range of the optical system or image sensor, the circuit makes the optical system or image sensor stand by at the boundary position until the reset operation is started, and controls the first drive mechanism based on the second signal. This may be configured to start the reset operation when the moving direction of the optical system or the image sensor approaches the first position.

The image pickup system according to one aspect of the present invention may include the control device, an image pickup device having an optical system, an image sensor, and a first drive mechanism, and a support mechanism.

The moving body according to one aspect of the present invention may be a moving body provided with the above-mentioned imaging system.

The control method according to one aspect of the present invention includes an image pickup device having a first drive mechanism for moving the optical system or an image sensor in a direction intersecting the optical axis of the optical system, and a support mechanism for rotatably supporting the image pickup device. It may be a control method for controlling an imaging system including. The control method is to control the support mechanism according to the first signal in the first frequency band based on the vibration signal indicating the vibration of the image pickup device, and to control the first drive mechanism according to the second signal in the second frequency band based on the vibration signal. Then, a step of performing image shake correction may be provided. The control method is that if the optical system or image sensor moves to the boundary position of the movable range while performing image shake correction, the optical system or image sensor reaches a predetermined first position within the movable range. It may be provided with a step of performing a reset operation to move the optics. The control method is a third frequency band including a first frequency band and a second frequency band based on a vibration signal after the optical system or the image sensor moves to a boundary position within the movable range until the reset operation is completed. By controlling the support mechanism according to the third signal of the above, a step of performing image shake correction may be provided.

The program according to one aspect of the present invention may be a program for operating a computer as the control device.

According to one aspect of the present invention, image shake correction can be further optimized in an image pickup system including a support mechanism that rotatably supports the image pickup device.

The outline of the above invention does not list all the necessary features of the present invention. A subcombination of these feature groups can also be an invention.

It is a figure which shows an example of the appearance of an image pickup system. It is a figure which shows an example of the functional block of an image pickup system. It is a figure which shows the state of the image shake correction in the pitch direction of an image pickup system schematically. It is a figure which shows the state of the time change of the runout angle required for image runout correction. It is a flowchart which shows an example of the processing procedure in the case of performing image runout correction of a gimbal and OIS. It is a figure which shows an example of the appearance of an unmanned aerial vehicle and a remote control device. It is a figure which shows an example of a hardware configuration.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential to the means of solving the invention. It will be apparent to those skilled in the art that various changes or improvements can be made to the following embodiments. It is clear from the description of the claims that the form with such changes or improvements may be included in the technical scope of the present invention.

The claims, description, drawings, and abstracts include matters subject to copyright protection. The copyright holder will not object to any person's reproduction of these documents as long as they appear in the Patent Office files or records. However, in other cases, all copyrights are reserved.

Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, wherein the block is (1) a stage of the process in which the operation is performed or (2) a device having a role of performing the operation. May represent a "part" of. Specific steps and "parts" may be implemented by programmable circuits and / or processors. Dedicated circuits may include digital and / or analog hardware circuits. It may include integrated circuits (ICs) and / or discrete circuits. The programmable circuit may include a reconfigurable hardware circuit. Reconfigurable hardware circuits include logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGA), programmable logic arrays (PLA), etc. It may include a memory element such as.

The computer-readable medium may include any tangible device capable of storing instructions executed by the appropriate device. As a result, the computer-readable medium having the instructions stored therein will include the product, including instructions that can be executed to create means for performing the operation specified in the flowchart or block diagram. Examples of the computer-readable medium may include an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, and the like. More specific examples of computer-readable media include floppy (registered trademark) disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), Electrically erasable programmable read-only memory (EEPROM®), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (RTM) disc, Memory sticks, integrated circuit cards, etc. may be included.

Computer-readable instructions may include either source code or object code written in any combination of one or more programming languages. Source code or object code includes traditional procedural programming languages. Traditional procedural programming languages are assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcodes, firmware instructions, state setting data, or Smalltalk®, JAVA®, C ++. It may be an object-oriented programming language such as, and a "C" programming language or a similar programming language. Computer-readable instructions are used locally or on a local area network (LAN), wide area network (WAN), etc., to the processor or programmable circuit of a general purpose computer, special purpose computer, or other programmable data processing device. ) May be provided. The processor or programmable circuit may execute computer-readable instructions to create means for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, and the like.

FIG. 1 is a diagram showing an example of the appearance of the image pickup system 1000 according to the present embodiment. The image pickup system 1000 includes a gimbal 50, an image pickup device 100, a support member 410, a pair of handle portions 400, a handle portion 420, and a display device 450.

The image pickup apparatus 100 is a camera for taking an image of a subject included in a desired imaging range. The gimbal 50 rotatably supports the image pickup device 100. The gimbal 50 supports the posture of the image pickup apparatus 100 in an adjustable manner. The gimbal 50 is an example of a support mechanism. For example, the gimbal 50 rotatably supports the image pickup device 100 about the yaw axis by using an actuator. The gimbal 50 further rotatably supports the image pickup device 100 around each of the pitch axis and the roll axis by using an actuator. The gimbal 50 may change the posture of the image pickup device 100 by rotating the image pickup device 100 around at least one of the yaw axis, the pitch axis, and the roll axis.

The support member 410 detachably supports the gimbal 50. The support member 410 has a T-shape and includes a rod-shaped member 412 extending in the pitch axis direction and a rod-shaped member 414 extending in the yaw axis direction from the central portion of the rod-shaped member 412. The pair of handle portions 400 are rotatably attached to the support member 410. The pair of handle portions 400 are attached to both ends of the rod-shaped member 412 with the gimbal 50 interposed therebetween. The pair of handle portions 400 are rotatably attached to the support member 410. The pair of handle portions 400 may be detachably provided on the support member 410. A gimbal 50 is detachably attached to one end of the rod-shaped member 414. The handle portion 420 extends in the roll axis direction and is provided at the other end of the rod-shaped member 414.

The rod-shaped member 414 is further provided with a display device 450. The display device 450 may be a touch panel display. The display device 450 is attached to the rod-shaped member 414 on the side opposite to the side where the lens portion 200 of the image pickup device 100 is provided. The display device 450 may be attached to the rod-shaped member 414 on the back side opposite to the front side where the lens portion 200 of the image pickup device 100 is provided. The display device 450 may be detachably provided on the support member 410. The image pickup system 1000 may be used with the display device 450 removed from the support member 410. The display device 450 may be provided on the support member 410 so that the angle of the display surface can be adjusted. The display device 450 may be provided on the support member 410 so as to be rotatable about the pitch axis.

The support member 410 and the display device 450 are examples of mounting members attached to the support member 410. The display device 450 in this embodiment is attached to the support member 410 as a separate body from the image pickup device 100. However, the display device 450 may be provided as a part of the image pickup device 100. The display device 450 may be supported by the support member 410 via the image pickup device 100. The display device 450 may be integrally provided with the image pickup device 100. The display device 450 may be provided in the image pickup device 100 so that the angle of the display surface can be adjusted with respect to the image pickup device 100.

FIG. 2 is a diagram showing an example of a functional block of the imaging system 1000. The image pickup system 1000 includes a gimbal 50, an image pickup device 100, a main control unit 600, a memory 610, a handle unit 400, and a display device 450.

The display device 450 displays an image captured by the image pickup device 100. The display device 450 may display a setting screen for setting various operating conditions of the gimbal 50 and the image pickup device 100. The display device 450 may be a touch display, and the user may instruct the operation of the gimbal 50 and the image pickup device 100 via the display device 450.

The main control unit 600 controls the entire image pickup system 1000. The main control unit 600 may be configured by a microprocessor such as a CPU or MPU, a microcontroller such as an MCU, or the like. The memory 610 stores programs and the like necessary for the main control unit 600 to control the gimbal 50, the image pickup device 100, and the handle unit 400. The memory 610 may be a computer-readable recording medium and may include at least one of flash memories such as SRAM, DRAM, EPROM, EEPROM, and USB memory. The memory 610 may be provided in the support member 410. The memory 610 may be provided so as to be removable from the support member 410.

The gimbal 50 includes a gimbal control unit 510, a yaw axis driver 512, a pitch axis driver 522, a roll axis driver 532, a yaw axis drive unit 514, a pitch axis drive unit 524, a roll axis drive unit 534, a yaw axis rotation mechanism 516, and a pitch axis. It has a rotation mechanism 526 and a roll shaft rotation mechanism 536.

The yaw axis rotation mechanism 516 rotates the image pickup device 100 around the yaw axis. The pitch axis rotation mechanism 526 rotates the image pickup apparatus 100 around the pitch axis. The roll axis rotation mechanism 536 rotates the image pickup device 100 around the roll axis. The gimbal control unit 510 outputs drive signals indicating the respective drive amounts to the yaw axis driver 512, the pitch axis driver 522, and the roll axis driver 532 in response to the drive signal of the gimbal 50 from the main control unit 600. do. The yaw axis driver 512, the pitch axis driver 522, and the roll axis driver 532 drive the yaw axis drive unit 514, the pitch axis drive unit 524, and the roll axis driver unit 534 according to the drive signal indicating the drive amount. The yaw axis rotation mechanism 516, the pitch axis rotation mechanism 526, and the roll axis rotation mechanism 536 are driven by the yaw axis drive unit 514, the pitch axis drive unit 524, and the roll axis drive unit 534 to rotate, and the posture of the image pickup apparatus 100. To change.

The image pickup apparatus 100 includes an image pickup unit 102 and a lens unit 200. The image pickup unit 102 includes an image sensor 120, an image pickup control unit 110, and a memory 130. The image sensor 120 may be configured by CCD or CMOS. The image sensor 120 outputs the image data of the optical image formed through the zoom lens 211 and the focus lens 210 to the image pickup control unit 110.

The image pickup control unit 110 may be configured by a microprocessor such as a CPU or MPU, a microcontroller such as an MCU, or the like. The memory 130 may be a computer-readable recording medium and may include at least one of flash memories such as SRAM, DRAM, EPROM, EEPROM, and USB memory. The memory 130 stores a program or the like necessary for the image pickup control unit 110 to control the image sensor 120 or the like. The memory 130 may be provided inside the housing of the image pickup apparatus 100. The memory 130 may be provided so as to be removable from the housing of the image pickup apparatus 100.

The lens unit 200 includes a focus lens 210, a zoom lens 211, a lens drive unit 212, a lens drive unit 213, and a lens control unit 220. The focus lens 210 and the zoom lens 211 may include at least one lens. At least a part or all of the focus lens 210 and the zoom lens 211 are arranged so as to be movable along the optical axis. The lens unit 200 may be an interchangeable lens that is detachably provided with respect to the image pickup unit 102. The lens driving unit 212 moves at least a part or all of the focus lens 210 along the optical axis via a mechanical member such as a cam ring and a guide shaft. The lens driving unit 213 moves at least a part or all of the zoom lens 211 along the optical axis via a mechanical member such as a cam ring and a guide shaft. The lens control unit 220 drives at least one of the lens drive unit 212 and the lens drive unit 213 according to a lens control command from the image pickup unit 102, and illuminates at least one of the focus lens 210 and the zoom lens 211 via a mechanical member. By moving along the axial direction, at least one of the zoom operation and the focus operation is executed. The lens control command is, for example, a zoom control command and a focus control command.

The lens unit 200 further includes a position sensor 214 and a position sensor 215. The position sensor 214 detects the position of the focus lens 210. The position sensor 214 may detect the current focus position. The position sensor 215 detects the position of the zoom lens 211. The position sensor 215 may detect the current zoom position of the zoom lens 211.

The lens unit 200 has an optical image stabilization mechanism (OIS). More specifically, the lens unit 200 includes a lens 231 for image shake correction, a lens drive unit 233, and a position sensor 235. Further, the image pickup unit 102 has a vibration sensor 250. The vibration sensor 250 outputs a vibration signal indicating the vibration of the image pickup apparatus 100. The vibration sensor 250 may be a gyro sensor that detects the vibration of the image pickup apparatus 100. The vibration sensor 250 may be an acceleration sensor that detects the vibration of the image pickup apparatus 100. The gyro sensor detects, for example, angular runout and rotational runout. The accelerometer detects, for example, shift deviation in the X direction or the Y direction. Even with a gyro sensor, angles and rotations can be converted into components in the X direction and components in the Y direction. The accelerometer can also convert shift blur in the X direction and Y direction into angular blur and rotational blur. The vibration sensor 250 may be a combination of an acceleration sensor and a gyro sensor.

The lens driving unit 233 moves the lens 231 in a direction orthogonal to the optical axis to perform image shake correction. The lens driving unit 233 may include an electric motor that drives the lens 231 in the X direction and an electric motor that drives the lens 231 in the Y direction. The electric motor may be a stepping motor. The electric motor may be a voice coil motor.

The main control unit 600 executes image shake correction based on the vibration signal from the vibration sensor 250. The main control unit 600 may generate a drive signal for the lens drive unit 233 for performing image shake correction based on the vibration signal from the vibration sensor 250. The lens drive unit 233 may move the lens 231 in a direction orthogonal to the optical axis based on the drive signal. The lens drive unit 233 may move the lens 231 in the X direction and the Y direction orthogonal to the optical axis based on the drive signal. The lens driving unit 233 may move the lens 231 in a direction orthogonal to the optical axis in a direction that reduces the influence of the vibration of the image pickup apparatus 100 based on the vibration signal from the vibration sensor 250.

The drive signal may indicate the amount of movement that moves the lens 231 in the X direction and the Y direction. The drive signal may indicate the drive amount of the electric motor for moving the lens 231 in the X direction and the drive amount of the electric motor for moving the lens 231 in the Y direction. The drive signal may indicate the current value input to each motor.

The position sensor 235 detects the position of the lens 231. The position sensor 235 may detect the position in the direction perpendicular to the optical axis of the lens 231. The position sensor 235 may detect the positions in the X and Y directions perpendicular to the optical axis of the lens 231.

The lens unit 200 is an example of an image shake correction device. The lens control unit 220 acquires a vibration signal indicating vibration from the vibration sensor 250, and based on the vibration signal, at least one of the X direction and the Y direction crossing the lens 231 with the optical axis via the lens drive unit 233. The image shake is corrected by vibrating to. The image sensor 120 captures an image formed through the zoom lens 211, the focus lens 210, and the lens 231.

The image pickup unit 102 further includes an in-body image shake correction mechanism (BIS). More specifically, the image pickup unit 102 further includes an image sensor drive unit 150 and a position sensor 152. The image sensor driving unit 150 moves the image sensor 120 in a direction intersecting the optical axis. The image sensor driving unit 150 moves the image sensor 120 in a direction orthogonal to the optical axis. The image sensor driving unit 150 moves the image sensor 120 in at least one of the X direction and the Y direction orthogonal to the optical axis. The image sensor driving unit 150 may include an electric motor that drives the image sensor 120 in the X direction and an electric motor that drives the image sensor 120 in the Y direction. The third motor and the fourth motor may be a stepping motor or a boil coil motor. The position sensor 152 detects the position of the image sensor 120. The position sensor 152 may detect a position in a direction perpendicular to the optical axis of the image sensor 120. The main control unit 600 may generate a drive signal for the image sensor drive unit 150 for performing image shake correction based on the vibration signal from the vibration sensor 250. The image pickup control unit 110 corrects image shake by vibrating the image sensor 120 in a direction intersecting the optical axis via the image sensor drive unit 150 based on the vibration signal from the main control unit 600. The image sensor drive unit 150 may move the image sensor 120 in a direction orthogonal to the optical axis based on the drive signal. The image sensor drive unit 150 may move the image sensor 120 in the X direction and the Y direction orthogonal to the optical axis based on the drive signal. The image sensor drive unit 150 may move the image sensor 120 in a direction orthogonal to the optical axis in a direction that reduces the influence of vibration of the image pickup apparatus 100 based on the drive signal.

The drive signal may indicate the amount of movement that moves the image sensor 120 in the X direction and the Y direction. The drive signal may indicate the drive amount of the electric motor for moving the image sensor 120 in the X direction and the drive amount of the electric motor for moving the image sensor 120 in the Y direction. The drive signal may indicate the current value input to each motor.

The image pickup apparatus 100 may have at least one of OIS and BIS. The image sensor drive unit 150 or the lens drive unit 233 is an example of the drive mechanism.

In the image pickup system 1000, the gimbal 50 controls the posture of the image pickup device 100. For example, the gimbal 50 uses the yaw axis rotation mechanism 516 via the yaw axis drive unit 514, the pitch axis drive unit 524, and the roll axis drive unit 534 so as to maintain the attitude of the image pickup apparatus 100 in a predetermined position. , Pitch shaft rotation mechanism 526, and roll shaft rotation mechanism 536 are controlled.

FIG. 3 schematically shows the state of image shake correction in the pitch direction of the image pickup system 1000. The image pickup system 1000 performs image shake correction by driving at least one of the lens 231 and the image sensor 120, and the gimbal 50 based on the vibration signal from the vibration sensor 250. The weights of the gimbal 50, the lens 231 and the image sensor 120 are different, and the ease of movement is also different. For example, the image sensor 120 is the lightest, then the lens 231 is the lightest, and the gimbal 50 is the heaviest. When the lens 231 is small, the lens 231 may be lighter than the image sensor 120. A light weight controlled object can vibrate at a higher frequency with lower power than a heavier weight controlled object. Therefore, it is possible to suppress the power consumption by performing the correction of the image shake at a high frequency with a controlled object having a light weight as compared with performing the correction with a controlled object having a heavy weight.

Therefore, in the present embodiment, the gimbal 50, the lens 231 and the image sensor 120 are set in consideration of dividing the vibration signal from the vibration sensor 250 into different frequency bands and suppressing power consumption in each frequency band. assign. Each of the gimbal 50, the lens 231 and the image sensor 120 performs image shake correction so as to cancel the vibration of the frequency in the assigned frequency band.

The main control unit 600 acquires a vibration signal indicating the vibration of the image pickup device 100 from the vibration sensor 250 via the image pickup control unit 110. The main control unit 600 acquires a first signal in the first frequency band and a second signal in the second frequency band from the vibration signal. The main control unit 600 may acquire a third signal in the third frequency band from the vibration signal. The main control unit 600 uses the filters of the first frequency band, the second frequency band, and the third frequency band to obtain the first signal of the first frequency band and the first signal of the second frequency band from the vibration signal. The two signals and the third signal in the third frequency band are separately acquired.

The main control unit 600 controls at least one of the yaw axis rotation mechanism 516, the pitch axis rotation mechanism 526, and the roll axis rotation mechanism 536 based on the first signal via the gimbal control unit 510. The main control unit 600 controls at least one of the yaw axis rotation mechanism 516, the pitch axis rotation mechanism 526, and the roll axis rotation mechanism 536 so as to cancel the vibration indicated by the first signal. The main control unit 600 controls the lens drive unit 233 based on the second signal via the lens control unit 220. The main control unit 600 controls the lens drive unit 233 so as to cancel the vibration indicated by the second signal. The main control unit 600 controls the image sensor drive unit 150 based on the third signal via the image pickup control unit 110. The main control unit 600 controls the image sensor drive unit 150 so as to cancel the vibration indicated by the third signal.

The main control unit 600 controls at least one of the yaw axis rotation mechanism 516, the pitch axis rotation mechanism 526, and the roll axis rotation mechanism 536 based on the first signal, and controls the lens drive unit 233 based on the second signal. Then, by controlling the image sensor driving unit 150 based on the third signal, image shake correction is executed.

The second frequency band for OIS may include a higher frequency than the frequency included in the first frequency band for the gimbal 50. The second frequency band for OIS may include a part of the frequency included in the first frequency band. The first frequency band may overlap with a part of the second frequency band. The third frequency band may include frequencies higher than those included in the second frequency band for OIS. The second frequency band may overlap with a part of the first frequency band and a part of the fourth frequency band. The second frequency band for OIS and the fourth frequency band for BIS may include frequencies higher than those included in the first frequency band for the gimbal 50.

The filters of the first frequency band, the second frequency band, and the fourth frequency band may be pre-designed according to the characteristics of the respective drive systems. The filters of the first frequency band, the second frequency band, and the fourth frequency band may be pre-designed according to the weight of the controlled object.

If the image pickup device 100 is detachably connected to the gimbal 50, the main control unit 600 acquires frequency band information indicating the frequency band for OIS and the frequency band for BIS from the image pickup device 100. , The frequency band information indicating the frequency band for the gimbal 50 is acquired from the memory 610, and the filters of the first frequency band, the second frequency band, and the fourth frequency band are set based on each frequency band information. good. The main control unit 600 acquires the filter information indicating the filter for OIS and the filter for BIS from the image pickup apparatus 100, acquires the filter information indicating the filter for the gimbal 50 from the memory 610, and is based on each filter information. , 1st frequency band, 2nd frequency band, and 4th frequency band may be set respectively.

The main control unit 600 reads out the filter information according to the frequency band shown in the frequency band information based on the plurality of filter information stored in the memory 610, and reads out the filter information, the first frequency band, the second frequency band, and the fourth. Each filter in the frequency band may be set.

In the case of an interchangeable lens in which the lens unit 200 is detachably attached to the image pickup unit 102, the main control unit 600 acquires the lens information of the interchangeable lens capable of specifying the frequency band or the filter, and based on the lens information, the main control unit 600 obtains the lens information of the interchangeable lens. A filter in the second frequency band may be set.

Here, by having the image pickup device 100 having OIS or BIS, the image shake is corrected by moving the lens 231 or the image sensor 120 in the direction of canceling the shake of the image pickup device 100. However, the movable range of the lens 231 or the image sensor 120 is physically limited. When the lens 231 or the image sensor 120 moves to a boundary position within the movable range, a reset operation for moving the lens 231 or the image sensor 120 to a predetermined first position within the movable range is executed. When the lens 231 or the image sensor 120 moves to a boundary position within the movable range, a reset operation for moving the lens 231 or the image sensor 120 to a predetermined initial position within the movable range may be executed. While the reset operation is being executed, the image shake correction cannot be executed by moving the lens 231 or the image sensor 120. On the other hand, the image shake correction by the gimbal 50 can be executed even while the reset operation of the lens 231 or the image sensor 120 is being executed.

Therefore, when the reset operation is executed while the lens 231 or the image sensor 120 is executing the image shake correction, the main control unit 600 is at least one of the frequency bands assigned to the lens 231 or the image sensor 120. The gimbal 50 may also perform image shake correction of the unit.

When the lens 231 or the image sensor 120 moves to the boundary position of the movable range while the image shake correction is being executed, the main control unit 600 moves the lens 231 to a predetermined first position within the movable range. Alternatively, a reset operation for moving the image sensor 120 may be executed. After the lens 231 or the image sensor 120 has moved to the boundary position within the movable range, the main control unit 600 has a first frequency band and a second frequency band based on the vibration signal from the vibration sensor 250 until the reset operation is completed. The third signal of the third frequency band including the frequency band may be acquired. The main control unit 600 may execute image shake correction by controlling the gimbal 50 according to the third signal.

When the lens 231 or the image sensor 120 moves to the boundary position within the movable range, the main control unit 600 may make the lens 231 or the image sensor 120 stand by at the boundary position until the reset operation is started. When the main control unit 600 controls the lens drive unit 233 or the image sensor drive unit 150 based on the second signal, the moving direction of the lens 231 or the image sensor 120 approaches the first position. The reset operation may be started.

When the main control unit 600 is executing the reset operation of the lens 231 or the image sensor 120 while the image shake correction is being executed, the frequency in the first frequency band from the vibration signal from the vibration sensor 250 and the first frequency band. Image shake correction is executed by acquiring the third signal of the third frequency band including at least a part of the frequencies of the second frequency band or the fourth frequency band and controlling the gimbal 50 based on the third signal. good. Further, the main control unit 600 may drive the gimbal 50 in a direction of canceling the reset operation of the lens 231 or the image sensor 120 while executing the image shake correction.

When the reset operation of the lens 231 is being executed, the main control unit 600 switches from the filter for the first frequency band and the filter for the second frequency band to the filter for the third frequency band, thereby switching to the third frequency. The third signal of the band may be acquired.

The main control unit 600 controls at least one of the yaw axis rotation mechanism 516, the pitch axis rotation mechanism 526, and the roll axis rotation mechanism 536 via the gimbal control unit 510 based on the third signal, and is based on the fourth signal. By controlling the image sensor 120 via the image sensor driving unit 150, image shake correction may be executed. When the lens 231 moves in the first direction by the first movement amount in the reset operation, the main control unit 600 moves the gimbal 50 so that the image pickup apparatus 100 moves in the second direction opposite to the first direction by the first movement amount. May be controlled. When the lens 231 moves by the first movement amount in the first direction by the reset operation, the main control unit 600 first moves the image pickup device 100 in the second direction opposite to the first direction while executing the image shake correction. The posture of the gimbal 50 may be controlled so as to rotate by a rotation amount corresponding to the amount. As a result, it is possible to suppress the displacement of the subject in the image due to the reset operation.

FIG. 4 shows how the runout angle required for image shake correction changes over time. FIG. 4 shows an example of performing image shake correction by controlling the drive of the gimbal 50 and the lens 231. (A) shows the image runout correction of the gimbal 50 and the total runout angle 800 of the OIS, and the runout angle 802 that should be originally corrected by the gimbal 50. In (b), the runout angle 802 originally to be corrected by the gimbal 50 and the runout angle 804 to be actually corrected by the gimbal 50 to which the runout angle of OIS is added according to the reset operation of OIS are shown. In (c), the runout angle 806 that should be originally corrected by OIS and the runout angle 808 that should be actually corrected by OIS due to the reset operation of OIS are shown.

In the first period 810, the main control unit 600 executes image shake correction of the gimbal 50 and OIS, respectively. In the first period 810, the main control unit 600 executes the image runout correction in the first frequency band by driving the gimbal 50. Further, the main control unit 600 performs the runout correction of the second frequency band higher than the first frequency band by OIS.

By executing the OIS, when the lens 231 moves to the boundary position of the movable range, in the second period 812, the main control unit 600 temporarily suspends the OIS and makes the lens 231 stand by at the boundary position. In the second period 812, the main control unit 600 adds the runout angle that should be originally corrected by the OIS, and executes the image shake correction by the gimbal 50.

After that, if the total runout angle 800 to be corrected is a runout angle in the direction in which the lens 231 moves so as to approach a predetermined first position, the main control unit 600 in the third period 814 A reset operation is executed to move the lens 231 to the first position. While the reset operation is being executed, the main control unit 600 adds the deflection angle of the lens 231 accompanying the reset operation to execute the image deflection correction of the gimbal 50. When the reset operation is completed, the image shake correction of the gimbal 50 and the OIS are executed again in the fourth period 816.

FIG. 5 shows an example of a processing procedure when performing image runout correction of the gimbal 50 and OIS.

The main control unit 600 drives the gimbal 50 based on the first signal in the first frequency band based on the vibration signal from the vibration sensor 250, and drives the lens 231 based on the second signal in the second frequency band based on the vibration signal. By driving, image shake correction is executed (S100). The main control unit 600 determines whether or not the lens 231 has moved to the boundary position of the movable range during the image shake correction (S102). If the lens 231 has not moved to the boundary position, the image shake correction for driving the gimbal 50 and the lens 231 is continued.

On the other hand, when the lens 231 moves to the boundary position, the main control unit 600 temporarily stops the OIS and makes the lens 231 stand by at the boundary position (S104). The main control unit 600 derives a runout angle that should be originally corrected by OIS (S106). The main control unit 600 adds the runout angle to be corrected by the OIS and executes the image runout correction by the gimbal 50 (S108).

Based on the second signal in the second frequency band, the main control unit 600 determines whether or not the runout angle that should be corrected by OIS indicates the angle at which the lens 231 moves in the direction approaching the first position ( S110). When the lens 231 moves in the direction approaching the first position, the main control unit 600 starts the reset operation of the OIS (S112). That is, the main control unit 600 starts driving the lens 231 in order to move the lens 231 to the first position.

While the reset operation is being executed, the main control unit 600 derives a runout angle that should be originally corrected by the OIS based on the second signal (S114). The main control unit 600 adds the runout angle to be corrected by the OIS and executes the image runout correction by the gimbal 50 (S116). The main control unit 600 executes image shake correction by the gimbal 50 so as to add the runout angle to be corrected by OIS and cancel the movement of the lens 231 due to the reset operation. When the reset operation is completed (S118). The main control unit 600 again executes image shake correction and OIS of the gimbal 50.

As described above, according to the present embodiment, image shake correction by the gimbal 50 is performed in addition to OIS or BIS, or OIS and BIS. Then, when the reset operation in OIS or BIS is executed, the gimbal 50 also executes the image shake correction that should be originally performed in OIS or BIS until the reset operation is completed. As a result, image shake correction can be performed with high accuracy even until the reset operation in OIS or BIS is completed. Power consumption is suppressed by performing image runout correction of high frequency components with OIS or BIS and image runout correction of low frequency components with gimbal 50. Moreover, when the image shake correction of the high frequency component cannot be executed by OIS or BIS due to the reset operation, the image shake correction of the low frequency and high frequency components is executed by the gimbal 50. Therefore, the accuracy of image shake correction can be improved.

The image pickup apparatus 100 as described above may be mounted on a moving body. The image pickup apparatus 100 may be mounted on an unmanned aerial vehicle (UAV) as shown in FIG. The UAV 10 may include a UAV main body 20, a gimbal 50, a plurality of image pickup devices 60, and an image pickup device 100. The gimbal 50 and the image pickup apparatus 100 are examples of an image pickup system. The UAV 10 is an example of a moving body propelled by a propulsion unit. The moving body is a concept including a UAV, a flying object such as another aircraft moving in the air, a vehicle moving on the ground, a ship moving on the water, and the like.

The UAV main body 20 includes a plurality of rotary wings. The plurality of rotor blades are an example of a propulsion unit. The UAV main body 20 flies the UAV 10 by controlling the rotation of a plurality of rotary blades. The UAV body 20 flies the UAV 10 using, for example, four rotor blades. The number of rotor blades is not limited to four. Further, the UAV 10 may be a fixed-wing aircraft having no rotor blades.

The image pickup apparatus 100 is a camera for taking an image of a subject included in a desired imaging range. The gimbal 50 rotatably supports the image pickup device 100. The gimbal 50 is an example of a support mechanism. For example, the gimbal 50 rotatably supports the image pickup device 100 on a pitch axis using an actuator. The gimbal 50 further rotatably supports the image pickup device 100 around each of the roll axis and the yaw axis by using an actuator. The gimbal 50 may change the posture of the image pickup device 100 by rotating the image pickup device 100 around at least one of the yaw axis, the pitch axis, and the roll axis.

The plurality of image pickup devices 60 are sensing cameras that image the surroundings of the UAV 10 in order to control the flight of the UAV 10. Two image pickup devices 60 may be provided in front of the nose of the UAV 10. Yet two other imaging devices 60 may be provided on the bottom surface of the UAV 10. The two image pickup devices 60 on the front side may be paired and function as a so-called stereo camera. The two image pickup devices 60 on the bottom side may also be paired and function as a stereo camera. Three-dimensional spatial data around the UAV 10 may be generated based on the images captured by the plurality of image pickup devices 60. The number of image pickup devices 60 included in the UAV 10 is not limited to four. The UAV 10 may include at least one image pickup device 60. The UAV 10 may be provided with at least one image pickup device 60 on each of the nose, tail, side surface, bottom surface, and ceiling surface of the UAV 10. The angle of view that can be set by the image pickup apparatus 60 may be wider than the angle of view that can be set by the image pickup apparatus 100. The image pickup apparatus 60 may have a single focus lens or a fisheye lens.

The remote control device 300 communicates with the UAV 10 to remotely control the UAV 10. The remote control device 300 may wirelessly communicate with the UAV 10. The remote control device 300 transmits to the UAV 10 instruction information indicating various commands related to the movement of the UAV 10, such as ascending, descending, accelerating, decelerating, advancing, reversing, and rotating. The instruction information includes, for example, instruction information for raising the altitude of the UAV 10. The instruction information may indicate the altitude at which the UAV 10 should be located. The UAV 10 moves so as to be located at an altitude indicated by the instruction information received from the remote control device 300. The instruction information may include an ascending instruction to elevate the UAV 10. The UAV10 rises while accepting the rise order. Even if the UAV10 accepts an ascending order, the ascending may be restricted if the altitude of the UAV10 has reached the upper limit altitude.

FIG. 7 shows an example of a computer 1200 in which a plurality of aspects of the present invention may be embodied in whole or in part. A program installed on a computer 1200 can cause the computer 1200 to function as an operation associated with a device according to an embodiment of the invention or as one or more "parts" of the device. Alternatively, the program may cause the computer 1200 to perform the operation or the one or more "parts". The program may cause the computer 1200 to perform a process according to an embodiment of the present invention or a step of the process. Such a program may be run by the CPU 1212 to cause the computer 1200 to perform certain operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

The computer 1200 according to this embodiment includes a CPU 1212 and a RAM 1214, which are connected to each other by a host controller 1210. The computer 1200 also includes a communication interface 1222, an input / output unit, which are connected to the host controller 1210 via an input / output controller 1220. The computer 1200 also includes a ROM 1230. The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit.

The communication interface 1222 communicates with other electronic devices via the network. A hard disk drive may store programs and data used by the CPU 1212 in the computer 1200. The ROM 1230 stores in it a boot program or the like executed by the computer 1200 at the time of activation and / or a program depending on the hardware of the computer 1200. The program is provided via a computer-readable recording medium such as a CR-ROM, USB memory or IC card or network. The program is installed in RAM 1214, which is also an example of a computer-readable recording medium, or ROM 1230, and is executed by the CPU 1212. The information processing described in these programs is read by the computer 1200 and provides a link between the program and the various types of hardware resources described above. The device or method may be configured to implement the operation or processing of information in accordance with the use of the computer 1200.

For example, when communication is executed between the computer 1200 and an external device, the CPU 1212 executes a communication program loaded in the RAM 1214, and performs communication processing with respect to the communication interface 1222 based on the processing described in the communication program. You may order. Under the control of the CPU 1212, the communication interface 1222 reads the transmission data stored in the transmission buffer area provided in the RAM 1214 or a recording medium such as a USB memory, and transmits the read transmission data to the network, or The received data received from the network is written to the receive buffer area or the like provided on the recording medium.

Further, the CPU 1212 makes the RAM 1214 read all or necessary parts of a file or a database stored in an external recording medium such as a USB memory, and executes various types of processing on the data on the RAM 1214. good. The CPU 1212 may then write back the processed data to an external recording medium.

Various types of information such as various types of programs, data, tables, and databases may be stored in recording media and processed. The CPU 1212 describes various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, and information retrieval described in various parts of the present disclosure with respect to the data read from the RAM 1214. Various types of processing may be performed, including / replacement, etc., and the results are written back to the RAM 1214. Further, the CPU 1212 may search for information in a file, database, or the like in the recording medium. For example, when a plurality of entries each having an attribute value of the first attribute associated with the attribute value of the second attribute are stored in the recording medium, the CPU 1212 specifies the attribute value of the first attribute. Search for an entry that matches the condition from the plurality of entries, read the attribute value of the second attribute stored in the entry, and associate it with the first attribute that satisfies the predetermined condition. The attribute value of the second attribute obtained may be acquired.

The program or software module described above may be stored on a computer 1200 or in a computer readable storage medium near the computer 1200. Also, a recording medium such as a hard disk or RAM provided in a dedicated communication network or a server system connected to the Internet can be used as a computer readable storage medium, thereby allowing the program to be transferred to the computer 1200 over the network. offer.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that the form with such changes or improvements may be included in the technical scope of the present invention.

The order of execution of each process such as operation, procedure, step, step in the device, system, program, and method shown in the claims, specification, and drawings is particularly "before" and "prior to". It should be noted that it can be realized in any order unless the output of the previous process is used in the subsequent process. Even if the scope of claims, the specification, and the operation flow in the drawings are explained using "first", "next", etc. for convenience, it means that it is essential to carry out in this order. It's not a thing.

10 UAV
20 UAV main unit 50 gimbal 60 image pickup device 100 image pickup device 102 image pickup unit 110 image pickup control unit 120 image sensor 130 memory 150 image sensor drive unit 152 position sensor 200 lens unit 210 focus lens 211 zoom lens 212, 213 lens drive unit 214, 215 position Sensor 220 Lens control unit 231 Lens 233 Lens drive unit 235 Position sensor 250 Vibration sensor 400 Handle unit 410 Support member 412 Rod-shaped member 414 Rod-shaped member 420 Handle unit 450 Display device 510 Gimbal control unit 512 Yaw axis driver 514 Yaw axis drive unit 516 Yaw axis rotation mechanism 522 Pitch axis driver 524 Pitch axis drive unit 526 Pitch axis rotation mechanism 532 Roll axis driver 534 Roll axis drive unit 536 Roll axis rotation mechanism 300 Remote control unit 600 Main control unit 610 Memory 1000 Imaging system 1200 Computer 1210 Host Controller 1212 CPU
1214 RAM
1220 Input / Output Controller 1222 Communication Interface 1230 ROM

Claims (8)

A control device that controls an image pickup system including an image pickup device having a first drive mechanism for moving an optical system or an image sensor in a direction intersecting the optical axis of the optical system, and a support mechanism for rotatably supporting the image pickup device. And,
Control the support mechanism according to the first signal in the first frequency band based on the vibration signal indicating the vibration of the image pickup device, and control the first drive mechanism according to the second signal in the second frequency band based on the vibration signal. With a circuit configured to perform image shake correction,
The circuit is
If the optical system or the image sensor moves to the boundary position of the movable range while the image shake correction is being performed, the optical system or the image sensor moves to a predetermined first position within the movable range. Perform a reset operation to move the image sensor,
After the optical system or the image sensor has moved to a boundary position within the movable range, the third frequency band including the first frequency band and the second frequency band based on the vibration signal is completed until the reset operation is completed. A control device configured to perform image shake correction by controlling the support mechanism according to a third signal in the frequency band. The control device according to claim 1, wherein the second frequency band includes a frequency higher than the frequency included in the first frequency band. The control device according to claim 1, wherein the second frequency band includes a part of frequencies included in the first frequency band. When the optical system or the image sensor moves to a boundary position within the movable range, the circuit causes the optical system or the image sensor to stand by at the boundary position until the reset operation is started, and causes the second signal. The reset operation is started when the moving direction of the optical system or the image sensor approaches the first position by controlling the first driving mechanism based on the first aspect. The control device described in. The control device according to any one of claims 1 to 4,
An image pickup device having the optical system, the image sensor, and the first drive mechanism,
An imaging system including the support mechanism. A mobile body that moves with the imaging system according to claim 5. A control method for controlling an image pickup system including an image pickup device having a first drive mechanism for moving an optical system or an image sensor in a direction intersecting the optical axis of the optical system, and a support mechanism for rotatably supporting the image pickup device. And,
Control the support mechanism according to the first signal in the first frequency band based on the vibration signal indicating the vibration of the image pickup device, and control the first drive mechanism according to the second signal in the second frequency band based on the vibration signal. At the stage of performing image shake correction,
If the optical system or the image sensor moves to the boundary position of the movable range while the image shake correction is being performed, the optical system or the image sensor moves to a predetermined first position within the movable range. The stage of performing the reset operation to move the image sensor and
After the optical system or the image sensor has moved to a boundary position within the movable range, the third frequency band including the first frequency band and the second frequency band based on the vibration signal is completed until the reset operation is completed. A control method including a step of performing image shake correction by controlling the support mechanism according to a third signal in the frequency band. A program for operating a computer as a control device according to any one of claims 1 to 4.

Images