JP2023134346A - Imaging apparatus and method for controlling the same - Google Patents

Abstract

To perform more appropriate distortion correction while improving the quality of a picked-up image.SOLUTION: An imaging apparatus has: an imaging unit that has an imaging optical system and an image pick-up device; attitude control means that moves the direction of the imaging unit in a pan direction and/or a tilt direction; shift control means that moves at least one of the imaging optical system and the image pick-up device in a plane parallel to an imaging surface; synchronization control means that, in driving one of the attitude control means and the shift control means, drives the other in synchronization with the drive of the one of the means so as to cancel a change in an imaging range of the imaging unit due to the drive of the one of the means; and determination means that sequentially determines the amount of distortion of a subject in a picked-up image obtained by the imaging unit. The synchronization control means stops the drive when the amount of distortion determined by the determination means becomes equal to or less than a predetermined threshold.SELECTED DRAWING: Figure 1

Description

The present invention relates to imaging control, and particularly to imaging control for reducing subject distortion.

When a subject is not directly facing the camera, such as when photographing a tall building from the ground, distortion occurs in the subject in the captured image. As a technique to solve this problem, the camera and subject are faced directly, and the image sensor (or optical system) is moved (shifted) parallel to the image plane to take the image (hereinafter referred to as "shift control"). It has been known.

Patent Document 1 discloses a technique for detecting the inclination of a subject with respect to a camera using distance measurement data obtained based on imaging, and performing direct facing correction. Further, it is disclosed that the direct facing correction is performed by controlling the rotation of the lens unit and controlling the shift of the image sensor. Further, Patent Document 2 discloses a technique for detecting distortion (angle) of an edge of a subject with respect to the vertical direction and shifting control of an optical system so that the detected distortion is canceled out.

Japanese Patent Application Publication No. 2003-185902 JP2011-059283A

However, in Patent Document 1, the tilt of the subject is detected based on the difference between a plurality of distance measurement data acquired at different image height positions on the image sensor. Therefore, if the distance measurement accuracy is low, the distortion of the subject may not be corrected appropriately. Further, in Patent Document 2, shift control of the optical system is performed to cancel edge distortion, but fluctuations in the imaging range due to shift control are not taken into consideration. Therefore, the position of the subject in the captured image may change during the period in which the shift control is being performed, and the quality may deteriorate.

The present invention has been made in view of such problems, and an object of the present invention is to provide a technique that enables more appropriate distortion correction while improving the quality of captured images.

In order to solve the above-mentioned problems, an imaging device according to the present invention has the following configuration. That is, the imaging device is
an imaging unit having an imaging optical system and an imaging element;
Attitude control means for moving the orientation of the imaging unit in a panning direction and/or a tilting direction;
Shift control means for moving at least one of the imaging optical system and the imaging element in a plane parallel to the imaging surface;
When driving one of the attitude control means and the shift control means, synchronization of driving the other in synchronization with the driving of the one so that a change in the imaging range of the imaging unit due to the driving of the one is offset. control means;
determining means for sequentially determining the amount of distortion of the subject in the captured image obtained by the imaging unit;
has
The synchronization control means stops driving when the amount of distortion determined by the determination means becomes less than or equal to a predetermined threshold.

According to the present invention, it is possible to provide a technique for performing more appropriate distortion correction while improving the quality of a captured image.

FIG. 1 is a block diagram illustrating the configuration of an imaging system. FIG. 3 is a diagram illustrating subject distortion. It is a figure explaining shift control. FIG. 6 is a diagram illustrating control for reducing subject distortion while maintaining the imaging range. It is a figure explaining drive amount calculation of rotation control and shift control. FIG. 6 is a diagram showing a temporal change in a captured image during distortion correction. FIG. 3 is a diagram illustrating a user interface for distortion correction. It is a figure explaining the control method in 1st synchronous control. It is a flowchart of the process in 1st synchronous control. It is a figure explaining the control method in 2nd synchronous control. It is a flowchart of the process in 2nd synchronous control. It is a flowchart of the process performed by the imaging device concerning a modification. FIG. 3 is a diagram illustrating distortion shape determination based on depth information. FIG. 3 is a diagram illustrating distortion shape determination based on edge detection. FIG. 6 is a diagram illustrating distortion shape determination when a plurality of subjects exist. FIG. 7 is a diagram illustrating correction target determination when a plurality of subjects exist. FIG. 7 is a diagram showing the relationship between rotation control and shift control for each distortion shape. FIG. 3 is a diagram illustrating distortion amount calculation based on depth information. FIG. 3 is a diagram illustrating distortion amount calculation based on edge detection. FIG. 2 is a diagram illustrating the hardware configuration of an imaging device.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

(First embodiment)
As a first embodiment of the imaging device according to the present invention, an imaging system will be described below as an example.

<System configuration>
FIG. 1 is a block diagram illustrating the configuration of an imaging system. The imaging system includes an imaging device and a monitoring device 111. The imaging device includes a camera unit 100 as an imaging section, an auto gain controller (AGC) 107, an analog-to-digital (AD) converter 108, a camera signal processing section 109, and a communication section 110. The imaging device further includes a drive amount determining section 112, a shift control section 113, a rotation control section 114, a shift drive section 115, and a rotation drive section 116.

The camera unit 100 includes an imaging optical system, a bandpass filter (BPF) 104, a color filter 105, and an image sensor 106. Here, the imaging optical system includes a zoom lens 101 that moves in the optical axis direction to change the focal length, a focus lens 102 that moves in the optical axis direction to adjust the focus, and an aperture unit 103 that adjusts the amount of light. Note that the imaging optical system may be configured integrally with the imaging device, or may be configured to be detachable from the imaging device.

The light that has passed through the imaging optical system forms a subject image as an optical image on the imaging element 106 via the BPF 104 and the color filter 105. The BPF 104 may be inserted into and removed from the optical path of the imaging optical system. The subject image is photoelectrically converted by the image sensor 106.

The analog electrical signal (imaging signal) output from the image sensor 106 is gain-adjusted by the AGC 107, converted into a digital signal by the AD converter 108, and then input to the camera signal processing unit 109. The camera signal processing unit 109 performs various image processing on the digital imaging signal to generate a video signal.

The video signal is output to a monitoring device 111 connected to the imaging device via wired or wireless communication via the communication unit 110. The monitoring device 111 also outputs control signals such as commands to the shift control section 113 and the rotation control section 114 via the communication section 110 in response to instructions from the user.

The drive amount determining unit 112 determines the drive amount of the other based on either the rotational drive amount when controlling the rotation of the camera unit 100 or the shift drive amount when shifting the image sensor 106. . Rotation control is posture control related to rotational movement in the pan direction and/or tilt direction. At this time, the determined drive amount is the other drive amount that performs a correction operation to offset a change in the imaging range caused by one (rotation control or shift control). That is, the other driving amount is determined so as to suppress a change in the position of the subject in the captured image. Details of the drive amount determination by the drive amount determination unit 112 will be described later with reference to FIGS. 2 to 5.

The shift control unit 113 instructs the image sensor 106 to shift drive based on the drive instruction (shift drive amount) determined by the drive amount determination unit 112 or received via the communication unit 110. The rotation control unit 114 instructs the camera unit 100 to rotate based on the rotational drive amount determined by the drive amount determination unit 112 or instructed via the communication unit 110.

The shift drive unit 115 drives the image sensor 106 based on the shift drive instructed by the shift control unit 113. The rotation drive unit 116 drives the camera unit 100 based on the rotation drive instructed by the rotation control unit 114. Here, the shift control unit 113 and the shift driving unit 115 instruct and drive the image sensor 106 to shift, but may also instruct and drive the optical system to shift.

<Shift control to compensate for subject distortion>
FIG. 2 is a diagram illustrating subject distortion. Here, a schematic diagram of a case where a building is photographed looking up from the ground and an image taken at that time are exemplarily shown. When the camera unit is not directly facing the subject 201 (that is, when the subject 201 (building wall) and the optical axis 204 are not perpendicular), the building is distorted into a trapezoid shape as shown in the captured image 200. (subject distortion).

FIG. 3 is a diagram illustrating shift control. Here, a schematic diagram in which the same subject 201 (building) as in FIG. 2 is directly faced and photographed by shift control, and a captured image at that time are exemplarily shown. Specifically, the optical axis 204 is perpendicular to the object 201 (the wall of the building), and the imaging surface 203 is adjusted (relative to the optical system 202) so that the object 201 is within the imaging range. It's shifting down. Through this shift control, as shown in the captured image 300, the object distortion is compensated for.

However, in order to capture an image such as the captured image 300 in which subject distortion is compensated for, it is necessary to appropriately control the camera unit 100 and the image sensor 106. Information is needed.

FIG. 4 is a diagram illustrating control for reducing subject distortion while maintaining the imaging range. Specifically, in the photographing state shown in FIG. 2, the imaging range is controlled while controlling the rotation of the camera unit (rotation amount α) so that the optical axis 204 approaches perpendicular to the object 201 (the wall of the building). This shows a state in which shift control (shift amount x) is performed to maintain the state.

That is, when the camera unit 100 is rotated (tilted) by a small amount of rotation α, the imaging range changes from the original imaging range 400 to the imaging range 401 when the camera unit is rotated α. For example, in the image captured by the imaging range 401, the ground surface is captured from the bottom 1/3, and the top of the building is missing (cut off). In order to eliminate such a change in the imaging range (maintain the imaging range 400), the imaging element 106 may be shifted downward within the imaging plane by the shift amount x.

FIG. 5 is a diagram illustrating drive amount calculation for rotation control and shift control. Specifically, a method of calculating the shift amount x of the image sensor 106 according to the rotation amount α of the camera unit 100 in order to maintain the imaging range as shown in FIG. 4 is explained.

Triangle OAB shown in FIG. 5 is an enlarged view of triangle OAB shown in FIG. 4. Here, α is the rotation amount of the camera unit 100, α ₀ is the vertical angle of view/2, x is the shift amount of the image sensor 106, x ₀ is the sensor vertical size/2, and l is the imaging position. At this time, the shift amount x of the image sensor when the camera unit 100 is rotated α is expressed by equation (1).

Here, the relationship of formula (2) holds true.

From formulas (1) and (2), when the camera unit 100 is rotated by α, the shift amount x of the image sensor to eliminate changes in the imaging range can be calculated using the sensor vertical size and the imaging position, using the formula ( 3).

That is, by performing rotation control and shift control using the relationship in equation (3), it is possible to reduce subject distortion. Furthermore, it is possible to suppress changes in the imaging range (that is, maintain the imaging range) during correction control (rotation control and shift control).

FIG. 6 is a diagram showing temporal changes in captured images during distortion correction. For example, it shows changes in the captured image displayed on the display unit of the monitoring device 111 from the start time to the completion time of the correction control (rotation control and shift control).

At the start of imaging (before the start of correction control), there is a distortion (hereinafter referred to as "upper distortion") in which the upper part of the building that is the object is shorter than the lower part. In this state, shift control of the image sensor 106 synchronized with the rotation control of the camera unit 100 is started using the relationship of formula (3). As a result, the distortion occurring in the subject is gradually corrected over time while the relative position of the subject in the captured image is maintained. Correction for this upper distortion will be referred to as "upper correction" below. Similarly, corrections for the lower part distortion, right part distortion, and left part distortion are called lower part correction, right part correction, and left part correction, respectively.

FIG. 7 is a diagram illustrating a user interface (UI) for distortion correction. For example, this UI may be provided in the monitoring device 111 as a physical button or a GUI on the display. The user determines which distortion correction (top correction, bottom correction, right correction, left correction) to perform according to the state of distortion of the subject in the captured image displayed on the display unit of the monitoring device 111, Press the corresponding button on the UI. As a result, a user instruction (drive instruction, drive start instruction, drive stop instruction) including information on the corresponding shift direction is transmitted.

If the captured image is as shown in FIG. 6, "upper part correction" is performed. Control of the camera unit 100 and the image sensor 106 is started based on equation (3) when the user starts pressing the button, and stops when the user finishes pressing the button.

The user can accurately correct the distortion of the subject by pressing the button while checking the temporally changing captured image as shown in FIG. For example, if the pressing time of the "upper correction" button is too long and overcorrection occurs, it is preferable to press the "lower correction" button.

<Effect>
Calculation of the shift amount expressed by the above-mentioned formula (3) does not require information such as the elevation angle of the imaging device or the subject distance. Specifically, the information necessary to calculate the shift amount using Equation (3) is the rotation amount of the camera unit 100, the sensor vertical size, and the imaging position. These pieces of information are known information for each imaging device or information obtained from control information. In this way, the shift control described above is useful in that it is not affected by the detection accuracy of information such as elevation angle and object distance.

Furthermore, as described with reference to FIG. 6, by synchronously controlling the rotation of the lens unit 100 and controlling the shift of the image sensor 106, it is possible to suppress changes in the imaging range during correction control. . That is, during the correction control, the subject position relative to the imaging range does not change, and only the distortion of the subject is corrected, so the quality is good.

<Operation of imaging device>
The synchronization control between the rotation control of the camera unit 100 and the shift control of the image sensor 106 described above will be further explained below. In particular, two different synchronous control methods will be explained.

<First synchronous control>
FIG. 8 is a diagram illustrating a control method in the first synchronous control. The first synchronous control is a method in which rotation control instructions for one step (predetermined amount) of the camera unit 100 are sequentially received, the shift amount of the image sensor 106 is sequentially determined, and synchronous control is performed. That is, the control target positions of the camera unit 100 and the image sensor 106 are specified in detail, and the drives of the two are synchronized.

FIG. 8 shows the relationship between the amount of rotation of the camera unit 100 and the amount of shift of the image sensor 106 under certain imaging conditions. Here, the amount of rotation of the camera unit 100 in one step is α', the amount of shift x of the image sensor 106 is calculated according to the amount of rotation from the reference position, and control is performed based on the calculation result.

FIG. 9 is a flowchart of processing in the first synchronous control. The following operations are performed by each section of the imaging device.

In step S900, the imaging apparatus obtains x ₀ (sensor vertical size/2), which is necessary to calculate the shift amount of the imaging element 106 based on equation (3).

In step S901, the imaging device determines whether the distortion correction button (FIG. 7) has been pressed by a user operation. For example, the determination is made based on whether a control signal transmitted from the monitoring device 111 and input via the communication unit 110 indicates that the distortion correction button has been pressed. If the button has been pressed, the process advances to step S902; if the button has not been pressed, the determination continues.

Of the four types of buttons shown in FIG. 7, one suitable for the shape of the distortion to be corrected is selectively pressed by the user's operation. The driving direction (shift direction) of the camera unit 100 and the image sensor 106 is uniquely determined by the type of correction selected here. For example, when the upper correction button is pressed, the rotation control is controlled in the downward tilt direction and the shift control is controlled in the downward direction. Further, when the lower correction is pressed, control is performed in the opposite direction to the upper correction. Furthermore, when the left correction button is pressed, the rotation control is controlled in the right direction for panning, and the shift control is controlled in the right direction. When the right correction is pressed, control is performed in the opposite direction to the left correction.

In step S902, the imaging device acquires the imaging position l. The imaging position l is the imaging distance between the OBs shown in FIG. 5, and indicates the imaging distance between the optical system 202 and the imaging surface 203. The imaging position l can be obtained by the focus lens position used to control the focus lens. It is only necessary to acquire the imaging position l once from when the user presses the distortion correction button until the press stops.

In step S903, the drive amount determination unit 112 sets the rotation amount α' of the camera unit 100 for one step. The smaller the rotation amount α' per step, the more effective the imaging range is maintained.

In step S904, the drive amount determination unit 112 sets the rotation target position α of the camera unit 100 in the rotation control unit 114. The rotation target position α is set based on the one-step rotation amount α' set in S903. The rotational target position α of the camera unit 100 is the amount obtained by adding one step of rotation amount α' each time the camera unit 100 rotates by one step with the rotational position of the camera unit 100 at the start of distortion correction as a reference.

In step S905, the drive amount determination unit 112 determines the shift target position x of the image sensor 106 based on the rotation target position α of the camera unit 100 set in S904. This determination is made based on formula (3), and the values obtained or set in S900, S902, and S904 are used for the necessary x ₀ , l, and α, respectively.

In step S906, the drive amount determination unit 112 sets the shift target position x of the image sensor 106 calculated in S905 to the shift control unit 113.

In step S907, the rotation control section 114 and the shift control section 113 control the rotation drive section 116 and the shift drive section 115, respectively, to drive the camera unit 100 and the image sensor 106. That is, the driving is performed based on the rotation target position α of the camera unit 100 set in S904 and the shift target position x of the image sensor 106 set in S906.

In step S908, the rotation control unit 114 and the shift control unit 113 each determine whether the control position of either the rotation drive of the camera unit 100 or the shift drive of the image sensor 106 has reached the mechanical drive end. . If it is determined that either of them has reached the drive end, the distortion correction control is ended. If neither has reached the drive end, the process advances to S909.

In step S909, the imaging device determines whether or not the distortion correction button has been pressed. For example, similarly to S901, the determination is made based on whether the control signal transmitted from the monitoring device 111 and input via the communication unit 110 indicates that the distortion correction button has been pressed. If the button has been pressed, the distortion correction control is ended; if the button has not been pressed, the process returns to S904 and the distortion correction control is continued.

If the distortion correction button continues to be pressed, by repeating the processes of S904 to S908, distortion correction can be performed while maintaining the relative position of the subject with respect to the imaging range. Here, the shift drive amount of the image sensor 106 is calculated in response to one step rotation control of the camera unit 100, but the shift drive amount of the image sensor 106 for one step is set, and the shift drive amount The amount of rotational drive may be calculated according to. Further, the shift control may be performed on the optical system 203 instead of the image sensor 106.

As described above, in the first synchronous control, it is possible to synchronize the driving of the camera unit 100 and the image sensor 106 while specifying the control target positions of the two in detail.

<Second synchronous control>
FIG. 10 is a diagram illustrating a control method in the second synchronous control. In the second synchronous control, the drive speeds of the rotation control of the camera unit 100 and the shift control of the image sensor 106 are specified at the start of the control, thereby synchronizing the control of both.

FIG. 11 is a flowchart of processing in the second synchronous control. The following operations are performed by each section of the imaging device. Note that S900 to S902, S907, and S909 are the same processes as the first synchronous control, so the explanation will be omitted.

In step S1100, the imaging device obtains the shift movement amount x _lim of the imaging element 106. The shift movable amount x _lim is the distance from the current position of the image sensor 106 to the mechanical drive end.

In step S1101, the imaging device obtains the amount of rotational movement α _lim of the camera unit 100. The rotational movable amount α _lim is the angle from the current position of the camera unit 100 to the mechanical drive end.

In step S1102, the drive amount determination unit 112 calculates the shift maximum drive amount x _max of the image sensor 106. The maximum shift drive amount x _max is the shift drive of the image sensor 106 necessary to maintain the relative position of the subject with respect to the imaging range when rotating the camera unit 100 by the rotational movement amount α _lim acquired in S1101. It's the amount. The maximum shift drive amount x _max is calculated based on Equation (3).

In step S1103, the drive amount determining unit 112 compares the shift movable amount x _lim calculated in S1100 with the shift maximum drive amount x _max calculated in S1102. If x _lim > x _max , the process advances to S1104. If x _lim > x _max , when the camera unit 100 is controlled to the mechanical drive end, even if the shift control of the image sensor 106 necessary to maintain the relative position of the subject is performed, the image sensor 106 is not mechanically driven. This is a case where the end is not reached. On the other hand, if x _lim ≦x _max , the process advances to S1106. If x _lim ≦x _max , when the camera unit 100 is controlled to the mechanical drive end, the image sensor 106 hits the mechanical drive end, and the relative position of the subject cannot be maintained.

In step S1104, the drive amount determination unit 112 sets the shift target position of the image sensor 106 to a position obtained by adding the shift maximum drive amount x _max calculated in S1102 to the current position x _now of the image sensor 106.

In step S1105, the drive amount determination unit 112 sets the rotational target position of the camera unit 100 to a position obtained by adding the rotational movable amount α _lim acquired in S1101 to the current position α _now of the camera unit 100.

In step S1106, the drive amount determination unit 112 calculates the rotational maximum drive amount α _max of the camera unit 100. The maximum rotational drive amount α _max is the rotational drive of the camera unit 100 necessary to maintain the relative position of the subject with respect to the imaging range when controlling the shift of the image sensor 106 by the shift movement amount x _lim acquired in S1100. It's the amount. The maximum rotational drive amount α _max is calculated based on Equation (3).

In step S1107, the drive amount determining unit 112 sets the shift target position of the image sensor 106 to a position obtained by adding the shift movable amount x _lim obtained in S1100 to the current position x _now of the image sensor 106.

In step S1108, the drive amount determination unit 112 sets the rotational target position of the camera unit 100 to a position obtained by adding the maximum rotational drive amount α _max calculated in S1106 to the current position α _now of the camera unit 100.

In step S<b>1109 , the drive amount determination unit 112 sets the rotation speed v _r of the camera unit 100 to the rotation control unit 114 . Note that the slower the rotation control speed v _r is, the easier it is for the user to make fine adjustments when instructing the start and end of distortion correction.

In step S1110, the drive amount determining unit 112 sets the shift speed _vs of the image sensor 106 in the shift control unit 113. At this time, the shift speed _vs is set so that the camera unit 100 and the image sensor 106 reach the target position at the same time (that is, the drive time becomes the same).

In step S1111, the rotation control section 114 and the shift control section 113 determine whether the camera unit 100 and the image sensor 106 have reached the target positions, respectively. If both have been reached, the distortion correction control ends. If at least one of them has not been reached, the process advances to S909.

In step S1112, the imaging device performs processing to stop distortion correction. The stop process is a process in which the image sensor 106 is shifted to a shift position where the imaging range is maintained with respect to the direction of the camera unit 100 at the timing when the distortion correction end instruction was received, and then the control is ended.

If the linearity between the amount of rotation of the camera unit 100 and the amount of shift of the image sensor 106 is low, the relative position of the subject with respect to the imaging range may not be completely maintained at a point on the way to the target position. Therefore, if the distortion correction ends before the control target position due to the user's operation, the imaging range is adjusted by the stop process in S1112.

As described above, in the first synchronous control, by specifying the driving speeds for controlling the camera unit 100 and the image sensor 106 at the start of control, it is possible to synchronize the driving of both.

Both the first synchronization control and the second synchronization control described above can synchronize the camera unit 100 and the image sensor 106. However, the first synchronous control is advantageous over the second synchronous control in that it is more effective in maintaining the relative position of the subject during distortion correction control. On the other hand, the second synchronous control requires fewer calculations and fewer drives (one time) than the first synchronous control, so it is advantageous under imaging conditions that require high-speed control.

<Hardware configuration of imaging device>
Next, an example of the hardware configuration of the imaging device will be described using the block diagram of FIG. 20. Note that the configuration shown in FIG. 20 is only an example of a configuration applicable to the imaging device, and can be modified/changed as appropriate.

The CPU 211 executes processing using computer programs and data stored in the main storage device 212. Thereby, the CPU 211 controls the operation of the entire imaging apparatus, and also executes or controls each of the processes described above as being performed by the imaging apparatus. For example, the CPU 211 executes processing using computer programs and data stored in the main storage device 212, thereby controlling the camera signal processing section 109, drive amount determining section 112, shift control section 113, and rotation control section shown in FIG. The functions of each functional unit of the control unit 114 are realized.

The main storage device 212 is a storage device such as Random Access Memory (RAM). The main storage device 212 has an area for storing computer programs and data loaded from the auxiliary storage device 213, images captured by the imaging unit 215, and various data received from the monitoring device 111 via the communication unit 110. . Furthermore, the main storage device 212 has a work area used when the CPU 211 executes various processes. In this way, the main storage device 212 can provide various areas as appropriate.

The auxiliary storage device 213 is a large-capacity information storage device such as a hard disk drive (HDD), a read only memory (ROM), or a solid state drive (SSD). The auxiliary storage device 213 stores an OS (operating system) and computer programs and data for causing the CPU 211 to execute or control each of the processes described above as being performed by the imaging apparatus. The auxiliary storage device 213 also stores data received from the monitoring device 111 via the communication unit 110. Computer programs and data stored in the auxiliary storage device 213 are loaded into the main storage device 212 as appropriate under the control of the CPU 211, and are subject to processing by the CPU 211.

The drive unit 214 drives the camera unit 100 based on the imaging parameters received from the monitoring device 111. For example, it corresponds to a shift drive unit 115 that shifts and controls the image sensor 106 and a rotation drive unit 116 that controls rotation of the camera unit 100. Note that the object to be controlled by the drive unit 214 is not limited to a specific object, but may be another object (for example, the position of the camera unit 100).

The camera unit 100 includes an image sensor and an optical system, and forms an image of a subject on the image sensor with the intersection of the optical axis of the optical system and the image sensor as the imaging center. Examples of the imaging device include CMOS (Complementary Metal-Oxide Semiconductor) and CCD (Charged Coupled Device). The communication unit 110 performs data communication with the monitoring device 111.

As described above, according to the first embodiment, when performing shift control of the image sensor, rotation control (tilt or pan) of the camera unit is performed in synchronization with the shift control of the image sensor. More specifically, rotation control is performed synchronously so as to offset changes in the imaging range caused by shift control. This synchronous control makes it possible to suppress changes in the imaging range during correction control, making it possible to provide high-quality images.

(Modified example)
In the modified example, an example will be described in which the imaging device determines the shape of distortion occurring in the subject based on the captured image, and controls the rotation of the camera unit 100 and the shift of the image sensor 106 so that the shape of the distortion is corrected. do.

FIG. 12 is a flowchart of processing executed by the imaging device according to the modification.

In step S1200, the imaging apparatus determines whether there is a distortion correction execution instruction from the user. If it is determined that there is a distortion correction instruction from the user, the process advances to S1201. On the other hand, if it is determined that there is no instruction, the CPU continues to wait for a distortion correction instruction.

In step S1201, the imaging apparatus determines whether a distorted subject (hereinafter referred to as a distorted subject) exists in the captured image. If it is determined that a distorted subject exists, the process advances to S1202, and if it is determined that a distorted subject does not exist, the process advances to S1203. As methods for determining the presence or absence of a distorted subject, a method based on depth information and a method based on edge detection will be described below, but other methods may be used to determine the presence or absence of a distorted subject.

FIG. 13 is a diagram illustrating distortion shape determination based on depth information. In a method for determining the presence or absence of a distorted subject based on depth information (such as a depth image in which each pixel has a depth value at that position), if the subject distance changes monotonically in a certain subject area, the subject is not connected to the imaging device. It is determined that the subject is not facing directly and is distorted.

FIG. 14 is a diagram illustrating distortion shape determination based on edge detection. In a method for determining the presence or absence of a distorted subject based on edge detection, a distorted subject is determined if the edge interval along the vertical direction (or horizontal direction) is not constant regarding a pair of edges detected for a certain subject area. .

In step S1202, the imaging apparatus sets the subject determined to be a distorted subject in step S1201 as a distortion correction target. Note that, depending on the scene, a plurality of distorted objects may exist in the captured image. In this case, it is preferable to set the subject to be corrected by receiving a selection of the subject to be corrected from the user.

FIG. 15 is a diagram illustrating distortion shape determination when a plurality of subjects exist. Further, FIG. 16 is a diagram illustrating correction target determination when a plurality of subjects exist. Specifically, FIG. 16 shows a screen that accepts the selection of a subject to be corrected from the user. FIG. 16 shows a display screen in which a plurality of polygonal regions divided into regions for each object determined to be a distorted subject in S1201 are superimposed and displayed on the captured image, and the selection of one polygonal region is accepted from the user. .

In step S1203, the imaging device notifies the user that there is no distorted subject (subject to be corrected) in the captured image.

In step S1204, the imaging device determines the distortion shape type (upper distortion, lower distortion, right distortion, left distortion) of the subject to be corrected. In the case of distortion shape determination as well, determination based on depth information or determination based on edge detection is possible.

For example, the depth information corresponding to the captured image 1300 in FIG. 13 shows that the depth (subject distance) with respect to the building area of the captured image 1300 changes from near to far from the bottom edge of the image toward the top edge. Due to perspective, the closer the object is, the larger it appears, and the farther it is, the smaller it appears. Therefore, it can be determined that the shape of distortion occurring in the building in the captured image 1300 is "top distortion" (larger in the bottom direction of the image and smaller in the upper direction).

Similarly, the depth information corresponding to the captured image 1301 shows that the depth (subject distance) with respect to the building area of the captured image 1301 changes from near to far from the left end of the image to the right end. Therefore, it can be determined that the shape of distortion occurring in the building in the captured image 1301 is "right-side distortion" (larger on the left side of the image and smaller on the right side). In this way, it is possible to use the depth information to determine the distorted shape based on the direction in which the depth of the object (building) in the captured image is changing.

Furthermore, in the captured image 1400 of FIG. 14, regarding a pair of edges detected for a subject in the captured image, the distance between the pair of edges becomes narrower from the bottom edge to the top edge of the image. There is. In this case, it can be determined that the distorted shape occurring in the subject is "top distortion." Furthermore, in the captured image 1401, regarding a pair of edges detected for a subject in the captured image, the distance between the pair of edges becomes narrower from the left end to the right end of the image. In this case, it can be determined that the shape of distortion occurring in the subject is "right-side distortion." In this way, it is possible to determine the distorted shape of the subject based on the direction in which the edge interval between a pair of edges detected for the subject is changing.

In step S1205, the imaging device sets driving directions for rotation control of the camera unit 100 and shift control of the image sensor 106 suitable for each distortion shape, based on the distortion shape determined in S1204.

FIG. 17 is a diagram showing the relationship between rotation control and shift control for each distorted shape. As shown in FIG. 17, the driving direction of the camera unit 100 and the image sensor 106 for correcting each distortion shape can be uniquely determined.

In step S1206, the imaging device performs distortion correction driving. Here, it is assumed that distortion correction driving (step driving) is performed by a predetermined amount and steps S1206 to S1208 are repeated. Note that this process is similar to the process (S902 to S907) described above with reference to FIG. 9, so a description thereof will be omitted.

In step S1207, the imaging device calculates the current amount of distortion occurring in the subject to be corrected. In the case of calculating the amount of distortion, calculation based on depth information or calculation based on edge detection is possible.

FIG. 18 is a diagram illustrating distortion amount calculation based on depth information. FIG. 18 exemplarily shows changes in the captured image during the distortion correction drive in S1206. The captured image 1800 shows that there is a large difference in depth (subject distance) between the upper and lower parts of the subject to be corrected. In this case, the amount of distortion is calculated as a large value. A captured image 1801 obtained after one or more steps of distortion correction driving has a smaller difference in depth between the upper and lower parts of the subject to be corrected, compared to the captured image 1800. Therefore, the amount of distortion is calculated as a smaller value compared to the captured image 1800. In the captured image 1802 obtained after further performing the distortion correction drive, the depths at the top and bottom of the subject to be corrected are approximately the same. Therefore, the amount of distortion is calculated as a value close to 0. When calculating the amount of distortion based on depth information in this way, it is possible to evaluate the depth difference between multiple regions (for example, upper and lower portions) of the subject to be corrected as the amount of distortion.

FIG. 19 is a diagram illustrating distortion amount calculation based on edge detection. FIG. 19 exemplarily shows changes in the captured image during the distortion correction drive in S1206. The captured image 1900 shows that there is a large difference in the edge spacing between a pair of edges at the top and bottom of the subject to be corrected. In this case, the amount of distortion is calculated as a large value. A captured image 1901 obtained after one or more steps of distortion correction driving has a smaller difference in edge spacing between the upper and lower parts of the subject to be corrected compared to the captured image 1900. Therefore, the amount of distortion is calculated as a smaller value compared to the captured image 1900. In the captured image 1902 obtained after further performing the distortion correction drive, the edge spacing at the top and bottom of the subject to be corrected is approximately the same. Therefore, the amount of distortion is calculated as a value close to 0. When calculating the amount of distortion based on edge detection in this way, it is possible to evaluate the difference in edge spacing between multiple regions (for example, upper and lower portions) of the subject to be corrected as the amount of distortion.

In step S1208, the imaging device determines whether the amount of distortion calculated in S1207 is less than or equal to a predetermined threshold. If the amount of distortion is not below the predetermined threshold, the distortion correction is insufficient and the process returns to S1206. On the other hand, if the amount of distortion is less than or equal to the predetermined threshold, the process advances to S1209.

In step S1209, the imaging device determines whether distortion shape determination (S1204) has been performed twice. This is because depending on the subject, there may be both vertical distortion (upper distortion or lower distortion) and horizontal distortion (right distortion or left distortion) in the captured image. In the correction based on one distortion shape determination (S1204 to S1208), only distortion in either the vertical direction or the horizontal direction is corrected, so it is determined whether or not the distortion shape determination has been performed twice.

As described above, the imaging device is capable of correcting the distortion occurring in the subject of the captured image based on the depth information corresponding to the captured image and the result of edge detection for the captured image.

The disclosure of this specification includes the following imaging device, control method, and program.
(Item 1)
an imaging unit having an imaging optical system and an imaging element;
Attitude control means for moving the orientation of the imaging unit in a panning direction and/or a tilting direction;
Shift control means for moving at least one of the imaging optical system and the imaging element in a plane parallel to the imaging surface;
When driving one of the attitude control means and the shift control means, synchronization of driving the other in synchronization with the driving of the one so that a change in the imaging range of the imaging unit due to the driving of the one is offset. control means;
determining means for sequentially determining the amount of distortion of the subject in the captured image obtained by the imaging unit;
has
The imaging device is characterized in that the synchronization control means stops driving when the amount of distortion determined by the determination means becomes less than or equal to a predetermined threshold.
(Item 2)
The shift control means moves at least one of the imaging optical system and the imaging element in a plane parallel to the imaging plane so that distortion of the subject in the captured image obtained by the imaging unit is compensated. The imaging device according to item 1.
(Item 3)
3. The imaging device according to item 1 or 2, wherein the synchronization control means determines the second drive amount in the other based on the first drive amount in the one.
(Item 4)
Item 3, wherein the synchronization control means further determines the second drive amount based on the size of the image sensor and the imaging distance from the imaging optical system to the image sensor. Imaging device.
(Item 5)
5. The imaging device according to any one of items 1 to 4, further comprising reception means for accepting user instructions for the shift control means.
(Item 6)
The accepting means is configured to sequentially accept drive instructions for driving the shift control means by a predetermined amount,
When the drive instruction is received by the reception means, the synchronization control means drives the shift control means by a predetermined amount based on the drive instruction, and controls the drive amount of the attitude control means in accordance with the drive of the predetermined amount. The imaging device according to item 5, characterized in that the imaging device is sequentially determined and driven.
(Item 7)
The receiving means is configured to receive a drive instruction to drive the shift control means,
The imaging device according to item 5, wherein the synchronization control means determines the drive speed of both the attitude control means and the shift control means and starts driving when the reception means receives the drive instruction. .
(Item 8)
An item characterized in that the synchronization control means determines the drive speed so that the drive time of both the attitude control means and the shift control means becomes the same when the drive instruction is received by the reception means. 7. The imaging device according to 7.
(Item 9)
The reception means is configured to further receive a drive stop instruction to stop the drive of the shift control means,
When the reception means receives the drive stop instruction while driving the attitude control means and the shift control means, the synchronization control means controls the orientation of the imaging unit at the timing when the drive stop instruction is received and 9. The imaging device according to item 7 or 8, wherein the other is adjusted based on one of the shift positions of the imaging element.
(Item 10)
10. The imaging device according to any one of items 6 to 9, wherein the drive instruction includes information on a shift direction of the imaging element by the shift control means.
(Item 11)
further comprising receiving means for receiving a drive start instruction to repeatedly drive the shift control means by a predetermined amount,
5. The imaging apparatus according to any one of items 1 to 4, wherein the shift control means starts driving when the reception means receives the drive start instruction.
(Item 12)
further comprising depth acquisition means for acquiring depth information corresponding to the captured image obtained by the imaging unit,
12. The imaging device according to item 11, wherein the determining means determines the distortion amount based on a difference in depth within a subject area included in the captured image.
(Item 13)
further comprising edge detection means for detecting edges of the subject included in the captured image obtained by the imaging unit,
12. The imaging device according to item 11, wherein the determining means determines the distortion amount based on a difference in edge spacing between a pair of edges detected for the subject by the edge detecting means.
(Item 14)
further comprising determining means for determining the type of distortion shape of the subject in the captured image obtained by the imaging unit,
The imaging device according to any one of items 1 to 4, wherein the driving direction in each of the attitude control means and the shift control means is determined based on the distortion shape type determined by the determination means. .
(Item 15)
further comprising depth acquisition means for acquiring depth information corresponding to the captured image obtained by the imaging unit,
15. The imaging device according to item 14, wherein the determining means determines the distortion shape type based on a direction of change in depth within a subject area included in the captured image.
(Item 16)
further comprising edge detection means for detecting edges of the subject included in the captured image obtained by the imaging unit,
Imaging according to item 14, wherein the determining means determines the distortion shape type based on a direction of change in an edge interval between a pair of edges detected for the subject by the edge detecting means. Device.
(Item 17)
Any one of items 14 to 16, further comprising accepting means for accepting selection of one subject to be determined by the determining means when there are a plurality of distorted subjects in the captured image. The imaging device described in item 1.
(Item 18)
A method for controlling an imaging device, the method comprising:
The imaging device includes:
an imaging unit having an imaging optical system and an imaging element;
Attitude control means for moving the orientation of the imaging unit in a panning direction and/or a tilting direction;
Shift control means for moving at least one of the imaging optical system and the imaging element in a plane parallel to the imaging surface;
has
The control method includes:
a determining step of sequentially determining the amount of distortion of the subject in the captured image obtained by the imaging unit;
When driving one of the attitude control means and the shift control means, synchronization of driving the other in synchronization with the driving of the one so that a change in the imaging range of the imaging unit due to the driving of the one is offset. control process;
including;
A control method characterized in that, in the synchronization control step, driving of the attitude control means and the shift control means is stopped when the amount of distortion determined in the determination step becomes equal to or less than a predetermined threshold value.
(Item 19)
A program for causing a computer to execute a control method for an imaging device,
The imaging device includes:
an imaging unit having an imaging optical system and an imaging element;
Attitude control means for moving the orientation of the imaging unit in a panning direction and/or a tilting direction;
Shift control means for moving at least one of the imaging optical system and the imaging element in a plane parallel to the imaging surface;
has
The control method includes:
a determining step of sequentially determining the amount of distortion of the subject in the captured image obtained by the imaging unit;
When driving one of the attitude control means and the shift control means, synchronization of driving the other in synchronization with the driving of the one so that a change in the imaging range of the imaging unit due to the driving of the one is offset. control process;
including;
A program characterized in that, in the synchronization control step, driving of the attitude control means and the shift control means is stopped when the amount of distortion determined in the determination step becomes equal to or less than a predetermined threshold value.

(Other examples)
The present invention provides a system or device with a program that implements one or more of the functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the following claims are hereby appended to disclose the scope of the invention.

100 Camera unit; 101 Zoom lens; 102 Focus lens; 103 Aperture unit; 104 Bandpass filter; 105 Color filter; 106 Image sensor; 107 AGC; 108 AD converter; 109 Camera signal processing section; 110 Communication section; 111 Monitoring device 112 Drive amount determining section; 113 Shift control section; 114 Rotation control section; 115 Shift drive section; 116 Rotation drive section

Claims (19)

an imaging unit having an imaging optical system and an imaging element;
Attitude control means for moving the orientation of the imaging unit in a panning direction and/or a tilting direction;
Shift control means for moving at least one of the imaging optical system and the imaging element in a plane parallel to the imaging surface;
When driving one of the attitude control means and the shift control means, synchronization of driving the other in synchronization with the driving of the one so that a change in the imaging range of the imaging unit due to the driving of the one is offset. control means;
determining means for sequentially determining the amount of distortion of the subject in the captured image obtained by the imaging unit;
has
The imaging device is characterized in that the synchronization control means stops driving when the amount of distortion determined by the determination means becomes less than or equal to a predetermined threshold. The shift control means moves at least one of the imaging optical system and the imaging element in a plane parallel to the imaging plane so that distortion of the subject in the captured image obtained by the imaging unit is compensated. The imaging device according to claim 1. 3. The imaging device according to claim 2, wherein the synchronization control means determines the second drive amount for the other based on the first drive amount for the one. 4. The synchronization control means further determines the second drive amount based on the size of the image pickup device and the imaging distance from the image pickup optical system to the image pickup device. imaging device. 5. The imaging apparatus according to claim 4, further comprising reception means for accepting user instructions for said shift control means. The accepting means is configured to sequentially accept drive instructions for driving the shift control means by a predetermined amount,
When the drive instruction is received by the reception means, the synchronization control means drives the shift control means by a predetermined amount based on the drive instruction, and controls the drive amount of the attitude control means in accordance with the drive of the predetermined amount. The imaging device according to claim 5, wherein the imaging device is sequentially determined and driven. The receiving means is configured to receive a drive instruction to drive the shift control means,
The imaging according to claim 5, wherein the synchronization control means determines the drive speed of both the attitude control means and the shift control means and starts driving when the drive instruction is received by the reception means. Device. The synchronization control means determines the drive speed so that the drive time of both the attitude control means and the shift control means becomes the same when the drive instruction is received by the reception means. The imaging device according to item 7. The reception means is configured to further receive a drive stop instruction to stop the drive of the shift control means,
When the reception means receives the drive stop instruction while driving the attitude control means and the shift control means, the synchronization control means controls the orientation of the imaging unit at the timing when the drive stop instruction is received and The imaging device according to claim 8, wherein the other is adjusted based on one of the shift positions of the imaging element. The imaging device according to claim 9, wherein the drive instruction includes information on a shift direction of the imaging element by the shift control means. further comprising receiving means for receiving a drive start instruction to repeatedly drive the shift control means by a predetermined amount,
The imaging apparatus according to claim 1, wherein the shift control means starts driving when the reception means receives the drive start instruction. further comprising depth acquisition means for acquiring depth information corresponding to the captured image obtained by the imaging unit,
The imaging apparatus according to claim 11, wherein the determining unit determines the distortion amount based on a difference in depth within a subject area included in the captured image. further comprising edge detection means for detecting edges of the subject included in the captured image obtained by the imaging unit,
The imaging apparatus according to claim 11, wherein the determining means determines the distortion amount based on a difference in edge spacing between a pair of edges detected for the subject by the edge detecting means. . further comprising determining means for determining the type of distortion shape of the subject in the captured image obtained by the imaging unit,
2. The imaging apparatus according to claim 1, wherein driving directions in each of the attitude control means and the shift control means are determined based on the distortion shape type determined by the determination means. further comprising depth acquisition means for acquiring depth information corresponding to the captured image obtained by the imaging unit,
15. The imaging apparatus according to claim 14, wherein the determining means determines the distortion shape type based on a direction of change in depth within a subject area included in the captured image. further comprising edge detection means for detecting edges of the subject included in the captured image obtained by the imaging unit,
15. The determining means determines the distortion shape type based on a direction of change in an edge interval between a pair of edges detected for the subject by the edge detecting means. Imaging device. 15. The imaging system according to claim 14, further comprising accepting means for accepting selection of one subject to be determined by the determining means when there are a plurality of subjects with distortion in the captured image. Device. A method for controlling an imaging device, the method comprising:
The imaging device includes:
an imaging unit having an imaging optical system and an imaging element;
Attitude control means for moving the orientation of the imaging unit in a panning direction and/or a tilting direction;
Shift control means for moving at least one of the imaging optical system and the imaging element in a plane parallel to the imaging surface;
has
The control method includes:
a determining step of sequentially determining the amount of distortion of the subject in the captured image obtained by the imaging unit;
When driving one of the attitude control means and the shift control means, synchronization of driving the other in synchronization with the driving of the one so that a change in the imaging range of the imaging unit due to the driving of the one is offset. control process;
including;
A control method characterized in that, in the synchronization control step, driving of the attitude control means and the shift control means is stopped when the amount of distortion determined in the determination step becomes equal to or less than a predetermined threshold value. A program for causing a computer to execute a control method for an imaging device,
The imaging device includes:
an imaging unit having an imaging optical system and an imaging element;
Attitude control means for moving the orientation of the imaging unit in a panning direction and/or a tilting direction;
Shift control means for moving at least one of the imaging optical system and the imaging element in a plane parallel to the imaging surface;
has
The control method includes:
a determining step of sequentially determining the amount of distortion of the subject in the captured image obtained by the imaging unit;
When driving one of the attitude control means and the shift control means, synchronization of driving the other in synchronization with the driving of the one so that a change in the imaging range of the imaging unit due to the driving of the one is offset. control process;
including;
A program characterized in that, in the synchronization control step, driving of the attitude control means and the shift control means is stopped when the amount of distortion determined in the determination step becomes equal to or less than a predetermined threshold value.

Images