JP5404160B2 - Imaging system and imaging method - Google Patents

Description

  The present invention relates to an imaging system and an imaging method.

  Conventionally, Patent Document 1 discloses a camera that performs imaging by tilting a lens when performing imaging while focusing on the entire area of an object that is inclined with respect to an imaging surface.

  In Patent Document 1, imaging is performed by tilting the lens so that the extension line of the main surface of the lens passes through the intersection of the extension line of the imaging surface and the extension line of the subject surface in accordance with the Scheinproof law. Therefore, such a camera is provided with a mechanism for tilting the lens.

JP 2000-056214 A

  Usually, the time from focusing to shooting is short. For this reason, in the above-described camera, when there is a difference in the posture of the camera with respect to the subject between the in-focus state and the shooting time (shooting moment), it is difficult to tilt the lens in accordance with the difference. As a result, there is a problem that it is difficult to obtain an image that is in focus on the entire captured image.

  The present invention was invented in order to solve such problems, and an object thereof is to obtain an image in which the entire captured image is in focus.

An image pickup system according to an aspect of the present invention includes an image pickup device that has a plurality of pixels and forms an image of a subject, a turn detection unit that detects a turn amount generated during shooting, and an image obtained by the image pickup device. An image processing unit that performs image processing based on the amount of turning of the data, and the image pickup device is a position of the image pickup device when a release button is pressed by a predetermined amount and an operation for focusing is performed during shooting. The rotation detection unit rotates with respect to the reference position about the inclination center, and the turn detection unit detects a plurality of pixels according to the inclination of the image sensor at the time of shooting with respect to the reference position and the distance from each inclination center of the plurality of pixels. The amount of twist is calculated for each.

According to another aspect of the present invention, an imaging method includes a plurality of pixels, captures a subject image with an imaging element on which an image of the subject is formed, detects the amount of wobbling that occurs during shooting, and obtains the image by the imaging element. A step of performing image processing on the obtained image data based on the amount of turning, and the image pickup device is a position of the image pickup device when a release button is pressed by a predetermined amount and an operation for focusing is performed during shooting. The rotation amount is rotated around the tilt center with respect to the reference position, and the amount of turning is determined for each of the plurality of pixels according to the tilt of the imaging element at the time of shooting with respect to the reference position and the distance from the tilt center of each of the plurality of pixels. Is calculated.

  According to these aspects, even when a blur occurs during imaging, an image focused on the entire area of the subject can be obtained by performing the image restoration process based on the blur amount. Therefore, for example, an image focused on the entire area of the subject can be obtained without using a mechanism for tilting the lens.

  According to the present invention, it is possible to obtain an image that is in focus over the entire captured image.

1 is a schematic block diagram of an imaging system according to a first embodiment. It is a flowchart which shows the control at the time of imaging | photography of 1st Embodiment. It is a graph which shows the relationship between the amount of curling and the amount of image restoration. It is a map explaining the relationship between the tilt of the camera and the image recovery amount. It is a schematic block diagram of the imaging system of 2nd Embodiment. It is a figure which shows the relationship between the amount of curling and the amount of defocusing. It is a flowchart which shows the control at the time of imaging | photography of 2nd Embodiment. It is a graph which shows the relationship between a defocus and MTF. It is a flowchart which shows the control at the time of imaging | photography of 3rd Embodiment. It is a schematic block diagram of the imaging system of 4th Embodiment. It is a flowchart which shows the control at the time of imaging | photography of 4th Embodiment. It is a schematic block diagram of the imaging system of 5th Embodiment.

  A first embodiment of the present invention will be described with reference to FIG. FIG. 1 is a schematic block diagram of an imaging system according to the first embodiment. Here, a digital still camera (hereinafter referred to as a camera) will be described as the imaging device, but the imaging device is not limited to this.

  The camera includes an optical system 1, a turn detection unit 2, an image sensor 3, an image processing unit 4, an image display unit 5, an image storage unit 6, and a control unit 7.

  The optical system 1 includes, for example, a plurality of lenses and a diaphragm. A part of the lens can be moved along the optical axis direction by an actuator or the like, and focus adjustment can be performed.

  The turning detection unit 2 detects the turning amount for each pixel of the image sensor 3. The amount of turning is an amount indicating a deviation of the posture of the camera at the time of shooting with respect to the reference position of the camera. The turn detection unit 2 is, for example, a tilt sensor, an angle sensor, an acceleration sensor, or a gravity sensor, and detects the tilt of the camera. Then, when detecting the tilt of the camera, the shift detection unit 2 calculates the shift amount in each pixel according to the distance from the tilt center of the camera. Here, the distance from the tilt center of the camera is the distance in the direction of arrow A in FIG. The amount of twist is the distance in the direction of arrow B in FIG. The reference position is the posture of the camera when the release button is pressed by a predetermined amount and focused. Even when the camera rotates around an arbitrary point, the camera tilts (posture shift). Therefore, the tilt of the camera can also be expressed as an angle. In FIG. 4A, the center of tilt of the camera is the center of rotation. Then, the deviation from the reference position in the rotation direction becomes the amount of twist when represented by an angle.

  The image sensor 3 outputs an electrical signal corresponding to light incident on the light receiving surface at a predetermined timing. The image pickup device 3 is an image pickup device of a format called a CCD (charge coupled device) or a CMOS (complementary metal oxide semiconductor) sensor, or other various types.

  The image processing unit 4 includes a recovery amount calculation unit 8 and an image recovery unit 9.

  The recovery amount calculation unit 8 calculates an image recovery amount for the electric signal output from the image sensor 3 for each pixel based on the amount of rotation detected by the rotation detection unit 2.

  The image recovery unit 9 performs an image recovery process on the electrical signal output from the image sensor 3 based on the image recovery amount calculated by the recovery amount calculation unit 8.

  In addition to this, the image processing unit 4 performs processing such as white balance adjustment and gradation / level correction. The image data processed by the image processing unit 4 is output to the image display unit 5 and the image storage unit 6.

  The image processing unit 4 includes a CPU, a ROM, a RAM, and the like. The ROM stores a control program and each data. Each function of the image processing unit 4 is exhibited by the CPU executing calculations based on the control program stored in the ROM.

  The image display unit 5 is a color liquid crystal display (LCD) panel or an organic EL (OEL) display panel. The image display unit 5 displays the captured image based on the image signal output from the image processing unit 4.

  The image storage unit 6 stores the image signal output from the image processing unit 4 as image data.

  The control unit 7 is connected to the blur detection unit 2, the image processing unit 4, the image display unit 5, and the image storage unit 6, and controls the entire camera including these.

  Next, control at the time of shooting according to the present embodiment will be described with reference to the flowchart of FIG.

  In step S100, when it is detected that the release button is pressed by a predetermined amount by the photographer (for example, half-pressed state), the optical system 1 is controlled to perform focus adjustment and focus on the subject.

  In step S101, the posture of the camera at the time of focusing is detected and the reference position is determined. Here, the posture of the camera at the time of focusing becomes the reference position. The reference position may be a position where the light receiving surface of the image sensor 3 is parallel to the vertical direction.

  In step S102, when it is detected that the release button is further pressed by the photographer (for example, in the fully pressed state), shooting is performed.

  In step S103, the posture of the camera at the time of shooting (shooting moment) is detected. During the period from focusing to shooting, the posture of the camera may change and the camera may turn. Therefore, in step S103, the posture of the camera at the time of shooting is detected.

  In step S104, the camera tilt angle at the time of shooting with respect to the reference position is detected from the camera posture at the time of focusing and the camera posture at the time of shooting, and the amount of turning at each pixel is calculated from the camera tilt angle.

  In step S105, an image restoration amount is calculated based on the calculated amount of turning. The image restoration amount is calculated using, for example, the graph shown in FIG. FIG. 3 is a graph showing the relationship between the amount of blurring and the amount of image recovery. In FIG. 3, the amount of image restoration increases as the absolute value of the amount of turning increases. In the present embodiment, a graph as shown in FIG. 3 is provided for each pixel. In calculating the image restoration amount, means other than the graph, for example, an equation corresponding to the graph or an approximate equation thereof may be used.

  The relationship between the camera tilt (swing) and the amount of image recovery will be described with reference to FIG. When the camera is turned from the reference position in FIG. 4A around one end (camera tilt center) on the back side of the camera as shown in FIG. 4B, the image recovery amount is As shown in 4 (c). Also, as shown in FIG. 4 (d) from the reference position, when a wrinkle occurs around the center on the back side of the camera (camera tilt center), the image recovery amount is as shown in FIG. 4 (e). In FIGS. 4C and 4E, gradation is given so as to become black as the image recovery amount increases. Thus, the image recovery amount increases as the distance from the camera tilt center increases. In FIGS. 4B and 4D, the camera at the reference position is indicated by a broken line.

  In step S106, based on the calculated image restoration amount, image processing is performed to obtain a restored image in which the influence of blurring is reduced using, for example, the differential equation shown in Equation (1).

f = g + a1 · g ′ + a2 · g ″ +... + an · g ′ (n) Equation (1)
In Equation (1), f is a restored image, g is a captured image, a1 to an are imaging characteristics of the optical system, and an image restoration amount. The method for image restoration processing of a captured image based on the image restoration amount is not limited to this.

  In step S <b> 107, an image that has been subjected to image processing based on the image recovery amount is displayed on the image display unit 5, and image data is stored in the image storage unit 6.

  With the above control, an image with reduced influence of turning can be obtained even when turning occurs during shooting.

  For example, the image processing unit 4 may store the amount of turning, the amount of image restoration, and the like. By storing the amount of turning, etc., when continuous shooting is performed by continuously pressing the release button, the time required for image processing can be shortened and the number of shots per hour can be increased.

  In the present embodiment, the amount of distortion is calculated for each pixel, and the image recovery amount is calculated. However, the light receiving surface of the image sensor 3 may be divided into a plurality of regions, and the image recovery amount may be calculated for each region. The amount of curling is calculated at at least two locations, and the image restoration amount is calculated based on the calculated amount of curling. Then, by performing image processing based on the calculated image restoration amount, it is possible to obtain an image in which the influence of curling is reduced.

  The effect of 1st Embodiment of this invention is demonstrated.

  When a curl occurs at the time of shooting, an amount of the curl is calculated, and image processing is performed based on the calculated curl amount, thereby obtaining an image in which the effect of the curl is reduced. Further, even when the optical system 1 has a correction mechanism, an image with reduced influence of turning can be obtained without using the correction mechanism.

  By performing image restoration processing based on the amount of turning on the image data obtained when the turning occurs, an image with reduced influence of turning can be obtained.

  By calculating the deviation from the reference position at the time of shooting as the amount of turning by using the camera posture at the time of focusing as a reference position, it is possible to restore the shot image to the image at the time of focusing even when the turning occurs. it can.

  By calculating the image recovery amount based on the amount of turning and performing the image restoration processing based on this amount of image restoration, it is possible to appropriately perform the image restoration processing according to the amount of turning.

  By calculating the amount of blur for each pixel and performing image restoration for each pixel, it is possible to obtain an image with reduced influence of the blur on the entire image.

  Next, a second embodiment of the present invention will be described with reference to FIG. FIG. 5 is a schematic block diagram of the imaging system of the second embodiment.

  The second embodiment includes a focus shift calculation unit 10 in addition to the imaging system of the first embodiment. Other configurations are the same as those in the first embodiment, and the same reference numerals as those in the first embodiment are attached, and the description thereof is omitted here. The recovery amount calculation unit 16 calculates an image recovery amount based on the focus shift amount.

  The focus shift calculation unit 10 calculates the focus shift amount from, for example, a look-up table shown in FIG. Note that the values in the look-up table shown in FIG. 6 are merely examples, and are not limited thereto. Further, the look-up table may have a larger amount of turning and a focus shift amount corresponding to the turning amount.

  Next, control during shooting according to the second embodiment will be described with reference to the flowchart of FIG.

  Since the control from step S200 to step S204 is the same as the control from step S100 to step S104 of the first embodiment, description thereof is omitted here.

  In step S205, the amount of focus shift is calculated based on the amount of turning.

  In step S206, an image restoration amount is calculated based on the focus shift amount. As a method for calculating the image restoration amount from the focus shift amount, the image restoration amount is calculated from a preset graph or the like as in step S105 of the first embodiment.

  In step S207, the image restoration amount is corrected according to the state of the optical system 1. The state of the optical system 1 is, for example, focal length information. For example, depending on whether the focal length of the optical system 1 is on the wide-angle side or the telephoto side, the amount of focus shift varies even with the same amount of turning. Therefore, the image restoration amount is corrected according to the state of the optical system 1. By correcting the image restoration amount according to the state of the optical system 1, an accurate image restoration amount can be calculated.

  Further, the image restoration amount is corrected according to the characteristics of the optical system 1. The characteristics of the optical system 1 are, for example, aberration information and MTF. When wobbling occurs, the light receiving surface of the image sensor 3 tilts from the in-focus position, so defocusing occurs due to the tilt of the light receiving surface. As shown in FIG. 8, when the focus is defocused by x, the MTF decreases by Mx from the peak of the MTF at the focused position. In such a case, an accurate image recovery amount can be calculated by correcting the image recovery amount so as to recover the lowered Mx.

  The state of the optical system 1 and the characteristics of the optical system 1 are stored as, for example, a lookup table. Further, when the characteristics of the optical system 1 can be expressed by an expression, this expression (including an approximate expression) may be stored.

  In step S208, image processing is performed to obtain a recovered image based on the corrected image recovery amount.

  In step S209, the image that has been subjected to image processing based on the image recovery amount is displayed on the image display unit 5, and the image data is stored in the image storage unit 6.

  Note that the amount of focus shift may be stored in the image processing unit 4 such as the recovery calculation unit 8. By storing the amount of focus shift and the like, when continuous shooting is performed by continuously pressing the release button, the time required for image processing can be shortened, and the number of shots per time can be increased.

  The effect of 2nd Embodiment of this invention is demonstrated.

  By calculating the focus shift amount from the amount of blur and calculating the image recovery amount based on the focus shift amount, an image recovery process corresponding to the focus shift amount can be performed.

  By correcting the image restoration amount based on the state of the optical system 1 and the characteristics of the optical system 1, it is possible to obtain an image in which the influence of the blur is further reduced.

  By storing the state of the optical system 1 and the characteristics of the optical system 1 in advance, the time required for image processing can be shortened.

  Next, an imaging system according to a third embodiment of the present invention will be described with reference to FIG. FIG. 9 is a flowchart showing control at the time of photographing according to the third embodiment.

  Since Steps S300 to S305 are the same control as Steps S200 to S205 of the second embodiment, description thereof is omitted here.

  In step S306, the amount of focus shift is compared with an allowable value. If the defocus amount is less than or equal to the allowable value, the process proceeds to step S310 without performing the image restoration process. If the focus shift amount is larger than the allowable value, the process proceeds to step S307 to perform image restoration processing. The allowable value is a value that is set in advance, and is a value that the user does not feel that there is a curl, for example, when an image is printed. The allowable value is set by the manufacturer at the time of design. The photographer may be settable.

  Since steps S307 to S309 are the same control as steps S206 to S208 of the second embodiment, description thereof is omitted here.

  In step S <b> 310, if the focus shift amount is equal to or smaller than the allowable value, the image captured by the image sensor 3 is displayed as it is on the image display unit 5, and the image data is stored in the image storage unit 6. When the amount of focus shift is larger than the allowable value, the image restored according to the focus shift amount is displayed on the image display unit 5 and the image data is stored in the image storage unit 6.

  In the present embodiment, it is determined whether to perform the image restoration process based on the focus shift amount, but it may be determined whether the image restoration process is performed based on the amount of blurring.

  The effect of the third embodiment will be described.

  When the amount of focus shift is larger than the allowable value, image recovery processing is performed based on the amount of focus shift. When the amount of focus shift is less than the allowable value, image recovery processing is not performed, and when there is almost no focus shift amount, the image Processing can be reduced.

  Next, an imaging system according to a fourth embodiment of the present invention will be described with reference to FIG. FIG. 10 is a schematic block diagram of the imaging system of the fourth embodiment. The imaging system according to the fourth embodiment includes an imaging device and a processing device. In the present embodiment, a case where a camera is used as the imaging device and a computer is used as the processing device will be described. However, it is not limited to these.

  The camera includes an optical system 1, an image sensor 3, a blur detection unit 2, a recovery amount calculation unit 8, a recording unit 11, and a control unit 7. Components having the same configurations as those of the first embodiment are denoted by the same reference numerals as those of the first embodiment, and description thereof is omitted here.

  The recording unit 11 records an electrical signal output from the image sensor 3 as image data. The recording unit 11 records the image recovery amount calculated by the recovery amount calculation unit 8. Note that the image data and the image restoration amount are recorded with correlation. In addition, the recording unit 11 stores image data and an image restoration amount in a storage medium. The storage medium is, for example, a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory. The storage medium is detachable from the camera and the computer.

  The computer includes an image processing unit 12, an image display unit 5, and an image storage unit 6. The image processing unit 12 includes an image recovery unit 13. The image recovery unit 13 reads the image data and the image recovery amount stored in the storage medium, and performs image recovery processing on the electrical signal output from the image sensor 3.

  The computer is composed of a CPU, ROM, RAM, and the like. The ROM stores a control program and each data. Each function as the processing device is exhibited when the CPU executes a calculation based on a control program stored in the ROM.

  Next, control during shooting according to the fourth embodiment will be described with reference to the flowchart of FIG.

  Since the control from step S400 to step S405 is the same as the control from step S100 to step S105 of the first embodiment, description thereof is omitted here.

  In step 406, the image data and the image recovery amount are stored in the storage medium.

  In the present embodiment, the image data and the image recovery amount stored in the storage medium are read out by a personal computer, an image recovery process is performed, and an image in which the influence of curling is reduced is created.

  The effect of the fourth embodiment will be described.

  The image data and the image restoration amount are stored in the storage medium by the recording unit 11. Then, the image data stored in the storage medium by the computer and the image restoration amount are read and the image restoration processing is performed, so that the processing in the camera can be reduced.

  Next, an imaging system according to a fifth embodiment of the present invention will be described with reference to FIG. FIG. 12 is a schematic block diagram of the fifth embodiment.

  The camera includes a focus shift calculation unit 10 and a first communication unit 14 in addition to the camera of the fourth embodiment. Since other configurations are the same as those in the fourth embodiment, description thereof is omitted here. The recording unit 15 records the image data and the image recovery amount. Further, the recovery amount calculation unit 16 calculates an image recovery amount based on the focus shift amount.

  The first communication unit 14 reads out the image data and the image recovery amount recorded in the recording unit 15, and sends a signal of the image data and the image recovery amount to the second communication unit 17 of the computer via wireless.

  The computer includes a second communication unit 17, an image processing unit 12, an image display unit 5, and an image storage unit 6. The second communication unit 17 receives the image data and the image restoration amount signal sent from the first communication unit 14. The received signal is sent to the image processing unit 12. The image processing unit 12 creates an image in which the influence of the blur is reduced based on the image data and the image restoration amount.

  In the fifth embodiment, image data and an image restoration amount are transmitted from the first communication unit 14 of the camera to the second communication unit 17 of the computer, and image processing is performed by a computer installed at a location different from the camera.

  In this embodiment, the recovery amount calculation unit 16 is provided in the camera. However, the recovery amount calculation unit 16 may be provided in the computer.

  The effect of 5th Embodiment is demonstrated.

  By sending the image data and the image recovery amount from the first communication unit 14 of the camera to the second communication unit 17 of the computer, the storage capacity of the camera can be reduced. In addition, image processing in the camera can be reduced. Further, image processing can be performed at any time by a computer set at a location away from the camera.

  The configuration of each embodiment is not limited to the above configuration, and can be realized by a combination of hardware and software. Moreover, it is also possible to combine each embodiment.

  It goes without saying that the present invention is not limited to the above-described embodiments, and includes various modifications and improvements that can be made within the scope of the technical idea.

DESCRIPTION OF SYMBOLS 1 Optical system 2 Shake detection part 3 Image pick-up element 4, 12 Image processing part 8, 16 Recovery amount calculation part (image recovery amount calculation part)
9, 13 Image recovery unit 10 Focus shift calculation unit 11, 15 Recording unit 14 First communication unit (transmission unit)
17 Second communication unit (receiving unit)

Claims (18)

An imaging device having a plurality of pixels and forming an image of a subject;
A turn detection unit that detects the amount of turn that occurred during shooting;
An image processing unit that performs image processing on the image data obtained by the imaging device based on the amount of turning,
The image sensor rotates around an inclination center with respect to a reference position that is a position of the image sensor when an operation for performing focusing by pressing a release button by a predetermined amount is performed,
The turn detection unit calculates the turn amount for each of the plurality of pixels according to the inclination of the image sensor at the time of shooting with respect to the reference position and the distance from the inclination center of each of the plurality of pixels. An imaging system characterized by that.   2. The imaging system according to claim 1, wherein the image processing unit performs an image restoration process on the image data obtained by generating the distortion so that the distortion is reduced based on the amount of the distortion. . The imaging system according to claim 1, wherein the reference position is a position of the imaging element when focusing is performed.   The imaging system according to claim 1, wherein the reference position is a position where a light receiving surface of the imaging element is in a vertical direction. An image restoration amount calculation unit that calculates an image restoration amount based on the amount of turning;
The imaging system according to claim 1, wherein the image processing unit performs the image processing based on the image restoration amount. The imaging system according to claim 5 , wherein the image restoration amount calculation unit stores the amount of blurring. A focus shift calculation unit for calculating a focus shift amount from the amount of turning;
The imaging system according to claim 1, wherein the image processing unit performs the image processing based on the focus shift amount. An image restoration amount calculation unit that calculates an image restoration amount based on the focus shift amount,
The imaging system according to claim 7, wherein the image processing unit performs image processing based on the image restoration amount.   The imaging system according to claim 8, wherein the image restoration amount calculation unit stores the focus shift amount. The imaging system according to claim 5 , wherein the image restoration amount calculation unit corrects the image restoration amount according to a state of an optical system. 11. The imaging system according to claim 5 , wherein the image restoration amount calculation unit corrects the image restoration amount according to a characteristic of an optical system.   The imaging system according to claim 10 or 11, wherein the image restoration amount calculation unit stores at least one of a state of the optical system and a characteristic of the optical system. The turn detection unit calculates a turn amount in at least two or more pixels of the image sensor,
The imaging system according to any one of claims 1 to 12, wherein the image processing unit performs the image processing based on at least two of the turning amounts. The image processing unit performs the image processing on the image data based on a value obtained by differentiating the image data obtained by the imaging device and the image restoration amount , and The imaging system according to any one of 8 to 12 .   The imaging system according to any one of claims 1 to 14, wherein the image processing unit performs the image processing based on the turning amount when the turning amount is larger than a predetermined value. An imaging device comprising: the imaging element; the blur detection unit; and a recording unit that records image data obtained from the imaging element and the image restoration amount based on the blur amount;
Claim 5, characterized in that and an image processing apparatus having the image processing unit which performs image processing based on the image data recorded in the image recovery amount by the recording unit, 6, 8 and 12 And the imaging system according to any one of 14 . The imaging apparatus includes a transmission unit that transmits the image data and the image recovery amount,
The imaging system according to claim 16, wherein the image processing apparatus includes a receiving unit that receives the image data transmitted from the transmitting unit. The subject image is captured by an imaging device having a plurality of pixels and on which the subject image is formed,
Detects the amount of distortion that occurred during imaging,
Performing image processing on the image data obtained by the imaging device based on the amount of turning;
The image sensor rotates around an inclination center with respect to a reference position that is a position of the image sensor when an operation for performing focusing by pressing a release button by a predetermined amount is performed,
The amount of turning is calculated for each of the plurality of pixels according to an inclination of the imaging element at the time of imaging with respect to the reference position and a distance from the inclination center of each of the plurality of pixels. Imaging method.