JP2016206281A - Imaging system, imaging device, imaging method, program and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it to take pictures at a focus position where deterioration due to deformation or blurring becomes minimum.SOLUTION: Images are taken by changing a focus position to acquire a plurality of taken images. An average image of the plurality of taken images is calculated. When a difference value between the average image of the taken images up to a current frame and the average image of the taken images is equal to and less than a threshold, it is determined that the average images up to the current frame are converged. A set of a focus position and a contrast value of the average image converged at each focus position is plurally collected, and then, as shown in a figure, an upward convex approximation curve can be plotted. The focus position where the contrast value becomes maximum is estimated to be an optimum focus position.SELECTED DRAWING: Figure 8

Description

  The present invention relates to an imaging system, an imaging apparatus, an imaging method, a program, and a storage medium.

  When a subject is photographed using a camera, image quality deterioration such as distortion or blur may occur in a photographed image due to disturbance such as a heat wave. The hot flame is a phenomenon in which air of different temperatures mixes, a difference in the density of the air locally occurs, light is refracted at the boundary, and the object appears to shake. When shooting is performed in an environment where a hot flame is generated, image quality deteriorates because fluctuations with a plurality of different sizes and orientations occur in the shot image. That is, for example, in a photographed image obtained by photographing a building or the like, a portion (window frame or the like) constituted by a straight line has a waved shape.

  In order to deal with this problem, there is an image processing technique that improves (restores) the image quality deteriorated by the heat so that the subject can be perceived well. That is, for example, in Patent Document 1, a reference image having no motion is estimated from an exponential moving average of a captured image, and a local fluctuation component (motion vector) is calculated based on the reference image and the captured image. There is described an image processing apparatus that generates a corrected image that removes fluctuations of a hot flame existing in a photographed image by moving a partial image of the photographed image so as to cancel the fluctuation component.

  However, in the image processing apparatus described in Patent Document 1, since the optical image from the subject is incident on the camera while temporally changing distortion and blur due to disturbance such as a hot flame, the focus position is adjusted to the in-focus position. I can't. For this reason, there is a problem that even if the deterioration due to the disturbance of the photographed image is restored by the image processing device, the image is only out of focus.

  The present invention has been made to solve such problems, and an object of the present invention is to enable photographing at a focus position at which deterioration due to deformation or blur is minimized.

  An imaging system according to the present invention includes an imaging unit capable of changing a focus position with respect to an imaging object, and a degree of focus at each of the plurality of focus positions from a captured image respectively acquired at the plurality of focus positions by the imaging unit. Focus information calculating means for calculating focus information to be indicated, and focus position determining means for determining a focus position for the object to be photographed based on focus information at each of a plurality of focus positions calculated by the focus information calculating means. It is a photographing system.

  According to the present invention, it is possible to take an image at a focus position at which deterioration due to deformation or blur is minimized.

1 is a block diagram illustrating a schematic configuration of a photographing system according to an embodiment of the present invention. It is a block diagram which shows the hardware constitutions of the lens part and camera part which comprise the camera main body in FIG. It is a block diagram which shows the hardware constitutions of the information processing apparatus in FIG. It is a figure which shows the picked-up image when it is in focus, and its time average image. It is a figure which shows the picked-up image when it is out of focus, and its time average image. It is a block diagram which shows the function structure of the optimal focus position estimation part in FIG. 1, each function block, and operation | movement of a camera part. 7 is a flowchart showing a flow of processing of each functional block and camera unit of the optimum focus position estimation unit shown in FIG. 6. It is a figure for demonstrating the method by which the focus position estimation part in FIG. 6 estimates an optimal focus position.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
<Schematic configuration of the shooting system>
FIG. 1 is a block diagram showing a schematic configuration of a photographing system according to an embodiment of the present invention.

  This imaging system includes a camera body 1 as an imaging unit and an information processing device 2. The camera body 1 includes a lens unit 11 and a camera unit 12, and the information processing apparatus 2 includes a captured image storage 21 and an optimum focus position estimation unit 22.

  The camera body 1 can change the focus position, captures the subject with an appropriate size, and generates image data that is an electrical signal corresponding to the captured image, that is, the optical image of the subject. The internal configurations of the lens unit 11 and the camera unit 12 will be described later.

  The information processing apparatus 2 is configured by, for example, a personal computer (hereinafter, PC). The captured image storage 21 stores a captured image generated by the camera body 1. The optimum focus position estimation unit 22 as a focus position determination unit estimates the optimum focus position of the camera unit 12 based on the captured image and notifies the camera unit 12 of the estimated focus position. The camera unit 12 sets the lens unit 11 at the notified optimal focus position. Here, the estimation of the optimum focus position is to determine a focus position for acquiring a captured image with minimal deterioration due to distortion or blur.

<Hardware configuration of camera body>
FIG. 2 is a block diagram showing a hardware configuration of the lens unit 11 and the camera unit 12 constituting the camera body 1 in FIG.

  The lens unit 11 includes a lens 31 and a lens driving unit 32. The camera unit 12 includes an image sensor 33, a camera signal processing unit 34, an I / F (interface) unit 35, a control unit 36, a display unit 37, and an operation unit 38.

  The lens 31 forms an optical image of the subject on the image sensor 33. The lens drive unit 32 adjusts the lens 31 to a desired focus position by displacing the lens 31 in the optical axis direction based on the focus position control signal from the control unit 36.

  The image sensor 33 is composed of an image sensor such as a CCD (Charge Coupled Device) or a MOS (Metal Oxide Semiconductor), and operates based on a drive control signal from the control unit 36 to convert an optical image into an analog electrical signal. (Analog image signal).

  The camera signal processing unit 34 performs predetermined camera signal processing such as noise removal, digitization, and gain adjustment on the analog image signal from the image sensor 33. The I / F unit 35 sends the image data output from the camera signal processing unit 34 to the information processing apparatus 2. In addition, various signals (described later in detail) exchanged between the control unit 36 and the information processing apparatus 2 are transmitted and received.

  The control unit 36 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), and controls the entire camera body 1. In the present embodiment, focus position control and zoom control of the lens unit 11, drive control of the image sensor 33, setting of processing parameters of the camera signal processing unit 34, and various signals with the information processing apparatus 2 via the I / F unit 35. Communicate.

  The display unit 37 includes a liquid crystal display device and the like, and is a user interface for the user to check the state of the camera body 1 and a captured image. The operation unit 47 includes various buttons and a touch panel formed on the display unit 37 and is a user interface for the user to operate the camera body 1.

<Hardware configuration of information processing device>
FIG. 3 is a block diagram showing a hardware configuration of the information processing apparatus 2 in FIG.
As illustrated, the information processing apparatus 2 can be realized by a general-purpose PC. That is, the CPU 41, the ROM 42, the RAM 43, the HDD (Hard Disk Drive) 44, and the I / F 15 are connected via the bus 40, and the display unit 46 and the operation unit 47 are connected to the I / F 45. Yes.

  The CPU 41 is a calculation unit and controls the operation of the information processing apparatus 2 as a whole. The ROM 42 is a read-only nonvolatile storage medium, and stores programs such as firmware. The RAM 43 is a volatile storage medium capable of reading and writing information at high speed, and is used as a work area when the CPU 41 processes information.

  The HDD 44 is a nonvolatile storage medium capable of reading and writing information, and stores an OS (Operating System), various control programs, application programs, and the like. These programs are input by each reading device from a portable computer-readable storage medium such as a CD-ROM, a DVD-ROM, or a flash memory. The HDD 44 functions as the captured image storage 21 in FIG.

  The I / F 45 connects and controls the bus 40 and various hardware and networks. Communication with the camera body 1 is also performed via the I / F 45. Note that communication with the camera body 1 may be wired communication or wireless communication.

  The display unit 46 includes a liquid crystal display device and the like, and is a user interface for the user to check the state of the information processing apparatus 2. The operation unit 47 is a user interface such as a keyboard and a mouse for the user to input information to the information processing apparatus 2.

  The optimum focus position estimation unit 22 in FIG. 1 is a functional block realized by the CPU 41 executing predetermined processing using the RAM 43 as a work area based on the various programs described above.

<Photographed image and its time average image>
FIG. 4 is a diagram showing a captured image and its time average image when the focus is in focus, and FIG. 5 is a diagram showing a captured image and its time average image when it is out of focus.

  When a captured image generated from an optical image of a stationary object that has changed over time due to atmospheric disturbance or the like is deteriorated due to distortion or blurring, the average image between frames gradually approaches an image of the correct shape. It is generally known. However, since the average image accumulates deformation and blur, the average image converges to an image having a low contrast as compared with an image having a correct shape.

  At this time, when the subject is in focus and when it is not in focus, the blur amount of the converged average image differs depending on the blur amount of the image before deterioration. That is, the focused image (FIG. 4) converges to a relatively clear average image with less blur than the unfocused image (FIG. 5).

  In the imaging system according to the present embodiment, the optimum focus position is estimated using the contrast value as focus information indicating the degree of focus. More specifically, by comparing the contrast values of the average images taken by changing the focus position and detecting the focus position where the contrast value is maximized, it is correct even when the deformation or blur changes over time. The focus position can be estimated.

<Functional configuration of optimal focus position estimation unit>
FIG. 6 is a block diagram illustrating a functional configuration of the optimum focus position estimation unit 22 in FIG. 1, each functional block thereof, and an operation of the camera unit 12.

  The optimum focus position estimation unit 22 includes a captured image storage unit 51, an average image calculation unit 52, an average image storage unit 53, an average image convergence determination unit 54, a contrast calculation unit 55, a contrast value / focus position storage unit 56, and a focus position estimation. A unit 57 and a focus position designation unit 58 are provided.

Before describing each part of the optimum focus position estimation unit 22, the general operation of the camera unit 12 and the optimum focus position estimation unit 22 will be described.
When the optimum focus position estimation unit 22 estimates the focus position, the camera unit 12 continuously performs photographing until an acquisition end signal is received from the optimum focus position estimation unit 22, and the captured image is displayed as the optimum focus position estimation unit 22. Send to. In response to reception of the acquisition end signal, shooting is temporarily stopped, and focus position information indicating the current focus position is transmitted to the optimum focus position estimation unit 22.

  Thereafter, the focus position information is received from the optimum focus position estimation unit 22, and the focus position is adjusted to that position. When the acquisition restart signal is received at the same time, that is, when the estimation of the optimum focus position is not completed (the received focus position information does not represent the optimum focus position), the optimum focus position is estimated. Take a picture for again.

  When the acquisition restart signal is not received at the same time as the focus position, that is, when the optimum focus position has been estimated (the received focus position information indicates the optimum focus position), the focus position is adjusted to that position. Then, the captured image is stored in the captured image storage 21.

Next, each part which comprises the optimal focus position estimation part 22 is demonstrated.
The captured image storage unit 51 stores captured images of a plurality of frames sequentially output from the camera unit 12. An average image calculation unit 52 serving as an average image calculation unit calculates an average image of the captured image group stored in the captured image storage unit 51. The average image storage unit 53 stores the average image calculated by the average image calculation unit 52.

  An average image convergence determination unit 54 serving as an average image convergence detection unit calculates an average image (current average image) currently calculated by the average image and a previous average image (previous average image) stored in the average image storage unit 53. A comparison value representing the degree of difference is calculated, and it is determined whether or not the average image has converged based on whether or not the comparison value is a predetermined threshold value or less. When it is determined that the comparison value is equal to or smaller than the predetermined value, that is, the current average image has converged, an acquisition end signal is transmitted to the camera unit 12 and a calculation start signal is transmitted to the contrast calculation unit 55. A specific example of the comparison value will be described later.

  A contrast calculation unit 55 as a focus information calculation unit receives the calculation start signal from the average image convergence determination unit 54 and calculates the contrast value of the current average image. The contrast value is calculated, for example, by calculating the sum of absolute values of luminance differences between adjacent pixels. The higher the contrast of the image, the greater the contrast value.

  The contrast value / focus position storage unit 56 associates the contrast value of the current average image calculated by the contrast calculation unit 55 with the focus position of the current photographed image output from the camera unit 12 and stores them as a set.

  The focus position estimation unit 57 displays the focus position information for specifying the next focus position until the predetermined number of contrast values and focus position sets are stored in the contrast value / focus position storage unit 56. 58.

  Then, when a set of a predetermined number of contrast values and focus positions is stored, an optimum focus determination signal indicating that the focus position having the maximum contrast value has been determined as the estimated value of the optimum focus position, and the optimum focus position are represented. The focus position information is sent to the focus position designation unit 58.

  When the focus position specifying unit 58 receives the focus position information, the focus position specifying unit 58 transmits the focus position information and the acquisition restart signal to the camera unit 12. When the focus position information and the optimum focus determination signal are received, the focus position information is transmitted to the camera unit 12.

<Operation of optimal position estimation unit>
FIG. 7 is a flowchart showing a process flow of the functional blocks of the optimum focus position estimation unit 22 and the camera unit 12 shown in FIG.

  First, the camera unit 12 performs zooming according to the object to be photographed (step S1). That is, when the user performs an input for a zoom operation from the operation unit 38 while viewing the image to be photographed displayed on the display unit 37, the control unit 36 controls the lens driving unit 32 to drive the lens 31. , Zoom. Next, in the camera unit 12, the lens drive unit 32 is controlled to drive the lens 31, and to adjust to an appropriate focus position (step S2).

  Next, the camera unit 12 takes an image (step S3). The captured image is transmitted to the optimum focus position estimation unit 22 and stored by the captured image storage unit 51. Next, the average image calculation unit 52 calculates an average image using the captured images stored so far (step S4). The calculated average image is stored by the average image storage unit 53.

  Next, the average image convergence determination unit 54 compares the current average image calculated this time with the previous average image stored in the average image storage unit 53, and calculates a comparison value (step S5). Here, as the comparison value, the total sum (cumulative value) of the difference values of the pixel values (luminance values, etc.) of each pixel of the current average image and the previous average image is used. Next, the average image convergence determination unit 54 performs average image convergence determination (determination as to whether the average image has converged) based on whether the comparison value is equal to or less than a predetermined threshold (step S6).

  As a result of the determination, if the comparison value is larger than the threshold value (step S6: No), it is determined that the current average image has not converged, and the process returns to step S3. As a result of the determination, if the comparison value is equal to or smaller than the threshold value (step S6: Yes), it is determined that the current average image has converged, an acquisition end signal is transmitted to the camera unit 12, and a calculation start signal is used as the contrast calculation unit. Then, the process proceeds to step S7.

  In step S7, the contrast calculator 55 calculates the contrast value of the current average image. Next, the contrast value / focus position storage unit 56 associates the contrast value of the current average image calculated by the contrast calculation unit 55 with the focus position transmitted from the camera unit 12 and stores it as a set (step S8).

  Next, has the focus position estimation unit 57 been able to sufficiently collect a set of samples for estimating the optimum focus position from a plurality of contrast values and focus position sets stored in the contrast value / focus position storage unit 56? It is determined whether or not (step S9, satisfaction determination of focus sample). Here, whether or not sufficient collection has been performed is determined based on whether or not the number of sets is equal to or greater than a predetermined value.

  If the number of samples is insufficient as a result of the determination (step S9: No), the focus position is slightly shifted (step S10), and the process returns to step S3. On the other hand, if the number of samples is sufficient (step S9: Yes), the process proceeds to step S11.

  In step S11, an optimum focus position is estimated. That is, the focus position estimation unit 57 estimates the focus position at which the contrast value is maximum as the optimum focus position using a set of a plurality of contrast values and focus positions stored in the contrast value / focus position storage unit 56. (The estimation method will be described later).

  Next, the camera unit 12 is adjusted to the optimum focus position (step S12). That is, the focus position designation unit 58 notifies the camera unit 12 of the focus position information representing the focus position estimated in step S11, and the camera unit 12 matches the notified focus position.

  By performing the above processing flow, even when the captured image is deteriorated due to temporally varying distortion or blur, it is possible to adjust to the correct focus position.

<Method of judging convergence of average image>
Regarding the convergence determination of the average image, the method of simply accumulating the difference value of the pixel values of each pixel of the current average image and the previous average image and comparing it with the threshold value was mentioned. There are two methods.

  The first is to measure and record the amount of deformation of a certain feature point in the image, using the property that the amount of deformation of the image deformation that varies with time, such as a heat wave, approaches a normal distribution. And the normal distribution are compared with a threshold value. When the degree of similarity exceeds a predetermined threshold, it is determined that the convergence has occurred. Here, as the feature point, one or more points on the outer edge of the subject (for example, the roof of the vehicle in the case of the images shown in FIGS. 4 and 5) can be adopted. As the deformation amount, the deformation amount with respect to the position of the feature point in the captured image of the first frame can be adopted. Also, the similarity with the normal distribution is determined as follows, for example, based on the kurtosis (the sharpness of the distribution) and the skewness (the left and right distortion of the distribution). That is, the difference between the kurtosis of the deformation sample and the kurtosis (= 3) of the normal distribution is within the threshold value, and the difference between the skewness of the deformation sample and the normal distribution is the threshold value (for example, 0.2 to If it is within about 0.4), it is determined that the distribution is similar to the normal distribution.

  The second is to accumulate the difference value of each pixel in a partial area of the current average image and the previous average image (for example, in the case of the images shown in FIGS. 4 and 5, a part of the image of the vehicle that is the subject), It is a method of comparing it with a threshold value. Although this method may reduce the accuracy, it is advantageous in that the calculation is lighter than the method of calculating the difference between the pixel values and the method of determining whether the measured deformation amount is close to the normal distribution. There is.

<Estimation method in the focus position estimation unit>
FIG. 8 is a diagram for explaining a method in which the focus position estimation unit 57 in FIG. 6 estimates an optimum focus position. In this figure, the horizontal axis represents the focus position, and the vertical axis represents the contrast value of the converged average image.

  When a set of the contrast value of the average image converged at the focus position and each focus position is collected a plurality of times, as shown in the figure, (horizontal axis, vertical axis) = (converged at each sampled focus position, each focus position) A convex approximate curve that passes through the contrast of the average image) can be drawn. In this approximate curve, the focus position where the contrast value is maximized can be obtained and estimated as the optimum focus position.

  As described above, according to the imaging system according to the embodiment of the present invention, when searching for an optimum focus position in focus, a sufficient number of images are captured at any of a plurality of focus positions, and an average is obtained from all the captured images. The contrast value of the image is calculated, and compared with the same calculation result at other focus positions, the focus value can be said to be in focus. An object whose blur varies with time can be photographed at an optimum focus position.

In the imaging system according to the embodiment of the present invention described above, the information processing device 2 separate from the camera body 1 has the captured image storage 21 and the optimum focus position estimation unit 22, but the following (1) It can also be configured as in (3).
(1) A configuration in which the camera body has a captured image storage unit and the information processing apparatus has an optimum focus position adjustment unit.
(2) A configuration in which the camera body includes a captured image storage unit and an optimum focus position adjustment unit. In this configuration, since the camera body as the photographing apparatus has the function of the information processing apparatus, the information processing apparatus becomes unnecessary.
(3) A configuration in which at least a part of the optimum focus position estimation unit is hardware.

  DESCRIPTION OF SYMBOLS 1 ... Camera body, 2 ... Information processing apparatus, 12 ... Camera part, 22 ... Optimum focus position estimation part, 52 ... Average image calculation part, 54 ... Average image convergence determination part, 55 ... Contrast calculation part, 56 ... Contrast value Focus position storage unit 57... Focus position estimation unit 58.

JP2012-104018A

Claims (10)

Photographing means capable of changing a focus position with respect to a photographing object;
Focus information calculating means for calculating focus information indicating the degree of focus at each of the plurality of focus positions from the captured images respectively acquired at the plurality of focus positions by the imaging means;
A focus position determination unit that determines a focus position for the object to be photographed based on focus information at each of a plurality of focus positions calculated by the focus information calculation unit;
A shooting system. In the imaging system according to claim 1,
The imaging system includes an average image calculation unit that calculates an average image of the acquired captured images, and the focus information calculation unit calculates the focus information from the average image. In the imaging system according to claim 2,
An imaging system comprising: average image convergence detection means for detecting convergence of the average image, wherein the focus information calculation means calculates the focus information from the average image whose convergence has been detected by the average image convergence detection means. In the imaging system according to claim 3,
The average image convergence detection means detects the convergence of the average image up to the current frame based on the degree of difference between the average image of the captured image up to the current frame and the average image of the captured image up to the previous frame. Shooting system. The imaging system according to claim 4,
The average image convergence detection means calculates a difference value between a difference value between at least some pixels of the average image of the photographed image up to the current frame and at least some pixels of the average image of the photographed image up to the previous frame. Shooting system. In the imaging system according to claim 3,
The average image convergence detection means is a photographing system that detects convergence of an average image based on a similarity between a distribution of deformation amounts of feature points in the photographed images acquired by the plurality and a normal distribution. Photographing means capable of changing a focus position with respect to a photographing object;
Focus information calculating means for calculating focus information indicating the degree of focus at each of the plurality of focus positions from the captured images respectively acquired at the plurality of focus positions by the imaging means;
A focus position determination unit that determines a focus position for the object to be photographed based on focus information at each of a plurality of focus positions calculated by the focus information calculation unit;
A photographing apparatus having An imaging method executed by an imaging means capable of changing a focus position with respect to an imaging object,
Taking a picture of the object to be photographed at a plurality of different focus positions and obtaining a photographed image respectively;
Calculating focus information indicating the degree of focus at each of the plurality of focus positions from the acquired captured images;
Determining a focus position for the object to be photographed based on focus information at each of the plurality of focus positions;
A photographing method comprising:   A program for causing a computer to function as focus information calculation means and focus position determination means in the photographing system according to claim 1. A computer-readable storage medium storing the program according to claim 9.

Images