JP7225281B2 - IMAGING DEVICE, IMAGING DEVICE CONTROL METHOD, COMPUTER PROGRAM AND RECORDING MEDIUM - Google Patents

Description

The present invention relates to the display of an imaging device, and more particularly to the display when imaging images with different focus positions.

When capturing images of multiple subjects at different distances from an image capturing device such as a digital camera, or when capturing an image of a subject that is long in the depth direction, the depth of field in the imaging optical system is insufficient, so some of the subjects may be captured. There are times when you can only focus on the subject. In order to solve this problem, Patent Document 1 discloses that a plurality of images are captured by changing the focus position, only the in-focus region is extracted from each image, combined into one image, and the entire imaging region is focused. A technique is disclosed for generating a composite image that

However, when performing the above-described imaging using the live view display of an imaging device such as a digital camera, the angle of view of the captured image may change as the focus position changes due to changes in the extension amount of the lens. . In order to generate a composite image, it is necessary to cut out other captured images according to the angle of view of the captured image with the narrowest angle of view. The angle of view may differ.

Patent Literature 2 discloses a method of generating image data for confirmation through simple pre-imaging and image processing before performing main imaging.

JP 2015-216532 A JP 2015-231058 A

However, with the method described in Patent Document 2, pre-imaging must be performed before main imaging, which is troublesome for the photographer and lengthens the entire imaging operation time.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an imaging apparatus capable of capturing a plurality of images with different focal positions and conveniently confirming the angle of view of a composite image.

The present invention comprises imaging means, an optical system capable of changing the focal position, and control means, wherein the control means is a plurality of images having different focal positions, which are used to generate a composite image. The optical system and the imaging means are controlled so as to capture the image at the predetermined focus position , and the control means controls the optical system before capturing the plurality of images with different focus positions. and controlling the image capturing means to perform live view display using the image captured by the image capturing means, wherein the control means changes the focus position so as to indicate the angle of view of the composite image. Provided is an imaging device characterized by controlling the live view display before imaging a plurality of images.

According to the imaging apparatus of the present invention, when combining a plurality of images with different focus positions, the angle of view of the combined image can be confirmed with good convenience.

1 is a block diagram showing the structure of a digital camera according to an embodiment of the present invention; FIG. 4 is a flowchart for explaining the first embodiment; FIG. 4 is a diagram for explaining a function representing the relationship between the focus position and the image magnification in the first embodiment; FIG. 4 is a diagram for explaining maximum set image magnification and LV image magnification in the first embodiment; FIG. 4 is a diagram for explaining an example of an array of sensors forming an imaging device capable of acquiring distance information of a subject; FIG. 4 is a diagram for explaining incidence of optical signals in a pixel having a plurality of photoelectric conversion units; FIG. 10 is a diagram for explaining a method of acquiring distance information of a subject using pixels having a plurality of photoelectric conversion units; 9 is a flowchart for explaining a second embodiment; (a) is a diagram showing a monotonous decrease in image magnification, and (b) is a diagram showing a monotonous increase in image magnification.

Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

(First embodiment)
FIG. 1 is a block diagram showing the structure of a digital camera according to this embodiment.

The control circuit 101 is, for example, a signal processor such as a CPU or an MPU, reads out a program prestored in a ROM 105 to be described later, and controls each part of the digital camera 100 . For example, as will be described later, the control circuit 101 issues commands to the imaging element 104, which will be described later, regarding the start and end of imaging. It also issues an image processing command to an image processing circuit 107, which will be described later, based on a program stored in the ROM 105. FIG. A user's command is input to the digital camera 100 by an operation member 110 (to be described later) and reaches each part of the digital camera 100 through the control circuit 101 .

The drive mechanism 102 is composed of a motor or the like, and mechanically operates an optical system 103 described later under the command of the control circuit 101 . For example, based on a command from the control circuit 101 , the drive mechanism 102 moves the position of the focus lens included in the optical system 103 to adjust the focal length of the optical system 103 .

The optical system 103 is composed of a zoom lens, a focus lens, an aperture, and the like. A diaphragm is a mechanism that adjusts the amount of light that is transmitted. By changing the position of the focus lens, the focus position can be changed.

The imaging element 104 is a photoelectric conversion element that performs photoelectric conversion to convert an incident optical signal into an electrical signal. For example, a CCD, CMOS sensor, or the like can be applied to the imaging device 104 .

The ROM 105 is a read-only non-volatile memory as a recording medium, and stores the operation program of each block provided in the digital camera 100 as well as the parameters required for the operation of each block. A RAM 106 is a rewritable volatile memory, and is used as a temporary storage area for data output during operation of each block of the digital camera 100 .

The image processing circuit 107 performs various image processing such as white balance adjustment, color interpolation, and filtering on the image output from the image sensor 104 or image signal data recorded in a built-in memory 109, which will be described later. . Also, the data of the image signal captured by the image sensor 104 is subjected to compression processing according to a standard such as JPEG.

The image processing circuit 107 is composed of an integrated circuit (ASIC) that is a collection of circuits that perform specific processing. Alternatively, the control circuit 101 may perform processing according to a program read from the ROM 105 so that the control circuit 101 may share some or all of the functions of the image processing circuit 107 . If the control circuit 101 shares all the functions of the image processing circuit 107, the image processing circuit 107 does not have to be provided as hardware.

A display 108 is a liquid crystal display, an organic EL display, or the like for displaying images temporarily stored in the RAM 106, images stored in a later-described internal memory 109, or setting screens of the digital camera 100. . The display 108 can reflect the image acquired by the imaging device 104 as a display image in real time, and can perform so-called live view display.

The built-in memory 109 is a place for recording the image captured by the image sensor 104, the image processed by the image processing circuit 107, information on the focus position at the time of image capturing, and the like. A memory card or the like may be used instead of the built-in memory.

The operation member 110 is, for example, a button, switch, key, mode dial, etc. attached to the digital camera 100, or a touch panel that is also used as the display 108, or the like. A command input by the user using the operation member 110 reaches the control circuit 101, and the control circuit 101 controls the operation of each block according to this command.

FIG. 2 is a flow chart for explaining this embodiment. First, this embodiment will be described with reference to FIG.

In step S201, the control circuit 101 acquires lens information. For example, in a digital camera with a non-interchangeable lens, lens information is stored in the ROM 105 in advance, and the control circuit 101 acquires the lens information in S201. Alternatively, in the case of a lens-interchangeable digital camera, the control circuit 101 reads the lens type through a mount provided in the optical system 103 or the like. Information for each type of lens is stored in advance in the ROM 105 , and the control circuit 101 compares it with the type of lens that has been read out, and reads information on the corresponding lens from the ROM 105 . Note that the lens information here refers to lens-specific information such as the relationship between the focal position and the image magnification when the lens is used, which will be described later.

In this embodiment, the control circuit 101 reads out a function indicating the relationship between the focus position and the image magnification included in the lens information. Since the image magnification and the angle of view of an image are inversely proportional, the control circuit 101 can obtain a function representing the relationship between the focus position and the angle of view from the function representing the relationship between the focus position and the image magnification. In general, the function indicating the relationship between the focus position and the image magnification is uniquely determined by the type of lens, and the function changes monotonously. It should be noted that the image magnification here refers to the ratio between the size of the subject image formed on the imaging device and the actual size of the subject.

FIG. 3 is a diagram for explaining a function showing the relationship between the focus position and the image magnification in this embodiment. In the function shown in FIG. 3, the image magnification monotonously decreases with respect to the focus position.

In step S202, the control circuit 101 sets a plurality of focus positions. For example, a user sets a focal area through a touch panel, and the optical system 103 measures a focus position corresponding to the focal area. The control circuit 101 sets a predetermined number of focus positions before and after the measured focus position. The control circuit 101 preferably sets the distance between adjacent focus positions so that the edges of the depth of field slightly overlap each other.

In step S203, the control circuit 101 calculates the image magnification. At step S203, the control circuit 101 uses the function indicating the relationship between the focus position and the image magnification obtained at step S201 to calculate the image magnification corresponding to each focus position set at step S202. For example, as shown in FIG. 3, the control circuit 101 sets focus positions 301 to 304 in step S202. According to the functions shown in FIG. 3, when the optical system 103 focuses on the focal positions 301 to 304, the image magnifications of the images captured by the imaging device 104 are 311 to 314, respectively.

In step S204, the control circuit 101 compares the maximum set image magnification and the live view image magnification (hereinafter referred to as LV image magnification). The maximum set image magnification here refers to the image magnification that is the largest among the image magnifications calculated in step S203 and serves as a reference. The LV image magnification refers to the image magnification corresponding to the focus position set when the display 108 is displaying the live view.

If the control circuit 101 determines in step S204 that the maximum set image magnification is greater than the LV image magnification, the process proceeds to step S205; otherwise, the process proceeds to step S206.

FIG. 4 is a diagram for explaining the maximum set image magnification and LV image magnification in this embodiment. FIG. 4(a) shows the angle of view of the image with the LV image magnification, and FIG. 4(b) shows the angle of view of the image with the maximum set image magnification. The angle of view of the image 401 shown in FIG. 4A is wider than the angle of view of the image 402 shown in FIG. It corresponds to the angle of view. Therefore, when the control circuit 101 attempts to synthesize the images 401 and 402, the portion outside the frame 403 of the image 401 must be cut off, and the portion outside the frame 403 is lost in the created synthesized image. . When the display 108 performs live view display using the image 401 as it is, a problem arises in that although the portion outside the frame 403 is displayed, that portion is not included in the synthesized image. In order to solve this problem, when the control circuit 101 determines that the maximum set image magnification is larger than the LV image magnification, the display 108 controls the live view display so that the portion outside the frame 403 of the image 401 is not live view displayed. Correct the image. Specifically, in step S205, the image processing circuit 107 cuts out the portion within the frame 403 from the image 401 shown in FIG.

In step S206, the display 108 displays a live view image. If the image processing circuit 107 corrects the live view in step S205, in step S206 the display 108 displays the corrected image instead of the pre-corrected image as the display image for live view display.

In step S<b>207 , the control circuit 101 determines whether or not there is an instruction to start imaging by operating a button provided on the operation member 110 by the user. If there is an instruction to start imaging, the process proceeds to step S207, and if there is no instruction to start imaging, the process proceeds to step S208. In step S208, the control circuit 101 determines whether or not to reset the focus position. If the focus position is to be reset, the process returns to step S202, and if the focus position is not to be reset, the process returns to step S204.

For example, when the digital camera 100 performs imaging in a tracking mode in which the focus position is changed in real time according to the position of the subject, the focus position also needs to be changed in real time. In such a case, if there is no instruction to start imaging in step S207, it is necessary to return to step S202 and set the focus position again. On the other hand, if the focus position is first set at a predetermined distance from the digital camera 100 in step S202 and is not changed thereafter, there is no need to set the focus position again. Therefore, in such a case, if there is no instruction to start imaging in step S207, the process returns to step S204 instead of step S202.

In step S209, main imaging is performed using the image sensor 104 at each of the plurality of focus positions set in step S202.

In step S210, the image processing circuit 107 performs synthesis using the image captured in step S209. An example of a method of synthesis is briefly described below. The image processing circuit 107 uses the contrast values obtained from each image to generate a composite MAP. Specifically, in each region or pixel of interest, the composite ratio of the image with the highest image contrast value among the plurality of images is set to 100%, and the other images are set to 0%. Note that when the composition ratio between adjacent pixels changes from 0% to 100% (or changes from 100% to 0%), unnaturalness at the composition boundary becomes conspicuous. Therefore, by applying a low-pass filter having a predetermined number of pixels (number of taps) to the composite MAP, the composite MAP is processed so that the composite ratio does not change abruptly between adjacent pixels. Further, based on the contrast value of each image in the region of interest or pixel, a composite map may be created in which the composition ratio is higher for images with higher contrast values.

As described above, according to the first embodiment, the angle of view of the synthesized image is taken into consideration, the live view display at the time of imaging is corrected, and the angle of view corresponding to the angle of view of the synthesized image is displayed to the user at the time of imaging. can be done.

In the description of step S202, a predetermined number of focus positions are set based on the focus position of the focus area set by the user through the touch panel, but the present invention is not limited to this. A region occupied by the subject for which the reference focus position is set may be determined from the image, and a plurality of focus positions may be set based on the distance information in this region.

FIG. 5 is a diagram for explaining an example of an array of sensors forming the imaging device 104 capable of acquiring distance information of a subject. FIG. 5A shows a structure in which a pixel 500 has two photoelectric conversion units 501 and 502 capable of reading optical signals independently of each other. The number of photoelectric conversion units included in each pixel is not limited to two, and may be three or more. For example, FIG. 5B shows a structure in which a pixel 510 has four photoelectric conversion units 511 to 514. FIG. The following description will be based on a structure in which one pixel has two photoelectric conversion units.

FIG. 6 is a diagram for explaining how an optical signal is incident on a pixel having a plurality of photoelectric conversion units.

In FIG. 6, a pixel array 601 includes microlenses 602 , color filters 603 , and photoelectric conversion units 604 and 605 . The photoelectric conversion units 604 and 605 belong to the same pixel and correspond to the common microlens 602 and color filter 603 . FIG. 6 is a top view of the digital camera 100 and shows that two photoelectric conversion units 604 and 605 corresponding to one pixel are arranged side by side. Of the luminous fluxes emitted from the exit pupil 606, the upper luminous flux (corresponding to the luminous flux from the area 607) with the optical axis 609 as a boundary is incident on the photoelectric conversion unit 605, and the lower luminous flux (corresponding to the luminous flux from the area 608) enters the photoelectric conversion unit 605. ) enters the photoelectric conversion unit 604 . That is, the photoelectric conversion units 604 and 605 receive light in different regions of the exit pupil of the imaging lens. Assuming that the signal received by the photoelectric conversion unit 604 is the A image and the signal received by the photoelectric conversion unit 605 is the B image, the amount of focus shift can be calculated based on the phase difference between the A image and the B image, and distance information can be obtained. can do. In particular, by arranging pixels having two photoelectric conversion units over the entire image sensor 104, the image sensor 104 can obtain distance information of an arbitrary location of the object.

FIG. 7 is a diagram for explaining a method of acquiring distance information of a subject using pixels having a plurality of photoelectric conversion units according to this embodiment. In FIG. 7A, a total of 25 pixels of 5×5 are shown on the screen for easy viewing, but the actual pixels are smaller and more in number. A pixel that overlaps with the object, such as a pixel 701, is drawn with a solid line, and a pixel that does not overlap, like a pixel 702, is drawn with a dotted line. By obtaining the distance information for all the solid-line pixels as described above, the distance information for the entire subject can be obtained.

The setting of the focus position in step S202 in this embodiment will be described below. The user performs a touch operation using the touch panel included in the operation member 110 in order to specify the area 703 that is part of the subject. When the area 703 is selected by this touch operation, the control circuit 101 detects the entire subject area 704 to which the area 703 belongs. For example, the control circuit 101 analyzes color information in the image through the image processing circuit 107 and detects a subject from the color information. Alternatively, the control circuit 101 may acquire edge information and distance information in the image through the image processing circuit 107, and then detect the entire area 704 of the object based on them.

Subsequently, as described above, the distance information of pixels overlapping the entire subject area 704, such as the pixel 701 in FIG. 7A, is acquired. The control circuit 101 sets the focus position at the distance of the farthest subject in the acquired distance information, and at the same time sets the focus position at the distance of the closest subject. Between these two focus positions, several focus positions are set by a predetermined method such as equidistant arrangement.

(Second embodiment)
In the second embodiment, unlike the first embodiment, the control circuit 101 does not automatically set the focus position, but the user manually moves the focus position to take an image. A second embodiment will be described below. Note that the description of the configuration similar to that of the first embodiment is omitted.

FIG. 8 is a flow chart for explaining this embodiment.

In step S801, the control circuit 101 acquires lens information, as in step S802 of the first embodiment.

In step S802, the control circuit 101 determines whether the image magnification is monotonically decreasing or monotonously increasing. FIG. 9A is a diagram showing how the image magnification monotonically decreases in this embodiment, and FIG. 9B is a diagram showing how the image magnification monotonously increases in this embodiment. The control circuit 101 analyzes the function indicating the relationship between the image magnification and the focus position based on the lens information acquired in step S801. If the image magnification always decreases as the focal position becomes distant, as shown by the curve 901 in FIG. 9A, the control circuit 101 determines that the image magnification monotonously decreases. Conversely, if the image magnification always increases as the focus position becomes farther, as shown by the curve 911 in FIG. 9B, the control circuit 101 determines that the image magnification monotonously increases. In step S802, if the control circuit 101 determines that the image magnification is monotonically decreasing, the process proceeds to step S803. If it is determined that the image magnification is monotonously increasing, the process proceeds to step S804.

In step S803, the control circuit 101 limits the driving direction of the focus position to the infinity side. That is, the control circuit 101 restricts the movement of the focus position from the focus position 902 to the focus position 903 along the arrow 904 as shown in FIG. If the focus position moves in the direction opposite to arrow 904, a warning is given as will be described later.

In step S804, the control circuit 101 limits the driving direction of the focus position to the close side. In other words, the control circuit 101 restricts the movement of the focus position from the focus position 913 to the focus position 912 along the arrow 914 as shown in FIG. If it moves in the opposite direction, it issues a warning as described below.

The user adjusts the focus position of the optical system 103 and moves it to the initial focus position. The initial focus position here can be set to any position while the user observes the degree of blurring of the subject through live view or an electronic viewfinder. In step S805, the control circuit 101 determines whether or not the user has given an instruction to take the first image.If so, the process proceeds to step S806.

In step S806, the image sensor 104 performs main imaging at the initial focus position, and the control circuit 101 acquires the image magnification corresponding to this focus position. The image magnification acquired by the control circuit 101 here corresponds to the "maximum set image magnification" in the first embodiment.

In step S807, control circuit 101 determines whether the user has moved the focus position in the direction restricted in step S803 or S804. For example, if the control circuit 101 determines in step S802 that the image magnification is monotonically decreasing and restricts the driving direction of the focus position to the infinity side in step S803, if the focus position has moved to the close side, step S807 display 108 displays a warning. Alternatively, display 108 displays the live-view image and displays the warning over the displayed live-view image in step S808. Alternatively, the digital camera 100 may issue a warning using a warning sound or the like. Conversely, if the focus position has moved to the infinity side in step S807, the process proceeds to step S809.

In step S809, the image processing circuit 107 corrects the live view image. By restricting the moving direction of the focus position in step S807, the angle of view of the live view image becomes larger than the angle of view of the image captured at the initial focus position. Therefore, as in step S205 of the first embodiment, the image processing circuit 107 performs enlargement processing to cut off portions that are not displayed in the composite image. A specific method is the same as step S205, and therefore omitted.

In step S810, the display 108 displays the live view image corrected by the image processing circuit 107 in step S809.

In step S811, the control circuit 101 determines whether or not the operation member 110 is instructed by the user to perform actual imaging. For example, it is determined whether or not the user has pressed a button provided on the operation member 110 . If there is an instruction to take an image, the process proceeds to step S812, and the imaging device 104 takes an image. If there is no instruction to take an image, the process returns to step S807.

After the image sensor 104 performs the main imaging in step S812, in step S813, the control circuit 101 determines whether or not there is an instruction to end imaging. Here, for example, it is determined whether the user has pressed a button other than the button in step S811. Alternatively, it is determined whether or not the user has given an instruction to end imaging through the touch panel. Alternatively, the user predetermines the number of images to be captured, and in step S813, the control circuit 101 determines whether or not the determined number of images have been actually captured. do. In step S813, if the control circuit 101 determines that imaging has ended, the process proceeds to step S814, and if it determines that imaging has not ended, the process returns to step S807.

In step 814, the image processing circuit 107 synthesizes a plurality of images captured by the image sensor 104 in steps S806 and S812. The combining method is the same as in step S209.

The above description is merely an example of this embodiment, and this embodiment can be modified in various ways. For example, in step S808, the control circuit 101 issues a warning not only when the focus position moves in a non-limiting direction, but also when the focus position moves out of the depth of field of the adjacent focus position. may be implemented. Alternatively, when the pixels 500 shown in FIG. 5 of the second embodiment are provided, subject detection is performed in the same manner as in the second embodiment, and when the focus position goes out of the distance range of the subject in the optical axis direction, , the control circuit 101 may be implemented to issue a warning.

According to the second embodiment, when the user manually takes an image at a plurality of focus positions, by restricting the movement direction of the focus position, the angle of view corresponding to the composite image is displayed to the user at the initial focus position. can do.

(Other embodiments)
Although the above embodiment has been described based on implementation in a digital camera, it is not limited to a digital camera. For example, it may be carried out by a mobile device having an image sensor built therein, or by a network camera capable of capturing an image.

In addition, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device executes the program. can also be realized by a process of reading and operating the It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

100 Digital Camera 101 Control Circuit 102 Driving Mechanism 103 Optical System 104 Imaging Device 105 ROM
106 RAMs
107 image processing circuit 108 display 109 built-in memory 110 operation member

Claims (8)

imaging means;
An optical system that can change the focus position,
a control means;
The control means controls the optical system and the imaging unit so that a plurality of images having different focus positions and used for generating a composite image are captured at the predetermined focus positions. control means and
The control means controls the optical system and the image capturing means to perform live view display using the images captured by the image capturing means before capturing the plurality of images with different focus positions. ,
The image capturing apparatus, wherein the control means controls the live view display before capturing the plurality of images with different focus positions so as to indicate the angle of view of the composite image. The angles of view of the plurality of images with different focus positions are angles of view that differ from each other according to the focus position,
2. The imaging apparatus according to claim 1, wherein the angle of view of said composite image is equal to the narrowest angle of view of the plurality of images with different focus positions. 2. The imaging apparatus according to claim 1, further comprising image processing means for generating said composite image using a plurality of images captured by said imaging means and having different focus positions. When the angle of view of the synthesized image is not narrower than the angle of view corresponding to the focus position of the live view display, the control means controls the live view display to indicate the angle of view corresponding to the focus position of the live view display. 4. The imaging apparatus according to any one of claims 1 to 3, characterized in that it controls the . 5. The imaging apparatus according to any one of claims 1 to 4, wherein the plurality of images have different focal positions in the optical axis direction. A control method for an imaging device comprising imaging means and an optical system capable of changing a focus position,
having a control step,
In the control step, the optical system and the controlling the imaging means;
before capturing the plurality of images with different focus positions, controlling the optical system and the imaging means to perform live view display using the images captured by the imaging means;
A control method, comprising: controlling the live view display before capturing the plurality of images having different focus positions so as to indicate the angle of view of the composite image. A program for causing a computer to operate a control method for an imaging device comprising imaging means and an optical system capable of changing a focus position,
having a control step,
In the control step, the optical system and the controlling the imaging means;
before capturing the plurality of images with different focus positions, controlling the optical system and the imaging means to perform live view display using the images captured by the imaging means;
A program for controlling the live view display before capturing the plurality of images with different focus positions so as to indicate the angle of view of the composite image. A computer-readable recording medium recording the program according to claim 7 .

Images