JP6257260B2 - Imaging apparatus and control method thereof - Google Patents

Description

  The present invention relates to an imaging apparatus capable of generating an image whose focus position can be changed after shooting.

  In recent years, an imaging apparatus equipped with a technique called Computational Photography has been studied. This is a technique in which, in addition to simple image brightness and color data, information that has not been recorded by a conventional imaging apparatus is captured by an image sensor and stored together with image data, whereby various processing is performed by an image processing LSI. It is.

  Typical information that could not be recorded by a conventional imaging apparatus is information on the incident angle and depth of light from the subject. By using these pieces of information, it is possible to accurately reproduce the focus, the depth of field by the iris, the focal length, and the like that have been uniquely determined in the past when the subject is imaged by subsequent image processing. By recording light ray information called a light field of these subjects, an imaging apparatus having a function that cannot be realized by a conventional imaging apparatus has begun to be proposed. A camera that can change the focus position (in-focus position) of an actually captured image later has been announced.

  Next, a method for acquiring images from multiple viewpoints will be described as a typical example of light field acquisition. For example, by acquiring a plurality of images by changing the focal length with a single imaging device and leaving the data as one image, the focal length of the image can be freely changed later by image processing. it can. Further, by moving the camera horizontally or vertically and acquiring a plurality of images, it is also possible to acquire subject distance information from the parallax of the images. Since these use a single imaging device (imaging device), there is a demerit that a light field cannot be effectively acquired for a dynamic subject.

  A method of using a plurality of imaging devices or imaging devices themselves has been proposed in order to acquire images from multiple viewpoints while compensating for the above-described disadvantages (Patent Document 1). By arranging a plurality of image pickup devices or image pickup devices in an array, it is possible to give the acquired images simultaneity and increase the amount of information obtained from the light field by acquiring a plurality of data.

JP 2008-257686A

  However, even in the conventional technique in which the focusing position can be changed later, the user cannot know the range in which focusing can be performed after shooting. For this reason, when the focus position cannot be focused at the stage of changing the focus position after shooting, there is a problem in that shooting must be performed again in order to obtain a shot image at the desired focus position.

  The present invention has been made to solve the above-described problems of the prior art, and an object of the present invention is to make it possible to recognize a focusable range after shooting together with the relationship with the subject distance during shooting. It is in.

In order to achieve the above object, the present invention provides a plurality of optical systems each including an image sensor and having different focal lengths, an acquisition unit for acquiring a subject distance of a subject in an image output from the image sensor, a range of the subject distance obtained on the basis of information about the optical system of the generated after shooting using an image output from the imaging element, images are focus position of the different focus positions from the image And a control means for displaying a bar having a length corresponding to a possible range on the monitor for each of the optical systems having different focal lengths together with the subject distance acquired by the acquisition means.

  According to the present invention, a focusable range after shooting can be recognized during shooting together with the relationship with the subject distance.

It is the front view (figure (a)) and rear view (figure (b)) of a camera. It is a figure which shows arrangement | positioning of the optical system by the front view of a photographic lens part. It is a schematic diagram of the structure of an optical system. FIG. 6 is a state diagram of the optical system when a subject at infinity is in focus. It is a state diagram of an optical system when a subject moves to a finite distance. FIG. 6 is a state diagram of the optical system when a focus group is moved to the subject side and focused. It is a block diagram of a camera. It is a block diagram of an imaging part and its related element. FIG. 2 is a diagram showing a principle of distance measurement (FIG. (A)) and a diagram (FIG. (B)) showing that the optical axes coincide when the captured image is viewed in the optical axis direction. It is a flowchart of an image output process. FIG. 2 is a diagram showing a distance measurement principle (FIG. (A)), and a diagram (FIG. (B)) in which a calculated subject is projected onto a virtual imaging device surface. It is a conceptual diagram of the recording format of a reconstructed image signal. It is a conceptual diagram of the format for communication of a reconfigure | reconstructed image signal. It is a figure which shows an example of the output image | video to a monitor. It is a figure which shows an example of the output image | video to a monitor. It is a figure which shows an example of the output image to the monitor in 2nd Embodiment. It is a figure which shows an example of the output image | video to a monitor.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
First, the overall configuration of the imaging apparatus according to the first embodiment of the present invention will be described with reference to FIGS. FIGS. 1A and 1B are a front view and a rear view of a camera 100 as an imaging apparatus.

  The camera 100 includes a photographic lens holding unit 3, a photographic lens unit 2, and a grip 6 on the front side of the camera body unit 1. A shutter button 4 and a zoom lever 5 are provided on the upper portion of the camera body 1. On the back side of the camera body 1, a monitor 43 for displaying images during shooting and after recording, a mode dial 8 for selecting a shooting mode, and an AE dial 9 for exposure correction are arranged. The photographic lens unit 2 is composed of a plurality (16) of optical systems.

  FIG. 2 is a diagram illustrating the arrangement of the optical system in front view of the photographic lens unit 2. “◯ (white circle)” in FIG. 2 are 16 optical systems constituting the taking lens unit 2. There are four types (units) having different focal lengths as optical systems, which are referred to as optical systems A, B, C, and D. The letter in “○” indicates the type. Each of the optical systems A, B, C, and D is composed of four optical systems. The total number of optical systems is 4 types × 4 = 16. Henceforth, when referring to the single optical system in each optical system (for example, optical system A), a number is added like optical system A1, A2, A3, A4.

  The four optical systems A are arranged so that the centroid positions of the four optical axes of the optical systems A1, A2, A3, and A4 coincide with the centroid indicated by O ′. Similarly, the centroid positions of the optical axes of the optical systems B, C, and D are all arranged so as to coincide with the centroid O ′. Accordingly, the centroid positions of all 16 optical systems also coincide with the centroid O ′.

  FIG. 3 is a schematic diagram of the structures of the optical systems A, B, C, and D. The configuration of optical systems of the same type is the same.

  The optical system A includes a concave lens 11a, a convex lens 12a, a convex lens 13a, and a concave lens 14a, which are arranged in order from the object side, that is, the subject side (the front side of the camera body 1). An image sensor 15a is disposed on the image plane. Similarly, the optical system B includes a concave lens 11b, a convex lens 12b, a convex lens 13b, and a concave lens 14b from the subject side, and an imaging element 15b is disposed on the image plane. The optical system C is composed of a convex lens 11c, a concave lens 12c, a concave lens 13c, and a convex lens 14c from the subject side, and an image sensor 15c is disposed on the image plane. The optical system D is composed of a convex lens 11d, a concave lens 12d, a concave lens 13d, and a convex lens 14d from the subject side, and an image sensor 15d is disposed on the image plane.

  The optical axes Oa, Ob, Oc, and Od of the optical systems A, B, C, and D are parallel to the front-rear direction of the camera body 1. A light amount adjusting element 17 is disposed between the convex lens 14 (14a to 14d) and the imaging element 15 (15a to 15d). The light amount adjusting element 17 is configured by an electroluminescence filter or the like, and is provided in common to each of the optical systems A to D, and is configured to act on each of the optical systems A to D with a common light reduction rate. .

  A plurality of optical systems are arranged close to each other. Therefore, in order to prevent unnecessary light from the adjacent optical system from reaching the image sensor 15, the effective range of light rays is limited between the lenses and between the lens and the image sensor 15 in the relationship between the adjacent optical systems. It is desirable to provide an aperture stop or a light shielding wall.

The focal lengths of the optical systems A, B, C, and D are assumed to be fa, fb, fc, and fd, respectively. The focal length between the optical systems is configured to have a relationship represented by the following mathematical formula 1.
[Equation 1]
fb / fa = fc / fb = fd / fc = 2.0
The convex lenses 11a to 11d and the concave lenses 12a to 12d are the focus group FG. The focus group FG moves in accordance with the subject distance for focusing during shooting. The operation of the focus group FG will be described with reference to FIGS.

  FIG. 4 is a state diagram of the optical systems A, B, C, and D when a subject at infinity is in focus. FIG. 5 is a state diagram of the optical systems A, B, C, and D when the subject has moved a finite distance from the state shown in FIG.

  In the state of FIG. 4, the imaging position in each optical system is indicated by a black arrow, and it can be seen that images are formed on the imaging elements 15 a to 15 d. As shown in FIG. 5, in the state where the subject has moved a finite distance from the state shown in FIG. 4 and the focus group FG has not moved, the imaging position is behind the imaging elements 15a to 15d (the side away from the subject). Shift. The white arrow shown in FIG. 5 indicates the imaging position (indicated by the black arrow in FIG. 4) when the subject distance is infinite, and the black arrow indicates the imaging position when moved to a finite distance. .

Since the focal lengths of the optical systems A, B, C, and D are different, the amount of movement of the imaging position (the interval between the black arrow and the white arrow) due to the change in the subject distance is different. The focus movement amount of the optical system A (the movement amount of the imaging position with respect to the movement amount of the subject) is assumed to be Δa. Similarly, the amounts of focus movement of the optical systems B, C, and D are denoted by Δb, Δc, and Δd. The relationship between the respective focus movement amounts is expressed by the following mathematical formulas 2, 3, and 4.
[Equation 2]
Δb / Δa = (fb / fa) ² = 4
[Equation 3]
Δc / Δb = (fc / fb) ² = 4
[Equation 4]
Δd / Δc = (fd / fc) ² = 4
FIG. 6 is a state diagram of the optical systems A, B, C, and D when the focus group FG is moved to the subject side and brought into focus from the state shown in FIG. In FIG. 6, a white arrow indicates an imaging position (shown by a black arrow in FIG. 5) before moving the focus group FG, and a black arrow indicates an imaging position after the focus group FG is moved.

The sensitivity β of the movement of the imaging position relative to the amount of movement in the optical axis direction of the combined optical system (a set of the convex lens 11 and the concave lens 12) of each of the optical systems A, B, C, and D in the focus group FG is βa, βb , Βc, βd. The sensitivity β is the amount of movement of the imaging position with respect to the amount of movement of the combining optical system. The relationship between the sensitivities β is expressed by the following mathematical formulas 5, 6, and 7.
[Equation 5]
βb / βa = (fb / fa) ² = 4
[Equation 6]
βc / βb = (fc / fb) ² = 4
[Equation 7]
βd / βc = (fd / fc) ² = 4
With this relationship, when the subject distance changes, the necessary amount of movement of the focus group FG necessary for focusing is the same. As a result, the optical systems can be moved together for focus adjustment, and the imaging positions of the optical systems can be made to coincide with the position of the image sensor 15 at the same time.

  FIG. 7 is a block diagram of the camera 100.

  The camera 100 includes a system control unit 21 that controls a shooting operation, an imaging unit 22, a transfer unit 23, an image restoration unit 42, a development unit 24, an image processing unit 25, a multi-view image processing unit 26, a monitor image processing unit 27, a distance. A calculation unit 28 and an AF / AE evaluation unit 29 are included. The camera 100 also includes an operation unit 30, a drive control unit 31, a drive driver 32, an encoder 33, an exposure control unit 34, a recording encoding unit 35, a recording unit 36, a communication encoding unit 37, a communication unit 38, an external output unit 39, and a camera shake. A detection unit 40 and a face detection unit 44 are provided.

  The system control unit 21 includes a CPU and a ROM (none of which is shown) that stores a control program executed by the CPU, and controls processing of the entire camera 100. The operation unit 30 includes input devices (including a shutter button 4 and a zoom lever 5) such as keys and buttons that are used by a user to give an instruction regarding shooting. The system control unit 21 supplies data for displaying a mode transition, a menu screen, various information, and the like according to the operation of these input devices to the monitor 43, and the data is displayed on the monitor 43 along with images at the time of shooting / playback. To display.

  FIG. 8 is a block diagram of the imaging unit 22 and its related elements. The operations of the imaging unit 22 and related elements will be described in detail with reference to FIGS.

  The imaging unit 22 includes the above-described optical systems A to D, the imaging device 15 corresponding to the optical systems A to D, the A / D conversion unit 41, and the light amount adjustment element 17. When shooting an image, the user issues a shooting instruction from the operation unit 30. When a shooting instruction is given, the system control unit 21 sends a shooting instruction signal to the imaging unit 22.

  An imaging instruction signal from the system control unit 21 is sent to each of the image sensors 15 and is exposed for a predetermined period, and electric charges are generated by photoelectric conversion. The image pickup device 15 is a CMOS image sensor, and an image pickup signal is read out in analog form at a predetermined read timing by a rolling shutter method, converted into a digital data string by the A / D conversion unit 41, and supplied to the transfer unit 23. The

  The transfer unit 23 appropriately adjusts the order and timing of data output for the imaging signals from the optical systems A to D according to the development and image processing in the post-process and the configuration of the processing unit. The image restoration unit 42 shown in FIG. 7 performs image processing for repairing noise and defects for each pixel on the image data transferred from the transfer unit 23 and supplies the processed image data to the developing unit 24 as so-called RAW data. The developing unit 24 generates an image signal by color interpolation processing for each pixel of the RAW data, and supplies the image signal to the image processing unit 25 as a digital imaging signal within a predetermined dynamic range. In the color interpolation processing, code processing is performed with colors so that RGB information is assigned to all pixels corresponding to the color filter structures of the image sensors 15a to 15d.

  When the shutter button 4 of the operation unit 30 is pressed, the system control unit 21 issues an image processing instruction to the image processing unit 25. The image processing unit 25 further performs image processing such as white balance correction, gamma correction, sharpness correction, and saturation correction.

  Next, creation of a distance map will be described with reference to FIGS. FIG. 9A is a diagram showing the principle of distance measurement. The distance calculation unit 28 calculates a distance map from the image signals of the same optical system having the same focal length from the image processing unit 25. A distance map is created for each of the optical systems A to D.

  FIG. 9A illustrates a state in which a distance map is created from two optical systems A1 and A2 having the same focal length as an example. The optical axes of the optical systems A1 and A2 are Oa1 and Oa2. Imaging elements 15a corresponding to the optical systems A1 and A2 are referred to as imaging elements S1 and S2, respectively. The subject P is imaged at the imaging positions p1 'and p2' in the imaging elements S1 and S2 by the two optical systems A1 and A2, and is read out as an imaging signal.

In the optical axis direction, the distance from the position of the principal point P0 of the optical systems A1 and A2 to the subject P is the subject distance L, and the distance from the principal point P0 to the image sensors S1 and S2 is fa ′. The distance between the two optical axes Oa1 and Oa2 is D, and the distances from the optical axes Oa1 and Oa2 to the imaging positions p1 ′ and p2 ′ are d1 and d2, respectively. The relationship of the following formula 8 holds for these.
[Equation 8]
L = D × fa ′ / (d1 + d2)
FIG. 9B is a diagram illustrating the captured images of the image sensors S1 and S2 so that the optical axes Oa1 and Oa2 coincide when viewed in the optical axis direction. (D1 + d2) in Expression 8 is equal to the interval Δ on the captured image at the imaging positions p1 ′ and p2 ′ of the subject, as shown in FIG. 9B.

  The interval Δ on the image sensor is calculated using a correlation detection type feature point matching method such as a block matching method for detecting corresponding points of an object from images of different optical axes. As a result, (d1 + d2) is determined. Thereafter, the distance calculation unit 28 uses the shooting state information (distance fa ′, optical axis interval D) previously recorded in the memory or the like according to the above equation (8). The subject distance L to the subject P can be acquired (acquisition means).

  A distance map can be generated by executing such calculation over the entire screen. Although the method for generating a distance map using two images has been described here, it is desirable to use more than two images in order to compensate for an image having no corresponding point due to the shadow of an object.

  The calculated distance map is fed back to the image processing unit 25 and the multi-view image processing unit 26 via the system control unit 21 and has special effects such as superimposing a blur corresponding to the subject distance on the reconstructed image signal. It is also used for image processing to be applied. Although two optical systems have been described in FIG. 9, the use of a plurality of optical systems improves the robustness against occlusion problems and optical system errors.

  7 generates an RGB image corresponding to an image photographed from the virtual optical axis Oa0 at the photographing field angle designated by the user with the zoom lever 5 of the operation unit 30. The multi-view image processing unit 26 illustrated in FIG. Hereinafter, this is referred to as a reconstructed image signal. Here, the virtual optical axis Oa0 is an optical axis when a virtual optical system is set at the center of gravity O ′ of the viewpoint shown in FIG. 2 (see also FIG. 11B). The multi-view image processing unit 26 generates, as a reconstructed image signal, an image signal in which the viewpoint is corrected to the virtual optical axis Oa0 from the RGB images of four viewpoints corresponding to the optical systems having the same focal length (generation unit). . Thus, a reconstructed image signal that can change the in-focus position after shooting is generated from the image being shot.

  Further, when the imaging field angle designated by the user does not match any of the focal lengths of the optical systems A to D, the multi-view image processing unit 26 is an optical system having a focal length that is wider by one step than the designated field angle. An area corresponding to the designated angle of view is cut out from the captured reconstructed image signal. This is electronic zoom processing.

  FIG. 10 is a flowchart of image output processing executed by the multi-view image processing unit 26.

  First, the multiview image processing unit 26 reads an image signal from the image processing unit 25 (step S101), and groups images having the same focal length (step S102). Next, the multi-view image processing unit 26 reads a calculation coefficient corresponding to each focal length (step S103). Here, the calculation coefficient is data necessary for correcting the viewpoint, such as the focal length, image size, and optical system interval, and the lens designation, lens information, and lens position in the header section described later with reference to FIG. Corresponding information.

  The multiview image processing unit 26 corrects the viewpoint using the calculation coefficient, and generates a reconstructed image signal (step S104). The multi-view image processing unit 26 determines whether or not the processing regarding all focal lengths has been completed (step S105), and if there is a focal length that has not been processed, the processing returns to step S104. When the processes related to all the focal lengths have been completed, the multi-view image processing unit 26 determines whether or not clipping is necessary based on the view angle instruction from the system control unit 21 (step S106). When the imaging angle of view specified by the user is different from the focal lengths of the optical systems A to D, it is determined that clipping is necessary.

  As a result of the determination, the multi-view image processing unit 26 advances the process to step S109 if the clipping is not necessary. On the other hand, if the clipping is necessary, the multi-view image processing unit 26 outputs a reconstructed image signal that is one step wider than the instructed angle of view. Read (step S107). Then, the multiview image processing unit 26 cuts out the image signal in accordance with the angle of view instructed from the read reconstructed image signal (step S108). In step S109, the multiview image processing unit 26 outputs a reconstructed image signal that matches the instructed angle of view, and the process ends.

  The reconstructed image can be generated by a known method. An example of a method for generating a reconstructed image by the multi-view image processing unit 26 and the like will be described in detail with reference to FIGS.

  As described above with reference to FIG. 9, the subject P is imaged at the imaging positions p1 'and p2' of the imaging elements S1 and S2 by the two optical systems, and is read out as an imaging signal. First, the distance calculation unit 28 extracts the imaging positions p1 'and p2' from the images generated by the different optical systems A1 and A2 based on the correlation between the luminance signal distribution and the color signal distribution of the image. The distance calculation unit 28 obtains the subject distance L from the principal point P0 to the subject P by the above formula 8.

  FIG. 11A is a diagram showing the principle of distance measurement. As shown in FIG. 11A, the multi-view image processing unit 26 is based on the origins of the images captured by the optical systems A1 and A2 (positions on the image sensors S1 and S2 through which the optical axes Oa1 and Oa2 pass). The coordinates (x1 ′, y1 ′) of the imaging position p1 ′ and the coordinates (x2 ′, y2 ′) of the imaging position p2 ′ are obtained. The multi-view image processing unit 26 uses the subject distance L, the distance fa ′, and the optical axis distance D, and the center of gravity O ′ that is also the origin of the imaging system (in FIG. 11A, the midpoint between the two optical systems A1 and A2). ) To calculate the coordinates (x ″, y ″, L) of the subject P ″.

  Here, the reverse trace straight lines R1 ″ and R2 ″ in FIG. 11A do not intersect in a three-dimensional space in the calculation due to an error in the focal length of the optical system, a positional deviation between the optical system and the image sensor, and the like. There is. However, in that case, a virtual plane may be set at a position where the two straight lines are closest to each other, and the midpoint of the intersection of the plane and each reverse trace straight line may be obtained as coordinates. When there are more than two optical systems, the same calculation is performed using straight lines obtained from the respective optical systems.

  FIG. 11B is a diagram in which the calculated subject P ″ is projected onto the virtual imaging device S0 surface. Consider a case where the calculated subject P ″ obtained above is projected onto the virtual imaging device S0 surface by the optical system A0 of the focal length fa ′ virtually arranged at the imaging system origin (center of gravity O ′). The multi-eye image processing unit 26 obtains the coordinates (x0 ′, y0 ′) of the virtual imaging position p0 ′ on the virtual imaging device S0 surface in this case. The multi-view image processing unit 26 performs such image processing on the entire image, and generates a reconstructed image of the virtual optical axis Oa0 by the virtual optical system A0.

  The recording encoding unit 35 shown in FIG. 7 encodes the reconstructed image signal output from the multiview image processing unit 26 into a predetermined signal format.

  FIG. 12 is a conceptual diagram of a recording format for the reconstructed image signal. FIG. 13 is a conceptual diagram of a communication format of the reconstructed image signal.

  As shown in FIG. 12, the recording format includes an image data portion and a distance map portion in addition to a header portion of a file storing shooting information. As an example, the data structure of the header part includes imaging information, pixel structure, and image format in addition to the above-described calculation coefficients (lens designation, lens information, lens positional relationship).

  In the specification of the lens in the header section, information related to the configuration of the optical system provided in the camera 100 (for example, the number of optical systems is 16 in the present embodiment) is stored. The lens information stores information on the angle of view of each optical system. Information on the positional relationship of the optical axes of each optical system is stored in the positional relationship of the lenses. The shooting information stores information on the angle of view when the user instructs shooting, information on the longitude and latitude of the shooting location, information on the time of the shooting location, and information on the direction of the camera when shooting. . The pixel structure stores information on the number of vertical and horizontal pixels of the recorded image data. The image format stores information on whether or not the image is compressed and the type of compression.

  The reconstructed image signal E generated at the angle of view designated by the user is recorded in the image data portion in the recording format, and the image signals (A1, A2, A3, A4) captured by the optical system A are then recorded. To be recorded. Next, image signals (B1, B2, B3, B4) imaged by the optical system B are recorded, and then image signals (C1, C2, C3, C4) imaged by the optical system C are recorded. The image signals (D1, D2, D3, D4) picked up by the optical system D are recorded in In FIG. 12, the image signals (A3, A4, B1 to B4, C1 to C4, and D1 to D3 are not shown).

  The distance map (A) from the image signal captured by the optical system A is recorded in the distance map portion in the recording format, and thereafter the distance maps from the image signals captured by the optical systems B, C, and D, respectively. (B, C, D) is recorded.

  The recording unit 36 shown in FIG. 7 records the reconstructed image signal encoded by the recording encoding unit 35 on a recording medium such as a memory card (not shown). The communication encoding unit 37 encodes the reconstructed image signal output from the multiview image processing unit 26 into the communication format shown in FIG.

  As shown in FIG. 13, the communication format includes an image data portion and a distance map portion in addition to a header portion of a file in which shooting information is stored. In order to reduce the amount of data during communication, the image data section includes a reconstructed image signal E generated at an angle of view designated by the user, an image signal (A) imaged by the optical system A and corrected for the viewpoint, and an optical system Image signals (B, C, and D) that have been picked up by B, C, and D and whose viewpoints have been corrected are recorded. Accordingly, the positional relationship of the lens is deleted from the header portion of the recording format (FIG. 12), and the lens designation, lens information, shooting information, pixel structure, and image format are stored in the header portion. Yes.

  The communication unit 38 transmits the reconstructed image signal encoded by the communication encoding unit 37 to an external server through wireless communication such as WiFi.

  Next, other operations will be described. First, autofocus control will be described with reference to FIG. The AF / AE evaluation unit 29 evaluates the luminance component of the image signal from the image processing unit 25 and determines whether or not the subject is in focus. The determined result is transmitted to the system control unit 21. When the determination result is out-of-focus, the system control unit 21 transmits an instruction to move the focus group FG of the imaging unit 22 and the movement amount and movement direction to the drive control unit 31.

  The operation of the drive control unit will be described with reference to FIG. In accordance with an instruction from the system control unit 21, the drive control unit 31 controls the drive driver 32 to drive a motor (not shown) to move the focus group FG in the in-focus direction. The movement amount of the focus group FG is detected by the encoder 33, and the detected movement amount is fed back to the drive control unit 31. The drive control unit 31 moves the focus group FG until the movement amount received from the encoder 33 matches the movement amount instructed from the system control unit 21.

  The automatic exposure control will be described with reference to FIG. The AF / AE evaluation unit 29 evaluates the luminance component of the image signal from the image processing unit 25 and determines whether the exposure is appropriate. The determined result is transmitted to the system control unit 21. When the determination result is poor exposure, the system control unit 21 transmits the exposure correction amount of the imaging unit 22 to the exposure control unit 34.

  The operation of the exposure control unit 34 will be described with reference to FIG. The exposure control unit 34 controls the exposure period by controlling the exposure period of each image sensor 15 and the visible light transmittance of the light amount adjusting element 17 in accordance with an instruction from the system control unit 21. Here, the control pattern of the image sensor 15 and the light amount adjusting element 17 is instructed by the user operating the mode dial 8 or the AE dial 9 of the operation unit 30 to select a known AE program diagram (not shown). To do.

  The control of camera shake correction will be described with reference to FIG. The camera shake detection unit 40 includes a so-called gyro sensor, an acceleration sensor, and the like, detects the amount of camera shake and the direction of camera shake at the time of shooting, and transmits the detection result to the system control unit 21. The system control unit 21 instructs the image sensor 15 to change the area for reading the image signal of the image sensor 15 according to the transmitted camera shake amount and the direction of the camera shake. The amount of change in the area from which the imaging signal is read varies depending on the focal length of the optical system, and the amount of change increases as the focal length increases.

  The control of the external output will be described with reference to FIG. The output of the image to the outside may be instructed directly by the user from the operation unit 30 or may be handled according to the mode transition of the camera system. An instruction from the user is transmitted to the system control unit 21 and is transmitted from the system control unit 21 to the external output unit 39.

  When outputting the recorded image, the external output unit 39 reproduces the file selected by the user from the image recorded by the recording unit 36 and reads the reconstructed image signal. Then, it is converted into an output destination signal format such as a PC (personal computer) or an external monitor and output from the video output terminal. When the captured image is directly output, the external output unit 39 converts the reconstructed image signal from the multi-view image processing unit 26 into an output destination signal format and outputs it from the video output terminal.

  The control for displaying the reproduced image on the monitor 43 will be described with reference to FIG. The display of the image on the monitor 43 may be instructed directly by the user or may be handled according to the mode transition of the playback device system. A thumbnail image of the recorded image is displayed on the monitor 43, and the user can directly select the image by touching the monitor 43. The thumbnail image is an image having an angle of view designated by the user at the time of shooting. An instruction from the user is sent to the system control unit 21 and is sent from the system control unit 21 to the monitor image processing unit 27.

  The face detection unit 44 performs a known face detection process on the image signal from the image processing unit 25 to detect a human face area included in the captured image of the image sensor 15. The detected result is sent to the system control unit 21. As a known face detection process, for example, there is a method of extracting a skin color region from the gradation color of each pixel represented by an image signal and detecting the face with a matching degree with a face contour plate prepared in advance. . Also disclosed is a method of performing face detection by extracting facial feature points such as eyes, nose and mouth using a well-known pattern recognition technique. Note that the face detection processing technique is not limited to the technique described above, and various known techniques can be used.

  In the present embodiment, an example in which a range that can be focused later is displayed on the monitor 43 using subject detection (face detection by the face detection unit 44) is shown. That is, the focusable range in the reconstructed image is displayed on the monitor 43 during shooting together with the acquired subject distance.

  14 and 15 are diagrams illustrating an example of an output video to the monitor 43. FIG.

  As shown in FIGS. 14 and 15, during photographing, the system control unit 21 causes the monitor 43 to display a photographed image acquired by any one of the optical systems A to D. For example, a captured image acquired from an optical system A to D having an angle of view specified by the user is selectively displayed. Subjects 101 and 102 are subjects detected by face detection. The face of the subject 101 is detected in the subject detection frame 113, and the face of the subject 102 is detected in the subject detection frame 114. In contrast to FIG. 14, the distance of the subject 102 from the camera 100 is longer in FIG. 15.

  The system control unit 21 further controls the monitor 43 to display a focus bar group 103 indicating a focusable range that can be focused later (during reproduction) by superimposing the image on the image being shot. The focus bar group 103 includes four focus bars 104, 105, 106, and 107 arranged in parallel with the vertical direction as the longitudinal direction. The lengths of the focus bars 104, 105, 106, and 107 indicate the focusable ranges of the optical systems A, B, C, and D having different focal lengths. The length of each focus bar corresponds to a range where focusing can be performed later in the corresponding optical system.

  14 and 15, “F” is the far side and “N” is the near side when viewed from the camera 100. Therefore, the F side end position (upper position) of each of the focus bars 104 to 107 indicates the farthest focusable position, and the N side end position (lower end position) indicates the closest focusable position. Taking the focus bar 104 as an example, the vertical length indicates the focusable range 108 of the optical system A. Similarly, the vertical lengths of the focus bars 105, 106, and 107 indicate the focusable ranges of the optical systems B, C, and D.

  The system controller 21 further controls the monitor 43 to display the distance information displays 111 (111A, 111B, 111C, 111D) and 112 (112A, 112B, 112C, 112D) together with the focus bar group 103 ( Control means). The distance information displays 111 and 112 are information indicating the distance (subject distance) of each of the subjects 101 and 102 from the camera 100, and positions corresponding to the subject distances of the subjects 101 and 102 in the length direction of the focus bar group 103. Is displayed.

  The distance information displays 111 </ b> A to 111 </ b> D are displayed at positions (including the extension) of the focus bars 104 to 107 corresponding to the optical systems A to D, respectively, and all are in the length direction of the focus bars 104 to 107. They are displayed at the same position in the (vertical direction). The positions of the distance information displays 111 and 112 in the vertical direction indicate subject distances 109 and 110 from the camera 100 to the subject 101, respectively. The subject distances 109 and 110 are acquired by the distance calculation unit 28 during shooting using the above-described method for calculating the subject distance L.

  However, of the four distance information displays 111A to 111D and 112A to 112D, only one distance information display 111 and 112 corresponding to the optical system from which the currently displayed image being captured is acquired These three are displayed in different display modes. As an example, as shown in FIGS. 14 and 15, when a captured image with the optical system B as an acquisition source is displayed, only the distance information displays 111 </ b> B and 112 </ b> B corresponding to the focus bar 105 are displayed in a dark color. . The distance information displays 111A, 111C, 111D and the distance information displays 112A, 112C, 112D other than these are displayed in a light color. Thereby, the optical system from which the captured image is acquired is notified.

  Referring to FIG. 14, the distance information displays 111 and 112 are within the full length range of the focus bars 104 to 107. From these, it is understood that the user can focus on both the subject 101 and the subject 102 after photographing at any angle of view of the optical systems A to D.

  On the other hand, referring to FIG. 15, the distance information displays 111 and 112 are within the total length of the focus bars 105 to 107, but only the distance information display 111 is within the focus bar 104, The distance information display 112 is deviated upward. That is, the distance information display 112 is out of the focusable range of the optical system A. From these, the user can focus on the subjects 101 and 102 after shooting at the angles of view of the optical systems B, C, and D. However, at the angle of view of the optical system A, the user can focus on the subject 102 after shooting. It can be seen that the focus cannot be adjusted appropriately.

  In this way, the user can recognize whether or not the plurality of detected subjects can be focused later for each of the plurality of optical systems A to D. The same can be considered when the number of subjects is three or more, and the distance information display may be increased corresponding to the number of subjects.

  According to the present embodiment, the focusable range in the reconstructed image generated from the image being shot is displayed on the monitor during shooting together with the subject distance. It can be recognized during shooting together with the relationship with the subject distance. As a result, it is possible to save troubles such as re-shooting without focusing on the intended subject after shooting.

  In addition, since the focus bars 104 to 107 are displayed for each of a plurality of optical systems (A to D) having different focal lengths, it is possible to know a focusable range (at each angle of view) for each optical system. . Moreover, since the length of the focus bar corresponds to a focusable range, it is easy to visually grasp the focusable range.

  In addition, since the distance information displays 111 and 112 are displayed for each subject, the user can determine whether or not each subject can be focused. Moreover, since the distance information displays 111 and 112 are displayed at positions corresponding to the subject distance in the range along the length direction of the focus bars 104 to 107, from the relative positional relationship between the focus bar and the distance information display, It is easy to visually recognize whether or not each subject can be focused.

  Further, since the optical system from which the captured image currently displayed on the monitor is acquired is notified based on the display color density of the distance information displays 111 and 112, the focusable range for the angle of view displayed on the monitor is recognized. Easy to do.

(Second Embodiment)
In the first embodiment, the subject distance is displayed for each subject whose face is detected in the monitor display. On the other hand, in the second embodiment of the present invention, the subject distance is acquired for each region obtained by dividing the captured image being captured into a plurality of regions, and the subject distance is displayed for each divided region. Other configurations are the same as those of the first embodiment. Therefore, the second embodiment will be described with reference to FIGS. 16 and 17 instead of FIGS.

  16 and 17 are diagrams illustrating an example of an output video to the monitor 43. FIG.

  In the first embodiment, the subject distance is calculated with respect to the region where the subject is detected. In the second embodiment, the distance calculation unit 28 calculates the subject distance for each divided region. Regions 213 and 214 are regions in which the subjects 101 and 102 are shown when the screen is divided. The method for dividing the region is not limited.

  In the example shown in FIGS. 16 and 17, the object distance is calculated only for the areas 213 and 214 in which the object is shown for simplification of description, and the distance information displays 111 and 112 are used as the distance information displays 111 and 112. An example to be displayed together is shown. The same applies to subjects in other divided areas. The focus bar group 103 displayed on the monitor 43 is the same as that of the first embodiment.

  In FIG. 16, the subject 101 in the region 213 and the subject 102 in the region 214 are displayed as distance information displays 111 and 112 in the focus bar group 103. From the relative positional relationship between the distance information displays 111 and 112 and the focus bar group 103, the focusable range can be visually grasped for each optical system and for each subject. It is the same as 15).

  According to the present embodiment, the same effect as in the first embodiment can be achieved with respect to recognizing a focusable range after shooting together with the relationship with the subject distance during shooting. In particular, in the present embodiment, the user can recognize whether or not the subject in each divided area in the screen can be focused later at each angle of view. Further, it is possible to notify the focusable range to a subject other than a person who cannot detect the subject.

  In the first and second embodiments, the focusable range after shooting is displayed on the monitor 43 so that at least the closest focusable position and the farthest focusable position are known. Should be adopted. Therefore, the display mode is not limited to the focus bar group 103. When the bar format is not used in this way, the distance information display may be displayed at a position that is relative to the display of the closest focusable position and the farthest focusable position. Good.

  In addition, as an aspect of notifying the optical system that is the acquisition source of the captured image currently displayed on the monitor, the distance information displays 111 and 112 are distinguished by the color intensity. However, the present invention is not limited to this, and the display mode of the distance information display may be distinguished between the acquisition source optical system and the other optical system. For example, you may distinguish with a display color and the mark to add.

  Further, it is not always necessary to display the distance information display (for example, 111A to 111D) corresponding to all the optical systems, only the distance information display (for example, 111A) corresponding to the optical system of the acquisition source is displayed, and otherwise ( For example, a method of not displaying 111B to D) is also conceivable. Alternatively, the distance information display may be abolished, and only the focus bar corresponding to the acquisition source optical system may have a display mode different in color or the like from the other focus bars. The setting of the longitudinal direction of the focus bar group 103 is not limited to the vertical direction.

  From the viewpoint of generating a reconstructed image from which an in-focus position can be changed after shooting from the image being shot, a configuration in which a lens array having a plurality of lenses arranged in the plane direction is arranged on the subject side of the image sensor 15 is adopted. May be.

  Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included.

A, B, C, D Optical system 21 System control unit 26 Multi-view image processing unit 28 Distance calculation unit 43 Monitor 103 Focus bar group 111, 112 Distance information display

Claims (8)

A plurality of optical systems having different focal lengths, each including an image sensor;
Obtaining means for obtaining a subject distance of a subject in an image output from the image sensor;
A range of the subject distance obtained based on the information about the plurality of optical systems, the generated after shooting using an image output from the image sensor, image Zogago different focus positions from the image Control means for displaying a bar having a length corresponding to a range that can be taken as a focal position on a monitor for each optical system having a different focal length together with the subject distance acquired by the acquisition means;
An imaging device comprising: The imaging apparatus according to claim 1, further comprising a generation unit configured to generate an image at a focus position different from the image output from the image sensor. The control means is acquired by the bar having a length corresponding to the range and the acquisition means when images are sequentially taken by the plurality of optical systems and images from the image sensor are sequentially displayed on the monitor. The imaging apparatus according to claim 1, wherein the subject distance is displayed on the monitor. Having a detecting means for detecting a human face included in the image being shot;
The imaging apparatus according to claim 1, wherein the acquisition unit acquires a face distance detected by the detection unit as the subject distance. The acquisition means can acquire subject distances of a plurality of subjects,
5. The imaging apparatus according to claim 1, wherein the control unit displays the acquired subject distances of the plurality of subjects on the monitor. 6. Wherein, in said monitor, the information indicating the object distance, along the length of the bar, any of claims 1 to 5, characterized in that to display in a position corresponding to the subject distance 1 The imaging device according to item . The monitor selectively displays a captured image acquired by one of the plurality of optical systems having different focal lengths,
It said control means, the image pickup apparatus according to any one of claims 1 to 6, characterized in that announcing optical system which is an acquisition source of the captured image currently displayed on the monitor. An acquisition step of acquiring subject distances of subjects in an image output from the imaging elements of a plurality of optical systems having different focal lengths, each including an imaging element;
A range of the subject distance obtained based on the information about the plurality of optical systems, the generated after shooting using an image output from the image sensor, image Zogago different focus positions from the image A control step of displaying a bar having a length corresponding to a range that can be taken as a focal position on a monitor for each optical system having a different focal length together with the subject distance acquired by the acquisition step;
A method for controlling an imaging apparatus, comprising:

Images