JP2015103879A - Image processing apparatus and control method thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus which allow the photographer to confirm the effect of image processing by a plurality of images in real time.SOLUTION: An image processing apparatus (1) includes a first imaging unit (11) and a second imaging unit (21) having optical axes different from each other, image processing means (30) performing image processing for the image obtained from at least one of the first imaging unit and second imaging unit, display means (50) for displaying the image subjected to image processing by the image processing means, and recording means (60) for recording the image subjected to image processing by the image processing means. The image processing means performs first image processing when displaying an image in the display means, and performs second image processing of longer processing time than that of the first image processing when recording an image in the recording means.

Description

  The present invention relates to an imaging apparatus having a plurality of imaging systems, and more particularly to image processing of a compound-eye imaging apparatus.

  Patent Document 1 discloses an imaging apparatus that acquires a plurality of images with a plurality of imaging systems and processes the images by image processing.

  Patent Document 2 discloses an imaging apparatus that adds blur to an image by image processing. Here, Patent Document 2 discloses a form in which an image to which image processing for adding blur is added is displayed on an electronic viewfinder corresponding to a camera monitor.

JP 2007-288799 A JP 2008-015754 A

  However, in Patent Document 1, image processing is performed at the time of recording an image, and since the photographed image is displayed on the camera monitor, the photographer cannot confirm the effect of the image processing. . In Patent Document 2, the photographer can check the result of the image processing to be added on the monitor. However, it takes time to perform complicated image processing, and the subject image is displayed on the monitor in real time. It was difficult. As described above, in the conventional technology disclosed in the above-described patent document, the photographer cannot confirm the effect of the image processing in real time.

  In view of the above problems, an object of the present invention is to provide an imaging apparatus and a control method for the imaging apparatus in which a photographer can confirm the effect of image processing using a plurality of images in real time.

  An imaging apparatus according to an aspect of the present invention is obtained from at least one of a first imaging unit and a second imaging unit having different optical axes, and the first imaging unit and the second imaging unit. Image processing means for performing image processing on the image, display means for displaying the image processed by the image processing means, and recording means for recording the image processed by the image processing means, The image processing means performs first image processing when displaying an image on the display means, and has a processing time longer than the processing time of the first image processing when recording an image on the recording means. 2 image processing is performed.

  According to the present invention, a photographer can confirm the effect of image processing using a plurality of images in real time.

It is a block diagram of the imaging device concerning the Example of this invention. It is a figure explaining the imaging operation concerning the Example of this invention. It is a figure of an example of the picked-up image and distance information map concerning the Example of this invention. It is a figure showing the area | region which integrated the block based on a distance information map in 1st image processing. (Example 1) It is a figure explaining the 1st image processing in each field which integrated a block. (Example 1) It is a figure of the image obtained by 1st image processing. (Example 1) It is a figure which shows the area | region which performs the electronic zoom concerning the Example of this invention. (Example 2) It is a figure showing the area | region by integrating the block based on a distance information map in 1st image processing. (Example 2) It is a figure explaining the 1st image processing in each field which integrated a block. (Example 2) It is a figure of the image obtained by 1st image processing. (Example 2)

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a block diagram of an imaging apparatus 1 in an embodiment of the present invention. As shown in FIG. 1, the imaging device 1 in the embodiment of the present invention is a compound eye imaging device including a first imaging unit 11 and a second imaging unit 21. In the drawing, the upper part shows the first photographing unit 11, and the lower part shows the second photographing unit 21. The first photographing unit 11 and the second photographing unit 21 have optical axes different from each other, and are provided at positions where a part of a subject area that is common to each other can be imaged.

  The first photographing unit 11 includes a first photographing lens 12, a first imaging element 13, a first imaging circuit 14, a first lens driving device 15, a first memory 16, and the like. Similar to the first imaging unit 11, the second imaging unit 21 includes a second imaging lens 22, a second imaging element 23, a second imaging circuit 24, a second lens driving device 25, and a second lens driving device 25. A memory 26 and the like are included. In the present embodiment, the first photographing lens 12 and the second photographing lens 22 are arranged so that their optical axes are substantially parallel to each other. Here, “substantially parallel” means that the case is completely parallel and the case where it is deviated from the case of being completely parallel within the allowable error range. The same applies to the following embodiments.

  The subject images formed on the light receiving surfaces of the image sensors 13 and 23 via the imaging lenses (imaging optical systems) 12 and 22 are photoelectrically converted by the image sensors 13 and 23 and sequentially read out as video signals (image signals). Video signals read from the image sensors 13 and 23 are A / D converted, converted into R, G, and B dot-sequential digital signals and output.

  The control circuit 30 performs overall control of the entire imaging apparatus 1 and controls the corresponding circuit based on various input signals input from the shutter 40, the mode selection unit 41, and the operation unit 42. Here, the mode selection unit 41 determines the shooting conditions (automatic exposure, aperture priority, shutter priority, etc.) and image processing of the captured image (image processing based on subject distance, tone conversion, monochrome conversion, etc.) for the user (photographer). Is a member for selection. That is, the control circuit 30 sets the aperture and shutter value of the imaging system, performs exposure control, and sets the white balance. The control circuit 30 controls the lens driving devices 15 and 25 to set the focus positions of the photographing lenses 12 and 22. The control circuit 30 is also used as an image processing circuit, and will be described later using the work area of the memory 33 based on the image signals output from the imaging circuits 14 and 24 based on the signals from the mode selection unit 41 and the operation unit 42. Image processing based on the subject distance to be performed is performed. In other words, the control circuit 30 functions as an image processing unit that performs image processing on an image obtained from at least one of the first photographing unit 11 and the second photographing unit 21.

  The distance information acquisition circuit 31 detects parallax (pixel shift) on the screen from the two video signals converted into digital signals by feature point correspondence processing or the like. Further, the distance information map is created by calculating the distance of the subject from the width (base line length) of the optical axes of the first photographing unit 11 and the second photographing unit 21. In other words, the distance information acquisition circuit 31 functions as a calculation unit that calculates the subject distance based on two image signals obtained from the first photographing unit 11 and the second photographing unit 21. Also, it functions as a generating unit that generates a distance information map based on the calculated subject distance.

  Image processing based on subject distance is the processing of an image captured through a taking lens by electrical processing, but the effect of image processing can be confirmed not only when recording an image but also before recording. Good. For this purpose, two methods are prepared for the image processing based on the subject distance: the first image processing with a short calculation time by simple image processing and the second image processing with a long calculation time by strict image processing. In the present invention, as will be described later, the first image processing is performed when an image is displayed on the liquid crystal monitor 50, and the processing time is longer than the processing time of the first image processing when the image is recorded in the external storage device 60. Second image processing with a long time is performed.

  The image reproducing device 32 supplies the image data processed based on the distance information map to the liquid crystal monitor 50 which is an image display means after the first image processing in the control circuit (image processing circuit) 30. In this way, an image captured by the image sensor is displayed on the screen of the liquid crystal monitor 50. In other words, the liquid crystal monitor 50 functions as a display unit that displays the first image processed image.

  Before receiving a shooting start signal generated by pressing the shutter 40 or the like, a preview image is displayed on the liquid crystal monitor 50 (for example, the first image processing has been performed). Screen freezes. In response to the reception of the imaging start signal, the image signals read from the imaging circuits 14 and 24 are subjected to second image processing by the control circuit (image processing circuit) 30 and then subjected to compression processing as necessary. Then, it is recorded in the external storage device 60 which is a recording means (recording medium). In other words, the external storage device 60 functions as a recording unit that records the image subjected to the second image processing. The control circuit 30 functions as a determination unit that determines the state of the shutter 40 (operation unit).

  Note that the form of the recording medium is not limited to the detachable external storage device 60, but may be an internal memory built in the imaging device 1. In addition, the shooting start signal may be applied from the outside of the imaging apparatus 1 by an operation unit having a shutter function such as a remote controller or an externally connected device. Furthermore, the screen of the liquid crystal monitor 50 is a touch panel, and may be emitted by the photographer's touch on the touch panel.

  When the above recording process is completed, the screen freeze is released and the display returns to the preview image display.

  The synchronization signal generation circuit 34 provides a synchronization signal for signal processing to the first imaging unit 11 and the second imaging unit 21.

  Next, image processing based on subject distance (first image processing and second image processing) will be described.

  The image processing based on the subject distance is image processing for processing an image according to the distance information of the distance information map for the video signal from the first photographing unit 11. This is particularly effective for image processing in which the specifications (focal length, F number), etc., of the photographing lens in the photographing unit can be virtually set. A mode for performing such image processing is an optical parameter expansion mode. The optical parameter expansion mode can be selected by the mode selection unit 41 that can select the imaging condition of the imaging device 1.

  The image processing in the present embodiment is image processing in which a blur effect corresponding to subject distance information is added to an image in accordance with an F number instruction from the operation unit 42.

  Next, the flow of image processing control based on the subject distance will be described with reference to the flowchart of FIG. The flowchart of FIG. 2 is executed based on a command from the control circuit 30.

  First, when power is supplied to the image pickup apparatus 1, the respective photographing units 11 and 21 are driven and controlled so that video signals from the image pickup elements 13 and 23 can be acquired (not shown).

  In S <b> 201, a first video signal of an image from the first imaging lens 12 is acquired from the first image sensor 13.

  In S <b> 202, the acquired first video signal is stored in the first memory 16.

  In S <b> 203, a second video signal of an image obtained by the second photographing lens 22 is acquired from the second image sensor 23.

  In S <b> 204, the acquired second video signal is stored in the second memory 26.

  In S205, the control circuit 30 determines whether or not the mode selection unit 41 is in the optical parameter expansion mode.

  If the optical parameter expansion mode is not set in S205 (No in S205), the process shifts to the normal shooting mode in S206, and the following image processing is not executed.

  If the optical parameter expansion mode is set in S205 (Yes in S205), the process proceeds to S207.

  In S207, F number information set by the photographer is acquired from the operation unit 42 as operation unit information. Further, the F number of the photographing lenses 12 and 22 of each photographing unit is changed to an F number that increases the depth of field.

  In S208, the first video signal is read from the first memory 16.

  In S209, the second video signal is read from the second memory 26.

  In S210, the control circuit 30 determines whether or not the shutter 40 has been cut.

  If the shutter 40 is not cut in S210 (No in S210), the process proceeds to S211 which is a flow for performing the first image processing.

  In S210, when the shutter 40 is off (Yes in S210), the process proceeds to S216, which is a flow for performing the second image processing.

  First, a flow for performing the first image processing will be described.

  In S211, the control circuit 30 sets a block of a predetermined area on the screen. Here, it is assumed that the block in the predetermined area has an area larger than at least the pixels of the image.

  Then, the distance information acquisition circuit 31 calculates distance information of the representative pixel (for example, the center pixel) of the block from the two video signals, and creates a distance information map.

  In S212, the control circuit 30 integrates blocks from the distance information for each block.

  In S213, the control circuit 30 performs the first image processing with blur added according to the distance information for each integrated block, and the video signal as a result of executing the image processing for all the regions (integrated block regions) is stored in the memory 33. To store.

  In S214, the video signal after the first image processing is read from the memory 33.

  In S215, the read video signal is displayed on the liquid crystal monitor 50, and the process returns to S201.

  Next, a flow for performing the second image processing will be described.

  In S216, the distance information of the pixel of the video signal from the first photographing unit 11 is calculated from the two video signals in the distance information acquisition circuit 31, and a distance information map is created.

  In S <b> 217, the control circuit 30 performs the second image processing with blurring according to the pixel distance information using the memory 33, and stores the video signal as a result of executing the image processing for all the pixels in the memory 33.

  In S218, the video signal after the second image processing is read from the memory 33.

  In S219, the read video signal is recorded in the external storage device 60, and the process returns to S201.

  Here, the first image processing will be described in detail below.

  FIG. 3A is an example of an image obtained from the first photographing unit 11, and FIG. 3B is a distance information map obtained from two images obtained from the first and second photographing units 11 and 21. FIG. FIG.

  In FIG. 3B, the distance from the focused subject is 0, and the direction approaching the imaging apparatus 1 is positive, and the direction away from the imaging apparatus 1 is negative. The unit is m, and the distance corresponding to infinity is represented by *. The subject distance information shown in FIG. 3B is obtained for each block obtained by dividing the screen into appropriate blocks. In FIG. 3B, the number of blocks is small for the sake of explanation, but it is actually desirable to divide into more blocks (for example, about 1/100 to 1/1000 of the sides of the imaging area). Here, it is assumed that the block having the divided predetermined area has an area larger than at least the pixels of the image.

  FIG. 4 shows a state of a divided area composed of blocks integrated based on the above-described distance information (that is, blocks separated by bold lines in FIG. 3B). Here, the area within a predetermined distance from the object being focused is divided into an in-focus area I and the other area is divided into an out-focus area. Further, the out-of-focus area is divided into four areas O1, O2, O3, and O4 according to the distance of each subject.

  Table 1 summarizes each distance range.

  Δ is the distance from the subject in focus. Δ ′ is distance information used when image processing is actually performed. Here, the distance information Δ ′ actually used when image processing is performed is a distance representing each of the regions I, O1, O2, O3, and O4. The distance representing each distance may be an average value calculated for each region or may be a preset representative distance.

  A blur function corresponding to the distance (Δ ′) representing the region and the F number from the operation unit 42 is convolved with respect to the image of the region O4 farthest from the imaging device 1. This is shown in FIG. 5 (a). The image thus obtained is recorded in the memory 33.

  Next, a blur function corresponding to the distance (Δ ′) representing the region and the F number from the operation unit 42 is convolved with respect to the image of the region O3 farthest from the imaging device 1 next to the region O4. This is shown in FIG. 5 (b). The image thus obtained is overwritten and recorded in the memory 33.

  Next, the image of the region I that is the farthest from the image capturing device 1 next to the region O3 is an in-focus region and is not processed. This is shown in FIG. 5 (c). This is overwritten in the memory 33 and recorded.

  Next, a blur function corresponding to the distance (Δ ′) representing the region and the F number from the operation unit 42 is convolved with respect to the image of the region O2 farthest from the imaging device 1 next to the region I. This is shown in FIG. 5 (d). The image thus obtained is overwritten and recorded in the memory 33.

  Next, a blur function corresponding to the distance (Δ ′) representing the region and the F number from the operation unit 42 is convolved with respect to the image of the region O1 farthest from the imaging device 1 next to the region O2. This is shown in FIG. 5 (e). The image thus obtained is overwritten and recorded in the memory 33.

  The image reproduction device 32 reads the image obtained by these image processings from the memory 33 and displays it on the liquid crystal monitor 50. This is shown in FIG.

  Next, the second image processing will be described in detail below.

  In the second image processing, distance information is obtained for each pixel. The sign of the subject distance is the same as in the first image processing.

  Here, as in the first image processing, the blur function corresponding to the distance and the F number from the operation unit 42 is convolved.

  In the second image processing, blur addition processing by convolution is sequentially performed for each pixel in the screen from an area farthest from the imaging device 1 and sequentially overwritten and recorded in the memory 33.

  After this image processing is executed for all pixels, it is recorded in the external storage device 60.

  In this way, image processing is simplified when displaying on a liquid crystal monitor during shooting, and an image can be displayed in real time while showing the effect instructed by the photographer. Execute image processing and add beautiful blur.

  In the first image processing, the area of the image is limited to five areas, and the same blur function is simply convolved in the four areas. Thus, the first image processing is more precise than the second image processing. However, the processing time can be greatly shortened. As a result, the effect of complicated image processing such as adding blur can be confirmed in real time on a liquid crystal monitor, and an image intended by the photographer can be appropriately set and recorded.

  In the above description, the distance information map is created with the actual distance from the subject in which the distance information is in focus, but the distance information map is created with the reciprocal of the actual distance, and the image is displayed according to the inverse of the actual distance. The method of processing may be used.

  In addition, when the image is recorded, a better image can be obtained by adjusting the apertures of the photographing lenses 12 and 22 and acquiring the image in the pan focus state and adding the blur calculated as the image size of 35 mm full size. can get.

  Alternatively, the recorded video signal may be compressed and displayed on the liquid crystal monitor 50 after recording.

  As a result, when an F-number brighter than the F-number of the photographing lenses 12 and 22 provided in the image pickup apparatus 1 is designated, the image is given a blurring effect corresponding to the designated F-number and optically blurred. It is possible to take a picture while checking the effect on the liquid crystal monitor 50.

  Hereinafter, a second embodiment of the present invention that effectively performs electronic zooming by image processing will be described.

  The configuration of the image pickup apparatus in the second embodiment is the same as that of the first embodiment shown in FIG.

  In the present embodiment, as shown in FIG. 7, a part of the acquired image (area surrounded by a dotted line) is enlarged to perform virtual zooming. The image processing in the present embodiment is image enlargement processing in which an enlargement process is performed on an image according to subject distance information in accordance with an instruction of a focal length from the operation unit 42.

  Hereinafter, the first image processing in the second embodiment will be described.

  The distance information map is created in the same manner as in the first embodiment, and the distance information obtained for each block obtained by dividing the screen into appropriate blocks is used to define a block integrated region as in FIG. .

  Here, the screen is divided into five regions P1, P2, P3, P4, and P5 according to the distance from the subject that is in focus. The image of the region P5 farthest from the imaging device 1 is enlarged according to the distance representing the region and the focal length from the operation unit 42, and data between the pixels is generated by linear interpolation. . At this time, the enlargement process does not simply enlarge the image, but the image is created in consideration of the vertical magnification according to the focal length, so that the effect of increasing the focal length of the lens can be added. This is shown in FIG. 9 (a). The image thus obtained is recorded in the memory 33. At this time, the image that protrudes from the necessary image size is ignored.

  Next, with respect to the image of the region P4 farthest after the region P5 from the imaging device 1, the image is enlarged according to the distance representing the region and the focal length from the operation unit 42, and data between the pixels Is generated by linear interpolation. This is shown in FIG. 9 (b). The image thus obtained is recorded in the memory 33. At this time, the image that protrudes from the necessary image size is ignored.

  Next, with respect to the image of the region P3 farthest after the region P4 from the imaging device 1, the image is enlarged according to the distance representing the region and the focal length from the operation unit 42, and data between the pixels Is generated by linear interpolation. This is shown in FIG. 9 (c). The image thus obtained is recorded in the memory 33. At this time, the image that protrudes from the necessary image size is ignored.

  Next, with respect to the image of the region P2 farthest after the region P3 from the imaging device 1, the image is enlarged according to the distance representing the region and the focal length from the operation unit 42, and data between the pixels Is generated by linear interpolation. This is shown in FIG. 9 (d). The image thus obtained is recorded in the memory 33. At this time, the image that protrudes from the necessary image size is ignored.

  Next, with respect to the image of the region P1 farthest after the region P2 from the imaging device 1, the image is enlarged according to the distance representing the region and the focal length from the operation unit 42, and data between the pixels Is generated by linear interpolation. This is shown in FIG. 9 (e). The image thus obtained is recorded in the memory 33. At this time, the image that protrudes from the necessary image size is ignored.

  An image obtained by this image processing is shown in FIG. The image reproducing device 32 reads out the video signal of the image from the memory 33 and displays it on the liquid crystal monitor 50.

  Next, the second image processing will be described.

  In the second image processing, distance information is obtained for each pixel. Also in the second image processing, the image is enlarged according to the distance and the focal length from the operation unit 42 as in the first image processing.

  In the second image processing, each pixel in the screen is enlarged in order from the area farthest from the imaging device 1 and sequentially overwritten in the memory 33. Further, the enlarged pixels are interpolated more smoothly than the first image processing by a method such as bicubic interpolation, and the video signal of the image obtained as a result of the final image processing is recorded in the external storage device 60.

  In this way, image processing is simplified when displaying on a liquid crystal monitor during shooting, and an image can be displayed in real time while showing the effect instructed by the photographer. Performs image processing and adds a sense of perspective that is magnified by a lens.

  In this description, the map is created with the actual distance from the subject whose distance information is in focus, but a map may be created with the inverse of the actual distance, and image processing may be performed according to the inverse of the actual distance.

  As a result, when a focal length longer than the focal length on the telephoto side of the photographic lens provided in the imaging apparatus is designated, the same effect as that obtained by optically enlarging the image with a burst effect corresponding to the designated focal length is provided. It becomes possible to take a picture while confirming on the LCD monitor.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  The compound eye imaging device of the present invention can be suitably used for an optical device such as a video camera or a compact digital camera.

First shooting unit 11
Second shooting unit 21
Control circuit 30
LCD monitor 50
External storage device 60

Claims (13)

A first photographing unit and a second photographing unit having different optical axes;
Image processing means for performing image processing on an image obtained from at least one of the first photographing unit and the second photographing unit;
Display means for displaying the image processed by the image processing means;
Recording means for recording the image processed by the image processing means,
The image processing means performs first image processing when displaying an image on the display means, and has a processing time longer than the processing time of the first image processing when recording an image on the recording means. 2 image processing,
An imaging apparatus characterized by that. Operating means having a shutter function;
Discriminating means for discriminating the state of the operating means,
The imaging apparatus according to claim 1, wherein the image processing unit performs the first image processing or the second image processing based on a state of the operation unit.   The image processing means performs the first image processing when the determining means determines that the operating means is not operated, and when the determining means determines that the operating means is operated, The imaging apparatus according to claim 2, wherein second image processing is performed. A calculation unit that calculates a subject distance based on two image signals obtained from the first imaging unit and the second imaging unit;
4. The imaging apparatus according to claim 1, wherein the first image processing and the second image processing perform image processing based on the subject distance. 5.   In the first image processing, the obtained image is divided into blocks each having a predetermined area, the subject distance is calculated for each of the divided blocks, and the subject distance calculated for each block is calculated. The image pickup apparatus according to claim 4, wherein the blocks are integrated.   The imaging apparatus according to claim 5, wherein the predetermined area is an area larger than a pixel of the obtained image.   The imaging apparatus according to claim 5, wherein the calculation unit calculates a subject distance of a representative pixel of the block.   The imaging apparatus according to claim 4, wherein the second image processing calculates the subject distance for each pixel of the obtained image.   The imaging according to any one of claims 4 to 8, wherein the first image processing and the second image processing are processing for adding a blur effect corresponding to the subject distance to an image. apparatus.   The imaging apparatus according to claim 5, wherein the first image processing performs a process of adding a blur effect corresponding to the subject distance for each of the integrated blocks.   The imaging apparatus according to claim 4, wherein the first image processing and the second image processing are image enlargement processing based on the subject distance and focal length.   The imaging apparatus according to claim 5, wherein the first image processing performs image enlargement processing based on the subject distance and focal length for each integrated block. A method of controlling an imaging apparatus including a first imaging unit and a second imaging unit having different optical axes,
An image processing step of performing image processing on an image obtained from at least one of the first photographing unit and the second photographing unit;
A display step for displaying the image processed by the image processing step;
A recording step of recording the image processed by the image processing step,
The image processing step performs first image processing when displaying an image in the display step, and processing time longer than processing time of the first image processing when recording an image in the recording step. 2 image processing,
And a method of controlling the imaging apparatus.